IRA Rollover & Taxes - Anyone had a smooth run with Gold/Silver?

• •I've been kicking around the idea of rolling over a significant chunk of my old 401k into a Precious Metals IRA.
• •We're talking somewhere in the neighborhood of $750k-$1M that's just sitting there, feeling a bit vulnerable in the current market.
• •I got out of active trading on Wall Street a few years back, and frankly, the volatility is doing less for my blood pressure now than it used to.

I've been kicking around the idea of rolling over a significant chunk of my old 401k into a Precious Metals IRA. We're talking somewhere in the neighborhood of $750k-$1M that's just sitting there, feeling a bit vulnerable in the current market. I got out of active trading on Wall Street a few years back, and frankly, the volatility is doing less for my blood pressure now than it used to. My portfolio's already heavy in metals, so this just feels like a natural progression to diversify further away from paper.

My main hang-up, of course, is the tax implications. I’ve done a fair amount of research, but it always feels like there's some hidden snag with the IRS, especially when you're dealing with larger sums and physical assets. I'm in NYC, so state taxes are always a fun added layer of complexity. I’ve heard horror stories about folks getting hit with unexpected penalties or improper classifications on their rollovers, even when they thought they were doing everything by the book.

Has anyone here, especially those with a substantial metal allocation, gone through a direct rollover from a traditional IRA/401k into a Gold/Silver IRA without any major tax headaches? What was your experience like? Did you use a particular custodian or precious metals dealer that made the process seamless tax-wise? Any specific forms or procedures you found particularly crucial to get right? I'm trying to avoid any nasty surprises down the line.

I’m looking for practical advice from those who've actually navigated this. Not interested in hypotheticals, just real-world experience. Tell me your war stories, good or bad!