Gold custodian vs self-directed IRA for a decent portfolio?

• •But now I'm wondering if I'm leaving flexibility on the table with a self-directed IRA.
• •The custodian I'm with is fine, but sometimes I feel a bit limited in terms of what I can hold beyond the absolute basics.
• •Or have you always used one?

I've been thinking a lot lately about the options for my gold IRA, especially as my portfolio has grown to a pretty respectable size (sitting comfortably in the upper half of the $500k-$1M range these days). I started with a pretty standard setup through a custodian a few years back, mainly because it felt straightforward and honestly, I didn't want to mess anything up. But now I'm wondering if I'm leaving flexibility on the table with a self-directed IRA.

My main reason for holding a good chunk of gold, besides it just being a smart financial move given some of the craziness in the world, is diversification. As a doctor here in Boston, my primary income is pretty steady, and my other investments are doing well, but having that gold base gives me some peace of mind. The custodian I'm with is fine, but sometimes I feel a bit limited in terms of what I can hold beyond the absolute basics. I've been eyeing some of the more niche precious metals and even a few specific types of gold coins that my current custodian doesn't make super easy to acquire through them.

For those of you who have a significant portion of your portfolio in gold, especially if you're in a similar bracket, did you ever make the switch to a self-directed gold IRA? Or have you always used one? I'm curious about the pros and cons you've experienced. Specifically, what were the major headaches, if any, of going the self-directed route? And conversely, for those sticking with a custodian, what are the big benefits that keep you there? I'm trying to weigh the potential for greater control against the added administrative burden.

Any insights, especially from folks who've navigated this with a portfolio of this size, would be greatly appreciated. I'm trying to figure out if it's worth the extra legwork to gain that flexibility, or if I'm just overthinking it and the custodian is perfectly adequate for long-term hold.