Custodian fees for my Gold IRA - what am I missing?

• •Okay, so I'm really trying to get a handle on all the costs for my Gold IRA.
• •I know it's not a huge portfolio yet, but every dollar obviously counts, especially with these fees stacking up year after year.
• •I've been looking at my statements, and the custodian fees seem significant.

Okay, so I'm really trying to get a handle on all the costs for my Gold IRA. I'm just starting out, only have about $20k in there right now (slowly building it up!), and I'm a young professional here in Charleston, SC, and want to get my retirement planning done right from the jump. I know it's not a huge portfolio yet, but every dollar obviously counts, especially with these fees stacking up year after year.

I've been looking at my statements, and the custodian fees seem significant. I'm currently with X company (won't name them yet, don't want to bias anyone), and their annual custodian fee is around $150. I know there are also storage fees on top of that, but I'm trying to deal with one thing at a time. I've poked around a few other companies' websites, and some are quoting similar, while others are a bit lower, like $100-$125. Is there a big difference in what these custodians actually do to warrant those variations?

This is probably a dumb question, but are there any hidden fees related to custodians that I should be aware of? Like, I've heard some talk about transaction fees if you want to rebalance or anything, but is that usually separate from the annual custodian fee, or sometimes wrapped up into it? My main goal is to minimize these recurring costs without sacrificing security, obviously. It stresses me out a bit because when you're just starting, that $150 feels like a significant chunk of my gains.

For those of you with more experience, especially if you have portfolios similar in size to mine (or perhaps even larger), what kind of custodian fees are you paying? Have you ever switched custodians purely to save on fees? Was it a massive headache? Any advice on how to effectively compare these costs across different providers would be hugely appreciated!