It’s pretty simple as a 401(k) plan sponsor, you have a fiduciary duty to only pay reasonable plan expenses.
When terminating your third-party administrator (TPA), you need to identify the costs of de-converting the plan to the next TPA. Hopefully, those costs are known to you when you signed that original TPA contract. If not, understand the costs and how it bears to the annual fee that you have been paying. While a de-conversion/termination fee is an acceptable fee in the 401(k) plan business, it should bear some reason for the annual TPA fee. De-converting shouldn’t be more than the annual fee, it shouldn’t be half the fee. A month or two, maybe. As a plan sponsor and fiduciary, you can’t pay unreasonable plan expenses, you’d be breaching your fiduciary duty if you did.