I have to agree with Ned. I've never shot or seen a. 5" group shot at 100 yards shoot 4.5" groups at 300 yards.
Well...something caused it. The phenomenon you're describing typically doesn't happen because of the load. It just doesn't. What you're describing suggests a change in the flight path of the bullet at some point after 100 yd. That is not likely to be caused by the load itself. It could be a component defect, but even that would be rarity. Loading the 185s would answer that question.