Do you use a concentricity gauge on your ammo. Last time I had an accuracy issue with 75+ i found that the bullets didn't fit my seating die well and concentricity was all over the place. If i sorted my ammo after; i found sub 1" groups at 209y, indicating this was a key loading component.
Barrel nut torque? Mil spec is 30-80ftlbs; most accurate assemblers that i know torque to 30 and increase until barrel nut clears the gas port with no gas tube contact. Or use a WOA like barrel nut that doesn't contact gas tube no matter what.
Facing off the receiver can help too; and I'll add that I've never found a barrel to shoot all heavies equally, Nosler, Sierra, Hornady, Berger try them all, the barrel will choose.
24.0gr Varget is 2650-2700fps from a 20" barrel. That is about as expected. If you want more velocity; you're asking for trouble when the weather changes. By all means, do what you gotta, just saying; consistency is key, my suggestion ignore velocity it's a gross measurement at best. The target tells all.
-Mac