One tenth grain powder is typically good for anywhere from about 5-10 fps velocity differential across a range of different cartridges, i.e. case volumes. Using a charge weight differential of 0.02 gr Varget is unlikely to create a difference in velocity that can even be measured reliably by most chronographs; i.e. less than 2-3 fps or so. I'm going to assume the 0.02 gr charge weight differential you described was due to day-to-day variance, as opposed to a selected charge weight increment.
Nonetheless, the good news is that something caused the 32 fps velocity change. In other words, it almost certainly was not the difference in charge weight of 0.02 gr Varget. Possible sources for the velocity difference might include different atmospheric conditions such as temperature, humidity, etc. Alternatively, variance in case volume, neck tension, primer brisance, powder moisture content, barrel lapping-in status etc., etc., etc., could also contribute to velocity variance. Unfortunately, sometimes the only way to know for sure is to test every possible variable independently until the culprit(s) are found.
Rather than testing a bazillion different potential sources for the velocity differential, I would suggest making sure the barrel is fully lapped-in and producing consistent velocity from a given charge weight before doing any further serious load development. In the event that the 0.02 gr differential you described was an intended charge weight increment, I would also suggest using a more realistic charge weight increment; i.e. not less than 0.1 gr. You can always go back and fine-tune charge weight later if you think it's necessary. Further, I would suggest not doing serious load development on days where the temperature varies markedly. If you can't get a sufficient number of days to test with reasonably stable temps, I'd wait until you can, or try to do more testing on a single day. If your velocity is varying by as much as 32 fps for a given charge weight on different days, load development will be challenging at best.