When you are shooting a really accurate rifle that has a stock that is typical of those used in benchrest , any extra effort required on the bolt handle will rock the rifle in its bags, and that will tend to cause sand to pile up in the middle of the front bag, which reduces roll stability and can open up groups. For those particular types of competition, unlike most other shooting, in order to be competitive you need a rifle that can shoot 25 shot averages (under the best conditions) that are well under .2000. In that situation, very small issues are likely to make a difference. Some can be compensated for, but the amount that it takes to do so, amounts to a disadvantage when conditions allow running a whole, or part of a group. Here is a good example of a top shooter shooting a tiny group at 200 yards at the Worlds Benchrest Championship.
Notice how smooth his movements are, and how little the operation of the rifle disturbs its interface with the bags. Rifles react to how they are supported, and at the top level of accuracy it is important, particularly when shooting free recoil that that support be consistent from shot to shot. BTW IMO many shooters think that a very light hold their rifle is how bench shooting is "supposed to be done". If the recoil, trigger weight, stock design, rest and sand bags are not just right for that style of shooting they are dead wrong. There are ways to hold a rifle that work, and some that do not give the best results, the only way to arrive at what works best for your situation is to experiment.