As stated, it almost always changes the POI. In rifles where adding a brake has opened up my groups (a rarity), I have always been able to tweak the load and tune them to shoot as well with the brake as without.
Would a POI change be the same as an accuracy change? Fortunately POI can be changed by adjusting the scope.