Ben,
the odds are the drift is, as others have said, almost entirely due to heat - plus, what nobody has said, Remington using hammer forged barrels. They're made by boring a short, fat blank, inserting a mandrel which has a mirror image of the barrel interior including the chamber machined into it, and feeding it into a hammer forging machine that belts it along the length of the blank with a series of opposed hammers around the circumference. What was short fat blank is now a long thin barrel and the mandrel is pulled out leaving it rifled and chambered. Obviously, it's a bit more complicated than this crude description, as seen in the 2nd of the links below, but that's the general principle.
http://www.rifleshootermag.com/gunsmithing/RSgunsmith1/
http://technology.calumet.purdue.edu/met/higley/NotesOnHammerForgedBarrels.pdf
Most factory rifle barrels are hammer-forged. The equipment is expensive and the operators need a lot of skill and experience, but once up and running it turns out long runs of barrels very cheaply, hence nearly all large scale sporting rifle manufacturers using it, and all military weapons now have barrels made this way.
Apart from low cost, pros are a very smooth and hard finish, so life is good. Cons are it is a brutal way to treat a chunk of steel and inserts enormous stresses into the metallic grain structure. Even with post manufacture inspection, stress relieving and straightening, they tend to bend a little as they heat up, hence wandering groups. Not all hammer forged barrels do this, so it's a lottery as to what you get. To be fair, people like Remington have been using this process so long and are so good at the inspection / rectification, it's a rare example that does it really noticeably nowadays, but many will do it a little after half a dozen to ten shots.
The manufacturer can say reasonably enough for sporters that it's first / second shot accuracy in a cold barrel that count, and as long as they're consistent, the rifle doies what it's intended to do.
If this is the problem and from your description it sounds like it, restocking or playing with the bedding won't help. The cure is either keeping the heat down by shooting slowly and/or having regular breaks to let the barrel cool down, or rebarrel with a cut or button rifled custom production job from a barrel-making shop.
Nobody now makes target rifle or really top quality varmint rifle barrels this way. The nearest you get are heavy barrel sniper rifles from the likes of the Austrian Steyr company with its SSG series, FN-Herstal and so on. They have a huge amount of experience and do a lot of testing / rectification before fitting their barrels, so the results are pretty good, but rarely up to Krieger or Bartlein standards. You get what you pay for at the end of the day!
The long defunct English company Parker-Hale Ltd of Birmingham made all its barrels this way including for its various target rifles. They mostly worked OK by the standards of the day, a few worked brilliantly, and a few were absolute dogs and just wouldn't shoot or suffered severe impact changes. P-H was very good at taking any such rifle back and rebarrellong it free of charge to keep the customer happy. I had an ex-police P-H M87 sniper rifle in .243W (actually still have it but now with a 6.5mm Bartlein on) and it did just what you describe, but not nearly as much, maybe moving a half inch to one side at 100yds. You just compensate for it once you know its little tricks. In your case I'd see if Remy will put it right if you complain as you've got a major POI shift there..
Laurie,
York, England