Jrkotz, you have a few viable options should you decide to go with a Savage 20BR. You could do what you have suggested and do a barrel swap on a factory F-Class of LRPV. You could buy a Precision Target action and start from there. Or you could go with the inexpensive route and buy a Steven's 200, sell the barrel and stock and build off that action.
The Steven's 200 is a pre-accutrigger Savage action with a skinny barrel and a plastic stock. The barrel and stock are essentially useless for your application, but the action is nearly ideal. The entire rifle can be had for around $300, and if you recover $80-$100 selling the barrel and stock, you have a very inexpensive action to build from at only $200-$220.
http://www.savagearms.com/images/firearms/finder/ginormous/stevens200short.png
You can buy a pre-fit barrel from most all the custom barrel makers and screw it on yourself with a barrel vise, barrel nut wrench and a headspace gauge, or take it to a smith and have it done for you for very little money.
Several nice triggers are available for short money. Timney, Rifle Basix, and Sharp Shooter Supply all make a good trigger, and most you can install yourself. The Rifle Basix SAV-2 trigger is adjustable from 4oz to 3lbs and costs $155.
http://www.riflebasix.com/index.php?main_page=product_info&cPath=1&selectOpts=1&products_id=182
For a stock, Sharp Shooter Supply makes excellent laminated stocks. They are available in several styles and colors and come inletted and pillar bedded. I have a Dog Tracker and I love it. It tracks great and is designed so that it doesn't torque in the bags.
http://www.sharpshootersupply.com/
Or you can look at what Stocky's Stocks has to offer, as they have a large selection.
http://www.stockysstocks.com/servlet/the-SAVAGE-CENTERFEED/Categories
If you choose you components carefully, you can have a complete semi-custom rifle for around $1000 that will drive nails.
I built a .222 on a Steven's action and it has been a great rifle. It will shoot .25MOA when I do my part, and I have just over $1000 in it including optics, though I bought my action used and my scope is far from top line. It has a Pac-Nor pre-fit barrel, Timney trigger, dog tracker stock, SSS single shot load ramp, Bench Source scope mount, Burris Z-Rings, and a Tasco 6-24x42 scope mounted on top. It is super accurate and is built almost exactly how I want it for less than even an F-Class Savage would cost, and it's probably just as accurate if not more so out to 300yds.
You don't have to spend a ton to get a nice accurate rifle. If you do some of the work yourself and choose components carefully, you will end up with an inexpensive rifle that will be very accurate and set up the way you want it, not how Savage decided to build it.