Go over to any decent gunsmith's shop, and take an inventory of the equipment you can see standing on the floor. End mill, lathe, grinders, drill press, bluing tanks, torch. Now check out the bench. Vises, dozens of files, hundreds of screwdrivers, punches, scrapers, stones, micrometers, dial indicators, taps, drills, books. You won't even see what's neatly filed away in drawers. Jigs for welding bolt handles, fixtures for doing action jobs on any of several dozen popular handguns, specialized tooling for procedures you've never heard of. None of it is cheap. None of it is unbreakable. All of it wears out eventually.
Of course, if your gunsmith is a rifle specialist, he probably has a bit less tooling, but he didn't get to be a rifle specialist without knowing how to do all the other stuff. Each job he does has to pay its share of the many, many thousands of dollars of tools and equipment in the shop, not to mention the investment in education and experience that got him there. Parts are expensive. Insurance is VERY expensive. He needs Federal, state, and local licenses, all of which have fees. Rent or mortgage on the building. Utilities. Taxes. And he still hasn't paid himself a wage yet. Custom gun work takes a lot of hours. Your $5,000 custom rifle very likely has several hundred hours in it. Do you really want a minimum-wage gunsmith building your custom masterpiece? I don't.
Custom rifles don't pay the bills. Bugatti used to build the fastest car in the world. They sold it for over a million dollars a copy, and lost money on every one. Like a custom rifle, they were built as a labor of love. Bugatti is owned by Volkswagen, and it's Golfs and Jettas that pay the bills. They built Veyrons just because they could, and tried to keep the losses under control by charging what the market would bear. Your local gunsmith keeps the lights on by doing DCOAs on Remington 870s and Savage 110s. Replacing broken firing pins. Troubleshooting old Marlins. The odd recoil pad. Mounting and boresighting scopes. If you paid full shop rate for all the hours in a custom rifle, you'd probably be looking at triple the price. The few custom riflesmiths that do only custom rifles have long, long waiting lists, full of people who really want to pay thousands for a custom rifle.