I agree that if barrel life or barrel heating is not a concern, as is with coyote hunting, a bigger 20 cal would be a wise choice.
For a rifle that could see 300+ rounds in one day like my rifles when gopher hunting, a smaller 20 cal would be nicer to you.
One thing to keep in mind is that if recoil and reloading cost is not a big deal, you can always load DOWN a big 20 cal to save barrel life when going after gophers and prairie dogs, then rev it back up for long range work.
Like I said, I only run a 34gr bullet from my .204 Ruger at 3750 fps in a 22" barrel because I mainly do a ton of shooting with it for prairie dogs and gophers and don't want to burn up my barrel too quickly. That load also works well for coyotes out to 400 yards or so, but drifts horribly in the wind past that due to light weight and low BC. I could run that bullet closer to 4000 fps, but gophers still come apart very nicely at 3750 fps
If I am specifically on a coyote hunt, I have another load for my .204 that uses the 39 gr Sierra BK at 3825 fps. Much better out to about 500 or 600 yards if the wind isn't blasting horribly.
So my .204 is pretty versatile. My 20 VT cannot be loaded UP a whole lot more. I could maybe get 3600 fps with the 34 gr bullet from my 20 VT in it's short 22" barrel, but that's not much gain from 3450 fps and I don't want to burn my barrel out too fast on that rifle either.
So you have to really think of not only the type of hunting you will be doing right now, but also about the type of hunting you MAY be doing in the future and choose a round that will do it all for you.
The other option is to build multiple rifles like many of us do : )