There is always a solution, depends on the goals and expectations.
Since you’re starting with a bolt action 90% of the problems are solved, you have kind of already stepped outside the box.
You need to decide which is more important, subs or supers. Build it to excel at one and accept what you get with the other. Since you don’t have to worry about powders and pressures that will cycle the action, any bullet choice can be shot subsonic, as long as the twist is fast enough.
The problem with the 300 Blackout and Ham’r is using it in an AR robs potential due to magazine length restrictions. So the first thing you will want to do is either single feed or set it up to use an AICS magazine. This will allow a loaded length of 2.550” + in a magazine. This will allow velocities will allow you to be competitive, with either cartridge.
Consider a Hornady 250 grain Atip in a 300 Blackout.
View attachment 1295190
One is loaded to 2.260” for AR magazine length, the other at 2.585”, where it hits the lands. The 2.585” round will feed from an AI magazine. The differences in powder capacity and velocity is about 700 fps.
View attachment 1295193
Depending on powder choice at 2.226” all you can fit is about 2-5 grains.
The other problem with the Ham’r and magazine length restrictions is what happens with a long bullet and not enough room between the end of the neck and the magazine. Here is that same Atip loaded at 2.226”, note the .308” portion of the bullet is below the neck line. This is why if you want to shoot long heavy bullets in the Ham’r, you might want to consider AI magazines. You will also be able to load them long enough to get enough powder to make it worthwhile.
View attachment 1295202
This will also come into play with lighter bullets that have a longer length like the 115 Berger. You can’t load the Berger with out the base of the bullet below the neck in the Ham’r at 2.226”. Ditching the AR mag will change both cartridges potential dramatically.
A 208 Amax does fine in a 1/10 out to about 300 yards, but remember, since its a bolt action, weight is not really a factor, 85 grain subs function just as well.
Case capacity is an issue. The Ham’r will provide more velocity for supers, the Blackout will be easier to tune for subs, where excess case capacity works against you for accuracy.
The velocity gain from the Ham’r for 2-300 yards probably isn’t the deciding factor with lighter bullets. It would be fun to see what you could sneak out of one with something like a 175 SMK, where 150-200 fps makes a difference.
