Nothing on my back except a camelback.
If you are referring to my comment, I think you might have misunderstood. I was referring to stuff on the back of the bike, not your back.
Can't use a handlebar bag due to the location of the brakes being on the inner handlebars. I would have to move them.
If you are referring to interupter levers, a lot of handlebar bags work fine with those. You don't need much space to use the levers and some handlebar bags have a long enough bracket. This is especially true if the levers are angled down rather than straight forward. Rotating them down may be needed.
I will have a rear rack and rear panniers.
Do I remember correctly that you are the guy with the super light bike who plans to carry 20 pounds of gear? If so why would you need front and rear panniers?
My brakes mount once in the middle of the fork above the tire. Sorry for not knowing the term of these brakes!
You might google "Sheldon fender nut" for one solution with that style brakes. The are usually used to mount fenders on road bikes, but can work for racks too. You may need to fabricate your own bracket though.
Where would I find these adapter plates?
I think Old Man Mountain might sell them. Some axiom racks use a plate like that, but I am not sure they sell them separately. I think Tubus may also have something.