Your message brought back VIVID memories of my first solo loaded tour, just a few months ago. My wife was watching as I road nervously down our street and wondered how on earth I'd make it to my campground that night.
FIRST, having ridden a thousand or so miles now with four loaded panniers, let me give the comforting news that it gets better, much better. I actually prefer riding loaded now, the bike feels more stable, especially on fast descents.
SECOND, for me, at least, what I've found is that the load amplifies my movements on the bike, and that the shaking that I felt that first morning back in June was largely self-created. Riding with loaded panniers requires a much lighter touch for me.
THIRD, my bike gets much more stable above 6 or 7 mph, although, oddly, I can climb a hill, peddling at a steady cadence, right down to 4 mph.
FOURTH, congratulations on setting out on this adventure! My journal for that first tour notes that for the first few miles I was sure my bike was about to fall apart, that after 5 miles I was starting to figure it out, and that after 20 miles the bike felt completely natural.
FINALLY, I find that I need to be more "intentional" with loaded panniers. Mounting the bike, starting out from an intersection, shifting early before the hill gets steep, all these things require thought at first but will become second nature, just like "riding a bike."
Sometimes I think a loaded bike needs a new name, it handles and rides so differently. And it's GREAT!