I think shimmy is a common problem in store bought touring bikes. If you read the fine print, they are called lite touring bikes. Some people can get away with putting panniers on them, and some can't. Lite touring bikes may have the mount points, but they are not really designed to take panniers front and rear. I had a Bianchi Volpe that wiggled like an exotic dancer, and that is why I spent the big bucks on a Waterford heavy touring bike. My base pannier pack is 15 pounds each (4X15, so 60 total), so I don't think I travel overly heavy. They do not sell ready made heavy touring bikes in bike store, although I hear good things about Long Haul Truckers.
My Waterford will flex but it immediately dampens out the vibrations. This is what generally makes it a rock solid ride. Once had the front wheel not fully seated, and the Waterford shimmied like my old Bianchi. Once I re-seated the wheel, the Waterford was back to being rock solid again.
This summer I am taking a tour and I am not taking my Waterford. I have been modifying an old steel KONA mountain bike that belonged to one of my kids. The KONA got drop handle bars, bar end shifters, and brake hoods. I also but one of the Schwalbe Marathon tire variants on, maybe 26x1.5. The KONA is not as stiff as my Waterford, but it is a lot stiffer than my Bianchi was. I had problems getting a front rack that worked but I finally ended up with a Minoura steel rack. The Minoura clamps to the brake boss and the axel. The KONA fork is rigid steel, but shaped like a suspension fork and has been a bear to fit a rack to.
You might try a better front rack as I believe that is where a lot of shimmy dynamics start.