Side winds will definitely push you around and they aren't going anywhere. However, stability has a lot to do with weight distribution on the bicycle. This includes the weight distribution of yourself as well as your gear.
Front panniers can help quite a bit as they take some of that weight that is on the back and put in on the front. This can help quite a bit, but might not be too beneficial if you just add front panniers then fill them up with more stuff to carry and still carry the same amount on the rear. I like to use two sets of front panniers, mounted on the front and rear. That way I have no extra capacity really so I avoid the temptation to carry more stuff but I have redistributed the weight over the front and rear. This works well for short, week long trips or less. It's probably not going to work for your epic tours and you will just have to take the large rear panniers and try to control yourself with what you carry.
Your positioning on the bike during descents is probably bigger effect than how your panniers are packed. I try to keep my body weight in what I call "inside the triangle" as I descend. What I mean by this is to keep your weight distributed inside the main triangle of the bike and not loading the back of the bike by hanging your butt off the back of the saddle. Slide forward on the saddle and bring your weight over the bottom bracket and stand on the pedals, weight off the saddle as much as you can. Your butt might not even have any weight on the saddle.
Get your hands in the drops and get your upper body pressing down into the handlebars, your arms should be pushing forward putting pressure on the handlebars. Place a knee against the top tube of the bike, this will help dampen any excessive vibrations in the bike frame. Keep your body relaxed as this will absorb the shocks of the bumps.
Lowering your handlebars will usually make a bicycle more stable. It makes it easier to get your body weight forward if they are in a lower position. If you drop your shoulders low enough you will feel the wind shift off your chest and begin to flow over your shoulders, this makes it feel like there is less wind pushing you around.
Look far down the road as you descend, then it looks like things are not coming up at you quite as fast as if you are just looking down the road a few feet. Try to relax and put your mind in it's peaceful place. Wider tires than what you are running would be better for a loaded touring bicycle.