Many, perhaps most, people leave the shoulder straps at home for exactly the reasons you cited. I, however, take them along. Why? First of all, the Ortlieb instructions tell you to use the shoulder strap to pull the ends of the roll down after rolling and clipping. Many people instead clip the two sides to each other, thus pulling the ends of the roll up. But I believe that the bags are more waterproof, especially from rain, when used as instructed. I've heard several people complain that Ortlieb panniers are not as waterproof as promised, but most of those people are clipping the ends together rather than using the shoulder strap to pull them down.
Next, I find the straps useful in a few situations. These situations don't come up that often, but they are very handy when they do. I've only done this once, but a pannier can be used to haul water and the shoulder strap is pretty important for that. More often, however, I've gone on a hike or another foot excursion where my handlebar bag wasn't big enough to carry what I needed. In that case, I take one or more of my panniers slung across my shoulders. It's a pretty comfortable way of carrying your stuff over long distances.
If you clip the ends of the roll to each other, you have to frequently clip and unclip the clips. If you use the shoulder strap, clipping and unclipping is much less frequent. I did have one clip break and I'd prefer to minimize the risk of that happening again.
Note that I don't think the handles are really designed for carrying, although I use them often for that if I don't have to go far. The main purpose of the handles is to attach and detach the pannier from the rack.