If the bike does not fit you properly, then it is not worth owning.
Lets talk about what happens once you put panniers on it. The bike still fits as good or as bad as it did before. The handling will be lethargic. The bike will be slower in general. If the bike frame was not designed for touring, it may also wiggle or shake (or it may not).
Are you concerned about keep in bike up right when you start or when you are stopped at a light? I don't really understand your fit concerns, but maybe you could tell us more. The long wheel base of a touring bike makes the bike pretty stable once it is moving...
Some loose rules on fit. The bike frame size is chosen based on your stand over, and then the stem (plus steering column spacers) and seat are adjusted to make the bike work for you. For many people, the frame size based on their stand over, can never be made to work for them because the stem would be insanely too long or too short, and that is why there is a market for custom bicycles. I generally fit a standard bike, but I have a friend that can only ride a custom bike.
You probably should get hooked up with a good bike store that knows how to fit you for a touring bike. Finding the right store can be a challenge. One of our local stores is known for their fit service, but their focus is on racing, and they don't know much about touring bikes or how to fit one. I bought two bikes from them, but my touring bike came from somewhere else. I am in the market for a cross bike, so I will probably go back there.
Where do you live? Are you in New York? Someone on this forum might know a good store in your area.