Like the OP, I used to ride a TT/tri bike. Likewise, I averaged ~20mph on nearly every ride and focused on keeping my speed up for training purposes. I competed in the weekly TT. Unfortunately, I got tired of being restricted to nicely, paved roads. I started riding more chip & seal and dirt. I ended up buying a cyclocross bike and setting up the cockpit like my TT bike, except with much higher stack height. It does have aerobars, fenders with flaps, and the occasional rack. It is much slower (~6mph less). But, it's much more comfortable over long distances, less terrain restricting, and altered my mindset when it comes to riding. I no longer care about speed. I just like to get out and comfortably cover as much area as I can, while enjoying nature.
At 22, I know I wasn't astute enough to realize there's a whole 'nother world in tour/adventure cycling. For that, you have my admiration.