Some years back, one of the guys I ride with decided to kick things up a notch. He started riding mountain bike trails on his Cannondale Toruing bike. Feeling pressured to keep up with him, I took a Bianchi Volpe to Maybury State Park, and tried riding the course. I bailed after doing 30% of the course, convinced that if I keep riding, I would end up with a broken collar bone. The Volpe was under-braked, and my center of gravity was too far forward to safely ride the course. And since the Volpe did not have a flat bar with bar ends, it was hard to muscle the bike over big exposed roots.
If the worst you have to ride is two track fire roads, then any bike could do. Racing road bikes might not be geared low enough for climbs that you will encounter, and their wheels might not be strong enough for dirt roads. I might encourage you to use 32mm or wider tires. I have 35mm Schwalbe Marathon tires on my touring bike. Schwalbe seems to have a bizarre sense of tire width, their tires have really big envelopes.
You might have to stage different tires for your off road segment. I cannot imagine riding cross country on knobby tires. There will not be any knobs left when you get to the other end.