Take the ferry from Manhattan to Atlantic Highlands NJ or the train (non-rush-hour) to Long Branch. Then just go down US9 to Cape May. It can be done in two days even with the detour through Egg Harbor City and Mays Landing. It's a fairly good bicycling road with plenty of services. I use the campground in Bayville, just south of Toms River and right on 9, and the one near downtown in Cape May.
Take the ferry from Cape May to Lewes and continue to the Chesapeake Bridge Tunnel. Motels enroute in Ocean City MD and Onley VA. and camping in Pocomoke State Park. Call ahead an hour or so for the free ride across the bridge in a maintenance pickup truck. This puts you right in Norfolk.