Personal anecdotal evidence only: I rode the same 20 miles in the same weather conditions, about 45F, cloud cover, no sun, 8-15mph swirling winds and spitting rain. Feet were ice cold on one trip, feet were fine on the other. When I took off socks, I could see the toes were pale and pushing on the moons indicated poor circulation.
The only thing that changed on the trips was swapping out my bike shorts for a swimming suit. My conclusion, the grippers on the bike shorts were restricting micro circulation just enough to make heat transfer into and out of the extremities difficult. Might have also involved how the chamois pad interacted with the seat on my recumbent. Toes were pink and moons recovered immediately clearly indicating adequate circulation.
I only know that using unrestricted, unconfined, unpadded shorts made a huge difference.
Your issue could be, as pointed out, circulation or nerve impingement but cold feet in 50F is not normal. Could be your clothing, your bike geometry, your physiology or an underlying condition you need to get looked at and treated. Maybe not because it's a dangerous situation but because it's uncomfortable.