You do need to be in relatively good physical condition, that's all dependent on how you want to push yourself..
We have 10 years old kids starting kitesurfing and the oldest person we taught was 69 years old...for now ! 
Every week in our stand Up Fitness classes we have persons from all different background and ages.
At 8 years we give them a stand up. At 4 a surfboard.
We have activities for all the fitness levels and ages.