I was forced to start piano lessons at 4th grade. I went 2 days a week and had many recitals etc. I didn't start really enjoying music until I started the band in Jr. High. What are your thoughts?
In my opinion, there's no point in forcing a child to take up something that they haven't shown an interest in. There's no right age but you can notice a child's talent at a very young age.
Otherwise, as you say, a child may end up detesting something you're forcing onto them.