I think you're getting rugby confused with handball.
I'm assuming your from England, because in Wales it is the national sport funded by local governments and such which Football isn't. Also football maybe more popular elsewhere because the rules are easy to grasp and you can play it anywhere. So most are very familiar with the concept from a young age but i'm not completely certain.
I question your premise -- participaition in Rugby Football in the UK is popular. Just not as popular as Association football. It's a little too rough to suit some people.
It is popular, probably the 2nd or 3rd most popular sport.
