I think padraig's post sums up the main (real) reason many people are opposed to it: it's "for toffs" so it "shouldn't be allowed". Of course the nominal main reason against it is that it is extremely cruel, and this is undeniable. But if you could quantify animal suffering (in Britain, at least) I think fox hunting would be quite far down on the list, after factory farming, animal testing for cosmetics and abusive/neglectful pet owners. Not to mention meat and fish imported from countries where animal welfare standards are much laxer or essentially non-existent. And from an ecological POV, foxes are hardly endangered and are a serious pest to farmers.
Personally, I think it's pretty fucked to get pleasure out of an animal's suffering - and fox hunting is fairly unique in this respect, in that if you go pheasant-shooting or whatever the animal dies a fairly clean death and is eaten afterwards, which is fine. From an animal-rights perspective, this kind of hunting is probably far preferable to the way most meat animals are raised and slaughtered in ordinary farming. And if you're serious about keeping fox numbers down you can of course just shoot them.
In the long run, I would have to say I probably support the ban, but with the reservation that many people who supported the ban did so for the wrong reason. As Rich said, the fact that you dislike someone or find them repugnant does not give you the right to dictate what they can and can't do - it should be the effects, if any, of their activities on other people that decide this. Or, in this case, the effects on animals (arguably). Whether the person doing the whatever-it-is grew up on a hereditary estate or a housing estate is irrelevant, or at least should be.
(BTW, hunting, shooting, falconry and so on were 'sports' for hundreds of years before the word came to mean stuff like football and tennis.)