In answer to tht's question, I'd say it was a bit of both, at least initially. The theory makes use of some maths that was develped by Gauss and later Riemann (both German mathematicians) in the 19th century purely for the purposes of abstract mathematics, with no expectation that these concepts would ever be used to create a physical theory (specifically, they were interested in hypothetical geometries that contradicted laws laid down in ancient times by Euclid). Einstein realised (some sixty years later) that Riemann's geometry, in a suitably modified form, could be used to describe the curvature of space-time in the presence of mass, with the degree of curvature coresponding to the strength of gravitational fields. At the time of the theory's publication, very few people were familiar with the necessary mathematics, as it had not been thought relevant to physics and had largely been filed filed away as an interesting but practically unfruitful footnote.
The other big barrier, as you say, was the sheer weirdness of the theory: it's a huge departure even from the Minkowskian space-time of special relativity that had become a familiar part of theoretical physics just a decade earlier, replacing the absolute space and time of the Newtonian worldview. However, just as many physicists were quick to learn the requisite maths to understand GR (and I can tell you from personal experience, the most intimidating thing about it - to someone with at least some experience with vector calculus - is its forbidding notation), the ideas themselves were rapidly absorbed into the corpus of scientific thought, especially after Arthur Eddington's 1919 expedition to measure the deflection of starlight by the Sun's mass during a total eclipse (although it is now thought the results from that were somewhat 'fudged' to fit the prediction, but that's for another thread...).
At the time a journalist told Eddington he'd heard that "only three people in the world understood GR", and asked for his comments; Eddington is said to have replied "I'm tring to think who the third person is" (the implication being that the other two were himself and of course Einstein). But as borderpolice points out, the theory is now taught to undergraduates, so I wouldn't say any 'eminent theoretical physicists' can't understand the theory - a few may have rival theories of their own, but so far not one of these has come close to matching GR for mathematical elegance or predictive power.
Interestingly, the other great theory Einstein helped found (but later disavowed) - namely quantum mechanics - causes controversy to this day, mainly because of the extreme difficulty found in interpreting its 'meaning' for our picture of reality (a feature not shared with GR). Richard Feynman (arguably the greatest physicist since Einstein) once said (in reference to Eddington's comment), that while GR is understood by undergraduate students all over the world, no-one really understands quantum mechanics.