Nah, that's not it at all. Would never've said that about UR, for example, and they were packed to the gills with concepts and agendas. It's just that most of IDM was so insufferably dull/vibeless (I liked some of it, obv.), and certainly didn't suffer from a surfeit of intelligence anyways. Its view of intelligence was a particular, very constricted view of what intelligence must be - this is for another thread, perhaps.
Yeah, I wouldn't actually associate James Blake or anyone else under disscussion here with concepts and agendas anyway. Probably the most conceptual / agenda oriented person in the dubstep / whatever millieu (and one of the most obviously intellectual as well) is Kode 9 who I have a frankly embarrasingly fanboyish devotion to.
The IDM fear is about the replacement of raw, exciting, mindbending music with something that is quite clever and uses all the right signifiers of good taste (while conspicuously avoiding anything that might be seen as cheesy) and does some vaguely innovative stuff, but without ever feeling like it
matters (there's an interesting discussion to be had about if and why a radical step forward in a scene with a reasonably well developed musical language feels so much more exciting than another eclectic mutation in a scene full of unrelated eclectic mutations) and without ever really being genuinely exciting.
I'm not saying that The Genre Formerly Known As Dubstep is like that at the moment, but it's a tendancy that people are quite reasonable to be worried about, particularly when a lot of the talk around it comes out of a distaste for 'stupid wobble' and a reverence for the 'proper musicality' of classic garage. (Also, I think 'intelligent drum and bass' and the whole jazzy fusiony breakbeaty 'thinking of putting a band together to do some of this stuff live' scene from the mid 90s is probably a better analogy than IDM?)