Interesting topic. I've nothing informed to say about it but here are some thoughts:
I'm not sure if IDM was about disavowing black music from the POV of its practitioners or something projected onto or discovered in it by music critics. Like I doubt Aphex Twin determinedly set out to make 'white jungle' or something, but he came from a different cultural background and that naturally pushed him in different directions. And avant garde electronic music has roots in the very white world of 20th century classical music (John Cage, Stockhausen, et al) and conceptual art.
Having said that, there is always a degree of this in certain areas of the nerdosphere - preferring to listen to Grime instrumentals (or dubstep, as Luka would say) than Grime with MCs. (Again, this usually wouldn't be a consciously racist thing so much as a matter of the culture people come from.)
Kanye is an interesting figure in this regard, especially given his recent attempt at divorcing himself in some way from his blackness - he started out sampling soul records and over the years has rebelled more and more against that traditional black music role to incorporate 'white' Euro influences.
I think people who make this music and listen to it are probably seen as weirdos from the POV of most ppl white and black but there's perhaps more freedom for white ppl to pursue these more abstract artforms without being accused of being effete or bourgeoisie or whatever.
As you say, this sort of art (not just music) is very middle-class, perhaps because it requires a certain degree of education/'cultivation' to understand why listening to e.g. abrasive white noise might be pleasurable or interesting and not just anhedonic and pretentious. The rise of Kanye et al in rap music is probably to do with the concurrent rise of the black middle class in America - and the (comparative) freedom that comes with this to explore and experiment with identity, sexuality, gender and sound.