Hard to approach this at my work desk (my hands roughened by the keyboard).
That it is a luxury which can only be attained by a removal from the world and the reality of people's lives? That it relies on everybody else's assumption of coarseness and unfeeling, so essentially you are talking about the difference between calloused rough strong hands of the worker and the soft dainty hands of the aristocrat.
There's truth in that, although I'd question why one person's "reality" - soft-handed or not - is more "real" than another's.
I think this accusation was levelled at Joyce, that he'd written a book ("Ulysses") about the common man which the common man would never be willing or able to read. And of course John Carey's view of modernism - that it was a conspiracy of sorts by the middle-upper classes to wrest literature from the grubby hands of an increasingly literature lower-middle class.
I was actually reading an essay that I'm sure would drive you up the wall by Philip Larkin last night, in which he attacks modernism for severing the link between the arts and the public. I think he's extremely narrow-minded and short-sighted on the "pros" of modernism (and this is a bit confusing, given that he clearly idolises Eliot) but there's obviously truth in what he's saying, insofar as there's a whole load of modern art that the average person is dumbfounded or bored by, and the more educated/aesthetic person could be said to be merely seduced or fooled into "appreciating".