Actually I'd argue the opposite: relativism is much less controversial in relation to aesthetics than ethics - you could make the case that the rise of moral relativism coincides with an aestheticisation of ethics...
A potential problem with a relativist/purely historicist approach to musical taste (and society for that matter) is that there needs to be some sort of transcendental frame in which you present the "people and cultural baggage and the rest" to explain what is going on, and this itself has to remain ahistorical - so, for example, when somone like Judith Butler says the history of society is just a collection of shifting performative practices, what remains unchanging throughout is this idea that our identity is performatively constructed: it's difficult to analyse or deconstruct or question the idea of identity-as-performance from inside this theory, to ask what are its historical conditions of possibility, because it functions as an epistemological a priori.
I think this is the criticism which mark k-punk makes of people who say that musical taste is all about individual enjoyment, and the stories we tell in order to explain/justify that enjoyment: I hope I'm not misrepresenting him when I suggest that what he wants to question is this notion of individual enjoyment as some "transcendental frame". For him "individual enjoyment" whose only committment is to itself is a historical phenomenon whose condition of possibility is late capitalism and the ethic of consumerism it promotes. The Marxist approach to aesthetic criticism such a complaint employs uses the economic sphere as its frame, but it does so self-consciously, and with the caveat that it attempts to be attentive to the historicity of its approach - the logic of capital is not a universal underlying cause, but a historically contingent one specific to our epoch. If we were to have a revolution, this universality itself would change, necessitating a new frame. Etc.
I broadly agree with this position; however I have some reservations about attempting to anchor considerations of music so sharply within this sort of political framework. To begin with, we should note that, as per Bourdieu, this approach seeks to reduce the aesthetic to the social/political (i.e. to be somewhat reductive, for Bourdieu music should be considered in terms of how it reinforces a sense of class distinction; for k-punk music should be considered in terms of how it reinforces or undermines a certain mode of subjectivity under late capitalism). I don't have a problem with that <i>per se</i>, but, somewhat contradictorally, this manoeuvre then doubles back on itself in order to reinforce pronouncements of individual enjoyment - e.g. "I enjoy this piece of music because it exists outside of Capital's vicious cycle of consumerism and conformity."
Such pronouncements do not really escape the problem of privileging enjoyment-perception - the sense that the music resists the logic of Capital has to, in the first place, be enjoyed/perceived.... unless you're basing yr music criticism solely on empirical facts such as the socio-economic status of the artist, modes of production etc... in which case there is no "music criticism" as such occurring at all. Having asserted that enjoyment is socially constructed and listeners are not the sole authority on the nature of their enjoyment means that we cannot silently pass over the "I enjoy this..." and pass directly to "because it..." - the post-facto rationalisation is precisely that, post-facto. And if we're serious that enjoyment is structured by the logic of Capital, then we have to, reluctantly, conclude that even when we champion some obscure grime MC or some left-wing post-punk band, our enjoyment is structured in advance <i>by the same logic</i>.
I guess what I mean by this is that you cannot have a post-capitalist mode of musical perception until you're in post-capitalism; our musical appreciation will adhere to a <i>form</i> structured by our pre-existing mode of subjectivity, even when the content we appreciate (e.g. music that seems connected to revolutionary social transformation, obliquely or explicitly) points to some alternative to this subjectivity. And even if our mode of subjectivity were to change, it would remain the case that our relationship to music would remain shaped by this enjoyment/perception threshold (itself transformed by the transformed social co-ordinates). I can't rule out the possibility that this mode of subjectivity might become a mode of <i>collectivity</i> (though I tend to consider "the subject" to be something that would survive pretty much any social transformation), but, even with such a shift to collectivities we would still be dealing with differences of perception-enjoyment.
The difference between music and ethics in this sense is that, while there's an argument that it is possible to consider abstractly the field of the social (if we hit upon the right formula), music criticism (when considered as the ethics of the perception-enjoyment of music) cannot abstract this perception-enjoyment threshold without becoming pure meta-criticism: the abstraction of the perception-enjoyment threshold disallows the critic from using their own perception-enjoyment as the secret underwriter for their pronouncements. So while, as I said before, I broadly agree with k-punk's thesis, I consider the naturalisation of the current mode of enjoyment to be the condition of possibility of saying anything intelligible about our relationship with a specific piece of music. By speaking about the music we like, we already accept the co-ordinates of a relativist approach to individual enjoyment, whatever we might say on the matter of relativism itself (litmus test for anyone's commitment to certainty in music: are you prepared to disavow your own opinions entirely?).
