Interesting piece, thanks for sharing. Hope you don't mind me jotting down some notes in no particular order.
First, it's often occurred to me to wonder what sort of political position Lovecraft would've occupied had he lived a bit longer - at least through WII, or even if he were somehow miraculously still alive today. His visceral racism certainly did mellow to some extent in his later life, although even on the day he died I'm sure he still had beliefs that any 21st-century person would regard as severely racist. I do wonder if the horrors of WWII and the Holocaust, which were surely as 'indescribable' as any entity he ever created, might have been enough of a shock to make him reconsider his position. And he was, in his own way, an avowed anti-capitalist and turned to moderate socialism in his 30s (although never communism, which he hated for its antipathy towards traditional culture, which for him was everything). If he could've got over his racism - a tall order, I know - I think it's not inconceivable that he could have found himself aligned with some elements of the radical left. Again, that's a very big 'if', of course.
You're dead right that he'd have had nothing to do with the AGW denial industry, that's for sure. The science he used for background in his stories was always rigorously up to date, with the glaring exception of the racial pseudoscience/social Darwinism, which was thoroughly rooted in the late 19th century and had begun to be abandoned by serious biologists even in the '20s and '30s while he was writing. I've not actually read that particular story but I see it's one of the Lovecraft-Barlow collaborations. Barlow's a fascinating character in himself, forming as he does the link between Lovecraft and Burroughs.
Not wholly convinced by "Soap operas and the modern novel dramatise everyday life in bourgeois society" - novels, maybe, but Coronation Street and Eastenders were not set in "bourgeois society" the last time I looked.
About Tolkien: with respect, if you think TLOTR is a 'comforting' book, you haven't understood it properly. I read a description of it once as "the saddest book in the world", and I think there's a lot to be said for that. At the end of the novel, after Sauron is defeated, his orcs and other monsters destroyed and the human nations he'd bewitched and enslaved set free, the elves have still lost, because they have the ultimatum of leaving Middle Earth forever or remaining behind and gradually fading into nonexistence. (This is personified in Elrond's giving away Arwen in marriage to Aragorn, knowing that she will become mortal and die as a result, and that he'll never see her again - and for elves, being immortal, this is a pretty big deal.) The hobbits presumably hung around long enough to inspire the tales of the 'little people' that are ubiquitous in European folklore, but also long ago vanished before the encroachment of modernity. The ents and dwarves likewise disappeared back into the forests and mountains they came from, before the forests themselves were hacked to pieces and the mountains dug out by great mines and quarries and crisscrossed by highways. And while the orcs are gone too, what do we have in their place? Human societies just as bad as, if not far worse than, Mordor and Isengard. The whole book, in fact everything Tolkien ever wrote, really, is one long lament for a spiritually connected and ecologically integral world that had mostly vanished long before he was born, at least in England, and which persisted here and there in a few places which were nonetheless being torn up and paved over even as he watched.
(And if you think Game of Thrones is 'comforting', I suggest you watch an episode - any episode - and you'll be disabused of that notion within the first five minutes. It's one of the bleakest things that's ever been on TV.)
BTW, your link to the Guardian piece on Tolkien is coming up with a 404.