It's actually a lot older than him; in the 90s, the music industry realized they wanted to get rap to conform to the older model of artist branding for other genres. I would say this began officially with Nas going to Columbia and being regarded as a 'voice of a generation' type figure. Before your other acts were either marketed in the touring rock group model (Public Enemy or Run DMC) or they were a 'pop star' (Will Smith, Hammer, Vanilla, etc.) and all the various acts beneath them never really worked to such industry terms with real ease. But with "Illmatic" changes everything. For one, DJs are no longer even a thought and producers are reduced to whores, whereas before you had a main producer who was an integral part of the group, either as a 'bandmate' ("he's the DJ, I'm the rapper") or the mastermind behind a roster of artists (your Herbie Lovebugs, Marley Marls). Obviously later the DJ/producer as auteur became a thing with Dre, but that was because he could also be marketed as an artist, like they'd do with The Neptunes and Timbaland later.
(Its what makes Mustard so unlikely... He can't do anything but make beats, yet they grant him more value than his artists Ty and YG. I guess its because now that rap is so vital to pop in the 21st century, he has a value that a Marley Marl could never achieve back in the 80s.)
Going back to "Illmatic", what happens here, with the precedent established by NWA and this concept later embodied by branding geniuses like Puffy w/ Biggie and Dame Dash w/ Jay is the marketing of the rapper as 'artiste'. Other rappers make singles, or serve as good rappers, the same way a musician isn't the best artist. But now these guys become brands of a new 'cool' aesthetic. They're movie stars, fashion icons, etc. etc. etc. The parts of rap that had defined it the decade before now get discarded or reduced drastically.
So then why does more rappers happen? Simply put, New York becomes too embedded in rivalry and panders too heavily to a R&B/pop brand that doesn't speak to AMERICA. At best it speaks to young america or black america but you're missing a huge fanbase who A) might not identify with this New York centricity or B) might honestly not identify with the imagery of lower-class urban African Americans. So you get Eminem, who adheres so subliminally to the ideas of rock as his basis, in addition to always ALWAYS thinking about pop sensibility before rap.
When Kanye comes out, he's possibly the first rapper after Eminem who's black who gets it: the labels do not want rappers, they do not want a classic album, they want to sell a career, a legacy. And Kanye establishes this so swiftly because he's attached to Dame Dash who has already worked at doing this with Jay to great success. But Dame also wants to do this for all kinds of rappers and establish huge rosters of careers, whereas the music industry wants to scale down: Keep the pantheon small. So when Dame gets clipped, Kanye comprehends this and jumps ship to Jay. Could Jay ever successfully manage or develop another artist? HELL FUCKING NO. Jay is a talent, but there is no entrepreneurial spirit to him. He's basically a rapping bimbo. Not that he has to be anything different.
So with the industry needing to downsize, do they get rid of the pantheon with all their expenses? No. They gut the part of them that relies on the lower-echelons of rappers. And to be fair there has always been middle-class poseurs in rap... Busta Rhymes loves to claim he was drug dealing when he comes from a solidly middle class part of Long Island, not even NYC. Prodigy went to art school, Ja Rule was went to private schools and came from a sheltered Jehova's Witness background.
But the problem with Atlanta in 2014 suffering this is that it deliberately ignores the lower-to-middle class audience who determines who are the hottest artists instead opting to let blogs lead them along to 'buzzing' artists determined by the internet; a temporal sort of presence as opposed to your more concrete campaigns made a decade ago by artists like TI who invested heavily in their communities to become regional and then national successes. But now radio has become graphically altered by Billboard placing Youtube Views on the same pedestal as radio spins, and thereby adheres to Youtube as a taste enforcer. And worse, iTunes is dictating certain songs that an urban radio would never play as Rap/Hip-Hop, and Clear Channel feels pressure to compete with the internet.
The result? You have American Rap Stations playing Macklemore and Katy Perry alongside their usual fare. Rappers now have to compete with not only the Rap Pantheon but ALL OF THE POP PANTHEON. How is your Young Thug going to independently make a dent in radio when he's struggling against big pop radio trying to consume everything in existence.
Which brings me to something that echoes Luka's ponderings about Gatekeepers.
For the past year now, Chief Keef has been silently releasing videos for singles he releases straight to iTunes or just to youtube with little fanfare but a lot of public acceptance. Like, go through his videos of the past year, and he racks up millions in views with relatively little internet buzz. For anyone else, this would mean his label should seize up on these songs doing their 'spins', and get them placed in radio rotation to further the expansion, the same way they've done for Macklemore who should be an infinitely harder sell for rap radio than Keef. But they don't get to radio. They are used as the background of dozens if not hundreds of instagram/vine clips the same way as other records such as Sage The Gemini's "Gas Pedal" do, so the fanbase is certainly receptive to a substantial enough degree, and the youtube views are a pretty solid indicator of his consistent following. But they don't get to radio. And now, Keef's been dropped from his label.
My theory is that management comes into play; recently I read this interview (
http://www.xxlmag.com/news/2014/10/cortez-bryant-lil-wayne-manager-interview/) about how Lil' Wayne's manager Cortez Bryant had to join a merger with Hip Hop Since 1979, Jay's management team, and that afterwards was Wayne's allowance to gain access to 'higher echelon' entertainment placement. Before then, he was a rapper, and afterwards, he gets to be the artist. If you look at a lot of these big managment firms, you see not only who are in charge of the biggest rappers in the game, but you see also who dominates the 'buzzing' artists. Kevin Gates for example has been managed by Young Money for approaching half a decade. Others are brought in under artists managed by the larger tycoons, such as Lil' Durk becoming a part of French Montana's team while French transitions from Bad Boy to GOOD Music. The various rappers all become like sports rosters, and where is Keef?
His manager is his uncle, a former gangbanger who appears to constantly mismanage Keef's money. And has no relations to people in the industry as opposed to King L(ouie) who's manager is the former Kanye manager John Monopoly. So King L maintains a deal and high-profile collabs despite having NOOOOOO hits of significance since "Val Venis" especially when compared to Keef's consistent buzz and string of cult hits. And look at Bobby Shmurda, who is not only already signed and put out a quick EP to compliment his hot singles, but is being assigned to remix other major label artists like TeeFlii. His current manager? Sha Money XL (former?) manager of 50 Cent and other successful artists.
Now tell a label group of executives you're going to interact with people they don't have relationships with, who they don't WANT to talk to based on behavior and aren't able to see the hits in those singles Keef provides, despite the audience being clearly there to build off of.
Rap is so fucked right now it's amazing. As an artform its still incredibly vital, but the industry is being savaged by wild decisions.