Four days before the episode was supposed to air on MTV2, there was an unprecedented rejection: SpongeBob wasn’t allowed to sing in somebody else’s voice.

THQ, the publisher behind the video game SpongeBob SquarePants: Lights, Camera, Pants!, had approved the final cut. Blink-182 had given MTV the green light to use “All the Small Things.” But SpongeBob creator Stephen Hillenburg didn’t want to see his creations performing the song virtually — despite the fact that, by 2005, SpongeBob had already sung in two other episodes of the show.

“This isn’t even someone that we usually have to get an approval from,” Frank Drucker, former producer at animation production company IBC Digital, told Polygon in a recent interview. “This is a whole third party, and it’s Nickelodeon. You’re the same company as MTV. In your own house, you got a rejection four days before it was supposed to air! I’m like, This is nuts.”

In the original rough cut, Sandy Cheeks played bass, Patrick Star stomped the kick pedal of a seashell-shaped drum set, and SpongeBob shredded a flying-V guitar while singing. After the rejection, IBC switched the concept from a band performance to a dance contest. This allowed the team to keep roughly half of what it had made and replace the instrument segments with more dancing shots. “It was, like, no sleep for four days to remake the video,” Drucker said. “Sometimes we hit hurdles in not the most expected places.”

Airing for two seasons on MTV2 from 2004 to 2005, Video Mods married licensed songs with video games, making game characters perform the tracks as if they were part of fictional bands. Treated as real music performances, the videos featured multiple cameras and light shows. Bands such as The Killers, Outkast, and Evanescence were paired up with game franchises such as Silent Hill, Star Wars, and The Sims.

At the time of this story going live, most Video Mods episodes from season 1 and season 2 are available to watch on YouTube. Prior to that, the occasional user would upload low-quality versions obtained from TV recordings almost two decades old, and they would often receive copyright strikes. It wasn’t until a few years ago that former employees began a proper preservation effort of what they described during interviews with Polygon as a complex and creatively fulfilling show that was ahead of its time.

Polygon recently spoke to 14 people who worked on the show — ranging from those who came up with the concept to former employees across MTV and the animation studios Big Bear Entertainment and IBC Digital — to find out about how the show was made, the obstacles of trying to match motion capture with game assets, how it went collaborating with people such as George Lucas and the Beastie Boys, and ideas for a season that never happened.

The concept

It all started with Tony Shiff and Kris Renkewitz. Shiff, a music video producer who had worked on hundreds of videos for artists like Madonna and Michael Jackson, was well acquainted with the industry. He also had experience in early 3D animation for the likes of Def Leppard and Peter Gabriel, and in full-motion video for Electronic Arts. These experiences led him to meet the manager of House of Moves, a motion capture studio in Los Angeles, as well as former MTV head of programming Alex Coletti, with whom he worked on a series called Revue that consisted of interviews with artists like Ozzy Osbourne.

Renkewitz had years of experience as a penciller and colorist at Marvel Comics and DC Comics, and as an artist and designer for games like Diablo 2 and 007: Tomorrow Never Dies. According to Renkewitz, prior to presenting the idea to MTV, the duo had started work on a “game version” of Revue where you made music videos, called DIReCTOR, in conjunction with a programmer friend of his. The game got picked up by Atari, according to Shiff, yet when Infogrames acquired the company soon afterward, it shut down the project before the duo could assemble a crew. This, alongside a pitch for a Comedy Central show based on video game characters about their “life after work” named GAMERS, which was approved and then canceled after the head of programming was replaced, inspired the pair to combine elements of both concepts and think of a show for a third pitch.

Shiff, having kept in touch with Coletti, proposed a meeting to pitch MTV, in which he mentioned how video game companies would spend millions of dollars creating assets, such as for characters and environments, then not do anything else with them. Using BloodRayne as an example, Shiff created a pitch video that featured the characters playing “Chop Suey” by System of a Down to illustrate the potential for a show, and showed it to MTV.

“It was pre-Twitch,” Coletti said. “No one was smart enough to think you can just actually stream video games and people would watch. But we thought, Hey, if we could marry music, which is our bread and butter, with video games, which our audience loves in a unique way, then this is a no-brainer.”

Shiff and Coletti both sat down with David Cohn, former general manager at MTV2, who bought the idea for Video Mods. Then EA sponsored a pilot featuring Missy Elliott and Outkast paired up with SSX 3 and The Sims Bustin’ Out, respectively.

The pilot, which was 22 minutes long, was completed in three weeks in Renkewitz’s garage. He produced most of the animation, with some help from friends, while Shiff did the final edit. “I literally stayed up 48 hours at a time,” Renkewitz said. “I created a render farm from old machines. It was a ton of work, and some of the videos were kind of meh, but it was 22 minutes of animation by one person in three weeks. The pilot aired a gazillion times and it got us a four-episode order, where [Shiff] and I assembled a team in San Francisco that could now help me build the show.”

Season 1

Shiff and Renkewitz then started Big Bear Entertainment, building a team of around eight people. A partnership with Nvidia helped the company secure equipment to render videos with dozens of frames, shots, and characters in a way that hadn’t been done before. This was especially important for a small team who had a month to work on each episode, juggling multiple videos simultaneously.

“The day-to-day was a lot of firefighting,” said former Big Bear character modeler/rigger and texture artist Lisette Titre-Montgomery. “We had to take all these assets in from a game engine, convert them, set them up, and have something hot within a week or two, which is… when you think about how long an actual music video takes, that’s a lot of work.”

Maya Zuckerman, former motion capture technical director and compositor, described her time at Big Bear as the de facto startup experience of the early 2000s. The team would do 12- to 14-hour days, six to seven days a week, to keep up with the tight schedule. Securing and clearing the licenses, which MTV2’s Cohn described as a “swamp of rights” between record labels and game companies, would sometimes delay the work on a specific video. Once the assets were in and the song licenses were cleared, it would be a rush to get things set up.

“We had, like, 20 or 30 Nvidia towers rendering all of those videos,” Zuckerman said. “We actually had a render farm, and it was hot, because we were hosting it in an office in downtown San Francisco. Summer there isn’t so bad, but we would get a few days where we couldn’t even sit in the rooms because of [the heat].”

As the first year of Video Mods — which featured pairings like “Shut Up” by the Black Eyed Peas and Leisure Suit Larry: Magna Cum Laude — was winding down, many were aware that the show was overly complex to make, especially for a small team. And there weren’t indications that MTV would increase the budget.

The show got attention in Los Angeles, and Shiff said that, thanks to certain relationships, Tom Petty’s manager connected Shiff with a “high-profile and powerful music lawyer” (whom he declined to name) who had represented artists like Justin Timberlake and Patrick Stewart, alongside Petty. The lawyer offered to improve Shiff’s deal with MTV, saying that he should be getting more money. The problem, Shiff added — which he said he wasn’t aware of at the time — was that MTV “hated” the lawyer.

MTV ended up hiring a different studio to produce the second season of Video Mods. Shiff said he was paid during the first few weeks of the handover, but that marked the end of Big Bear Entertainment’s involvement — and the lawyer dropped Shiff as a client during the transition as well. “I created it, I sold it to MTV, and that’s what TV channels do,” Shiff said. “They can do that. They went for the cheapest solution.” When asked whether he thought it was unfair, he said he’s remained friends with the MTV staff since. “It’s just business,” he said.

MTV approached Frank Drucker, a producer and director at IBC Digital who was a former director at the network. Even though MTV trusted him, it asked for a test video to see if the team would be a good fit for what was presented as a quite unusual job.

IBC’s concept video, which was completed in around two days, featured the series Outlaw Golf. “It was a real mad dash to get that test done,” former IBC visual effects supervisor Daniel Smith said. “And then we presented it, and we won the job, basically.”

Season 2

IBC team members described Video Mods season 2 in many of the same ways Big Bear described season 1: MTV would obtain the rights from artists and game companies, get the song pairings and game assets to IBC, and then the team would produce the videos, finding similar struggles around software limitations and mocap not always aligning with complex character models. Instead of having an in-house render farm, however, IBC partnered with the University at Buffalo’s supercomputing center, which would render videos in around an hour. It would then take a full day to transfer all the frames to a portable hard drive, after which an IBC editor would put all the pieces together for another rendering pass.

Most companies and artists remained fairly hands off, only giving approvals on storyboards and rough cuts, but there was more direct input from certain parties during the second season. Aside from the aforementioned SpongeBob incident, LucasArts had a particular request in exchange for partnering with MTV and IBC to use footage from Star Wars: Battlefront 2 and Star Wars Episode III: Revenge of the Sith. Whereas IBC usually had the liberty to decide the direction of each Video Mod, a certain percentage of the LucasArts videos had to feature gameplay footage, which the company provided.

“It was very interesting to know that somewhere in a room at Skywalker Ranch, George Lucas had watched our rough cut and gave it a thumbs-up,” former IBC president Benjamin Porcari said. There was a similar scenario with Marc Ecko around the video for “Mountain Song” by Jane’s Addiction, which featured Marc Ecko’s Getting Up: Contents Under Pressure. The fashion designer and artist requested that the motion capture be authentic, and invited one of his graffiti artist friends to the studio for it. “It was incredible meeting him and hearing the stories from him on what it was like in the early days of graffiti culture,” Porcari said.

In addition, the Beastie Boys were adamant in directing the Video Mod for “Oh Word?” featuring the avatars of the band that were unlockable characters in NBA Street V3. They wanted to do the motion capture themselves, as the band insisted that no one was allowed to imitate them. They also requested a fish-eye lens, an effect that wasn’t possible with the software that IBC was using. The team had to mimic the look by distorting the image.

The workaround for motion capture proved more difficult. IBC worked with Elektrashock in Los Angeles, but the band’s schedule didn’t allow them to fly there. Meanwhile, no motion capture studios in New York would rent their spaces. As such, IBC had to set up a mocap room by shipping the equipment from Los Angeles, and one of the computers broke in transit.

All the computers had proprietary software, so the team couldn’t just replace the machine with any old hardware. They got in touch with the software company, which was based in Spain, and received a temporary license to use after finding a matching computer. The Beastie Boys spent most of the time just talking and making jokes with each other, waiting for the troubleshooting to finish. Eventually, they asked the team if it thought the issues would be fixed by the end of the day; IBC said it was certain of that. The band said that it was no problem, and that they’d return the next day.

The issue was that the temporary license, which had worked just fine that evening, expired at midnight. “They come the next morning and it doesn’t work,” Drucker said. “I’m like, What the hell? I’m cursed. So we had to get on the phone with the company, and they sent us another demo and we got it done.”

Everyone we talked to who worked on season 2 of Video Mods recalled their time fondly, noting the creative freedom of pitching concepts to MTV, as the company went with most of the team’s ideas. But as much as the process didn’t change much from the first season, neither did the long hours and tight deadlines. “At any time, we were juggling three balls at the same time,” Porcari said. “We had 80-, 90-hour weeks. We were going home just to sleep, working through weekends. It was pretty intense. But even then, I knew the minute it was over, I’d miss it.”

Unplugged

Quite a few ideas for Video Mods ended up on the cutting room floor. There was supposed to be one more episode for the second season, and there had been months of discussions for a third season.

Some of the artists being considered — the criteria being songs that had been in the top 10 most listened list but were on their way out — included Daddy Yankee, the Dixie Chicks, and 50 Cent. For the latter, Drucker described a meeting with members of G-Unit, who pitched ideas like “two gangs squaring off with machine guns” as well as a “whorehouse,” both of which would have “never made it past the censors” on TV. Not many recalled the games in consideration, with the exception of Gears of War.

Another question was where to take the show’s direction once the novelty began to wear off. Many people mentioned the site Machinima as an ongoing influence, particularly with having more user-generated content on the show. This had already happened once with the Video Mod for “So Cold” by Breaking Benjamin, featuring Half-Life 2. That video was originally made by Paul Marino, who told Polygon that he didn’t receive compensation, only exposure, but said that it helped him showcase the potential of platforms like Machinima to people outside the modding community.

At the end of the day, Video Mods wasn’t a ratings bonanza. “It wasn’t like Cribs, or a show that was getting picked up and talked about in papers or on Entertainment Tonight,” Coletti said. “It was a decision of where to allocate money, production money. It felt like after two seasons, the show was kind of what it was, and it wasn’t gonna do any better or worse.”

Most people who had worked on Video Mods stayed with Big Bear Entertainment and IBC Digital after the deals with MTV had ended; the studios continued operating until 2007 and 2013, respectively. Afterward, many went on to pursue jobs in the games industry, while others transferred their skills to TV and film. Interviewees for this piece went on to work for companies like HBO, BioWare, and Vice Media; left their touch on games like Psychonauts 2 and Prince of Persia: The Forgotten Sands; and were part of shows and films like The Great North and Insidious: The Red Door.

Now, two decades after Video Mods went off the air, Shiff and Renkewitz are working on what Shiff considers a “Video Mods reboot,” using the Unreal Engine and Renkewitz’s home motion capture setup. The project consists of two games, which the duo is hoping to pitch to game publishers for funding: Flashmobs and Supergroup. They’re working with professional dancers for Flashmobs and rock bands for Supergroup to create performances, with a focus on social media and influencers. Shiff mentioned being in talks with a “major metal band,” as well as working closely with dance influencers — not just for the games, but to create Video Mods-style clips for their TikTok and YouTube channels.

The virality that clips like SpongeBob’s “All the Small Things” have found in recent years provides a glimpse of what the reception could have been. People modding characters from other games, films, and TV shows into Guitar Hero, and the concept behind Fortnite Festival (and its potential for funny clips), are inherently keeping the Video Mods spirit alive.

“I think, 20 years later, it’s still an innovative concept, even though the graphics have changed and people have a higher fidelity of expectation,” Titre-Montgomery said. “I still believe that games are the best medium for self-expression, and what was really fun about Video Mods is that it was really positive; there was always something fun going on, and I think we’ve lost that in the video game space. It’s become really toxic and sad, and pockets of fun are found in different places, and I’d love for people to really think about how we can go back to that.”