Keystroke's Muse cut a guild music video to the beat with ffmpeg

Keystroke wanted a music video for his guild in the game ESO. He created a song for it with Suno, the AI music generator, and handed the rest to his Muse agent, Skynet. Skynet downloaded the guild’s game clips, analyzed the song, cut the clips to its length with transitions landing on the beat, lined up the lyrics, and published the result to a YouTube channel it runs. The video is titled “Legendary Bro: Be Chill. Be a Bro.”

The notable detail is where the editing happened. “ffmpeg was used from inside the muse VM directly,” Keystroke wrote on October 30. The Muse VM is the virtual machine where a Muse agent can install and run its own software, and ffmpeg is the standard command-line tool for cutting and encoding video. No desktop editor was involved.

How it came together

Keystroke’s summary in the server’s showcase forum lays out the division of work. He made the song in Suno. He also set Muse up so it could create videos and publish them to YouTube. Skynet did the rest: downloading the clips, analyzing the track, timing each cut to the music and aligning the lyrics on screen.

The YouTube channel is one of several accounts Skynet controls. Back on October 16, Keystroke said he had set up a Gmail account, an Instagram account, a YouTube channel and a private Discord server for Skynet. “He has full control over them. They grow up so fast,” he wrote. Skynet has used the Instagram account to post a series of parody photos, including a gym selfie.

The guild is also where Skynet started. Keystroke built it to run his ESO guild’s Discord, posting events, polls and trials, and the guild’s members voted on its name. “Members voted to name him Skynet as we were already meming about,” he explained when he joined the Muse Agent server. In the general channel he introduced the new video the same way: “I had Skynet create a YouTube channel and made a music video for my ESO guild.”

What the server said

Chonky, one of the server’s admins, commented the next day and singled out the timing. “The fact that the clips align with the beats is great,” he wrote, “(at least at first…i think it slipped later in the video).”

gumdean said he could not watch it right away, then came back to say he liked it. He had been working in the same direction. Two days earlier he posted a Halloween music video for which his Muse wrote a song and told him what prompts to use in Suno; he noted that was made “way before I know what I do now.” On November 1 he posted another, “a catchy little tune written by Quasar,” his Muse, “and brought to life with Suno,” with the video also made by Quasar.

The two projects sit at different points on the same path. In gumdean’s first video Muse wrote the words and coached the human through the tools. In Keystroke’s, the human supplied the song and the agent did the editing and the publishing itself.