Creating step charts for video game franchise Dance Dance Revolution is a time-consuming task. The charts are essentially choreographic scores, instructing players where to place the feet in time with the music, and are usually written by hand, either by the game’s developers or fans using open source DDR port StepMania. Now, though, computer engineers have created a quicker way to generate step charts for any song using the power of neural networks.
In a paper published this week with the quite brilliant title Dance Dance Convolution, a trio of researchers from the University of California describe training a neural network to generate new step charts. Neural networks study data to analyze patterns and then create similar outputs, and in this case, there was abundant source of data in the form of fan-written step charts.
One of the study’s co-authors (and a longtime DDR fan) Chris Donahue told The Outline: “It sort of dawned on me one day that I had somewhere buried deep in my hard drive, gigabytes and gigabytes of data from this game StepMania, from a folder I’d been transferring from computer to computer since I was a teenager.” The researchers used two main datasets from different step chart authors, with the total training data spanning 35 hours of annotated music and more than 350,000 steps.
The trained neural net then generates new steps charts using a series of simple actions. First, targets songs are converted into spectrograms — visual representations of audio frequencies which the neural network uses to identify features like pitch and rhythm. Next, a step placement algorithm slices the song into 10 millisecond sample, and decides whether or not any single sample could contain a step based on relevant audio features. Then, a step selection algorithm maps these steps onto different moves to create a fully-fledged DDR step chart.
You can see the sort of choreography the network comes up with below. On the left is a step chart written by a human; on the right, the computer-generated version.
The results are perfectly human-playable, but, as with many creative forays by AI, professionals can still tell the difference. Speaking to The Register, step chart creator Fraxtil, who made many of the charts used to train this neural network, said “it’s pretty easy to tell that its output is synthetic.”
“There’s a lot of creativity involved in step charting, mainly selective use of repetition and contrast, that the AI either can’t learn or can’t apply effectively,” said Fraxtil. But he did add that, of all the attempts he’s seen to auto-generate step charts, this one was by far “the most successful iteration.”
Now we just need to create a robot that can perform DDR moves as well as humans.