← Home

Workstream Proposal - MIDI

December 2, 2024

We have line of sight to general and robust automatic midi transcription. MIDI is the best way for computers to understand music; MIDI can serve as the foundation for music visualizers and attribution. It also enables precise control of notes in a language musicians understand.

Why

Visualization

A fundamental limitation of audio exploration is that it must be listened to in real time, as opposed to glanced over like images. The grand challenge of the field is to design a visual representation of music that allows users to make fast informed decisions on what to listen to next. I believe MIDI is a necessary component of such a visualizer.

MIDI is an ideal format for visualization. Light shows are a trivial extension of the MIDI language. I validated this by converting beatsaber maps into the MIDI specification, and successfully trained GPT style models on it.

ChatBots

MIDI can be fed into LLMs in raw text form. With finetuning, I expect LLM's can describe, edit, and generate MIDI based on natural language instructions.

Control

The most obvious use case is MIDI control for generations. This becomes more interesting combined with stem control. You will be able to create each stem with complete control of pitch and rhythm.

Attribution

Though imperfect, MIDI is a legible way of measuring the similarity between two songs. With general transcription this can be scaled to all recorded music. MIDI is likely to be a large factor in the future of automated attribution.

How

MT3 showed that transcription can be formulated as a language modeling problem. This greatly differs from far more common approach of producing a transcript in a single model pass. Being autoregressive, MT3 is far more expensive than its competition. However, autoregression allows the use of noisy and diverse data during pretraining. MT3 was trained on every accessible dataset, and achieved SOTA yet unusable results.

I reproduced MT3 and scaled up the data by 1-2 OOMs. Much of this data was extremely noisy, but importantly it was real non-synthetic data. The resulting model was able to understand pretty much any song I tried, and output a reasonable MIDI arrangement of it. I named the model arranger since its best at arranging a song to any subset of instruments while taking artistic liberties. You can listen to the results at https://platers.github.io/arrange_demo.

Arranger is cool, but isn't a reliable transcriber due to the noisy pretraining data. The obvious solution is to fine tune it on some clean data. I estimate it would cost $10-40k to acquire enough clean data to solve the problem.

Its also possible that recent advancements in source separation can also solve the reliability issue. Arranger is more reliable on single stems, and its outputs can be more easily checked allowing for filtering of the dataset or DPO.

Roadmap