Workstream Proposal - Decomposition

We would like to automatically decompose any song into its constituent components. In rough order of importance this includes stems, samples, midi, lyrics, and effects. A full decomposition should contain similar information as an ableton project. Our decomposition ability will form the foundation of our editing capabilities. We now have enough stem and sample data to make such a project feasible.

What is a Decomp?

A decomp is a file format similar to an ableton project file. It contains everything needed to construct the final song audio, as well as metadata. The more granular we can make the components the better.

Concretely a v0 decomp should include labeled stems and samples. v1 may add perfectly aligned lyrics. v2 may add per stem MIDI transcripts. v4 may decompose stems further into dry samples with effect chains.

Generality and accuracy are important for decomps to serve as a foundation for downstream use cases like editing and visualization. We will need to largely "solve" several grand challenges of MIR.

How

Synthetic data.

Now I know what you're thinking, where is Victor and what have you done to him? I trapped him in a metal tube at 30,000 ft. But hear him out.

We now have a medium sized library of rerecorded karaoke stems as well as a large library of all kinds of samples. All have good labels. With just the karaoke stems we saw surprising generalization to real songs. By mixing in samples in moderately musical ways we can construct synthetic songs to train our decomposition model.

This type of synthetic data should be fine for the decomposition task. Since the stems and samples themselves are high quality, even if most mixes are musically dubious real music should still be in distribution.

Effects can be synthetically added randomly as well.

To create synthetic mixes, I suspect random mixing will suffice. If not, LLM's may be able to select reasonable things to mix.

Why

Make wavtool great again!

To spell out a few capabilities decomposition would unlock:

• Reliable, granular, general, source separation
• Conditioned on some stems, add a bass track
• Add a guitar solo in the style of artist X
• Cover your guitar/vocals into a professional recording
• Create a song utilizing a few samples you recorded
• Know which samples are used in a song, and where else they appear
• More powerful search based on individual stems
• Input for visualizers
• MIDI transcription becomes more tractable
• We have another tool for attribution

Stem control is the one of, if not the top requested feature from artists. I believe all artists would find such tools useful, and could be convinced to upload their songs to Suno.

A successful decomp would deeply impress the MIR community and obsolete quite a few companies.