Making music with magenta.js

Magenta.js is a JavaScript library that helps you generate art and music on the web. In this tutorial, we'll talk about the music generation bits in @magenta/music -- how to make your browser sing, and in particular, how to make your browser sing like you!

As a library, @magenta/music can help you:

1. make music in the browser by having some neat abstractions over the WebAudio API.
2. use Machine Learning models to generate music in the browser.

Table of contents

• Step 0: First things first!
• Step 1: Making sounds with your browser
• Playing a NoteSequence
• Visualizing a NoteSequence
• Useful helpers
• Step 2: Using Machine Learning to make music
• MusicRNN
• MusicVAE
• Creating new sequences
• Interpolating between two sequences

Step 0: First things first!

If you're going to use Magenta, you need to add it to your page. Add this somewhere in your page's head element (it's also available as a module which you can install via npm):

<script src="https://cdn.jsdelivr.net/npm/@magenta/music@^1.0.0"></script>

Step 1: Making sounds with your browser

Everything in @magenta/music is centered around NoteSequences. This is an abstract representation of a series of notes, each with different pitches, instruments and strike velocities, much like MIDI.

For example, this is a NoteSequence that represents "Twinkle Twinkle Little Star". Try changing the pitches to see how the sound changes!

TWINKLE_TWINKLE = { notes: [ {pitch: 60, startTime: 0.0, endTime: 0.5}, {pitch: 60, startTime: 0.5, endTime: 1.0}, {pitch: 67, startTime: 1.0, endTime: 1.5}, {pitch: 67, startTime: 1.5, endTime: 2.0}, {pitch: 69, startTime: 2.0, endTime: 2.5}, {pitch: 69, startTime: 2.5, endTime: 3.0}, {pitch: 67, startTime: 3.0, endTime: 4.0}, {pitch: 65, startTime: 4.0, endTime: 4.5}, {pitch: 65, startTime: 4.5, endTime: 5.0}, {pitch: 64, startTime: 5.0, endTime: 5.5}, {pitch: 64, startTime: 5.5, endTime: 6.0}, {pitch: 62, startTime: 6.0, endTime: 6.5}, {pitch: 62, startTime: 6.5, endTime: 7.0}, {pitch: 60, startTime: 7.0, endTime: 8.0}, ], totalTime: 8 };

A special feature of NoteSequences is how they keep time. Sequences can be:

• unquantized -- which means the notes are defined as starting and ending at a particular time interval, in seconds. The "Twinkle Twinkle Little Star" sequence above was an example of this.
• quantized -- which means the notes are defined in terms of "steps"; the duration of the steps can be configured per quarter note. For example, in the unquantized sequence below, there are 4 steps per quarter note, and the duration is determined by the qpm (quarter notes per minute). Try adjusting the qpm value to hear the tempo change.

DRUMS = { notes: [ { pitch: 36, quantizedStartStep: 0, quantizedEndStep: 1, isDrum: true }, { pitch: 38, quantizedStartStep: 0, quantizedEndStep: 1, isDrum: true }, { pitch: 42, quantizedStartStep: 0, quantizedEndStep: 1, isDrum: true }, { pitch: 46, quantizedStartStep: 0, quantizedEndStep: 1, isDrum: true }, { pitch: 42, quantizedStartStep: 2, quantizedEndStep: 3, isDrum: true }, { pitch: 42, quantizedStartStep: 3, quantizedEndStep: 4, isDrum: true }, { pitch: 42, quantizedStartStep: 4, quantizedEndStep: 5, isDrum: true }, { pitch: 50, quantizedStartStep: 4, quantizedEndStep: 5, isDrum: true }, { pitch: 36, quantizedStartStep: 6, quantizedEndStep: 7, isDrum: true }, { pitch: 38, quantizedStartStep: 6, quantizedEndStep: 7, isDrum: true }, { pitch: 42, quantizedStartStep: 6, quantizedEndStep: 7, isDrum: true }, { pitch: 45, quantizedStartStep: 6, quantizedEndStep: 7, isDrum: true }, { pitch: 36, quantizedStartStep: 8, quantizedEndStep: 9, isDrum: true }, { pitch: 42, quantizedStartStep: 8, quantizedEndStep: 9, isDrum: true }, { pitch: 46, quantizedStartStep: 8, quantizedEndStep: 9, isDrum: true }, { pitch: 42, quantizedStartStep: 10, quantizedEndStep: 11, isDrum: true }, { pitch: 48, quantizedStartStep: 10, quantizedEndStep: 11, isDrum: true }, { pitch: 50, quantizedStartStep: 10, quantizedEndStep: 11, isDrum: true }, ], quantizationInfo: {stepsPerQuarter: 4}, tempos: [{time: 0, qpm: 120}], totalQuantizedSteps: 11 };

Playing a NoteSequence

When you pressed the "Play" button above, this started or stopped a Player. There are several kinds of players in @magenta/music-- the default player uses a built in "synth" sound to make the sounds. A different kind of player is a SoundFontPlayer, which lets you use real sounds for any of the notes played.

In the example below, try uncommenting the SoundFont player to see how that affects the sound of the NoteSequence. We're still using the "Twinkle Twinkle Little Star" from above, and whatever changes you've made to it are persisisting. If you accidentally broke the sequence, just refresh the page! 😅

player = new mm.Player(); // player = new mm.SoundFontPlayer('https://storage.googleapis.com/magentadata/js/soundfonts/sgm_plus'); // Players can also play at a different tempo //player.setTempo(200);

And you control the player with:

player.start(TWINKLE_TWINKLE); player.stop();

Players also have the ability to call a callback method after every note that is played. This is extremely useful if you want to update a visualization as a result of playing a NoteSequence, which we will see below.

Visualizing a NoteSequence

Listening to NoteSequences is great, but sometimes it's useful to look at a piano roll representing the notes. @magenta/music has a built-in Visualizer for that, which paints the notes to a canvas, and updates them using a Player's callback:

viz = new mm.Visualizer(TWINKLE_TWINKLE, document.getElementById('canvas')); vizPlayer = new mm.Player(false, { run: (note) => viz.redraw(note), stop: () => {console.log('done');} });

You can configure a visualizer's appearance, such as the size and colours of the notes. Try changing the values below and seeing how the piano roll is updated!

config = { noteHeight: 6, pixelsPerTimeStep: 30, // like a note width noteSpacing: 1, noteRGB: '8, 41, 64', activeNoteRGB: '240, 84, 119', } // Don't edit this line unless you want to break things. :) visualizer = new mm.Visualizer(TWINKLE_TWINKLE, document.getElementById('canvas'), config);

Useful helpers

There are a lot of other helper methods sprinkled around the @magenta/music codebase that you might need but not know where to find. Here are some of my favourites:

• converting between MIDI and NoteSequences
• quantizing an unquantized NoteSequence
• recording events from a MIDI instrument

Step 2: Using Machine Learning to make music

@magenta/music has several Machine Learning models, each with different strengths:

• MusicRNN - you give it a NoteSequence, and it continues it in the style of your original NoteSequence.
• MusicVAE - generates brand new NoteSequences or interpolates between two sequences.
• Onsets and Frames -- transcribes piano audio

The models are built with Tensorflow.js, so they run directly in the browser, using WebGL shaders (so that they won't be unbelievably slow)

Now that we know how to use NoteSequences and Players, adding some basic Machine Learning is a continuation of that. The pattern for using any of these models is:

1. Load @magenta/music (which we already know how to do!)
2. Create a model from a checkpoint (i.e. where the weights, or the encoding, of the model lives)
3. Ask the model to do something.

MusicRNN

A MusicRNN is an LSTM-based language model for musical notes -- it is best at continuing a NoteSequence that you give it.

To use it, you need to give it a sequence to continue -- when it's ready, the model will return a Promise containing the following sequence.

// Initialize the model. music_rnn = new mm.MusicRNN('https://storage.googleapis.com/magentadata/js/checkpoints/music_rnn/basic_rnn'); music_rnn.initialize(); // Create a player to play the sequence we'll get from the model. rnnPlayer = new mm.Player(); function play() { if (rnnPlayer.isPlaying()) { rnnPlayer.stop(); return; } // The model expects a quantized sequence, and ours was unquantized: const qns = mm.sequences.quantizeNoteSequence(ORIGINAL_TWINKLE_TWINKLE, 4); music_rnn .continueSequence(qns, rnn_steps, rnn_temperature) .then((sample) => rnnPlayer.start(sample)); }

With Music RNN, you can configure the number of steps the new sequence will be, as well as the "temperature" of the result -- the higher the temperature, the more random (and less like the input) your sequence will be. You can play around with these values and see how the resulting sequences are different:

rnn_steps = 20; rnn_temperature = 1.5;

Music RNN has other checkpoints you can use, which are trained on different melodies and instruments. For example, drum_kit_rnn makes new drum sequences.

MusicVAE

A MusicVAE is a variational autoencoder made up of an Encoder and Decoder -- you can think of the encoder as trying to summarize all the data you give it, and the decoder as trying to recreate the original data, based on this summarized version. As a generative model, you can think of a VAE as coming up with new sequences that could be a decoding of some summarized version.

The Music VAE implementation in @magenta/musicin particular does two things: it can create new sequences (which are reconstructions or variations of the input data), or it can interpolate between two.

Creating new sequences

// Initialize the model. music_vae = new mm.MusicVAE('https://storage.googleapis.com/magentadata/js/checkpoints/music_vae/mel_4bar_small_q2'); music_vae.initialize(); // Create a player to play the sampled sequence. vaePlayer = new mm.Player(); function playVAE() { if (vaePlayer.isPlaying()) { vaePlayer.stop(); return; } music_vae .sample(1, vae_temperature) .then((sample) => vaePlayer.start(sample[0])); }

As before, changing the temperature changes the randomness of the result.

vae_temperature = 1.5;

Interpolating between two sequences

// We are reusing the same music_vae and vae_player from the previous example. function playInterpolation() { if (vaePlayer.isPlaying()) { vaePlayer.stop(); return; } // Music VAE requires quantized melodies, so quantize them first. const star = mm.sequences.quantizeNoteSequence(TWINKLE_TWINKLE, 4); const teapot = mm.sequences.quantizeNoteSequence(LITTLE_TEAPOT, 4); music_vae .interpolate([star, teapot], 4) .then((sample) => { const concatenated = mm.sequences.concatenate(sample); vaePlayer.start(concatenated); }); }

---

You're now ready to build your own amazing, Machine Learning powered, music instrument! If you want more information, you can check out:

• some demos #MadeWithMagenta
• the documentation
• the Magenta blog, which talks about all the mathy bits we skipped.

Have fun! 💕