Remember the #BachDoodle? We’re excited to release paper on Behind-the-Scenes design, #ML, scaling it up, and dataset of 21.6M melodies from around the world!
📜 arxiv.org/abs/1907.06637
w/ @fjord41 @ada_rob @notwaldorf @bengiswex Leon Hong @jaxcooo

tl; dr
1/ In three days, people spent 350 years worth of time playing with the Bach Doodle, and the “harmonize” button was clicked more than 55 million times.

2/ The model Coconet 🥥 is an instance of OrderlessNADE and uses Gibbs sampling to generate the harmonizations through rewriting.
📜 Previous blogpost: g.co/magenta/coconet
📝 Paper from #ISMIR 2017: arxiv.org/abs/1903.07227

3/ We sped up Coconet from 40s to 2s in #TensorFlowJS by using dilated depth-wise separable convolutions, which requires less layers and are more accelerated then conventional convolutions. Also by fusing ops that are used in every Gibbs steps (@GreenBeanDou).

4/ We reduced the model download size to approximately 400KB through post-training weight quantization.

https://twitter.com/nsthorat/status/1109231029036949504

5/ We calibrated a speed test based on partial model evaluation time to determine if the harmonization request should be performed locally using #TensorFlowJS or sent to remote #TPU servers. @jaxcooo was able to get a nearly 50/50 split!

6/ As the model was trained on Bach Chorales, melodies outside of the soprano range is “out of distribution” and harder to harmonize.

See how Paul Davids discovers this at 6:22 via @YouTube

7/ We found that parallel fifths and octaves were more common when user input was out of distribution, and fewer parallel fifths and octaves were correlated with positive user feedback.

8/ Checkout @notwaldorf’s amazing interactive visualizations to explore the 21.6M melodies, the dataset and also hear new harmonizations.

https://twitter.com/notwaldorf/status/1151552258191941633