What is the AI that turns music into sheet music?

Artificial intelligence has had a long and fascinating history intersecting with music. Early applications of AI in music focused on music transcription, or converting audio recordings into sheet music notation. In 1897, the invention of the oscilloscope allowed researchers to visualize sound waves for the first time. This sparked continued advancements in transcribing recorded music into written notation. In the 1950s-60s, early computer systems were developed that could transcribe monophonic melodies with limited accuracy. Since then, rapid progress in machine learning and neural networks has enabled AI transcription systems to more accurately parse polyphonic music and handle complex instrumentation. Today’s AI can not only transcribe music, but also generate original compositions, remix tracks, and understand nuances of musical expression. This article will provide an overview of the leading AI technology that automates music transcription.

What is Music Transcription?

Music transcription is the process of converting recorded audio into musical notation in the form of sheet music or tablature (Verbit.ai, 2022). It involves listening to a piece of music and manually notating the melody, harmony, rhythm, lyrics, and other musical elements by ear into a readable format like standard musical notation, guitar tabs, or MIDI files (Espresso Translations, 2022).

Transcription can be applied to all genres and styles of music, from classical and jazz to rock and pop. It enables music to be preserved, analyzed, edited, and performed in written form by musicians around the world (Routenote, 2022). The end result of music transcription is a detailed musical score that recreates what was played in the recording. This allows others to study, play, or arrange the music.

How Does the AI Work?

AI-powered music transcription relies on advanced machine learning algorithms that are trained on large datasets of audio recordings and corresponding sheet music. The AI analyzes the audio signal, detects notes, rhythms, instrumentation, and other musical elements. It then generates a musical score that represents what it hears in the audio.

Some of the key steps in the transcription process include:

• Splitting the audio into short frames, like 0.1 seconds long.
• Extracting audio features like pitch, timbre, rhythm, melody for each frame using signal processing.
• Identifying onsets – when notes begin – and durations of each note.
• Classifying the instrument making each note using machine learning models.
• Applying music language models to create musically plausible transcriptions.
• Converting the identified notes, rhythms, instruments into standard musical notation.
• Refining the transcription using scoring, error checking, and post-processing rules.

The AI continuously improves its transcription abilities through deep learning techniques as it processes more audio data. However, it still struggles with complex polyphonic music and accurately inferring note durations. The technology continues to advance but has limitations in handling ambiguous musical elements.

Accuracy and Limitations

Current AI transcription tools can achieve decent accuracy when transcribing simple melodies and instrumentals, but still struggle with more complex polyphonic music (music with multiple melodic parts). According to research from Stanford University, AI-based music transcription tools had 73% accuracy at identifying the correct notes in monophonic music (single melodic lines), but just 58% accuracy for polyphonic music (source).

There are also challenges transcribing the timing of notes correctly, with tools often struggling to identify precise rhythm and tempo. Research suggests AI transcription has around 80-85% accuracy at placing notes at the right time (source).

Accuracy at identifying instrumentation and orchestration is also limited. Most systems are focused solely on the pitch content and struggle to distinguish between instruments.

Perhaps the biggest limitation is transcribing vocals and lyrics from songs. According to AssemblyAI, background instrumentation dramatically reduces accuracy: “If we were to strip out the background noise/music, and isolate just the vocals, we’d expect the lyrics to be transcribed much more accurately” (source). So transcription works best on acapella recordings.

Overall, AI music transcription is still an emerging technology with room for improvement. But accuracy continues to increase each year as the machine learning models are refined with more training data. Most experts believe AI will reach professional music transcription quality within 5-10 years.

Major AI Transcription Tools

There are a few leading AI transcription tools that can convert music audio into sheet music and guitar tabs with incredible speed and accuracy thanks to advancements in artificial intelligence and deep learning algorithms:

AnthemScore

According to source, AnthemScore by Lunaverus is one of the top ranked AI music transcription technologies. Key features include:

• Transcribes polyphonic music
• Extracts multiple instruments parts into separate staves
• Supports a range formats including MP3, M4A, WAV, etc
• Outputs formats such as MIDI, MusicXML, guitar pro tab, etc
• Very fast processing time

Noteflight

Noteflight is another powerful online music notation service that utilizes AI algorithms to transcribe music uploaded in MP3 format into sheet music. Key highlights:

• Browser-based so works on any device
• Transcribes both melody and harmony into sheet music
• Syncs sheet music display with original audio track
• Tools to edit notes, playback audio, print and share sheet music

AnthemScore

AnthemScore is an AI-powered software that automatically transcribes audio files like MP3s and WAVs into musical notation (AnthemScore Review, Pricing and Alternatives 2023). It utilizes advanced machine learning algorithms to analyze audio tracks and convert them into sheet music that can be edited.

Some key features of AnthemScore include the ability to recognize multiple instruments in a song, identify key signatures and time signatures, and export sheet music as MusicXML, MIDI, and PDF files. The software also allows you to make edits to fix any transcription errors.

In testing, AnthemScore has proven highly accurate in transcribing simple piano melodies and guitar riffs. However, it still struggles with more complex polyphonic music across multiple instruments. The software works best with clearly recorded audio without too much background noise.

Overall, AnthemScore is a powerful AI tool that makes transcribing music significantly easier. It can save musicians, composers, and students many hours of manual notation. However, the technology still has limitations, so human review of transcriptions is recommended (Unleash Your Music Potential with AI: AnthemScore Review).

Noteflight

Noteflight is an AI-powered music notation software that allows users to turn recorded music into musical scores through audio transcription ¹. It utilizes advanced machine learning algorithms to listen to audio files and automatically determine the pitch, rhythm, tempo, key signature, and more to generate sheet music.

The transcription engine in Noteflight leverages deep neural networks trained on a large dataset of audio recordings mapped to their corresponding sheet music. This allows it to accurately identify notes, chords, and other musical elements from new recordings. Even for complex polyphonic music with multiple instruments, Noteflight can produce high quality transcriptions.

Reviews of Noteflight have been generally positive, with the full-featured program complemented for its transcription capabilities, although considered a little expensive given limited budgets ². The free version, however, provides sufficient functionality for more basic needs. Some users found the extensive options overwhelming initially.

Applications

Music transcription AI has opened up new opportunities for musicians, researchers, composers, educators, and other professionals to work with sheet music. Some key applications include:

Musicians use transcription tools to convert their improvisations, compositions, and recordings into sheet music or guitar tabs. This allows them to analyze their playing, publish scores, teach students, register copyright, and more. Popular tools like AnthemScore generate results faster and more accurately than attempting to manually notate music.

Music researchers employ AI transcription to convert large volumes of audio recordings into symbolic notation. By systematizing melodic and rhythmic content into musical scores, researchers gain expanded capabilities for analyzing patterns and structures within genres, artists’ catalogs, or other groupings.

Composers apply intelligent transcription tools as aids for arranging and orchestrating their ideas. assisted by AI, composers can swiftly translate their mental musical concepts into written compositions ready for human refinement.

Educators supplement instruction with automatically generated scores depicting students’ performances, model interpretations of pieces, historical recordings, and other educational materials. AI-enabled assessment provides students detailed personalized feedback.

AI music transcription uproots long-standing constraints around working with sheet music, bringing written notation into closer parity with audio. Continued progress promises more seamless integration between the auditory and symbolic dimensions of music.

Future Outlook

AI music transcription is an emerging technology that is expected to continue improving in accuracy and capability. According to one source, future AI systems may be able to transcribe more complex musical arrangements with multiple instruments and vocals. Machine learning techniques will enable the systems to better handle nuances like expression and articulation markings.

Innovations in the field could lead to new creative tools for musicians, composers, and music producers. For example, advanced transcription coupled with generative models may one day allow realistic-sounding instrumental or vocal parts to be generated from sheet music. This could save time and provide more flexibility during music production. AI transcription could also open up new applications in music education, analysis, and accessibility.

While current AI transcription tools have limitations, rapid progress in machine learning will likely lead to accurate transcription of more diverse musical styles. As the technology continues advancing, it promises to shape the future of how music is created, learned, and enjoyed around the world.

Conclusion

AI tools that transcribe music into sheet music have come a long way, providing useful assistance to musicians and music learners. While not perfect, the transcriptions can be very close to what a human would notate. AnthemScore and Noteflight are leading options among a growing landscape of automated music transcription tools.

The AI technology behind these solutions leverages machine learning models trained on vast libraries of sheet music and audio files. This allows the systems to recognize instruments, melodies, chords, tempo, time signatures and translate the music into notation. The more data they train on, the smarter the models get.

There remain some key limitations currently, especially with handling more complex polyphonic pieces, but the transcriptions work well for simpler compositions. As research continues, accuracy will further improve.Overall, AI-powered music transcription offers an efficient way to notate compositions, supporting creative workflows and music practice. We can expect even more breakthrough innovations in this space as the underlying technology advances.