Canary 1B

Model Architecture

Canary is an encoder-decoder model with FastConformer [1] encoder and Transformer Decoder [2]. With audio features extracted from the encoder, task tokens such as <source language>, <target language>, <task> and <toggle PnC> are fed into the Transformer Decoder to trigger the text generation process. Canary uses a concatenated tokenizer from individual SentencePiece [3] tokenizers of each language, which makes it easy to scale up to more languages. The Canay-1B model has 24 encoder layers and 24 layers of decoder layers in total.

Performance

In both ASR and AST experiments, predictions were generated using beam search with width 5 and length penalty 1.0.

ASR Performance (w/o PnC)

The ASR performance is measured with word error rate (WER), and we process the groundtruth and predicted text with whisper-normalizer.

WER on MCV-16.1 test set:

WER on MLS test set:

More details on evaluation can be found at HuggingFace ASR Leaderboard

AST Performance

We evaluate AST performance with BLEU score, and use native annotations with punctuation and capitalization in the datasets.

BLEU score on FLEURS test set:

BLEU score on COVOST-v2 test set:

BLEU score on mExpresso test set:

NVIDIA Riva: Deployment

NVIDIA Riva, is an accelerated speech AI SDK deployable on-prem, in all clouds, multi-cloud, hybrid, on edge, and embedded. Additionally, Riva provides:

• World-class out-of-the-box accuracy for the most common languages with model checkpoints trained on proprietary data with hundreds of thousands of GPU-compute hours
• Best in class accuracy with run-time word boosting (e.g., brand and product names) and customization of acoustic model, language model, and inverse text normalization
• Streaming speech recognition, Kubernetes compatible scaling, and enterprise-grade support

Although this model isn’t supported yet by Riva, the list of supported models is here.
Check out Riva live demo.

References

[1] Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition

[2] Attention is all you need

[3] Google Sentencepiece Tokenizer

[4] NVIDIA NeMo Toolkit

Licence

License to use this model is covered by the CC-BY-NC-4.0. By downloading the public and release version of the model, you accept the terms and conditions of the CC-BY-NC-4.0 license.