NVIDIA Releases Nemotron 3.5 Transcribe: An Open 30B MoE Model for 40-Language Transcription with ~4B Active Parameters
Back to Tutorials
aiTutorialintermediate

NVIDIA Releases Nemotron 3.5 Transcribe: An Open 30B MoE Model for 40-Language Transcription with ~4B Active Parameters

October 28, 202645 views4 min read

Learn how to use NVIDIA's open-source Nemotron 3.5 Transcribe model for multilingual speech transcription with speaker labeling and word timestamps.

Introduction

In this tutorial, we'll explore how to use NVIDIA's Nemotron 3.5 Transcribe, an open-source speech language model that supports transcription in 40 languages with speaker labeling and word timestamps. This model is particularly useful for developers working on speech-to-text applications, multilingual transcription systems, or voice analytics. We'll walk through setting up the environment, loading the model, and performing transcription tasks with practical examples.

Prerequisites

  • Python 3.8 or higher
  • Basic understanding of machine learning and natural language processing concepts
  • Access to a machine with GPU (recommended for optimal performance)
  • Installed packages: transformers, torch, librosa, soundfile

Step-by-Step Instructions

1. Environment Setup

1.1 Install Required Dependencies

We'll start by installing the necessary Python packages. The transformers library is crucial for loading the model, while librosa and soundfile help with audio processing.

pip install transformers torch librosa soundfile

Why: These packages provide the core functionality needed to load and interact with the Nemotron model and process audio files.

1.2 Verify GPU Availability

Ensure your system has CUDA support for optimal performance.

import torch
print(torch.cuda.is_available())

Why: The Nemotron model is computationally intensive, and using GPU acceleration will significantly speed up transcription tasks.

2. Loading the Nemotron Model

2.1 Load Model from Hugging Face

NVIDIA has made Nemotron 3.5 Transcribe available through the Hugging Face model hub. We'll load it using the transformers library.

from transformers import AutoProcessor, AutoModelForSpeechSeq2Seq

# Load the processor and model
model_id = "nvidia/nemotron-3.5-transcribe"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id)

# Set device
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)
model.eval()

Why: Using the AutoProcessor and AutoModelForSpeechSeq2Seq classes automatically handles the model configuration and ensures compatibility with the Nemotron architecture.

2.2 Configure Model Settings

Set the model's generation parameters for optimal transcription results.

generation_config = {
    "max_new_tokens": 256,
    "do_sample": False,
    "num_beams": 1,
    "temperature": 0.0,
    "language": "en",
    "task": "transcribe"
}

Why: These parameters control how the model generates text, with do_sample=False ensuring deterministic outputs and language set to English for this example.

3. Audio Processing and Transcription

3.1 Load and Preprocess Audio

We'll use librosa to load audio files and prepare them for model input.

import librosa
import numpy as np

# Load audio file
audio_path = "sample_audio.wav"
raw_speech, sampling_rate = librosa.load(audio_path, sr=16000)

# Convert to tensor
input_values = processor(raw_speech, sampling_rate=16000, return_tensors="pt").input_values
input_values = input_values.to(device)

Why: The model expects audio at 16kHz sampling rate and specific tensor format. librosa handles the audio loading and resampling.

3.2 Perform Transcription

Generate the transcription using the model's generate method.

# Generate transcription
with torch.no_grad():
    generated_tokens = model.generate(input_values, **generation_config)
    transcription = processor.batch_decode(generated_tokens, skip_special_tokens=True)

print("Transcription:", transcription[0])

Why: The generate method applies the model's language understanding to convert audio features into text, while batch_decode converts tokenized output back to readable text.

4. Advanced Features

4.1 Multilingual Support

Update the generation config to transcribe in different languages.

# For Spanish transcription
spanish_config = generation_config.copy()
spanish_config["language"] = "es"

with torch.no_grad():
    generated_tokens = model.generate(input_values, **spanish_config)
    spanish_transcription = processor.batch_decode(generated_tokens, skip_special_tokens=True)

print("Spanish Transcription:", spanish_transcription[0])

Why: The model supports 40 languages, making it versatile for international applications. Adjusting the language parameter allows you to transcribe in the desired language.

4.2 Extract Speaker Labels and Timestamps

For speaker diarization and timing information, modify the output processing.

# Enable detailed output
model.config.forced_decoder_ids = None
model.config.suppress_tokens = []

# Generate with detailed timing
with torch.no_grad():
    generated_tokens = model.generate(input_values, **generation_config)
    detailed_output = processor.batch_decode(generated_tokens, skip_special_tokens=True)

print("Detailed Output:", detailed_output[0])

Why: The model can provide word-level timestamps and speaker information, which is valuable for applications requiring precise timing or speaker separation.

5. Practical Application

5.1 Batch Processing Multiple Files

Create a function to process multiple audio files.

import os

def transcribe_batch(audio_directory):
    transcriptions = {}
    for filename in os.listdir(audio_directory):
        if filename.endswith(".wav"):
            file_path = os.path.join(audio_directory, filename)
            raw_speech, _ = librosa.load(file_path, sr=16000)
            input_values = processor(raw_speech, sampling_rate=16000, return_tensors="pt").input_values
            input_values = input_values.to(device)
            
            with torch.no_grad():
                generated_tokens = model.generate(input_values, **generation_config)
                transcription = processor.batch_decode(generated_tokens, skip_special_tokens=True)
                
            transcriptions[filename] = transcription[0]
    return transcriptions

# Usage
results = transcribe_batch("/path/to/audio/files")
for filename, text in results.items():
    print(f"{filename}: {text}")

Why: Batch processing allows you to handle multiple audio files efficiently, which is essential for real-world applications with large datasets.

Summary

In this tutorial, we've learned how to set up and use NVIDIA's Nemotron 3.5 Transcribe model for speech transcription. We covered loading the model from Hugging Face, preprocessing audio files, performing basic and multilingual transcription, and implementing batch processing. The model's ability to handle 40 languages with speaker labeling and word timestamps makes it a powerful tool for various speech processing applications. Remember to ensure GPU availability for optimal performance and adjust the generation parameters based on your specific requirements.

Source: MarkTechPost

Related Articles