Hot French startup ZML releases free product to speed inference across lots of AI chips

Introduction

In this tutorial, you'll learn how to leverage ZML's open-source inference optimization software to accelerate AI model execution across multiple hardware platforms. ZML/LLMD is designed to optimize large language model inference by automatically selecting the most efficient execution paths across different AI chips, potentially reducing costs and improving performance. This hands-on guide will walk you through setting up the ZML inference optimizer and running a sample model to demonstrate its capabilities.

Prerequisites

• Basic understanding of Python programming and machine learning concepts
• Access to a system with at least one GPU (NVIDIA recommended) or CPU
• Python 3.8 or higher installed
• Git installed for cloning repositories
• Basic familiarity with Docker (optional but recommended)
• Access to a Hugging Face account (for model downloading)

Step 1: Environment Setup and Installation

1.1 Clone the ZML Repository

First, we'll clone the ZML/LLMD repository from GitHub to access the core optimization software:

git clone https://github.com/ZML-LLMD/zml-llmd.git
 cd zml-llmd

This step gives us access to the core ZML inference optimization framework and its associated tools.

1.2 Create a Virtual Environment

It's good practice to create an isolated Python environment to manage dependencies:

python -m venv zml_env
source zml_env/bin/activate  # On Windows: zml_env\Scripts\activate

This ensures that ZML's dependencies don't conflict with your existing Python packages.

1.3 Install Required Dependencies

Install the necessary Python packages for ZML and model inference:

pip install -r requirements.txt
pip install torch transformers accelerate

These packages provide the foundation for running AI models and the optimization capabilities that ZML provides.

Step 2: Configure ZML Optimization Settings

2.1 Initialize ZML Configuration

Create a basic configuration file that tells ZML how to optimize your model execution:

import json

cfg = {
    "optimization_level": "auto",
    "target_hardware": "auto",
    "batch_size": 1,
    "precision": "fp16",
    "enable_quantization": True,
    "enable_kernel_fusion": True
}

with open('zml_config.json', 'w') as f:
    json.dump(cfg, f, indent=2)

This configuration enables automatic optimization level selection and kernel fusion, which are key features that help ZML optimize performance across different hardware.

2.2 Set Hardware Detection

ZML automatically detects available hardware. Verify that your system is properly detected:

from zml_llmd.core.hardware import HardwareDetector

hardware = HardwareDetector()
print(f"Detected hardware: {hardware.get_hardware_info()}")

This step ensures that ZML knows what hardware it's working with, which is crucial for making optimal optimization decisions.

Step 3: Model Loading and Optimization

3.1 Load a Sample Model

Load a pre-trained language model using Hugging Face Transformers:

from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "gpt2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# Set pad token for GPT-2
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

We're using GPT-2 as a sample model because it's lightweight and readily available, making it ideal for demonstrating ZML's optimization capabilities.

3.2 Apply ZML Optimization

Now we'll apply ZML's optimization to our loaded model:

from zml_llmd.core.optimizer import ModelOptimizer

# Initialize optimizer with our configuration
optimizer = ModelOptimizer(config_path='zml_config.json')

# Optimize the model
optimized_model = optimizer.optimize(model, tokenizer)
print("Model optimization complete!")

ZML's optimization process analyzes the model architecture and hardware capabilities to apply the most effective optimization techniques automatically.

Step 4: Run Inference with Optimized Model

4.1 Prepare Input Data

Create a sample input for inference:

input_text = "The future of artificial intelligence is"
input_ids = tokenizer.encode(input_text, return_tensors='pt')
print(f"Input tokens: {input_ids.shape}")

This creates a tokenized input that we can feed into our model for inference.

4.2 Execute Optimized Inference

Run inference using the ZML-optimized model:

from zml_llmd.core.inference import InferenceEngine

# Initialize inference engine
inference_engine = InferenceEngine(optimized_model, tokenizer)

# Run inference
with torch.no_grad():
    outputs = inference_engine.generate(
        input_ids,
        max_length=50,
        num_beams=1,
        temperature=0.8
    )

# Decode results
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(f"Generated text: {generated_text}")

This step demonstrates how ZML's optimization directly improves the inference performance while maintaining model accuracy.

Step 5: Performance Comparison

5.1 Benchmark Both Models

Compare performance between the original and optimized models:

import time

# Time original model
start_time = time.time()
with torch.no_grad():
    original_output = model.generate(input_ids, max_length=30)
original_time = time.time() - start_time

# Time optimized model
start_time = time.time()
with torch.no_grad():
    optimized_output = inference_engine.generate(input_ids, max_length=30)
optimized_time = time.time() - start_time

print(f"Original model time: {original_time:.4f}s")
print(f"Optimized model time: {optimized_time:.4f}s")
print(f"Speed improvement: {original_time/optimized_time:.2f}x")

This comparison shows the tangible performance benefits that ZML's optimization provides, which can be especially significant on larger models or production systems.

Summary

In this tutorial, you've learned how to implement ZML's inference optimization software to accelerate AI model execution. You've set up the ZML environment, configured optimization parameters, loaded and optimized a language model, and demonstrated performance improvements. The key advantages of using ZML include automatic hardware detection, kernel fusion optimization, and quantization techniques that can significantly reduce inference time while maintaining model accuracy. This optimization approach is particularly valuable for deploying large language models across diverse hardware platforms, as it automatically selects the most efficient execution paths for each specific system configuration.

Remember that ZML's effectiveness can vary depending on your specific hardware setup and model complexity. For production deployments, consider running more extensive benchmarks to determine the optimal configuration for your particular use case.