Liquid AI Open-Sources Pipette: A Reproducible Benchmarking Suite That Measures On-Device Models, Quantization, Runtime and Hardware Together

Introduction

In the world of artificial intelligence, models often perform well on powerful servers but struggle when deployed on mobile devices or edge hardware. This is where Pipette, an open-source benchmarking suite developed by Liquid AI, comes into play. Pipette helps developers understand how AI models behave in real-world, on-device conditions by testing them across different hardware, quantization levels, and runtime environments.

In this tutorial, you'll learn how to set up and use Pipette to benchmark a simple AI model on a simulated edge device. We'll walk through each step, from installing the tools to running your first benchmark, so you can start measuring model performance on devices like phones or tablets.

Prerequisites

To follow this tutorial, you'll need:

• A computer running Linux, macOS, or Windows
• Python 3.7 or higher installed
• Basic understanding of machine learning concepts (no prior experience with Pipette required)
• Access to a model file (we'll use a sample model in this tutorial)

Step-by-Step Instructions

1. Install Pipette

First, you'll need to install the Pipette library using pip. Open your terminal or command prompt and run:

pip install pipette

Why: This installs the core Pipette package that will allow us to benchmark models on different devices.

2. Prepare Your Model

For this tutorial, we'll use a simple pre-trained model. You can either use an existing model or create a dummy one for testing. Let's assume you have a model file named model.onnx in your working directory. If you don't have one, you can create a simple dummy model using the following Python code:

import torch
import torch.nn as nn

class DummyModel(nn.Module):
    def __init__(self):
        super(DummyModel, self).__init__()
        self.layer = nn.Linear(10, 1)

    def forward(self, x):
        return self.layer(x)

# Save the model
model = DummyModel()
torch.onnx.export(model, torch.randn(1, 10), "model.onnx", export_params=True)

Why: We need a model to benchmark. This dummy model will help us test Pipette's functionality without needing a complex AI model.

3. Create a Benchmark Configuration

Next, we'll define how we want to benchmark our model. Create a file named config.yaml with the following content:

model_path: "model.onnx"
hardware:
  - name: "mobile_cpu"
    type: "cpu"
    cores: 4
    memory: "4GB"
quantization:
  - type: "none"
  - type: "int8"
  - type: "float16"
runtime:
  - name: "onnxruntime"
    version: "1.15.0"
  - name: "tflite"
    version: "2.13.0"

Why: This configuration tells Pipette which model to test, what hardware to simulate, what quantization methods to use, and which runtimes to benchmark against.

4. Run the Benchmark

Now that everything is set up, we can run the benchmark. In your terminal, run:

pipette run --config config.yaml

Why: This command executes the benchmark process using the configuration file we created, testing the model across different hardware, quantization, and runtime conditions.

5. View Results

Once the benchmark finishes, Pipette will generate a report. You can view it by running:

pipette report

Why: This command displays the results of your benchmark, showing how the model performed under different conditions.

6. Interpret the Results

The output will include performance metrics such as:

• Inference time
• Memory usage
• Accuracy (if applicable)

These metrics help you understand how your model behaves on real devices. For example, a model might run fast on a server but be too slow or memory-heavy for a mobile device.

Why: Interpreting results is crucial for making decisions about model deployment and optimization. It helps you choose the best model configuration for your target hardware.

Summary

In this tutorial, we learned how to set up and use Pipette, an open-source tool for benchmarking AI models on edge devices. We installed Pipette, prepared a sample model, configured the benchmark settings, ran the benchmark, and interpreted the results. This process allows developers to make informed decisions about how to optimize their models for real-world deployment on devices like smartphones or tablets.

By understanding how models behave in different conditions, you can ensure that your AI applications perform well and efficiently in real-world scenarios.