Liquid AI Releases LFM2.5-VL-3B: A 3B Vision-Language Model That Reads Screens, Grounds Objects, and Calls Tools On-Device

Introduction

In this tutorial, you'll learn how to work with vision-language models (VLMs) that can process visual input and generate text responses, similar to the LFM2.5-VL-3B model released by Liquid AI. This model is designed for on-device deployment, meaning it can run locally on hardware like Apple M5 Max without requiring cloud connectivity. You'll build a simple application that demonstrates core capabilities like screen reading, object grounding, and function calling using Python and Hugging Face's Transformers library.

By the end of this tutorial, you'll have a working prototype that can process images, understand visual content, and simulate tool calling—key features of modern VLMs.

Prerequisites

• Python 3.8 or higher installed on your system
• Basic understanding of machine learning concepts and Python programming
• Access to a machine with internet connectivity (for downloading models)
• Optional: Apple M5 Max or similar hardware for on-device inference (not required for this tutorial)

Step-by-Step Instructions

1. Set Up Your Python Environment

First, create a new virtual environment and install the required packages:

python -m venv vlm_env
source vlm_env/bin/activate  # On Windows: vlm_env\Scripts\activate
pip install transformers torch pillow

Why: This creates an isolated Python environment to avoid conflicts with other projects. The transformers library provides easy access to pre-trained models, while torch handles the deep learning operations.

2. Download a Pre-trained Vision-Language Model

We'll use a lightweight model like facebook/blip2-opt-2.7b as a demonstration. While it's not exactly LFM2.5-VL-3B, it demonstrates similar functionality:

from transformers import Blip2Processor, Blip2ForConditionalGeneration

processor = Blip2Processor.from_pretrained("facebook/blip2-opt-2.7b")
model = Blip2ForConditionalGeneration.from_pretrained("facebook/blip2-opt-2.7b")

Why: This downloads and loads a pre-trained model that can generate text descriptions of images, which is foundational to vision-language tasks.

3. Prepare an Image for Processing

Load an image and prepare it for input to the model:

from PIL import Image
import requests

# Load image from URL or local file
image_url = "https://example.com/sample_image.jpg"
image = Image.open(requests.get(image_url, stream=True).raw)

# Process image
inputs = processor(image, return_tensors="pt")

Why: The processor converts the image into a format the model can understand, including resizing and normalization.

4. Generate Text Description from Image

Use the model to generate a text description of the image:

generated_ids = model.generate(**inputs, max_new_tokens=20)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
print(generated_text)

Why: This simulates the model's ability to read and interpret visual content—similar to how LFM2.5-VL-3B reads screens.

5. Simulate Object Grounding

Object grounding involves identifying specific items in an image. While the model doesn't directly support this, we can simulate it by prompting:

# Example prompt for grounding
prompt = "What is the red ball in the image?"
inputs = processor(image, prompt, return_tensors="pt")
generated_ids = model.generate(**inputs, max_new_tokens=10)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
print(generated_text)

Why: This demonstrates how models can be prompted to focus on specific visual elements, mimicking the grounding capabilities of LFM2.5-VL-3B.

6. Implement Function Calling Simulation

Function calling allows models to interact with external tools. Here's a simple simulation:

def simulate_tool_call(tool_name, parameters):
    print(f"Calling tool: {tool_name} with parameters: {parameters}")
    # Simulate tool response
    return f"Tool {tool_name} executed with {parameters}"

# Example function call
tool_response = simulate_tool_call("get_weather", {"location": "New York"})
print(tool_response)

Why: This simulates how LFM2.5-VL-3B can call tools on-device, enabling actions like retrieving data or controlling hardware.

7. Combine All Components into a Single Pipeline

Now, let's create a simple pipeline that integrates all components:

def vlm_pipeline(image_path):
    # Load image
    image = Image.open(image_path)
    
    # Generate description
    inputs = processor(image, return_tensors="pt")
    generated_ids = model.generate(**inputs, max_new_tokens=20)
    description = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
    
    # Simulate grounding
    grounding_prompt = "Identify the main object in the image"
    inputs = processor(image, grounding_prompt, return_tensors="pt")
    generated_ids = model.generate(**inputs, max_new_tokens=10)
    ground_object = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
    
    # Simulate tool call
    tool_response = simulate_tool_call("analyze_image", {"object": ground_object})
    
    return {
        "description": description,
        "grounded_object": ground_object,
        "tool_response": tool_response
    }

# Run pipeline
result = vlm_pipeline("sample_image.jpg")
print(result)

Why: This creates a cohesive workflow that mimics the capabilities of LFM2.5-VL-3B—processing images, grounding objects, and calling tools.

Summary

In this tutorial, you've learned how to work with vision-language models using the Hugging Face Transformers library. You've simulated core capabilities like screen reading, object grounding, and function calling—features that make models like LFM2.5-VL-3B powerful for on-device AI applications. While this tutorial uses a smaller model for demonstration, the principles apply to larger, more advanced models like those from Liquid AI.

Next steps could include exploring more advanced models, optimizing for on-device performance, or integrating with actual hardware APIs for real tool calling.