PrismML Releases Bonsai 27B: 1-bit and Ternary Builds of Qwen3.6-27B That Run on Laptops and Phones

Introduction

In this tutorial, you'll learn how to work with low-bit language models, specifically the Bonsai 27B models released by PrismML. These models are compressed versions of the Qwen3.6-27B model that can run on regular laptops and phones, making powerful AI accessible to everyone. We'll explore how to load, run, and test these models using Python and Hugging Face's Transformers library.

Prerequisites

Before starting this tutorial, you should have:

• A computer with Python 3.8 or higher installed
• Basic understanding of Python programming
• Internet access to download model files
• At least 8GB of RAM (16GB recommended)

Step-by-Step Instructions

Step 1: Install Required Libraries

First, we need to install the necessary Python packages. The main library we'll use is Hugging Face's Transformers, which provides easy access to pre-trained models.

pip install transformers torch accelerate

Why: The transformers library provides a simple interface to load and run pre-trained models, while torch handles the deep learning operations, and accelerate helps with optimization.

Step 2: Import Required Modules

Now, let's create a Python script to import the necessary modules:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

Why: These modules allow us to tokenize text and load the model for inference. AutoTokenizer handles text processing, while AutoModelForCausalLM loads the language model.

Step 3: Load the Bonsai 27B Model

We'll start by loading the ternary version of Bonsai 27B. This model uses 1.71 bits per weight, making it much smaller than the original.

# Load tokenizer and model
model_name = "PrismML/Bonsai-27B-ternary"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16, device_map="auto")

Why: We specify torch.float16 to use half-precision floating point numbers, which saves memory and speeds up computation. The device_map="auto" automatically distributes the model across available devices.

Step 4: Test the Model

Let's create a simple test to see how the model works:

# Define a test prompt
prompt = "The future of AI is"

# Tokenize the input
inputs = tokenizer(prompt, return_tensors="pt")

# Generate text
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=50)
    generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)

print(generated_text)

Why: This code demonstrates the basic workflow: tokenize input text, generate new text using the model, and decode the output back to readable text.

Step 5: Try Different Models

PrismML also released a 1-bit version of Bonsai 27B. Let's see how to load that version:

# Load the 1-bit version
model_name_1bit = "PrismML/Bonsai-27B-1bit"
tokenizer_1bit = AutoTokenizer.from_pretrained(model_name_1bit)
model_1bit = AutoModelForCausalLM.from_pretrained(model_name_1bit, torch_dtype=torch.float16, device_map="auto")

Why: The 1-bit version uses binary weights (−1, +1) and is even more compressed than the ternary version, making it ideal for devices with very limited resources.

Step 6: Compare Performance

Let's create a simple performance comparison between the two models:

import time

# Test prompt
prompt = "Machine learning is"
inputs = tokenizer(prompt, return_tensors="pt")

# Time the ternary model
start_time = time.time()
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=30)
end_time = time.time()

print(f"Ternary model took {end_time - start_time:.2f} seconds")

# Time the 1-bit model
start_time = time.time()
with torch.no_grad():
    outputs = model_1bit.generate(**inputs, max_new_tokens=30)
end_time = time.time()

print(f"1-bit model took {end_time - start_time:.2f} seconds")

Why: This comparison helps us understand the trade-offs between model size and generation speed.

Step 7: Experiment with Different Prompts

Try different prompts to see how the models respond:

# Test various prompts
prompts = [
    "Artificial intelligence will",
    "The impact of technology on society",
    "Future of work in the age of AI"
]

for prompt in prompts:
    inputs = tokenizer(prompt, return_tensors="pt")
    with torch.no_grad():
        outputs = model.generate(**inputs, max_new_tokens=40)
        generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
    print(f"Prompt: {prompt}")
    print(f"Generated: {generated_text}\n")

Why: Testing with different prompts helps understand how the models handle various topics and contexts.

Step 8: Save Your Custom Generation

If you want to save your generated text to a file:

# Save generated text to file
with open("generated_text.txt", "w") as f:
    f.write(generated_text)

Why: This allows you to preserve and review the outputs from your model experiments.

Summary

In this tutorial, you've learned how to work with PrismML's Bonsai 27B models, which are compressed versions of Qwen3.6-27B designed to run on laptops and phones. You've installed the required libraries, loaded both ternary and 1-bit versions of the model, tested their performance, and generated text with different prompts. These low-bit models represent an exciting development in making powerful AI accessible to everyday users without requiring high-end hardware.