Introduction
In this tutorial, you'll learn how to work with the Inkling model released by Thinking Machines Lab. Inkling is a 975 billion parameter multimodal model that can process text, images, and audio inputs. It's designed as a customization base with controllable thinking effort, making it perfect for learning how to work with large-scale AI models. We'll walk through setting up your environment, loading the model, and running basic inference tasks with text and image inputs.
Prerequisites
Before starting this tutorial, you'll need:
- A computer with at least 16GB of RAM (32GB recommended)
- Python 3.8 or higher installed
- Basic understanding of machine learning concepts
- Access to a GPU with at least 8GB VRAM (NVIDIA recommended)
Step-by-Step Instructions
1. Set Up Your Python Environment
First, create a virtual environment to keep your project dependencies isolated:
python -m venv inkling_env
source inkling_env/bin/activate # On Windows: inkling_env\Scripts\activate
Why: Using a virtual environment prevents conflicts with other Python packages on your system.
2. Install Required Dependencies
Install the necessary packages for working with Inkling:
pip install torch transformers accelerate datasets
Why: These libraries provide the core functionality needed to load and run large language models, including PyTorch for deep learning operations and Hugging Face's Transformers for model loading.
3. Download the Inkling Model
Since Inkling is an open-weight model, you can download it from the Hugging Face Hub:
from huggingface_hub import snapshot_download
# Download the Inkling model
model_path = snapshot_download(repo_id="thinkingmachines/inkling", revision="main")
Why: This downloads the complete model files, including weights and configuration files, to your local machine.
4. Load the Model and Tokenizer
Now, load the model and tokenizer for text processing:
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load tokenizer and model
model_name = "thinkingmachines/inkling"
# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_name)
# Load model with proper device mapping
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype=torch.float16,
device_map="auto"
)
Why: The tokenizer converts text into tokens that the model can understand, while the model itself processes these tokens to generate responses.
5. Prepare Text Input for Testing
Prepare a simple text prompt to test the model:
prompt = "Explain how multimodal AI works in simple terms."
input_ids = tokenizer.encode(prompt, return_tensors="pt")
Why: This converts your text prompt into a format that the model can process, which is a sequence of numerical tokens.
6. Generate Text Response
Run the model to generate a response to your prompt:
from transformers import GenerationConfig
# Configure generation settings
generation_config = GenerationConfig(
max_new_tokens=100,
temperature=0.7,
top_p=0.9,
do_sample=True
)
# Generate response
with torch.no_grad():
outputs = model.generate(
input_ids,
generation_config=generation_config
)
# Decode the response
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)
Why: This runs the model's inference process to generate text based on your input, using parameters that control the creativity and coherence of the output.
7. Test with Image Input
For multimodal input, you'll need to handle both text and image processing:
from PIL import Image
import requests
# Load an image
image_url = "https://example.com/sample-image.jpg"
image = Image.open(requests.get(image_url, stream=True).raw)
# For multimodal processing, you'll need a specialized processor
from transformers import AutoProcessor
processor = AutoProcessor.from_pretrained("thinkingmachines/inkling")
# Process image and text together
inputs = processor(text=prompt, images=image, return_tensors="pt")
Why: This prepares both image and text inputs for processing by the multimodal model, which can understand visual and textual information simultaneously.
8. Run Multimodal Inference
Execute the multimodal inference with both text and image inputs:
# Generate response for multimodal input
with torch.no_grad():
outputs = model.generate(
**inputs,
generation_config=generation_config
)
# Decode the multimodal response
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)
Why: This demonstrates how Inkling can process both text and image inputs together, showcasing its multimodal capabilities.
9. Experiment with Thinking Effort Control
Inkling features controllable thinking effort, which affects how much processing the model does:
# Adjust temperature to control thinking effort
# Lower temperature = more focused, less creative
# Higher temperature = more creative, less focused
low_effort_config = GenerationConfig(
max_new_tokens=100,
temperature=0.3,
do_sample=True
)
high_effort_config = GenerationConfig(
max_new_tokens=100,
temperature=0.9,
do_sample=True
)
# Generate with low thinking effort
with torch.no_grad():
low_effort_output = model.generate(input_ids, generation_config=low_effort_config)
# Generate with high thinking effort
with torch.no_grad():
high_effort_output = model.generate(input_ids, generation_config=high_effort_config)
print("Low effort response:", tokenizer.decode(low_effort_output[0], skip_special_tokens=True))
print("High effort response:", tokenizer.decode(high_effort_output[0], skip_special_tokens=True))
Why: Controlling thinking effort allows you to balance between precise, focused responses and more creative, exploratory outputs.
Summary
In this tutorial, you've learned how to set up and work with the Inkling model from Thinking Machines Lab. You've downloaded the model, loaded it into your environment, and run both text-only and multimodal (text + image) inference tasks. You've also explored how to control the model's thinking effort to get different types of responses. While Inkling isn't the most powerful model available, its open weights and customization capabilities make it an excellent learning platform for understanding large-scale multimodal AI systems.
This hands-on experience gives you a foundation for working with advanced AI models and understanding how different parameters affect model behavior. As you continue exploring, you can experiment with different prompts, inputs, and generation parameters to see how they affect the model's outputs.
