Introduction
In this tutorial, you'll learn how to run the Underdog Saluki 27B model, a 2-bit quantized version of Qwen3.8-27B, using the llama.cpp framework. This model is optimized for tool-calling tasks and can run efficiently on consumer hardware. We'll walk through the entire process from downloading the model to running it with tool-calling capabilities.
Prerequisites
Before starting this tutorial, you'll need:
- A computer with at least 8GB of RAM (16GB recommended)
- Git installed on your system
- Basic command-line knowledge
- Python 3.7 or higher installed
Step-by-Step Instructions
1. Setting Up the Environment
1.1 Clone the llama.cpp Repository
The first step is to get the llama.cpp codebase, which provides tools to run quantized models like Underdog Saluki 27B.
git clone https://github.com/ggerganov/llama.cpp.git
Why: This repository contains all the necessary tools to run and manage LLMs on local hardware.
1.2 Navigate to the Directory
After cloning, move into the repository directory:
cd llama.cpp
Why: We need to work within this directory to compile and run the tools.
2. Compiling llama.cpp
2.1 Build the Project
Now, we'll compile the llama.cpp project to create the necessary executables:
make
Why: This step compiles the C++ code into executables that can run on your machine.
2.2 Verify the Build
After compilation, verify that the required binaries are present:
ls -la build/bin/
Why: This confirms that the build was successful and we have the necessary executables.
3. Downloading the Underdog Saluki 27B Model
3.1 Find the Model File
The Underdog Saluki 27B model is typically distributed in GGUF format, which is optimized for local inference. You'll need to download it from a source like Hugging Face or the model's official repository.
curl -L -o models/underdog-saluki-27b.gguf https://huggingface.co/underdog-labs/underdog-saluki-27b/resolve/main/underdog-saluki-27b.gguf
Why: GGUF files are specifically designed for efficient inference on local hardware and are compatible with llama.cpp.
3.2 Create Model Directory
If the models directory doesn't exist, create it:
mkdir -p models
Why: This ensures we have a proper location to store our model files.
4. Running the Model with Tool-Calling
4.1 Basic Inference
To run the model, we'll use the main executable with appropriate parameters:
./build/bin/main -m models/underdog-saluki-27b.gguf -p "You are a helpful assistant." --color -n 128
Why: This command starts the model with a prompt, enables color output, and limits the output to 128 tokens.
4.2 Enable Tool-Calling
For tool-calling capabilities, we'll need to provide a JSON schema that defines the available tools:
./build/bin/main -m models/underdog-saluki-27b.gguf --tool-calling --tools "[{\"name\": \"get_weather\", \"description\": \"Get weather information\", \"parameters\": {\"type\": \"object\", \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The city to get weather for\"}}, \"required\": [\"location\"]}}]" -p "What's the weather in London?"
Why: This enables the model to understand and use the defined tools, allowing it to call functions like a weather API.
5. Understanding GGUF Headers
5.1 Inspect Model Metadata
To understand the model's structure, we can use the gguf-info tool:
./build/bin/gguf-info models/underdog-saluki-27b.gguf
Why: This shows us the model's quantization details, dimensions, and other metadata that's crucial for understanding its capabilities.
5.2 Check Memory Usage
Quantized models like Underdog Saluki 27B are designed to use minimal memory:
./build/bin/main -m models/underdog-saluki-27b.gguf --memory-fraction 0.5 -p "Hello, how are you?"
Why: This limits the memory usage to 50% of available RAM, making it possible to run on lower-end hardware.
6. Testing Tool-Calling Performance
6.1 Create a Simple Tool-Calling Test
Let's create a test to verify tool-calling works correctly:
echo '{"prompt": "Use the get_weather tool to find the weather in Tokyo", "tools": [{"name": "get_weather", "description": "Get weather information", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "The city to get weather for"}}, "required": ["location"]}}]}' > test.json
Why: This creates a structured test case that we can use to verify the model's tool-calling behavior.
6.2 Run the Test
Now, run the test with the model:
./build/bin/main -m models/underdog-saluki-27b.gguf --tool-calling --tool-json test.json -p "Use the get_weather tool to find the weather in Tokyo"
Why: This demonstrates how the model can process tool-calling instructions and respond appropriately.
Summary
In this tutorial, we've walked through the complete process of setting up and running the Underdog Saluki 27B model using llama.cpp. We've covered:
- Cloning and building the llama.cpp framework
- Downloading and organizing the model files
- Running basic inference and tool-calling capabilities
- Understanding GGUF headers and memory usage
This setup allows you to run powerful language models locally with tool-calling capabilities, making it ideal for applications that require function calling without relying on external APIs.