How to Run Underdog Saluki 27B: 2-Bit Qwen3.8-27B Tool-Calling Guide
Back to Tutorials
aiTutorialbeginner

How to Run Underdog Saluki 27B: 2-Bit Qwen3.8-27B Tool-Calling Guide

October 24, 202629 views4 min read

Learn how to set up and run the Underdog Saluki 27B 2-bit quantized model using llama.cpp, including tool-calling capabilities and memory optimization.

Introduction

In this tutorial, you'll learn how to run the Underdog Saluki 27B model, a 2-bit quantized version of Qwen3.8-27B, using the llama.cpp framework. This model is optimized for tool-calling tasks and can run efficiently on consumer hardware. We'll walk through the entire process from downloading the model to running it with tool-calling capabilities.

Prerequisites

Before starting this tutorial, you'll need:

  • A computer with at least 8GB of RAM (16GB recommended)
  • Git installed on your system
  • Basic command-line knowledge
  • Python 3.7 or higher installed

Step-by-Step Instructions

1. Setting Up the Environment

1.1 Clone the llama.cpp Repository

The first step is to get the llama.cpp codebase, which provides tools to run quantized models like Underdog Saluki 27B.

git clone https://github.com/ggerganov/llama.cpp.git

Why: This repository contains all the necessary tools to run and manage LLMs on local hardware.

1.2 Navigate to the Directory

After cloning, move into the repository directory:

cd llama.cpp

Why: We need to work within this directory to compile and run the tools.

2. Compiling llama.cpp

2.1 Build the Project

Now, we'll compile the llama.cpp project to create the necessary executables:

make

Why: This step compiles the C++ code into executables that can run on your machine.

2.2 Verify the Build

After compilation, verify that the required binaries are present:

ls -la build/bin/

Why: This confirms that the build was successful and we have the necessary executables.

3. Downloading the Underdog Saluki 27B Model

3.1 Find the Model File

The Underdog Saluki 27B model is typically distributed in GGUF format, which is optimized for local inference. You'll need to download it from a source like Hugging Face or the model's official repository.

curl -L -o models/underdog-saluki-27b.gguf https://huggingface.co/underdog-labs/underdog-saluki-27b/resolve/main/underdog-saluki-27b.gguf

Why: GGUF files are specifically designed for efficient inference on local hardware and are compatible with llama.cpp.

3.2 Create Model Directory

If the models directory doesn't exist, create it:

mkdir -p models

Why: This ensures we have a proper location to store our model files.

4. Running the Model with Tool-Calling

4.1 Basic Inference

To run the model, we'll use the main executable with appropriate parameters:

./build/bin/main -m models/underdog-saluki-27b.gguf -p "You are a helpful assistant." --color -n 128

Why: This command starts the model with a prompt, enables color output, and limits the output to 128 tokens.

4.2 Enable Tool-Calling

For tool-calling capabilities, we'll need to provide a JSON schema that defines the available tools:

./build/bin/main -m models/underdog-saluki-27b.gguf --tool-calling --tools "[{\"name\": \"get_weather\", \"description\": \"Get weather information\", \"parameters\": {\"type\": \"object\", \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The city to get weather for\"}}, \"required\": [\"location\"]}}]" -p "What's the weather in London?"

Why: This enables the model to understand and use the defined tools, allowing it to call functions like a weather API.

5. Understanding GGUF Headers

5.1 Inspect Model Metadata

To understand the model's structure, we can use the gguf-info tool:

./build/bin/gguf-info models/underdog-saluki-27b.gguf

Why: This shows us the model's quantization details, dimensions, and other metadata that's crucial for understanding its capabilities.

5.2 Check Memory Usage

Quantized models like Underdog Saluki 27B are designed to use minimal memory:

./build/bin/main -m models/underdog-saluki-27b.gguf --memory-fraction 0.5 -p "Hello, how are you?"

Why: This limits the memory usage to 50% of available RAM, making it possible to run on lower-end hardware.

6. Testing Tool-Calling Performance

6.1 Create a Simple Tool-Calling Test

Let's create a test to verify tool-calling works correctly:

echo '{"prompt": "Use the get_weather tool to find the weather in Tokyo", "tools": [{"name": "get_weather", "description": "Get weather information", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "The city to get weather for"}}, "required": ["location"]}}]}' > test.json

Why: This creates a structured test case that we can use to verify the model's tool-calling behavior.

6.2 Run the Test

Now, run the test with the model:

./build/bin/main -m models/underdog-saluki-27b.gguf --tool-calling --tool-json test.json -p "Use the get_weather tool to find the weather in Tokyo"

Why: This demonstrates how the model can process tool-calling instructions and respond appropriately.

Summary

In this tutorial, we've walked through the complete process of setting up and running the Underdog Saluki 27B model using llama.cpp. We've covered:

  • Cloning and building the llama.cpp framework
  • Downloading and organizing the model files
  • Running basic inference and tool-calling capabilities
  • Understanding GGUF headers and memory usage

This setup allows you to run powerful language models locally with tool-calling capabilities, making it ideal for applications that require function calling without relying on external APIs.

Source: MarkTechPost

Related Articles