Sakana AI's Fugu orchestrates multiple LLMs to match Anthropic's Fable and Mythos benchmarks

Learn to build a simple multi-LLM orchestration system that dynamically selects the best AI model for each question, similar to Sakana AI's Fugu system.

Introduction

In this tutorial, you'll learn how to build a simple multi-LLM orchestration system similar to what Sakana AI's Fugu does. This system will coordinate multiple language models to answer questions, making decisions about which model to use based on the question type. This approach helps reduce dependency on any single AI provider while potentially improving performance.

By the end of this tutorial, you'll have a working prototype that demonstrates how to combine different AI models in a smart way, just like the Fugu system described in the news article.

Prerequisites

To follow this tutorial, you'll need:

A computer with Python 3.7 or higher installed
Basic understanding of Python programming
Access to at least one language model API (we'll use OpenAI's GPT and Hugging Face's models)
Internet connection

Step-by-Step Instructions

1. Set Up Your Python Environment

First, create a new Python project directory and install the required packages:

mkdir multi_llm_orchestrator
cd multi_llm_orchestrator
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install openai transformers torch

Why this step? We need to create a separate Python environment to avoid conflicts with other projects, and install packages for both OpenAI API access and local Hugging Face models.

2. Get Your API Keys

Sign up for an OpenAI account and get your API key from https://platform.openai.com/api-keys. Create a file called .env in your project directory:

OPENAI_API_KEY=your_openai_api_key_here

Why this step? We need API access to use OpenAI's language models, which will be one of our orchestrator's components.

3. Create the Main Orchestrator Class

Create a file called orchestrator.py and add the following code:

import os
from openai import OpenAI
from transformers import pipeline
import random

class MultiLLMOrchestrator:
    def __init__(self):
        # Initialize OpenAI client
        self.openai_client = OpenAI(api_key=os.getenv('OPENAI_API_KEY'))
        
        # Initialize local model
        self.local_model = pipeline('text-generation', model='gpt2')
        
        # Define model capabilities
        self.models = {
            'gpt-3.5-turbo': {
                'type': 'openai',
                'capabilities': ['general_qa', 'creative_writing', 'translation']
            },
            'gpt-4': {
                'type': 'openai',
                'capabilities': ['complex_reasoning', 'technical_analysis', 'data_interpretation']
            },
            'gpt2': {
                'type': 'local',
                'capabilities': ['basic_qa', 'summarization']
            }
        }
        
        # Define question classification rules
        self.question_rules = {
            'technical': ['calculate', 'analyze', 'explain', 'solve'],
            'creative': ['write', 'create', 'compose', 'imagine'],
            'general': ['what', 'how', 'why', 'when']
        }

    def classify_question(self, question):
        question_lower = question.lower()
        
        # Check for technical terms
        for term in self.question_rules['technical']:
            if term in question_lower:
                return 'technical'
        
        # Check for creative terms
        for term in self.question_rules['creative']:
            if term in question_lower:
                return 'creative'
        
        # Default to general
        return 'general'

    def select_best_model(self, question):
        # Classify the question
        question_type = self.classify_question(question)
        
        # Find models that can handle this question type
        suitable_models = []
        for model_name, model_info in self.models.items():
            if question_type in model_info['capabilities']:
                suitable_models.append(model_name)
        
        # If no specific model found, use a default
        if not suitable_models:
            suitable_models = list(self.models.keys())
        
        # Return a random suitable model (in a real system, you might use more sophisticated logic)
        return random.choice(suitable_models]

    def query_model(self, model_name, question):
        if model_name == 'gpt2':
            # Use local model
            result = self.local_model(question, max_length=100, num_return_sequences=1)
            return result[0]['generated_text']
        else:
            # Use OpenAI model
            response = self.openai_client.chat.completions.create(
                model=model_name,
                messages=[{"role": "user", "content": question}]
            )
            return response.choices[0].message.content

    def process_question(self, question):
        # Select the best model for this question
        model_name = self.select_best_model(question)
        
        # Query the selected model
        result = self.query_model(model_name, question)
        
        return {
            'question': question,
            'selected_model': model_name,
            'answer': result
        }

Why this step? This creates the core logic of our orchestrator - it can determine which model is best for a given question and then query that model.

4. Create a Simple Interface

Create a file called main.py:

from orchestrator import MultiLLMOrchestrator
import os
from dotenv import load_dotenv

# Load environment variables
load_dotenv()

# Create orchestrator instance
orchestrator = MultiLLMOrchestrator()

# Simple interactive loop
print("Multi-LLM Orchestrator - Type 'quit' to exit")

while True:
    question = input("\nAsk a question: ")
    
    if question.lower() == 'quit':
        break
    
    result = orchestrator.process_question(question)
    print(f"\nSelected Model: {result['selected_model']}")
    print(f"Answer: {result['answer']}")

Why this step? This provides a user-friendly way to interact with our orchestrator system.

5. Test Your System

Run your system with:

python main.py

Try asking questions like:

"Explain quantum computing in simple terms"">
"Calculate the area of a circle with radius 5"">
"Write a poem about technology"">

Observe how the system selects different models based on your question.

Why this step? Testing helps you understand how the system works and identify any issues before deployment.

6. Extend the System

Enhance your orchestrator by adding more models or capabilities:

# Add to the models dictionary in orchestrator.py
'claude-3': {
    'type': 'openai',
    'capabilities': ['complex_reasoning', 'ethics_discussion', 'data_interpretation']
}

You can also add more sophisticated classification rules or use machine learning to improve model selection.

Why this step? This demonstrates how the system can be expanded to handle more complex scenarios and additional models.

Summary

In this tutorial, you've built a simple multi-LLM orchestration system that demonstrates key concepts from Sakana AI's Fugu. You learned how to:

Set up a Python environment for AI development
Integrate with both local and cloud-based language models
Classify questions and select appropriate models
Create an interactive system that can coordinate multiple AI models

This system mimics the core functionality of Fugu by dynamically choosing which AI model to use based on the question type, helping reduce dependence on any single provider while potentially improving performance through model specialization.

While this is a simplified version, it demonstrates the fundamental principles behind advanced systems like Fugu, which can be expanded with more sophisticated model selection algorithms, better classification systems, and additional AI models.