Context Engineering Inside the Harness: 4 Mechanisms That Beat Context Overflow and Goal Loss on Long-Horizon Tasks

Introduction

Large Language Models (LLMs) are powerful tools, but they face significant challenges when working on long, complex tasks. Two major issues are context overflow and goal loss. Context overflow happens when an LLM's input context window is exceeded, causing it to lose information from earlier parts of the conversation. Goal loss occurs when the model loses track of the original task objective over time. This tutorial will show you how to build a simple context management system that addresses these problems using four key mechanisms: context summarization, task state tracking, dynamic context trimming, and goal reinforcement.

By the end of this tutorial, you'll understand how to create a basic system that helps LLMs maintain context and task focus during long conversations, similar to what's used in advanced agents like LangChain Deep Agents and Claude Code.

Prerequisites

To follow this tutorial, you'll need:

• A basic understanding of Python programming
• Python 3.7 or higher installed on your system
• Access to an LLM API (we'll use OpenAI's API for this example)
• Basic knowledge of how LLMs work (prompting, context windows, etc.)

Step-by-Step Instructions

1. Set Up Your Environment

First, we need to install the required Python packages. Open your terminal and run:

pip install openai python-dotenv

This installs the OpenAI Python library and dotenv for managing API keys securely.

2. Create Your API Key File

Create a file named .env in your project directory and add your OpenAI API key:

OPENAI_API_KEY=your_api_key_here

Replace your_api_key_here with your actual API key from OpenAI.

3. Initialize Your Context Manager

Let's create a basic context manager class that will track conversation history and apply our four mechanisms:

import os
import openai
from dotenv import load_dotenv

# Load environment variables
load_dotenv()

# Initialize OpenAI client
openai.api_key = os.getenv('OPENAI_API_KEY')


class ContextManager:
    def __init__(self, max_context_tokens=2000, summary_threshold=1000):
        self.conversation_history = []
        self.max_context_tokens = max_context_tokens
        self.summary_threshold = summary_threshold
        self.task_goal = ""
        
    def add_message(self, role, content):
        """Add a message to the conversation history"""
        self.conversation_history.append({
            'role': role,
            'content': content
        })
        
    def set_task_goal(self, goal):
        """Set the main task goal for the conversation"""
        self.task_goal = goal
        
    def get_context(self):
        """Get current context with applied mechanisms"""
        # Apply dynamic context trimming
        if len(self.conversation_history) > 0:
            # Summarize old context if needed
            if self._should_summarize():
                self._summarize_context()
            
            # Reinforce goal
            self._reinforce_goal()
            
        return self.conversation_history
    
    def _should_summarize(self):
        """Check if context needs summarization"""
        # Simple heuristic: if we're approaching max tokens
        current_tokens = self._count_tokens()
        return current_tokens > self.summary_threshold
    
    def _count_tokens(self):
        """Simple token counter (in practice, use a proper tokenizer)"""
        # This is a simplified token count
        total = 0
        for msg in self.conversation_history:
            total += len(msg['content']) // 4  # Approximate tokens
        return total
    
    def _summarize_context(self):
        """Summarize the conversation history to reduce context length"""
        # Get the conversation history to summarize
        messages_to_summarize = self.conversation_history[:-2]  # Leave last 2 messages
        
        if len(messages_to_summarize) > 0:
            # Create a summary prompt
            summary_prompt = f"Summarize the following conversation in 2 sentences:\n\n"
            for msg in messages_to_summarize:
                summary_prompt += f"{msg['role']}: {msg['content']}\n\n"
            
            # Get summary from LLM
            try:
                response = openai.ChatCompletion.create(
                    model="gpt-3.5-turbo",
                    messages=[
                        {"role": "system", "content": "You are a helpful assistant that summarizes conversations."},
                        {"role": "user", "content": summary_prompt}
                    ],
                    max_tokens=100
                )
                
                # Replace old messages with summary
                summary_content = response.choices[0].message.content
                self.conversation_history = [self.conversation_history[-2], self.conversation_history[-1]]
                self.conversation_history.insert(0, {"role": "assistant", "content": f"Summary of previous conversation: {summary_content}"})
                
            except Exception as e:
                print(f"Error during summarization: {e}")
                
    def _reinforce_goal(self):
        """Reinforce the task goal in the context"""
        if self.task_goal and len(self.conversation_history) > 0:
            # Add goal reminder at the beginning
            goal_prompt = f"Remember: Your task is to {self.task_goal}."
            
            # Check if goal is already present
            goal_present = any('Remember:' in msg['content'] and self.task_goal in msg['content'] 
                              for msg in self.conversation_history)
            
            if not goal_present:
                self.conversation_history.insert(0, {"role": "system", "content": goal_prompt})

# Initialize the context manager
context_manager = ContextManager(max_context_tokens=2000, summary_threshold=1000)

4. Test Your Context Manager

Now let's create a simple test to see how the context manager works:

# Set up the context manager
context_manager = ContextManager(max_context_tokens=2000, summary_threshold=1000)

# Set task goal
context_manager.set_task_goal("analyze customer feedback and generate a summary report")

# Add some messages
context_manager.add_message("user", "I want to analyze customer feedback about our new product.")
context_manager.add_message("assistant", "I can help with that. What specific aspects would you like to focus on?")
context_manager.add_message("user", "I want to focus on product features, pricing, and user experience.")

# Get current context
context = context_manager.get_context()
print("Current context:")
for msg in context:
    print(f"{msg['role']}: {msg['content']}")

5. Simulate Long Conversations

Let's simulate a long conversation to see how our context manager handles it:

def simulate_long_conversation():
    # Set up
    context_manager = ContextManager(max_context_tokens=2000, summary_threshold=1000)
    context_manager.set_task_goal("research and summarize the impact of AI on healthcare")
    
    # Simulate adding many messages
    for i in range(50):
        user_input = f"This is message number {i} about AI in healthcare. It contains some information about how artificial intelligence is changing medical diagnosis."
        context_manager.add_message("user", user_input)
        
        # Simulate assistant response
        assistant_response = f"I understand message {i}. This is a response to your input about AI in healthcare."
        context_manager.add_message("assistant", assistant_response)
        
        # Print context status every 10 messages
        if i % 10 == 0:
            print(f"After {i} messages:")
            context = context_manager.get_context()
            print(f"Total messages in context: {len(context)}")
            print(f"Context tokens: {context_manager._count_tokens()}")
            print("---")

# Run simulation
simulate_long_conversation()

6. Add Task State Tracking

Let's enhance our context manager with task state tracking to prevent goal loss:

class EnhancedContextManager(ContextManager):
    def __init__(self, max_context_tokens=2000, summary_threshold=1000):
        super().__init__(max_context_tokens, summary_threshold)
        self.task_state = {}
        
    def update_task_state(self, key, value):
        """Update the task state"""
        self.task_state[key] = value
        
    def get_task_state(self):
        """Get current task state"""
        return self.task_state
        
    def _reinforce_goal(self):
        """Enhanced goal reinforcement with task state"""
        if self.task_goal and len(self.conversation_history) > 0:
            # Add goal reminder at the beginning
            goal_prompt = f"Remember: Your task is to {self.task_goal}."
            
            # Add task state reminder
            if self.task_state:
                state_str = ", ".join([f'{k}: {v}' for k, v in self.task_state.items()])
                goal_prompt += f"\nCurrent task state: {state_str}"
            
            # Check if goal is already present
            goal_present = any('Remember:' in msg['content'] and self.task_goal in msg['content'] 
                              for msg in self.conversation_history)
            
            if not goal_present:
                self.conversation_history.insert(0, {"role": "system", "content": goal_prompt})

Summary

In this tutorial, you've learned how to build a basic context management system that addresses two major challenges in long-horizon LLM tasks: context overflow and goal loss. We've implemented four key mechanisms:

1. Context summarization: Automatically summarize old conversation history when it approaches token limits
2. Task state tracking: Maintain and update the current state of the task
3. Dynamic context trimming: Remove unnecessary information while preserving key details
4. Goal reinforcement: Continuously remind the model of the original task objective

These mechanisms work together to help LLMs maintain context and task focus, similar to what's used in advanced agent systems. While this is a simplified implementation, it demonstrates the core concepts that power systems like LangChain Deep Agents and Claude Code. In practice, you'd want to use more sophisticated token counting, better summarization models, and more robust state management.