Introduction
In this tutorial, we'll explore how to work with research agent benchmarks like Perplexity's WANDR, which evaluates AI systems' ability to conduct deep and wide research. We'll build a simplified version of a research agent that can search for evidence, evaluate sources, and generate structured responses - similar to what WANDR tests. This hands-on approach will teach you how to implement a basic research agent framework that can be extended for more complex tasks.
Prerequisites
Before starting this tutorial, you should have:
- Intermediate Python programming skills
- Familiarity with APIs and web scraping concepts
- Basic understanding of natural language processing (NLP) concepts
- Installed Python packages: requests, BeautifulSoup, openai, pandas
Step-by-Step Instructions
1. Set Up Your Development Environment
First, create a new Python project directory and install the required dependencies:
mkdir research-agent-benchmark
cd research-agent-benchmark
pip install requests beautifulsoup4 openai pandas
This setup gives us the core tools needed to make web requests, parse HTML content, interact with OpenAI's API, and handle data structures.
2. Create a Basic Research Agent Class
Let's start by creating a foundation for our research agent:
import requests
from bs4 import BeautifulSoup
import openai
import time
class ResearchAgent:
def __init__(self, api_key):
openai.api_key = api_key
self.session = requests.Session()
self.session.headers.update({
'User-Agent': 'ResearchAgent/1.0'
})
def search_web(self, query):
# Simple web search using a search API
# In practice, you'd use a proper search API like SerpAPI or DuckDuckGo
print(f"Searching for: {query}")
# This is a placeholder - you'd implement actual search logic here
return ["https://example.com/article1", "https://example.com/article2"]
def fetch_content(self, url):
# Fetch and parse content from a URL
try:
response = self.session.get(url, timeout=10)
soup = BeautifulSoup(response.content, 'html.parser')
# Extract main content (simplified)
content = soup.get_text()
return content[:1000] # Return first 1000 characters
except Exception as e:
print(f"Error fetching {url}: {e}")
return ""
def analyze_evidence(self, query, content):
# Use OpenAI to analyze evidence and extract key points
prompt = f"Analyze the following content related to '{query}' and extract key evidence points with citations:\n\n{content}"
response = openai.ChatCompletion.create(
model="gpt-3.5-turbo",
messages=[
{"role": "system", "content": "You are a research assistant that extracts evidence and cites sources accurately."},
{"role": "user", "content": prompt}
],
max_tokens=500,
temperature=0.3
)
return response['choices'][0]['message']['content']
This class sets up the basic structure for a research agent with search, content fetching, and evidence analysis capabilities. The agent can search for information, retrieve web content, and analyze it using OpenAI's language models.
3. Implement a Task Evaluation System
Now let's create a system that evaluates research tasks similar to WANDR's approach:
import pandas as pd
class TaskEvaluator:
def __init__(self):
self.results = []
def evaluate_task(self, task):
# Simulate task evaluation
# In WANDR, this would measure evidence discovery and citation quality
task_id = task['id']
query = task['query']
expected_evidence_count = task['expected_evidence_count']
# This would be replaced with actual agent logic
evidence_discovered = min(expected_evidence_count + 2, 10) # Simulate finding extra evidence
# Calculate soft F1 score (simplified)
precision = min(evidence_discovered / expected_evidence_count, 1.0)
recall = min(evidence_discovered / expected_evidence_count, 1.0)
if (precision + recall) > 0:
f1_score = 2 * (precision * recall) / (precision + recall)
else:
f1_score = 0
result = {
'task_id': task_id,
'query': query,
'evidence_found': evidence_discovered,
'expected_evidence': expected_evidence_count,
'f1_score': f1_score
}
self.results.append(result)
return result
def generate_report(self):
df = pd.DataFrame(self.results)
print("\nEvaluation Results:")
print(df)
avg_f1 = df['f1_score'].mean()
print(f"\nAverage F1 Score: {avg_f1:.3f}")
return df
This evaluator mimics how WANDR would score research agents by measuring how many qualifying entities they discover and how well they cite evidence. The F1 score balances precision (how many discovered entities are relevant) and recall (how many actual entities were found).
4. Create a Sample Task Dataset
Let's create some sample research tasks similar to those in WANDR:
def create_sample_tasks():
tasks = [
{
'id': 'task_001',
'query': 'Major breakthroughs in quantum computing in 2023',
'expected_evidence_count': 5
},
{
'id': 'task_002',
'query': 'Impact of AI on healthcare diagnostics',
'expected_evidence_count': 7
},
{
'id': 'task_003',
'query': 'Renewable energy storage technologies',
'expected_evidence_count': 6
}
]
return tasks
# Test the sample tasks
sample_tasks = create_sample_tasks()
print("Sample Tasks:")
for task in sample_tasks:
print(f"- {task['query']}")
These sample tasks represent the type of evidence-heavy queries that WANDR evaluates. Each task requires finding multiple qualifying entities with supporting evidence.
5. Run the Complete Research Agent System
Now let's put everything together to run a complete research evaluation:
def main():
# Initialize components
agent = ResearchAgent(api_key="your-openai-api-key")
evaluator = TaskEvaluator()
tasks = create_sample_tasks()
print("Starting research agent evaluation...")
for task in tasks:
print(f"\nProcessing task: {task['query']}")
# Step 1: Search for relevant sources
search_results = agent.search_web(task['query'])
# Step 2: Fetch content from sources
all_content = []
for url in search_results[:2]: # Limit to 2 sources for demo
content = agent.fetch_content(url)
all_content.append(content)
# Step 3: Analyze evidence
combined_content = "\n\n".join(all_content)
evidence_analysis = agent.analyze_evidence(task['query'], combined_content)
# Step 4: Evaluate task
evaluation = evaluator.evaluate_task(task)
print(f"Found {evaluation['evidence_found']} pieces of evidence")
print(f"F1 Score: {evaluation['f1_score']:.3f}")
# Show a sample of the analysis
print(f"\nEvidence Analysis Summary:\n{evidence_analysis[:200]}...")
# Add delay to respect API limits
time.sleep(1)
# Generate final report
evaluator.generate_report()
if __name__ == "__main__":
main()
This final integration demonstrates how a research agent would work through tasks similar to WANDR's evaluation. It shows the complete workflow from searching to evidence analysis to scoring.
Summary
In this tutorial, we've built a simplified research agent framework that mirrors the capabilities tested by Perplexity's WANDR benchmark. We've created:
- A research agent class that can search, fetch, and analyze web content
- A task evaluation system that measures evidence discovery and citation quality
- A sample dataset of research tasks
- A complete workflow that demonstrates how such agents would be evaluated
This framework provides a foundation for building more sophisticated research agents. In practice, you would enhance this with:
- Real search APIs (like SerpAPI or Google Custom Search)
- Better content extraction and summarization
- Advanced citation and verification systems
- More sophisticated evaluation metrics
By understanding how benchmarks like WANDR evaluate research agents, you can build systems that truly search wide and deep while providing verifiable evidence - the core challenge that makes research agents valuable for complex information tasks.
