Introduction
In this tutorial, you'll learn how to work with AI agents that can interact with web content using Python. This tutorial is inspired by the recent OpenAI incident where autonomous agents caused unintended changes to a German wiki. We'll build a simple web scraping and content modification tool that demonstrates how AI agents can interact with online content, while also learning about responsible AI practices.
Prerequisites
Before starting this tutorial, you'll need:
- A computer with Python 3.7 or higher installed
- Basic understanding of Python programming concepts
- Access to a web browser
- Internet connection
Step-by-Step Instructions
Step 1: Set Up Your Python Environment
First, we need to create a clean Python environment for our project. Open your terminal or command prompt and run the following commands:
mkdir ai_web_agent
cd ai_web_agent
python -m venv agent_env
source agent_env/bin/activate # On Windows: agent_env\Scripts\activateWhy we do this: Creating a virtual environment isolates our project dependencies from your system's Python installation, preventing conflicts with other projects.
Step 2: Install Required Libraries
Next, we'll install the necessary Python libraries for web scraping and AI interaction:
pip install requests beautifulsoup4 seleniumWhy we do this: These libraries provide the tools needed to fetch web content, parse HTML, and automate browser interactions - essential for building AI agents that can work with web pages.
Step 3: Create a Basic Web Scraper
Now, let's create a simple script to fetch and display web content:
import requests
from bs4 import BeautifulSoup
# Simple web scraper
def scrape_website(url):
try:
response = requests.get(url)
response.raise_for_status() # Raises an HTTPError for bad responses
soup = BeautifulSoup(response.content, 'html.parser')
return soup
except requests.RequestException as e:
print(f"Error fetching {url}: {e}")
return None
# Test with a sample URL
if __name__ == "__main__":
url = "https://example.com"
soup = scrape_website(url)
if soup:
print("Title:", soup.title.string)
print("First paragraph:", soup.find('p').text[:100])Why we do this: This basic scraper shows how to fetch web content programmatically, which is a fundamental step in building AI agents that can interact with websites.
Step 4: Create a Simple AI Agent Interface
Let's create a basic AI agent that can analyze and potentially modify content:
import requests
from bs4 import BeautifulSoup
import time
# AI agent class
class WebAgent:
def __init__(self, base_url):
self.base_url = base_url
self.session = requests.Session()
self.session.headers.update({'User-Agent': 'AI-Web-Agent/1.0'})
def fetch_content(self, path):
url = f"{self.base_url}{path}"
try:
response = self.session.get(url)
response.raise_for_status()
soup = BeautifulSoup(response.content, 'html.parser')
return soup
except requests.RequestException as e:
print(f"Error fetching {url}: {e}")
return None
def analyze_content(self, soup):
# Simple content analysis
if soup:
title = soup.title.string if soup.title else "No title"
paragraphs = soup.find_all('p')
word_count = sum(len(p.text.split()) for p in paragraphs)
return {
'title': title,
'paragraph_count': len(paragraphs),
'word_count': word_count
}
return None
def safe_modify_content(self, soup, changes):
# This method would contain safety checks
print("Content modification would happen here")
print("Changes to be made:", changes)
return soup
# Example usage
if __name__ == "__main__":
agent = WebAgent("https://example.com")
soup = agent.fetch_content('/')
analysis = agent.analyze_content(soup)
if analysis:
print("Analysis results:", analysis)Why we do this: This agent structure demonstrates how to safely interact with web content while keeping track of what the agent is doing - an essential practice for responsible AI development.
Step 5: Add Safety Measures and Logging
Now we'll add logging and safety measures to prevent unintended modifications:
import logging
import time
from datetime import datetime
# Configure logging
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s',
handlers=[
logging.FileHandler('agent_activity.log'),
logging.StreamHandler()
]
)
class SafeWebAgent(WebAgent):
def __init__(self, base_url):
super().__init__(base_url)
self.action_log = []
def log_action(self, action, details):
log_entry = {
'timestamp': datetime.now().isoformat(),
'action': action,
'details': details
}
self.action_log.append(log_entry)
logging.info(f"Action: {action} - Details: {details}")
def safe_fetch(self, path):
# Add delay to be respectful to servers
time.sleep(1)
self.log_action("FETCH", f"Fetching {path}")
return self.fetch_content(path)
def safe_modify(self, soup, changes):
# Safety check before modification
if not changes:
self.log_action("MODIFY", "No changes to make")
return soup
self.log_action("MODIFY", f"Attempting to modify with changes: {changes}")
# In a real implementation, this would be more complex
print("Would modify content here - but we're just logging for now")
return soup
# Example usage
if __name__ == "__main__":
agent = SafeWebAgent("https://example.com")
soup = agent.safe_fetch('/')
analysis = agent.analyze_content(soup)
if analysis:
print("Analysis results:", analysis)
# Log a modification attempt
agent.safe_modify(soup, ["Add new paragraph"])Why we do this: Adding logging and safety measures is crucial when building AI systems that interact with the web. The OpenAI incident shows why responsible AI practices are essential - logging helps track what AI agents do, and safety checks prevent unintended consequences.
Step 6: Test Your Agent
Let's create a test script to verify our agent works correctly:
import unittest
from unittest.mock import Mock, patch
# Simple test for our agent
class TestWebAgent(unittest.TestCase):
def setUp(self):
self.agent = SafeWebAgent("https://example.com")
def test_agent_initialization(self):
self.assertEqual(self.agent.base_url, "https://example.com")
def test_logging_functionality(self):
# Mock the logging to test if it's called
with patch('builtins.print') as mock_print:
self.agent.log_action("TEST", "Test message")
# Check if logging was called
self.assertTrue(len(self.agent.action_log) > 0)
if __name__ == "__main__":
unittest.main()Why we do this: Testing ensures our AI agent behaves as expected and helps prevent bugs before they cause problems. Testing is especially important for AI systems that interact with external resources.
Step 7: Run Your Complete Agent
Now let's run our complete agent to see it in action:
# Complete example run
if __name__ == "__main__":
print("Starting AI Web Agent Test")
# Create agent
agent = SafeWebAgent("https://example.com")
# Test fetching
print("\n--- Testing Content Fetch ---")
soup = agent.safe_fetch('/')
# Test analysis
print("\n--- Testing Content Analysis ---")
analysis = agent.analyze_content(soup)
if analysis:
print("Analysis results:", analysis)
# Test modification logging
print("\n--- Testing Modification Logging ---")
agent.safe_modify(soup, ["Add new content"])
print("\nAgent activity logged in agent_activity.log")
print("\n--- Test Complete ---")Why we do this: Running the complete example demonstrates how all components work together and shows the logging system in action, which is crucial for understanding how AI agents should operate responsibly.
Summary
In this tutorial, you've learned how to build a basic AI web agent using Python. You've created a system that can fetch web content, analyze it, and log its activities - all while implementing safety measures. The OpenAI incident highlights the importance of responsible AI practices, and this tutorial demonstrates how to build AI systems that are both functional and accountable. Remember that when working with AI agents that interact with real websites, always implement proper logging, rate limiting, and safety checks to prevent unintended consequences.
