Introduction
In this tutorial, you'll learn how to analyze web content for AI-generated text using Python and natural language processing techniques. This builds on recent findings from the Pew Research Center, which revealed that over a third of web pages published since ChatGPT's launch contain machine-written text. By the end of this tutorial, you'll have created a tool that can detect potential AI-generated content in web pages.
Prerequisites
- Python 3.7 or higher installed on your system
- Basic understanding of web scraping and HTML parsing
- Intermediate knowledge of natural language processing concepts
- Required Python packages: requests, BeautifulSoup, nltk, transformers
Step-by-Step Instructions
1. Set Up Your Python Environment
First, create a virtual environment and install the required packages. This ensures you have a clean, isolated environment for this project.
python -m venv ai_content_detector
source ai_content_detector/bin/activate # On Windows: ai_content_detector\Scripts\activate
pip install requests beautifulsoup4 nltk transformers torch2. Download Required NLTK Data
Download the necessary NLTK datasets for text processing. These will help with identifying linguistic patterns commonly found in AI-generated text.
import nltk
nltk.download('punkt')
nltk.download('stopwords')
nltk.download('vader_lexicon')3. Create the Main Analysis Class
Now, we'll create a class that will handle the core functionality of detecting AI-generated text. This class will use multiple heuristics to identify suspicious patterns.
import requests
from bs4 import BeautifulSoup
from collections import Counter
import nltk
from nltk.sentiment import SentimentIntensityAnalyzer
from nltk.corpus import stopwords
import re
class AIContentDetector:
def __init__(self):
self.stop_words = set(stopwords.words('english'))
self.sia = SentimentIntensityAnalyzer()
def extract_text_from_html(self, html_content):
soup = BeautifulSoup(html_content, 'html.parser')
# Remove script and style elements
for script in soup(['script', 'style']):
script.decompose()
# Get text and clean it
text = soup.get_text()
# Break into lines and remove leading/trailing space on each
lines = (line.strip() for line in text.splitlines())
# Break multi-headlines into a line each
chunks = (phrase.strip() for line in lines for phrase in line.split(" "))
# Drop blank lines
text = ' '.join(chunk for chunk in chunks if chunk)
return text
def analyze_readability(self, text):
# Simple readability metrics
sentences = nltk.sent_tokenize(text)
words = nltk.word_tokenize(text)
# Average sentence length
avg_sentence_length = len(words) / len(sentences) if sentences else 0
# Average word length
avg_word_length = sum(len(word) for word in words) / len(words) if words else 0
return {
'avg_sentence_length': avg_sentence_length,
'avg_word_length': avg_word_length,
'sentence_count': len(sentences),
'word_count': len(words)
}
def analyze_sentiment(self, text):
scores = self.sia.polarity_scores(text)
return scores
def analyze_word_frequency(self, text):
words = nltk.word_tokenize(text.lower())
# Remove stop words and non-alphabetic words
words = [word for word in words if word.isalpha() and word not in self.stop_words]
# Get frequency distribution
freq_dist = Counter(words)
# Check for unusual frequency patterns
most_common = freq_dist.most_common(10)
return {
'most_common_words': most_common,
'unique_word_ratio': len(freq_dist) / len(words) if words else 0
}
def detect_ai_patterns(self, text):
# This is where we implement various heuristics
patterns = {}
# Analyze readability
readability = self.analyze_readability(text)
patterns['readability'] = readability
# Analyze sentiment
sentiment = self.analyze_sentiment(text)
patterns['sentiment'] = sentiment
# Analyze word frequency
frequency = self.analyze_word_frequency(text)
patterns['frequency'] = frequency
# Check for common AI-generated patterns
patterns['has_repetitive_phrasing'] = self.check_repetitive_phrasing(text)
patterns['has_excessive_punctuation'] = self.check_excessive_punctuation(text)
return patterns
def check_repetitive_phrasing(self, text):
# Look for repeated phrases or structures
sentences = nltk.sent_tokenize(text)
if len(sentences) < 5:
return False
# Simple check for very similar sentence structures
sentence_lengths = [len(nltk.word_tokenize(sent)) for sent in sentences]
if len(set(sentence_lengths)) == 1 and sentence_lengths[0] > 10:
return True
return False
def check_excessive_punctuation(self, text):
# AI-generated text often has unusual punctuation patterns
exclamation_count = text.count('!')
question_count = text.count('?')
total_punctuation = exclamation_count + question_count
word_count = len(nltk.word_tokenize(text))
if word_count > 0 and total_punctuation / word_count > 0.05:
return True
return False4. Integrate with Hugging Face Transformers
To enhance our detection capabilities, we'll use a pre-trained model from Hugging Face that can help identify AI-generated text patterns.
from transformers import pipeline
class AdvancedAIContentDetector(AIContentDetector):
def __init__(self):
super().__init__()
# Load a pre-trained text classification model
self.classifier = pipeline("text-classification", model="facebook/bart-large-mnli")
def classify_text(self, text):
try:
# Use the model to classify the text
result = self.classifier(text[:512]) # Limit to 512 tokens
return result
except Exception as e:
print(f"Error in classification: {e}")
return None5. Create a Web Page Analyzer
Now we'll create a function that can fetch a web page and analyze its content using our detector.
def analyze_webpage(url):
try:
# Fetch the web page
response = requests.get(url, timeout=10)
response.raise_for_status()
# Create detector instance
detector = AdvancedAIContentDetector()
# Extract text from HTML
text = detector.extract_text_from_html(response.text)
# Analyze the text
patterns = detector.detect_ai_patterns(text)
# Get model classification
classification = detector.classify_text(text)
return {
'url': url,
'text_length': len(text),
'patterns': patterns,
'classification': classification,
'content': text[:500] + '...' if len(text) > 500 else text
}
except Exception as e:
return {'error': str(e)}6. Run the Analysis
Finally, let's create a simple test to see how our detector works on a sample web page.
def main():
# Example usage
test_urls = [
'https://www.example.com', # Replace with actual URLs
'https://www.wikipedia.org',
]
for url in test_urls:
print(f"Analyzing: {url}")
result = analyze_webpage(url)
if 'error' in result:
print(f"Error: {result['error']}")
else:
print(f"Text length: {result['text_length']}")
print(f"Readability - Avg sentence length: {result['patterns']['readability']['avg_sentence_length']:.2f}")
print(f"Sentiment scores: {result['patterns']['sentiment']}")
print(f"Word frequency unique ratio: {result['patterns']['frequency']['unique_word_ratio']:.2f}")
print(f"Has repetitive phrasing: {result['patterns']['has_repetitive_phrasing']}")
print(f"Has excessive punctuation: {result['patterns']['has_excessive_punctuation']}")
print("---")
if __name__ == "__main__":
main()Summary
This tutorial has taught you how to build a web content analysis tool that can detect potential AI-generated text patterns. You've learned how to:
- Extract text from HTML content using BeautifulSoup
- Analyze readability and linguistic patterns
- Use sentiment analysis to identify unusual emotional tones
- Check for repetitive phrasing and punctuation patterns
- Integrate with Hugging Face's transformers for advanced text classification
While this tool provides a foundation for detecting AI-generated content, it's important to note that AI-generated text detection is an evolving field. As AI models become more sophisticated, detection methods must also evolve. This tool gives you a starting point that you can enhance with more advanced machine learning techniques and larger datasets for better accuracy.
