Introduction
In this tutorial, we'll build a practical AI text detection system inspired by Pangram's advanced text classifier. You'll learn how to implement a machine learning model that can distinguish between human-written and AI-generated text using Python and scikit-learn. This system will help you understand the core concepts behind AI text detection while building a functional prototype.
Prerequisites
- Python 3.7 or higher installed
- Basic understanding of machine learning concepts
- Knowledge of Python programming and data manipulation
- Installed packages: scikit-learn, pandas, numpy, matplotlib
Why these prerequisites? Understanding basic ML concepts will help you grasp how our model learns to differentiate between text types. The Python packages provide the tools needed for data processing, model training, and visualization.
Step-by-Step Instructions
1. Set Up Your Development Environment
First, create a new Python environment and install the required packages:
pip install scikit-learn pandas numpy matplotlib2. Create Sample Data
We'll generate synthetic training data that mimics human and AI-generated text patterns:
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
# Generate synthetic dataset
np.random.seed(42)
# Create features that might distinguish human vs AI text
n_samples = 10000
# Features: word length, punctuation, sentence complexity
human_text_features = {
'avg_word_length': np.random.normal(5, 1.5, n_samples),
'punctuation_count': np.random.poisson(5, n_samples),
'sentence_length': np.random.normal(15, 5, n_samples),
'repetition_ratio': np.random.uniform(0.01, 0.2, n_samples),
'complexity_score': np.random.uniform(0.1, 0.9, n_samples)
}
ai_text_features = {
'avg_word_length': np.random.normal(4, 0.8, n_samples),
'punctuation_count': np.random.poisson(3, n_samples),
'sentence_length': np.random.normal(12, 3, n_samples),
'repetition_ratio': np.random.uniform(0.05, 0.3, n_samples),
'complexity_score': np.random.uniform(0.2, 0.8, n_samples)
}
# Combine into dataframes
human_df = pd.DataFrame(human_text_features)
ai_df = pd.DataFrame(ai_text_features)
# Add labels
human_df['label'] = 0 # Human
ai_df['label'] = 1 # AI
data = pd.concat([human_df, ai_df], ignore_index=True)
print("Dataset shape:", data.shape)
print("Label distribution:")
print(data['label'].value_counts())3. Prepare the Data
Split the dataset into training and testing sets, and scale the features:
# Separate features and target
X = data.drop('label', axis=1)
y = data['label']
# Split the data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Scale features
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
print("Training set shape:", X_train_scaled.shape)
print("Test set shape:", X_test_scaled.shape)4. Train the AI Text Detector Model
Use a Random Forest classifier, which is effective for text classification tasks:
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix
# Initialize and train the model
rf_model = RandomForestClassifier(n_estimators=100, random_state=42)
rf_model.fit(X_train_scaled, y_train)
# Make predictions
y_pred = rf_model.predict(X_test_scaled)
# Evaluate performance
accuracy = accuracy_score(y_test, y_pred)
print(f"Model Accuracy: {accuracy:.4f}")
print("\nClassification Report:")
print(classification_report(y_test, y_pred))5. Test the Model with New Text Samples
Create a function to test new text samples and predict whether they're human or AI-generated:
def predict_text_type(text_features, model, scaler):
"""
Predict if text is human or AI-generated
text_features: dict with avg_word_length, punctuation_count, etc.
"""
# Convert to dataframe
features_df = pd.DataFrame([text_features])
# Scale features
features_scaled = scaler.transform(features_df)
# Make prediction
prediction = model.predict(features_scaled)[0]
probability = model.predict_proba(features_scaled)[0]
return "AI-generated" if prediction == 1 else "Human-written", probability
# Test with sample texts
sample_texts = [
{'avg_word_length': 4.5, 'punctuation_count': 3, 'sentence_length': 12, 'repetition_ratio': 0.1, 'complexity_score': 0.3},
{'avg_word_length': 5.2, 'punctuation_count': 6, 'sentence_length': 18, 'repetition_ratio': 0.05, 'complexity_score': 0.7}
]
for i, text in enumerate(sample_texts):
prediction, probability = predict_text_type(text, rf_model, scaler)
print(f"Sample {i+1}: {prediction}")
print(f"Confidence: {max(probability):.4f}")
print("---")6. Visualize Results
Create visualizations to better understand model performance:
import matplotlib.pyplot as plt
# Confusion Matrix
plt.figure(figsize=(8, 6))
conf_matrix = confusion_matrix(y_test, y_pred)
plt.imshow(conf_matrix, interpolation='nearest', cmap=plt.cm.Blues)
plt.title('Confusion Matrix')
plt.colorbar()
plt.xlabel('Predicted Label')
plt.ylabel('True Label')
plt.show()
# Feature importance
feature_names = ['avg_word_length', 'punctuation_count', 'sentence_length', 'repetition_ratio', 'complexity_score']
importances = rf_model.feature_importances_
indices = np.argsort(importances)[::-1]
plt.figure(figsize=(10, 6))
plt.title("Feature Importance")
plt.bar(range(len(importances)), importances[indices])
plt.xticks(range(len(importances)), [feature_names[i] for i in indices], rotation=45)
plt.tight_layout()
plt.show()7. Improve Model Performance
Enhance the model by tuning hyperparameters:
from sklearn.model_selection import GridSearchCV
# Define parameter grid
param_grid = {
'n_estimators': [50, 100, 200],
'max_depth': [None, 10, 20],
'min_samples_split': [2, 5, 10]
}
# Perform grid search
grid_search = GridSearchCV(
RandomForestClassifier(random_state=42),
param_grid,
cv=5,
scoring='accuracy',
n_jobs=-1
)
grid_search.fit(X_train_scaled, y_train)
print("Best parameters:", grid_search.best_params_)
print("Best cross-validation score:", grid_search.best_score_)
# Use best model
best_model = grid_search.best_estimator_
y_pred_best = best_model.predict(X_test_scaled)
accuracy_best = accuracy_score(y_test, y_pred_best)
print(f"Improved model accuracy: {accuracy_best:.4f}")Summary
In this tutorial, you've built a practical AI text detection system that can distinguish between human and AI-generated text. You learned how to prepare data, train a machine learning model, evaluate its performance, and improve it through hyperparameter tuning. This approach mirrors the techniques used by companies like Pangram in their advanced text detection systems.
The system you've created demonstrates core concepts of text classification, including feature engineering, model training, and evaluation metrics. While our synthetic dataset is simplified, real-world applications would use more sophisticated features like n-gram analysis, linguistic patterns, and transformer-based embeddings to achieve the high accuracy levels mentioned in the news article.
Remember that building production-grade text detectors requires large, high-quality datasets and may involve more complex architectures like neural networks or transformer models for optimal performance.
