A third of ChatGPT ads appear in irrelevant conversations

Introduction

In this tutorial, we'll explore how to analyze and evaluate the relevance of AI-generated advertisements within conversational AI systems like ChatGPT. Based on recent findings that a third of ChatGPT ads appear in irrelevant conversations, we'll build a tool that can assess ad relevance by examining conversation context and ad content. This is crucial for understanding AI advertising effectiveness and improving user experience in conversational interfaces.

Prerequisites

• Python 3.7 or higher installed
• Basic understanding of natural language processing concepts
• Knowledge of API interactions and JSON data handling
• Installed libraries: openai, scikit-learn, numpy, pandas
• Access to OpenAI API key

Step-by-Step Instructions

1. Set up your development environment

First, create a new Python project directory and install the required dependencies:

mkdir chatgpt-ad-analyzer
 cd chatgpt-ad-analyzer
pip install openai scikit-learn numpy pandas

This creates a clean project space and installs all necessary libraries for our analysis.

2. Initialize OpenAI client and configure API access

Create a Python file called ad_analyzer.py and start by setting up your OpenAI client:

import openai
import os
from dotenv import load_dotenv

# Load environment variables
load_dotenv()

# Initialize OpenAI client
openai.api_key = os.getenv('OPENAI_API_KEY')

# Set up model parameters
MODEL = "gpt-4-turbo"
MAX_TOKENS = 1000

This sets up our connection to OpenAI's API, which we'll use to analyze both conversations and ad relevance.

3. Create conversation and ad data structures

Define the data structure to hold conversation and ad information:

class Conversation:
    def __init__(self, id, messages):
        self.id = id
        self.messages = messages
        
    def get_context(self):
        # Extract relevant conversation context
        return " ".join([msg['content'] for msg in self.messages[-3:]])

    def get_full_text(self):
        return " ".join([msg['content'] for msg in self.messages])


class Ad:
    def __init__(self, id, content, target_audience):
        self.id = id
        self.content = content
        self.target_audience = target_audience

This structure allows us to organize conversation data and ad information for analysis.

4. Implement ad relevance scoring function

Create a function that uses OpenAI to determine how relevant an ad is to a conversation:

def evaluate_ad_relevance(conversation_context, ad_content):
    """Evaluate the relevance of an ad to a conversation context"""
    prompt = f"""
    You are an AI assistant that evaluates ad relevance to conversations.
    
    Conversation context: {conversation_context}
    
    Ad content: {ad_content}
    
    Rate the relevance of this ad to the conversation on a scale of 1-10 (1 = completely irrelevant, 10 = perfectly relevant).
    
    Provide only a number between 1-10 and explain your reasoning briefly.
    """
    
    try:
        response = openai.ChatCompletion.create(
            model=MODEL,
            messages=[
                {"role": "system", "content": "You are a helpful assistant that evaluates ad relevance."},
                {"role": "user", "content": prompt}
            ],
            max_tokens=MAX_TOKENS,
            temperature=0.3
        )
        
        result = response['choices'][0]['message']['content']
        
        # Extract numerical score from response
        import re
        score_match = re.search(r'(\d+)', result)
        score = int(score_match.group(1)) if score_match else 5
        
        return score, result
    except Exception as e:
        print(f"Error evaluating relevance: {e}")
        return 5, "Error in evaluation"

# Example usage
conversation = Conversation("conv_123", [
    {"role": "user", "content": "I'm looking for a new laptop for graphic design work"},
    {"role": "assistant", "content": "What specific features are you looking for in a laptop?"}
])

ad = Ad("ad_456", "Get 20% off our premium gaming laptops with powerful graphics cards", "gaming enthusiasts")

score, explanation = evaluate_ad_relevance(conversation.get_context(), ad.content)
print(f"Relevance score: {score}/10")
print(f"Explanation: {explanation}")

This function uses OpenAI's reasoning capabilities to assess relevance, which is essential for understanding why ads appear in irrelevant conversations.

5. Build a batch analysis function

Create a function to analyze multiple conversations and ads:

def analyze_ad_conversation_pairs(conversations, ads):
    """Analyze multiple ad-conversation pairs"""
    results = []
    
    for conv in conversations:
        for ad in ads:
            context = conv.get_context()
            score, explanation = evaluate_ad_relevance(context, ad.content)
            
            result = {
                'conversation_id': conv.id,
                'ad_id': ad.id,
                'relevance_score': score,
                'explanation': explanation,
                'conversation_context': context[:100] + "..." if len(context) > 100 else context
            }
            
            results.append(result)
            
            # Add delay to avoid rate limiting
            import time
            time.sleep(0.5)
    
    return results

# Example data
conversations = [
    Conversation("conv_1", [
        {"role": "user", "content": "I need help finding a good restaurant for dinner"},
        {"role": "assistant", "content": "What type of cuisine are you interested in?"}
    ]),
    Conversation("conv_2", [
        {"role": "user", "content": "Can you recommend a good book to read?"},
        {"role": "assistant", "content": "What genre do you prefer?"}
    ])
]

ads = [
    Ad("ad_1", "New smartphone with advanced camera features", "tech enthusiasts"),
    Ad("ad_2", "Best restaurants in downtown for date night", "food lovers")
]

# Run analysis
results = analyze_ad_conversation_pairs(conversations, ads)
for result in results:
    print(f"Conversation {result['conversation_id']} - Ad {result['ad_id']}: {result['relevance_score']}/10")

This batch processing function allows us to systematically evaluate multiple ad-conversation pairs, which is crucial for large-scale analysis.

6. Create data visualization and reporting

Visualize the analysis results to identify patterns:

import pandas as pd
import matplotlib.pyplot as plt

def generate_report(results):
    """Generate a summary report from analysis results"""
    df = pd.DataFrame(results)
    
    # Basic statistics
    avg_score = df['relevance_score'].mean()
    total_pairs = len(df)
    
    # Count of highly relevant vs irrelevant ads
    relevant_count = len(df[df['relevance_score'] >= 7])
    irrelevant_count = len(df[df['relevance_score'] <= 3])
    
    print(f"\n=== AD RELEVANCE ANALYSIS REPORT ===")
    print(f"Total ad-conversation pairs analyzed: {total_pairs}")
    print(f"Average relevance score: {avg_score:.2f}/10")
    print(f"Highly relevant ads (7-10): {relevant_count}")
    print(f"Irrelevant ads (1-3): {irrelevant_count}")
    
    # Create visualization
    plt.figure(figsize=(10, 6))
    plt.hist(df['relevance_score'], bins=10, alpha=0.7, color='blue')
    plt.xlabel('Relevance Score')
    plt.ylabel('Frequency')
    plt.title('Distribution of Ad Relevance Scores')
    plt.axvline(avg_score, color='red', linestyle='--', label=f'Average: {avg_score:.2f}')
    plt.legend()
    plt.grid(True, alpha=0.3)
    plt.savefig('ad_relevance_distribution.png')
    plt.show()
    
    return df

# Generate report from our analysis
report_df = generate_report(results)
print(report_df[['conversation_id', 'ad_id', 'relevance_score']])

This visualization helps identify the proportion of irrelevant ads, directly addressing the issue mentioned in the news article.

Summary

In this tutorial, we've built a practical tool to analyze ad relevance in conversational AI systems like ChatGPT. By leveraging OpenAI's language understanding capabilities, we can evaluate how well ads match conversation contexts. This approach helps identify the root cause of why a third of ChatGPT ads appear irrelevant - it's not just about the AI's understanding, but also about the contextual matching process. The tool provides actionable insights for improving AI advertising systems and enhancing user experience in conversational interfaces.

The key takeaway is that while AI systems can understand conversation content, they still require proper contextual matching to ensure ad relevance. This analysis can be extended to monitor real-time ad performance and improve targeting algorithms.