Swedish court orders Google to pay Klarna nearly $2 billion in antitrust damages

Introduction

In this tutorial, we'll explore how to build a web scraper that analyzes search engine results to detect potential anti-competitive behavior, similar to what was investigated in the Swedish antitrust case against Google. This involves understanding how search results are structured, how to extract relevant data, and how to analyze patterns that might indicate favoritism. We'll create a Python-based tool that can scrape search results and compare them for bias.

Prerequisites

• Python 3.7 or higher installed
• Basic understanding of web scraping concepts
• Knowledge of HTML and CSS selectors
• Understanding of Python requests library
• Basic familiarity with data analysis libraries (pandas, numpy)

Step-by-Step Instructions

Step 1: Set Up Your Development Environment

First, we need to install the required Python libraries. Open your terminal or command prompt and run:

pip install requests beautifulsoup4 pandas numpy

Why: These libraries provide the foundation for web scraping, data manipulation, and analysis. Requests handles HTTP communication, BeautifulSoup parses HTML, and pandas/numpy help with data processing.

Step 2: Create the Search Result Scraper

Let's start by creating a basic scraper that can fetch and parse search results:

import requests
from bs4 import BeautifulSoup
import time
import random

class SearchResultsScraper:
    def __init__(self):
        self.session = requests.Session()
        self.session.headers.update({
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
        })

    def scrape_search_results(self, query, num_results=10):
        url = f'https://www.google.com/search?q={query}&num={num_results}'
        try:
            response = self.session.get(url)
            response.raise_for_status()
            soup = BeautifulSoup(response.content, 'html.parser')
            
            results = []
            for item in soup.find_all('div', class_='g'):
                title_element = item.find('h3')
                link_element = item.find('a')
                
                if title_element and link_element:
                    title = title_element.get_text()
                    link = link_element.get('href')
                    
                    # Extract domain from URL
                    domain = self.extract_domain(link)
                    
                    results.append({
                        'title': title,
                        'link': link,
                        'domain': domain
                    })
            
            return results
        except Exception as e:
            print(f"Error scraping results: {e}")
            return []

    def extract_domain(self, url):
        if url and url.startswith('http'):
            from urllib.parse import urlparse
            return urlparse(url).netloc
        return 'unknown'

# Test the scraper
scraper = SearchResultsScraper()
results = scraper.scrape_search_results('shopping comparison sites', 10)
for result in results:
    print(f"{result['title']} - {result['domain']}")

Why: This creates a foundation for analyzing search results, which is crucial for detecting patterns of favoritism. We're using Google's search results page structure to extract relevant information.

Step 3: Implement a Bias Detection Algorithm

Now, let's build a system that can detect potential bias by analyzing the distribution of results:

import pandas as pd
from collections import Counter

class BiasAnalyzer:
    def __init__(self):
        self.scraper = SearchResultsScraper()

    def analyze_bias(self, query, competitor_domains, num_searches=5):
        all_results = []
        
        for i in range(num_searches):
            # Add random delay to avoid being blocked
            time.sleep(random.uniform(1, 3))
            results = self.scraper.scrape_search_results(query, 10)
            all_results.extend(results)
            
        # Convert to DataFrame for analysis
        df = pd.DataFrame(all_results)
        
        # Count occurrences of each domain
        domain_counts = Counter(df['domain'])
        
        # Calculate percentage distribution
        total_results = len(df)
        
        # Analyze competitor vs. non-competitor domains
        competitor_count = sum(domain_counts[domain] for domain in competitor_domains if domain in domain_counts)
        non_competitor_count = total_results - competitor_count
        
        competitor_percentage = (competitor_count / total_results) * 100
        
        return {
            'query': query,
            'total_results': total_results,
            'competitor_results': competitor_count,
            'competitor_percentage': round(competitor_percentage, 2),
            'domain_distribution': dict(domain_counts),
            'bias_score': self.calculate_bias_score(competitor_percentage, total_results)
        }

    def calculate_bias_score(self, competitor_percentage, total_results):
        # Simple bias score calculation
        # If competitor sites appear more than 50% of the time, it might indicate bias
        if competitor_percentage > 50:
            return 'High bias detected'
        elif competitor_percentage > 30:
            return 'Moderate bias possible'
        else:
            return 'Low bias observed'

# Example usage
analyzer = BiasAnalyzer()
competitor_domains = ['klarna.com', 'pricerunner.com', 'shopzilla.com']
analysis = analyzer.analyze_bias('shopping comparison sites', competitor_domains)
print(f"Analysis for '{analysis['query']}':")
print(f"Total results: {analysis['total_results']}")
print(f"Competitor results: {analysis['competitor_results']} ({analysis['competitor_percentage']}%)")
print(f"Bias score: {analysis['bias_score']}")

Why: This algorithm helps identify when certain competitors are disproportionately favored in search results, which is a key indicator of anti-competitive behavior similar to what was found in the Swedish case.

Step 4: Add Data Visualization

Let's enhance our tool with visualization capabilities to better understand the results:

import matplotlib.pyplot as plt

# Add this method to the BiasAnalyzer class
    def visualize_results(self, analysis):
        domains = list(analysis['domain_distribution'].keys())
        counts = list(analysis['domain_distribution'].values())
        
        # Create bar chart
        plt.figure(figsize=(12, 6))
        bars = plt.bar(range(len(domains)), counts)
        plt.xlabel('Domains')
        plt.ylabel('Number of Results')
        plt.title(f'Search Results Distribution for: {analysis["query"]}')
        plt.xticks(range(len(domains)), domains, rotation=45, ha='right')
        
        # Color competitors differently
        for i, (bar, domain) in enumerate(zip(bars, domains)):
            if domain in ['klarna.com', 'pricerunner.com']:
                bar.set_color('red')
            else:
                bar.set_color('blue')
        
        plt.tight_layout()
        plt.show()

# Usage example
analyzer.visualize_results(analysis)

Why: Visualization makes it easier to spot patterns and biases at a glance. Red bars for competitor domains help quickly identify if these sites are being favored in search results.

Step 5: Build a Comprehensive Analysis Tool

Let's create a complete tool that can analyze multiple queries and provide a comprehensive report:

class CompetitionAnalyzer:
    def __init__(self):
        self.analyzer = BiasAnalyzer()
        self.results = []

    def run_comprehensive_analysis(self, queries, competitor_domains):
        for query in queries:
            print(f"Analyzing query: {query}")
            analysis = self.analyzer.analyze_bias(query, competitor_domains)
            self.results.append(analysis)
            print(f"Bias Score: {analysis['bias_score']}")
            print("---")
        
        return self.results

    def generate_report(self):
        df = pd.DataFrame(self.results)
        print("\n=== COMPREHENSIVE ANALYSIS REPORT ===")
        print(df[['query', 'competitor_percentage', 'bias_score']])
        
        # Summary statistics
        avg_competitor_percentage = df['competitor_percentage'].mean()
        print(f"\nAverage competitor percentage across queries: {avg_competitor_percentage:.2f}%")
        
        high_bias_queries = df[df['bias_score'] == 'High bias detected']
        print(f"Queries with high bias: {len(high_bias_queries)}")

# Example usage
queries = ['shopping comparison sites', 'online shopping', 'price comparison tools']
competitor_domains = ['klarna.com', 'pricerunner.com', 'shopzilla.com']

analyzer_tool = CompetitionAnalyzer()
results = analyzer_tool.run_comprehensive_analysis(queries, competitor_domains)
analyzer_tool.generate_report()

Why: This comprehensive tool allows for systematic analysis of multiple search queries, helping to build a case for potential anti-competitive behavior by identifying consistent patterns of favoritism.

Summary

In this tutorial, we've built a web scraping and analysis tool that can detect potential anti-competitive behavior in search results. By scraping search results, analyzing their distribution, and visualizing patterns, we've created a system that could be used to investigate cases like the Swedish court ruling against Google. The tool can help identify when competitors are disproportionately favored in search results, which is a key indicator of potential antitrust violations.

This approach demonstrates how technology can be used to monitor and analyze market competition, providing evidence that could be valuable in antitrust investigations and legal proceedings.