Introduction
In this tutorial, you'll learn how to create a smart outdoor deals tracker using Python and web scraping techniques. This tool will help you monitor Memorial Day deals on outdoor equipment like lawn mowers and power banks, similar to what you might see in the ZDNet article. We'll build a system that automatically fetches deal data from online retailers and presents it in an organized way.
Prerequisites
- Python 3.7 or higher installed on your system
- Familiarity with Python programming concepts
- Basic understanding of web scraping and HTML structure
- Required Python libraries: requests, BeautifulSoup4, pandas, schedule
Step-by-Step Instructions
1. Setting Up Your Development Environment
1.1 Install Required Libraries
First, you need to install the necessary Python libraries for web scraping and data handling. Open your terminal or command prompt and run:
pip install requests beautifulsoup4 pandas scheduleWhy this step? These libraries provide the core functionality we need: requests for making HTTP requests to websites, BeautifulSoup for parsing HTML content, pandas for data manipulation and storage, and schedule for automating our scraping tasks.
1.2 Create Project Structure
Create a new directory for your project and set up the following files:
outdoor_deals_tracker/
├── main.py
├── deal_scraper.py
├── data_processor.py
└── deals.csvThis structure will help organize our code and make it maintainable.
2. Building the Web Scraper
2.1 Create the Deal Scraper Module
Create deal_scraper.py with the following code:
import requests
from bs4 import BeautifulSoup
import time
def scrape_lawn_mower_deals():
# Simulated scraping function
# In a real implementation, you would target actual retailer websites
deals = [
{
'product': 'Lawn Mower XYZ-2023',
'price': '$299.99',
'original_price': '$399.99',
'discount': '25% off',
'store': 'HomeDepot',
'url': 'https://homedepot.com/lawnmower'
},
{
'product': 'Electric Lawn Mower Pro',
'price': '$199.99',
'original_price': '$249.99',
'discount': '20% off',
'store': 'Amazon',
'url': 'https://amazon.com/lawnmower'
}
]
return deals
def scrape_power_bank_deals():
deals = [
{
'product': 'Power Bank 20000mAh',
'price': '$39.99',
'original_price': '$59.99',
'discount': '33% off',
'store': 'BestBuy',
'url': 'https://bestbuy.com/powerbank'
}
]
return deals
def scrape_outdoor_deals():
print('Scraping outdoor deals...')
all_deals = []
# Scrape lawn mower deals
mower_deals = scrape_lawn_mower_deals()
all_deals.extend(mower_deals)
# Scrape power bank deals
power_deals = scrape_power_bank_deals()
all_deals.extend(power_deals)
return all_dealsWhy this step? This module sets up the foundation for scraping deals from different retailers. In a real-world scenario, you would replace the simulated functions with actual scraping logic that targets specific retailer websites.
3. Creating the Data Processor
3.1 Implement Data Processing Module
Create data_processor.py to handle data manipulation and storage:
import pandas as pd
import os
from datetime import datetime
def save_deals_to_csv(deals, filename='deals.csv'):
# Convert deals to DataFrame
df = pd.DataFrame(deals)
# Add timestamp
df['scraped_at'] = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
# Save to CSV
if os.path.exists(filename):
# Append to existing file
df.to_csv(filename, mode='a', header=False, index=False)
else:
# Create new file
df.to_csv(filename, index=False)
print(f'Saved {len(deals)} deals to {filename}')
def display_deals(deals):
if not deals:
print('No deals found')
return
df = pd.DataFrame(deals)
print('\n=== Current Outdoor Deals ===')
print(df.to_string(index=False))
def get_best_deals(deals, min_discount=20):
# Filter deals based on minimum discount percentage
df = pd.DataFrame(deals)
# Extract discount percentage
df['discount_percent'] = df['discount'].str.extract(r'(\d+)%').astype(int)
# Filter by minimum discount
best_deals = df[df['discount_percent'] >= min_discount]
return best_deals.to_dict('records')Why this step? This module handles the data processing and storage. It converts scraped data into a structured format, saves it to CSV files for persistence, and provides functionality to filter and display deals based on criteria like discount percentage.
4. Main Application Logic
4.1 Create the Main Application
Create main.py to orchestrate the entire process:
import schedule
import time
from deal_scraper import scrape_outdoor_deals
from data_processor import save_deals_to_csv, display_deals, get_best_deals
def job():
print('Running scheduled deal check...')
# Scrape deals
deals = scrape_outdoor_deals()
# Save to CSV
save_deals_to_csv(deals)
# Display current deals
display_deals(deals)
# Show best deals
best_deals = get_best_deals(deals, min_discount=25)
if best_deals:
print('\n=== BEST DEALS (25%+ off) ===')
display_deals(best_deals)
else:
print('\nNo deals found with 25%+ discount')
def main():
# Run immediately
job()
# Schedule to run every 2 hours
schedule.every(2).hours.do(job)
print('Deal tracker started. Press Ctrl+C to stop.')
# Keep the script running
try:
while True:
schedule.run_pending()
time.sleep(1)
except KeyboardInterrupt:
print('\nStopping deal tracker...')
n = __name__
if main():
main()Why this step? This is the core of our application that ties everything together. It schedules regular deal checks, runs the scraping process, and provides a user-friendly display of results.
5. Running Your Deal Tracker
5.1 Execute the Application
Run your application with:
python main.pyThe application will immediately check for deals and then continue running, checking for new deals every 2 hours.
5.2 View Results
After running, you'll see output similar to:
Running scheduled deal check...
Saved 3 deals to deals.csv
=== Current Outdoor Deals ===
product price original_price discount store url
Lawn Mower XYZ-2023 $299.99 $399.99 25% off HomeDepot https://homedepot.com/lawnmower
Electric Lawn Mower Pro $199.99 $249.99 20% off Amazon https://amazon.com/lawnmower
Power Bank 20000mAh $39.99 $59.99 33% off BestBuy https://bestbuy.com/powerbank
=== BEST DEALS (25%+ off) ===
product price original_price discount store url
Lawn Mower XYZ-2023 $299.99 $399.99 25% off HomeDepot https://homedepot.com/lawnmowerWhy this step? This demonstrates the complete workflow of your deal tracking system, showing how it collects, processes, and displays information in a useful format.
6. Enhancing Your Deal Tracker
6.1 Add Email Notifications
For a more advanced version, you could add email notifications for significant deals:
import smtplib
from email.mime.text import MIMEText
def send_email_notification(subject, message):
# Configure your email settings
# This is a simplified example - you'd need to configure real email credentials
print(f'Would send email: {subject}')
print(message)Why this step? Adding email notifications would make your deal tracker more proactive, alerting you to the best deals as they're discovered rather than requiring you to check manually.
Summary
In this tutorial, you've built a smart outdoor deals tracker that scrapes and organizes Memorial Day deals for lawn mowers, power banks, and other outdoor equipment. You've learned how to structure a web scraping application, process and store data using pandas, and automate regular checks using the schedule library. This system can be extended to include more retailers, additional deal types, and even notifications when specific deals are found. The modular approach makes it easy to add new features or scrape from different sources, making it a flexible tool for monitoring online deals.
