Anthropic acquires computer-use AI startup Vercept after Meta poached one of its founders

Introduction

In this tutorial, you'll learn how to create a basic computer-use AI agent that can interact with graphical user interfaces (GUIs) using Python and Selenium. This technology, similar to what Vercept developed, enables AI systems to perform tasks within applications just like a human would. You'll build a simple web automation agent that can navigate websites, fill forms, and click buttons - the foundation of what's being developed in the AI computer-use space.

Prerequisites

Before starting this tutorial, you'll need:

• Python 3.7 or higher installed on your system
• Basic understanding of Python programming concepts
• Internet connection for downloading dependencies
• Text editor or IDE (like VS Code or PyCharm)

Step-by-Step Instructions

1. Set up your Python environment

First, create a new virtual environment to keep your project dependencies isolated:

python -m venv computer_agent_env
computer_agent_env\Scripts\activate  # On Windows
# or
source computer_agent_env/bin/activate  # On macOS/Linux

Why: Using a virtual environment ensures that your project dependencies don't interfere with other Python projects on your system.

2. Install required packages

Install Selenium and related dependencies:

pip install selenium webdriver-manager
pip install pillow

Why: Selenium provides the web automation capabilities, webdriver-manager handles browser driver management, and Pillow helps with image processing tasks.

3. Create the basic agent structure

Create a new Python file called computer_agent.py and start with the basic imports:

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager


class ComputerUseAgent:
    def __init__(self):
        self.driver = None
        self.setup_driver()

    def setup_driver(self):
        # Set up Chrome options
        chrome_options = webdriver.ChromeOptions()
        chrome_options.add_argument('--no-sandbox')
        chrome_options.add_argument('--disable-dev-shm-usage')
        
        # Initialize the driver
        service = Service(ChromeDriverManager().install())
        self.driver = webdriver.Chrome(service=service, options=chrome_options)

    def navigate_to(self, url):
        self.driver.get(url)
        print(f'Navigated to {url}')

Why: This creates a reusable class structure that can be extended with more sophisticated computer-use capabilities.

4. Add basic interaction methods

Add methods to interact with web elements:

    def find_and_click(self, element_locator, locator_type=By.ID):
        try:
            element = WebDriverWait(self.driver, 10).until(
                EC.element_to_be_clickable((locator_type, element_locator))
            )
            element.click()
            print(f'Clicked on element: {element_locator}')
            return True
        except Exception as e:
            print(f'Failed to click element: {e}')
            return False

    def fill_form_field(self, element_locator, text, locator_type=By.ID):
        try:
            element = WebDriverWait(self.driver, 10).until(
                EC.presence_of_element_located((locator_type, element_locator))
            )
            element.clear()
            element.send_keys(text)
            print(f'Filled {element_locator} with: {text}')
            return True
        except Exception as e:
            print(f'Failed to fill form field: {e}')
            return False

    def close(self):
        if self.driver:
            self.driver.quit()

Why: These methods provide the fundamental building blocks for GUI interaction - finding elements and performing actions like clicking and typing.

5. Create a sample task execution

Add a method to demonstrate how the agent might complete a simple task:

    def search_github(self, query):
        # Navigate to GitHub
        self.navigate_to('https://github.com')
        
        # Find and fill search box
        search_box = 'query-builder-search'
        self.fill_form_field(search_box, query)
        
        # Click search button
        search_button = 'search-button'
        self.find_and_click(search_button)
        
        # Wait for results
        time.sleep(3)
        print(f'Search completed for: {query}')

Why: This demonstrates how your agent can perform a real-world task similar to what Vercept's agents can do - completing multi-step operations within applications.

6. Test your agent

Create a main execution block to test your agent:

if __name__ == '__main__':
    # Create agent instance
    agent = ComputerUseAgent()
    
    try:
        # Execute a simple task
        agent.search_github('python automation')
        
        # Wait to see results
        time.sleep(5)
        
    except Exception as e:
        print(f'Error during execution: {e}')
    finally:
        # Always close the driver
        agent.close()

Why: This structure ensures your agent works properly and cleans up resources properly after execution.

Summary

In this tutorial, you've built a foundational computer-use AI agent using Selenium WebDriver. This agent can navigate websites, fill forms, and click buttons - the core capabilities that enable AI systems to interact with graphical interfaces. While this is a simplified version of what companies like Vercept are developing, it demonstrates the fundamental building blocks of GUI interaction automation. As you continue developing, you can extend this agent with more sophisticated features like image recognition, natural language processing, and complex multi-step workflows that mirror human-like computer use.

The technology you've learned about is at the forefront of AI development, enabling systems to perform tasks in real applications rather than just processing text or data in isolation.