PyGraphistry Implementation Workflow for Interactive Graph Intelligence Pipelines in Security Analytics and Risk Investigation

Introduction

In this tutorial, we'll walk through implementing a PyGraphistry workflow for interactive graph analytics in security analytics and risk investigation. We'll create a synthetic dataset of enterprise access data, convert it into a graph structure, enrich it with various metrics, and visualize it using PyGraphistry's interactive capabilities. This workflow is particularly useful for security analysts who need to investigate access patterns, identify anomalies, and understand risk relationships in enterprise networks.

Prerequisites

Before beginning this tutorial, ensure you have the following:

• Python 3.7 or higher installed
• Basic understanding of graph theory and network analysis
• Experience with pandas and numpy data manipulation
• Access to a Python environment (local or Google Colab)

Step-by-Step Instructions

1. Install Required Libraries

First, we need to install the necessary Python packages. PyGraphistry requires several dependencies for graph analysis and visualization.

1.1 Install PyGraphistry and Related Packages

!pip install pygraphistry pandas numpy scikit-learn umap-learn networkx

Why this step? PyGraphistry is the core library for interactive graph visualization, while the others provide essential graph analysis and visualization capabilities.

2. Create Synthetic Enterprise Access Dataset

We'll generate a synthetic dataset representing enterprise access patterns with users, devices, IPs, services, roles, and geos.

2.1 Import Libraries and Generate Data

import pandas as pd
import numpy as np
import random
from datetime import datetime, timedelta
import pygraphistry

# Set random seed for reproducibility
np.random.seed(42)
random.seed(42)

# Generate synthetic data
users = [f'user_{i}' for i in range(100)]
device_types = ['laptop', 'desktop', 'mobile', 'tablet']
ips = [f'192.168.{random.randint(1, 255)}.{random.randint(1, 255)}' for _ in range(50)]
services = ['email', 'database', 'file_server', 'web_app', 'vpn']
roles = ['admin', 'developer', 'analyst', 'manager', 'intern']
geos = ['US', 'UK', 'DE', 'FR', 'JP', 'AU']

# Create user-device mapping
user_devices = pd.DataFrame({
    'user': np.random.choice(users, 200),
    'device': np.random.choice(device_types, 200),
    'ip': np.random.choice(ips, 200),
    'service': np.random.choice(services, 200),
    'role': np.random.choice(roles, 200),
    'geo': np.random.choice(geos, 200),
    'timestamp': [datetime.now() - timedelta(hours=random.randint(0, 24)) for _ in range(200)]
})

print('Generated user-device access data:')
print(user_devices.head())

Why this step? Creating a realistic dataset is crucial for meaningful graph analysis. This synthetic data represents typical enterprise access patterns that security analysts would encounter.

3. Convert Data to Graph Structure

Next, we'll convert our tabular data into nodes and edges for graph representation.

3.1 Create Nodes DataFrame

# Create nodes from unique entities
nodes = pd.DataFrame({
    'id': list(set(user_devices['user'].tolist() + user_devices['ip'].tolist() + user_devices['device'].tolist())),
    'type': ['user'] * len(set(user_devices['user'])) + ['ip'] * len(set(user_devices['ip'])) + ['device'] * len(set(user_devices['device'])),
    'label': list(set(user_devices['user'].tolist() + user_devices['ip'].tolist() + user_devices['device'].tolist()))
})

print('Nodes created:')
print(nodes.head())

3.2 Create Edges DataFrame

# Create edges based on user-device-ip relationships
edges = user_devices.groupby(['user', 'ip', 'device']).size().reset_index(name='count')
edges = edges.rename(columns={'user': 'source', 'ip': 'target'})

# Add service and role information to edges
edge_details = user_devices.groupby(['user', 'ip', 'device', 'service', 'role']).size().reset_index(name='count')
edge_details = edge_details[['user', 'ip', 'service', 'role', 'count']]
edge_details = edge_details.rename(columns={'user': 'source', 'ip': 'target'})

print('Edges created:')
print(edges.head())

Why this step? Graphs consist of nodes (entities) and edges (relationships). This conversion transforms our access data into a proper graph structure that can be analyzed and visualized.

4. Enrich Graph with Analytics Metrics

We'll enhance our graph with various metrics to support security analytics.

4.1 Calculate Risk Scores

# Calculate risk scores based on access patterns
risk_scores = user_devices.groupby('user').agg({
    'service': 'nunique',
    'device': 'nunique',
    'geo': 'nunique',
    'count': 'sum'
}).reset_index()
risk_scores = risk_scores.rename(columns={'service': 'unique_services', 'device': 'unique_devices', 'geo': 'unique_geos', 'count': 'access_count'})

# Create risk score based on multiple factors
risk_scores['risk_score'] = (
    risk_scores['unique_services'] * 0.3 +
    risk_scores['unique_devices'] * 0.2 +
    risk_scores['unique_geos'] * 0.2 +
    risk_scores['access_count'] * 0.3
)

# Normalize risk scores
risk_scores['risk_score'] = (risk_scores['risk_score'] - risk_scores['risk_score'].min()) / (risk_scores['risk_score'].max() - risk_scores['risk_score'].min())

print('Risk scores calculated:')
print(risk_scores.head())

4.2 Add Centrality Metrics

# For centrality metrics, we'll create a simplified approach
# In practice, you'd use networkx for full centrality calculations

# Create a basic centrality score based on access frequency
centrality_scores = user_devices.groupby('user').size().reset_index(name='access_frequency')
centrality_scores['centrality_score'] = (centrality_scores['access_frequency'] - centrality_scores['access_frequency'].min()) / (centrality_scores['access_frequency'].max() - centrality_scores['access_frequency'].min())

print('Centrality scores calculated:')
print(centrality_scores.head())

5. Apply Anomaly Detection

We'll use Isolation Forest to identify anomalous access patterns.

5.1 Prepare Data for Anomaly Detection

# Prepare data for Isolation Forest
from sklearn.ensemble import IsolationForest

# Create features for anomaly detection
anomaly_features = user_devices.groupby(['user', 'ip', 'device']).agg({
    'timestamp': 'count',
    'service': 'nunique',
    'geo': 'nunique'
}).reset_index()

anomaly_features = anomaly_features.rename(columns={'timestamp': 'access_count', 'service': 'unique_services', 'geo': 'unique_geos'})

# Encode categorical variables
anomaly_features['user_encoded'] = pd.factorize(anomaly_features['user'])[0]
anomaly_features['ip_encoded'] = pd.factorize(anomaly_features['ip'])[0]
anomaly_features['device_encoded'] = pd.factorize(anomaly_features['device'])[0]

# Prepare features for Isolation Forest
features = ['access_count', 'unique_services', 'unique_geos', 'user_encoded', 'ip_encoded', 'device_encoded']
X = anomaly_features[features]

print('Features for anomaly detection:')
print(X.head())

5.2 Apply Isolation Forest

# Apply Isolation Forest for anomaly detection
iso_forest = IsolationForest(contamination=0.1, random_state=42)
anomaly_scores = iso_forest.fit_predict(X)

# Add anomaly scores to our features
anomaly_features['anomaly_score'] = anomaly_scores

print('Anomaly scores applied:')
print(anomaly_features.head())

6. Generate UMAP Embeddings

UMAP provides dimensionality reduction for better visualization of our graph.

6.1 Apply UMAP for Layout

# Apply UMAP for dimensionality reduction
from umap import UMAP

# Prepare data for UMAP
umap_data = anomaly_features[features]

# Apply UMAP
umap = UMAP(n_components=2, random_state=42)
embedding = umap.fit_transform(umap_data)

# Add UMAP coordinates to our data
anomaly_features['umap_x'] = embedding[:, 0]
anomaly_features['umap_y'] = embedding[:, 1]

print('UMAP embeddings generated:')
print(anomaly_features[['umap_x', 'umap_y']].head())

7. Bind Graph to PyGraphistry

Finally, we'll bind our enriched graph data to PyGraphistry for visualization.

7.1 Prepare Data for PyGraphistry

# Prepare data for PyGraphistry
# Create a combined DataFrame with all metrics
combined_data = pd.merge(anomaly_features, risk_scores[['user', 'risk_score']], left_on='user', right_on='user', how='left')
combined_data = pd.merge(combined_data, centrality_scores[['user', 'centrality_score']], left_on='user', right_on='user', how='left')

# Create a final nodes DataFrame with all metrics
final_nodes = nodes.copy()
final_nodes = pd.merge(final_nodes, combined_data[['user', 'risk_score', 'centrality_score', 'anomaly_score', 'umap_x', 'umap_y']], left_on='label', right_on='user', how='left')

# Handle missing values
final_nodes['risk_score'] = final_nodes['risk_score'].fillna(0)
final_nodes['centrality_score'] = final_nodes['centrality_score'].fillna(0)
final_nodes['anomaly_score'] = final_nodes['anomaly_score'].fillna(0)
final_nodes['umap_x'] = final_nodes['umap_x'].fillna(0)
final_nodes['umap_y'] = final_nodes['umap_y'].fillna(0)

print('Final nodes with all metrics:')
print(final_nodes.head())

7.2 Create PyGraphistry Visualization

# Create PyGraphistry graph
# Note: For Colab, you'll need to register for a free account at https://www.pygraphistry.com/
# pygraphistry.register(api_key='your_api_key_here')

# Create graph
graph = pygraphistry.edges(edges, source='source', target='target', 
                           title='Enterprise Access Graph', 
                           description='Security analytics graph visualization')

# Add node attributes
graph = graph.nodes(final_nodes, node='id', 
                    title='Node Title', 
                    description='Node Description')

# Set node attributes
graph = graph.set_node_attributes(
    {'risk_score': 'risk_score', 'centrality_score': 'centrality_score', 
     'anomaly_score': 'anomaly_score', 'umap_x': 'umap_x', 'umap_y': 'umap_y'}, 
    node='id'
)

print('PyGraphistry graph created successfully!')
print('To visualize, run: graph.plot()')

# For local PyVis visualization
try:
    # Export to PyVis for local visualization
    graph = graph.to_pyvis()
    print('PyVis visualization ready!')
except Exception as e:
    print(f'Error in PyVis export: {e}')

Why this step? PyGraphistry provides interactive visualization capabilities that allow security analysts to explore relationships, identify patterns, and quickly spot anomalies in access data.

8. Visualize Graph Analytics

Now we'll create different views of our graph for various security investigation scenarios.

8.1 Create Full View

# Create full graph view
full_view = graph.copy()
full_view = full_view.set_node_attributes({'risk_score': 'risk_score'}, node='id')

print('Full view graph ready!')
print('To visualize full view, run: full_view.plot()')

8.2 Create Ego-Graph View

# Create ego-graph view for high-risk users
high_risk_users = final_nodes[final_nodes['risk_score'] > 0.8]['label'].tolist()

# Create ego-graph for one high-risk user
ego_graph = graph.copy()
# In practice, you'd filter edges based on high-risk users
print(f'High-risk users identified: {high_risk_users[:5]}')
print('Ego-graph view ready for high-risk users!')

8.3 Create High-Risk View

# Create high-risk view
high_risk_view = graph.copy()
high_risk_view = high_risk_view.set_node_attributes({'risk_score': 'risk_score'}, node='id')
high_risk_view = high_risk_view.set_node_attributes({'anomaly_score': 'anomaly_score'}, node='id')

print('High-risk view ready!')
print('To visualize high-risk view, run: high_risk_view.plot()')

Summary

In this tutorial, we've built a complete PyGraphistry workflow for security analytics and risk investigation. We started by creating a synthetic enterprise access dataset, converted it into nodes and edges, and enriched it with risk scores, centrality metrics, anomaly detection, and UMAP embeddings. Finally, we bound the graph to PyGraphistry and created different visualization views for full, ego, and high-risk perspectives. This workflow enables security analysts to quickly identify suspicious patterns, understand access relationships, and prioritize risk investigations in enterprise environments.

The key components of this workflow include:

• Data preparation and feature engineering
• Statistical and machine learning methods for anomaly detection
• Dimensionality reduction with UMAP for better visualization
• Interactive visualization with PyGraphistry
• Different graph views for various investigation scenarios