Introduction
In this tutorial, you'll learn how to use the open-source Flora system developed by Nvidia for robot video labeling. Flora is a powerful data engine that can label robot videos at a fraction of the cost of competing systems like GPT-6 Astra. You'll set up a basic Flora environment, understand its core components, and run a simple labeling task to see how it works.
This tutorial is designed for beginners with no prior experience in robotics or AI data labeling. By the end, you'll have a working understanding of how Flora processes robot video data and how it can be integrated into your own projects.
Prerequisites
To follow along with this tutorial, you'll need:
- A computer with internet access
- Python 3.8 or higher installed
- Basic knowledge of command-line operations
- Some familiarity with AI concepts (no deep technical knowledge required)
Note: Flora is an open-source system, so you'll need to download and install it from Nvidia's repository.
Step-by-Step Instructions
1. Install Required Dependencies
First, we need to set up our Python environment. Open your terminal or command prompt and run the following commands:
pip install torch torchvision
pip install transformers
pip install opencv-python
pip install numpy
Why: These packages are essential for handling video data, running AI models, and processing images. PyTorch is the deep learning framework used by Flora, while OpenCV helps with video processing.
2. Clone the Flora Repository
Next, we'll download the Flora source code from Nvidia's GitHub repository:
git clone https://github.com/NVIDIA/flora.git
cd flora
Why: This gives us access to all the Flora components, including the pre-trained models and example scripts needed to run labeling tasks.
3. Install Flora in Development Mode
With the repository cloned, we need to install Flora so we can use it in our projects:
pip install -e .
Why: The -e flag installs the package in 'editable' mode, meaning any changes you make to the source code will be reflected immediately without reinstalling.
4. Prepare a Sample Robot Video
For testing purposes, create a simple folder structure and add a sample video file:
mkdir test_videos
# Place a sample robot video file (e.g., robot_demo.mp4) in this folder
Why: Flora needs video input to process and label. This step simulates the real-world scenario where you'd have robot footage to analyze.
5. Configure Flora Parameters
Now, create a basic configuration file for Flora. Create a file named config.yaml with the following content:
model:
name: "qwen3.5"
checkpoint: "qwen3.5-robot-labeler"
input:
video_path: "test_videos/robot_demo.mp4"
frame_rate: 30
output:
label_path: "output_labels.json"
format: "json"
processing:
batch_size: 8
device: "cuda"
Why: This configuration tells Flora how to process your video, what models to use, and where to save the results. It's the key to customizing Flora for your specific needs.
6. Run the Flora Labeling Process
With everything set up, we can now run Flora on your video:
python -m flora.label --config config.yaml
Why: This command executes the labeling process using the configuration we just created. Flora will process your video, detect robot actions, and generate labeled data.
7. View the Output Labels
Once the process completes, check the output file:
cat output_labels.json
Why: The JSON file contains the labels generated by Flora. You'll see how the system identified different robot actions in your video, such as 'grasping', 'moving', or 'releasing' objects.
8. Analyze Performance
Flora's performance is measured by both cost and accuracy. To compare, you can create a simple Python script to analyze the output:
import json
with open('output_labels.json', 'r') as f:
labels = json.load(f)
print(f"Total frames labeled: {len(labels)}")
print(f"Unique actions detected: {set([label['action'] for label in labels])}")
Why: This analysis helps you understand how well Flora performed on your video. It shows both the quantity of labeled frames and the variety of actions detected.
Summary
In this tutorial, you've learned how to set up and use Nvidia's open-source Flora system for robot video labeling. You installed the required dependencies, downloaded the Flora code, configured it for a sample video, and ran a labeling task. You now understand how Flora can reduce the cost of robot video labeling from $88 per hour to just 72 cents while maintaining high accuracy.
Flora represents a significant advancement in robotics data processing, offering a cost-effective solution for labeling robot actions. As you continue working with Flora, you can explore more advanced features like custom model training and integration with robotic control systems.