AutoNav Demo

AI Resnet Autonomous RC Car lane-following vision models

End-to-End Pipeline for NVIDIA Jetson Nano and Rockchip NPU Devices. This project covers data collection, model training, inference, and fleet workflows for autonomous RC car lane following.

GitHub Repo Back to Projects

Demo Video

README

AutoNav is an end-to-end pipeline for data collection, model training, and model inference for a deep learning self-driving lane-following model.

It is designed for a single board computer with CUDA cores or a Rockchip NPU, a camera, and an RC car.

What you can do

  • Collect data by driving the car around a track made of tape or another material.
  • Use a data management frontend to clean your data and generate augmented images.
  • Train a model to predict steering and/or throttle values.
  • Run the model and watch the car autonomously drive around your track.
  • Use the fleet management frontend to remotely start or stop the vehicle, load new models, view the camera feed, and inspect logs.

Top-level layout

tests

Hardware checks and test scripts.

setup

Jetson setup and build scripts, including bundled RealSense artifacts.

inference

Model optimization and on-car autonomous runtime scripts.

data_collection

Data recording, dataset management frontend, and augmentation utilities.

model_training

Model training code for both legacy RGB-only and new sensor-combination workflows.

fleet

Fleet-facing client and host application code.

Quick start path

  1. Start with setup docs in setup/README.md.
  2. Verify controls and hardware using scripts in tests/README.md.
  3. Collect data and interact with the data management frontend using data_collection/README.md.
  4. Train models from model_training/README.md.
  5. Optimize and deploy with inference/README.md.
  6. Run fleet workflows and manage cars from fleet/fleet_management_app/README.md.

Hardware

Jetson Nano prototype

  • Jetson Nano 4GB Developer Kit with ARM Cortex-A57 CPU and Fan-4020-PWM-5V, Ubuntu + JetPack
  • LaTrax Rally 1/18 RC car
  • Intel RealSense D435i for sidecar depth and IMU
  • Front CAM0 fisheye camera for primary forward RGB
  • Rear CAM1 camera for preview and reverse-only support
  • TP-Link TL-WN725N USB WiFi adapter
  • PCA9685 16-channel servo driver
  • Pololu 4-Channel RC Servo Multiplexer

Rockchip 5B prototype

  • Radxa Rock 5B with Rockchip RK3588 SoC and Radxa Heatsink 4012, Rock 5B Armbian
  • Raspberry Pi Pico
  • Generic $10 Toy RC Car
  • L298N motor driver module
  • TP-Link TL-WN725N USB WiFi adapter
  • Generic $5 USB Webcam

CAM0 is the primary forward RGB source for lane following. RealSense remains active for depth-stop and IMU context, and CAM1 is reserved for rear-preview scaffolding.

Software

  • Ubuntu 18.04.6 LTS
  • Jetpack 4.6.1 SDK
  • Python 3.6.9

Core features

  • Data collection from teleoperated driving
  • ResNet-based steering model with PyTorch and TensorRT
  • Autonomous lane following on an indoor track
  • CAM0 fisheye preview and lane-follow training pipeline
  • RealSense depth ROI measurement for safety and debugging
  • REST API for live predictions
  • Host dashboard for fleet and operator monitoring
  • Dockerfile support for deployment workflows

Model architecture

Supports several ResNet variants: Resnet18, Resnet34, Resnet50, Resnet101, and Resnet152.

The project defines multiple model variants through an EXPERIMENTS list so different sensor combinations can be trained and evaluated without changing the core training code.

EXPERIMENTS = [
    {"id": 1, "desc": "Front+Back + all sensors", "csv": AUGMENTED_CSV, "features": ['rgb_path', 'cam1_path', 'ir_path', 'depth_path']},
    {"id": 2, "desc": "Front only + all sensors", "csv": AUGMENTED_CSV, "features": ['rgb_path', 'ir_path', 'depth_path']},
    {"id": 3, "desc": "Front only RGB only",     "csv": AUGMENTED_CSV, "features": ['rgb_path']},
    {"id": 4, "desc": "Front+Back RGB only",      "csv": AUGMENTED_CSV, "features": ['rgb_path', 'cam1_path']},
    {"id": 5, "desc": "Front+Back + all sensors (Cleaned)",   "csv": CLEANED_CSV,   "features": ['rgb_path', 'cam1_path', 'ir_path', 'depth_path']},
    {"id": 6, "desc": "Front+Back RGB only (Cleaned)",        "csv": CLEANED_CSV,   "features": ['rgb_path', 'cam1_path']}
]

Experiment 5 and 6 mirror 1 and 2 respectively, but use non-augmented images only and therefore perform worse.

Troubleshooting

If the car is not moving when the model is running, the README includes an I2C warm-up sequence that directly writes raw register values to wake up the PCA9685, set it to 50 Hz, sweep the steering servo, and return the throttle channel to neutral.