# NVIDIA AI Infrastructure for   
Autonomous Vehicles  

The [NVIDIA AI Enterprise Software Suite](https://www.nvidia.com/en-us/data-center/products/ai-enterprise/) delivers the essential tools you need for streamlining the development and deployment of AV software. This includes everything from data preparation and training to optimizing for inference and deploying at scale.

[![NVIDIA AI Enterprise Software Suite streamlines AV software development and deployment](https://developer.download.nvidia.com/images/drive/auto-ecosystem-diagram(1).png)](https://developer.download.nvidia.com/images/drive/auto-ecosystem-diagram(1).png)

Click to enlarge

  
Note: Before getting started, make sure you have the [latest NVIDIA products](https://www.nvidia.com/en-us/data-center/products/) (e.g. DGX) and the corresponding[drivers](https://www.nvidia.com/Download/index.aspx?lang=en-us) for your NVIDIA GPU(s). 

* * *

## Step 1: Data Preprocessing and Extract, Transform, and Load (ETL)

### Transforming Raw Sensor Data  

Use [Apache Spark](https://www.nvidia.com/en-us/deep-learning-ai/solutions/data-science/apache-spark-3/), plus the NVIDIA [RAPIDS](https://rapids.ai/)™ suite of open source libraries and APIs, to help accelerate cleaning, filtering, and transforming your raw sensor data into a more structured format.   
  
Analyze your workloads using the [Accelerated Spark Analysis Tool](https://www.nvidia.com/en-us/lp/deep-learning-ai/solutions/data-science/spark-accelerator/).

### Video Processing

Decode and convert compressed data to frames using [NVIDIA Video Processing Framework](https://github.com/NVIDIA/VideoProcessingFramework) (VPF).  
  
Adjust image size and aspect ratio, standardize pixel values, and enhance image diversity and clarity by using [NVIDIA® CV-CUDA](https://developer.nvidia.com/cv-cuda)™ — an open-source, GPU-accelerated library for large-scale image-processing tasks.

### Auto-Labeling

Use [NVIDIA Triton™ Inferencing Server](https://developer.nvidia.com/triton-inference-server) for your auto-labeling DNN models, preferably in [NVIDIA TensorRT](https://developer.nvidia.com/tensorrt)™ format to get the highest throughput. You may also execute pre-processing (e.g., decoding, re-shaping, filtering, etc.) and post-processing (e.g., encoding bounding boxes, reshaping, etc.) operations on the same GPU for the end-to-end maximum auto-labeling speed and high GPU utilization. You can use the tools introduced in the Video Processing section on this page.

### Scene Mining  

It’s critical to minimize the time spent on extracting relevant scenes to build training or test datasets. In addition to keyword search within the meta data, DNN models can help with synthetic and semantic mining of scenes. Use [Triton Inferencing Server](/triton-inference-server) for your auto-labeling DNN models, preferably in [NVIDIA TensorRT](/tensorrt) format to get the highest throughput. Similar to auto-labeling, you can use the tools introduced in the Video Processing section on this page.

* * *

## Step 2: Training and Optimizations

NVIDIA offers [Docker containers](https://docs.docker.com/get-docker/) for your preferred [deep learning framework](https://developer.nvidia.com/deep-learning-frameworks) through the [NGC catalog](https://catalog.ngc.nvidia.com/?filters=&orderBy=weightPopularASC&query=).

### GPU Acceleration

Enable GPU acceleration using [NVIDIA CUDA](https://developer.nvidia.com/cuda-toolkit)® Toolkit and NVIDIA [CUDA Deep Neural Network library](https://developer.nvidia.com/cudnn) (cuDNN). The toolkit includes GPU-accelerated libraries, debugging and optimization tools, a C/C++ compiler, and a runtime library. While cuDNN offers GPU-accelerated library of primitives for deep neural networks.

### Multi-GPU Acceleration   

Achieve multi-GPU acceleration using NVIDIA [Magnum IO](https://github.com/NVIDIA/MagnumIO)™ and [NCCL](https://developer.nvidia.com/nccl). A variety of software and techniques support [distributed training](https://docs.nvidia.com/ai-enterprise/deployment-guide-multi-node/0.1.0/workflows.html), allowing multiple GPUs to train larger deep learning models. These libraries are compatible with all NVIDIA networking solutions, including NVLink™ and NVSwitch™.

### Mixed-Precision Training  

[Mixed-precision training](https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html) is a technique that uses lower-precision floating-point numbers (e.g., half-precision) to accelerate training by using [Tensor Cores](https://www.nvidia.com/en-us/data-center/tensor-cores/). Learn more about using [automatic mixed precision for major deep learning frameworks](https://developer.nvidia.com/automatic-mixed-precision).

### NVIDIA DALI®

[NVIDIA Data Loading Library (DALI®)](https://developer.nvidia.com/dali) is a portable, open-source software library that can be used for decoding and augmenting images, videos, and speech. This helps accelerate DLI applications by reducing data access latency and training time.

* * *

## Step 3: Simulation and Testing

### Simulation

[Simulation](https://www.nvidia.com/en-us/use-cases/autonomous-vehicle-simulation/) of real-world scenarios isn't only critical for training for unforeseen scenarios, but also vital for testing and validation of autonomous vehicles. Deploying autonomy in a safe and trustworthy manner requires achieving real-world fidelity, which is incredibly challenging without simulation of rare real-world scenarios. It requires accurate modeling of the physics and behavior of an autonomous systems sensors and surroundings.  
  
[NVIDIA Omniverse Cloud Sensor RTX](https://nvidianews.nvidia.com/news/omniverse-cloud-apis-industrial-digital-twin) for AV simulation deliver large-scale, high-fidelity sensor simulation to accelerate ‌AV development, giving you a seamless way to effectively test countless “what if'' scenarios and diverse environmental conditions.

### Replay

Playing back recorded logs and recorded sensor data can be used to analyze and diagnose issues that may have occurred during testing or in the field. Sensor data, system logs, and other data can be replayed to recreate the conditions under which a problem occurred and help identify the root cause. Replay can also be used to evaluate the performance of different algorithms and systems under different conditions.