UAV3D: A Large-scale 3D Perception Benchmark for Unmanned Aerial Vehicles

Hui Ye1, Raj Sunderraman1, Shihao Ji2
1Georgia State University, 2University of Connecticut
arXiv UAV3D Code UAV3D

UAV3D Demo

Introduction

UAV3D is a public large-scale benchmark designed for 3D perception tasks from Unmanned Aerial Vehicle (UAV) platforms. This benchmark comprises the synthetic data and 3D perception algorithms, aiming to facilitate research in both single UAV and collaborative UAVs 3D perception tasks.

The UAV3D dataset comprises 1,000 scenes (700 scenes for training, 150 scenes for validation, and 150 scenes for test) with 500k RGB images and 3.3 million 3D boxes. The dataset is organized in the format of nuScenes dataset, with the compatibility to the well-established nuScenes-devkit.

UAV3D dataset

Scene Planning

  • Locations: urban areas (Towns 3 and 10) and suburban areas (Towns 6 and 7) in Carla.
  • Flight routes: 250 routes from the bottom left to the top right of each map.
  • Scenes: 700 training, 150 for validation, and 150 for testing.
UAV3D dataset

Sensor Setup

  • Positions of RGB cameras: front, left, right, center, and back.
  • Rotation angle: bottom camera provides a bird’s eye view, while the other four are a pitch angle of -45 degrees.
  • Resolution: 800x450 pixels.
Sensor Setup

UAV Formation

  • Cross-shaped formation: front, left, right, center, and back.
  • Distance: each with 20 meters from the center drone.
  • Altitude: the UAV swarm maintains an altitude of 60 meters.
Sensor Setup

Data Format

  • Database schema: nuScenes schema.
  • Annotations: 3D bounding boxes, pixel-wise semantic labels.

Experiments

We benchmark four standard perception tasks for UAVs: single-UAV 3D object detection, single-UAV object tracking, collaborative-UAV 3D object detection, and collaborative-UAV object tracking.

Table 1: 3D object detection results on the validation set of UAV3D.
detection
Table 2: 3D object tracking results on the validation set of UAV3D.
tracking
Table 3: Collaborative 3D object detection results on the validation set of UAV3D.
detection
Table 4: Collaborative 3D object tracking results on the validation set of UAV3D.
tracking

Code

You can find our code from Github.

Dataset

The dataset is available for download from Google Drive or Baidu Netdisk.

Citation

If you find the UAV3D dataset and/or code useful, please consider citing this paper.

@inproceedings{uav3d2024,
    title={UAV3D: A Large-scale 3D Perception Benchmark for Unmanned Aerial Vehicles},
    author={Hui Ye and Raj Sunderraman and Shihao Ji},
    booktitle={The 38th Conference on Neural Information Processing Systems (NeurIPS)},
    year={2024}
}

Acknowledgement

The software and data were created by Georgia State University Research Foundation under Army Research Laboratory (ARL) Award Numbers W911NF-22-2-0025 and W911NF-23-2-0224. ARL, as the Federal awarding agency, reserves a royalty-free, nonexclusive and irrevocable right to reproduce, publish, or otherwise use this software for Federal purposes, and to authorize others to do so in accordance with 2 CFR 200.315(b).