How Dedrone Just Delivered Its Fastest and Most Accurate Airspace Security Solution Yet

Written by: Rafael Turner (Director of Artificial Intelligence, Dedrone), Alex Kimiavi (Computer Vision Engineer, Dedrone), Ivan Lebedev (Simulation Team Lead, Dedrone)

Key Take-Aways

1. Today, Dedrone has delivered its fastest and most accurate lower airspace awareness solution by leveraging PyTorch, Weights & Biases, and Nvidia’s H100 to create a new model we call Pythagoras 1
2. Three fundamental building blocks must come together to create the world’s fastest and most accurate airspace security solution. These include hardware, neural network architecture, and data. In each case, Dedrone has brought together the most capable components available.
3. This core engine can be applied both to protect against unauthorized drones while also enabling productive drones to navigate safely through a more and more complex airspace
4. Pythagoras 1 has already delivered massive jumps in performance including speed and detection accuracy for the identification of birds, helicopters, quad-copter drones, fixed wing drones and airplanes.

Introduction

Today, we’re excited to share that the next generation of computer vision (CV) models from Dedrone are available for broad use by our customers. We call it Pythagoras 1, named after PyTorch, the fully featured framework for building deep learning models used at Dedrone, and, in part after the famous Greek mathematician. This release features improvements in precision on all object classes and recall on most object classes. The release also includes a significant improvement to the runtime of the model. This next generation model is now powering all of Dedrone’s products, including our latest sensor-fusion drone protection solutions as well as DedroneBeyond.  

Dedrone’s dedicated Machine Learning (ML) team leveraged three core triggers to deliver one of the industry's fastest and most accurate sensor-fusion based airspace security solutions in the world.  

Firstly, we started with the right GPU, the H100. This machine is well known to deliver today’s top of the line capabilities as a machine learning training machine. We now own several of these machines and they are running 24 hours a day to continuously improve our solution.

Next, we leverage PyTorch as the framework for DedroneTracker.AI’s computer vision models, we set out to build the best models for lower airspace security while still maintaining our performance for the counter-drone mission. We wanted to address customer feedback on further reducing false alarm rates and improving our ability to detect flying objects well beyond traditional drone quadcopters. PyTorch enabled us to utilize a completely new neural network architecture. Pythagoras 1 is a major leap forward and is truly state-of-the-art for object detectors in lower airspace awareness.  

Finally, we took the impressive library of seven years of already built-up data sets and augmented it even further through both simulated data sets and the integration of active learning of the most interesting cases and consequently drove those back into our model training process.

What Were the Performance Improvement Results?

Pythagoras 1 has delivered an average 20% speed increase of our video tracker in addition to a significant improvement in accuracy, driving down both false positives as well as false negatives. An illustration detailing scenarios for True Positives, False Positives, and False Negatives is shown below:

‍

These improvements can be measured by improved Mean Average Precision (mAP) and Mean Average Recall (mAR). Recall and Precision are the two key metrics used to assess detector performance. In practice, there is a trade-off between these metrics. Increasing the threshold for classifying an object in the airspace will result in fewer false positives, thus improving precision. However, this tends to allow for more false negatives in practice, so recall is now lower. This works both ways, too. The two equations below explain these metrics:

‍

Furthermore, we see a 14x improvement in our average precision for extremely small drones. These improvements were observed across all objects and several spatial scales. The detection metrics are detailed in the tables below:

‍

‍

How Were These Results Achieved?

The Need for Speed: Implemented Top of the Line Infrastructure

Dedrone is committed to improving our end-to-end AI solution, and to do this rapidly we have purchased a new computer cluster leveraging NVIDIA’s flagship ML Graphics Processing Unit (GPU), the H100. We worked with Lambda Labs to build out our new computer cluster. This new hardware decreased the time it takes to train our model across millions of frames of video since we adapted our training code to use multiple GPUs. In turn, this enabled our team to run experiments at a much higher rate. Said experiments involved neural network architecture modifications along with the guided tuning of hyperparameters. Additionally, our evaluation infrastructure is based on Weights & Biases, allowing us to easily visualize the results of our model and understand where we can improve. Furthermore, Weights & Biases comes with a hyperparameter tuning framework, Sweeps, which is applied to every architecture we train.

New Neural Network Architecture

Pythagoras 1 is a full upgrade from our previous model, leveraging a new neural network architecture, training framework, and deployment framework. The neural network architecture allows for features to be related more easily and gets rid of some assumptions other detectors make that can confuse the network. The training framework is much more robust than our previous framework, allowing us to quickly prototype and experiment with new ideas. The new inference engine we use greatly reduces the time it takes for our model to execute on a video, which, in turn, allows us to leverage more “neurons” in our neural network for a similar cost of frames per second when compared to our previous model. Additionally, the network can leverage more pixels of information without taking a large hit in runtime when compared to our previous deployment method.  This change has enabled us to quickly process and infer what is in 4k video, thus allowing us to be ready for new airspace awareness challenges like Drone as First Responder (DFR).

It’s All About the Data: Good Data Beats More Data

In machine learning, data is everything. Good data can significantly enhance the performance of a model, while poor data can lead to suboptimal results, regardless of the model's complexity. This concept is encapsulated in the adage:

"Good data beats more data."

Our journey began with a baseline dataset with the object distribution seen in the image below. The data was collected from a variety of sites, but contained mostly drones. These frames were annotated using Dedrone's internal tool, forming the foundation of our training dataset.  To improve the performance of what our model could achieve, we needed to diversify and improve the quality of our data. Understanding the lack of richness of our data set, led us to adopt various innovative approaches for Data Curation and Perfection (DCAP), which we will discuss in the following sections.

Data Curation And Perfection (DCAP)

One of our key strategies to enhance data quality has been the integration of active learning into our model training process. Active learning involves intelligently selecting the most informative data points for annotation, thereby improving model performance with fewer data samples.

‍

Our active learning loop involves identifying failing cases including false alerts, incorrect classifications, and missed detections including in instances with multiple objects in the scene. Failing cases were crucial for understanding where our system needed improvement.

This methodology of improvement continues today; by actively seeking out failing cases, we can curate a more comprehensive and effective training dataset. The identified failing cases are sent to an outsourcing annotation company for thorough annotation and quality assurance. Once this process is complete, the new data is integrated into our training set, ready for the next iteration of active learning. This iterative process ensures continuous improvement in our model's performance. We recently added an additional 2M annotated images through this methodology.

Artificial Data

Artificial data is crucial in ML and CV as it helps to fill gaps in real-world datasets, enhances diversity, and balances class distributions. By generating synthetic images, we can simulate various scenarios, augment existing data, and create challenging cases that improve model robustness and accuracy. This becomes even more important when real data is scarce or difficult to obtain.

Therefore, to further enhance Dedrone’s dataset, we employ data augmentation techniques using our AutoKat tool. AutoKat augments existing images by inpainting artificial objects into them, either with or without existing annotations. For this project, we acquired various models, including 21 helicopters, 7 planes, and 11 drones (including quadcopters, fixed-wing drones and even three Group-3 drones), which can be scaled, oriented, and placed in any position within an image.

This method allows us to create a diverse set of images, particularly useful for balancing our dataset by generating many helicopter and plane annotations. It also helps in addressing the object size distribution, which is crucial for advancing our models to work with 4K images. While AutoKat-generated images might not fully capture the noise, lighting, and sharpness of real-world images, they significantly contribute to enhancing our dataset. Below are some examples of images extended for the Pythagoras 1 project.

Another promising approach is the generation of fully artificial images using simulated environments and objects. Our simulation team is working on creating realistic images where everything is rendered together, providing comprehensive ground truth information. Simulated environments offer the flexibility to experiment with various lighting conditions, weather scenarios, and unique situations that are challenging to replicate in real life, such as kamikaze drones flying over cities. The main challenge lies in matching the simulated sensor and lens configurations with those used in real-life. Despite this difficulty, we believe that in the future, this simulated data approach will further enhance our model's performance. Here are some images showcasing the simulated world we've developed. This approach is still in development and will be used for the further stages of improvement of Pythagoras 1.

Data is what brings any ML model to life. With redundant, or poor data, the model may learn undesirable behavior or not generalize to unseen instances, well. Through our ablation process we identified key data augmentations that improved our model. The data our model leverages is also challenging to collect, such as planes and helicopters at a very far distance from a camera. To account for this in our dataset, the simulation team was able to generate synthetic data. Dedrone’s Data Curation and Perfection team (DCAP) has collected and  continues to deliver desirable and diverse data which Pythagoras can be trained on. These continuous improvements in our data  continue to fuel Pythagoras’ improvement  in performance.

Result

After we improved our neural network architecture, optimized the hyperparameters, ablated the data, and filled in the gaps with simulated data. The mAP of our model improved by 24.3%!

What to Expect Next from Dedrone

Our vision is to detect, track, and classify every flying object in the sky. With Pythagoras 1, we have taken a big step toward this goal.  Pythagoras 1 brings forth the world’s fastest and most accurate solution for lower airspace awareness and counter-UAS by leveraging the fastest processing technology, the latest in ML model research and development, and the backing of a strong data curation, perfection, and simulation team. We’re committed to the continued growth and development of Pythagoras 1 and will be releasing thermal versions soon.