AI landmine detection reveals bigger problems in computer vision testing 

Researchers at Binghamton University are using computer vision and machine-learning to detect landmines from aerial images. Their results reveal the importance of out-of-sample (OOS) testing, which evaluates a model on independently collected images beyond the dataset used to train and tune it. 

“It’s easy to get numbers in the 80 or 90 percent range if you’re not doing out-of-sample testing,” said Thomas Pingel, associate professor of Geography at Binghamton University and corresponding author of the paper, in an interview with R&D World.   

“By using out-of-sample tests, meaning an entirely separate data collection, you get a better reflection of what this is like in the field,” he said.  

Mapping mines from individual drone images 

The researchers trained You Only Look Once (YOLO)’s object detection algorithm to detect PFM-1 landmines from images. The researchers trained the model on PFM-1 imagery, then used an inert PFM-1 mine and 3D printed, paint-matched replicas to test how well it performed on mines it hadn’t seen during training.  

The model was slightly better at identifying replicas than real inert mines, the paper notes. The replicas showed precision, recall and mAP (mean average precision) between 75% and 80% while inert mines scored between 60% and 70%. The authors suggest this could be due to imperfect paint matching or the Cyrillic Y cut into real inert mines that designates them as training devices acting as unintended visual features the model learned to identify.  

PFM-1s are widely used scatterable landmines, which are designed to be deployed over wide areas. PFM-1s are engineered to fall like maple seeds from the sky and can generally escape detection.  

In post-conflict regions, these landmines may become buried over time. Since they are relatively small, drones must fly low to the ground for their sensors to detect the mines.  

Rather than stitching drone images into a single composite map before running detection, a common approach that requires heavy computer processing, the researchers ran YOLO directly on the raw drone photos, then pinpointed mine locations using the drone’s position and camera angle at the moment each image was captured. 

Why out of sample testing matters 

The researchers used two models: one trained on PFM-1 images only, and one trained on PFM-1 images plus the COCO dataset, a collection of labeled images of everyday items.  

For this study, precision refers to whether there was a landmine where the model flagged one and recall refers to how many of the landmines the model flagged overall.  

During testing, the PFM-1 model showed precision of 57% and recall of 93%. However, the recall dropped to 24% during OOS testing. The precision during OOS testing increased to 74%.  

The model also trained on the COCO dataset showed a precision of 94% and recall of 85% during testing, but the precision dropped to 80% and the recall to just 14% for OOS testing.  

“I think it’s a wider problem in most computer vision. I think this is changing, people are getting more sensitive to the fact that out-of-sample testing needs to be part of every computer vision project. But we’ve certainly found that in our work, without it, the numbers given suggest it’s going to operate at an 80 or 90 percent level, and then you run it and it doesn’t. That happens over and over again,” Pingel said.  

A widely cited 2018 paper by Caltech researchers tested animal-detection algorithms across twenty camera trap locations and found that models performed extremely well when tested at the same locations where they were trained, but generalized poorly to new, unseen locations. 

A systematic review published in the Korean Journal of Radiology in 2019 found that only 6% of AI publications in medical imaging included external validation at all. A separate review found that a study of 86 deep-learning radiology algorithms reported 81% saw declining accuracy on external data.