Chandra Krintz and Rich Wolski Help Ecologists with their Project: "Where's the Bear?"
Facing a daunting, digital pile millions of photographs high, how would you find the few images you need? Could you, even? It’s a classic needle-in-a-haystack conundrum, and it’s a blaring reality for ecologists, resource managers and farmers who use camera traps to track wildlife.
Consider Sedgwick Ranch Reserve, part of UC Santa Barbara’s Natural Reserve System. A sprawling and pristine 6,000 acres and nine square miles, the protected land used for research and teaching is a veritable nirvana for animals of all kinds. Mountain lions and black bears and deer, oh my. And they are all represented many times over in the reserve’s massive image archive — millions of pictures of thousands of animals, captured by multiple camera traps and dating back more than a decade. But who has time to sort them all?
Cue the computer scientists. More specifically, UCSB computer science professors Chandra Krintz and Rich Wolski. Their Sedgwick-inspired new endeavor — aptly named “Where’s the Bear?” — is training technology to wrangle and analyze those images, leaving researchers more time for, well, research. The pair conceived and co-lead the effort that employs machine learning to identify and classify animals caught on camera.
By assigning to computers a vexing task that until now was the sole purview of people, the project saves major manpower; the Sedgwick team has spent days, weeks, even months, sorting images. But the innovation also holds potential far beyond Sedgwick to other reserves, and beyond ecology to farming, to agriculture, even to medical imaging.
“Not only does ‘Where’s the Bear’ work, but it works really well,” said Krintz, vice chair of UCSB’s undergraduate program in computer science. “We don’t get any coyote wrong. We don’t get any bears wrong. We get about 12 percent error on deer — there are lots of deer — and we are trying to improve on that. Now all the ecologists are saying, ‘Count deer, count bear. Tell me if the bear is healthy. Is it the same bear, is it the same deer? How many deer are there with antlers?’
“The low-hanging fruit was easy and we can save people thousands and thousands of hours, literally,” Krintz continued. “So now we can start asking those harder questions by working with ecologists and scientists on how to design that system.”
The multitier, cloud, edge and sensing system that is “Where’s the Bear” integrates recent advances in machine learning-based image processing to automatically classify animals in images captured by remote, motion-triggered camera traps. To build a “training” set, Krintz and Wolski overlaid thousands of stock photos of wildlife onto background images from Sedgwick watering holes that are monitored by cameras.
Using the new, synthetic images to teach their novel device how to accomplish the task at hand — i.e., to automatically and accurately identify and classify those animals — the system has so far helped the Sedgwick team aggregate and analyze more than 1 million images dating back years. And the hardware lives at Sedgwick, meaning all that data processing is done within yards of where the data is collected.
“Everybody uses images to do analysis now,” Krintz said. “It doesn’t have to be animals. It could be people or how many cars pass by a particular street corner. All of that analysis can now be done where the data is being collected. That’s our real advance, because that wasn’t possible before. Hopefully by working with Sedgwick we can eventually go to all the reserves in the UC system and help them, their researchers and their students answer interesting questions and see how far we can push this.”
“This is hugely powerful technology and we want to bring it to bear on important problems,” she added. “It saves tons of time — what once took 14 days we can do in 3 hours — and it saves money, communications, energy, carbon footprint. The main idea is to help scientists who gather the data, in the way of pictures, and use those pictures to guide their science.”
According to Wolski, the project can inform research in endless ways, from identifying which species are present — thereby indicating, too, which are absent — to illuminating the effects of drought by revealing how animals respond when there is more, or less, water available. Further, the technology can enhance understanding of the health of, say, the bear population, or enable more accurate deer counts to better inform the number of hunting licenses issued each year. Ranchers can use it to monitor livestock; farmers can use it to monitor their crops.
Eventually, the system could even include a notification function, using real-time image analysis to alert property managers or authorities to the presence of poachers, or alert visitors — hikers, researchers, students — that there may be wildlife nearby.
“‘Where’s the Bear’ is innovative both in the remoteness of where the images are being captured, and also in the software architecture that we have developed to manage the process of gathering the images, classifying them accurately, then distributing them to interested scientists,” Wolski explained. “There is nobody bridging the gap between what the reserves are doing and what scientists need to do to consume what is a valuable scientific resource. We’re hoping we’re providing the technology conduit that allows that to happen, so researchers all over the world will eventually have access to those images.”
For Kate McCurdy, longtime resident director at Sedgwick Reserve, it’s coming none too soon. Amassing photos for more than 10 years, she said she was all in from the moment “Where’s the Bear” was first proposed. And she quickly involved her volunteer research assistant, Grant Canova Parker, who manages the cameras in the field and has logged thousands of hours on the project.
“We’ve done data handoffs before to researchers, but it’s pretty unwieldy to give someone five hard drives of digital images and say, ‘Good luck finding the thing that you’re looking for,’” McCurdy said. “We need to do better, to be able to hand off a spreadsheet that’s been processed and say, ‘Here’s the data. Here are all the pictures of the deer we’ve taken over the last five years — you can crunch your own numbers and find the trends.’ To be able to develop non-invasive tools that are cheap and easy to use for land managers, property owners, ecologists and students is huge. Being on the forefront of this new technology is really exciting.”
Wolski and Krintz intend to make the project software open-source, available and accessible to all who are interested — starting with some of Sedgwick’s counterparts.
“We hope to work with UCSB to bring this to the other reserves,” Wolski said. “Ecology and agriculture are related areas where this technology could have high impact, and we would love to see the pervasiveness of this work throughout the UC system.”