How Songtrail identifies birds, right on your phone

Two machine-learning models do this work, one for sight and one for sound. They're the kind of models that would normally live on a server, and getting them to run on a phone instead took some fun engineering. Here's how it works.

Two ways in

You can identify a bird two ways. With Spot, you take a photo and the image model works out the species. With Listen, you record a few seconds of audio and the sound model works out the species.

Both run right on your device. None of the identification happens in the cloud. The models and the logic around them are bundled into the app and run locally, the image side through Apple's Core ML (which can use the Neural Engine built into modern iPhones) and the sound side through TensorFlow Lite. That constraint shapes everything else. The models have to be small and fast, and there's no giant server model standing by to rescue a hard case.

A ranked list, not a verdict

This is the idea the whole app is built around. An app that announces one confident answer, and is quietly wrong a good share of the time, just teaches you the wrong bird. So Songtrail doesn't pretend to be surer than it is. Instead of "This is a House Finch," it shows a short list of candidates with confidence scores, most likely at the top.

That's honest about uncertainty, and it leaves you as the final judge, which for a birder is half the fun anyway. Everything the models produce feeds that ranked list rather than a single take-it-or-leave-it label.

Two open models, running on your phone

The biggest decision behind Songtrail's accuracy is that we don't train the identification models ourselves. We run two of the best openly published bird models there are, directly on your phone.

For sound, that's Perch, Google's open-source bioacoustics model, trained on a huge library of bird recordings. For sight, it's BioCLIP, a vision model from the Imageomics Institute trained on millions of images across the whole tree of life. Both come under permissive open-source licenses (Perch under Apache 2.0, BioCLIP under MIT), and full credit goes to the teams who built them. Leaning on their work means Songtrail inherits the accuracy of models far larger than anything we could build on our own.

The catch is size. These models are big and phones are small, so most of the engineering below is about closing that gap without ever phoning home.

Spot: a photo into a bird

The image side works in two steps, and the first one is easy to overlook.

The first step is finding the bird. Most of a bird photo isn't bird. There's sky, branches, a feeder, a blurry background. Before trying to name anything, a fast object detector scans the frame, picks out the most bird-like thing in it, and crops to it. Now the model gets a clean, centered subject instead of a speck lost in a landscape, which makes a real difference for distant or off-center birds.

The second step is identifying the crop. The cropped photo goes into BioCLIP, running on the phone through Core ML. Rather than spit out "species number 47," BioCLIP turns the photo into a kind of visual signature, a list of numbers that captures what the bird looks like, and Songtrail matches that against a written description of every species it knows. Whichever description fits best rises to the top, and how well each one fits becomes the starting point for the ranked list. Because BioCLIP was trained across the entire tree of life, it already has a deep sense of what tells one bird from another, with no fine-tuning from us.

Listen: a sound into a bird

The audio side has a neat trick at its core. It turns sound into a picture and reads it with a vision model.

It starts by drawing the sound. A raw waveform is hard for a model to read, so it gets turned into a spectrogram, a picture where time runs left to right, pitch runs bottom to top, and brightness is loudness. Drawn this way, a bird's song becomes a recognizable shape: the rising sweep of one call, the buzzy smear of another, the neat row of dots of a third. (Perch uses a mel spectrogram, which spaces out the pitch axis the way animal hearing actually works, giving more room to the low end where most of the action is.)

Next it finds the call. A real recording is mostly silence, wind, and traffic, with the bird chiming in now and then. Songtrail slides a series of windows across the clip, scores each one, and keeps the moments where something is actually singing. The dead air gets thrown out, and the model only looks at the good parts.

Then it identifies the sound. Each of those windows goes into Perch, running on the phone. Perch can recognize thousands of species worldwide, so Songtrail narrows its output to the birds it covers and pools the windows into one answer for the clip.

Getting Perch onto a phone

Identifying a bird from a clean, close-up recording is a solved problem. Identifying one from a phone held up to a windy field, with two other birds singing and a lawnmower going two houses over, is much harder, and it's the only kind of recording that really happens. Perch is good at exactly this, because it learned from a vast and messy real-world collection of bird sound. That's why we wanted it.

The tricky part was running it on a phone without changing its answers. Perch expects the audio prepared in a very particular way before it ever sees it, a precise recipe for turning the waveform into that spectrogram. We rebuilt that recipe in the app's own native code and checked it against the original model down to the last decimal, then handed the prepared sound to Perch itself. What runs in your pocket is the real Perch, offline, giving the same answers it would give on a research workstation.

Context makes it smarter

A model on its own looks at a bird in isolation. A good birder never does. A flash of red in a Vermont backyard in January is almost certainly a cardinal, not some tropical rarity it happens to resemble. Songtrail builds in that same instinct.

Once the model has its scores, Songtrail re-ranks them using what it knows about the moment:

• Where you are. Every species has a range, drawn from millions of public bird sightings pooled by GBIF, the Global Biodiversity Information Facility. One that's common where you're standing gets nudged up; one never seen within a thousand miles gets nudged down.
• The time of year. Plenty of birds are seasonal. A species that shouldn't be around this month gets pushed down.
• The time of day. An owl at 2 in the afternoon is a lot less likely than an owl at 2 in the morning, so night birds and day birds are weighted by the clock.
• The habitat. A duck deep in a forest, or a woodpecker out over open water, is worth a second look. Songtrail reads the land around you from OpenStreetMap (park, forest, water, city) and quietly favors the birds that belong there.

None of these overrule the model. They're gentle nudges, and together with the model's own confidence they tilt the list toward what's actually plausible where you are right now. A lot of the app's real-world accuracy lives here, in pairing a strong model with good local knowledge. Deny location access and the model just falls back on its own read. The context is a bonus, never a requirement.

Knowing when there's no bird

The models are trained to name birds, so left alone they'll try to name one even when you've photographed a fence post or recorded a passing bus. That's a fast way to look confidently foolish, so Songtrail runs a separate check for whether there's a bird there at all before it trusts any ID.

For photos, the same detector that crops the bird doubles as the gate: nothing bird-shaped in the frame, no ID. For sound, Songtrail leans on Apple's built-in sound classifier, a general "what am I hearing" model that ships with iOS, to tell bird song from speech, music, and silence, and it checks whether the recording is genuinely tonal rather than flat noise. When nothing convincing is there, the app says so. Being willing to say "I don't see a bird" is part of being honest.

Why keep it all on the phone

Doing everything locally costs us some effort. The models have to be compressed and tuned to run fast on phone hardware. But you get a lot back. The ID still works deep in a forest with no bars. It's instant, with no upload and no waiting in a server queue. And your photos, recordings, and locations never leave your phone.

For an app you use standing in a marsh at sunrise, that last part isn't a nice-to-have. It's the whole idea.

In short

Every identification follows the same path. Songtrail isolates the subject, cropping to the bird or finding the call. An open model does the heavy lifting, BioCLIP for photos and Perch for sound, running entirely on your phone. It weighs the result against context, where you are and the season, time, and habitat. And before any of that, it checks there's really a bird there. What you get back is an honest ranked list, and the last call is yours.