Advancing the Photo Mission: Part 2

Applying machine learning to Alarmy's Photo Mission

In the previous post, ‘Advancing the Photo Mission: Part 1’, we discussed various methods for comparing the similarity between two photos—the reference image saved when setting the alarm, and the image taken to dismiss it—along with the limitations of each approach.

By using the "Multi Image Classification" method in ML, we prepped a dataset of photos users would likely take for their missions (e.g., bathroom sinks, beds, dining tables) and trained a model to match input images with their correct labels. Using this trained model, we could feed both the saved reference image and the newly captured image into the system. If they were classified as the same object, the mission was a success; if classified differently, it was a failure. This is how we initially got the Photo Mission working.

However, this multi-class classification approach has a built-in flaw: it can only classify images it has actually seen before. The algorithm we trained only knows about the dataset we provided and the labels we assigned (e.g., labeling a sink image as "sink"). Feed it anything else, and it’s completely clueless. An algorithm trained only on sink images will just know that a non-sink image "isn't a sink"—it won't have any idea what the object actually is.

To overcome this limitation, this post will introduce how to implement the Photo Mission using a "One-class Image Classification" approach.

Unlike Multi-class Classification, One-class Classification is an algorithm designed to determine whether a given image matches a single specific class. Multi-class classification works by training on all the different types (classes) of images you want to categorize, so when a new image comes in, the model can successfully classify it if it already knows the answer (like recognizing a "sink" because it learned what a sink looks like). In contrast, One-class Classification only trains on the single type of image you want to recognize. When the trained model is given a new image, the problem is compressed into a simple binary classification: is this the target image or not (A / ~A)?

So, if we apply this One-class Classification method to our Photo Mission, we can train the model specifically on the image the user saves when setting the alarm. Then, when the user takes a picture to dismiss the alarm upon waking up, we feed it into the trained model and simply run a binary check: is it classified as the same class or not? Instead of trying to train the model on every conceivable object a user might photograph in the morning (like we did with multi-class), we only train it on the one specific object they actually photographed. After that, we just have to judge whether any future image belongs to that same exact category.

There is a catch, though. To achieve high classification accuracy, a model needs a sufficient amount of training data. But in this scenario, the training data is generated right when the user snaps a photo to set up their mission. This means we aren't starting out with a massive pre-built dataset like in the multi-class approach; our entire training set consists of exactly one single photograph.

To solve this problem, we can use data augmentation. When given a single image data point, we can programmatically manipulate it—by applying rotation, brightness and saturation adjustments, flipping, shifting, and so on. This creates the effect of having photographed the same object across many different times and environments.

Now, thanks to data augmentation, we can generate a robust dataset from just one initial photo—more than enough to train our one-class image classification algorithm. With the data scarcity issue out of the way, we can train the model using this artificially expanded dataset. Once fully trained, the model will evaluate new images: if an image falls within the learned boundaries, it's classified as a match; if it falls outside, it's something else. Applied to our Photo Mission logic, the former equates to mission success, and the latter to mission failure.

We've now explored various ways to execute the Photo Mission—from a pixel-by-pixel direct comparison, to a multi-class approach that classifies two images independently and compares the results, all the way to a one-class approach that simply sorts two images into A or ~A. The most straightforward method, direct image comparison, suffered from relying on simplistic feature comparisons, which meant entirely different objects with similar colors could be mistakenly classified as the same thing. Meanwhile, the multi-class approach was limited to classifying only the specific images it was trained on. By adopting the one-class image classification method, we successfully condensed the challenge into a binary problem, verifying whether a given image matched the training data or not. Given the image data in the top left, we achieved the following results.

Across "Advancing the Photo Mission: Part 1" and "Part 2," we’ve unpacked the flaws in our existing methods and explored potential improvements. Unfortunately, since Firebase ML Kit is currently in beta, it doesn't support on-device model training yet, meaning we haven't been able to apply this to the live Alarmy app. They do plan to roll out mobile training capabilities in the future, so I'll be eagerly waiting for that day as I wrap up this journey.

We're looking for people to share valuable experiences with us at DelightRoom