Advancing the Photo Mission: Part 1

Applying Machine Learning to Alarmy's Photo Mission

Alarmy offers various missions to help users wake up completely and start a successful morning. Among them, Alarmy's photo mission requires you to wake up and take a picture of the same location or object you registered when setting the alarm. If the two photos match, the alarm turns off. For example, you can take a picture of your bathroom sink in advance and save it as your photo mission when setting your alarm. Later, when the saved alarm rings, you take a picture of the sink from the exact same spot to dismiss it. If you have a specific object or place that guarantees you'll wake up, you can take a picture of it to use the photo mission more effectively. Just like the example, preparing a cold cup of coffee and a warm brunch before taking a picture could make the mission process really helpful for a sure wake-up.

The critical issue in implementing this feature is figuring out how to measure the match, or similarity, between two photos. The simplest approach could be comparing the bitmap forms of two images pixel by pixel to determine similarity via Euclidean distance, or comparing HSV values. However, these methods have a flaw: two completely different photos might score a high similarity just because their overall colors are similar.

The problem with the methods just mentioned is that the feature values extracted to measure similarity are limited to color—meaning whatever value represents the image's characteristics is confined solely to its hues. To overcome this, we could try extracting and comparing a more diverse set of image features. Expanding on this a bit, we could build an image classifier that extracts features from a pre-prepared set of images and learns from them. For instance, if we extract features from a sink image that allow it to be classified as a sink and train our classifier on them, feeding a sink image into the classifier later will correctly identify it as a sink. In other words, given two images, we can classify what each one is through an image classification process, and then implement the photo mission by checking whether the two images are the same.

Image classification has long been studied in ML (machine learning), and thanks to its rich history, there are plenty of algorithms available—from logistic regression, SVM, and decision trees, to more recent approaches utilizing deep learning. Since the algorithms for image classification already exist in abundance, it seems all we need to do is prepare the dataset to train the algorithm.

A dataset is constructed by preparing multiple photos of specific objects and labeling what each photo represents. For example, you could gather 1,000 diverse images of sinks, input them into the algorithm, and tell it in advance that these 1,000 photos are sinks. This forces the algorithm to learn the features extracted from those 1,000 sink images.

Now, the algorithm knows the characteristics of a sink image and has acquired the intelligence to classify whether an image is a sink or not when presented with new photos. If we repeat this process for images of other objects, our trained algorithm becomes an excellent AI model equipped with the intelligence to classify any objects provided in the training data.

Therefore, we compile a training dataset consisting of labeled photos by shooting objects users are likely to snap when they wake up—such as sinks, toilets, dressing tables, beds, and living room windows—from as many different angles and distances as possible. Labeling means we pre-match a sink image with the correct answer, "sink." When we feed this training dataset into our algorithm, it turns into an image classification model capable of sorting out what object a specific image shows. Now, using this trained model, we can execute the photo mission by inputting both the designated image from the alarm setup and the newly taken image from the morning into the classifier, and then comparing their labels. If the labels of the two images match, they'll be judged as the same image and the alarm will turn off normally. If not, the mission fails and the alarm will keep ringing.

However, this kind of multi-class image classification faces a limitation: it can only classify images it was trained on. Our trained algorithm only knows about the datasets we presented during training and the labels we provided (the ground truth, e.g., the label "sink" for sink images). When given other data, it's completely clueless. An algorithm trained strictly on sink images will only know that a non-sink image is "not a sink," but it won't be able to tell you what object it actually is. I'll cover the methods to solve this problem in the next post.

We're looking for people to share valuable experiences with us at DelightRoom