Shazam for food! HealthifyMe’s Snap AI tool makes tracking calories fun and easy

Remember SeeFood? The fictitious "Shazam for food" application featured in the HBO sitcom Silicon Valley. Well it seems to be a reality now with HealthifyMe's new AI feature - Snap!

If you're weary of the gruelling process of manually recording your meals and tracking calorie consumption through various health applications, you might find HealthifyMe's newest innovation, Snap, quite appealing. Snap is a food recognition system that harnesses artificial intelligence to recognise and monitor your food intake using photographs. You don't even need to open the app for its operation. Just capture an image of your meal from your phone's lock screen, and let Snap handle the rest later.

Abhijit Khasnis, VP - Engineering, HealthifyMe had a chat with Business Today on how Snap simplifies and makes tracking of nutritional information super easy. 

How does Snap leverage AI for food recognition? What makes it significant from a technical point of view?

Snap, the world’s first 'Fire & Forget' food recognition system, automates the nutrition tracking process, by using groundbreaking computer vision AI to identify and track foods directly from the photo gallery. It mixes light, efficient AI models on a user's phone with larger, more complex foundational  AI models on HealthifyMe's servers that have been fine-tuned using several thousands of food photos. Users only need to take a photo of their food plate, which they can do from their phone's lock screen without even launching the HealthifyMe app. The advanced AI capabilities of HealthifyMe then take over, automatically detecting the food items, to log the meal facts. It overlays heuristic data on top to personalise the results. Once the meals are logged, our AI systems also provide feedback on the foods consumed. This includes insights on macros and micros, along with recommendations and nutritional facts that are specific to the user.

What prompted you to launch Snap?

Nutrition tracking is an important tool to achieve one’s fitness goals. However, unlike steps or heart rate, it requires significant user discipline and involvement. With Snap, we make meal tracking as easy as clicking a picture. This ease when tracking food is a game changer for anyone looking to elevate their fitness game. As a result, clients are more aware of their food intake patterns, allowing them to make healthier choices on their own or with the assistance of their HealthifyMe coaches. 

How do you ensure the accuracy of the calories and macros, especially when it comes to health-related stuff?

The Snap AI model has access to heuristic analytics to personalise Snap recommendations using billions of previously logged foods, as well as humans-in-the-loop whenever necessary. The hybrid technique used produces a remarkable 75% accuracy rate, which is expected to increase as HealthifyMe Snap processes more and more images. Snap is currently tailored for Indian cuisine, which is typically a difficult cuisine to recognise due to a lack of reference images/data and the diversity of foods in India. With HealthifyMe’s extensive knowledge and model training, Snap is capable of recognising a variety of popular Indian dishes from all over the country. It can also recognise packaged food and drinks usually found on store shelves and is well-trained to detect popular international foods in India, such as pastas, burgers, and salads. Additionally, each food image is shown to the user when they open the app for review to ensure accuracy. This ensures the power is in the hands of the user with Snap simply doing the first round of detection to save time and effort for the user.

How has the initial response/ traction been for Snap?

Customers' meal tracking consistency has already increased by more than 80%. We are also doing a pilot that offers these capabilities on WhatsApp in order to enable more customers to experience SNAP.

Here's how it works:
You click a food picture or upload it to HealthifySnap's number.
Snap will send you Calories, PFCF and insights about your food.
If you have registered an account on Healthifyme with the same Whatsapp Number, it auto-tracks the food for you.

How is user privacy maintained with Snap?

HealthifyMe gives utmost importance to protecting user privacy. Snap is designed to ensure that only photos recognised as food items are uploaded to the app, thanks to an "on-device" model, after explicit user permissions. No picture that is not food is uploaded to HealthifyMe servers. Also, the capability to automatically analyse photos is accessible only to those who have granted Snap access to their camera roll. 

What data did you train your AI model on?

Over the years, HealthifyMe had attempted to bring image-based food recognition; however, the development of more advanced foundation AI models enabled the creation of Snap. We trained Snap on thousands of Indian foods and hundreds of thousands of pictures of these foods. Currently, 150,000 Indian cuisine items can be recognised by the feature thanks to training. Human reviewers examine and rectify erroneous recognitions. In order to enhance the model, users can also manually identify these incorrectly identified images.

What new innovations does HealthifyMe envision from their learnings from Snap?

We are monitoring the feedback received from the usage to increase the number of foods detected across global cuisines, which can aid us in our upcoming international launches,  and improve the food quantity recognition. Also, Snap is a step forward towards automatic tracking of key data. We have done this with SmartScale already. We will continue to add more health attributes to this “passive tracking” system over time.