ChatGPT can now see images; from cooking to office work, here's how you can put it to use

OpenAI made an interesting announcement on Monday when they claimed that ChatGPT can now 'see', 'hear' and 'speak'. But what does this mean? It hasn't grown a set of eyes and ears yet but what OpenAI wants to highlight is its ability to view and analyse images and also provide voice output. The new features bring ChatGPT closer to AI assistants we've seen in sci-fi movies. But how can these features be used on a daily basis? 

Read More

ChatGPT will be able to use visual information. So, you can share one or more images with the AI to address a wide range of tasks. Troubleshoot issues like a malfunctioning gadget, or even analyze complex graphs for work-related data, or plan your meals by scanning your fridge and pantry. Within the image, you can even help ChatGPT focus on specific details within an image with the help of the drawing tool available in the mobile app.

To explain the new feature, here's one example of how it can be used to create a custom recipe using only the ingredients you have. 

Step 1: Capturing the images

Imagine you've just returned from work, and you're contemplating what to prepare for dinner. Instead of scanning through the fridge, you decide to take the help of ChatGPT. Here's how you can use the image-based feature:

• Open the ChatGPT app on your mobile device.
• Tap the plus button to initiate image input.
• Use your device's camera to capture images of the contents of your fridge and pantry. You can take separate pictures of each if they are located in different places.

Step 2: Uploading the images

Once you've captured the images, ChatGPT will prompt you to upload them. You can review the images to ensure they are clear and include all relevant items.

Step 3: Guiding ChatGPT with the drawing tool

ChatGPT might ask if there are specific items you'd like to focus on or any particular dietary preferences or restrictions. To guide ChatGPT, you can use the drawing tool within the app. Circle specific items or ingredients in the images that you want to incorporate into your meal. Add labels or notes to clarify your preferences. For instance, you could circle a pack of chicken breasts and write "Use for protein."

Step 4: Engaging with ChatGPT

ChatGPT will analyze the images and your annotations to suggest suitable recipes based on the ingredients you have available. You can ask for step-by-step instructions, cooking tips, or even nutritional information.

Step 5: Exploring recipe options and fine tuning

ChatGPT's image-based feature goes beyond providing a single recipe. It can offer multiple recipe options based on your ingredients and preferences. You can select your preferred cuisine to help finalize the recipe. 

Step 6: Meal planning and cooking

Once you've chosen a recipe, ChatGPT can provide detailed instructions, including cooking times, temperatures, and techniques. You can follow along and prepare your meal with confidence.

Can it help with office work as well?

This ChatGPT feature can let you analyse graphs and tables by just clicking a picture. It can simplify the data for you or even draw inferences from it. However, OpenAI has been very vocal about the chances of errors and inaccuracies.

Availability of new ChatGPT Features

Initially, the voice and image features will be available to Plus and Enterprise users over the next two weeks. OpenAI plans to introduce these capabilities to other user groups, including developers, in the near future.

Also read: Spotify tests voice translation feature for podcasts with help from Microsoft-backed OpenAI

Also read: After GPT-4, OpenAI launches Dall-E 3, its newest text-to-image tool; see what's new