Fill an online grocery cart from a handwritten list with ChatGPT agent

Posted: Jul 22, 2025 by Timothy B. Lee
Last checked: Oct 16, 2026
Method:: How we verify

Evidence: Article or interviewA newsletter, journalist or interview account of what happened.

PartialThe source says it only partly worked, or had a significant problem a person had to deal with.

What went wrong: Missed one of 16 items and gave up at the store login until the user stepped in; the author called it still not very useful.

Outcome

ChatGPT agent transcribed 15 of 16 items from a photographed shopping list (missing onions), found the nearest Harris Teeter and added them to the cart; a safety monitor first blocked the Kroger login, but after the user said they'd log in themselves and entered the password, it ordered the groceries.

Agent

ChatGPT agent

Prompt Improved prompt

Strengthened by us to avoid the problem in the source. Not tested by the poster. It doesn't change what happened in the source.

Here's a task I'd like your help with. Read it all the way through before you start.

Goal: Turn a photo of my handwritten shopping list into a grocery cart at my nearest store, ready for me to check out.

Check with me first: Which store, and which items on the list should you add (for example, only the ones marked in red)?

What I'll give you:
- A clear photo of my list
- My store or ZIP code ([STORE / ZIP])
- My store account (I'll log in myself)

Rules:
- First, write out every item you can read from the photo and number them. Show me the list and the total count before you shop.
- Mark any word you can't read clearly and ask me about it instead of guessing.
- If the site asks me to sign in, stop and tell me. I'll log in, then you carry on from where you stopped.
- Add each item, then tick it off. Before finishing, compare the cart against the numbered list and name anything missing.
- Pick the closest match for brand-free items and note any substitutions.
- Don't check out or pay. Stop at the cart.

Output: The numbered list you read, the cart with every item, and a short note of anything missing, swapped or unclear.

You'll know it worked when: the number of items in the cart matches the number on my list, with no surprises.

Try this in:

ChatGPT Used in source Gemini Grok

This prompt has blanks to fill in, like [STORE / ZIP]. We'll copy it instead of filling it in, so you can edit it before you send.

Opens in a new tab. We don't see what you send.

These links are only a shortcut. We're not affiliated with OpenAI, Google or xAI, and your results may not match the source.

The prompt they actually used (our rebuild)

The source didn't share its exact prompt, so we wrote this from what it describes.

Here is a photo of my shopping list. Add the items marked in red to my cart at my nearest grocery store's website and get it ready to order.

Optional stronger version

Our rewrite with extra checks. The poster didn't test it.

Here's a task I'd like your help with. Read it all the way through before you start.

Goal: Turn a photo of my handwritten list into a complete cart at my local store, with nothing missed, ready for my checkout.

Quick check first: tell me if anything below is missing or unclear.

Inputs (fill in or confirm):
- A clear photo of the list
- Store name and ZIP ([STORE], [ZIP])
- Which items count (e.g. only ticked or red items)
- Brand or size preferences and items to never substitute

Rules:
- First transcribe every line into a numbered list and show it to me; mark anything you can't read with '?'.
- Add items in list order and tick each one off the numbered list as it goes in.
- For out-of-stock items, pick the closest match only if I allowed substitutes; otherwise skip and note it.
- Before finishing, compare cart count with the list count and add anything missing.

What went wrong before, and how to avoid it:
- Last time one of 16 items was missed: reconcile cart vs list before you stop.
- Last time it gave up at the store login: hand the login to me and resume rather than ending the task.

Checkpoint: Stop at checkout. Don't choose a delivery slot, tip or pay.

If you get stuck: When the store asks you to sign in, pause immediately and ask me to log in (including any CAPTCHA or two-step code), then continue from where you stopped instead of quitting; never guess or store passwords.

Verify before you finish: Show list items vs cart items side by side; every line must be 'added', 'substituted' or 'skipped (reason)'.

Output: The reconciled list table, cart total and a link to the cart.

Done when: every item on the list is accounted for and the cart is ready to pay.

Source

https://www.understandingai.org/p/chatgpt-agent-a-big-improvement-but

Order groceries from H-E-B with ChatGPT agent (access denied)Improved promptEvidence: Screenshots or logsFailure
The poster tried to order groceries from H-E-B in Austin with ChatGPT agent, but it failed because access was denied.
ChatGPT agent · Shopping · Source date: Jul 25, 2025
Prep an Instacart order from history and sales with ChatGPT agent (odd picks)Recreated promptEvidence: Described by poster
Given access to Instacart and told to prep an order from the poster's history and current sales, ChatGPT agent picked items with reasoning the poster found oddly generic ("cereal, which is on sale and a usual part of groceries").
ChatGPT agent · Shopping · Source date: Jul 29, 2025
Order groceries from a 16-item list with OpenAI OperatorRecreated promptEvidence: Article or interview
Operator misread parts of a 16-item grocery list and added only 13 items to the cart even after corrections; the author concluded it wasn't ready for real use.
OpenAI Operator · Shopping · Source date: Jun 26, 2025