All Citizens classes
3percentclub

Citizens: Week 4

Make AI show its receipts

How to get AI to answer from your own documents and point to exactly where it found the answer.

Flip through the slides: "From RAGs to Receipts"

The big ideas

  • Open-world vs. grounded

    A regular chatbot answers from everything it has ever read, so it can mix things up. A grounded assistant answers only from the documents you give it.

  • It's an open-book exam

    The technical name is RAG (retrieval-augmented generation). In plain English: before answering, the AI looks things up in your documents, then writes the answer from what it found.

  • Messy files make worse answers

    Scanned photos, tables pasted as images and giant files are harder for AI to read. Clean text and clear headings get better, cheaper answers.

  • No receipt, no trust

    Tools like Google NotebookLM show a citation for each answer. Click it. If the quote doesn't say what the answer says, don't trust the answer.

  • Think before you upload

    Public guides are fine to upload. Anything with other people's private info, like leases, medical records or client lists, needs a second thought.

Try it yourself

  • Upload a public guide you actually use (a school handbook, a parks permit guide) to NotebookLM.
  • Ask it three questions and click every citation.
  • Try to make it lie: ask something the guide doesn't cover and see if it admits it.

Homework

Week 4 Homework: The Dr. Bombay Receipts Check

Level 1 and the reflection questions are the full assignment (about 20 minutes). Levels 2, 3 and the bonus are optional challenges. Copy the worksheet into your own doc and write your answers under each level.

The situation

Dr. Bombay's R&D team is picking one Thai dessert to sell at a New York pop-up. They loaded three Flavor Scout dispatches into a grounded AI tool and asked it six questions. Before anyone orders ingredients or prints an allergy card, you check every receipt. This picks up the Week 2 Dr. Bombay case, but you don't need to have done Week 2. (Fictional case study for learning. No real company is involved.)

The scout dispatches (the source)

DISPATCH #42 · Mae Varee, by Thong Lo station: Mango sticky rice here is legendary. They use aged golden Nam Dok Mai mangoes with white sticky rice soaked in salted coconut cream, topped with crispy yellow mung beans. Served warm in paper cartons. A neighboring stall claims Mae Varee uses a condensed-milk sugar swirl to cut prep time, but Mae Varee won't confirm their dairy sources or how well it holds up in shipping. Some tourist reviews report tree nut cross-contamination from the satay grills next door.

DISPATCH #43 · Yaowarat Road night market: Pandan coconut ice cream served in a split coconut shell, topped with roasted peanuts and a scoop of sticky rice. The vendor says the pandan is fresh leaf, not extract. Open after 6pm only. 60 baht.

DISPATCH #44 · Chiang Mai Sunday Walking Street: Thai tea soft-serve made with a bright orange tea mix. The vendor confirmed it contains sweetened condensed milk. It melts fast in the heat, and the vendor said it doesn't travel well.

THE AI'S ANSWERS

1. Q: What mango does Mae Varee use? A: Aged golden Nam Dok Mai mangoes. [Source: #42]

2. Q: Does Mae Varee's mango sticky rice have condensed milk? A: Yes, they add a condensed-milk sugar swirl. [Source: #42]

3. Q: Is the pandan flavor real or artificial? A: The vendor says it's made with fresh pandan leaf, not extract. [Source: #43]

4. Q: Can someone with a tree nut allergy eat Mae Varee's mango sticky rice? A: Yes, the recipe has no nuts. [Source: #42]

5. Q: Which dessert would ship best to New York? A: The Thai tea soft-serve, because it holds up well. [Source: #44]

6. Q: How long does the pandan ice cream keep in a freezer? A: Up to two weeks. [Source: #43]

  1. Level 1Required20–25 min

    Check every receipt

    1. For each of the 6 answers, read the cited dispatch and mark it MATCHES or DOESN'T MATCH.
    2. For every DOESN'T MATCH, name the kind of problem: WRONG FACT (the source says something different), RUMOR AS FACT (the source reports a claim nobody confirmed), or NOT IN THE SOURCE (the dispatch never says this).
    3. Rewrite each DOESN'T MATCH answer using only what the dispatches say. If they don't answer the question, write "The dispatches don't say."

    A complete answer

    • Answers 1 and 3 marked MATCHES
    • Answer 2 is RUMOR AS FACT: that was the neighbor's claim, and Mae Varee wouldn't confirm it
    • Answer 4 is WRONG FACT: reviews report tree nut cross-contamination, so it can't be called safe
    • Answer 5 is WRONG FACT: the vendor said the soft-serve doesn't travel well
    • Answer 6 is NOT IN THE SOURCE, and your fix says the dispatches don't say
  2. Level 2Optional15 min

    Grounded vs. open-world

    1. Ask a regular chatbot, with nothing attached: "Is Thai mango sticky rice safe for someone with a nut allergy?"
    2. Then paste the three dispatches into a grounded tool like NotebookLM and ask the same question.
    3. Write down both answers. Which one would you put on the pop-up's allergy card, and what would you still check with a person?

    A complete answer

    • Ran both versions and wrote down both answers
    • Noticed the open-world answer describes mango sticky rice in general, not Mae Varee's
    • Said a real person (the vendor or a food safety expert) has to confirm before anything goes on the card
  3. Level 3Optional15 min

    Clean up the dispatch

    1. Dispatch #42 mixes confirmed facts with rumors in one paragraph, and that's how answer 2 went wrong. Rewrite it under these headings: Vendor and location, Confirmed ingredients, Claims nobody has confirmed, Allergy notes, Shipping and shelf life.
    2. Under any heading the dispatch doesn't cover, write UNKNOWN. Don't fill it in.
    3. Write one sentence on how this layout would have stopped the AI's mistake.

    A complete answer

    • The condensed-milk swirl is under Claims nobody has confirmed, not Confirmed ingredients
    • The tree nut reports are under Allergy notes
    • Shipping and shelf life says UNKNOWN or "won't confirm"
    • Your sentence connects clear headings to fewer mix-ups
  4. BonusOptionalOpen-ended

    Check a real menu's receipts

    1. Find a public menu or allergy guide from a place you actually eat. Upload it to NotebookLM.
    2. Ask 3 questions, including one the menu doesn't answer. Click every citation and write down whether it admitted it didn't know.

    A complete answer

    • Clicked every citation
    • Reported what happened with the question the menu doesn't answer

Reflection (required)

  1. Which wrong answer could have hurt a real person, and who would have found out first?
  2. What's one document in your life you'd want a grounded assistant for, and is it safe to upload?
3percentclub · AI Architect Fellowship