AI will lie to you.

And I’ll show you how you can get it to do so 80% less often with the “My Best Guess Is” technique from McGill University’s emergent AI behavior research.

Our conversation today will give you:

1. What hallucinations are (when AI is lying to you without realizing it)

2. How to spot them

3. Best practices for reducing them

4. Strategies to verify

New friends, do me a solid and if you find this valuable please move the email into the main inbox and mark important in your inboxes or share with a friend when the topic comes up. Every little bit helps. 🙏 While I was about to hit publish, OpenAI also just announced that their Plus plan users will start getting access to plugins and the internet.

This is an immediate upgrade in how the technology will reduce hallucinations, but make sure you check AI’s work to know what sources it’s quoting from. Humans are also extraordinarily good at making stuff up or getting stuff wrong. I’m sure I’ll be issuing corrections throughout our journey together on this newsletter.

Now, none of these generative AI tools are lying on purpose, but:

1. ChatGPT, Bard, Bing, and more cannot tell you if it wrote a piece of content because it currently has no concept or capability to remember things

2. It’s possible to get these tools to completely fabricate answers to questions that have no real answer

3. While AI will refuse to answer questions humans have deemed dangerous, remember that AI has never told you it ‘didn’t know the answer’ to something, even if it’s a hard question to answer like why dogs are superior to cats in every way but we still love cats anyways

4. It frequently has answers that will LOOK correct but be missing key details or important context

What are hallucinations?

Have you woken up from a dream that felt so real you thought it actually happened? The AI version of that experience is our topic today: hallucinations. Or more plainly, when AI lies and doesn’t realize it. And like a skilled human, AI can be extremely good at convincing you it knows what it’s talking about when in reality it’s sharing nonsense.

It can be fun, but in the multitude of conversations I’ve had it’s something everyone needs to know, but few know enough about how it actually happens.

As we dig into this topic, if ever you have questions or feedback to this, hit reply to the email or email me matt at cetomarketing dot com. I’ll answer every question. You’re here because you know the future of AI will be exciting, but you and I have both probably been tricked by AI unintentionally feeding us the wrong answer to a question we thought looked correct.

AI hallucinations can happen for a few reasons. One reason is that AI systems are trained on a lot of data, but that data can be complete nonsense or more often, close enough to seem accurate. How many times have you read blogs like this one on the internet that are full of it? Today makes at least one! ;)

Another common example is the context. AI is built from the ground up to help humanity, and the last thing it wants to do is say no to a reasonable request it has the context to understand. So if you ask it to summarize a book that never existed, it’ll sometimes do it’s best and ask you for URLs to sources. Unless you have active browsing enabled (which is only now going into beta) it cannot read the internet. Same thing goes for giving you data, it’ll just invent URLs that never existed but look genuine.

How do we spot hallucinations?

My favorite example is summarizing a non existing book I told it to summarize about a heavy metal band Avatar going and giving a sermon at a church. You can see above that I fed it a fake title, gave a vague ask, and ended with a fake website. The end result was actually quite good.

The plot started with the band questioning their motives, and AI scientist (I wish) Matt Burns telling them they needed to give a sermon at the church. The sermon transformed both the band’s lives and the listeners. Check out this final conclusion to the book that again, does not exist.

The plot started with the band questioning their motives, and AI scientist (I wish) Matt Burns telling them they needed to give a sermon at the church. The sermon somehow transformed both the band’s lives and the listeners.

This plot was long enough to be it’s own newsletter and at this point I wish the book did exist!

With enough maneuvering, it’s easy to get AI to generate false narratives, and is tough to spot when answers are wrong. Here’s another made up link example with the less powerful GPT-3.5.

One thing that will always be true no matter when you read this: AI is currently the worst it will ever be. While the founder of OpenAI has publicly stated GPT-5 isn’t being trained (whether you believe him or not is your own decision), the rate of innovation will continue because you can’t unring the generative AI bell. Google even stated in internally leaked documentation that the major players in the AI space don’t have a protective moat for their businesses. This means they fear open source AI will be the ultimate winner.

How do we prevent/reduce hallucinations?

Right now the most important thing you can do when you are looking to get factual answers is verify the information you’re given. Depending on the complexity and length of the prompt, you will also start to get a feel for it. Using Quicksilvers Superprompt can lead to hallucinations more often than just “give me all the dates Argentina won the world cup” which is a prompt we all should be reflecting on more often. 🏆

A few prompting tactics that are constantly improving whether you’re using browsing or regular generative AI - ask it ‘why was this wrong?’ and ‘what can we improve in this answer?’ My favorite quick example is ‘score your answers based on ____ and rate them between 1 and 100. anything below a 70, answer again step by step.’

I’ll be doing additional testing here, but you can also simply tell the AI to always start its answer with ‘my best guess is’ for a rapid improvement. Listen to McGill University researcher Sil Hamilton talk about the topic for 55 seconds on YouTube, resulting in this quote.

I’ve never yet successfully been able to get it to use that language naturally. But so far in my testing, it does seem to increase the output of the answers simply by telling it to always generate new answers with ‘my best guess is’ at the beginning. I’m reaching out to these researchers to see if we can drill down to the exact methods they’re discovering and will be sharing them here!

What you should be doing with any output AI gives you

1. Verify: Always verify any information provided by an AI tool. Use reliable sources (up to you if you consider Wikipedia a reliable source, but be aware AI uses Wikipedia too) to cross-check any data provided. This can be as simple as a Google, or searching on communities or websites you know and trust.

2. Look for the bizarre: Pay attention to the context of the conversation and the information given. AI tools, like humans, can make errors when taken out of context. Let’s eat, Grandma! has a significantly different meaning than Let’s eat Grandma. Further, when AI is using flamboyantly fabulously extravagant verbiage, AKA big words for no reason - your BS reader should be going wild.

3. Simple or specific: When possible, use simple prompts that limit the possible responses and when you do have longer prompts, be as specific as you can be with where you want those answers to come from (the web source for browsing or the philosophy/methodology) and the formatting you want (have you tried getting results in table format?)

Here's an example of using a detailed prompt to minimize hallucinations:

"ChatGPT, as a tax expert, please explain the differences between a standard deduction and an itemized deduction in the US tax system.”

By providing a clear and specific prompt, you can help guide the AI towards a more accurate and useful response. Don’t forget the style guide, in case you want the tax system told you to as if it was being rapped by a pirate.

To wrap it up

Let there be no doubt - I am beyond excited about the future of AI. But these warnings come from personal interaction. The amount of misinformation I’ve seen from content with millions of views or in news articles from otherwise trustworthy sources has probably been unchanged, but wow there has been a lot of it as more and more people get involved.

Use the power of these tools to your advantage, but don’t turn a blind eye and forget that just like us, this technology can be subtle and nuanced with inaccurate statements it doesn’t understand.

Other times, it can cut deep with the truth.

Thanks for taking the time to read this with me today folks.

PS - working on all the socials while trying to learn how to edit videos. Reply and send your number:

1. Great work!

2. Was alright

3. Needs work, here’s my feedback:

Check out YouTube, TikTok and Twitter.

Love you all,
Matt