Search for a command to run...
I work as a Tech Arch Senior Manager at Accenture and am a mother to two wonderful kids who test my patience and inspire me to be curious. I love cooking, reading, and painting.
No comments yet. Be the first to comment.
A Tale of Protocol Quirks, JSON Wrestling, and the Joy of Finding the Bug
In 2018, OpenAI pledged its "primary fiduciary duty is to humanity." Seven years later, the company launches erotic AI companions while families testify in courtrooms about children who never came home. We're conducting the largest unregulated behavi...
Did you know that 68% of organizations now identify data silos as their top operational concern? A 7% increase from 2023, according to a 2024 DATAVERSITY® Trends in Data Management Survey. In today's hyper-personalized business landscape, enterprises...
The Changing Search Paradigm This week, OpenAI introduced shopping capabilities within ChatGPT's search feature, coinciding with Mastercard unveiling Agent Pay, a new payment approach that enables AI agents to securely complete purchases. These devel...
Last week, the AI coding assistant tool Cursor encountered a crisis when its front-line AI support bot named “Sam” confidently provided a user with a completely fabricated policy.
The user raised a support ticket to resolve an issue with ending Cursor sessions when switching between multiple devices. Sam responded, “Cursor is designed to work with one device per subscription as a core security feature,” according to a now-deleted Reddit post. The incident quickly gained traction among frustrated developers on Reddit and Hacker News, with many threatening to cancel their subscriptions. Cursor’s leadership issued a retraction explaining that Sam, a front-line AI support bot, had “hallucinated” the policy. It had essentially fabricated information that didn’t exist.
This incident involving Cursor’s AI support bot, ‘Sam,’ highlights critical issues facing organizations deploying AI:
Sam’s response did not indicate that it was generated by an AI system. The lack of clear AI labeling led to further assurance that the response is made by a human. The company has since confirmed it will rectify it in future communications.
The fact that AI hallucinations aren’t isolated incidents but a systemic challenge, even for AI-focused companies.
This isn’t a unique incident. In February 2025, the Morgan & Morgan law firm admitted to citing fabricated legal cases made up by AI in a lawsuit against Walmart. This incident is described as one of the most shocking instances where AI confidently generated false information and illustrates potential consequences such as courtroom sanctions, professional embarrassment, and potential job losses. Similarly, Air Canada lost a legal battle in 2024 after its AI-powered chat assistant fabricated a refund policy that did not exist. The misinformation caused confusion and inconvenience for the traveler. These examples emphasize the tangible risks of LLM hallucinations, where AI confidently outputs information ungrounded in its knowledge base.
Let’s first dig into the term “AI hallucination” to understand it more before we dive into how enterprises can address this issue.
The term “hallucination” in the AI context refers to a phenomenon where an artificial intelligence system, particularly large language models (LLMs) and generative AI, generates outputs that are incorrect, misleading, or entirely fabricated. These outputs often appear plausible, coherent, or confident, especially in customer interactions, even though they are not grounded in factual accuracy or real-world information.
It's an anthropomorphic term that implies the AI is experiencing something similar to human hallucinations—seeing or believing things that aren't there. However, this anthropomorphizing is misleading. What's actually happening is a fundamental error in the model's pattern recognition and text generation processes.
When pressed for an answer, these models will attempt to provide one based on statistical patterns they've observed, even when they have insufficient data to form a reliable response. They don't have a concept of "I don't know" unless explicitly trained to admit uncertainty. Large language models aren't "imagining" things. They're failing to properly map their outputs, resulting in erroneously constructed responses (“confabulation”) rather than perceived experiences.
Personally, I think the term “hallucination” is misleading or personifies AI systems. This tendency to anthropomorphize is widespread. In fact, Sam Altman from OpenAI even noted the cost of users treating ChatGPT politely, saying ‘please’ and ‘thank you,’ is costing them millions of dollars of compute power. It is evident how easily we project human traits onto these systems. Are we failing to see the machines for what they are? Another term suggested to describe this phenomenon is “confabulation,” which means representing a process of creative gap-filling. Why not simply state them as “factual error”? Too techy?
Hallucinations are not unique to Cursor - they affect virtually all generative AI systems:
In a benchmarking study, OpenAI’s technology had a hallucination rate of about 3%, Meta’s systems around 5%, Anthropic’s Claude 2 over 8%, and Google’s Palm Chat as high as 27%.
Even top-performing models like GPT-4 and Gemini 2.0 Flash Exp exhibit hallucination rates between 1.3% and 1.8%.
One analysis found factual errors in 46% of chatbot outputs, and by 2023, chatbots were estimated to hallucinate up to 27% of the time depending on the model and task.
Concerningly, as models become more fluent and confident, their errors can become more difficult to detect because they're presented with the same authoritative tone as factual information.
Recognizing when an AI system is hallucinating requires vigilance and often domain expertise. Here are some key indicators:
Overly specific details that seem improbable or too convenient
References to non-existent sources, studies, or publications
Inconsistencies within the same response or across multiple responses
Perfect but suspicious answers to ambiguous or challenging questions
Confident assertions about recent events that fall outside the model's training data
Too-good-to-be-true solutions to complex problems
As a technical architect using AI in daily work tasks, I've found that cross-referencing critical information and implementing systematic verification processes are essential when leveraging AI outputs for business decisions.
For organizations racing to adopt AI technologies, hallucinations represent a significant business risk. Here's how enterprises can navigate this challenge:
Cross-checking LLM outputs against external databases or the web in real time to validate the generated response is correct reduces hallucinations by 20–30%. This verification can happen in real-time where the RAG retrieves the relevant data, updates the query and presents the more accurate response. For example, RAG verifies current data (e.g., who is the current premier of New South Whales) instead of relying on outdated training sets.
Implement self-verification mechanisms within your system architecture to ensure AI outputs are validated prior to delivery. A few techniques to leverage:
Chain of Verification (CoVe) encourages models to draft verification questions and independently answer those questions to check consistency before generating the final verified prompt response.
Real-time Verification and Rectification (EVER) employs external evidence retrieval and verification steps. Corrects outputs by validating the responses through multiple verification questions that are posed in parallel using chain-of-thought prompting. Responses that result in a false answer are rectified based on the evidence.
Develop a database of factual information relevant to your industry domain, products, and operations. This database should include edge cases and common misconceptions. The knowledge base must be updated regularly as the industry facts evolve.
Design prompts that probe the boundaries of the AI’s knowledge. The prompts should include variations of queries with verifiable answers alongside questions where the expected response should be “I don’t know.”
Track both false positives (hallucinated information) and false negatives (refusal to answer known information). Measure hallucination severity using a scoring system to define a benchmark.
Use the domain-specific benchmark before your system deployment as an entry criterion.
Develop feedback loops where hallucinations are identified to improve both the model and the benchmark.
A quick win that requires no architectural redesign is to develop chain-of-thought prompting and negative prompting:
Chain-of-thought prompting is where your query guides the AI models to break down their reasoning process into explicit steps, slowing down inference and creating verification points before arriving at a conclusion.
Example:
Instead of asking, "What caused the 2008 financial crisis?"
Change your prompt to:
Let's think step by step about the causes of the 2008 financial crisis. First, identify the
main sectors involved. Then, analyze the key events chronologically. Finally, explain how these
factors interconnected to create the crisis.
Negative prompting involves setting explicit boundaries in the instruction, priming for uncertainty, and establishing fallback procedures.
Example:
Answer the following question about our company policy. If you're unsure about any details or
if the information might not be in your training data, explicitly state that you don't know
rather than attempting to guess. Do not reference specific policy numbers, dates, or provisions
unless you are certain they exist. If you need to make an assumption, clearly label it as
such.
💡Bonus: For those interested, Hugging Face maintains a leaderboard that tracks, ranks, and evaluates hallucinations in LLMs - click here.
The Cursor incident demonstrates that even AI-focused companies are not immune to the risks posed by AI hallucinations. As enterprises accelerate AI adoption, robust architectural governance becomes increasingly critical.
The solution isn't to abandon AI adoption but to approach it with an architectural perspective. By designing systems that complement AI capabilities with appropriate verification mechanisms and human oversight, organizations can harness the benefits while mitigating the risks of hallucinations.
As technology architects, our role has expanded beyond traditional system design to include crafting resilient sociotechnical systems where AI and human intelligence work together, each compensating for the limitations of the other.
What strategies has your organization implemented to address AI hallucinations? I'd love to hear your experiences in the comments below.
https://www.nytimes.com/2023/11/06/technology/chatbots-hallucination-rates.html
https://en.wikipedia.org/wiki/Hallucination_(artificial_intelligence)
https://www.visualcapitalist.com/ranked-ai-models-with-the-lowest-hallucination-rates/
https://cacm.acm.org/news/shining-a-light-on-ai-hallucinations/
https://the-learning-agency.com/the-cutting-ed/article/hallucination-techniques/
Enjoy my blog? For more such awesome blog articles - follow, subscribe, and let's connect.
Disclaimer: The views expressed in this blog are my own; AI was used solely for editing and spell-checking purposes, and image generation is done through a beta AI tool and Canva.