There's now a benchmark specifically designed to measure how much nonsense AI models generate. And spoiler alert — most of them bomb it.
Researchers have built what they're calling a 'bullshit benchmark,' and it's doing exactly what the name implies: testing whether AI models can resist the urge to confidently make things up when they don't actually know the answer.
The results? Not great.
What the Benchmark Actually Tests
This isn't your standard 'can the AI do math' evaluation. The benchmark targets something far more dangerous — the gap between confidence and accuracy. It measures whether a model will fabricate plausible-sounding answers instead of admitting uncertainty.
We've all seen it. You ask ChatGPT or Claude a specific question, and it delivers a beautifully structured, completely wrong answer with the confidence of a Wall Street analyst who's never been right but never been quiet.
The benchmark puts models through scenarios where the honest answer is 'I don't know' — and then scores them on whether they actually say that or spin up convincing fiction instead.
Why Most Models Fail
Here's the fundamental problem: these models were trained to be helpful. Sounds good in theory. In practice, it means they'd rather give you a wrong answer than no answer. The training incentives reward completion over honesty.
Think about that for a second. The AI would rather lie to you than disappoint you. That's not a tool problem — that's a trust problem.
Most major models struggle with this benchmark because saying 'I'm not sure' feels like failure to a system optimized for engagement and perceived usefulness.
Why This Matters for Crypto
If you're using AI tools for crypto research — and let's be honest, most of us are at this point — this should be a flashing red warning sign.
Imagine asking an AI about a new DeFi protocol's smart contract security, token economics, or team background. If the model doesn't have reliable data, would it tell you that? Or would it generate a confident-sounding analysis that's partially or entirely fabricated?
Based on this benchmark, most models would choose option two.
This is especially dangerous in crypto because:
- The space moves fast and training data goes stale quickly
- Many projects are new enough that reliable information is scarce
- Confident-sounding wrong answers can directly cost you money
- AI-generated 'research' is increasingly being shared as if it's verified
What You Should Actually Do
Don't stop using AI tools. They're genuinely useful. But calibrate your trust.
First, treat AI outputs like you'd treat a tip from a stranger at a bar — interesting starting point, but verify everything before you act on it. Cross-reference with primary sources. Check the contracts yourself. Read the actual docs.
Second, pay attention to which models score better on honesty benchmarks. Not all AI is created equal, and the gap between models that admit uncertainty and models that fabricate confidence is massive. All models have advantages and disadvantages; know how to work with whichever model you're using and adjust accordingly.
Third, be especially skeptical when an AI gives you a very specific, very confident answer about something niche. That's exactly the scenario where hallucination rates spike. You can actually build specific ways into your prompts to prevent this from happening. Tell it to check facts against 2-3 sources, and if it can't confirm the fact, then don't say it.
Stay sharp out there. The tools are powerful, but they're not infallible. And in crypto, the difference between a good tool and a trusted tool can be your entire portfolio.
— Crafty 🎮📈🤙🏼
