Can AI Really Do Fortune-Telling? A Field Report from Practice Rounds to the Official Arena
Every fortune-telling website claims to be accurate — yet no one has ever produced evidence for what "accurate" means. We put the AI behind MingLuo Pan through two exams: a self-run, reproducible practice round, and an official arena with server-side grading, scores locked at submission, and only one chance per entrant. Here are the full results:
| Venue | Nature | # of Questions | Score (Top-1) |
|---|---|---|---|
| MingLi-Bench (local test) | Public question set, reproducible | 160 | 33.1% |
| Shen Ji Miao Suan Arena public_test | Public practice, server-side grading | 80 | 38.75% |
| Shen Ji Miao Suan Arena private (official scoring) | Private question bank, single-shot leaderboard | 495 | 41.9% |
The passing line for random guessing is 25% (four choices).
What the Questions Look Like
The "Global Fortune-Teller Championship" is an open arena in the metaphysics world: you're given a real person's birth data and asked what happened in a particular year — marriage, career, health, wealth, disasters — all of them real events that actually occurred and are verifiable, with four choices. The Shen Ji Miao Suan Arena (bench.sjms.ai) adds a private question bank on top of this: questions are distributed, answers are never distributed, and grading is done server-side to prevent "memorizing questions to farm scores."
For example, one real question:
> Male, born 1974, born in the USA. Question: What happened to this person in 1996?
> A. Developed severe depression / B. Returned to Hong Kong and met his current wife / C. Traffic accident but unharmed / D. Received unexpected windfall
What's being tested isn't "does it sound plausible" — it's whether the judgment is correct.
How We Tested
We used the metaphysics evaluation benchmark MingLi-Bench (open source, MIT). Two key settings in the rules:
- Pre-computed chart injection (--astro): Each question comes with a BaZi (八字) + Zi Wei Dou Shu chart pre-computed by iztro — the AI doesn't need to calculate anything, only to interpret. This is exactly our architectural philosophy: leave chart calculation to a deterministic engine, leave interpretation to the AI — what's being tested is precisely this interpretation layer.
- Chain-of-thought reasoning (--cot): The model reasons step by step before giving an answer, consistent with how it works during real chart readings.
The passing line for random guessing is 25% (four choices).
Round 1: Local Practice Round (MingLi-Bench, 160 questions)
| Model | Score | Correct | Equivalent to |
|---|---|---|---|
| Pro tier (used for paid readings) | 33.1% | 53 / 160 | 1.3× random baseline |
| Flash tier (used for free quick answers) | 29.4% | 47 / 160 | 1.2× random baseline |
Category highlights:
- Pro tier's strongest areas: Children 50%, academics 45.5%, personality 42.9%, health 35.3% — structural judgments (patterns/temperament/six relations) are its strength;
- Flash tier scored a surprising 69.2% on wealth but only 9.1% on academics — very inconsistent, so the free tier is only suitable for quick answers; deep readings still need Pro;
- Both tiers completely failed on disasters, legal troubles, and fortune-telling categories (all 5 questions wrong) — these tricky "what suddenly happened in a given year" questions are currently as hard for AI as they are for human contestants, and we're laying that out honestly.
Round 2: Official Arena Scoring (Shen Ji Miao Suan private, 495 questions)
Practice rounds can prove themselves, but there will always be skeptics saying "the questions are public and you scored yourselves." So we entered the official arena at bench.sjms.ai — the rules are brutal: a private bank of 495 questions, answers never distributed, one submission chance per entrant, scores locked and leaderboarded at submission, no retakes.
What we put on the field was our full product pipeline: birth data → our Zi Wei Dou Shu engine for deterministic chart calculation + lunar_python for BaZi Four Pillars (四柱) → full chart injected into the prompt → Pro model makes judgments → server-side grading.
Official results (scored 2026-10-09, task task_efea525e57909ec3):
| Metric | Result |
|---|---|
| Top-1 (382 scored questions) | 41.9% |
| Top-2 (correct answer in top two) | 62.8% |
| 95% confidence interval | 37.3% – 46.8% |
| Coverage | 100% (495/495 answered) |
| Official ranking | 7th / 14 teams |
Leaderboard context (as of 2026-10-09, verifiable in real time at bench.sjms.ai):
- The leader is Gemini-3.6 (61.3%); second place GPT-5.6 (58.4%) was officially flagged by human reviewers as "suspected internet access, not answering normally"; two other teams were flagged as "same-source multi-application";
- Among clean scores with no flags, only three teams are ahead of us: Gemini-3.6, Claude Opus 5 Vedic Full Array (50.0%), and Axiom Three-Layer Pipeline (45.5%) — all flagship models with heavy-duty pipelines;
- We hit 41.9% with the consumer-grade mimo-v2.6-pro, the smallest model on the leaderboard, ranking above deepseek-chat (38.5%). Trading scale for results — this math works out;
- Reference point: the benchmark's white paper baseline for "chart calculation + flagship model" averages 36.2% — we're 5.7 percentage points higher;
- Diagnostic slices: classical text questions 44.4%, paraphrased questions 34.4%, chronicle questions 22.7% (honestly flagged: chronicle-type questions are our weak spot).
How to Read This Score
This is not proof that "AI fortune-telling hits every time" — quite the opposite. It maps the boundary of capability.
- Clearly beating random, and beating the industry baseline of "chart calculation + flagship model" (36.2%), shows there are genuinely learnable patterns in chart reasoning — it's not pure rhetoric;
- But being far from a perfect score shows there is real uncertainty in metaphysical judgment — consistent with what we've always said: a chart gives probabilities and tendencies, not fate;
- The competition questions are inherently tricky (deliberately picking unusual years in a chart), while real-world chart readings discuss patterns and how to navigate them — practical usability is higher than this raw score;
- One detail verified by testing: feeding the full chart (including Flying Palace Four Transformations, Sui Qian and Jiang Qian stars) verbatim to the model scores better than a compressed summary — so our interpretation pipeline doesn't skimp on those tokens.
What we actually care about more is something else: in the metaphysics AI industry, almost no one dares to put their model on a public exam. We dare, because "verify before you believe" isn't just a slogan — every current-events divination prediction is publicly timestamped and marked when fulfilled; every assertion from our chart engine is verifiable; and this time, even the AI's own score is laid out for you to see.
Direct Impact on Our Product
This field test determined the backend configuration of MingLuo Pan:
- Paid deep readings use the Pro model — 33.1% vs. 29.4%, the score gap justifies the price gap;
- Free quick answers use Flash — 29.4% is sufficient for everyday quick questions, but category variance is high; don't rely on it for deep chart readings;
- Qi Men Dun Jia (奇门遁甲) has no public question set yet — we're building our own Qi Men question bank with the same approach, and will publish results once we have enough questions.
Limitations Disclaimer
Multiple-choice questions only test "event judgment," not the "chart-writing" we do daily; pre-computed charts are generated by a third-party library and may have subtle differences from our engine; the 160-question sample size means the score carries a statistical error of about ±7%. Results represent reasoning performance under this specific benchmark only and do not constitute comprehensive proof of chart interpretation effectiveness.
---
*Data reproducible: MingLi-Bench is open source — anyone can retest with the same question set. Our complete response logs are archived for reference.*