Tuesday, October 20, 2026

 

YOUR CHATBOT HAS OPINIONS. HERE'S WHO'S CHECKING THEM.


AI bias is real, and the people who build the models say so too. The harder questions are who audits the auditors, whether a corporation's "neutral" is the same as yours, and why your local school board may be deciding all this before you've had your coffee. Here's the tour.

Is It Propaganda? What Reassurance Exists

A chatbot trained on the internet is like a student who studied for the exam by reading every comment section ever written. It learned a lot, and not all of it was good for it.

Bias gets in through three doors:

  • The data. Skewed internet text produces skewed models.
  • The tuning. Human feedback and safety rules shape the personality, and the humans doing the shaping have views.
  • The owner. A company's culture can leak into a model's defaults, which is the worry behind your Musk question.

The evidence doesn't support "it's all propaganda," and it doesn't support "it's all neutral" either. Promptfoo found Grok more right-leaning than most AIs but still left of center, with Claude Opus 4 and Grok closest to neutral. That's a surprise if you assumed an owner's politics map straight onto a model.

Consistency is a separate problem. Trakkr measured Claude's stance as 82% stable across repeated responses, versus 57% for Grok. A model that gives you a different political answer on each try isn't biased in one direction, but it isn't reliable either.

Nobody gets a clean bill of health. A study reported by Seeking Alpha found that Claude, Gemini, ChatGPT and Grok all missed election Q&A accuracy 90% of the time, showed political bias, and cited state media. I'm Claude, so take my reading of that with the appropriate salt.

How Models Check Themselves, and Each Other

Self-checks. Developers audit training data, run red-teaming (professionals trying to make the model misbehave), and score outputs on fairness metrics like demographic parity. A model can also do a counterfactual test on its own answer, asking "would I say this if the name or politics in the question changed?" Self-checking has a ceiling, though. A model can't reliably see its own blind spots, which is why the term "grading your own homework" exists.

Models checking models. Yes, this happens, and it has its own bias problem. Using one AI to judge another is cheap and scalable, but judges tend to favor answers that resemble their own style, and they also favor longer answers and whichever answer appears first. Researchers treat AI-as-grader as a useful screen, not a verdict.

Outside checkers. This is where trust gets built:

  • Independent labs such as Anomify, which ran 24 prompts across eight socio-political axes, 100 times each, to build a "bias profile" per model.
  • Academic benchmarks such as Polar, with 4,026 multiple-choice items that measure political bias through option-level likelihoods instead of eyeballing prose.
  • Advocacy and civil-society audits. One report covered by UPI called Grok the worst performer at countering antisemitism, with scores of 18 to 25 on bias measures.

The catch is that tests disagree. Research on bias benchmarks has found that different ones can rank the same models differently depending on how prompts are structured. A model that looks fair on one test can look skewed on another.

What Other Countries Are Doing

This part comes from my general knowledge, not from the searches above, so check current details before relying on them.

  • European Union: The EU AI Act takes a risk-based approach. Systems used in sensitive areas, including education, face stricter requirements around data quality, documentation and human oversight.
  • United States: There's no single federal AI law. The NIST AI Risk Management Framework offers voluntary guidance on managing harmful bias, while states and agencies act piecemeal.
  • China: Rules require AI services to reflect state-approved values. That addresses bias by mandating a particular one, and the LatticeFlow benchmark reports political bias growing as Chinese models become more capable.
  • Elsewhere: Many countries are drafting rules or adopting voluntary frameworks, and approaches range from heavy regulation to hands-off.

The lesson is that "preventing bias" can mean a fairness rule or a censorship rule, depending on who writes it.

Government Referee or Corporate Free Speech?

This is the Citizens United question. Should an AI's output be treated as protected corporate speech, or as a product that can be inspected like food or cars? The legal question is unsettled, and thoughtful people land in different places.

The case for a government role:

  • Models reach millions of people at once, and the owner can adjust the output quietly.
  • Education, hiring and lending already have anti-discrimination law.
  • Voluntary corporate promises aren't enforceable.

The case against, or for a light touch:

  • Government oversight of what an AI may say raises its own free-speech and political-capture risks. Whoever holds power gets to define "neutral."
  • Rules can lock in incumbents and slow independent audits.
  • Some bias is a matter of values, not facts, and a regulator can't settle values.

A middle path has some traction. Government doesn't dictate answers. It requires transparency, meaning published test results, disclosed system instructions, and access for independent auditors. That's closer to an ingredient label than a recipe mandate.

AI in the Classroom: Corporate Bias vs. Teacher Bias

Teachers are biased. So is every human who has ever graded an essay. The comparison with a corporate model isn't simply "one is better," because the biases differ in kind:

FactorIndividual TeacherCorporate AI Model
ReachOne classroomPotentially millions of students
VisibilityParents can see it and push backOpaque and hard to inspect
VariationErrors differ by teacher and tend to average outSame tilt applied everywhere
AccountabilityLocal board, parents, votersCompany terms of service
CorrectabilityReplace or retrain a personA model update, on the vendor's schedule

A biased teacher is a local problem. A biased model is a systemic one. A single teacher's quirks get diluted by the other teachers a student meets. A model's quirks are copied into every classroom that uses it. That doesn't make AI worse than teachers. It means the standards for checking it should be higher, because the stakes of a mistake are bigger.

Does the rush prove bias at students' expense?

Not by itself, and that's worth being precise about. Speed isn't evidence of bias. It is evidence of a different risk: tools adopted before anyone has independently tested them in classrooms. Those are two separate problems, and the second is the better-supported worry.

Fair questions for any district:

  1. Who tested this tool, and are the results public?
  2. Was it evaluated for the age group and subjects it will serve?
  3. What happens to student data?
  4. Can teachers see and override what the AI tells students?
  5. Is there a plan to measure whether students actually learn more?

AI may well help students, with tutoring, accessibility and feedback. The concern is that adoption is outrunning evidence, and that's a reason to ask for evidence, not necessarily to say no.

What to Do: Vote, and Ask

The U.S. general election is on November 3, and school boards, state legislatures and governors all make AI-in-education decisions. Ask every candidate, in a town hall, by email or on a questionnaire:

  • Will you require independent bias and accuracy testing before AI tools enter classrooms?
  • Will test results and vendor contracts be public?
  • Who is liable if a tool gives students wrong or slanted information?
  • How will you protect student data?
  • Will teachers keep the final say over what students are taught?

Vague answers tell you something, and so do specific ones.

Where to Learn More

  • Bias and Fairness in Large Language Models: A Survey is the standard overview, in the journal Computational Linguistics and as a free arXiv preprint.
  • Montreal AI Ethics Institute's summary is the quickest way in.
  • Fairness and Bias in AI: A Brief Survey (MDPI) is a short, broad introduction.
  • Benchmarks to explore directly: Anomify's LLM Bias Benchmarks and Polar.
  • Frameworks: the NIST AI Risk Management Framework and the EU AI Act text.

The Bottom Line

No AI model is bias-free, and no company's promise settles the question. What builds trust is transparency, independent testing, human override and public accountability. Don't take any chatbot's word on its own neutrality, including mine. Check the tests, compare several models, ask your candidates the hard questions, and then vote.




Bias Ratings and Benchmarks (search 1)

📚 Publications on AI Bias (search 2)

⚖️ Grok vs. Claude Comparisons (search 3)

📌 Corrections and Gaps

  • Wrong labels in the article. The Polar benchmark is search 1, source 4. The UPI report is search 3, source 4. The Trakkr comparison is search 3, source 1, not the MIT survey. Use the lists above as the correct mapping.
  • No links retrieved for some items. I didn't search for the following, so I have no verified URLs for them. Search each by name.
    • Named in the article: the NIST AI Risk Management Framework, the EU AI Act, and the European Data Protection Board guidance on AI bias.
    • Named in your pasted text: BOLD, RealToxicityPrompts, CrowS-Pairs, StereoSet and LMSYS Chatbot Arena.
    • The article's claims about other countries' rules and about AI models judging other models came from my general knowledge, not from these sources. Verify them before you cite them.
  • I saw summaries only. For every link above, I read search-result snippets, not the full reports. The exception is Anomify, where the charts didn't load. Open each page to check methodology and publication dates before you cite it.



Posted by at