Close Menu
World Forbes – Business, Tech, AI & Global Insights
  • Home
  • AI
  • Billionaires
  • Business
  • Cybersecurity
  • Education
    • Innovation
  • Money
  • Small Business
  • Sports
  • Trump
What's Hot

Trump lashes out at ‘crazy’ Putin after deadly Russian air raids on Ukraine | Donald Trump News

May 26, 2025

Endurance swimmer closes in circumnavigation of Martha’s Vineyard ahead of ‘Jaws’ 50th

May 26, 2025

Your guide to the national spelling bee

May 26, 2025
Facebook X (Twitter) Instagram
Trending
  • Trump lashes out at ‘crazy’ Putin after deadly Russian air raids on Ukraine | Donald Trump News
  • Endurance swimmer closes in circumnavigation of Martha’s Vineyard ahead of ‘Jaws’ 50th
  • Your guide to the national spelling bee
  • Powell defends Federal Reserve in Princeton speech amid onslaught of attacks from Trump
  • What’s open and closed on Memorial Day?
  • Strauss’ ‘Blue Danube’ waltz is launching into space to mark his 200th birthday
  • Foreign universities hope to lure scientists from the US after Trump research cuts
  • Meet Hercules and Ned, the border collies fending off wildlife at West Virginia’s busiest airport
World Forbes – Business, Tech, AI & Global InsightsWorld Forbes – Business, Tech, AI & Global Insights
Monday, May 26
  • Home
  • AI
  • Billionaires
  • Business
  • Cybersecurity
  • Education
    • Innovation
  • Money
  • Small Business
  • Sports
  • Trump
World Forbes – Business, Tech, AI & Global Insights
Home » OpenAI’s o3 AI model scores lower on a benchmark than the company initially implied
AI

OpenAI’s o3 AI model scores lower on a benchmark than the company initially implied

adminBy adminApril 20, 2025No Comments3 Mins Read
Facebook Twitter Pinterest LinkedIn Tumblr WhatsApp Telegram Email
Share
Facebook Twitter LinkedIn Pinterest Email
Post Views: 34


A discrepancy between first- and third-party benchmark results for OpenAI’s o3 AI model is raising questions about the company’s transparency and model testing practices.

When OpenAI unveiled o3 in December, the company claimed the model could answer just over a fourth of questions on FrontierMath, a challenging set of math problems. That score blew the competition away — the next-best model managed to answer only around 2% of FrontierMath problems correctly.

“Today, all offerings out there have less than 2% [on FrontierMath],” Mark Chen, chief research officer at OpenAI, said during a livestream. “We’re seeing [internally], with o3 in aggressive test-time compute settings, we’re able to get over 25%.”

As it turns out, that figure was likely an upper bound, achieved by a version of o3 with more computing behind it than the model OpenAI publicly launched last week.

Epoch AI, the research institute behind FrontierMath, released results of its independent benchmark tests of o3 on Friday. Epoch found that o3 scored around 10%, well below OpenAI’s highest claimed score.

OpenAI has released o3, their highly anticipated reasoning model, along with o4-mini, a smaller and cheaper model that succeeds o3-mini.

We evaluated the new models on our suite of math and science benchmarks. Results in thread! pic.twitter.com/5gbtzkEy1B

— Epoch AI (@EpochAIResearch) April 18, 2025

That doesn’t mean OpenAI lied, per se. The benchmark results the company published in December show a lower-bound score that matches the score Epoch observed. Epoch also noted its testing setup likely differs from OpenAI’s, and that it used an updated release of FrontierMath for its evaluations.

“The difference between our results and OpenAI’s might be due to OpenAI evaluating with a more powerful internal scaffold, using more test-time [computing], or because those results were run on a different subset of FrontierMath (the 180 problems in frontiermath-2024-11-26 vs the 290 problems in frontiermath-2025-02-28-private),” wrote Epoch.

According to a post on X from the ARC Prize Foundation, an organization that tested a pre-release version of o3, the public o3 model “is a different model […] tuned for chat/product use,” corroborating Epoch’s report.

“All released o3 compute tiers are smaller than the version we [benchmarked],” wrote ARC Prize. Generally speaking, bigger compute tiers can be expected to achieve better benchmark scores.

Granted, the fact that the public release of o3 falls short of OpenAI’s testing promises is a bit of a moot point, since the company’s o3-mini-high and o4-mini models outperform o3 on FrontierMath, and OpenAI plans to debut a more powerful o3 variant, o3-pro, in the coming weeks.

It is, however, another reminder that AI benchmarks are best not taken at face value — particularly when the source is a company with services to sell.

Benchmarking “controversies” are becoming a common occurrence in the AI industry as vendors race to capture headlines and mindshare with new models.

In January, Epoch was criticized for waiting to disclose funding from OpenAI until after the company announced o3. Many academics who contributed to FrontierMath weren’t informed of OpenAI’s involvement until it was made public.

More recently, Elon Musk’s xAI was accused of publishing misleading benchmark charts for its latest AI model, Grok 3. Just this month, Meta admitted to touting benchmark scores for a version of a model that differed from the one the company made available to developers.





Source link

Share. Facebook Twitter Pinterest LinkedIn Tumblr Email
admin
  • Website

Related Posts

After Klarna, Zoom’s CEO also uses an AI avatar on quarterly call

May 23, 2025

Anthropic CEO claims AI models hallucinate less than humans

May 22, 2025

Anthropic’s latest flagship AI sure seems to love using the ‘cyclone’ emoji

May 22, 2025

A safety institute advised against releasing an early version of Anthropic’s Claude Opus 4 AI model

May 22, 2025

Anthropic’s new AI model turns to blackmail when engineers try to take it offline

May 22, 2025

Meta adds another 650 MW of solar power to its AI push

May 22, 2025
Add A Comment
Leave A Reply Cancel Reply

Don't Miss
Billionaires

Apple Design Guru Jony Ive To Become A Billionaire Thanks To OpenAI

May 23, 2025

Longtime Apple designer Jony Ive holds more than 12,000 patents related to the user interface…

This Puzzling Metaverse Company Just Renamed Itself Napster

May 22, 2025

This Trump Building Appears To Be Deeply Underwater

May 21, 2025

Elon Musk Will Stay Tesla CEO For Next Five Years And Cut Political Spending

May 20, 2025
Our Picks

Trump lashes out at ‘crazy’ Putin after deadly Russian air raids on Ukraine | Donald Trump News

May 26, 2025

Endurance swimmer closes in circumnavigation of Martha’s Vineyard ahead of ‘Jaws’ 50th

May 26, 2025

Your guide to the national spelling bee

May 26, 2025

Powell defends Federal Reserve in Princeton speech amid onslaught of attacks from Trump

May 25, 2025

Subscribe to Updates

Subscribe to our newsletter and never miss our latest news

Subscribe my Newsletter for New Posts & tips Let's stay updated!

About Us
About Us

Welcome to World-Forbes.com
At World-Forbes.com, we bring you the latest insights, trends, and analysis across various industries, empowering our readers with valuable knowledge. Our platform is dedicated to covering a wide range of topics, including sports, small business, business, technology, AI, cybersecurity, and lifestyle.

Our Picks

After Klarna, Zoom’s CEO also uses an AI avatar on quarterly call

May 23, 2025

Anthropic CEO claims AI models hallucinate less than humans

May 22, 2025

Anthropic’s latest flagship AI sure seems to love using the ‘cyclone’ emoji

May 22, 2025

Subscribe to Updates

Subscribe to our newsletter and never miss our latest news

Subscribe my Newsletter for New Posts & tips Let's stay updated!

Facebook X (Twitter) Instagram Pinterest
  • Home
  • About Us
  • Advertise With Us
  • Contact Us
  • DMCA Policy
  • Privacy Policy
  • Terms & Conditions
© 2025 world-forbes. Designed by world-forbes.

Type above and press Enter to search. Press Esc to cancel.