This “cheap” open source model in fact burns your calculation budget

Research has examined 19 different AI models in three tasks categories: basic questions, mathematical problems and logical puzzles. The team has measured “the efficiency of the tokens” – how many models of calculation units use in relation to the complexity of their solutions – a metric which has received little systematic study despite its significant cost implications.

“The effectiveness of the tokens is a critical metric for several practical reasons,” noted the researchers. “Although accommodation of open weight models can be cheaper, this cost advantage could be easily offset if they require more tokens to reason from a given problem.”

The ineffectiveness is particularly pronounced for important reasoning models (LRM), which use extensive “thought chains” to solve complex problems. These models, designed to reflect on step -by -step problems, can consume thousands of tokens reflecting on simple questions that should require a minimum calculation.

For the basic questions of knowledge as “what is the capital of Australia?” The study revealed that the reasoning models pass “hundreds of tokens reflecting on matters of simple knowledge” to which one could answer in one word.

What models have delivered to Bang for your money

Research has revealed striking differences between model suppliers. Openai models, in particular its O4-Mini variants and Open-Source GPT-OS variants, have demonstrated an exceptional token efficiency, especially for mathematical problems. The study revealed that the OpenAi models “are distinguished by the effectiveness of extreme tokens in mathematical problems”, using up to three times less tokens than other commercial models.

Among the Open Source options, Llama-3.3-Nemotron-Super-49B-V1 of Nvidia has become “the most effective open weight model in all areas”, while new models of companies and the masterful showed “exceptionally high use in tokens” as exterior.

The difference in efficiency varied considerably depending on the type of task. While the open models have used about twice as many tokens for mathematical and logical problems, the difference has increased simple knowledge questions where effective reasoning should be useless.

What business leaders must know about IA IT costs

The results have immediate implications for the adoption of corporate AI, where IT costs can evolve rapidly with use. Companies that assess AI models often focus on precision references and prices by Token, but can ignore total IT requirements for real world tasks.

“The best efficiency of chips of closed weight models often compensates for higher API pricing of these models,” noted the researchers during the analysis of total inference costs.

The study has also revealed that closed -source model suppliers seem to actively optimize efficiency. “The closed weight models have been optimized iteratively to use fewer tokens to reduce inference costs”, while open-source models “increased their use of tokens for more recent versions, possibly reflecting a priority towards better reasoning performance”.

How researchers fell in love with the IA effectiveness code

The research team has faced unique challenges in the measurement of efficiency in different model architectures. Many models of closed source do not reveal their gross reasoning processes, providing compressed summaries of their internal calculations to prevent competitors from copying their techniques.

To remedy this, the researchers used completion tokens – the total calculation units billed for each request – as an indirect indicator of the reasoning effort. They discovered that “the most recent closed source models will not share their traces of raw reasoning” and “use smaller language models instead to transcribe the chain of thinking or compressed representations”.

The study methodology included tests with modified versions of well -known problems to minimize the influence of memorized solutions, such as the modification of variables in mathematical competition problems of the American Invitational Mathematics Examination (AIM).

The future of the effectiveness of AI: what will follow

The researchers suggest that the effectiveness of the tokens should become a main optimization target in parallel with the precision for the future development of the model. “A densified bed bed will also allow more effective use of context and can counter the degradation of the context during difficult reasoning tasks,” they wrote.

The release of Openai Open-Source GPT-Ass models, which demonstrate the efficiency of the cutting edge of technology with a “free accessible COT”, could serve as a reference point to optimize other open source models.

The complete search data set and the evaluation code are available on GitHub, allowing other researchers to validate and extend the results. While the AI industry rushes towards more powerful reasoning capacities, this study suggests that real competition may not be on whom can build the most intelligent AI – but which can build the most effective.

After all, in a world where each token counts, the most wasted models can find themselves getting out of the market, whatever they can think.

Daily information on business use cases with VB daily

If you want to impress your boss, VB Daily has covered you. We give you the interior scoop on what companies do with a generative AI, from regulatory changes to practical deployments, so that you can share information for a maximum return on investment.

Read our privacy policy

Thank you for subscribing. Discover more VB newsletters here.

An error occurred.