Wolfram Launches LLM Benchmark on Wolfram Language

🤖 Wolfram Launches LLM Benchmark on Wolfram Language

The project evaluates the ability of models to generate code, splitting the evaluation into syntactic correctness and functional accuracy in solving tasks. The leaders in accuracy were Anthropic's Claude Fable 5 and Claude Opus 4.7, as well as Google's Gemini 3 Pro.

🌍 The benchmark reveals a gap between correct syntax and the actual ability of models to solve complex logical problems, which is critical for the development of reasoning methods.

👤 Even advanced models cannot yet handle complex programming 100% of the time, so the choice of neural network now heavily depends on the selected operating mode (thinking/reasoning).

Source 1: https://www.wolfram.com/llm-benchmarking-project/