Anthropic trained their LLMs with copyrighted stuff but distillation is bad. Anthropic has closed models, China releases as open weight. DeepSeek even allows distilling their models, but China = bad. Understood.
Mistral has solid OCR, STT and TTS models and I would love to support them by switching with all of our business workloads to Mistral... but their LLM models are sadly not competitive at all. In our business benchmarks their Mistral Medium 3.5 with reasoning is worse than Gemma 4 31B and Glimmer 30B. It's a 128B dense model that's priced accordingly! Mistral Small 4 is way worse than Gemma 4 26B A4B. I applaude the effort that they release those models as open weight but Gemma 4 models are currently way easier to run with more tok/s and less hardware. Their API pricing is just insane for what you get. But I guess enterprise customers don't care about it, this is why they are probably not lowering it.
> Mistral Small 4 is way worse than Gemma 4 26B A4B
Depends for what purpose? I found large Mistrals are massively better than Gemma 4 at creative writing: have more natural tone, better consistency than 26B as it is MoE.
Mistral Small 4 is also a MoE model, with way more params, so I would expect it to perform better. Our benchmark involves around 10% communication and writing in German and English and Gemma 4 26B A4B beats it although creative writing is the only category in our benchmark which might be a bit subjective.
For me Opus 5 was the nail in the coffin. Fable without the restrictions was a great model, but became unusable with the security guardrails. Opus 5 became so bad and slow it's unbearable. And since Fable falls back to Opus all the time it was time to switch. Me and my friends are calling it Slowpus by now... Up until some weeks ago Anthropic was the king, but we all switched to Grok 4.6. It became slow as well but is not down all the time, is a solid model and paired with another reviewer model it's a great daily driver.
How long have you been testing this? Have you noted a large improvement? I tested Opus for this quite a while ago (maybe 4.5? Whatever was out about a year ago), and it performed quite poorly on my use case.
I have put together an internal benchmark on 1000s of business documents with weird tables, structure, etc. that I run on every relevant model release. Opus 4.8 performs very very well. But it is obviously overkill for the task (and expensive at doing so). I just wanted to respond to the OP.
I'm assuming that the reason I didn't have good success rate is because it was not scanned documents, but photographs, and lighting conditions weren't always ideal. I think scanned business documents are a happy-case scenario in a way. (obv, you seem to run it against some complex documents, so that's impressive)
> (I originally was going to say a computer that plays chess, but computers play chess with no intuition or instinct--they just search a gigantic solution space very quickly.)
Isn't that how LLM models are trained right now? Trying to predict the next word within a "gigantic solution space". Interesting.
Which even shows Sam has no idea about AI, as the best chess engine at that point in time Komodo 8 was trained and developed primarily through the efforts of GM Larry Kaufman and Mark Lefler, focusing on refining the engine's evaluation function and search accuracy rather than relying on deep, brute-force calculation.
In one sense, all intelligence is a search in a gigantic solution space.
But the difference is:
What Deep Blue did was (if the Wikipedia page is correct) Alpha-beta pruning[0], where some humans came up with the function for what "better" and "worse" board states look like.
And what LLMs do (at least the end models) includes at least some steps where there's an AI trying to learn what human preferences are in the first place, in order to maximise the human evaluation scores. Some of those things are good, like "what's the right answer to the trolley problem?" and "which is the better poem?", but some are bad such as "what answer best flatters the ego of the user without any regard for truth?"
The former is exactly like route-finding, in that you could treat travel time as your score of better-worse and the moves as if they're on a map rather than a chess board.
The latter is like being dumped into a new video game with no UI and all NPCs interact with you only in a language you don't know such as North Sentinelese.
It's neither how computer chess works or how LLMs are trained.
Computer chess uses various tricks to prune the search space of board states, where the search is guided by the "value" of each board state. Neural networks can be used (and probably was at the time) to approximate this value, but there can be hand coded algorithms with learned statistics or even lookup tables for smaller games than chess.
In Germany we have several accounting software solutions like that for 5-10+ years that integrate with bank accounts, paypal, etc. - automatically suggests booking accounts and exports it via a REST API to the software accountants use. Accountants have access to it. Is this basically the same as your solution?
I will never use dark mode if I can avoid it. The idea that it's somehow better is a bit shady [0][1] (save for mobile devices where arguably it may help save some energy), but I absolutely understand that it can be a personal preference.
reply