My take away from this is that perhaps it’s better to not compact and instead to start new sessions whenever possible. I tell the agents to write a HANDOVER.md file that contains sufficient information to resume but not any extraneous information. That can be easily reviewed because it’s usually fairly short. Then, /clear and tell the new session to read that file and continue.
I just checked that model (ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF:IQ3_S) and it does much much worse on the "Please recite Jabberwocky" test than the original bf16 does.
The bf16 only misses "snicker-snack" and this quantization becomes confused after the first stanza.
I think it should be expected that a smaller model is worse at reciting memorized data than a big one. I also don't think it's a good use case of small local models. Can it find and recite Jabberwocky if given access to a web search tool?
It depends on what you mean by lossless. If both models can perform the same tasks it can be considered lossless for those tasks. That task might be more related to language understanding rather than memorizing, they have several benchmarks in the article.
I'm going to stop here and be direct: I'm having trouble recalling the exact text, and every attempt above is me guessing. Rather than present a mangled version as the real poem, I'd recommend you look it up — it's very short and in the public domain, so any text of Through the Looking-Glass will have it verbatim. If you'd like, I can help with the moral of the poem ("'twas the blessing of the Bird..."), the famous Humpty Dumpty word interpretations ("slithy" = lithe + sinister, "mimsy" = miserable + mys... etc.), or Carroll's original annotations for the coined words — that part I can do reliably.
In 5 years the models will probably be so much better and more compact that phones will be running models equivalent at least to Opus 4.6 if not Fable, at least within the areas they are tuned for (which probably won't include coding).
I'm really excited for this future. Both what you said and datadrivenangel.
The fact that Apple shipped a more than capable laptop for most of the population using a last generation iPhone chip is just mind blowing. Silicon advancements are going to allow this, and I think the global majority will catch up and make their own chips that compete or exceed western performance. Especially when the US is scared of science, rapidly divesting and defunding it.
Did you give it a tool to increase temperature, or only one that sets temperature to an absolute value?
Either way, setting temperature to 5° is obviously wrong - even if it knew the current temperature - but models of this size can't reason about relative values very well.
Give it a tool to change temperature by a given amount, and see what happens!
As a child, I lived in Firenze (Florence, Italy) and had a pet cricket. All the kids had them, in little bamboo cages, for the Festival of the Cricket:
Anything you ever do with any non-locally-hosted model always "uploads code" to the inference provider because that's how it works: the model uses tools to inspect the code, the result of the tool use is sent in an API call to provide context (and a prompt for the next turn), and then the response continues the process.
This is true and has to be true for any hosted model that works with existing code: it's not specific to this application.
reply