Rendered at 04:10:03 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
chvid 16 minutes ago [-]
“The fact that a 17GB file can do all of this stuff on my home machines is a miracle. Once again, I’m delighted and amazed at how much progress local models have made this year.”
I think that should be the blinking headline - this shows what can be done with consumer hardware.
jatora 1 hours ago [-]
All current era models overthink as it's a product of their RL incentives (or distillation of models with them...)
From my reading of the Fable 5 and Opus 5 System cards, my reconstruction is something like:
Finish the task → make externally observable evidence that it is finished → check your own work → fix problems → don't stop prematurely → satisfy the evaluator comprehensively.
That is fantastic for SWE benchmarks and autonomous agents. It also naturally creates pathologies:
under-answering is expensive; over-answering is cheap.
dannyw 8 minutes ago [-]
The recent 'Stolen Thoughts'[1] paper shows many excerpts of private reasoning for frontier models.
For a complex maths problem, Sol reasoned in 367 tokens before working:
We need solve. Need interpret no person sits next to two other people = among binary string length16 weight8, no occupied chair whose both neighbors occupied, equivalently ab 111 substring. Endpoints cannot have two neighbors anyway; only avoid 111. Count binary strings length16 weight8 avoiding 111. Need N mod1000. Compute stepwise perhaps runs of 1 length max2. Count via runs.
[... cut in half for HN readability ... ]
Check interpretation potentially "no person sits next to two other people": no seated person's chair adjacent to two occupied chairs. Exactly no three consecutive chairs selected. yes.
Need reason step by step final boxed 907. Explain runs. Ensure people each select chair distinct subset (given subset count). Let's present.
That doesn't look like an overthinker to me, and matches my experiences. There's plenty of papers and research on reducing thinking verbosity/length while keeping as much quality as possible.
I think one of the bigger problems is that verbose, `max`-style thinking does generally lead to higher benchmark scores. And model vendors are incentivised to for benchmarks (at least to some extent).
All these weird partial language thought patterns im surprised none of the teams have taught the models to think in something like court stenography or some very dense pattern (i mean they even tried caveman language at one point)
nojs 46 minutes ago [-]
For a model this small it’s also a way of trading supply-constrained VRAM for inference time, which for self hosting consumers (and probably Chinese companies subject to export controls) is mostly good.
I can imagine a curve where for a given level of “intelligence” you either need model size, or inference time (“test time compute”), and can somewhat trade one for the other.
atif089 5 minutes ago [-]
I believe this is what Meta is doing as they started recording their SWE screens some time ago.
jongjong 6 minutes ago [-]
This is a good way to think about it.
For coding, this is very interesting because the same incentives were present for humans before AI. Tech companies which had a culture of rewarding complexity would see huge Pull Requests and a lot of unnecessary complexity. I've worked in companies which would require a thousand lines of code to implement a feature which would require only a hundred or so lines at a different company. The shorter one was more reliable too. Code begets more code. The incentives created by the company culture had a massive impact... And the culture was heavily determined by whether or not the company had a market monopoly. More monopoly -> more unnecessary complexity (presumably so that employees could achieve better lock-in/job security as a result of being the only one who could understand some critical part of the system or complex integration).
So it's not surprising that the same dynamics are at play with AI. Now, because it's happening so rapidly, the effects have become much more obvious; many senior engineers and CTOs will echo my point; but I suspect most engineers and crucially, most managers, still don't get it...
Something tells me that the AI companies supplying the models are well aware of the tradeoff. When you can dial up the complexity of the LLM's output by 5% and see a 5% immediate increase in your revenue from a large segment of your users, that's a very tempting knob to dial up! Now when you learn that this complexity compounds and next year's revenue will be 10% higher (as a result of your users now having to maintain that additional complexity); this is extremely tempting! Especially in the context of users who are largely ignorant as to the true cost of the unnecessary complexity they are adding... Now factor in the monopolistic tendencies of those tech markets... It's a real gold mine.
Revenue from downstream corporate users comes in regardless of compounding code complexity; those companies just keep hiring more people, spending more on tokens. Enshittification takes place but the downstream end user has nowhere else to go.
jedbrooke 55 minutes ago [-]
I feel like the current “reasoning” that LLMs are doing has got to be a dead end eventually. Every time I have to read another answer with “but wait” and “Actually,” as they “reason” their way to a (sometimes) better answer, I feel like there’s got to be a way to just shortcut to the actual correct answer instead of burning all these token going in circles mimicking actual thought
nsingh2 25 minutes ago [-]
One natural evolution seems to be toward some form of latent-space reasoning, as in [1]. Natural language seems like a relatively low-bandwidth channel for reasoning.
So-called "caveman" thinking attempts to address this.
The important part of "actually wait, I really need to XYZ" is just "XYZ".
The model can attend to just "do XYZ" and produce almost the same vector modifications as full verbose "reasoning".
russfink 53 minutes ago [-]
It “thinks out loud” to populate its token space. Asking it to shortcut risks truncating that process.
jauntywundrkind 48 minutes ago [-]
Yeah. It's "thinking" in absurd massive vectors. It needs to assess a couple to weigh out. That's the compression. That's the nature. It looks ridiculous when thinking traces render out such simple statements ('reassessing ..') but I expect this is far deeper an assessment than it can fully reflect to us on, and I expect its a huge part of their thinking.
NewJazz 47 minutes ago [-]
Hardcode their "thoughts" in your agents.md... But they might still reason through it anyway.
RachelF 2 hours ago [-]
To me, the amazing thing is that we now have local models that rival the reasoning of high end models from about a year ago.
I hope this trend continues.
refactor_master 2 hours ago [-]
Unlike cloud infra in general which offers things like automatic backups, regional redundancy, and effectively unlimited scalability, it seems like the value proposition of cloud LLM gets ever shakier.
* Many businesses don't need frontier level intelligence anyway.
* It's completely stateless. If your local LLM machine catches fire? Nothing was lost. Buy another.
SV_BubbleTime 15 minutes ago [-]
[dead]
xlayn 2 hours ago [-]
I have this branch of llama.cpp that among other things (like patching the template to not break the kv cache, and saving conversations to disk so you can resume quickly days after) also accept the reasoning effort flag here https://github.com/alainnothere/llama.cpp/tree/disk-cache-ev...
I did testing and the reasoning effort can be set per message, I was not aware of the option of none mentioned by @xscott, I tested but didn't see any change, I think there are just 3 values, xhigh, medium and low as per https://huggingface.co/Qwen/Qwen3.8-27B-FP8 , I did testing and the thing can do it's "I'll speak 10 million words to myself to ensure I'm not missing something" and then switch to a faster model, then switch... I did a test and the thing keep coherence and follow it's train of though-kens, you can see the result here... https://github.com/alainnothere/llama.cpp/blob/disk-cache-ev...
Gracana 1 hours ago [-]
What’s that about the template breaking the kv cache?
xlayn 1 hours ago [-]
this is my understanding, the default template keeps the thinking part but only for the last message, so the harness has to play along with the template and strip and add to keep the conversation matching what's there on the llama.cpp cache, but if the harness sends the thinking in every turn, then you break what llama.cpp expects, the conversation doesn't match anymore what you have on cache and it reprocesses again the whole conversation
Gracana 1 hours ago [-]
Oh, interesting. I want to say there’s a “preserve thinking” option in mainline llama.cpp, but I’m seeing mixed results in my searches and my in-flight internet is not good enough to dig into it properly. Good to hear you have a solution though, because that is likely something I’ll run into as well.
hedgehog 57 minutes ago [-]
Look for improved templates by "froggeric" on Hugging Face. I use Qwen 3.6 a fair amount using his template and it fixes some issues I saw with the upstream versions.
johnnyApplePRNG 1 hours ago [-]
According to the paper "Stealing reasoning traces from proprietary llms" [0] all frontier models overthink.
Thinking is good.
You just don't see it in proprietary harnesses because it's literally cryptographically hidden from you.
It won't satisfy the people who just want to drop a model into their existing toolset and run, but I think there are a lot of ways to deal with this overthinking problem.
For instance, it's a step backward, but I put {"reasoning_effort":"none"} and led it by the nose:
User: We're going to make <silly demo>. Please create a plan, but do not write code yet.
Agent: <short and reasonable plan>
User: Now please follow that plan and write the code. No other chat.
Agent: <reasonable code in reasonable time>
Maybe this can be fixed with Jinja templates or something, or maybe it's a hack to your harness, but it shows you can get the model to reason reasonably.
hedgehog 53 minutes ago [-]
To be fair a lot of models have quirks, I've never found a model swap that was transparent.
adam_arthur 1 hours ago [-]
Yes, if you set reasoning to none you can force the granularity of the thinking.
It will actually adhere to your request for e.g. 3 sentences max.
Thinking mode will override any instructions in the prompt (at least for other models in my experience).
Of course this will probably hurt performance, but works great for easy tasks that you know are trivial. Tons of pipeline, image recognition etc use cases where this works well.
I'd be curious to see Qwen 3.8 27B low thinking benchmarks though.
andy99 4 hours ago [-]
The big problem with overthinking on a dense model is obviously the speed hit you take. Going from Qwen 35BA3B to 27B for me is about 7-8x slower (should be ~9x?). This makes me a lot less patient for useless thinking tokens.
I’d want to compare this to the new Muse 30B model which is super terse and has a whole different way of thinking (no “Wait,”) and in my experiments was way more token efficient to the point that the absolute tok / s didn’t really matter.
simonw 3 hours ago [-]
Comparing with Muse Glimmer is a good idea. I ran the same exact HTML tool generating prompt against both Glimmer 30B and Qwen 3.8 27B. Results:
Both applications work correctly and fulfill the requirements. The Qwen one (which used the default xhigh reasoning setting) is massively over-engineered. The Glimmer one used whatever their default in LM Studio is and I would argue is a tiny bit under-engineered.
Weirdly the Glimmer one doesn't work with images on other domains like https://static.inaturalist.org/photos/714731804/large.jpg - it fails with a CORS error, but you don't need CORS to load images and detect their width and height, and the Qwen one handles that URL just fine.
That's because Glimmer added this unnecessary line:
img.crossOrigin = 'anonymous';
NitpickLawyer 45 minutes ago [-]
Yesterday I tried both as well. I do a quick "explain this repo" + "any security issues" convo to do a "vibe check" on the models and make sure everything works w/ serving and harness. Both qwen and glimmer explained it pretty well, and both accepted the security question without any issues, flagged a few things left there on purpose (hardcoded tokens, single auth, no logs, etc).
I like the style of glimmer more. Much terser language, no adjectives, no fluffy claude-like language. ("Images are written to...", "Tasks are stored in SQLite...", "Docker image is built from ...")
In contrast, qwen is a bit more flowery. ("Unbounded image processing / resource exhaustion — preprocess() opens whatever was downloaded with no size/dimension/format validation before the VAE encodes it..." , "SQLite as a queue — fine at this scale, but...", "Debug info leakage — exceptions are re-raised as...".
But both flagged pretty much the same stuff, just ordered / styled differently. Mighty impressive understanding for a thing that I can run locally. Qwen served in fp8 w/ full kv cache, glimmer in w4a16 (the fp8 weights wouldn't serve for whatever reason), both at full supported context in 48GB of VRAM.
bogzz 3 hours ago [-]
I love reading Glimmer's "thoughts". Why use many word when few do trick?
bblb 40 minutes ago [-]
Me machine, no human. Why waste token.
Do fast, deliver.
Gracana 1 hours ago [-]
I’ve noticed dsv4 do that as well, but inconsistently. I thought it was broken at first, but no, it’s just kind of shorthand that it does while thinking.
dofm 2 hours ago [-]
It’s also a little bit snarky, almost. The stuff it thought during the car wash puzzle made me laugh.
lostmsu 2 hours ago [-]
Glimmer is stupider than 3.6 27B. You can't compare its speed to 3.8 and be done.
atif089 7 minutes ago [-]
So if I have to set this up on my 24GB MBP what'd the right configuration and tuning look like?
nharziro 2 hours ago [-]
I do agree that Qwen 3.8 27B is excellent but slow and very token inefficient. My benchmark places it near opus 4.6 and codex 5.3 performance. 3.6 27B couldn't even complete the benchmark. Please see below for details:
Opus 4.6 performance with a local model that can be hosted on consumer hardware is an incredible result!!
doginasuit 2 hours ago [-]
To be fair, Opus 5 overthinks things on a regular basis. I interact with the LLM almost entirely through the prompt interface vs. some agentic harness, so I have a lot of granular exposure to its reasoning. For almost every code analysis, it flags all the important issues and at least one non-issue. It suggests some impractical and unnecessary fix for the non-issue that would categorically be a regression.
I've learned that medium effort can improve the outcome relative to higher settings. But I suspect the phenomenon is an artifact of a misguided effort to fix inherent LLM limitations. At least some of its reasoning will miss the target, and more bad reasoning is not the remedy.
SwellJoe 3 hours ago [-]
This is true, but I think it understates the problem. I did a task I've done with a bunch of small models lately (https://github.com/swelljoe/flar/pull/17), and it did an excellent job, the best of any self-hostable model. But, it took eleven (11!) hours on my dual GPU setup. It really chewed on it, and spent a lot of time checking and re-checking. It is by far the slowest model I've used for the task. GPT 5.5 did a similar task in about 20 minutes. Most big models took about an hour or so, and most small models needed a couple of hours (but did a worse job).
simonw 3 hours ago [-]
Was that with the default xhigh reasoning setting? I suggest trying again with reasoning set to low or turned off entirely.
dofm 1 hours ago [-]
You now have me testing it with reasoning turned off, which I have never bothered much with on any other local models because it's rarely worth it.
The result appears to be almost as good as Qwen 3.6 35B A3B on medium thinking mode.
It second-guesses a little, it gives broader/more speculative answers, of course, and it missed the nuance of one of my prompts, but this gives me a lot more confidence that the Low reasoning effort is going to be as good as they say, and perhaps in some cases non-thinking looks like it would be enough.
Really useful, thanks.
anon373839 16 minutes ago [-]
> Qwen 3.6 35B A3B on medium thinking mode
Qwen 3.6 doesn’t have configurable reasoning effort, does it?
SwellJoe 2 hours ago [-]
Yes, default everything, no tuning, 8_K_XL Unsloth quantization on dual Radeon V620 GPUs (which aren't blazing, but faster than the Strix Halo).
syntaxing 2 hours ago [-]
What speed do you get on this setup? Im tempted to use the same GPU.
SwellJoe 31 minutes ago [-]
It varies widely based on a bunch of factors. With this specific model at 8-bit quantization and MTP, it starts out at about 25 t/s for basic chat, but for agentic tasks with long context it slows way down to something like 12-15. I don't see a big difference in token rates based on any config changes I have tried, or going to the smaller 6-bit quantization, so far, though I haven't spent a ton of time on experimenting.
If you already have one or more of them, then, yeah, you can use them for this model or any other at around this size or smaller, but I wouldn't recommend you buy them (or anything else, right now, everything is just too overpriced). You can run better models for less money at higher speeds. I bought mine before they got more expensive, but I wish I'd just bit the bullet and bought newer/faster cards before they got more overpriced. Or, the actual smart money, even back then was to just use cloud models and forget about self-hosting.
fermuch 2 hours ago [-]
xhigh tells it to overthink and re check everything. Low tells it to only do the minimum thinking necessary. I would suggest to give qwen medium which doesn't inject any thinking directives into it and also to give as much context as you can, ideally around 500k tokens or even 1M if you can. Big complex tasks like these make the model hit the compaction trigger a lot and they end up re thinking the same thing several times in my experience.
kennywinker 13 minutes ago [-]
Doesn’t it max out its context at like 256k?
teravor 35 minutes ago [-]
when you distill a thinking LLM past its capacity it will default to overthinking because during training that was the only way for a chance at a reward on many tasks.
you can generally avoid this if you specialize it on a domain that is within its capacity.
matheusmoreira 45 minutes ago [-]
Am I the only one who enjoys it when LLMs overthink everything?
Opus 4.8 would spend like 10 minutes thinking and then go out there and do an excellent job. Only Fable 5 seems to be smart enough to just know everything it needs to immediately start working without any reasoning or verification. Opus 5 tries to be relentless like Fable, but it's not as smart as Fable and I have to constantly challenge and correct its unfounded assumptions. Sol is somewhere between Fable and Opus 5, it's smart but it's not Fable, it keeps making assumptions that I have to correct.
After trying all these models, I find that I miss Opus 4.8's overthinking. Sure it's slow, but it actually gets things right.
blagui 2 hours ago [-]
You have 4 thinking levels.
You can disable it. It's well known issue in Qwen, previous releases I would disable it by default.
Also xhigh seem a new thing.
dofm 2 hours ago [-]
Yes. Though the chat template doesn’t tell LM Studio to offer the little dropdown. You can bodge the template in the load parameters.
Unsloth Studio / Desktop has it working really well with their version of the weights.
jakswa 1 hours ago [-]
I went back to Glimmer 30b for my 20GB of VRAM. Just a better experience fit-wise and speed-wise and tone-/voice-wise.
cyanydeez 3 hours ago [-]
--thinking-budget and --thinking-message is all you need in llamacpp to keep it progressing.
the message can be some combination of tool calling, summarizing, etc. It's overthinking often is a bunch of recursion, so simply stopping t and redirecting is all you need to do.
If someones building a harness for llamacpp, you can set this per message, so it's possible to dynamically control it by watching for the expansion of the thinking traces, and redirecting it.
I use the message to tell it to use subagents, add additional logging and to use opencode's dynamic context pruning.
As such, we'll just whisper here _skill issue_.
dofm 2 hours ago [-]
Unfortunately in xhigh thinking it goes down rabbit holes in such an extreme depth-first way, that whenever you choose to cut it off, there is a very good chance it will not have got round to musing on even half of the prompt! It doesn’t really obviously loop in xhigh, so I am not sure if an “overthinking guard” proxy would have much to go on, but it does obsessively ruminate on edge cases. I have seen it overcomplicate simple code as a result even in my limited testing.
Probably the better solution if you want it to be quicker but still fairly thorough appears to be to configure reasoning effort instead of thinking budget. It seems to do very well still even on the Low setting; on the Medium setting it can get stuck in loops like 3.6 does.
I think xhigh reasoning effort was an absurd choice for a default, and so was not sorting out the chat template so LM Studio could offer the reasoning effort dropdown.
cyanydeez 2 hours ago [-]
to the point though: most of that overthinking is useless if you have a proper redirect message. So setting arbitrary budget and getting it a good message will do the trick regardless of what type of thinking it's doing. The reason thinking seems to work is that it's just trying to find an optimum outside the local optimum, and the thinking trace helps find it.
The only think I could think that'd be better than the --reasoning-budget would bet a budget jitter just in case it really is repeating a pattern and you want to escape it arbitrarily, otherwise yes, it could keep looping if you're always cutting at the wrong time.
dofm 2 hours ago [-]
> The reason thinking seems to work is that it's just trying to find an optimum outside the local optimum, and the thinking trace helps find it.
Yes, I think I finally have an intuitive sense for that. But surely on a longer prompt it is still better for the final response if the thinking has at least brushed past all of the prompt?
One of the things I witnessed with xhigh is that while the thinking trace starts out intending an overview of the prompt, it actually can go fully down a rabbit hole off one of the first two or three bullet points even when it was seemingly intending not to.
It’s basically a lot like me. Gets sidetracked by the interesting bits.
bitexploder 2 hours ago [-]
Yeah, but be fair. Working with small models is a different ball game. Not all the batteries come included :)
bellowsgulch 2 hours ago [-]
This is definitely such a cool feature that I wish cloud providers would expose.
deadcatfound 3 hours ago [-]
For agents, token efficiency is an operating cost. I’d rather have a terse model that escalates hard cases than one that overthinks every tool call.
kamranjon 2 hours ago [-]
A no-thinking pelican! I hope to see more, it's surprisingly good for just 2 minutes.
2 hours ago [-]
1 hours ago [-]
LoganDark 2 hours ago [-]
I hope Apple does end up moving to HBM. Unified memory has been a huge godsend, but the low memory bandwidth is just such a killer. Even/especially on M5, where the available compute is starting to starve incredibly badly on ML workloads.
dofm 2 hours ago [-]
AFAIK that is initially only for the iPhone?
LoganDark 60 minutes ago [-]
Apple is reportedly considering skipping the higher-end M6 chips altogether, which could potentially give enough time for higher-end M7 (in over a year) to use HBM
kennywinker 10 minutes ago [-]
Considering, or being forced to consider due to the hardware crunch?
javchz 3 hours ago [-]
I wonder if this can be fixed with LORAs.
bitexploder 2 hours ago [-]
I had to fix this on 35B A3B -- I have a proxy that just shuts it down if it gets to 2K thinking tokens and injects something like "We have thought enough, let's begin working." and it almost always finishes the turn then. It rarely needs more than 2K thinking tokens and if it does there is always next turn. I would need to see what 27B is actually doing, but these smaller Qwen models seem prone to this.
dofm 2 hours ago [-]
Unfortunately in xhigh reasoning effort it will burn through 2K tokens before it has even finished its bullet point overview. It really is intense and obsessive. You might need ten times more!
Your strategy would likely help in medium reasoning effort (because there it gets caught up in the very typical Qwen looping).
Not seen looping in the “low” reasoning effort mode.
bitexploder 57 minutes ago [-]
I have been using Muse Glimmer for a few days instead of A3B. It gets the job done quicker than A3B despite being several times slower.
logicallee 2 hours ago [-]
>I had to fix this on 35B A3B -- I have a proxy that just shuts it down if it gets to 2K thinking tokens and injects something like "We have thought enough, let's begin working." and it almost always finishes the turn then.
I think that should be the blinking headline - this shows what can be done with consumer hardware.
From my reading of the Fable 5 and Opus 5 System cards, my reconstruction is something like:
Finish the task → make externally observable evidence that it is finished → check your own work → fix problems → don't stop prematurely → satisfy the evaluator comprehensively.
That is fantastic for SWE benchmarks and autonomous agents. It also naturally creates pathologies:
under-answering is expensive; over-answering is cheap.
For a complex maths problem, Sol reasoned in 367 tokens before working:
That doesn't look like an overthinker to me, and matches my experiences. There's plenty of papers and research on reducing thinking verbosity/length while keeping as much quality as possible.I think one of the bigger problems is that verbose, `max`-style thinking does generally lead to higher benchmark scores. And model vendors are incentivised to for benchmarks (at least to some extent).
[1] https://stolen-thoughts.com/
I can imagine a curve where for a given level of “intelligence” you either need model size, or inference time (“test time compute”), and can somewhat trade one for the other.
For coding, this is very interesting because the same incentives were present for humans before AI. Tech companies which had a culture of rewarding complexity would see huge Pull Requests and a lot of unnecessary complexity. I've worked in companies which would require a thousand lines of code to implement a feature which would require only a hundred or so lines at a different company. The shorter one was more reliable too. Code begets more code. The incentives created by the company culture had a massive impact... And the culture was heavily determined by whether or not the company had a market monopoly. More monopoly -> more unnecessary complexity (presumably so that employees could achieve better lock-in/job security as a result of being the only one who could understand some critical part of the system or complex integration).
So it's not surprising that the same dynamics are at play with AI. Now, because it's happening so rapidly, the effects have become much more obvious; many senior engineers and CTOs will echo my point; but I suspect most engineers and crucially, most managers, still don't get it...
Something tells me that the AI companies supplying the models are well aware of the tradeoff. When you can dial up the complexity of the LLM's output by 5% and see a 5% immediate increase in your revenue from a large segment of your users, that's a very tempting knob to dial up! Now when you learn that this complexity compounds and next year's revenue will be 10% higher (as a result of your users now having to maintain that additional complexity); this is extremely tempting! Especially in the context of users who are largely ignorant as to the true cost of the unnecessary complexity they are adding... Now factor in the monopolistic tendencies of those tech markets... It's a real gold mine.
Revenue from downstream corporate users comes in regardless of compounding code complexity; those companies just keep hiring more people, spending more on tokens. Enshittification takes place but the downstream end user has nowhere else to go.
[1] https://github.com/sapientinc/HRM-Text
The important part of "actually wait, I really need to XYZ" is just "XYZ".
The model can attend to just "do XYZ" and produce almost the same vector modifications as full verbose "reasoning".
I hope this trend continues.
* Many businesses don't need frontier level intelligence anyway.
* It's completely stateless. If your local LLM machine catches fire? Nothing was lost. Buy another.
I did testing and the reasoning effort can be set per message, I was not aware of the option of none mentioned by @xscott, I tested but didn't see any change, I think there are just 3 values, xhigh, medium and low as per https://huggingface.co/Qwen/Qwen3.8-27B-FP8 , I did testing and the thing can do it's "I'll speak 10 million words to myself to ensure I'm not missing something" and then switch to a faster model, then switch... I did a test and the thing keep coherence and follow it's train of though-kens, you can see the result here... https://github.com/alainnothere/llama.cpp/blob/disk-cache-ev...
Thinking is good.
You just don't see it in proprietary harnesses because it's literally cryptographically hidden from you.
[0] https://arxiv.org/pdf/2608.09867
For instance, it's a step backward, but I put {"reasoning_effort":"none"} and led it by the nose:
Maybe this can be fixed with Jinja templates or something, or maybe it's a hack to your harness, but it shows you can get the model to reason reasonably.It will actually adhere to your request for e.g. 3 sentences max.
Thinking mode will override any instructions in the prompt (at least for other models in my experience).
Of course this will probably hurt performance, but works great for easy tasks that you know are trivial. Tons of pipeline, image recognition etc use cases where this works well.
I'd be curious to see Qwen 3.8 27B low thinking benchmarks though.
I’d want to compare this to the new Muse 30B model which is super terse and has a whole different way of thinking (no “Wait,”) and in my experiments was way more token efficient to the point that the absolute tok / s didn’t really matter.
Qwen: https://gist.github.com/simonw/121ad098860028b2fab603fa12da1... - 17,576 reasoning tokens, produced this HTML result: https://static.simonwillison.net/static/2026/qwen-over-think...
Glimmer: https://gist.github.com/simonw/51e8ddb2ee597a5005fa63bd4927d... 1,021 reasoning tokens, this HTML: https://static.simonwillison.net/static/2026/glimmer-bbox.ht... - ugly but functional.
In both cases paste in the URL https://static.simonwillison.net/static/2026/two-pelicans-on... to see them work.
Both applications work correctly and fulfill the requirements. The Qwen one (which used the default xhigh reasoning setting) is massively over-engineered. The Glimmer one used whatever their default in LM Studio is and I would argue is a tiny bit under-engineered.
Weirdly the Glimmer one doesn't work with images on other domains like https://static.inaturalist.org/photos/714731804/large.jpg - it fails with a CORS error, but you don't need CORS to load images and detect their width and height, and the Qwen one handles that URL just fine.
That's because Glimmer added this unnecessary line:
I like the style of glimmer more. Much terser language, no adjectives, no fluffy claude-like language. ("Images are written to...", "Tasks are stored in SQLite...", "Docker image is built from ...")
In contrast, qwen is a bit more flowery. ("Unbounded image processing / resource exhaustion — preprocess() opens whatever was downloaded with no size/dimension/format validation before the VAE encodes it..." , "SQLite as a queue — fine at this scale, but...", "Debug info leakage — exceptions are re-raised as...".
But both flagged pretty much the same stuff, just ordered / styled differently. Mighty impressive understanding for a thing that I can run locally. Qwen served in fp8 w/ full kv cache, glimmer in w4a16 (the fp8 weights wouldn't serve for whatever reason), both at full supported context in 48GB of VRAM.
Do fast, deliver.
https://gist.github.com/nharziro/aed0c364ce2f295a493494c6f1b...
I've learned that medium effort can improve the outcome relative to higher settings. But I suspect the phenomenon is an artifact of a misguided effort to fix inherent LLM limitations. At least some of its reasoning will miss the target, and more bad reasoning is not the remedy.
The result appears to be almost as good as Qwen 3.6 35B A3B on medium thinking mode.
It second-guesses a little, it gives broader/more speculative answers, of course, and it missed the nuance of one of my prompts, but this gives me a lot more confidence that the Low reasoning effort is going to be as good as they say, and perhaps in some cases non-thinking looks like it would be enough.
Really useful, thanks.
Qwen 3.6 doesn’t have configurable reasoning effort, does it?
If you already have one or more of them, then, yeah, you can use them for this model or any other at around this size or smaller, but I wouldn't recommend you buy them (or anything else, right now, everything is just too overpriced). You can run better models for less money at higher speeds. I bought mine before they got more expensive, but I wish I'd just bit the bullet and bought newer/faster cards before they got more overpriced. Or, the actual smart money, even back then was to just use cloud models and forget about self-hosting.
you can generally avoid this if you specialize it on a domain that is within its capacity.
Opus 4.8 would spend like 10 minutes thinking and then go out there and do an excellent job. Only Fable 5 seems to be smart enough to just know everything it needs to immediately start working without any reasoning or verification. Opus 5 tries to be relentless like Fable, but it's not as smart as Fable and I have to constantly challenge and correct its unfounded assumptions. Sol is somewhere between Fable and Opus 5, it's smart but it's not Fable, it keeps making assumptions that I have to correct.
After trying all these models, I find that I miss Opus 4.8's overthinking. Sure it's slow, but it actually gets things right.
You can disable it. It's well known issue in Qwen, previous releases I would disable it by default.
Also xhigh seem a new thing.
Unsloth Studio / Desktop has it working really well with their version of the weights.
the message can be some combination of tool calling, summarizing, etc. It's overthinking often is a bunch of recursion, so simply stopping t and redirecting is all you need to do.
If someones building a harness for llamacpp, you can set this per message, so it's possible to dynamically control it by watching for the expansion of the thinking traces, and redirecting it.
I use the message to tell it to use subagents, add additional logging and to use opencode's dynamic context pruning.
As such, we'll just whisper here _skill issue_.
Probably the better solution if you want it to be quicker but still fairly thorough appears to be to configure reasoning effort instead of thinking budget. It seems to do very well still even on the Low setting; on the Medium setting it can get stuck in loops like 3.6 does.
I think xhigh reasoning effort was an absurd choice for a default, and so was not sorting out the chat template so LM Studio could offer the reasoning effort dropdown.
The only think I could think that'd be better than the --reasoning-budget would bet a budget jitter just in case it really is repeating a pattern and you want to escape it arbitrarily, otherwise yes, it could keep looping if you're always cutting at the wrong time.
Yes, I think I finally have an intuitive sense for that. But surely on a longer prompt it is still better for the final response if the thinking has at least brushed past all of the prompt?
One of the things I witnessed with xhigh is that while the thinking trace starts out intending an overview of the prompt, it actually can go fully down a rabbit hole off one of the first two or three bullet points even when it was seemingly intending not to.
It’s basically a lot like me. Gets sidetracked by the interesting bits.
Your strategy would likely help in medium reasoning effort (because there it gets caught up in the very typical Qwen looping).
Not seen looping in the “low” reasoning effort mode.
that is amazing, thanks for sharing.