A lot of this is over my head but why would you do compression when GPU time is the most expensive thing in the world right now?
KV can be trivially stored on ram or even a spinning disk and retrieved on the order of milliseconds. See LM cache for vLLM for example. In fact it’s so easy it kinda shocks me when Claude Code will sit and recompute my entire KV on a new session after a couple of hours, I guess Anthropic infra is not as optimized as it would seem.
Think about the problem from first principles:
Storing a few GB per user at scale isn’t that hard and was solved years ago. Let’s say I have 20 chat sessions open and the session persists for a day or two, this seems negligible to me as a systems design problem.
I created a patch for llama.cpp to store on disk instead of deleting the kv cache as well as the checkpoints... there is this bug on llama.cpp if you have more than one instance going on of chats... and that causes the kv cache to be lost between changes of chat...
And I can tell you, using Qwen3.627B after one day of use you can have 120-200Gb of chats on disk.
And yes it's way way faster, even if you get it from a spinning disk it's still faster than re-computing the whole thing...
I guess for a 300B parameter or more and couple million users with the price of storage increasing as part of ramagedon this is also not viable...
Qwen 27B maxes out at a 16GB context. A nice thing about DeepSeek V4, especially Flash, is that its context size stays tiny even at 1M tokens! Which in turn opens up wide batching on common consumer platforms.
You need wider batches to get effective reuse of experts in any given layer, but you absolutely can. DeepSeek V4 has tiny KV caches that make this quite feasible. When targeting consumer platforms that only have a limited amount of compute headroom to begin with, the approach is quite reasonable.
What do you mean by doing the math? If you repeatedly sample n_active experts out of n_total, why wouldn't you expect to get some meaningful probability of reuse/overlap once your batch grows past size 5 or so (for the sparsest MoE models in common use)? And you only need enough reuse to fill the compute headroom which is quite small on consumer platforms (we won't have huge TOPS numbers for the typical integrated GPU in Strix Halo or even the upcoming RTX Spark). Plus if you're a single user running multiple streams in parallel the choice of experts will be highly biased leading to more reuse.
There's been some very rough experiments with batching on Apple Silicon (and that's not a highly suitable platform since the compute/thermals bottleneck hits sooner than elsewhere) that seem to be broadly consistent with what I argued, showing as much as 2x total decode throughput with an 8-wide batch. That's substantial in this context.
Assuming you magically use all 128GiB of xRAM you need to read ~32GiB per token in batched mode. On a good SSD that would be 1/3 tokens per second. Cool, 2x that you can do 2/3 tokens per second. Let's assume you are lucky and can actually do 6/7 tokens per second. That's still an extremely far cry from 20+ tokens per second of 27B before any batching.
Before batching. The slowdown is because the model does not fit the xRAM so experts will have to be read from SSD on every forward pass. That's why it is impractically slow.
Batching could allow you to generate 10 tokens for 10 different conversations at the time, but it also means that you need to load different experts for different tokens, so it does not help as much as it does for dense models.
But IIUC the point is that each expert gets used for more than just the one token. So yes, the tps of a given thread takes a hit because now you're sometimes going to schedule in unrelated experts and it will have to pause. But overall you're utilizing the hardware much more efficiently and so in aggregate there's a speedup.
On top of that (as previously pointed out by zoz) for a single user running a single overarching task the choice of experts is expected to be highly biased.
> Why do you think that's the case? Part of the training is balancing load between experts.
That is a fair point. That expectation may have been misplaced on my part. I'm not sufficiently familiar with the details of MoE training.
> The slowdown is because the model does not fit the xRAM so experts will have to be read from SSD on every forward pass.
> 20+ tokens per second of 27B before any batching.
Does the model fit in RAM or not? What is your justification for your stated expectation that the unbatched model will perform 20x faster than the aggregate tps (note, not the single stream tps) of the batched model?
My expectation is that if the unbatched model is 20 tps and batching provides a 2x speedup then each individual stream will be slower but the aggregate throughput should rise to 40 tps. What do you believe me to be missing here?
It has been quantized to 80GB (2-bit quantization for experts) with limited degradation. Certainly competitive with a 27B model, and especially useful in a size range where few "native" models exist.
> Why? Why do you think that's the case? Part of the training is balancing load between experts.
The training balances expert choice across the entire scope of the model. Experiments have consistently shown that within a given session or topic (taken in a broad sense) expert choice is biased in a way that's likely to make caching useful and reuse across a user-specific batch realistic.
While prefill is bottlenecked by GPU compute time, decode might be bottlenecked by GPU memory bandwidth, as you basically need to go through entire KV cache for each new token. So compression can make it faster - you will use more GPU compute but less memory bandwidth for attention calculation
Host to device bandwidth (ram to vram) is 128Gb/s for PCIe Gen 6. VRAM to GPU bandwidth is 1.8Tb/s for GDDR 7 (5090), and 8Tb/s for HBM3e (B200). So it can be faster to recompute than offload kv cache.
While this might seem to be true for casual users, I recall that one of the reasons for Anthropic's recent changes for only retaining KV cache for an hour or so, was that many users just have one massive ongoing session that they continue on with multiple unrelated queries (as one would in a single-thread "group chat"). And this is hard to distinguish from someone who wants that context for their seemingly-unrelated query to apply tone etc.
So in practice, there are many casual users who are typing their Google-esque searches against a 100k+ token context window - and it's at that point where things balloon into 300GB+ KV caches to maintain.
I wouldn't be surprised if we see new UX's around subsidized plans starting to encourage resetting the context window more often.
Because you need kv proportional to context length during inference of a single token to avoid quadratic recomputation. So compressing the kv lets you handle longer contexts in the same amount of vram.
Why is it such a dramatic statement for Boris to claim that he no longer writes code?
Are people on HN still typing out functions by hand one character at a time?
It would be like a developer in 2020 claiming that he only writes assembly because compilers can’t be trusted. No one is taking that person seriously. If you chose a career in tech you made a decision to work in one of the fastest moving fields in human history. Now it’s time to get over it, learn the new tools and adapt.
> Now it’s time to get over it, learn the new tools and adapt.
No, thank you. I have used the new tools, determined that they aren't helpful to me, and set them aside as I would with any other bad tool. I don't feel the need to let hype take the steering wheel.
> Now it’s time to get over it, learn the new tools and adapt.
Exactly. You are free to use openclaw or a coding agent to build a competing bank, hedge-fund, hospital or even a new airliner because the previous ones were built by humans. Surely an AI can do it better by itself.
> Are people on HN still typing out functions by hand one character at a time?
Yes, me. Yes, I tried LLMs for what I am doing and will try again in few months. No, there was no noticeable or clear improvement over doing it manually.
Yes, I am using some LLMs for some purposes but Claude Code had slight improvement, if any, not worth introducing proprietary dependency.
> Why is it such a dramatic statement for Boris to claim that he no longer writes code?
Because we can actually see the disjointed slop that Anthropic produces. And when issues happen, they can't fix them for weeks on end because no one understands what code does anymore, and all of their "hard problems causing issues" they blog about are literally "if we had actual engineers this wouldn't even be an issue to begin with". Like this bullshit they had in spring: https://www.anthropic.com/engineering/april-23-postmortem
> It would be like a developer in 2020 claiming that he only writes assembly because compilers can’t be trusted.
LLMs are not compilers. For a few very obvious reasons I'll leave as an exercise to figure out
You answered your own question... you are needed to tell it what to do. Let's not pretend that someone with prior software skills will be able to produce larger scale and/or higher quality work compared to someone with no experience.
> Let's not pretend that someone with prior software skills will be able to produce larger scale and/or higher quality work compared to someone with no experience.
This seems like a really confident prediction. It isn't true right now, why do you think it will be true in the future? Right now having knowledge and experience is a huge benefit to steering the LLM (it makes dumb decisions all the time still).
I totally intended to say "let's not pretend that ... will NOT be able to". Typo and poorly worded, my mistake. I completely agree that the prior experience significantly amplifies what you can do with an LLM over someone without experience.
>Are people on HN still typing out functions by hand one character at a time?
Well I use tab completion, of course. And I copy-paste snippets from LLM more often than from SO now. But otherwise not much has changed in my career in the last 5 years. Is this different for you?
I'm not fundamentally opposed to code generation, and I use LLMs for some taks, but I don't see myself vibecoding whole pages of production code. I vibecoded a throwaway note-taking app for myself though.
Yes. If you're copy posting code from LLMs you're in the minority of people who are "in the past". Most people are generating most of their code in ~1-200 LOC chunks, reviewing it, adjusting as needed (usually with another prompt) and then opening a PR, which gets reviewed both my LLM and other teammates.
Citations very much needed on the "most" here. I'm in a huge corpo with ~1500ish devs and the large majority of code is still very much handwritten, and we have access to all the AI tools imaginable (it's not forced on us because they treat us like professionals that will use the most appropriate tools to do the job, thankfully).
Our whole team has transitioned to agentic engineering around the start of this year. We've been heavily investing in harness engineering to ensure final outputs are production quality and not slop.
These days it very much feels like the task has shifted from "building the system" to "building the system that builds the system".
It is because HN is contrarian and behind the times.
I work at a big tech company and I don't know a single person that still hand writes code. Most people haven't hand written code for at least half a year now.
I do wonder what sort of bug is making its rounds on HN that people here find this so shocking and unbelievable.
The bug is called "applying actual engineering principles and critical thinking".
The absolute vast majority of people who point out AI's downsides have used it and use it. People who uncritically write things like "I work at a big tech company and I don't know a single person that still hand writes code." scare the shit out of us for a good reason.
Using it is not enough. You have to explore its boundaries, see what it can do when cost is not a constraint. This is only really possible at the big tech companies and well funded startups.
Any eng that is only using Claude Code or Codex or whatever, is frankly not entitled to talk about AI's limits since they are using the most basic harnesses. They literally don't know better.
When I see Claude Code or Codex users on HN talking about how coding with AI is risky, it's like watching someone that has only ever seen a catapult argue about how space travel must be impossible.
> Any eng that is only using Claude Code or Codex or whatever, is frankly not entitled to talk about AI's limits since they are using the most basic harnesses. They literally don't know better.
I really doubt big tech has much better harnesses than are publicly available. Definitely not "catapult vs space travel". They have the same base models we all have access to.
They are utterly trivial tools, and most of them are extremely over-engineered to the point of absurdity (that was the most embarassing about the Claude Code source map leak imho).
That the for loop concatenating LLM output from POST calls to an internal buffer has more code than what is doing training or inference should tell them something.
Hermes is the only one that tried to do something novel (low bar), and at least when I looked at it it hadn't succumbed to the vibes yet.
You only explore boundaries of an AI if you are actually training and evaluating your own models.
Harnesses aren't testing boundaries of AI. They inject "make no mistakes" in various forms and provide some session management tools.
And, as with any people fully buying into and promoting hype, "my harness is in another castle" (they never show anything they boast about") lathered with huge amounts of crude demagoguery and analogies.
Would you care to share the name of a good harness which might qualify someone to talk about AI's limits? There's quite a lot of big tech companies and well funded startups using Claude Code and Codex, although I suppose it's possible that none of them know what they're doing.
I'd probably break NDA if I said anything about ours, sadly. I don't know of any publicly available harness on the same level as what big tech companies use internally.
solenoid0937 is not working at any big tech company, instead they are being paid by Anthropic to spread unfounded LLM-hype. They have a history of doing that. See https://news.ycombinator.com/item?id=48270186 for more info.
I installed 4 mini splits with it two years ago and they are still working great and blowing ice cold air.
It walked me through measuring refrigerant, subcool and superheat, pulling the vacuum, brazing the lines, exactly what tools to buy, I even input the numbers from the meter and it told me how much to add and so on. And this was with GPT4 or something far less intelligent.
In the past I tried to learn this stuff but the HVAC community are massive gatekeepers and try to hide information behind paywalls or spread FUD even though anyone could do it with the right tools and a little bit of knowledge.
Saying that society is divided on AI therefore HN, being society, should also be divided is an absurd take.
I expect the people in here to be domain experts, understand simple concepts like closed loop water cooling, deterministic vs non-deterministic systems, maybe some basic concept of how a GPU and vector math works and most notably the exponential pace that it’s becoming both more capable and more efficient.
Unfortunately, like OP that’s not the case and it’s the same talking points I could read in my local paper. Then everyone’s talking points change in unison like they are waiting on the latest instructions from headquarters.
You are getting negative feedback, maybe because of tone; but I am up-voting for the message which is in line with HN guidelines and expectations:
1. Comments should be thoughtful even if not coming from a place of expertise
2. Better comments have something useful to share behind the passion
3. "Me too" comments are usually not as interesting to read and rarely add to the discussion
I have personally been both praised and attacked for making detailed, reasoned comments. It's more about the mood and attitude of the reader(s) that determines which way it goes.
AI is a complicated issue because it is creating change very quickly--as many predicted it would 20-30 years ago--and that change is still accelerating.
In your words, the change is exponential--and this may indeed be demonstrated by the performance data over the past several years. It's hard to contemplate or predict when that change will plateau, but we definitely aren't there yet.
The ultimate impact on the world society is likely to be profound. Whether you like it or hate it, it is happening. Those who lose a job to AI may find that their mother is saved by a medicine or doctor aided by AI. It's complicated.
Perhaps there are ways to legislate compromises into its use or expansion in some parts of the world--but it is likely a technology that has at least as much positive potential for humanity as it does negative--so legislation is going to have to address the good and the bad and no country is an island unto itself.
I find that the more you know about a subject the worst impression you will have of AI. What it knows is kinda shallow and basic, and if you have a more difficult question the answers are confidently random.
That's something we've probably all noticed - Gell-Mann amnesia à la LLM.
But the opposite is also true: shallow basic knowledge about almost anything, instantly available in a form that directly applies to one's problem, is an incredible resource. Not only can you get pretty far on that alone, it also gets you moving in a way that inertia would otherwise have prevented. That's my experience at least.
The short answer is all those problems have already been solved.
Israel desalinates 75-85% of its drinking water. The problem is political and economic dysfunction.
California for example could be doing widespread desalination with nuclear power and technology from the 1970s. They could also greatly expand reservoirs and waterways, but don’t do it. Very similar to Rome in the 400s, when people were using aqueducts built by a past civilization but lost the ability to construct them.
Nuclear is very expensive per MWh and thus per litre of water generated
Solar on the other hand is very cheap, and you don't need to desalinate 24/7 -- just do it when power is cheapest (which is during the sunny times if you have large amounts of solar, during windy if you have large amounts of wind, etc)
I’d love more datacenters so I can colo my pet projects and startup like I was doing in the early 2000s
The point where we decided we would put all infrastructure in N. Virginia, stop owning hardware and rent it from a corporation charging 10x markup was right about when the Internet started going downhill.
I live in rural Ohio and there is like 6? Colo places within an hour’s drive.
When I lived outside of Orlando there was like a dozen in Orlando alone.
If you look up Colo options in your area, what is lacking? These new datacenters are high density AI hypercalers. They are not traditional Colo DC’s. It’s more like a whole new AWS AZ is getting slapped down. More of the same cloud computing you’re complaining about.
If anything, existing colos are going to be hurt by what's going on right now, as they're getting all of the political backlash from the AI hyperscaler bullshit going on.
> Don’t be fooled, they already 100% do that if you use any of these products.
Just to clarify for anyone not paying attention -- Anthropic has written postmortems detailing their Claude Code monitoring and how they "coordinated with authorities" as they "gathered actionable intelligence" from users creating bad content [0].
Not sure how this doesn’t seem to get more attention. Sensitive queries can undoubtedly end up flagged and eventually in front of a human, apparently with the capability to then explore all other submissions from you.
>Not sure how this doesn’t seem to get more attention
Because it's obvious? Any interaction on the cloud = someone else's computer has 0 expectation of privacy. Are we next gonna pretend being shocked that google queries aren't private.
Claude is something like $35 per million tokens. If I was using API pricing I could trivially spend $100 in a single hour long coding session, with /fast turned on in about 10 minutes. Not sure how you guys are using it.
Or you could let information be free, at least the stuff that’s on the public net.
As for issues like bots overloading websites or using too many resources scaling laws will take care of it quickly, it’s not like you can’t serve thousands of RPS from a Raspberry Pi these days.
I feel like these arguments are always framed as an evil corporation wants to take advantage of consumers. Except that's misdirection. The guilty party isn't the corporation, it's you, the consumer. And the corporations are already regulated. Heavily.
You want Gore-Tex (expanded PTFE) boots, Cobalt EV batteries (Child labor in the DRC), Solar Panels (Open pit quartz mines), Wind Turbine Blades (Epoxy Resins & glass-like fibers), and so on. All those things sound nice and good for the environment but don't appear out of some magical horn of plenty. All those things require intensive chemical and industrial processes that cost a lot of money.
"Just make the government solve the problem by criminalizing their entire operation" isn't a serious solution. It's a generic anti-corporation/NIMBY argument to outsource uncomfortable things to another country without labor or safety protections. Consumers need to accept that if they want nice things those things come with some amount of cost to the environment and level of risk. The government needs to work with corporations to find the safest _practical_ mitigation that doesn't bankrupt them. If that's done correctly you will actually avoid accidents like this because everyone is working together on the same page.
You’re reversing causality. People don’t want gore-tex, and they don’t want cobalt batteries. They want dry boots and transportation.
If some corporation comes along and says they have dry boots and electric cars, it is not realistic to expect every single consumer in a society to become expertly informed on fluorochemistry or the economics of mining, and then also expect them to make the decision that is best for all of us.
Exactly, the burden for better behavior lies with the people with the power, agency, and information.
The consumer is did not decide on the formulation of Chemical X, they weren't there to see the day it accidentally melted through the floor, and if the customer was somehow a hyper-motivated scientist with the right training, the company isn't gonna share the data.
You write as if it would not be possible to work with these chemicals safely at a reasonable cost, and that's just not true. Other jurisdictions manage this.
Corporations naturally seek to improve margins, all the time, constantly. They will push and push against rules and regulations. It's the proper role of government to balance the costs to the corporation against the interests of the public. And it can be done well. But in the US, it's becoming more and more rare.
I have read the rules are tighter in most EU nations.
There is jurisdiction shopping of course. If china or wherever wants to have really lax rules, and that means production moves there, I’m not sure what the answer is.
But, for this product (making plexiglass like things), I expect all the consumer production has gone overseas anyway. This is defense / aerospace, so it probably can’t move.
The answer is actually really easy, and it's been implemented successfully before: selective import taxes.
You set import taxes so that they offset price advantages. If a country has shitty environmental laws, crappy labor protections, etc. then you prices that into their import taxes. That way they don't gain any advantages in a race to the bottom based on things that you care about in your own country.
If a country adopts better environment laws, labor projections, etc. then you lower the import tariffs you charge on that country's relevant goods
They are not saying that we have to do everything here, they are saying: If you want to move your mfg. to a country that has less environmental regulation enforcement, labor law enforcement and other things that we care about here, then the goods you are shipping back should be tariffed accordingly.
If we as a people pass environmental laws & labor laws because we feel that these things are important then why are we accepting products that are made in violation of our standards.
Snark aside, the widespread lack of understanding of basic economic concepts is actually a real problem. On one side, you got an utter buffoon like Trump acting like taxes and tariffs are his personal revenge sledgehammer to wield however he wants, and on the other side you got (an awful lot of) short-term rewarders who will accept just about any predictable long-term consequence for short-term "lower prices for consumers".
Wow that is a hell of a lot of responsibility to heap on the consumer. I think the right/rational argument is properly regulated safety procedures for storing large quantities of extremely hazardous chemicals. There is a middle ground. This is in my view a regulatory failure if I ever saw one… who was inspecting this tank and what were they looking for? I am willing to bet the gas pump nearest me gets more attention from whoever is responsible for weights and measures.
You present this as if consumers were truthfully informed of all the ecological and labor impacts of products they buy. In many cases contrary is true, companies don't inform the customers, try to hide the impacts or downplay the impacts. Using outsourcing and very difficult to trace supply chains is often way to prevent informed public.
The idea that "If corporations can't do whatever they want, including put everyone's lives at risk, nobody will be able to have anything nice" is a commonly seen argument but of course it's a lie. Companies might make less profit if they had to act responsibly but they'd still make profits. Those that failed to don't deserve to exist and should get out of the way so that a more efficient and capable company can take their place.
The same argument could be made for all kinds of unreasonable demands. If there were products that couldn't be profitably made and sold without slavery do you think we should all just accept that slavery? We, as a society, make choices all the time that certain products, industries, and practices aren't worth the costs. Sometimes it's perfectly fine to bankrupt companies and kill entire industries to do it.
By all indications child porn could be a massively profitably industry. For a long time it was. As a society we decided that crossed a line, and we petitioned our government to outlaw it and enforce that regulation on the porn industry. The economy didn't collapse when we did. It's just as reasonable for the public to decide to demand better safety from the chemical industry and ask the government to regulate that as well. It's been done many times in various forms already. The economy didn't collapse then either.
We'd agree that there is a middle ground to aim for most of the time though. The problem we have now is that government is being bribed to ignore what voters want and roll back many of the regulations we demanded. Incidents like this one remind us that companies should be expected and required to do better, but as long as government can keep accepting piles of cash in exchange for ignoring the rest of us it's not going to be easy to convince the government to do their job. There's also been an increasing amount of voter suppression to make it harder to fire and replace corrupt government with people who will do their job.
It may well be quite valid in context to let a company or even an entire industry go bankrupt if the net negative is large enough and give zero fucks if the mitigation required is practical or affordable. It may also be valid from the perspective of one group of citizens to foist the cost and risk on another nations less organized or represented citizens in another nation. Unkind but we don't pay our lawmakers to represent our citizens and theirs equally.
The average person is dumber than a box of rocks and the ones that aren't have limited time, attention, and expertise. They can't be relied on to make practical decisions while shopping on amazon regarding the practical effects of their buying power. The only hope to have sane decision making is by subject matter experts which is why we are a nation of laws which basically say follow the rules set down by these unlected assholes who actually know <insert subject> because it is literally the only practical way forward in our nation of 338M stupid assholes.
I don't want any of those things, really (besides solar panels I suppose). I avoid plastic as much as I can. But, let's take your boots example. I recently went looking for a pair of well-made boots that don't contain plastic. But that eliminates something like 99% of the available offerings, and most of the remaining are luxury brands that can cost upwards of 600 dollars. I don't have that kind of budget, so I had to compromise. Do you see the problem here? If I want decent boots without a luxury brand fee, I HAVE to give these chemical companies my money. Extend that to clothing, groceries, furniture, devices, etc etc.
I avoid this stuff as much as I can without upending my life, and I'm still forking over much of my spending to companies that can pollute my land, water, and air with near impunity. I didn't choose this shit!
KV can be trivially stored on ram or even a spinning disk and retrieved on the order of milliseconds. See LM cache for vLLM for example. In fact it’s so easy it kinda shocks me when Claude Code will sit and recompute my entire KV on a new session after a couple of hours, I guess Anthropic infra is not as optimized as it would seem.
Think about the problem from first principles:
Storing a few GB per user at scale isn’t that hard and was solved years ago. Let’s say I have 20 chat sessions open and the session persists for a day or two, this seems negligible to me as a systems design problem.
reply