If you want frontier-level, the economically reasonable option is OpenRouter or a direct sub to frontier-of-your-choice.
The reality is that they do not offer configurations that would allow a consumer to run that much VRAM on a single setup to protect datacenter margins. Apple used to, and they stopped, those devices are going for ~$20k+ each on ebay now.
You can get very, very capable models on a 3090/4090/5090/6000 series card. But if you want 'frontier level' you are investing ~22k at a bare minimum if you go new. Used you can probably build your own server for much cheaper up-front cost but it's likely going to be 4-6x+ electricity usage.
I truly think by 2028 we'll have integrated chip systems that'll be able to run opus 4.8 level models at ~500 watts at acceptable performance. Honestly I think now is the worst time to invest in AI hardware. Get your harness ready and processes perfected with hosted models, and wait a few years to buy hardware to transition to running models locally
Trying Taalas is almost scary, there is something unsettling with that speed! Even with that small model, because of the speed, you could run hundreds of sample runs in a second, and pick from the best.
Right now, we seem to be shambling toward a war which would hit globalized industrial processes very hard. Buying decent hardware now might wind up looking like good insurance against that.
the difference is that Putin's hand was forced by age, (possibly) illness, and the last several decades of how he chose to run his country. Putin's power base is a relatively small group of elites and oligarchs who would happily snuff out the man who pushes them out of windows if they get too uppity, if they were given the chance. He needed the cover of war to maintain the fiction of his type of strongman "only I can save us" leadership.
Xi's power base is the simple fact that his leadership has transformed China into the #2, and now because of Trump possibly soon the #1 world superpower. He has also acted aggressively in the last decade to find and remove corruption and prevent individuals from accumulating the kind of wealth and influence that could threaten his power from outside official Party channels. Of course, as I'm not Chinese myself, I have no clue what the internals of Party politics actually look like. But as an outside observer it seems clear that Xi et. al. do not actually need Taiwan for anything other than national pride. They know the US would go to the mat to protect it as TSMC is extremely vital to US military power. And since China cannot compete in that arena and has too much to lose, they instead have focused on weakening the US from within, quite successfully of late.
By the time China finally takes Taiwan it will be with little fanfare and little consequence - they won't touch it until the US either has lost its military capabilities, or the US has its own internal chip industry. Anything else is an existential risk for the coastal cities that are China's entire economic advantage.
There are also significant economies of scale (namely: utilization and batching), which tend to make inference on a shared server more economical even after the operator takes a cut.
You can use batching on consumer hardware, it just requires a KV-cache efficient model (or short context only) and keeping multiple inference flows running in parallel. This is most useful in combination with streamed inference, since the compute intensity of decode with those newer KV-compressed models is high enough that you have limited compute headroom when running at the speed of RAM.
Ya that'd be an awesome project, the only issue is how do you verify it's not being poisoned? To actually validate it would require more analysis than the training took to run. It would require a trusted network, not an open one, unless that can get solved somehow.
The same way Anthropic is making it difficult to compete with them. They intentionally train the model (via PEFT, as called out in the model card) to be dumber when attempting to do things Anthropic doesn't want — in this case, competing with them, but you could apply the same training process for other domains such as actually-malicious use cases.
The same way pursuing a bachelor's degree in order to achieve a nefarious end goal does. Refuse to handhold the user on risky topics and outright refuse to answer if an explicit scenario that appears to be harmful is provided. Provide only textbook level technical explanations for such topics the same as any STEM student has ready access to.
It's a relatively new benchmark but from what I can tell it has serious cred behind it. I assume it will be picked up as part of the standard suite of CS-related benchmarks soon enough.
Yeah, Claude, Grok and Gemini immediately recognized it as a SUBLEQ VM (and worked out from there all the details), only chatgpt got stumped and just said it was a "visual simulator".
All the big labs have plenty of proprietary (i. e. paying PhD holders good money for writing stuff) and synthetic training data now compensating for the lack of "naturally occuring" SUBLEQ and similar stuff.
Your hostility isn't helping things either. "You've been here long enough to understand" followed by an incorrect usage of "goal post moving" makes you look like an agitator.
reply