It's absolutely true. I have three different setups here, a rig with 8 gpus that I use in either one of two modes (all 8, very heavy model or just one much lighter) and two DGX sparks. I haven't used a paid model in months. Sure, they use a few more tokens and the results may be slightly less good (but this doesn't really matter because I'm still in the loop so it is easy to correct). But the fact that I'm not sharing my work with the paid providers is much more to my liking than having big brother looking over my shoulder at every step and occasionally giving me the 'I can't let you do that, Jacques' treatment.
Working on a blog post to document it all, but that will take a few more days. The short version: the big rig is probably $25K, ttft is dependent on how much prefill you have but typically a couple of seconds (2K prefill/sec in one setup, about 500 in the other), tgs 100 to 150 or so depending on what it is doing and how well the speculative decoder works with the current context. Electricity is pretty bad, full tilt it's about 2500W or so but I've power limited it to 1500 to keep it civil. It is not cheaper than using a frontier lab (or at least, I don't think it is), but the independence bit is the important part for me and I don't care if that carries a surcharge, even if it is a substantial one.
The benefit of the agent working firstly for you rather than for Anthropic and OpenAI will become increasingly apparent. We're already seeing signs: remember when Anthropic openly promised to have Claude sabotage people using it to create competitors? And OpenAI is looking for revenue opportunities from advertising, which can't end well for their users.
Just think of how user-hostile other online services have gotten, and imagine how acutely painful the proprietary LLM providers could make theirs (if they weren't counterbalanced by open models).
My local model never says no. I never get banned. If I want to do security stuff, reverse engine, there it goes! Do I wanna chat about politics? No problem! Do I want it to analyze medical stuff? I don't get to hear that it's not a doctor and I should call 911. Do I want to explore legal docs, no problem! There are lots of things that cloud models will say no to! I read that don't even agree to help with advanced biology and chemistry topic much. I also never run out of tokens, no 5 hr limit. I can run it 24/7. I don't run out of tokens. I can run 16 sub agents at the same time, non stop! 1-4 billion tokens a month? No problem. Any agent harness I want! No restrictions. Privacy all day. My own true system prompt, not something someone has determined, no filters. Furthermore, there are many open weight cloud models that don't have equivalent online. For a while before cloud models got good at math, the best model at math was DeepSeekv2 Math and there was no API. The only way to use it was to run local. All sorts of audio models, image/video models, etc. No multiple subscription needed. Private UI for my family, kids included (I can guard what kids infer). Got a friend that really needs some tokens? I give em access through private VPN. Telling someone that they are paying for decades of Claude for local AI is like telling someone they are paying for decades of public transport when they buy a car. Sure, you are right, but ...
If I want to go out of business that is what I would do. Most of the data I have here could never be shipped to a service provider, I'd be in lawsuits for the next decade or so and die bankrupt.
Companies pretty much give us the full overview of what they have from a technical perspective (and often much more than that), we sign NDAs with investors that have serious teeth in them. The data they share with us would not be disclosed under any other circumstance and even our reports are shared only with the clients.
It's a nice little business. I'm not going to risk offending my customers for convenience or to save some money.
If you know you will pay Claude a few thousand bucks a year a decade adds up too, owning the hardware could just as well end up cheaper because you can sell it or as others have alluded the price to rent could get much worse.
The bet we (self-hosters) are making is that once these companies IPO, they will be forced to do whatever they have to do to make money. i.e. price increases, more throttling of subscription non-usage-based accoutns, etc.
They are subsidizing all of us to the tune of $billions in losses per month. That can't go on for a decade. Probably not even for another 18 months.
I still use all the tokens Claude will give me on the max plan. But I also run deepseek v4 locally for overflow and for... contingencies.
I mean, I just have 4xV100S GPU and a Macbook Pro w/48GB of RAM and I find it kind of amazing.
My server can run Qwen Flash Next at 500 t/s aggregate (about 200 t/s single user). It's pretty great. I use it for local coding and all my worker agents. It is about as reliable as Sonnet 5 for agentic coding. I make 3D models, keep the home network and server updated, fully built ansible files.
You do NOT need 20K in hardware to run these models reliably. I built my own inference engine from scratch for fun. You can run Strata on 16-32GB cards at very respectable token rates if you have the standard RAM to go with it. I strongly encourage people to go use this model on OpenRouter and try it out for tasks.
I also have a somewhat lobotimized quant of GLM Flash 5.3 working and it's pretty decent too.
V100S is a very good card for some workloads. And they were cheap as well.
I have 240G VRAM total which is a good reason why that rig cost as much as it does, and if I were to build it at today's prices it would be even more expensive.
I got mine just a few weeks ago for about $750 each. I am a decent programmer and have Opus optimizing the inference engine. I know enough to keep it on track, but GPU programming is something I have literally never done, before, but now I know enough about Volta and how to optimize it to be dangerous.
Kind of ironic, but it's the only way. I just really like this as an option. For a household LLM I have the IQ4_K_XL quant and plenty of capacity for everything you could want in a household LLM and it is trustworthy enough to not rm -rf /, etc.
And yeah that is a really nice amount! You can run and compose a lot of interesting models, vision capabilities. IDK. Plus this level of model is far more interactive in the sense that it can implement a well scoped ticket or unit of work but its not going to magically just figure some things out without guidance or a spec :)
I tried getting some 170's when I first heard about that unlock but they were already unobtanium by then. That would have been a nice way to add a lot of VRAM.
GLM 5.3 runs pretty comfortably on this rig, even with large context.
> Sure, they use a few more tokens and the results may be slightly less good (but this doesn't really matter because I'm still in the loop so it is easy to correct)
I can't wait to use models that are 'slightly less good' when I am at work instead of something like Fable that lets me focus on the problem at hand as much as possible rather than also spending additional effort correcting for the tool I am using...
You need to reread what was written. No one is proposing to take away your right to leak proprietary edge to the rapacious frontier labs that need to eat most of the economy in order to survive.
"Affordable" and "IBM" don't belong in the same sentence together, but they're happy to sell you their inference-capable hardware if you've got the wallet for it.
I don't agree with that. They made the PC revolution possible by making hardware that was at least affordable enough to be bought by a large number of businesses and individuals. It also set the stage for the clone market. I'm pretty sure that if IBM really set their mind to it that they could do this.
That doesn't detract from your overall point that they tend to make overpriced stuff, but I don't think they have to do that. Besides, what's overpriced? NVidia's GB300 workstation sells for 95K euros, a single RTX 6000 is now 20K. Surely even IBM can compete at that level.
True, even so: They still make some pretty impressive hardware. They have access to fab capacity, they know how to make chips. They have the capital. If they really wanted they could do it.
In a way it is, but I would see that as a starting point, not as the ready solution. They'd need to do what NV has done for their eco-system: fantastic libraries, a credible future upgrade path, easy ordering, manufacturing partnerships, the works. I figure besides the performance gap that they are at least 5 to 10 years behind NV with their present offering. But NVidia is dropping stitches and IBM should take advantage of that.
I think IBM sees the Spyre as a reason to sell more Power hardware but that's just not where the industry is right now and I doubt it will be enough to create sufficient demand. Who knows, I don't have any insight in what they're doing there, but there is massive gap between A GB300 workstation and what IBM currently offers. At the same time: the RAM crunch hits all parties and that's a major obstacle as well.
Yeah. For all the jokes we make about paperclip maximizer, we've figured it out. It's not paperclips, it's RAM.
That's the thing though. Software is cheap with AI, but Nvidia has Cuda. If software doesn't matter anymore (it does) because it's so cheap now, then Nvidia's moat evaporates to IBM's Spyre. In reality "just a little bit of software" has inertia and cannot be dislodged quickly, quietly, or easily. cf Microsoft.
That. And not starting every sale with 'call us' where they first want you to tell them how much you can afford to spend. I buy a fair amount of gear and if the sales process starts with some sleazeball sales guy instead of a mouseclick on a local webshop then I'm out.
As for CUDA: I think CUDA is a formidable moat but I also think that that is precisely the kind of thing where the likes of IBM could (if they really wanted to) make things happen. They're already in the space with Redhat and they have ample knowledge about drivers and low level code. Putting together a team that either makes a drop-in replacement or alternatively, that starts supporting their custom hardware for the big players in the inference space (VLLM, llama.cpp and Torch) would make all the difference. They'd have to keep doing it though, even if the initial results were lower than expected because that market is not going to turn on a dime.
You'd think that with a 250K workforce and an active line of hardware including an AI accelerator that those people are still working there. I could be wrong, but someone has to be doing that work unless IBM in turn outsources it.
I know. But: they have gobs of money, they still have some talent, they still have a hardware department that I'm pretty sure would jump at a chance to give NV a run for their money. This would require a massive long term investment and a commitment that I'm not sure they would have the stomach for.
They do? They are a spec in the ocean relative to big players offering inference and training their own models. Pretty sure one thing they are struggling with is the money to compete successfully in the ai game this time around.
10 billion give or take, market cap 250 billion, 250K employees. That's not a 'spec in the ocean' by normal standards, and IBM is a mature company, not an inflated balloon.
When that happened I'd been working at Tandy (EU version of Radio Shack) for a while and even though we sold computers they were not taken serious until IBM came along and then suddenly we couldn't supply them fast enough.
Yes, that was the first time I had a 'proper' processor to play with. A Dragon 32 (CoCo clone from the UK).
I've never seen Tandy used outside of Europe but then again, I had not traveled as much then as now so it's definitely possible they used the brand on other continents as well. I wonder how many trees were cut for those catalogs :)
Yes but unironically? IBM should be what NVIDIA would be competing with their NVL72 and DGX products. It doesn't make sense that IBM is just watching NVIDIA take its lunch.
Considering the disclosed numbers for DeepInfer and the usefulness/popularity of DeepSeek 4.1 Flash, I expect it is inevitable that more companies are going to consider the self-hosting option seriously, and perhaps play with the idea of offering inference as a vendor.
While he thinks most businesses can build useful AI infrastructure in just two or three racks, he feels many will struggle to get it running.
“The question becomes: how do you do that?” he said. “The question that the enterprise struggles with is: ‘Where are my kernel engineers? Where are the people who worry about the underlying network? Who can make sure that the model is clustered on GPUs that are close to each other in network proximity? Who can make sure that you’re doing the right caching of results so you get an incredible response?’”
Has he heard of “agentic AI”? You simply fire up Claude or ChatGPT and ask it how to do all of the above.
It's not a one time thing though - onprem inference requires continuous monitoring and tuning to get the most out of your GPUs. Looking at 25-50% performance benefits. Today, it does require dedicated Inference Engineers...
That's based on the idea that humans can learn and adapt and solve problems, and that's not what companies want to think about when they're seeking a profitable transaction such as hiring.
Do not do that if you can't prompt it correctly. It is a potential disaster waiting to happen. LLMs are likely to pull things off the shelf without real defense in depth and that's a problem. Maybe slightly less if it's only in an internal network, but still.
What happens when your competitors invest in better and more expensive tools or materials in general? They (potentially) make better products, but those products also cost more. It's turtles all the way down.
Can we please stop giving weight to CEOs talking their book? Like of course IBM wants you to think this, it would be great for them if it’s true. And maybe it is true! But IBM’s CEO saying it’s true tells me just as much as if a frontier lab said it’s not: nothing.
I’ve been in tech for a long time… I have no idea what IBM does. Now, I get that I’m not the customer, that their customer is “business” but not my business… no… bigger business. The biggest of businesses I assume. I think it must either be a club I’m not in, or IBM is three half racks in a trenchcoat submitting invoices for millions to companies where people just pay it because “it’s IBM, it must be important”.
Ironically one of the smaller places I've worked at tried to use an IBM product once - I'm still not clear on what it was meant to do or how it was better than any cheaper alternative, so I assume someone in leadership was fooled by a very impressive pitch.
Lots of work went into integration and it never went live.
OpenAI most likely pushed Cerebras into a contractual agreement not to do business with Anthropic through 2030. ("Most likely" because the names of the companies Cerebras is contractually obligated not to sell to are redacted in public filings)
Just think of how user-hostile other online services have gotten, and imagine how acutely painful the proprietary LLM providers could make theirs (if they weren't counterbalanced by open models).
I've heard that there are legal software platforms like Clio that offer AI products, not sure what just eavesdropping on some lawyers at the bar.
Companies pretty much give us the full overview of what they have from a technical perspective (and often much more than that), we sign NDAs with investors that have serious teeth in them. The data they share with us would not be disclosed under any other circumstance and even our reports are shared only with the clients.
It's a nice little business. I'm not going to risk offending my customers for convenience or to save some money.
They are subsidizing all of us to the tune of $billions in losses per month. That can't go on for a decade. Probably not even for another 18 months.
I still use all the tokens Claude will give me on the max plan. But I also run deepseek v4 locally for overflow and for... contingencies.
Are you comparing to subscription prices or API prices? Because if it's the former it won't last forever.
My server can run Qwen Flash Next at 500 t/s aggregate (about 200 t/s single user). It's pretty great. I use it for local coding and all my worker agents. It is about as reliable as Sonnet 5 for agentic coding. I make 3D models, keep the home network and server updated, fully built ansible files.
You do NOT need 20K in hardware to run these models reliably. I built my own inference engine from scratch for fun. You can run Strata on 16-32GB cards at very respectable token rates if you have the standard RAM to go with it. I strongly encourage people to go use this model on OpenRouter and try it out for tasks.
I also have a somewhat lobotimized quant of GLM Flash 5.3 working and it's pretty decent too.
I have 240G VRAM total which is a good reason why that rig cost as much as it does, and if I were to build it at today's prices it would be even more expensive.
Kind of ironic, but it's the only way. I just really like this as an option. For a household LLM I have the IQ4_K_XL quant and plenty of capacity for everything you could want in a household LLM and it is trustworthy enough to not rm -rf /, etc.
And yeah that is a really nice amount! You can run and compose a lot of interesting models, vision capabilities. IDK. Plus this level of model is far more interactive in the sense that it can implement a well scoped ticket or unit of work but its not going to magically just figure some things out without guidance or a spec :)
I tried getting some 170's when I first heard about that unlock but they were already unobtanium by then. That would have been a nice way to add a lot of VRAM.
GLM 5.3 runs pretty comfortably on this rig, even with large context.
I can't wait to use models that are 'slightly less good' when I am at work instead of something like Fable that lets me focus on the problem at hand as much as possible rather than also spending additional effort correcting for the tool I am using...
That doesn't detract from your overall point that they tend to make overpriced stuff, but I don't think they have to do that. Besides, what's overpriced? NVidia's GB300 workstation sells for 95K euros, a single RTX 6000 is now 20K. Surely even IBM can compete at that level.
I think IBM sees the Spyre as a reason to sell more Power hardware but that's just not where the industry is right now and I doubt it will be enough to create sufficient demand. Who knows, I don't have any insight in what they're doing there, but there is massive gap between A GB300 workstation and what IBM currently offers. At the same time: the RAM crunch hits all parties and that's a major obstacle as well.
That's the thing though. Software is cheap with AI, but Nvidia has Cuda. If software doesn't matter anymore (it does) because it's so cheap now, then Nvidia's moat evaporates to IBM's Spyre. In reality "just a little bit of software" has inertia and cannot be dislodged quickly, quietly, or easily. cf Microsoft.
As for CUDA: I think CUDA is a formidable moat but I also think that that is precisely the kind of thing where the likes of IBM could (if they really wanted to) make things happen. They're already in the space with Redhat and they have ample knowledge about drivers and low level code. Putting together a team that either makes a drop-in replacement or alternatively, that starts supporting their custom hardware for the big players in the inference space (VLLM, llama.cpp and Torch) would make all the difference. They'd have to keep doing it though, even if the initial results were lower than expected because that market is not going to turn on a dime.
They do? They are a spec in the ocean relative to big players offering inference and training their own models. Pretty sure one thing they are struggling with is the money to compete successfully in the ai game this time around.
Yeah they were only half a decade late to the PC revolution back in the day...
I've never seen Tandy used outside of Europe but then again, I had not traveled as much then as now so it's definitely possible they used the brand on other continents as well. I wonder how many trees were cut for those catalogs :)
If you have ever worked at IBM, it makes total sense. Completely dysfunctional middle management and on up.
we technologists tend to extend what we see and think to other realms where it doesn't make sense
https://karlbode.com/ceo-said-a-thing-journalism/
Lots of work went into integration and it never went live.
Cerebras with Claude 8?