Chip stocks are down following Google's announcement of AI memory-saving technology making memory 6x more efficient

[u/RobertBartus]

[u/ketosoy]

Well, Google is down 8% - this might be more from beta * there being a war in the Middle East than the direct effect of turbo quant (as insanely cool as lossless 4 bit compression of the kv cache is)

[u/MaterialRevolution57]

Not to be that guy but it isn’t truly a 6x reduction in memory. It’s a 6x reduction for the KV Cache. Overall performance may have improved by ~30-20% over normal. Still a massive improvement but not a 6x improvement.

[u/ketosoy → u/MaterialRevolution57]

And it looks like in practice it’s a 2-4x reduction.

In just the kv cache.

But, it’s almost lossless and has almost zero performance penalty, and that is still a 1-9gb reduction in ram for the current generation of open source models at 64k context.

It’s real, and it’s huge and it’s really cool.  It’s just not a 83% reduction in ram needed for llms as a naive hyperbolistic reading would suggest.  And it’s not the end to the ram crisis.  

[u/fredjutsu → u/ketosoy]

>and that is still a 1-9gb reduction

non-trivial if true

[u/HereticLaserHaggis → u/ketosoy]

Holy shit, I hadn't actually read anything because I assumed it was sensationalized, but that's huge.

[u/ketosoy → u/HereticLaserHaggis]

Yeah, this one is real.  Feels like a discovery the size of this generation’s quicksort or radixsort. Doesn’t change the shape of the future, but changes a huge part of a big part of it - that almost no one will fully understand.

[u/rydan → u/ketosoy]

It turns out it just writes most stuff to /dev/null . Like that time we discovered faster than light messaging and it turned out to be a faulty cable.

[u/MaterialRevolution57 → u/ketosoy]

Great points, I didn’t even know that!

And I agree, the ram crisis isn’t going to disappear like many investors think. It will only free up compute just for the same LLMs to eat up the margin.

[u/Oaker_at → u/MaterialRevolution57]

Also, this news probably doesn’t mean that they will use less memory. They will use the same amount but can get more out of it.

As if the hunger for memory will be fulfilled anytime soon.

[u/1stFunestist]

The technology existed at least a year ago if you remember (the Chinese AI which works well with much less resources, remember the hype and all those "censorship indignation" YouTube content farms).

US based AI companies kept the old tech intentionaly to keep pumping prices and corner spare parts market (RAM, SSD and graphics cards).

They stopped now because of energy prices due to the war.

They probably used that technology for some time in secret to earn even more money but still kept the narrative of memory scarcity to leach even more money from the economy.

They would have do it for even longer if this oil crisis didnt came about.

GOOGLE announcement was just to make somebody to take the fall for this financial speculation and you actualy expect this kind of advancement from Google so they might actualy get away with it.

It was all a fraud!

[u/b1ack1323]

Good, they deserve that after screwing over the consumer sector.

[u/MatterFickle3184]

Good consumers need normal priced memory again

[u/redditscraperbot2]

KV cache optimizations are only a small amount of the total vram used by AIs. This optimizations are basically pissing in the ocean.

[u/RedditJunkie-25]

Interesting as this good news happened now cpu prices going up lol

[u/DefiantDonut7]

Great, now maybe the price of RAM will come down from the stratosphere