Don’t feed the trolls.

- Internet rule #1 since 1983.

When the AI bubble burst, they’ve already made their cash selling shovels (being very anticompetitive) and walk away. Their startup competitors wither, and they are set for the next “thing.”

They could be the first generation to grow up “GenAI Savvy” kind of like how early internet kids developed pretty decent online critical thinking compared to previous (and unfortunately, subsequent) generations.

I think it’s more “there’s no such thing as bad attention.”

Any engagement compounds, and at some point, turns into money. It’s not a mystery either, it’s a systemic issue from the way people are fed information now, thanks to the engagement optimization race to the bottom.

Celebrities can certainly fall, but it’s only if they’re boring.

Unfortunately Nvidia is, by fair, the best choice for local LLM coder hosting, and there are basically two tiers:

• Buy a used 3090, limit the clocks to like 1400 Mhz, and then host Qwen 2.5 coder 32B.

• Buy a used 3060, host Arcee Medius 14B.

Both these will expose an OpenAI endpoint.

Run tabbyAPI instead of ollama, as it’s far faster and more vram efficient.

You can use AMD, but the setup is more involved. The kernel has to be compatible with the rocm package, and you need a 7000 card and some extra hoops for TabbyAPI compatibility.

Aside from that, an Arc B570 is not a terrible option for 14B coder models.

It’s not functional yet.

This is true. Some elements of JC Avatar were super interesting and detailed. Like the spaceship, that was *ridiculously well-thought-out for such a brief appearance, and no telling how much time was spent on the fauna.

But… they made the overarching story and characters so unremarkable.

I felt something similar watching Black Panther: Wakanda Forever. Talokan (the underwater city) was breathtaking and incredible, no telling how much labor was put into it… only for that gorgeous setting to be used for a brief swim through and never seen again.

This is true, I remember it being lauded in theaters.

I mean, I guess not everyone is media savvy and the story could have felt “new” to some, but there wasn’t a character that stood out or anything, no cast stealing the show…

Yep. I didn’t mean to process shame you or anything, just trying to point out obscure but potentially useful projects most don’t know about :P

https://en.m.wikipedia.org/wiki/External_memory_algorithm

Unfortunately that’s not really relevant to LLMs beyond inserting things into the text you feed them. For every single word they predict, they make a pass through the multi-gigabyte weights. Its largely memory bound, and not integrated with any kind of sane external memory algorithm.

There are some techniques that muddy this a bit, like MoE and dynamic lora loading, but the principle is the same.

A1111

Eh, this is a problem because the “engine” is messy and unoptimized. You could at least try to switch to the “reforged” version, which might preserve extension compatibility and let you run features like torch.compile.

Oh you should be able to batch the heck out of that on a 4080. Are you not using HF diffusers or something?

I’d check out stable-fast if you haven’t already:

https://github.com/chengzeyi/stable-fast

VoltaML is also old at this point, but it has really fast AITemplate implementation for SD 1.5: https://github.com/VoltaML/voltaML-fast-stable-diffusion

Oh, 16GB should be plenty for SDXL.

For flux, I actually use a script that quantizes it down to 8 bit (not FP8, but true quantization with huggingface quanto), but I would also highly recommend checking this project out. It should fit everything in vram and be dramatically faster: https://github.com/mit-han-lab/nunchaku

You don’t want it to anyway, as “automatic” spillover with an LLM painfully slow.

The RAM/VRAM split is manually configurable in llama.cpp, but if you have at least 10GB VRAM, generally you want to keep the whole model within that.

Try a new quantization as well! Like an IQ4-M depending on the size of your GPU, or even better, an 4.5bpw exl2 with Q6 cache if you can manage to set up TabbyAPI.

Depends which 14B. Arcee’s 14B SuperNova Medius model (which is a Qwen 2.5 with some training distilled from larger models) is really incrtedible, but old Llama 2-based 13B models are awful.

But the original Jurassic Park was fun. The writing was sharp and memorable, the cast charismatic, even if the plot is not that important (which is fine).

I did love JC Avatar’s alien flora and fauna, and some small details like the realistic spaceship, but I guess it feels much less exciting in hindsight without anything to “attach” it to.

And again… the IP its name collides with is nothing to sneeze at, visually. You can pause it almost anywhere, even in “mundane” scenes, and get gorgeous fantasy shots and incredible music:

No, all the weights, all the “data” essentially has to be in RAM. If you “talk to” a LLM on your GPU, it is not making any calls to the internet, but making a pass through all the weights every time a word is generated.

There are system to augment the prompt with external data (RAG is one word for this), but fundamentally the system is closed.

There is no movie in Ba Sing Se.

Here, we are safe. Here, we are free.

Oh I didn’t mean “should cost $4000” just “would cost $4000”

Ah, yeah. Absolutely. The situation sucks though.

I wish that the vram on video cards was modular, there’s so much ewaste generated by these bottlenecks.

Not possible, the speeds are so high that GDDR physically has to be soldered. Future CPUs will be that way too, unfortunately. SO-DIMMs have already topped out at 5600, with tons of wasted power/voltage, and I believe desktop DIMMs are bumping against their limits too.

But look into CAMM modules and LPCAMMS. My hope is that we will get modular LPDDR5X-8533 on AMD Strix Halo boards.