Local SDXL on 4 GB of VRAM: designing around a constraint
I had an idea for a small online shop, and it needed product cards. The obvious answer is to shoot them on a phone. The problem is that a phone shot is fine for a catalogue and not for marketing.
Cloud services exist, but they want a subscription and an account. I wanted something that runs locally with no sign-up at all.
And that is where the main constraint appeared: the laptop's graphics card has only 4 GB of memory.
This article is about designing under that constraint. It is not really about image generation — it is about how engineering decisions get made when a resource limit is hard.
Put the constraint first
The usual approach is: pick the best tool, then try to run it. I went the other way — I wrote the 4 GB constraint down first and passed every decision through that filter.
The effect was immediate. The newest and most powerful models dropped off the list: they simply do not run on this machine. Disappointing, but useful — I did not spend time evaluating them.
The second filter was the account requirement. I wanted to download models without authentication, so sources demanding a token fell away too.
Together the two filters cut the options down to a handful. That is not a loss — it is shrinking the decision space, and it helps in any project.
Why this particular tool
There is exactly one practical way to run SDXL models in 4 GB: paging weights between video memory and system memory. If the model does not fit on the GPU as a whole, parts of it get moved in as needed.
That is slower, but it works. And "works slowly" is infinitely better than "does not work".
The interface I chose had two branches. I made the newer one primary and kept the older as a fallback. Models are shared between them via symlinks — storing the same files twice is not a 4 GB problem, but it is tens of gigabytes of disk.
The reason was simple: the face-enhancement extension I needed is officially supported in the newer branch, while the older one only had an unsupported fork.
What did not work
This is the interesting part.
I originally wanted to add pose control to the pipeline — an extra model that preserves the body position from the source photo. Technically it looked like the right move.
I removed it entirely, for two reasons.
The first was speed. It made the pipeline 2.7 times slower. On 4 GB every additional model is expensive, because it increases the amount of paging.
The second, and more important, was that it worked against the goal. I did not need the original pose preserved. Quite the opposite: I wanted the pose to change — same clothing, different position, different framing. Pose control blocked exactly that.
This turned out to be the most useful lesson for me. A tool being technically good does not mean you need it. The requirement was "the pose should change", and I was trying to add a tool that says "the pose should be preserved".
Before adding any component it is worth asking: does this serve my actual requirement, or am I adding it because it is the "proper" thing to do?
The pipeline that works
The final result is two stages, and deliberately simple:
- Generation — at a medium size, with face enhancement. This stage takes about 77 seconds.
- Upscaling — as a separate step, doubling the size. That is 21 seconds.
Around 99 seconds per card in total.
Why two stages? Because on 4 GB, generating directly at the large size does not fit in memory. Generating smaller and upscaling with a separate tool is the correct way around the limit.
The pattern generalises: splitting big work into two small steps is often cheaper than one big step.
The retreat ladder
Working with 4 GB, running out of memory is inevitable. So I wrote down a retreat ladder in advance — in order, what to do when memory runs out.
Step one: reduce the size. Step two: drop the number of simultaneously generated images to one. Step three: disable extra extensions. Step four: move to a smaller model.
Because it is a list, I do not deliberate when an error appears — I walk down it, starting from step one.
That habit is useful in software too. When a problem hits and you have an ordered fallback plan, you act by procedure instead of panic.
Verification gates
Before starting the project I wrote five "gates" — specific checks that had to pass. For instance: does the model load at all, does a single image generate, does face enhancement work, does upscaling fit, is the total pipeline time acceptable.
That list gave me two things.
First, I always knew where I had stopped. If gate three has not passed, spending time on gate four is pointless.
Second, "the project is done" acquired a precise meaning. Five gates passed means done. That is far better than a vague "it looks decent".
Conclusion: a constraint is not an enemy
4 GB of video memory is a serious limit. But it did not break the project — it simplified it.
Without it I would probably have tried the most powerful models, kept pose control, complicated the pipeline, and ended up spending three minutes per card.
The constraint forced me to ask every component "is this actually needed?". And most of the time the answer was no.
That, I think, is the most valuable thing about resource limits. They push you toward the simple solution — and the simple solution usually turns out to be the right one.
If you are also on a weak machine, do not treat it as an obstacle. Use it as a design filter: write the constraint down on day one and pass every decision through it. The result will come faster and be easier to understand.