Why I Gave Up Running My Own AI Locally, and What I Built Instead

Overview

I recently spent a few days building a personal knowledge base on Telegram: an assistant I could talk to, send documents to, and have it quietly organize everything into my own private wiki. The idea itself wasn’t complicated. Getting the privacy right, without turning the project into a second job, was the harder part. I tried running the whole thing myself on spare hardware before switching to a hybrid setup that actually stuck.

The Challenge

• Since this was a personal project handling my own notes and documents, privacy was non-negotiable from the start. That ruled out the usual cloud AI subscriptions outright, since none of them offered the guarantee I needed: that my data wouldn’t be logged or used to train anything.
• Running everything locally looked like the obvious fix, so I set it up on a spare RX6800 graphics card connected to a Mini PC over Oculink (a cable connection that lets an external graphics card plug into a small computer), with 16GB of RAM to work with.
• In practice, the RAM limited which models I could run at all, and the hardware itself became its own project: watching temperatures, keeping drivers up to date, and never quite finding a comfortable middle ground between pushing it hard enough to be useful and having it crash if I left it under load for too long.
• On top of that, Singapore’s electricity costs made running a model round the clock hard to justify, and tuning it for my hardware meant learning about things like quantization (compressing a model so it runs on smaller, less powerful hardware). That’s a reasonable rabbit hole for someone who likes tinkering, but not something an everyday user should have to sign up for.

My Approach

• I shifted from local-only to Zero Data Retention (ZDR), a setting offered by some cloud AI providers where your data is processed in memory to generate a response and then discarded immediately, with no logs kept and nothing used for training.
• I chose OpenRouter as the AI provider because it gave me two things at once: a single subscription covering Gemini, Claude, and GPT-5 alongside cheaper open models instead of juggling separate billing across providers, and the ability to switch on ZDR for almost every major model it offers.
• I split the system into two parts. The same Mini PC now runs Nanobot, an on-premise agent that handles the logic and stores everything on my own hard drive, without the GPU. The cloud model is only ever called for a split second, to help organize and link pages into the wiki. The data lives with me; the AI is just borrowed briefly to do the organizing.

Impact & Outcomes

• I now have a working assistant on Telegram and Discord that remembers everything from random membership numbers to detailed project notes, all searchable and linked in my own wiki.
• The data never leaves my control in any lasting way. Nothing is logged by the cloud provider, and the only permanent storage is the hard drive sitting next to me.
• I avoided the maintenance overhead that comes with running a local model full time, the RAM ceiling, the driver upkeep, the crashes, without giving up the privacy that made local models appealing in the first place.

Going through the local setup first was worth it, if only because it showed me exactly what I was trying to avoid. Local LLMs are genuinely good for tinkering, but for an everyday user, that tech isn’t fuss-free yet. A ZDR-enabled cloud and on-premise hybrid solved my actual problem with far less friction. The point of building this in the first place was to make my life easier, and it’s worth remembering that when the tinkering starts pulling in the opposite direction.

All opinions and perspectives expressed here are solely my own and do not represent the views of my current or past employers.