- tenancy isolationHow this was measured — Queries for chats and history are scoped to the authenticated owner in the repository layer, alongside route-level authentication.
- per-user
- LLM routingHow this was measured — Hosted models and a local option are routed behind one interface, with per-provider rate-limit and quota errors surfaced cleanly to the user instead of failing the request.
- cloud + local
Context
"Wrap an LLM API in a chat UI" is a weekend project. Turning that into something multiple people can actually sign into is where the engineering lives: passwords hashed correctly, sessions that can be revoked, chat queries scoped to the authenticated owner, a budget on a shared key, and streaming through a reverse proxy.
Constraints
One shared model key powers everyone, so abuse control is mandatory. It ships on a $0 stack (a Hugging Face Docker Space + Neon Postgres), and the frontend is intentionally framework-free to keep the container tiny and the ChatGPT pixel-match tractable.
System architecture
A strictly layered FastAPI backend serving a vanilla-JS SPA: routes → services → an LLM router → a repository → the database. Tenancy is enforced at the repository layer, streaming is SSE tuned for proxies, and the same SQLAlchemy code runs over Postgres in production and SQLite in tests.
System map
Tap to inspect
- 01
Dependency-free frontend; auth gate + per-user history over an HttpOnly session cookie.
- 02
Argon2id hashing; opaque server-side session tokens, revocable on logout.
- Instead of
- Stateless JWTs, and bcrypt for hashing
- Why
- A DB-backed opaque session token (HttpOnly, SameSite=Lax) gives real revocation — logout, suspend, or compromise kills the session with a single row change, which stateless JWTs can't do before expiry. Argon2id is memory-hard, which resists GPU brute-forcing far better than bcrypt.
- Tradeoff
- A DB read per authenticated request — cheap, and worth it for revocation on a multi-user app.
Per-user daily budget on the shared key, checked before the LLM call so over-quota returns a clean 429.
- Instead of
- Metering tokens after the call, or no quota at all
- Why
- One shared model key powers everyone, so abuse control is mandatory. Each user's daily budget is checked and atomically incremented before the model call — an over-quota user gets a clean 429 instead of burning an API call on the shared key.
- Tradeoff
- A coarse per-message budget rather than per-token metering — simpler, and enough to keep a free-tier key alive.
Streams tokens as text/event-stream with X-Accel-Buffering: no so proxies don't buffer.
- Instead of
- WebSockets, or the OpenAI SDK's built-in retry/backoff
- Why
- Token streaming is one-directional, so SSE (Cache-Control: no-cache, X-Accel-Buffering: no) is exactly right and survives a reverse proxy. The provider client sets max_retries=0 with explicit quota/timeout classification, so a 429 surfaces as a clear 'quota exhausted' message instead of the SDK silently backing off into a confusing timeout.
- Tradeoff
- ~20 lines of explicit retry/branching in chat_service — for correct, legible errors.
Ownership enforced on every query; a chat owned by someone else is silently refused.
- Instead of
- Trusting the auth middleware and hand-written query filters
- Why
- Tenancy is enforced where data is accessed: every list/upsert/delete filters by owner_id, and a write to someone else's chat is silently refused. Belt-and-suspenders on top of the auth gate, so a bug in any one service can't leak another user's data. A test proves user B can't clobber user A's chat by reusing its id.
- Tradeoff
- A couple of extra filter calls per repository function — cheap insurance against a whole class of tenant-isolation bugs.
- 03
Gemini, Cloudflare, Groq, Mistral, GLM, NVIDIA — all via the OpenAI protocol.
One OpenAI-compatible client factory; add a model without touching the streaming code.
- Instead of
- A separate vendor SDK per provider
- Why
- One client factory returns an OpenAI-compatible client for every cloud model (Gemini, Groq, Mistral, GLM, NVIDIA) and local Ollama, so adding a model never touches the streaming path.
- Tradeoff
- Providers must speak the OpenAI protocol — true for all the ones that matter here.
Local models over the same interface.
- 04
Identical SQLAlchemy 2.0 path over Postgres (prod) and SQLite (local + tests).
- Instead of
- Forcing Postgres (or Docker) for local dev
- Why
- One SQLAlchemy path selects the driver by URL, so developers 'just run' on SQLite while production stays on Neon Postgres. This once hid a real bug — millisecond-epoch timestamps overflowed Postgres INT4 (register 500'd in prod) while SQLite's 64-bit ints passed the tests — now locked down with BigInteger columns and a regression test.
- Tradeoff
- DB-specific behavior needs a real-Postgres check — the INT4 overflow is exactly why.
Hover a node to preview · click one to see the decision behind it
Use the arrow keys, Home, or End to move between nodes. Press Enter or Space to pin a node's decision, and Escape to release it.
- Vanilla-JS SPA: Dependency-free frontend; auth gate + per-user history over an HttpOnly session cookie.
- Argon2id + sessions: Argon2id hashing; opaque server-side session tokens, revocable on logout. Why: A DB-backed opaque session token (HttpOnly, SameSite=Lax) gives real revocation — logout, suspend, or compromise kills the session with a single row change, which stateless JWTs can't do before expiry. Argon2id is memory-hard, which resists GPU brute-forcing far better than bcrypt. Instead of: Stateless JWTs, and bcrypt for hashing. Tradeoff: A DB read per authenticated request — cheap, and worth it for revocation on a multi-user app.
- Daily quota: Per-user daily budget on the shared key, checked before the LLM call so over-quota returns a clean 429. Why: One shared model key powers everyone, so abuse control is mandatory. Each user's daily budget is checked and atomically incremented before the model call — an over-quota user gets a clean 429 instead of burning an API call on the shared key. Instead of: Metering tokens after the call, or no quota at all. Tradeoff: A coarse per-message budget rather than per-token metering — simpler, and enough to keep a free-tier key alive.
- chat_service: Streams tokens as text/event-stream with X-Accel-Buffering: no so proxies don't buffer. Why: Token streaming is one-directional, so SSE (Cache-Control: no-cache, X-Accel-Buffering: no) is exactly right and survives a reverse proxy. The provider client sets max_retries=0 with explicit quota/timeout classification, so a 429 surfaces as a clear 'quota exhausted' message instead of the SDK silently backing off into a confusing timeout. Instead of: WebSockets, or the OpenAI SDK's built-in retry/backoff. Tradeoff: ~20 lines of explicit retry/branching in chat_service — for correct, legible errors.
- LLM router: One OpenAI-compatible client factory; add a model without touching the streaming code. Why: One client factory returns an OpenAI-compatible client for every cloud model (Gemini, Groq, Mistral, GLM, NVIDIA) and local Ollama, so adding a model never touches the streaming path. Instead of: A separate vendor SDK per provider. Tradeoff: Providers must speak the OpenAI protocol — true for all the ones that matter here.
- Cloud LLMs: Gemini, Cloudflare, Groq, Mistral, GLM, NVIDIA — all via the OpenAI protocol.
- Ollama: Local models over the same interface.
- Repository layer: Ownership enforced on every query; a chat owned by someone else is silently refused. Why: Tenancy is enforced where data is accessed: every list/upsert/delete filters by owner_id, and a write to someone else's chat is silently refused. Belt-and-suspenders on top of the auth gate, so a bug in any one service can't leak another user's data. A test proves user B can't clobber user A's chat by reusing its id. Instead of: Trusting the auth middleware and hand-written query filters. Tradeoff: A couple of extra filter calls per repository function — cheap insurance against a whole class of tenant-isolation bugs.
- Neon ⇄ SQLite: Identical SQLAlchemy 2.0 path over Postgres (prod) and SQLite (local + tests). Why: One SQLAlchemy path selects the driver by URL, so developers 'just run' on SQLite while production stays on Neon Postgres. This once hid a real bug — millisecond-epoch timestamps overflowed Postgres INT4 (register 500'd in prod) while SQLite's 64-bit ints passed the tests — now locked down with BigInteger columns and a regression test. Instead of: Forcing Postgres (or Docker) for local dev. Tradeoff: DB-specific behavior needs a real-Postgres check — the INT4 overflow is exactly why.
Data flow
A request authenticates against a server-side session, the daily quota is checked before the model call, chat_service streams tokens over SSE, the router picks an OpenAI-compatible client for the chosen model, and the repository persists the turn scoped to the owner.
Key decisions
DB-backed sessions over stateless JWTs
Instead of: stateless JWTs
Argon2id hashing and an opaque server-side session token (HttpOnly, SameSite=Lax) give real revocation, session tracking, and clean expiry.
Trade-off accepted — A DB read per authenticated request — worth more than shaving a query for a chat app.
Tenancy enforced at the repository layer, not in routes
Instead of: ownership checks sprinkled across routes
Ownership is enforced where data is accessed: lists filter by owner, and upsert/delete silently refuse a chat owned by someone else — one auditable choke point.
Trade-off accepted — The 'silent skip' is less chatty than a 403; repository-level checks can be reviewed and tested together.
Quota checked before the stream, not after
Instead of: meter tokens after the call, no quota
Each user gets a daily budget keyed by user + date, atomically incremented and checked before the LLM call, so an over-quota user gets a clean 429 instead of burning an API call.
Trade-off accepted — A coarse per-message budget rather than per-token metering — simpler and enough to keep a shared free-tier key alive.
OpenAI-compatible routing over per-vendor SDKs
Instead of: a separate SDK per provider
One client factory returns an OpenAI-compatible client for cloud models (Gemini, Groq, Mistral, GLM, NVIDIA) and local Ollama, so adding a model never touches the streaming code.
Trade-off accepted — Providers must speak the OpenAI protocol.
SSE tuned for real proxies
Instead of: WebSockets, plain SSE
Streaming sends text/event-stream frames with Cache-Control: no-cache and X-Accel-Buffering: no so a reverse proxy doesn't buffer the token stream.
Trade-off accepted — One-directional SSE rather than WebSockets — exactly right for token streaming.
What failed
Millisecond-epoch timestamp columns were overflowing 32-bit INT4in Postgres (values near 2.1×10⁹) while SQLite silently tolerated it — the exact class of bug that's invisible locally and obvious against a real database. The columns are now BigInteger.
Cause — ms-epoch values overflowed 32-bit INT4 in Postgres; SQLite silently tolerated it
Fix — widened the columns; caught only by deploying against the real production database
Performance & cost
| Metric | Value | Method |
|---|---|---|
| tests passing | 31 | pytest + ruff gating CI, plus pip-audit + gitleaks + Dependabot. |
| database path | dual | Identical SQLAlchemy 2.0 code over Postgres (prod, pooled) and SQLite (local + tests), so tests need zero external services. |
| hardening | CSP + limits | Sliding-window rate limits on auth/reset/upload, a strict CSP (object-src none, frame-ancestors none), HSTS, and structured per-request logging. |
| document grounding | 10 MB cap | .txt/.md/.pdf/.docx uploads are extension- and size-limited, extracted, and truncated to a 200k-char cap before use as context. |
Operations
Shipped as a single Docker container; sessions and usage live in Postgres. The ownership check is visible at the repository boundary:
# a chat owned by someone else is silently refused — one auditable choke point
if existing and existing.owner_id not in (None, owner_id):
continue # never touch another tenant's rowWhat I'd do differently
A side-by-side compare mode (stream two models to the same prompt) is on the roadmap; the shipped app streams one model per request. I'd also move the coarse per-message quota to real per-token metering once usage justifies the added bookkeeping.
Evidence
The app is live and the auth, tenancy, and routing code are in the repository above.
Changelog
End-to-end account flows (change/forgot password, delete account) and SEO/PWA metadata.
A ChatGPT-parity UI overhaul and clearer provider rate-limit / quota error handling.
Rebuilt a single-user chat app into a multi-user SaaS — per-user auth + history, Dockerized, CI/CD, hardened.