Cloud VPS for AI backends: sizing, setup and cost for the server that sits in front of the model

What the backend actually does

Between a browser and an LLM there is a service that holds the API key, checks the user's session, enforces per-user limits, pulls context from a database or vector store, assembles the prompt, calls the model and relays the stream back. None of that is compute-heavy; the server spends most of its time waiting on sockets. That has two consequences for sizing: CPU matters less than the number of connections you can hold open, and memory per connection (the buffered stream plus request state) sets the ceiling. The CPU Cloud plans are the right class of machine for this; GPUs belong on the model side.

Sizing by concurrency

A streamed completion occupies a connection for the whole generation (seconds to a minute), so "concurrent" means users mid-answer, not requests per second. Memory per open stream is small (tens of kilobytes) unless you buffer whole responses; relay chunks as they arrive. The live plan list with current prices is below.

Why per-minute billing suits this workload

Production servers run all month, but the servers around them do not: staging, per-branch preview environments, load-test targets and one-off migration boxes live for hours. With per-minute billing a preview environment that exists for a four-hour review costs four hours, and the cheapest plan runs at $0.0104 per hour. The discipline is the same as with GPU pods: create from a script, destroy from the same script. The always-on production figure for the smallest plan is about $6.47 per month, which sets a floor for the architecture: at that price there is no reason to co-host your database and your API on one box.

A setup that holds up

1. Image. Ubuntu 24.04 LTS or Debian 12; both are offered at deploy time. Pick LTS and enable unattended security upgrades.
2. Access. SSH keys only, password login off, a non-root deploy user, and ufw allowing 22, 80 and 443.
3. Runtime. Docker with Compose, one container per service: API, reverse proxy, optional Redis. Pin image tags.
4. Reverse proxy. Caddy or Nginx terminating TLS, forwarding to the API with buffering disabled on the streaming routes so server-sent events pass through immediately.
5. Secrets. The FastInfra API key in an environment file readable only by the deploy user; never in the image.
6. Observability. Structured request logs with the model id, token usage from the usage block and time to first token; ship them off-box.

Disable proxy buffering deliberately: with buffering on, Nginx holds the entire streamed response and the user sees nothing until the model finishes, which defeats streaming.

Latency: where the time goes

A typical streamed request spends 50–200 ms in your backend (auth, retrieval, prompt assembly), then waits on the model API for time to first token, then relays tokens as they arrive. The model dominates. Placing the VPS near your users minimises the round trips you control: TLS handshake, the first request byte, and the chunk relay. Keeping a connection pool warm to the API host removes a handshake per request. Beyond that, latency improvements come from the model choice on the pricing page, not from a larger server.

Scaling up versus out

The first bottleneck is usually the single process, not the machine: run one worker per vCPU (Gunicorn with Uvicorn workers, or Node cluster mode) before resizing. The second is redundancy: two small servers behind a load balancer survive a reboot; one large server does not. Move state out of the box early — sessions and rate-limit counters in Redis, documents in a managed or dedicated database server — so that adding a second API node is a config change.

A cost sketch

Production: two 2 vCPU / 4 GB nodes always on. Staging: one node, business hours only (about 220 hours). Preview environments: ten per month at four hours each. The variable items add up to roughly a third of one always-on node, and the whole stack stays far below the cost of a single idle GPU hour per day. The CPU cloud guides cluster continues with hardening and sizing for specific runtimes.

Deploying and updating without downtime

With Docker Compose on the VPS, a zero-downtime deploy is a pull and a rolling restart: start the new API container on a second port, point the reverse proxy at it, then stop the old one. Health-check the new container by making a real request to a free-tier model through it, so the deploy verifies the FastInfra key, network path and streaming relay in one go rather than just that the process started. Keep the previous image tag so a rollback is one command.

Secrets rotate the same way: create a new API key in the console, update the environment file, roll the container, then delete the old key. Because the API accepts any active key for the account, the rotation is invisible to users. Schedule it, and schedule OS patching with unattended upgrades, so the security work on the box is routine rather than remembered.