Local, Private Cloud or Hosted AI? How to Choose

Name the constraint before the architecture.

Write each concern as a testable condition. “No training use” needs a review of the contract and the services actually used. “In this region” needs a data-flow map, including logs and support. “Only these staff” needs identity and access tests. “We can leave the supplier” needs an export and replacement exercise. Give each condition an owner who can accept the evidence.

Separate inference from the rest of the application. Documents may pass through extraction, embedding, retrieval, tracing and backup services before or after a model call. List those paths, the data each carries and its deletion route. A locally served model does not, by itself, establish that the whole application stays local.

Five options. Eight questions.

Treat A and B as different contract choices, C as a managed service choice, and D and E as operating choices. A service marketed as zero retention still needs a review of its scope, exceptions and enabled features. A VPC connection is a network control; verify processing, retention and access separately.

Option E includes on-premises hardware that buyers may choose to isolate from external networks. Offline operation needs a plan for software and model imports, patching, licence checks and recovery without online dependencies. It is included here for comparison. If your environment requires full network isolation, raise it at scoping. Feasibility is assessed per engagement; it is not a standard part of this service.

Make model quality a task test.

Freeze a set of representative inputs, difficult cases and cases that should be refused. Have the people who do the work describe an acceptable answer before running candidates. Compare the hosted baseline and candidate open-weight models using the same source snapshot, permissions and output rubric. Keep a separate held-out set for the final comparison.

Record the model version, serving precision, prompt, retrieval configuration, response time and failed requests. A model tested in one serving configuration is not evidence for another. Assess completion quality alongside latency at the intended concurrency, long-input behaviour and refusal. For a narrow task, a candidate may be useful despite weaknesses elsewhere; the test set must make that scope explicit.

Use the acceptance scorecard to document the pass rules and the evidence. Do not infer task quality from parameter count, deployment location or a general benchmark ranking.

Compare full cost at a useful workload.

Choose one accounting period and define a unit of accepted work, such as a completed, reviewed case. Count input and output use, retries and rejected work for hosted candidates. For self-hosting, count capacity, storage, networking, equipment allocation where relevant, operator time, evaluation and model upgrades. Include initial migration and eventual exit separately. Count the application costs on both sides; cancel only costs that are truly shared.

For an approximately linear range, write hosted cost as Fh + vh × Q and self-hosted cost as Fs + vs × Q. F is fixed cost in the period, v is cost per accepted unit and Q is accepted units. When vh exceeds vs, their crossing is Q = (Fs − Fh) ÷ (vh − vs). If vh is no greater than vs, increasing usage does not recover a higher self-hosted fixed cost in this model; compare the totals directly.

Run a low-use case as well as an expected-use case. Check whether a longer prompt, a different model or more review changes cost per accepted result. A cheaper generated response is not useful evidence if it shifts work to the reviewer or fails the quality gate.

Budget the operating work.

For D and E, name owners for identity, credential rotation, access removal, logging and retention, patching, capacity and model updates. Keep configuration and model versions reproducible. Before an upgrade, run the acceptance suite and rehearse returning to the last accepted version. Define who can pause inference and how requests behave when capacity or a dependency fails.

List the state needed to recover: configuration, source data, indexes, retained conversations, authorised model artefacts and secrets recovery procedures. Decide what may be rebuilt and what must be backed up. Test a restore into a clean environment, including identity and a representative task. For A, B and C, check the provider’s recovery scope in the contract and still test recovery of your application and data.

Plan the exit while the choice is reversible.

For A and B, retain your prompts, evaluation cases and data in formats you control; inventory provider-specific tools and conversation state. Check export, termination and deletion terms. For C, add cloud identity, networking and service-specific APIs to the migration list. An API adapter may reduce code changes, but does not establish equivalent outputs or permissions.

For D, retain deployment configuration and the right to use the selected weights under their licence; rehearse a move to another environment. For E, include hardware replacement, compatible runtimes and recovery media. In each case, run the same acceptance suite after a move, and confirm deletion of retained copies through the relevant owners and contracts.

A mixed deployment needs an explicit boundary.

One possible pattern is a contracted hosted service for general drafting and a model in your cloud account for restricted retrieval. Define the routing rule before use: which data may cross which boundary, who decides, and what happens when classification is uncertain. Test attachments, copied passages, logs and failure paths. A fallback to another model service is a new data flow that needs approval.

Make the next decision in four questions.

If a hosted service meets the data rules, passes the task tests and has an acceptable exit path, keep it on the shortlist. If local control is necessary, establish operating capacity before committing to hardware or infrastructure. Bring the constraint map and evaluation cases to a discussion of private AI infrastructure.