Private LLM or public API? How to decide where your AI runs

A plain decision guide for choosing between a hosted AI API, a private cloud deployment and self-hosted models, based on data, cost, speed and control.

“Can we use this with our data?” is usually the first question a risk team asks about an AI project, and the honest answer is that it depends on where the model runs. There are three broad options, and choosing between them is more a business decision than a technical one.

What are the three options?

  • A public hosted API. You send text to a provider’s service and get a response. Nothing to run yourself.
  • A private deployment in your cloud. The same class of model, offered through your own cloud account, inside your network boundary and under your access controls.
  • A self-hosted model. An open-weight model running on infrastructure you operate, in the cloud or on your own hardware.

How do they compare?

FactorPublic APIPrivate cloud deploymentSelf-hosted model
Data leaves your boundaryYes, under the provider’s termsNo, stays in your accountNo
Cost at low volumeLowestModerateHighest, capacity sits idle
Cost at high, steady volumeGrows with every requestGrows with useCan be lower once capacity is used well
Response timeVaries with shared loadMore consistentYours to tune
Control over model versionProvider decidesYou choose when to moveFull
Effort to runMinimalSome cloud workA team that operates it

Start with the data, not the model

Before comparing options, sort the information the system will touch into classes. Three is usually enough:

  1. Public or low-risk: published content, generic drafting.
  2. Internal: plans, procedures, non-personal business data.
  3. Restricted: personal information, health or financial records, anything under a contract or regulation.

Then ask what your contracts, your regulators and your own policies say about each class. Many organizations find that the first class can go anywhere, the second needs contractual assurances about retention and training, and the third must stay inside their own boundary.

What should you check with a hosted provider?

  • Is our data used to train or improve models? Can that be switched off, and is it off by default?
  • How long are prompts and outputs kept, and can we set that period?
  • In which region is the data processed?
  • Who at the provider can see it, and under what circumstances?
  • What notice do we get before a model version is retired?

Get the answers in the agreement, not from a sales call.

When does self-hosting make sense?

Running your own model is justified in a narrow set of cases: data that may not leave an isolated environment, very high and steady volume where your own capacity costs less than paying per request, or a need for response times you cannot get from a shared service.

It also brings work that is easy to underestimate. Someone has to size the hardware, apply updates, monitor quality and keep it available. If you do not already run comparable infrastructure, count that team in the cost.

A practical way to decide

  1. Classify the data the use case touches.
  2. Rule out any option your obligations do not allow for that class.
  3. Estimate realistic volume for the first year, not the pilot.
  4. Choose the simplest remaining option.
  5. Put a thin layer between your application and the model so you can change your mind later.

The last step matters most. Models and prices change quickly. A system built so the model can be swapped keeps your options open, and it lets you route simple requests to a small model and hard ones to a larger one.

Where the model runs is one control among several. Access rules, logging, testing and human review are still needed whichever option you choose.

Common questions

What is a private LLM?

A language model that runs inside your own cloud account or infrastructure, so prompts and outputs stay within your network boundary and under your access controls.

Is it safe to send company data to a public AI API?

It depends on the data and the provider’s terms. Check whether data is used for training, how long it is retained, where it is processed and who can access it, then compare that with your own obligations for that class of data.

Is self-hosting an AI model cheaper than using an API?

Only at high, steady volume, and only once you include the people needed to operate it. At low or uneven volume a hosted service usually costs less.

Can we change AI models later?

Yes, if the system is designed for it. Keeping a thin layer between your application and the model makes switching providers or versions a contained change.

Talk to us

Want this applied to your organization?

Tell us the business area you have in mind. We aim to reply within one business day with how we would approach it and whether an assessment fits.

Take the AI readiness check

We use these details only to reply to this request. No newsletters, no list. We aim to reply within one business day. See our Privacy Policy.

Book a callTalk to our team