Private LLMs vs ChatGPT: why your company shouldn't use the OpenAI API for sensitive data

Every time your team pastes a contract, an internal email, or customer data into ChatGPT, that information enters OpenAI’s infrastructure. The debate about exactly what OpenAI does with that data is long, but the relevant question for any company is simpler: do you need to take that risk?

The answer is almost always no. The reason many companies still do it is that nobody has explained that a viable alternative exists.

What actually happens with your data at OpenAI

Using the OpenAI API under standard terms is not the same as using the web interface. OpenAI states that data sent via API is not used for training by default — but that can change, and in any case the data leaves your perimeter and passes through US third-party infrastructure, with all the GDPR, NIS2, and EU AI Act implications that brings.

For many industries — legal, healthcare, fintech, public administration — this is not a theoretical problem. It is a potential compliance breach.

The alternative: fine-tuned local models

For many specific business tasks, a well-configured open-weight model on your own hardware is enough. You do not need a frontier model like OpenAI’s GPT-6 and GPT-6.1.

Models I regularly deploy for clients:

• Qwen3.6 and Qwen3.8 27B (Alibaba), in tuned versions: good reasoning, multilingual and, quantized, they fit on one 16 GB GPU.
• Qwen3.5 4B and 9B, for fast tasks: classifying, extracting and summarizing.

All can run on local hardware — from a workstation with an RTX GPU to a dedicated server — without sending any data to the internet.

The real cost of local inference

The most common argument against local models is hardware cost. It is a valid argument if you run the numbers incorrectly.

Our inference server has two 16 GB RTX 5080 cards: a quantized 27B model takes one, and the other handles image, video and voice. The API charges per token and the GPU is a fixed cost, so with high, steady volume the hardware ends up paid for. When that happens depends on your volume and current prices: run the numbers with your real data.

Fine-tuning on your own data also gives significantly better results on specific business tasks than a general-purpose model.

When the OpenAI API makes sense

Not everything is black and white. External API makes sense when:

• Rapid prototyping where data is public or anonymous
• Very low volumes with no sensitive data
• Teams without the technical infrastructure to manage local models

But if your company handles customer data, financial data, intellectual property, or regulated information — and wants GDPR and EU AI Act compliance — the conversation about private AI infrastructure should have happened long ago.

Angel Sulev is a security and AI architect with 30+ years of experience. Founder of PROTISEC, specializing in MLSecOps and private LLM deployments for B2B companies.