If you’ve played around with ChatGPT or Claude, you know AI can talk. It types out answers to your questions, writes emails, or summarizes documents. That’s pretty powerful, right?
But what if AI could do things instead of just talking? Imagine an AI that doesn’t just draft an email, but actually looks up a customer’s purchase history in your CRM, filters the results, then drafts the email, and then sends it through your marketing platform. Or an AI that analyzes sensitive financial reports, identifies anomalies, and then initiates an alert to your compliance team.
Here’s the thing. When AI moves from being a helpful chatbot to an active participant in your business operations, it starts touching very sensitive data and proprietary processes. This creates a whole new set of security challenges. How do you protect that customer data, those financial reports, or your company’s secret sauce (the AI model itself) while the AI is actively working on it?
Most people think about securing data by encrypting it when it’s sitting on a hard drive (data “at rest”) or when it’s traveling across the internet (data “in transit”). And those are critical steps, of course. You wouldn’t leave a physical contract sitting unsecured, would you?
But here’s the weird part: what about when the data is actually being used? When your AI model is actively working on it, making calculations, generating responses? That’s “data in use,” and it’s historically been the hardest to protect. Think of it like a confidential document. You lock it in a safe (at rest) and send it in an armored car (in transit). But eventually, someone needs to read and work on that document. Traditionally, once it’s out of the safe and the armored car, it’s vulnerable on the desk.
NVIDIA Confidential Computing (CC) changes that. It’s like giving your AI a tiny, super-secure, transparent vault that travels with the document even as it’s being read and processed. This “vault” protects your data and your AI model itself while they are actively running – what we call inference (the moment an AI model makes a prediction or generates an answer based on new input).
Why does this matter to you, a business leader? Well, protecting your proprietary AI models means safeguarding your intellectual property. Securing sensitive client data is non-negotiable for regulatory compliance (think GDPR, HIPAA, or emerging AI regulations). And honestly, building trust in your AI systems is paramount if you want widespread adoption within your organization. Confidential Computing offers a way to scale your AI ambitions without compromising on security.
How does this secure “vault” actually work? It starts with the hardware. NVIDIA Blackwell GPUs, like the powerful HGX B300, are engineered with Confidential Computing embedded directly into their chips. This isn’t a software add-on; it’s a fundamental security layer built from the ground up. Each GPU has a hardware root of trust, a private signing key fused into the silicon during manufacturing. This key is never exposed to software, making it an unchangeable security anchor.
Before any sensitive AI workload can even begin, it goes through a rigorous verification process called attestation. Imagine a bouncer checking IDs before letting anyone into a VIP room. In this case, a Confidential VM (CVM) – a special virtual computer running your AI model – needs to prove its identity and integrity. It sends a signed evidence bundle, which is essentially a report on the GPU’s hardware and the CPU’s secure environment, to the NVIDIA Remote Attestation Service (NRAS).
*The attestation process ensures a Confidential VM is verified before receiving sensitive secrets.*The NRAS then checks this bundle against a known-good reference. Only once the Confidential VM is confirmed to be in a verified, untampered state can secrets, such as model decryption keys or sensitive data, be deployed into that secure environment. This attestation handshake typically happens just once when the AI workload starts up, so it doesn’t add latency to individual inference requests later on. For setups with multiple GPUs, like those found in larger AI systems, NVIDIA’s NVLink encryption ensures that data moving between these connected GPUs is also fully secured.
Now, I know what you’re probably thinking: “Sure, more security sounds great, but won’t this slow everything down?” It’s a fair question. Historically, adding layers of security often comes with a performance cost. But this is where NVIDIA Confidential Computing truly shines.
The whole point of this technology is to deliver robust security without crippling the performance needed for real-world AI applications. Honestly, for many business users, a security solution that makes their AI models run 20% slower is a non-starter. You need both speed and safety.
NVIDIA’s benchmarks on their Blackwell Ultra (HGX B300) GPUs, running a large language model called Qwen 3.5-397B-A17B-FP8, showed that enabling Confidential Computing produced minimal overhead. We’re talking about maintaining up to 98% of the performance compared to running without this security layer. The impact on throughput (how many tokens per second the AI can process) and the time per output token (TPOT, how long it takes to generate each piece of the answer) was typically under 8%.
Yes, there are some small overheads. Things like secure work submission latency (the extra time it takes to launch a secure task) and reduced host-to-device CPU-to-GPU bandwidth (encrypted data transfers can be a bit slower). But NVIDIA has baked in several innovations to mitigate these. For instance, tools like FlashInfer and SGLang have been optimized with features like CC-safe autotuning and piecewise CUDA graph support to make sure these overheads stay tiny, even for complex AI workloads.
Look at the numbers. In various test scenarios, from moderate to heavy loads, the performance impact remained surprisingly low. For example, during high-concurrency workloads, the maximum observed throughput overhead was about 7.5%, and the maximum time per output token (TPOT) overhead was around 8.1%. However, in many other common scenarios, these overheads were much smaller, often less than 4% or even below 1%. This means your AI models can run almost as fast as they would without the extra security.
Sources: developer.nvidia.com