If you’ve used ChatGPT, you know it types an answer back. It’s a fantastic tool for getting information or drafting text. But a newer kind of AI does more than just talk. It can do things for you.
Imagine an AI that doesn’t just write an email, but actually looks up customer data, filters it by purchase history, and then drafts a personalized message. This is what we call agentic AI (AI that can act on its own, often by using other tools).
Here’s the thing, though. What if that “doing” involves your company’s deepest secrets? Or customer data protected by strict rules like GDPR or HIPAA? The moment AI starts handling sensitive information, a big question pops up: How do you keep that data truly safe?
The hidden problem with AI security
Most people think about data security in two main ways. First, data “at rest” – that’s information sitting on your hard drive, encrypted. Second, data “in transit” – that’s data moving across a network, also encrypted. We’re pretty good at those.
But there’s a third, often overlooked, state: data in use. This is when your AI model is actively processing information, crunching numbers, making decisions, or generating responses. It’s like a chef preparing a meal with secret ingredients. While those ingredients are being chopped and mixed, they’re vulnerable.
This blind spot can be a real headache for CEOs, CFOs, and investors. You want to use AI for powerful business insights, but you can’t risk a data breach or regulatory non-compliance. Many assume that adding more security will inevitably slow everything down to a crawl. And for a long time, that was often true.
Securing AI without the usual slowdown
The good news is that new technology is tackling this specific challenge. It’s called Confidential Computing (CC), and NVIDIA has built it directly into their latest AI hardware. Think of it like a bank vault for your data, but this vault isn’t just for storage. It’s where your most trusted accountant (the AI) can actively work on your sensitive financial records, fully protected, without anyone else peeking in or tampering with the process.
This isn’t just about software tricks. It’s about security baked right into the physical components of the computer. This approach protects your enterprise data, your proprietary AI model’s inner workings (called “weights”), and the code itself, all while the AI is actively running, or what we call inference (the process where an AI model makes predictions or generates outputs based on new data).
The big takeaway? This hardware-level security is incredibly strong. And here’s the weird part that used to confuse me for years: it barely slows down the AI. Honestly, it’s pretty impressive.
How this hardware-level protection works
So, how does NVIDIA pull this off? It comes down to a few core ideas, all rooted in the hardware itself.
First, there’s a hardware root of trust. This is the absolute foundation. NVIDIA Blackwell GPUs, like the HGX B300 and HGX B200, have security features embedded right into the silicon when they’re made. Each GPU has a unique, private signing key fused into it during manufacturing. This key is like an unchangeable serial number and digital signature for that specific piece of hardware. No software, no firmware, not even the host computer can ever see or access this key. It’s the ultimate ID card.
Then comes attestation. Before any confidential workload (your AI model with its sensitive data) starts running, it undergoes a rigorous check called remote attestation. The NVIDIA Remote Attestation Service (NRAS) acts like a security guard. It verifies a digital “evidence bundle” – essentially a report from the GPU’s hardware combined with measurements from the CPU’s Trusted Execution Environment (TEE, a secure area within the CPU, like AMD SEV-SNP or Intel TDX). This check makes sure the Confidential VM (CVM, a secure, isolated virtual machine where the AI runs) is exactly as it should be, untampered and unmodified. Only once it’s verified, like a security guard checking credentials and inspecting a secure room, can secrets like model decryption keys be deployed into the CVM. This whole attestation handshake usually happens just once when the AI system starts up, so it doesn’t add any delay to individual AI responses.
*The NVIDIA Remote Attestation Service (NRAS) verifies evidence from the NVIDIA Blackwell GPU, then issues cryptographic proof to the Confidential VM (CVM).*Beyond that, the NVIDIA Blackwell GPUs also use NVLink encryption. This ensures that data moving between multiple GPUs in a powerful AI system is also fully encrypted. It’s like having armored trucks moving your sensitive data between those secure bank vaults, completely protected from interception. This whole system forms a security layer that spans the silicon chip itself, the connections between chips, and the underlying system software.
The performance numbers that matter
“That sounds great,” you might be thinking, “but how much does it actually slow things down?” This is where NVIDIA’s Confidential Computing really shines. Most companies assume a significant performance hit for this level of security. But the impact is surprisingly small.
NVIDIA ran benchmarks on their HGX B300 system using a large language model called Qwen 3.5-397B-A17B-FP8. They measured throughput (how many tokens the AI can process per second) and per-token latency (how long it takes to generate each output token). Across a range of different scenarios, the overhead from enabling Confidential Computing was minimal.
Look at these numbers. The throughput overhead (the percentage reduction in tokens per second) was typically under 8%. For example, with 4 concurrent requests, throughput dropped by only 2.0%, and the time per output token (TPOT) increased by just 1.6%. Even with a heavy load of 128 concurrent requests, throughput was down by 7.5% and TPOT by 8.1%. These are tiny numbers in the grand scheme of things, especially considering the security benefits.
*Throughput overhead for NVIDIA Confidential Computing on an HGX B300 system, showing minimal impact across various concurrency levels.*This means that you can get the security benefits of Confidential Computing while still achieving roughly 98% of the performance you’d get without it. That’s a huge shift. You can read more about the detailed benchmarks in NVIDIA’s blog post here.
How they keep it fast
Achieving this low overhead wasn’t accidental. It required clever engineering. There were two main areas where security could have slowed things down:
- Secure work submission latency: This is the extra time it takes to encrypt and launch individual tasks to the GPU. Smaller tasks are more affected.
- Reduced host-to-device bandwidth: Moving data from the main computer (host) to the GPU (device) is slower when it’s encrypted.
To counter these, NVIDIA and others developed smart optimizations. For example, the FlashInfer library uses CC-safe autotuning, which helps the AI system choose the fastest way to run tasks even with the security overhead. And tools like SGLang use an async D2H copy worker to read back tokens without blocking the main process, plus piecewise CUDA graph support to reduce the kernel launch overhead that security can amplify. These are technical details, sure, but they’re the reasons why the performance hit is so small.
A critical layer for enterprise AI
So, what does this all mean for you as a business leader? It means that the biggest hurdles for adopting powerful AI – data privacy, regulatory compliance, and protecting your valuable intellectual property – are getting much, much smaller. You no longer have to choose between cutting-edge AI capabilities and ironclad security.
Confidential Computing lets you deploy sophisticated AI models, even large language models (LLMs, the kind of AI that powers ChatGPT and other conversational agents), to handle your most sensitive customer data, financial forecasts, or proprietary research, all while knowing it’s protected from tampering or unauthorized access during its active use. This is especially important for the rise of agentic AI, which will be making decisions and acting on your behalf. You can learn more about NVIDIA’s broader Confidential Computing offerings here.
This isn’t a solution for every single AI task out there. But for any business dealing with sensitive, regulated, or proprietary data, hardware-rooted security like NVIDIA’s Confidential Computing changes the game. It allows you to confidently build and deploy the next generation of AI applications, knowing your most valuable assets are safe, without having to compromise on speed.