If you’ve used ChatGPT, you know it types an answer back. That’s a powerful kind of AI, but a newer kind of AI does things instead of just talking. It can analyze huge datasets, predict market trends, or even design new molecules.
These “doing” AIs need serious horsepower. They run on specialized computer chips called GPUs (Graphics Processing Units). Think of a GPU as a super-fast engine built specifically for AI math.
Here’s the thing: these engines are expensive. Really expensive. And many companies buying them don’t actually know if they’re running at full throttle, or just idling in the garage.
Why your expensive AI engines might be idling
Imagine you’ve invested millions in a fleet of high-performance sports cars for your team. You’ve got a sophisticated garage, a team of drivers, and lots of important deliveries to make. But you have no idea which cars are actually out on the road, which ones are sitting in the parking lot, or if a driver is using a whole car just to move a single box.
That’s the situation many businesses face with their AI infrastructure. They use Kubernetes (a system that acts like an orchestra conductor, managing thousands of individual AI jobs and their hardware) to deploy their AI projects. But often, they have almost no idea what their GPUs are actually doing. They don’t know who’s using them, how much memory each AI job needs, or if their AI tasks are waiting endlessly in a queue.
This lack of visibility leads to two major headaches, both costly.
First, over-provisioning. Your AI teams often request an entire GPU for a project, just to be safe. It’s like booking out an entire stadium for a small team meeting. The problem? Many AI models, especially during development or for simpler tasks, only use about 30% to 50% of a GPU’s actual capacity. The rest of that expensive hardware sits idle, doing nothing, but you’re still paying for it. You have high nominal demand but low effective utilization.
Second, scheduling blind spots. When AI jobs need GPUs, they often wait in a digital queue. If too many jobs are waiting and not enough GPUs are available, your projects stall. This delay, sometimes called “pod starvation” (a “pod” is a small, self-contained AI task in Kubernetes), means your teams aren’t getting results, and your investment isn’t paying off. These bottlenecks usually only surface when someone complains, not when an alert goes off. Honestly, that’s not how you want to discover a problem.
The standard tools that monitor Kubernetes just don’t show specific GPU details. It’s like having a car dashboard that only tells you if the engine is on or off, but not how fast you’re going or how much gas is left.
A clear view into your AI operations
Here’s where it gets interesting. NVIDIA, a leading maker of GPUs, recognized this gap. They developed an open-source tool called the GPU Usage Monitor. Think of it as that missing dashboard for your AI engines. It gives you a clear, real-time view of every GPU in your Kubernetes clusters.
This tool brings together several specialized components into one simple package. It’s like getting a pre-built, fully functional control panel instead of having to wire every gauge yourself. With a single command, you can deploy it and start seeing actionable insights within minutes.
So, what kind of insights can you get?
It shows you GPU allocation trends. You can track which teams or projects are holding onto GPUs over time. If a team requested 8 GPUs for a project but only ever used 2, you’ve just found 6 idle GPUs that can be reclaimed. That’s capacity you already own, ready for new projects, instead of buying more.
You also see compute utilization with thresholds. This means you see the actual percentage of work each GPU is doing. If a GPU is consistently running at, say, 37% utilization when it could be doing much more, you know there’s room for optimization. You can even set alerts for when GPUs are nearing full capacity to prevent slowdowns.
Crucially, it shows memory usage per workload. This is a big one for cost savings. For example, if a specific AI workload consistently uses just 12 GB of memory on an 80 GB NVIDIA GPU, you’re effectively wasting 68 GB of expensive memory. Knowing this lets you adjust resource requests, freeing up those wasted gigabytes for other tasks.
*This chart illustrates how an AI workload allocated an 80 GB GPU might only use 12 GB, leaving 68 GB idle.*Finally, you can see running and pending pod counts. This is your early warning system. If you see 20 AI jobs stuck in a “Pending” state while only 5 are “Running,” you know you have a bottleneck. You can address it before users report slow results or stalled projects. The dashboard even lets you filter metrics by NVIDIA GPU platform (like Hopper or Blackwell) if you have a mix of different hardware.
How it works (the technical bits you can skip)
For those curious about the nuts and bolts, the GPU Usage Monitor is actually a clever combination of four key pieces of software, all working together. It’s like a specialized team.
First, the DCGM Exporter (NVIDIA Data Center GPU Manager Exporter) acts as the sensor. It sits on each server with GPUs and directly collects hardware metrics, like how much compute power and memory each individual GPU is using.
Then, kube-state-metrics is the Kubernetes reporter. It tells you about the status of your AI jobs (pods) within Kubernetes itself – are they running, waiting, or failed?
*The GPU Usage Monitor combines DCGM Exporter, kube-state-metrics (not pictured, also feeding Prometheus), Prometheus, and Grafana to provide unified visibility.*All these metrics then flow into Prometheus, which acts as the data collector and historian. It gathers all the numbers and stores them efficiently. Finally, Grafana is the dashboard creator. It takes all the raw data from Prometheus and turns it into the easy-to-read charts and graphs you see in the GPU Usage Monitor dashboard. The genius here is that this entire integrated stack can be deployed with a single command using Helm (a package manager for Kubernetes). You can find the project on GitHub under an Apache 2.0 license.
The catch (and how to get started)
This tool is incredibly powerful for visibility, but it’s not a magic bullet that optimizes your GPUs automatically. You still need to act on the insights it provides. For instance, if you see a team consistently underutilizing GPUs, you’ll need to communicate with them about right-sizing their resource requests.
To get started, you’ll need a Kubernetes cluster (version 1.19 or later) and Helm (version 3.0 or later) already set up. Crucially, the DCGM Exporter needs to be running on your GPU nodes. If you’re managing a production environment, remember to update the default admin/admin credentials for Grafana before exposing the dashboard to your wider teams.
The GPU Usage Monitor also fits well into existing setups. If your organization already uses a central Prometheus instance, you can configure the chart to ship GPU metrics there instead of deploying a new one. This keeps your monitoring centralized, which is usually a good thing.
Whether you’re running a small set of GPUs for a single machine learning team or managing a vast AI platform supporting hundreds of workloads, complete GPU observability isn’t just a nice-to-have. It’s a prerequisite for operating that infrastructure efficiently and profitably. It helps you stop paying for idle hardware and keeps your AI projects moving forward, which means better returns on your significant AI investments.
Sources: developer.nvidia.com