编写一个 Shell 脚本,检查 Kubernetes 节点上的 GPU 资源使用情况并输出告警
脚本示例:
#!/bin/bash
set -uo pipefail
GPU_UTIL_THRESHOLD=${GPU_UTIL_THRESHOLD:-85}
GPU_MEM_THRESHOLD=${GPU_MEM_THRESHOLD:-90}
nodes=$(kubectl get nodes -l nvidia.com/gpu=true -o jsonpath='{.items[*].metadata.name}')
for node in $nodes; do
echo "checking node: $node"
gpu_info=$(ssh "$node" 'nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total --format=csv,noheader,nounits' 2>/dev/null)
if [ $? -ne 0 ]; then
echo "ERROR: cannot query nvidia-smi on $node"
continue
fi
while IFS=, read -r util mem_used mem_total; do
util=$(echo "$util" | tr -d '[:space:]')
mem_used=$(echo "$mem_used" | tr -d '[:space:]')
mem_total=$(echo "$mem_total" | tr -d '[:space:]')
if [ "$mem_total" -eq 0 ]; then
mem_percent=0
else
mem_percent=$((100 * mem_used / mem_total))
fi
echo "$node GPU util=${util}% mem=${mem_percent}%"
if [ "$util" -gt "$GPU_UTIL_THRESHOLD" ] || [ "$mem_percent" -gt "$GPU_MEM_THRESHOLD" ]; then
echo "WARNING: $node GPU utilization or memory exceeds threshold"
fi
done <<< "$gpu_info"
done
考察点:kubectl 节点标签查询、nvidia-smi 输出解析、循环与阈值判断、远程执行。生产环境更推荐使用 DCGM Exporter + Prometheus 采集 GPU 指标。
题目与答案就在本页;收藏、批注和错题重练在应用内