AI 系统性能检查清单（175+ 项）

附录

APPENDIX

AI 系统性能检查清单（175+ 项）

AI Systems Performance Checklist (175+ Items)

This extensive checklist covers both broad process-level best practices and detailed, low-level tuning advice for AI systems performance engineers. Each of these checklist items serves as a practical reminder to squeeze maximum performance and efficiency out of AI systems.这份详尽的检查清单既涵盖流程层面的通用最佳实践，也涵盖面向 AI 系统性能工程师的底层细粒度调优建议。清单中的每一项都是实用提醒，帮助从 AI 系统中榨取最大性能与效率。

Use this guide when debugging, profiling, analyzing, and tuning one’s AI systems. By systematically applying these tips—from low-level OS and CUDA tweaks up to cluster-scale optimizations—an AI systems performance engineer can achieve both lightning-fast execution and cost-effective operation on modern NVIDIA GPU hardware using many AI software frameworks, including CUDA, PyTorch, OpenAI’s Triton, TensorFlow, Keras, and JAX. The principles in this checklist will also apply to future generations of NVIDIA hardware, including their GPUs, ARM-based CPUs, CPU-GPU superchips, networking gear, and rack systems.在调试、性能剖析、分析和调优 AI 系统时，可使用本指南。通过系统性地应用这些技巧——从底层操作系统与 CUDA 微调，一直到集群规模的优化——AI 系统性能工程师可以在现代 NVIDIA GPU 硬件上，借助 CUDA、PyTorch、OpenAI 的 Triton、TensorFlow、Keras 和 JAX 等众多 AI 软件框架，实现极快的执行速度与高性价比的运行。本清单中的原则同样适用于未来几代 NVIDIA 硬件，包括其 GPU、基于 ARM 的 CPU、CPU-GPU 超级芯片、网络设备以及机架系统。

性能调优与成本优化思维

Performance Tuning and Cost Optimization Mindset

A pragmatic, documented loop—quick wins before deep work—turns engineering time into measurable ROI. Start by targeting the biggest runtime and cost drivers, and always profile before and after to verify impact.一个务实且有记录的循环——先做快速见效的优化，再做深入的工作——能把工程时间转化为可衡量的 ROI。首先针对最大的运行时与成本驱动因素，并且始终在优化前后做性能剖析以验证效果。

Combine auto-tuning, framework upgrades, cloud pricing levers, and utilization dashboards for high-ROI wins, documenting results and favoring simple, maintainable fixes. Tune throughput-sensitive hyperparameters when accuracy allows. Here are some tips on the performance tuning and cost optimization mindset:将自动调优、框架升级、云定价手段和利用率仪表盘结合起来，以获得高 ROI 收益，同时记录结果，并优先选择简单、易维护的修复方案。在精度允许的情况下，调优对吞吐量敏感的超参数。以下是关于性能调优与成本优化思维的一些建议：

Optimize the expensive first优先优化开销最大的部分

Use the 80/20 rule. Find the top contributors to runtime and focus on those. If 90% of the time is in a couple of kernels or a communication phase, it’s better to optimize those deeply than to microoptimize something taking 1% of the time. Each chapter’s techniques should be applied where they matter most. For example, if your training is 40% data loading, 50% GPU compute, and 10% communication, then first fix data loading, as you can maybe halve the overhead. Then look at GPU kernel optimization.运用 80/20 法则。找出运行时的最大贡献者并聚焦于此。如果 90% 的时间花在少数几个内核或某个通信阶段上，那么深入优化这些部分，要比对只占 1% 时间的东西做微优化好得多。每一章的技术都应应用在最关键的地方。例如，如果你的训练中 40% 是数据加载、50% 是 GPU 计算、10% 是通信，那么先解决数据加载，因为这部分开销或许能减半。然后再看 GPU 内核优化。

Profile before and after优化前后都要做性能剖析

Whenever you apply an optimization, measure its impact. This sounds obvious, but often tweaks are made based on theory and might not help—or even hurt— in practice. Consider a scenario where your workload is not memory-limited, but you decide to try enabling activation checkpointing for your training job. This may actually slow down the job by using extra compute to reduce memory. In other words, always compare key metrics like throughput, latency, and utilization before and after making changes. Use the built-in profilers for simple timing, such as average iteration time over 100 iterations.每当你应用一项优化，都要测量其影响。这听起来显而易见，但人们常常基于理论做调整，而实践中这些调整可能没有帮助，甚至有害。设想这样一个场景：你的工作负载并不受内存限制，但你决定为训练任务启用激活检查点（activation checkpointing）。这实际上可能因为用额外计算换取内存减少而拖慢任务。换言之，始终在做出改动前后比较吞吐量、时延和利用率等关键指标。对于简单的计时，可使用内置性能剖析器，例如 100 次迭代的平均迭代时间。

Embrace adaptive autotuning feedback loops拥抱自适应自动调优反馈闭环

Implement advanced autotuning frameworks that leverage real-time performance feedback—using techniques like reinforcement learning or Bayesian optimization—to dynamically adjust system parameters. This approach enables your system to continuously fine-tune settings in response to changing workloads and operating conditions.实现先进的自动调优框架，利用实时性能反馈——采用强化学习或贝叶斯优化等技术——动态调整系统参数。这种方法使系统能够针对不断变化的工作负载和运行条件持续微调设置。

Budget for optimization time为优化预留时间预算

Performance engineering is an iterative investment. There are diminishing returns—pick the low-hanging fruit like enabling AMP and data prefetch. These might give 2× easily. Harder optimizations like writing custom kernels might give smaller increments. Always weigh the engineering time versus the gain in runtime and cost saved. For large recurring jobs like training a flagship model, even a 5% gain can justify weeks of tuning since it saves maybe millions. For one-off or small workloads, focus on bigger wins and be pragmatic.性能工程是一项需要迭代投入的工作。收益会递减——先摘取低垂的果实，比如启用 AMP 和数据预取。这些可能轻松带来 2 倍提升。更难的优化（例如编写自定义内核）带来的增量可能更小。始终要权衡工程投入与运行时间缩短、成本节省所带来的收益。对于像训练旗舰模型这样大规模、反复运行的任务，即使 5% 的提升也足以证明数周的调优是值得的，因为它可能节省数百万美元。对于一次性或小规模工作负载，应聚焦于更大的收益并保持务实。

Stay updated on framework improvements持续跟进框架改进

Many optimizations we discussed, such as mixed precision, fused kernels, and distributed algorithms, continue to be improved in deep learning frameworks and libraries. Upgrading to the latest PyTorch or TensorFlow can sometimes yield immediate speedups as they incorporate new fused ops or better heuristics. Leverage these improvements, as they are essentially free gains. Read release notes for performance-related changes.我们讨论过的许多优化，例如混合精度、融合内核和分布式算法，仍在深度学习框架和库中不断改进。升级到最新的 PyTorch 或 TensorFlow 有时能立即带来加速，因为它们纳入了新的融合算子或更好的启发式策略。要利用这些改进，因为它们本质上是免费的收益。阅读发布说明中与性能相关的变更。

Codesign collaboratively with vendors and community members与厂商和社区成员协同设计

Stay connected with hardware vendors and the broader performance engineering community to align software optimizations with the latest hardware architectures. This codesign approach can reveal significant opportunities for performance gains by tailoring algorithms to leverage emerging hardware capabilities. Regularly review vendor documentation, participate in forums, and test beta releases of drivers or frameworks. These interactions often reveal new optimization opportunities and best practices that can be integrated into your systems. Integrating new driver optimizations, library updates, and hardware-specific tips can provide additional, sometimes significant, performance gains.与硬件厂商和更广泛的性能工程社区保持联系，使软件优化与最新硬件架构保持一致。这种协同设计（codesign）方法通过调整算法以利用新兴硬件能力，可以揭示显著的性能提升机会。定期查阅厂商文档、参与论坛，并测试驱动或框架的 beta 版本。这些互动常常揭示可集成到你的系统中的新优化机会和最佳实践。集成新的驱动优化、库更新和硬件特定技巧，可以带来额外的、有时是显著的性能提升。

Leverage cloud flexibility for cost利用云的灵活性控制成本

If running in cloud environments, use cheaper spot instances or reserved instances wisely. They can drastically cut costs, but you may lose the spot instances with a few minutes’ notice. Also consider instance types, as sometimes a slightly older GPU instance at a fraction of the cost can deliver better price/performance if your workload doesn’t need the absolute latest. Our discussions on H800 versus H100 showed it’s possible to do great work on second-best hardware with effort. In the cloud, you can get similar trade-offs. Evaluate cost/performance by benchmarking on different instance configurations, including number of CPUs, CPU memory, number of GPUs, GPU memory, L1/L2 caches, unified memory, NVLink/NVSwitch interconnects, network bandwidth and latency, and local disk configuration. Calculate metrics like throughput per dollar to guide your optimization decisions.如果在云环境中运行，请明智地使用更便宜的竞价实例或预留实例。它们可以大幅降低成本，但竞价实例可能在几分钟通知后就被收回。同时要考虑实例类型，因为如果你的工作负载不需要绝对最新的硬件，有时稍旧的 GPU 实例以几分之一的价格就能提供更好的性价比。我们关于 H800 与 H100 的讨论表明，通过努力，在次优硬件上也能做出出色的工作。在云中，你可以获得类似的权衡。通过在不同实例配置上进行基准测试来评估性价比，包括 CPU 数量、CPU 内存、GPU 数量、GPU 显存、L1/L2 缓存、统一内存、NVLink/NVSwitch 互连、网络带宽与时延，以及本地磁盘配置。计算每美元吞吐量等指标，以指导你的优化决策。

Monitor utilization metrics监控利用率指标

Continuously monitor GPU utilization, SM efficiency, memory bandwidth usage, and, for multinode, network utilization. Set up dashboards using DCGM exporter, Prometheus, etc., so you can catch when any resource is underused. If GPUs are at 50% utilization, dig into why. It’s likely data waiting/stalling and slow synchronization communication. If the network is only 10% utilized but the GPU waits on data, maybe something else like a lock is the issue. These metrics help pinpoint which subsystem to focus on.持续监控 GPU 利用率、SM 效率、显存带宽使用情况，对于多节点场景还要监控网络利用率。使用 DCGM exporter、Prometheus 等搭建仪表盘，以便在任何资源未被充分利用时及时发现。如果 GPU 利用率只有 50%，就要深入排查原因。很可能是数据等待/停滞以及同步通信缓慢。如果网络利用率只有 10%，但 GPU 却在等待数据，那么问题可能是锁之类的其他因素。这些指标有助于定位应重点关注哪个子系统。

Iterate and tune hyperparameters for throughput迭代并调优超参数以提升吞吐量

Some model hyperparameters, such as batch size, sequence length, and number of MoE active experts, can be tuned for throughput without degrading final accuracy. For example, larger batch sizes give better throughput but might require tuning the learning rate schedule to maintain accuracy. Don’t be afraid to adjust these to find a sweet spot of speed and accuracy. This is part of performance engineering too—sometimes the model or training procedure can be adjusted for efficiency, like using activation checkpointing or more steps of compute for the same effective batch. You might tweak the training learning rate schedule to compensate for this scenario.一些模型超参数，例如批大小、序列长度和 MoE 活跃专家数，可以在不降低最终精度的情况下针对吞吐量进行调优。例如，更大的批大小能带来更好的吞吐量，但可能需要调整学习率调度以保持精度。不要害怕调整这些参数，以找到速度与精度的最佳平衡点。这也是性能工程的一部分——有时可以为了效率调整模型或训练流程，例如使用激活检查点（activation checkpointing），或在相同的有效批大小下使用更多计算步数。你可能需要微调训练学习率调度来补偿这种情况。

Document and reuse记录并复用

Keep notes of what optimizations you applied and their impact. Document in code or in an internal wiki-like shared knowledge-base system. This builds a knowledge base for future projects. Many tips are reusable patterns, like enabling overlapping and particular environment variables that help on a cluster. Having this history can save time when starting a new endeavor or when onboarding new team members into performance tuning efforts.记录你应用了哪些优化及其影响。在代码中或内部的类 wiki 共享知识库系统中进行文档化。这为未来的项目构建了知识库。许多技巧都是可复用的模式，例如启用重叠（overlap）以及某些在集群上有帮助的特定环境变量。在启动新项目或将新团队成员引入性能调优工作时，拥有这些历史记录可以节省时间。

Balance optimizations with complexity在优化与复杂度之间取得平衡

Aim for the simplest solution that achieves needed performance. For example, if native PyTorch with torch.compile meets your speed target, you might not need to write custom CUDA kernels. This will help avoid extra maintenance. Over-optimizing with highly custom code can make the system brittle. There is elegance in a solution that is both fast and maintainable. Thus, apply the least-intrusive optimization that yields the required gain, and escalate to more involved ones only as needed.以最简单的方案达成所需性能为目标。例如，如果原生 PyTorch 配合 torch.compile 就能满足你的速度目标，你可能不需要编写自定义 CUDA 内核。这有助于避免额外的维护工作。用高度定制化的代码过度优化会使系统变得脆弱。既快速又可维护的解决方案自有其优雅之处。因此，应应用能带来所需收益的侵入性最小的优化，仅在需要时才升级到更复杂的优化。

Optimize AI-driven performance优化 AI 驱动的性能

Leverage machine learning models to analyze historical telemetry data and predict system bottlenecks, enabling automated adjustments of parameters in real time to optimize resource allocation and throughput.利用机器学习模型分析历史遥测数据并预测系统瓶颈，从而实现参数的实时自动调整，以优化资源分配和吞吐量。

可复现性与文档化最佳实践

Reproducibility and Documentation Best Practices

Performance wins don’t stick unless they’re reproducible, versioned, and continuously checked, or they’ll regress quietly over time. Treat docs, CI benchmarks, and shared knowledge as the glue that preserves speedups and accelerates onboarding and audits.性能收益不会自动保持——除非它们可复现、有版本管理并被持续检查，否则会随时间悄然回退。请将文档、CI 基准测试和共享知识视为维系加速效果、加速新人上手与审计的黏合剂。

Lock down versions, configs, and benchmarks in source control so experiments are repeatable and regressions traceable. Bring performance checks into CI/CD, instrument end-to-end monitoring and alerts, and pair optimization with security and thorough documentation to create a durable, auditable practice. The following is a list of tips to improve reproducibility and documentation:将版本、配置和基准测试锁定在源码控制中，使实验可重复、回退可追溯。把性能检查纳入 CI/CD，为端到端监控和告警配置埋点，并将优化与安全性和完善的文档相结合，从而形成持久、可审计的实践。以下是提升可复现性与文档化的一些技巧：

Rigorous version control严格的版本控制

Maintain comprehensive version control for all system configurations, framework/driver versions, OS settings, optimization scripts, and benchmarks. Use Git (or a similar system) to track changes and tag releases. This way, experiments can be reproduced exactly—and performance regressions can be easily identified.对所有系统配置、框架/驱动版本、操作系统设置、优化脚本和基准测试保持全面的版本控制。使用 Git（或类似系统）跟踪变更并给发布打标签。这样，实验可以被精确复现——性能回退也能被轻松识别。

Continuous integration for performance regression针对性能回退的持续集成

Integrate automated performance benchmarks and real-time monitoring into your CI/CD pipelines. This ensures that each change—from code updates to configuration changes—is validated against a set of performance metrics, helping catch regressions early and maintaining consistent and measurable performance gains. Adopt industry-standard benchmarks, such as MLPerf, to establish a reliable performance baseline and track improvements over time.将自动化性能基准测试和实时监控集成到 CI/CD 流水线中。这确保每一次变更——从代码更新到配置变更——都针对一组性能指标进行验证，有助于尽早发现回退，并保持一致且可度量的性能提升。采用行业标准基准测试（如 MLPerf），以建立可靠的性能基线并跟踪随时间的改进。

End-to-end workflow optimization端到端工作流优化

Ensure that optimizations are applied holistically across the entire AI pipeline— from data ingestion and preprocessing through training and inference deployment. Coordinated, cross-system tuning can reveal synergies that isolated adjustments might miss, resulting in more significant overall performance gains.确保在整个 AI 流水线上整体地应用优化——从数据摄取与预处理，一直到训练与推理部署。协调一致的跨系统调优能够揭示孤立调整可能错过的协同效应，从而带来更显著的整体性能提升。

Automated monitoring and diagnostics自动化监控与诊断

Deploy end-to-end monitoring solutions that collect real-time metrics across hardware, network, and application layers. Integrate these with dashboards, such as Prometheus/Grafana, and configure automated alerts to promptly detect anomalies, such as sudden drops in GPU utilization or spikes in network latency.部署端到端监控方案，采集跨硬件、网络与应用层的实时指标。将这些指标与 Prometheus/Grafana 等仪表盘集成，并配置自动告警，以便及时检测异常，例如 GPU 利用率的骤降或网络时延的尖峰。

Fault tolerance and automated recovery容错与自动恢复

Incorporate fault tolerance into your system design by using distributed checkpointing, redundant hardware configurations, and dynamic job rescheduling. This strategy minimizes downtime and maintains performance even in the face of hardware or network failures.通过分布式检查点、冗余硬件配置和动态作业重调度，将容错能力纳入系统设计。该策略可最大限度减少停机时间，即使在硬件或网络故障时也能保持性能。

Compiler and build optimizations编译器与构建优化

Leverage aggressive compiler flags and profile-guided optimizations during the build process to extract maximum performance from your code. Regularly update and tune your build configurations, and verify the impact of each change through rigorous benchmarking to ensure optimal execution.在构建过程中利用激进的编译器标志和性能剖析引导优化（profile-guided optimization），从代码中榨取最大性能。定期更新并调优构建配置，并通过严格的基准测试验证每一项改动的影响，以确保最优执行。

Security, compliance, and performance安全、合规与性能

Integrate and codesign security, compliance, and performance. Regularly audit configurations, enforce access controls, and maintain industry-standard safeguards, including encryption, secure data channels, zero-trust networking, hardware security modules (HSMs), and secure enclaves. And make sure that performance tuning never compromises system security. Similarly, make sure security doesn’t incur unnecessary performance overhead.将安全、合规与性能进行集成与协同设计。定期审计配置、实施访问控制，并维护行业标准的防护措施，包括加密、安全数据通道、零信任网络、硬件安全模块（HSM）和安全飞地。并确保性能调优绝不损害系统安全。同样，确保安全措施不会带来不必要的性能开销。

Comprehensive documentation and knowledge sharing完善的文档与知识共享

Maintain detailed records of all optimization steps, system configurations, and performance benchmarks. Develop an internal knowledge base to facilitate team collaboration and rapid onboarding, ensuring that best practices are preserved and reused across projects.详细记录所有优化步骤、系统配置和性能基准。建立内部知识库以促进团队协作和快速上手，确保最佳实践得以保留并在各项目间复用。

Future-proofing and scalability planning面向未来与可扩展性规划

Design modular, adaptable system architectures that can easily incorporate emerging hardware and software technologies. Continuously evaluate scalability requirements and update your optimization strategies to sustain competitive performance as your workload grows.设计模块化、可适配的系统架构，能够轻松纳入新兴的硬件与软件技术。持续评估可扩展性需求，并更新优化策略，以在工作负载增长时保持有竞争力的性能。

系统架构与硬件规划

System Architecture and Hardware Planning

Your hardware, interconnects, and data paths set the ceiling for performance and cost-efficiency—no software tweak can outrun a starved GPU. Plan for goodput per dollar/watt by matching accelerators, CPU/DRAM/I/O, and cooling/power to the workload to avoid bottlenecks from the start.你的硬件、互连与数据路径决定了性能与成本效益的上限——任何软件层面的微调都无法超越一块处于饥饿状态的 GPU。应以每美元/每瓦有效吞吐为目标进行规划，将加速器、CPU/DRAM/I/O 以及散热/供电与工作负载相匹配，从一开始就避免瓶颈。

Specifically, design for goodput—useful work per dollar/watt—and not just raw FLOPS. Match accelerators and interconnects to workload, right-size CPU/ memory/I/O to keep GPUs fed, keep data local, and plan power/cooling so hardware sustains peak clocks. Evaluate scaling efficiency before adding more GPUs. Here are some tips for optimizing system architecture and improving hardware planning efficiency:具体而言，要面向有效吞吐——即每美元/每瓦的有用工作量——进行设计，而不只是追求原始 FLOPS。让加速器与互连匹配工作负载，合理配置 CPU/内存/I/O 的规模以持续喂饱 GPU，保持数据本地性，并规划供电/散热，使硬件能够维持峰值时钟频率。在增加更多 GPU 之前，先评估扩展效率。以下是优化系统架构、提升硬件规划效率的一些建议：

Design for goodput and efficiency面向有效吞吐与效率进行设计

Treat useful throughput as the goal. Every bit of performance gained translates to massive cost savings at scale. Focus on maximizing productive work per dollar/ watt—and not just raw FLOPS.把有用吞吐量作为目标。所获得的每一点性能提升，在规模化时都会转化为巨大的成本节约。专注于最大化每美元/每瓦的有效工作量——而不只是原始 FLOPS。

Choose the right accelerator选择合适的加速器

Prefer modern GPUs for superior performance-per-watt and memory capacity. Newer architectures offer features like native FP8 and FP4 precision support— along with much faster interconnects. These produce big speedups over older-generation GPUs and systems.优先选择现代 GPU，以获得更优的每瓦性能与显存容量。更新的架构提供原生 FP8 与 FP4 精度支持等特性，并具备快得多的互连。相比老一代 GPU 与系统，这些能带来大幅加速。

Leverage high-bandwidth interconnects利用高带宽互连

Use systems with NVLink/NVSwitch, such as GB200/GB300 NVL72, instead of PCIe-only connectivity for multi-GPU workloads. NVLink 5 provides up to 1.8 TB/s bidirectional GPU-to-GPU bandwidth (over 14× PCIe Gen5), enabling near-linear scaling across GPUs. NVLink Switch domains can be scaled with second-level switches to connect up to 576 GPUs in one NVLink domain. This对于多 GPU 工作负载，应使用具备 NVLink/NVSwitch 的系统（例如 GB200/GB300 NVL72），而不是仅靠 PCIe 连接。NVLink 5 提供高达 1.8 TB/s 的双向 GPU 到 GPU 带宽（超过 PCIe Gen5 的 14 倍），使跨 GPU 的近线性扩展成为可能。NVLink Switch 域可通过二级交换机扩展，在单个 NVLink 域内连接多达 576 块 GPU。这

enables hierarchical collectives that stay on NVLink as long as possible before falling back to the inter-rack fabric.使得分层集合通信尽可能长时间地留在 NVLink 上，然后再回退到跨机架网络架构。

Balance CPU/GPU and memory ratios平衡 CPU/GPU 与内存配比

Provision enough CPU cores, DRAM, and storage throughput per GPU. For example, allocate ~1 fast CPU core per GPU for data loading and networking tasks. Ensure system RAM and I/O can feed GPUs at required rates on the order of hundreds of MB/s per GPU to avoid starvation.为每块 GPU 配置足够的 CPU 核心、DRAM 与存储吞吐能力。例如，为每块 GPU 分配约 1 个高性能 CPU 核心用于数据加载与网络任务。确保系统内存与 I/O 能以每块 GPU 数百 MB/s 量级的所需速率喂饱 GPU，避免出现饥饿。

Plan for data locality规划数据局部性

If training across multiple nodes, minimize off-node communication. Whenever possible, keep tightly coupled workloads on the same NVLink/NVSwitch domain to exploit full bandwidth, and use the highest-speed interconnect that you have access to. Ideally, this is NVLink for intranode and intra-rack communication and InfiniBand for inter-rack communication.如果跨多个节点训练，应尽量减少节点间通信。只要可能，就把紧耦合的工作负载保持在同一个 NVLink/NVSwitch 域内，以充分利用全部带宽，并使用你能获得的最高速互连。理想情况下，节点内与机架内通信用 NVLink，机架间通信用 InfiniBand。

Avoid bottlenecks in the chain避免链路中的瓶颈

Identify the slowest link—be it CPU, memory, disk, or network—and scale it up. For instance, if GPU utilization is low due to I/O, invest in faster storage or caching rather than more GPUs. An end-to-end design where all components are well-matched prevents wasted GPU cycles.找出最慢的一环——无论是 CPU、内存、磁盘还是网络——并对其进行扩容。例如，如果 GPU 利用率因 I/O 而偏低，应投资更快的存储或缓存，而不是增加更多 GPU。让所有组件良好匹配的端到端设计可以避免浪费 GPU 周期。

Choose an appropriate cluster size选择合适的集群规模

Beware of diminishing returns when adding GPUs. Past a certain cluster size, overheads can grow—ensure the speedup justifies the cost. It’s often better to optimize utilization on N GPUs by reaching 95% usage, for example, before scaling to 2*N* GPUs.增加 GPU 时要注意收益递减。超过某个集群规模后，开销可能会增长——确保加速比足以证明成本合理。通常更好的做法是先把 N 块 GPU 的利用率优化到 95%，然后再扩展到 2*N* 块 GPU。

Design for cooling and power针对散热与供电进行设计

Ensure the data center can handle GPU thermal and power needs. High-performance systems like GB200/GB300 have very high TDP. Provide adequate cooling (likely liquid-based) and power provisioning so the GPUs can sustain boost clocks without throttling.确保数据中心能够满足 GPU 的散热与供电需求。GB200/GB300 等高性能系统的 TDP 非常高。提供充足的散热（很可能是液冷）与供电，使 GPU 能够维持加速时钟而不发生降频。

统一 CPU-GPU“超级芯片”架构

Unified CPU-GPU “Superchip” Architecture

Unified memory and on-package links let you fit larger models and cut copy overhead when you place the right data in the right tier. Using Grace for preprocessing and HBM for “hot” tensors turns the superchip into a tightly coupled engine with fewer stalls.统一内存与封装内链路让你在把正确的数据放在正确的层级时，能够容纳更大的模型并减少拷贝开销。用 Grace 做预处理、用 HBM 存放“热”张量，可将超级芯片变成一台停顿更少的紧耦合引擎。

On Grace Blackwell Superchips, treat CPU and GPU as a shared-memory complex. Keep hot weights/activations in HBM and overflow or infrequent data in Grace LPDDR via NVLink-C2C. Use the on-package Grace CPU for preprocessing/在 Grace Blackwell 超级芯片上，应将 CPU 与 GPU 视为一个共享内存复合体。把热权重/激活值保留在 HBM 中，把溢出数据或不常访问的数据通过 NVLink-C2C 放在 Grace LPDDR 中。使用封装内的 Grace CPU 进行预处理/

orchestration and prefetch or pipeline-managed memory to hide latency for ultralarge models. Take advantage of the superchip architecture as follows:编排，并使用预取或流水线管理的内存来为超大模型隐藏时延。按如下方式利用超级芯片架构：

Utilize unified CPU-GPU memory利用统一的 CPU-GPU 显存

Exploit the Grace Blackwell (GB200/GB300) Superchip’s unified memory space. Two Blackwell GPUs and a 72-core Grace CPU share a coherent memory pool with NVLink-C2C (900 GB/s). Use the CPU’s large memory (e.g., 480 GB LPDDR5X) as an extension for oversize models while keeping “hot” data in the GPUs’ HBM for speed.利用 Grace Blackwell（GB200/GB300）超级芯片的统一内存空间。两块 Blackwell GPU 与一颗 72 核 Grace CPU 通过 NVLink-C2C（900 GB/s）共享一个一致性内存池。将 CPU 的大容量内存（例如 480 GB LPDDR5X）用作超大模型的扩展，同时把“热”数据保留在 GPU 的 HBM 中以获得速度。

Place data for locality按局部性放置数据

Even with unified memory, prioritize data placement. Put model weights, activations, and other frequently accessed data on GPU HBM3e (which has much higher local bandwidth), and let infrequently used or overflow data reside in CPU RAM. This ensures the 900 GB/s NVLink-C2C link isn’t a bottleneck for critical data.即使有统一内存，也要优先考虑数据放置。将模型权重、激活值以及其他频繁访问的数据放在 GPU HBM3e 上（其本地带宽高得多），让不常使用或溢出的数据驻留在 CPU RAM 中。这样可确保 900 GB/s 的 NVLink-C2C 链路不会成为关键数据的瓶颈。

Take advantage of CPU-GPU direct memory access when available在可用时利用 CPU-GPU 直接显存访问

Use the GPU’s ability to directly access CPU memory on combined CPU-GPU superchips like the GB200 and GB300. GPUs can read and write Grace LPDDR memory coherently over NVLink-C2C without staging over host PCIe. Bandwidth and latency are still lower than HBM, so prefetch managed pointers, stage data, and pipeline transfers to hide latency. As such, it’s recommended to keep hot activations and KV cache in HBM and use CPU memory as a lower-tier cache with explicit prefetch.在 GB200 和 GB300 等 CPU-GPU 一体化超级芯片上，利用 GPU 直接访问 CPU 内存的能力。GPU 可以通过 NVLink-C2C 一致地读写 Grace LPDDR 内存，而无需经由主机 PCIe 暂存。带宽和时延仍低于 HBM，因此要预取托管指针、暂存数据并对传输进行流水线化，以隐藏时延。因此，建议将热激活值和 KV 缓存保留在 HBM 中，并将 CPU 内存用作更低层级的缓存，配合显式预取。

Use the Grace CPU effectively高效利用 Grace CPU

The on-package Grace CPU provides 72 high-performance cores—utilize them! Offload data preprocessing, augmentation, and other CPU-friendly tasks to these cores. They can feed the GPUs quickly using NVLink-C2C, essentially acting as an extremely fast I/O and compute companion for the GPU.封装内的 Grace CPU 提供 72 个高性能核心——充分利用它们！将数据预处理、数据增强以及其他适合 CPU 的任务卸载到这些核心上。它们可以通过 NVLink-C2C 快速为 GPU 供给数据，实质上充当 GPU 的极速 I/O 与计算伙伴。

Plan for ultralarge models为超大规模模型做好规划

For trillion-parameter model training that exceeds GPU memory, GB200/GB300 systems allow you to train using CPU memory as part of the model’s memory pool. Prefer framework caching allocators and use cudaMallocAsync in custom code to minimize fragmentation and enable graph capture. Use CUDA Unified Memory or managed memory APIs to handle overflow gracefully, and consider explicit prefetching (e.g., cudaMemPrefetchAsync) of upcoming layers from CPU → GPU memory to hide latency.对于超出 GPU 显存的万亿参数模型训练，GB200/GB300 系统允许将 CPU 内存作为模型显存池的一部分来训练。优先使用框架的缓存分配器，并在自定义代码中使用 cudaMallocAsync，以尽量减少碎片并支持 CUDA 图捕获。使用 CUDA 统一内存或托管内存 API 来优雅地处理溢出，并考虑对即将到来的层进行显式预取（例如 cudaMemPrefetchAsync），从 CPU 内存 → GPU 显存，以隐藏时延。

Consider superchip-optimized algorithms考虑针对超级芯片优化的算法

SuperOffload is an example of a superchip-optimized set of algorithms focused on improving efficiency of offload and tensor cast/copy strategies. Innovations include speculation-then-validation (STV), heterogeneous optimizer computation, and anSuperOffload 就是一组针对超级芯片优化的算法示例，专注于提升卸载以及张量类型转换/拷贝策略的效率。其创新包括先推测后验证（speculation-then-validation，STV）、异构优化器计算，以及一个

ARM-based CPU optimizer. Designed specifically for NVIDIA superchips (e.g., Grace Hopper, Grace Blackwell, Vera Rubin), SuperOffload increases token-processing throughput and chip utilization relative to traditional offload strategies.基于 ARM 的 CPU 优化器。SuperOffload 专为 NVIDIA 超级芯片（例如 Grace Hopper、Grace Blackwell、Vera Rubin）设计，相对于传统卸载策略，可提升 token 处理吞吐量和芯片利用率。

多 GPU 扩展与互连优化

Multi-GPU Scaling and Interconnect Optimizations

Scaling pays only when communication is fast and topology-aware—otherwise added GPUs just wait on one another. Lean on NVLink/NVSwitch bandwidth, modern collectives, and fabric-aware placement to approach linear speedups.只有当通信快速且拓扑感知时，扩展才有收益——否则新增的 GPU 只会相互等待。依托 NVLink/NVSwitch 带宽、现代集合通信以及感知网络结构的放置策略，才能接近线性加速。

Specifically, exploit NVLink/NVSwitch domains (e.g., NVL72) for near-linear scaling, and choose parallelism strategies that fit the fabric. Use topology-aware placement, updated NCCL collectives (e.g., PAT), and telemetry to verify you’re using the ~1.8 TB/s bidirectional throughput per-GPU bandwidth effectively. Plan hierarchical communications as you expand. The following are tips on utilizing multi-GPU scaling through interconnect and topology optimizations:具体而言，利用 NVLink/NVSwitch 域（例如 NVL72）实现接近线性的扩展，并选择与网络结构相匹配的并行策略。使用拓扑感知的放置、更新后的 NCCL 集合通信（例如 PAT）以及遥测数据，来验证你是否有效利用了每 GPU 约 1.8 TB/s 的双向吞吐带宽。随着规模扩大，要规划分层通信。以下是关于通过互连与拓扑优化来利用多 GPU 扩展的提示：

Design for high-speed all-to-all topology面向高速全对全拓扑进行设计

On NVL72 NVSwitch clusters with 72 fully interconnected GPUs, for example, any GPU can communicate with any other at full NVLink 5 speed. At the fabric level, the NVLink Switch domain is nonblocking. Application-level throughput can vary with concurrent traffic and path scheduling, so verify behavior using DCGM NVLink counters and Nsight Systems traces before assuming per-pair saturation. Take advantage of this topology by using parallelization strategies, such as data parallel, tensor parallel, and pipeline parallelism, that would be bottlenecked on lesser interconnects.例如，在拥有 72 个全互连 GPU 的 NVL72 NVSwitch 集群上，任意 GPU 都能以完整的 NVLink 5 速度与任何其他 GPU 通信。在网络结构层面，NVLink Switch 域是无阻塞的。应用层吞吐量会随并发流量和路径调度而变化，因此在假定任意一对 GPU 都能饱和之前，请使用 DCGM NVLink 计数器和 Nsight Systems 跟踪来验证实际行为。利用这一拓扑，采用那些在较低规格互连上会成为瓶颈的并行化策略，例如数据并行、张量并行和流水线并行。

Utilize topology-aware scheduling采用拓扑感知调度

Always colocate multi-GPU jobs within an NVLink Switch domain if possible. Keeping all GPUs of a job on the NVL72 fabric means near-linear scaling for communication-heavy workloads. Mixing GPUs across NVLink domains or standard networks will introduce bottlenecks and should be avoided for tightly coupled tasks.尽可能将多 GPU 作业放在同一个 NVLink Switch 域内。把作业的所有 GPU 都保留在 NVL72 互联结构上，意味着通信密集型工作负载可以获得接近线性的扩展。将 GPU 跨 NVLink 域或跨标准网络混布会引入瓶颈，对于紧耦合任务应避免这种做法。

Leverage unprecedented bandwidth利用前所未有的带宽

Recognize that NVLink 5 has 900 GB/s per GPU in each direction, which doubles the per-GPU bandwidth versus the previous generation. An NVL72 rack provides ~130 TB/s total intra-rack bandwidth in aggregate. This drastically reduces communication wait times, as even tens of gigabytes of gradient data can be all-reduced in a few milliseconds at 1.8 TB/s. Design training algorithms, such as gradient synchronization and parameter sharding, to fully exploit this relatively free communication budget.要认识到 NVLink 5 为每个 GPU 提供每个方向 900 GB/s 的带宽，使单 GPU 带宽相比上一代翻倍。一个 NVL72 机架在机架内提供约 130 TB/s 的总聚合带宽。这大幅缩短了通信等待时间：即便是数十 GB 的梯度数据，在 1.8 TB/s 下也只需几毫秒即可完成全归约。设计训练算法（例如梯度同步与参数分片）时，应充分利用这份相对“免费”的通信预算。

Embrace modern collective algorithms拥抱现代集合通信算法

Use the latest NVIDIA NCCL library optimized for NVSwitch. Specifically, enable the parallel aggregated tree (PAT) algorithm, which was introduced for NVLink Switch topologies. This further reduces synchronization time by taking advantage of the NVL72 topology to perform reductions more efficiently than other tree/ring algorithms.使用针对 NVSwitch 优化的最新 NVIDIA NCCL 库。具体而言，应启用为 NVLink Switch 拓扑引入的并行聚合树（parallel aggregated tree，PAT）算法。该算法利用 NVL72 拓扑以比其他树形/环形算法更高效的方式执行归约，从而进一步缩短同步时间。

Consider fine-grained parallelism考虑细粒度并行

With full-bandwidth all-to-all connectivity, consider fine-grained model parallelism that wasn’t feasible before. For example, layer-wise parallelism or tensor parallelism across many GPUs can be efficient when each GPU has 1.8 TB/s bidirectional throughput to every other. Previously, one might avoid excessive cross-GPU communication, but NVL72 allows aggressive partitioning of work without hitting network limits.在全带宽的全互连（all-to-all）连接下，可以考虑以往不可行的细粒度模型并行。例如，当每个 GPU 与其他每个 GPU 之间都有 1.8 TB/s 的双向吞吐时，跨多个 GPU 的逐层并行或张量并行可以很高效。过去人们可能会避免过多的跨 GPU 通信，而 NVL72 允许对工作进行激进划分而不触及网络上限。

Monitor for saturation监控饱和情况

Although NVL72 is extremely fast, keep an eye on link utilization in profiling. If your application somehow saturates the NVSwitch using extreme all-to-all operations, for example, you might need to throttle communication by aggregating gradients, etc. Use NVIDIA’s tools or NVSwitch telemetry to verify that communications are within the NVLink capacity, and adjust patterns if needed. For instance, you can stagger all-to-all exchanges to avoid network contention. DCGM exposes NVLink counters that can help verify link balance and detect hotspots during collectives.尽管 NVL72 极快，在性能剖析中仍要关注链路利用率。例如，如果你的应用以极端的全互连（all-to-all）操作把 NVSwitch 打满，可能就需要通过聚合梯度等方式来限制通信。使用 NVIDIA 的工具或 NVSwitch 遥测来确认通信处于 NVLink 容量之内，并在需要时调整通信模式。例如，可以错开全互连交换以避免网络争用。DCGM 暴露了 NVLink 计数器，有助于在集合通信期间验证链路均衡并检测热点。

Plan for future expansion为未来扩展做好规划

Be aware that NVLink Switch can scale beyond a single rack—up to 576 GPUs in one connected domain using second-level switches. If you operate at that ultrascale, plan hierarchical communication using local NVL72 inter-rack collectives first, then use inter-rack interconnects only when necessary. This helps to maximize intra-rack NVLink usage first. This ensures you’re using the fastest links before resorting to inter-rack InfiniBand hops.要注意 NVLink Switch 可以扩展到单机架之外——借助二级交换机，单个连通域最多可容纳 576 个 GPU。如果你在该超大规模下运行，应规划分层通信：先做本地 NVL72 的机架内集合通信，仅在必要时才使用机架间互连。这有助于优先最大化机架内 NVLink 的使用，确保在退回到机架间 InfiniBand 跳转之前先用上最快的链路。

Identify opportunities for federated and distributed optimizations识别联邦式与分布式优化的机会

For deployments that span heterogeneous environments, such as multicloud or edge-to-cloud setups, adopt adaptive communication protocols and dynamic load balancing strategies. This minimizes latency and maximizes throughput across distributed systems, which ensures robust performance even when resources vary in capability and capacity.对于跨异构环境的部署，例如多云或边缘到云的架构，应采用自适应通信协议与动态负载均衡策略。这可以最小化分布式系统中的时延并最大化吞吐量，从而在资源能力与容量参差不齐时仍能保证稳健的性能。

操作系统与驱动优化

Operating System and Driver Optimizations

OS jitter, NUMA misses, and driver mismatches quietly drain throughput and create variability you can’t tune around. Hardening the stack (huge pages, affinities, consistent CUDA/driver, persistence) creates a stable, high-performance baseline.操作系统抖动、NUMA 未命中以及驱动不匹配会悄无声息地消耗吞吐量，并带来无法通过调优消除的波动性。加固整个软件栈（大页、亲和性、一致的 CUDA/驱动、持久化模式）可建立稳定、高性能的基线。

Run a lean, HPC-tuned Linux. Set NUMA/IRQ affinities and enable THP and high memlock. Keep NVIDIA drivers/CUDA consistent across nodes. Isolate system jitter, tune CPU libraries/storage, set container limits correctly, and keep BIOS/firmware/ NVSwitch fabric up to date for predictable throughput. Here are some host, OS, and container optimizations that you should explore in your environment:运行精简且针对 HPC 调优的 Linux。设置 NUMA/IRQ 亲和性，并启用 THP 和较高的 memlock 上限。保持各节点上的 NVIDIA 驱动/CUDA 版本一致。隔离系统抖动、调优 CPU 库/存储、正确设置容器限制，并保持 BIOS/固件/NVSwitch 网络结构为最新版本，以获得可预测的吞吐量。以下是一些你应在自己的环境中探索的主机、操作系统与容器优化项：

Use a Linux kernel tuned for HPC使用针对 HPC 调优的 Linux 内核

Ensure your GPU servers run a recent, stable Linux kernel configured for high-performance computing. Disable unnecessary background services that consume CPU or I/O. Use the “performance” CPU governor—versus “on-demand” or “power-save”—to keep CPU cores at a high clock for feeding GPUs.确保你的 GPU 服务器运行较新且稳定的、针对高性能计算配置的 Linux 内核。禁用消耗 CPU 或 I/O 的不必要后台服务。使用“performance”CPU 调频策略——而非“on-demand”或“power-save”——使 CPU 核心保持高主频，为 GPU 供给数据。

Disable swap for performance-critical workloads为性能关键型工作负载禁用 swap

Disable swap on training servers to avoid page thrashing, or, if swap must remain enabled, lock critical buffers using mlock or cudaHostAlloc to ensure they stay in RAM.在训练服务器上禁用 swap 以避免页面颠簸；或者，如果必须保留 swap，则使用 mlock 或 cudaHostAlloc 锁定关键缓冲区，确保它们常驻内存。

Avoid memory fragmentation with aggressive preallocation通过激进预分配避免显存碎片

Preallocate large, contiguous blocks of memory for frequently used tensors to reduce runtime allocation overhead and fragmentation. This proactive strategy ensures more stable and efficient memory management during long training runs.为频繁使用的张量预分配大块连续内存，以减少运行时分配开销和碎片。这种主动策略可在长时间训练运行期间确保更稳定、更高效的内存管理。

Optimize environment variables for CPU libraries为 CPU 库优化环境变量

Fine-tune parameters, such as OMP_NUM_THREADS and MKL_NUM_THREADS, to better match your hardware configuration. Adjusting these variables can reduce thread contention and improve the parallel efficiency of CPU-bound operations.微调 OMP_NUM_THREADS 和 MKL_NUM_THREADS 等参数，使其更好地匹配你的硬件配置。调整这些变量可减少线程争用，并提升 CPU 密集型操作的并行效率。

Design for NUMA awareness面向 NUMA 感知进行设计

For multi-NUMA servers, pin GPU processes/threads to the CPU of the local NUMA node. Use tools like numactl or taskset to bind each training process to the CPU nearest its assigned GPU. Similarly, bind memory allocations to the local NUMA node (numactl --membind) so host memory for GPU DMA comes from the closest RAM. This avoids costly cross-NUMA memory traffic that can halve effective PCIe/NVLink bandwidth.对于多 NUMA 服务器，将 GPU 进程/线程绑定到本地 NUMA 节点的 CPU 上。使用 numactl 或 taskset 等工具，将每个训练进程绑定到离其分配 GPU 最近的 CPU。同样，将显存分配绑定到本地 NUMA 节点（numactl --membind），使供 GPU DMA 使用的主机内存来自最近的 RAM。这样可避免代价高昂的跨 NUMA 内存流量——它可能使有效 PCIe/NVLink 带宽减半。

Utilize IRQ affinity for network and GPU tasks利用 IRQ 亲和性处理网络与 GPU 任务

Explicitly bind NIC interrupts to CPU cores on the same NUMA node as the NIC, and similarly pin GPU driver threads to dedicated cores—including those from long-running services like the nvidia-persistence service daemon. This strategy minimizes cross-NUMA traffic and stabilizes performance under heavy loads.将 NIC 中断显式绑定到与 NIC 处于同一 NUMA 节点的 CPU 核心上，同样地，将 GPU 驱动线程固定到专用核心——包括来自 nvidia-persistence 服务守护进程等长期运行服务的线程。该策略可最大限度减少跨 NUMA 流量，并在高负载下稳定性能。

Enable transparent hugepages启用透明大页

Turn on transparent hugepages (THP) in always or madvise mode so that large memory allocations use 2 MB pages. This reduces TLB thrashing and kernel overhead when allocating tens or hundreds of GBs of host memory for frameworks. Verify THP is active by checking for /sys/kernel/mm/transparent_huge page/enabled. With THP enabled, your processes are using hugepages for big allocations. Prefer THP in madvise mode if your workload is latency-critical and you observe jitter.将透明大页（THP）设为 always 或 madvise 模式开启，使大块显存分配使用 2 MB 页。当为框架分配数十或数百 GB 的主机内存时，这可减少 TLB 抖动与内核开销。通过检查 /sys/kernel/mm/transparent_huge page/enabled 确认 THP 已启用。启用 THP 后，进程的大块分配即会使用大页。若工作负载对时延敏感且观察到抖动，优先使用 madvise 模式的 THP。

Increase max locked memory提高最大锁定内存上限

Configure the OS to allow large pinned (aka page-locked) allocations. GPU apps often pin memory for faster transfers—set ulimit -l unlimited or a high value so your data loaders can allocate pinned buffers without hitting OS limits. This prevents failures or fallbacks to pageable memory, which would slow down GPU DMA.配置操作系统以允许大型锁页（又称页锁定）分配。GPU 应用常锁定内存以实现更快的传输——设置 ulimit -l unlimited 或一个较高的值，使数据加载器能够分配锁页缓冲区而不触及操作系统限制。这可避免失败或回退到可分页内存，后者会拖慢 GPU DMA。

Use the latest NVIDIA driver and CUDA stack使用最新的 NVIDIA 驱动与 CUDA 软件栈

Keep NVIDIA drivers and CUDA runtime up-to-date (within a tested stable version) on all nodes. New drivers can bring performance improvements and are required for new GPUs’ compute capabilities. Make sure all nodes have the same driver/CUDA versions to avoid any mismatches in multinode jobs. Enable persistence mode on GPUs at boot (nvidia-smi -pm 1) so the driver stays loaded and GPUs don’t incur re-init delays. Update the NVIDIA driver and toolkit on all nodes to inherit bug fixes and performance improvements.在所有节点上保持 NVIDIA 驱动与 CUDA 运行时为最新（处于经过测试的稳定版本）。新驱动可带来性能改进，并且是新 GPU 计算能力的必要条件。确保所有节点使用相同的驱动/CUDA 版本，以避免多节点作业中出现任何不匹配。在启动时对 GPU 启用持久化模式（nvidia-smi -pm 1），使驱动保持加载、GPU 不会产生重新初始化延迟。在所有节点上更新 NVIDIA 驱动与工具包，以继承缺陷修复与性能改进。

Enable GPU persistence when using a MIG configuration使用 MIG 配置时启用 GPU 持久化

With persistence mode enabled, the GPU remains “warm” and ready to use, reducing startup latency for jobs. This is especially crucial if using a Multi-Instance GPU (MIG) partitioning—without persistence, MIG configurations would reset on every job, but keeping the driver active preserves the slices. Always configure persistence mode when using MIG.启用持久化模式后，GPU 保持“热”状态并随时可用，从而降低作业启动时延。若使用多实例 GPU（MIG）分区，这一点尤为关键——若不启用持久化，MIG 配置会在每次作业时重置，而保持驱动活跃则可保留各切片。使用 MIG 时务必配置持久化模式。

Isolate system tasks隔离系统任务

Dedicate a core—or small subset of cores—on each server for OS housekeeping, such as interrupt handling and background daemons. This way, your main CPU threads feeding the GPU are not interrupted. This can be done using CPU isolation or cgroup pinning. Eliminating OS jitter ensures consistent throughput.在每台服务器上划出一个核心——或一小组核心——专用于操作系统日常事务，例如中断处理与后台守护进程。这样，为 GPU 供给数据的主 CPU 线程就不会被打断。可通过 CPU 隔离或 cgroup 绑核实现。消除操作系统抖动可确保吞吐量稳定。

Optimize system I/O settings优化系统 I/O 设置

If your workload does a lot of logging or checkpointing, mount filesystems with options that favor throughput. Consider using noatime for data disks and increase filesystem read-ahead for streaming reads. Ensure the disk scheduler is set appropriately to use mq-deadline or noop for NVMe SSDs to reduce latency variability.若工作负载涉及大量日志记录或检查点写入，请以偏向吞吐量的选项挂载文件系统。考虑对数据盘使用 noatime，并为流式读取增大文件系统预读。确保磁盘调度器设置得当，对 NVMe SSD 使用 mq-deadline 或 noop，以降低时延波动。

Perform regular maintenance执行定期维护

Keep BIOS/firmware updated for performance fixes. Some BIOS updates improve PCIe bandwidth or fix input–output memory management unit (IOMMU) issues for GPUs. Also, periodically check for firmware updates for NICs and NVSwitch/Fabric if applicable, as provided by NVIDIA, such as Fabric Manager upgrades, etc. Minor firmware tweaks can sometimes resolve obscure bottlenecks or reliability issues.保持 BIOS/固件更新以获取性能修复。某些 BIOS 更新可提升 PCIe 带宽，或修复 GPU 的输入输出内存管理单元（IOMMU）问题。此外，若适用，应定期检查 NVIDIA 提供的网卡与 NVSwitch/Fabric 固件更新，例如 Fabric Manager 升级等。细微的固件调整有时能解决难以定位的瓶颈或可靠性问题。

Tune Docker and Kubernetes configurations for maximum performance调整 Docker 与 Kubernetes 配置以获得最大性能

When running in containers, add options, such as --ipc=host for shared memory, and set --ulimit memlock=-1 to prevent memory locking issues. This guarantees that your containerized processes access memory without OS-imposed restrictions.在容器中运行时，添加相应选项，例如用于共享内存的 --ipc=host，并设置 --ulimit memlock=-1 以避免内存锁定问题。这可确保容器化进程访问内存时不受操作系统施加的限制。

GPU 资源管理与调度

GPU Resource Management and Scheduling

Smarter placement and partitioning raise utilization without buying new hardware— and protect predictability for mixed workloads. Respect topology, use MPS/MIG where appropriate, and control clocks/power to minimize contention and tail latency.更智能的放置与分区可在不购置新硬件的前提下提升利用率，并为混合负载保障可预测性。遵循拓扑结构，在适当场景使用 MPS/MIG，并控制时钟/功耗以尽量减少争用与尾时延。

Schedule with GPU/NUMA/NVLink topology in mind, and use MPS or MIG to raise utilization for smaller jobs while retaining ECC and persistence for reliability. Lock clocks or power limit for stability when needed, avoid CPU oversubscription, and pack jobs intelligently to maximize ROI without contention. Here are some GPU resource management and scheduling tips:调度时考虑 GPU/NUMA/NVLink 拓扑，并使用 MPS 或 MIG 提升较小作业的利用率，同时保留 ECC 与持久化模式以保障可靠性。需要时锁定时钟或限制功耗以保证稳定性，避免 CPU 超额订阅，并智能地打包作业，在无争用的前提下最大化投资回报。以下是一些 GPU 资源管理与调度技巧：

Topology-aware job scheduling拓扑感知的作业调度

Ensure that orchestrators like Kubernetes and SLURM are scheduling containers on nodes that respect NUMA and NVLink boundaries to minimize cross-NUMA and cross-NVLink-domain memory accesses. This alignment reduces latency and improves overall throughput.确保 Kubernetes、SLURM 等编排器将容器调度到遵循 NUMA 与 NVLink 边界的节点上，以尽量减少跨 NUMA 与跨 NVLink 域的显存访问。这种对齐可降低时延并提升整体吞吐量。

Multi-Process Service (MPS)多进程服务（MPS）

Enable NVIDIA MPS when running multiple processes on a single GPU to improve utilization. MPS allows kernels from different processes to execute concurrently on the GPU instead of time-slicing. This is useful if individual jobs在单个 GPU 上运行多个进程时启用 NVIDIA MPS 以提升利用率。MPS 允许来自不同进程的内核在 GPU 上并发执行，而非分时执行。当单个作业

don’t fully saturate the GPU—for example, running 4 training tasks on one GPU with MPS can overlap their work and boost overall throughput.无法完全占满 GPU 时，这很有用——例如，使用 MPS 在一个 GPU 上运行 4 个训练任务，可以让它们的工作相互重叠，从而提升整体吞吐量。

Multi-Instance GPU (MIG)多实例 GPU（MIG）

Use MIG to partition high-end GPUs into smaller instances for multiple jobs. If you have many light workloads like inferencing small models or running many experiments, you can slice a GPU to ensure guaranteed resources for each job. For instance, modern GPUs can be split into multiple MIG slices (up to 7). Do not use MIG for tightly coupled parallel jobs, as those benefit from full GPU access. Deploy MIG for isolation and maximizing GPU ROI when jobs are smaller than a full GPU.使用 MIG 将高端 GPU 划分为更小的实例以承载多个作业。如果你有许多轻量负载，例如推理小模型或运行大量实验，可以切分 GPU 以确保每个作业获得有保障的资源。例如，现代 GPU 可被切分为多个 MIG 切片（最多 7 个）。不要将 MIG 用于紧耦合的并行作业，因为这类作业受益于对整块 GPU 的访问。当作业规模小于整块 GPU 时，部署 MIG 以实现隔离并最大化 GPU 投资回报。

Persistence for MIGMIG 的持久化

Keep persistence mode on to maintain MIG partitions between jobs. This avoids repartitioning overhead and ensures subsequent jobs see the expected GPU slices without delay. Configure MIG at cluster boot and leave it enabled so that scheduling is predictable, as changing MIG config on the fly requires resetting the GPU, which can disrupt running jobs. Plan for maintenance windows as MIG device partitions are not persisted by the GPU across reboot. Use NVIDIA’s MIG Manager to automatically recreate the desired layout on boot.保持持久化模式开启，以便在作业之间维持 MIG 分区。这可以避免重新分区的开销，并确保后续作业无需等待即可看到预期的 GPU 切片。在集群启动时配置 MIG 并保持启用，从而使调度可预测，因为动态更改 MIG 配置需要重置 GPU，可能中断正在运行的作业。由于 GPU 在重启后不会保留 MIG 设备分区，需要规划维护窗口。使用 NVIDIA 的 MIG Manager 在启动时自动重建所需布局。

GPU clock and power settingsGPU 时钟与功耗设置

Consider locking GPU clocks to a fixed high frequency with nvidia-smi -lgc/ -lmc if you need run-to-run consistency. By default, GPUs use auto boost, which is usually optimal, but fixed clocks can avoid any transient downclocking. In power-constrained scenarios, you might slightly underclock or set a power limit to keep GPUs in a stable thermal/power envelope—this can yield consistent performance if occasional throttling was an issue.如果需要逐次运行的一致性，可以考虑使用 nvidia-smi -lgc/-lmc 将 GPU 时钟锁定在固定的高频率。默认情况下，GPU 使用自动加速，这通常是最优的，但固定时钟可以避免任何瞬时的降频。在功耗受限的场景中，可以略微降频或设置功耗上限，使 GPU 保持在稳定的热/功耗包络内——如果偶发的降频曾是个问题，这可以带来一致的性能。

ECC memoryECC 内存

Keep ECC enabled on data center GPUs for reliability unless you have a specific reason to disable it. The performance cost is minimal—on the order of a few percent loss in bandwidth and memory—but ECC catches memory errors that could otherwise corrupt a long training job. Most server GPUs ship with ECC on by default. Leave it on to safeguard multiweek training.除非有特定理由禁用，否则在数据中心 GPU 上保持 ECC 启用以确保可靠性。性能代价很小——大约百分之几的带宽和内存损失——但 ECC 能捕获内存错误，否则这些错误可能损坏长时间的训练作业。大多数服务器 GPU 默认开启 ECC。保持开启以保障持续数周的训练。

Job scheduler awareness作业调度器感知

Integrate GPU topology into your job scheduler, such as SLURM and Kubernetes. Configure the scheduler to allocate jobs on the same node or same NVSwitch group when low-latency coupling is needed. Use Kubernetes device plugins or SLURM Gres to schedule MIG slices for smaller jobs. A GPU-aware scheduler prevents scenarios like a single job spanning distant GPUs and suffer-ing bandwidth issues.将 GPU 拓扑集成到作业调度器中，例如 SLURM 和 Kubernetes。当需要低时延耦合时，配置调度器将作业分配到同一节点或同一 NVSwitch 组。使用 Kubernetes 设备插件或 SLURM Gres 为较小的作业调度 MIG 切片。GPU 感知的调度器可以避免诸如单个作业跨越相距较远的 GPU 而遭受带宽问题的情况。

CPU oversubscriptionCPU 超额订阅

When scheduling jobs, account for the CPU needs of each GPU task, such as data loading threads, etc. Don’t pack more GPU jobs on a node than the CPUs can handle. It’s better to leave a GPU idle than to overload the CPU such that all GPUs become underfed. Monitor CPU utilization per GPU job to inform scheduling decisions.调度作业时，要考虑每个 GPU 任务的 CPU 需求，例如数据加载线程等。不要在节点上堆积超过 CPU 处理能力的 GPU 作业。宁可让一个 GPU 空闲，也不要让 CPU 过载以致所有 GPU 都供给不足。监控每个 GPU 作业的 CPU 利用率，以指导调度决策。

Use NVIDIA Fabric Manager for NVSwitch为 NVSwitch 使用 NVIDIA Fabric Manager

On systems with NVSwitch, the GB200/GB300 NVL72 racks ensure NVIDIA Fabric Manager is running. It manages the NVSwitch topology and routing. Without it, multi-GPU communication might not be fully optimized or could even fail for large jobs. The Fabric Manager service typically runs by default on NVSwitch-equipped servers, but you should double-check that it’s enabled and running—especially after driver updates.在配备 NVSwitch 的系统上，GB200/GB300 NVL72 机架要确保 NVIDIA Fabric Manager 正在运行。它管理 NVSwitch 拓扑与路由。没有它，多 GPU 通信可能无法完全优化，甚至对大型作业可能失败。Fabric Manager 服务通常在配备 NVSwitch 的服务器上默认运行，但你应再次确认它已启用并正在运行——尤其是在驱动更新之后。

Job packing for utilization通过作业打包提升利用率

Maximize utilization by intelligently packing jobs. For example, on a 4-GPU node, if you have two 2-GPU jobs that don’t use much CPU, running them together on the same node can save resources and even use the faster NVLink for communication if running together inside the same compute node or NVLink-enabled rack. Conversely, avoid colocating jobs that collectively exceed the memory or I/O capacity of the node. The goal is high hardware utilization without contention.通过智能地打包作业来最大化利用率。例如，在一个 4-GPU 节点上，如果你有两个不占用太多 CPU 的 2-GPU 作业，将它们一起运行在同一节点上可以节省资源，并且如果它们运行在同一计算节点或支持 NVLink 的机架内，甚至可以使用更快的 NVLink 进行通信。反之，避免将合计超出节点内存或 I/O 容量的作业放在一起。目标是在无争用的情况下实现高硬件利用率。

I/O 优化

I/O Optimization

If data can’t keep up, GPUs idle—often the largest, cheapest speedups come from fixing input, not math. Parallelism, pinned memory, async transfers, and fast storage ensure the model is continuously fed.如果数据供给跟不上，GPU 就会空闲——通常最大、最廉价的加速来自修复输入，而非数学计算。并行化、锁页内存、异步传输和快速存储可确保模型持续获得数据供给。

Keep the GPUs fed by parallelizing data loaders, using pinned memory and async transfers, and storing data on fast NVMe—preferably with GPUDirect Storage. Stripe, cache, and compress wisely. Measure end-to-end throughput so I/O scales with cluster size, and write checkpoints/logs asynchronously. Here are some tips on I/O optimizations for your data pipeline:通过并行化数据加载器、使用锁页内存与异步传输，以及将数据存放在快速 NVMe 上（最好配合 GPUDirect Storage），让 GPU 始终有数据可算。合理地进行条带化、缓存与压缩。测量端到端吞吐量，使 I/O 能随集群规模扩展，并异步写入检查点/日志。以下是关于数据管道 I/O 优化的一些建议：

Load data in parallel并行加载数据

Use multiple workers/threads to load and preprocess data for the GPUs. The default of one to two data loader workers may be insufficient. Profile and increase the number of data loader processes/threads using PyTorch Data Loader(num_workers=N), for example, until the data input is no longer the bottleneck. High core-count CPUs exist to feed those GPUs, so make sure you utilize them.使用多个工作进程/线程为 GPU 加载和预处理数据。默认的一到两个数据加载器工作进程可能不够。用性能剖析器测量，并通过 PyTorch 的 Data Loader(num_workers=N) 等方式增加数据加载器进程/线程数，直到数据输入不再是瓶颈。高核心数的 CPU 正是为了给这些 GPU 供数据而存在的，务必充分利用它们。

Pin host memory for I/O为 I/O 锁定主机内存

Enable pinned (aka page-locked) memory for data transfer buffers. Many frameworks have an option like PyTorch’s pin_memory=True for its DataLoader to allocate host memory that the GPU can DMA from directly. Using pinned memory significantly improves H2D copy throughput. Combine this with asynchronous transfers to overlap data loading with computation.为数据传输缓冲区启用锁页（pinned，又称 page-locked）内存。许多框架都提供类似 PyTorch DataLoader 的 pin_memory=True 选项，用于分配 GPU 可直接 DMA 访问的主机内存。使用锁页内存可显著提升 H2D 拷贝吞吐量。将其与异步传输结合，可使数据加载与计算重叠。

Overlap compute and data transfers重叠计算与数据传输

Pipeline your input data. While the GPU is busy computing on batch N, load and prepare batch N+1 on the CPU and transfer it in the background using CUDA streams and nonblocking cudaMemcpyAsync. This double buffering hides latency—the GPU ideally never waits for data. Ensure your training loop uses asynchronous transfers. For example, in PyTorch, you can copy tensors to GPU with non_blocking=True. Asynchronous transfer allows the CPU to continue running while the data transfer is in progress in the background. This will improve performance by overlapping computation with data transfer.对输入数据进行流水线化。当 GPU 忙于计算第 N 个批次时，在 CPU 上加载并准备第 N+1 个批次，并使用 CUDA 流和非阻塞 cudaMemcpyAsync 在后台传输。这种双缓冲可隐藏时延——理想情况下 GPU 永远不必等待数据。确保训练循环使用异步传输。例如，在 PyTorch 中可以用 non_blocking=True 将张量拷贝到 GPU。异步传输允许 CPU 在数据传输于后台进行时继续运行。这将通过重叠计算与数据传输来提升性能。

Use fast storage (NVMe/SSD)使用快速存储（NVMe/SSD）

Store training data on fast local NVMe SSDs or a high-performance parallel filesystem. Spinning disks will severely limit throughput. If available, enable GPU-Direct Storage (GDS) so that GPUs can stream data directly from NVMe or network storage—bypassing the CPU. This further reduces I/O latency and CPU load when reading large datasets. For large datasets, consider each node having a local copy or shard of the data. If using network storage, prefer a distributed filesystem like Lustre with striping or an object store that can serve many clients in parallel.将训练数据存放在快速的本地 NVMe SSD 或高性能并行文件系统上。机械硬盘会严重限制吞吐量。如果可用，启用 GPU-Direct Storage（GDS），使 GPU 能够绕过 CPU 直接从 NVMe 或网络存储流式读取数据。这在读取大型数据集时进一步降低 I/O 时延和 CPU 负载。对于大型数据集，可考虑让每个节点持有数据的本地副本或分片。如果使用网络存储，优先选择支持条带化的分布式文件系统（如 Lustre），或能够并行服务众多客户端的对象存储。

Tune I/O concurrency and striping调优 I/O 并发与条带化

Avoid bottlenecks from single-file access. If one large file is used by all workers, stripe it across multiple storage targets or split it into chunks so multiple servers can serve it. For instance, break datasets into multiple files and have each data loader worker read different files simultaneously. This maximizes aggregate bandwidth from the storage system.避免单文件访问造成的瓶颈。如果所有工作进程都使用同一个大文件，可将其条带化到多个存储目标上，或切分成多个块，以便多台服务器共同提供。例如，将数据集拆分为多个文件，让每个数据加载器工作进程同时读取不同的文件。这样可最大化存储系统的聚合带宽。

Optimize small files access优化小文件访问

If your dataset consists of millions of small files, mitigate metadata overhead. Opening too many small files per second can overwhelm the filesystem’s metadata server. Solutions pack small files into larger containers, such as tar or RecordIO files; use data ingestion libraries that batch reads; or ensure metadata caching is enabled on clients. This reduces per-file overhead and speeds up epoch start times.如果数据集由数百万个小文件组成，需缓解元数据开销。每秒打开过多小文件会压垮文件系统的元数据服务器。解决方案包括：将小文件打包进更大的容器，如 tar 或 RecordIO 文件；使用批量读取的数据摄取库；或确保客户端启用了元数据缓存。这可减少单文件开销，并加快训练轮次（epoch）的启动时间。

Use client-side caching when available在可用时使用客户端缓存

Take advantage of any caching layer. If using NFS, increase the client cache size and duration. For distributed filesystems, consider a caching daemon or even manually caching part of the dataset on a local disk. The goal is to avoid repeatedly reading the same data from a slow source. If each node processes the same files at different times, a local cache can drastically cut redundant I/O.利用任何缓存层。如果使用 NFS，则增大客户端缓存大小和持续时间。对于分布式文件系统，可考虑使用缓存守护进程，甚至手动将部分数据集缓存到本地磁盘。目标是避免反复从慢速数据源读取相同数据。如果每个节点在不同时间处理相同的文件，本地缓存可以大幅减少冗余 I/O。

Compress data wisely明智地压缩数据

Store the dataset compressed if I/O is the bottleneck, but use lightweight compression, such as LZ4 or Zstd fast mode. This trades some CPU to reduce I/O volume. If the CPU becomes the bottleneck due to decompression, consider multithreaded decompression or offloading to accelerators. Also, overlap decompression with reading by using one thread to read compressed data and another thread to decompress the data in parallel. Modern GPUs can perform on-the-fly data decompression using GPU computing resources (or specialized decoders for image/visual data) when paired with GPUDirect Storage and the cuFile I/O stack.如果 I/O 是瓶颈，就以压缩形式存储数据集，但应使用轻量级压缩，例如 LZ4 或 Zstd 快速模式。这以少量 CPU 开销换取 I/O 数据量的减少。如果解压缩导致 CPU 成为瓶颈，可考虑多线程解压缩或将其卸载到加速器。此外，通过用一个线程读取压缩数据、另一个线程并行解压缩数据，使解压缩与读取重叠进行。现代 GPU 在与 GPUDirect Storage 和 cuFile I/O 栈配合时，可利用 GPU 计算资源（或针对图像/视觉数据的专用解码器）执行即时数据解压缩。

Measure throughput and eliminate bottlenecks测量吞吐量并消除瓶颈

Continuously monitor the data pipeline’s throughput. If GPUs aren’t near 100% utilization and you suspect input lag, measure how many MB/s you’re reading from disk and how busy the data loader cores are. Tools like dstat or NVIDIA’s DCGM can reveal if GPUs are waiting on data. Systematically tune each component by bumping up prefetch buffers, increasing network buffer sizes, optimizing disk RAID settings, etc. Do this until the input pipeline can feed data as fast as GPUs consume it. Often, these optimizations raise GPU utilization from ~70% to > 95% on the same hardware by removing I/O stalls.持续监控数据流水线的吞吐量。如果 GPU 利用率未接近 100%，且你怀疑存在输入滞后，就测量从磁盘读取的 MB/s 以及数据加载器核心的繁忙程度。dstat 或 NVIDIA 的 DCGM 等工具可以揭示 GPU 是否在等待数据。通过增大预取缓冲区、增加网络缓冲区大小、优化磁盘 RAID 设置等，系统地调优每个组件。持续这样做，直到输入流水线供给数据的速度能赶上 GPU 消费数据的速度。通常，这些优化通过消除 I/O 停顿，可在相同硬件上将 GPU 利用率从约 70% 提升到 > 95%。

Scale I/O for multinode为多节点扩展 I/O

At cluster scale, ensure the storage system can handle aggregate throughput. For example, 8 GPUs consuming 200 MB/s each is 1.6 GB/s per node. Across 100 nodes, that’s 160 GB/s needed. Very few central filesystems can sustain this. Mitigate by sharding data across storage servers, using per-node caches, or preloading data onto each node’s local disk. Trading off storage space for throughput (e.g., multiple copies of data) is often worth it to avoid starving expensive GPUs.在集群规模下，确保存储系统能够承受聚合吞吐量。例如，8 块 GPU 每块消耗 200 MB/s，即每节点 1.6 GB/s。跨 100 个节点，则需要 160 GB/s。很少有集中式文件系统能维持这一水平。可通过将数据分片到多台存储服务器、使用每节点缓存，或将数据预加载到每个节点的本地磁盘来缓解。以存储空间换取吞吐量（例如保存多份数据副本）通常是值得的，可避免昂贵的 GPU 因缺少数据而空转。

Minimize checkpointing and logging overhead最小化检查点与日志记录开销

Write checkpoints and logs efficiently. Use asynchronous writes for checkpoints if possible, or write to local disk, then copy to network storage to avoid stalling training. Compress checkpoints or use sparse storage formats to reduce size. Limit logging frequency on each step by aggregating iteration statistics and logging only every Nth iteration rather than every iteration. This will greatly reduce I/O overhead.高效地写入检查点和日志。尽可能对检查点使用异步写入，或先写入本地磁盘，再复制到网络存储，以避免训练停顿。压缩检查点或使用稀疏存储格式以减小体积。通过聚合迭代统计信息并仅每 N 次迭代记录一次日志（而非每次迭代都记录），限制每一步的日志记录频率。这将大幅降低 I/O 开销。

You can also suspend a running GPU process with cuda-checkpoint and Checkpoint/Restore in Userspace (CRIU) to persist the process image. When ready to resume, the CUDA driver can restore device memory and CUDA state—even on to other GPUs of the same device type. Treat this as complementary to your model’s state-dict or sharded checkpoint files rather than a replacement.你还可以使用 cuda-checkpoint 和用户空间检查点/恢复（CRIU）挂起正在运行的 GPU 进程，以持久化进程映像。准备恢复时，CUDA 驱动可以恢复设备显存和 CUDA 状态——甚至可以恢复到相同设备类型的其他 GPU 上。应将其视为对模型 state-dict 或分片检查点文件的补充，而非替代。

数据处理流水线

Data Processing Pipelines

The format, layout, and locality of data determine how smoothly the pipeline runs at scale. Binary formats, sharding, caching, and prioritized threads turn I/O from a bottleneck into a steady stream.数据的格式、布局和局部性决定了流水线在规模上运行的顺畅程度。二进制格式、分片、缓存和线程优先级可将 I/O 从瓶颈转变为稳定的数据流。

Convert datasets to binary or memory-mapped formats, shard across storage and nodes, and raise thread priorities or move simple augments to the GPU to prevent stalls. Cache hot data/KV states, prefetch and buffer aggressively, and size batches to keep the pipeline smooth from disk to device. The following are tips for improving your data processing:将数据集转换为二进制或内存映射格式，跨存储和节点进行分片，并提高线程优先级或将简单的数据增强移到 GPU 上，以防止停顿。缓存热数据/KV 状态，积极预取和缓冲，并调整批大小，使流水线从磁盘到设备保持顺畅。以下是改进数据处理的建议：

Use binary data formats使用二进制数据格式

Convert datasets to binary formats, such as TFRecords, LMDB, or memory-mapped arrays. This conversion reduces the overhead associated with handling millions of small files and accelerates data ingestion.将数据集转换为二进制格式，例如 TFRecords、LMDB 或内存映射数组。这种转换可降低处理数百万个小文件所带来的开销，并加速数据摄取。

Tune the file system调优文件系统

In addition to mounting file systems with noatime and increasing read-ahead, consider sharding data across multiple storage nodes to distribute I/O load and prevent bottlenecks on a single server.除了以 noatime 挂载文件系统并增大预读之外，还应考虑将数据分片到多个存储节点，以分散 I/O 负载，避免单台服务器出现瓶颈。

Disable hyperthreading for CPU-bound workloads对 CPU 密集型工作负载禁用超线程

For data pipelines that are heavily CPU-bound, disabling hyperthreading can reduce resource contention and lead to more consistent performance. This is especially beneficial on systems where single-thread performance is critical.对于高度受限于 CPU 的数据流水线，禁用超线程可以减少资源争用，带来更稳定的性能。在单线程性能至关重要的系统上，这一点尤其有益。

Elevate thread priorities提升线程优先级

Increase the scheduling priority of data loader and preprocessing CPU threads using tools, such as chrt or pthread_setschedparam. By giving these threads higher priority, you ensure that data is fed to the GPU with minimal latency, reducing the chance of pipeline stalls.使用 chrt 或 pthread_setschedparam 等工具提高数据加载器和预处理 CPU 线程的调度优先级。通过赋予这些线程更高的优先级，可确保以最小时延向 GPU 供给数据，减少流水线停顿的可能性。

Cache frequently used data缓存频繁使用的数据

Leverage operating system page caches or a dedicated RAM disk to cache frequently accessed data. This approach is especially beneficial in applications like NLP, where certain tokens or phrases are accessed repeatedly, reducing redundant processing and I/O overhead.利用操作系统页缓存或专用的 RAM 磁盘来缓存频繁访问的数据。这种方法在 NLP 等应用中尤其有益——其中某些 token 或短语会被反复访问，从而减少重复处理与 I/O 开销。

Prefetch and buffer data预取并缓冲数据

Always load data ahead of the iteration that needs it. Use background data loader threads or processes, such as PyTorch DataLoader with prefetch_factor. For distributed training, use DistributedSampler to ensure each process gets unique data to avoid redundant I/O.始终在需要数据的迭代之前提前加载数据。使用后台数据加载线程或进程，例如带 prefetch_factor 的 PyTorch DataLoader。在分布式训练中，使用 DistributedSampler 确保每个进程获得唯一数据，避免冗余 I/O。

Parallelize data transformations并行化数据变换

If CPU preprocessing—such as image augmentation and text tokenization—is heavy, distribute it across multiple worker threads/processes. Profile to ensure the CPU isn’t the bottleneck while GPUs wait. If it is, either increase workers or move some transforms to GPU, as libraries like NVIDIA’s DALI can do image operations on a GPU asynchronously.如果 CPU 预处理（例如图像增强和文本分词）负载较重，应将其分散到多个工作线程/进程上。通过性能剖析确认在 GPU 等待时 CPU 并非瓶颈。若确实是瓶颈，可增加工作进程数，或将部分变换迁移到 GPU——NVIDIA 的 DALI 等库可以在 GPU 上异步执行图像操作。

Cache model states and outputs缓存模型状态与输出

When inferencing with LLMs, it’s beneficial to cache the embeddings and V cache for frequently seen tokens to avoid having to recompute them repeatedly. Similarly, if an LLM training job reuses the same dataset multiple times (called epochs), you should leverage OS page cache or RAM to store the hot data.使用 LLM 进行推理时，缓存常见 token 的嵌入和 V 缓存是有益的，可避免反复重新计算。同样，如果 LLM 训练任务多次复用同一数据集（称为*轮次*（epoch）），则应利用操作系统页缓存或 RAM 存储热数据。

Shard data across nodes跨节点分片数据

In multinode training, give each node a subset of data to avoid every node reading the entire dataset from a single source. This scales out I/O. Use a distributed filesystem or manual shard assignment with each node reading different files. This speeds things up and naturally aligns with data parallelism since each node processes its own data shard. DeepSeek’s Fire-Flyer File System (3FS) is one example of a distributed dataset sharding filesystem. DeepSeek’s 3FS achieves multiterabyte-per-second throughput by distributing dataset shards across NVMe SSDs on each node—while minimizing traditional caching. This design feeds each GPU with local high-speed data, avoiding I/O bottlenecks.在多节点训练中，为每个节点分配一部分数据，避免每个节点都从单一数据源读取整个数据集。这样可以横向扩展 I/O。使用分布式文件系统或手动分片分配，让每个节点读取不同的文件。这能加快速度，并且天然契合数据并行，因为每个节点处理自己的数据分片。DeepSeek 的 Fire-Flyer File System（3FS）就是分布式数据集分片文件系统的一个例子。DeepSeek 的 3FS 通过将数据集分片分布到每个节点上的 NVMe SSD，实现了每秒数 TB 的吞吐量，同时尽量减少传统缓存。这种设计为每个 GPU 提供本地高速数据，避免 I/O 瓶颈。

Monitor pipeline and adjust batch size监控流水线并调整批大小

Sometimes increasing batch size will push more work onto GPUs and less frequent I/O, improving overall utilization—but only up to a point as it affects convergence. Conversely, if GPUs are waiting on data often, and you cannot speed I/O, you might actually decrease batch size to shorten each iteration and thus reduce idle time or do gradient accumulation of smaller batches such that data reads are more continuous. Find a balance where GPUs are nearly always busy.有时增大批大小会把更多工作推给 GPU、降低 I/O 频率，从而提升整体利用率——但仅在一定程度上有效，因为它会影响收敛。反之，如果 GPU 经常等待数据，且无法加快 I/O，你实际上可以减小批大小以缩短每次迭代，从而减少空闲时间，或者对小批量做梯度累加，使数据读取更连续。找到一个让 GPU 几乎始终处于忙碌状态的平衡点。

Apply data augmentation on GPU在 GPU 上执行数据增强

If augmentation is simple but applied to massive data, like adding noise or normalization, it might be worth doing on GPU to avoid saturating CPU. GPUs are often underutilized during data loading, so using a small CUDA kernel to augment data after loading can be efficient. But be careful not to serialize the pipeline. Use streams to overlap augmentation of batch N+1 while batch N is training.如果增强操作很简单但应用于海量数据，例如添加噪声或归一化，那么在 GPU 上执行可能值得，以避免 CPU 饱和。数据加载期间 GPU 往往未被充分利用，因此在加载后使用一个小型 CUDA 内核来增强数据是高效的。但要注意不要让流水线串行化。使用流在批次 N 训练的同时重叠执行批次 N+1 的增强。

Utilize GPU-accelerated libraries like NVIDIA DALI to perform these tasks asynchronously. This helps maintain a smooth and high-throughput data pipeline.利用 NVIDIA DALI 等 GPU 加速库异步执行这些任务。这有助于保持数据流水线顺畅且高吞吐。

Focus on end-to-end throughput (e.g., tokens per second)关注端到端吞吐量（例如每秒 token 数）

Remember that speeding up model compute doesn’t help if your data pipeline cuts throughput in half. Always profile end-to-end, not just the training loop isolated. Use Nsight Systems and Nsight Compute to measure kernel timelines and stalls, or the PyTorch profiler for framework-level attribution. Then compare iteration time with synthetic versus real data to see how much overhead data loading introduces. Aim for less than 10% overhead from ideal. If it’s more than that, invest time in pipeline optimization; it often yields large “free” speedups in training.请记住，如果数据流水线把吞吐量削减一半，加快模型计算也无济于事。始终进行端到端剖析，而不只是孤立地剖析训练循环。使用 Nsight Systems 和 Nsight Compute 测量内核时间线和停顿，或使用 PyTorch profiler 进行框架级归因。然后比较合成数据与真实数据的迭代时间，看看数据加载引入了多少开销。目标是将开销控制在理想值的 10% 以内。如果超过这个比例，就投入时间优化流水线；这往往能带来可观的“免费”训练加速。

性能剖析、调试与监控

Performance Profiling, Debugging, and Monitoring

You can’t optimize what you don’t measure; profiling reveals if you’re compute-bound, memory-bound, I/O-bound, or network-bound so you target the right fix. Continuous telemetry and regression tests keep wins from eroding as code, drivers, and data evolve.无法优化你未测量的东西；性能剖析能揭示你是受计算限制、受内存限制、受 I/O 限制还是受网络限制，从而对症下药。持续遥测和回归测试可防止随着代码、驱动和数据演进，已有收益被侵蚀。

Specifically, use Nsight Systems/Compute and framework profilers with NVTX to determine whether you’re compute-bound, memory-bound, I/O-bound, or communication-bound. Trim Python overhead, watch utilization gaps, balance work across ranks, track memory/network/disk health, and gate changes with performance regression tests and alerts. Use the following guidance to profile, monitor, and debug the performance of your AI workloads:具体来说，使用 Nsight Systems/Compute 和带 NVTX 的框架性能剖析器，判断你是受计算限制、受内存限制、受 I/O 限制还是受通信限制。削减 Python 开销，留意利用率空档，在各 rank 之间平衡工作，跟踪内存/网络/磁盘健康状况，并用性能回归测试和告警来把关变更。使用以下指导来剖析、监控和调试 AI 工作负载的性能：

Profile to find bottlenecks and root cause analysis通过性能剖析定位瓶颈并开展根因分析

Regularly run profilers on your training/inference jobs. Use NVIDIA Nsight Systems to get a timeline of CPU and GPU activity. You can also use Nsight Compute or the PyTorch profiler to drill down into kernel efficiency. Identify whether your job is compute bound, memory bound, or waiting on I/O/communication. Target your optimizations accordingly. For example, if your workload is memory bound, focus on reducing memory traffic rather than implementing compute-bound optimizations. Combine with machine-learning–driven analytics to predict and preempt performance bottlenecks. This can help in automating fine-tuning adjustments in real time. When using GPUDirect Storage, enable GDS tracing to correlate cuFile activity with kernel gaps.定期对训练/推理作业运行性能剖析器。使用 NVIDIA Nsight Systems 获取 CPU 与 GPU 活动的时间线。也可以使用 Nsight Compute 或 PyTorch 性能剖析器深入分析内核效率。判断作业是受计算限制、受内存限制，还是在等待 I/O/通信。据此有针对性地开展优化。例如，若工作负载受内存限制，应聚焦于减少内存流量，而不是实施面向计算受限场景的优化。结合机器学习驱动的分析来预测并提前化解性能瓶颈。这有助于实时自动化地完成微调调整。使用 GPUDirect Storage 时，启用 GDS 跟踪，将 cuFile 活动与内核空档关联起来。

Eliminate Python overhead消除 Python 开销

Profile your training scripts to identify Python bottlenecks—such as excessive looping or logging—and replace them with vectorized operations or optimized library calls. Minimizing Python overhead helps ensure that the CPU does not become a hidden bottleneck in the overall system performance.对训练脚本进行性能剖析，找出 Python 瓶颈——例如过多的循环或日志记录——并用向量化操作或优化过的库调用替换它们。尽量减少 Python 开销，有助于确保 CPU 不会成为整体系统性能中的隐藏瓶颈。

Measure GPU utilization and idle gaps测量 GPU 利用率与空闲空档

Continuously monitor GPU utilization, SM efficiency, memory bandwidth usage, etc. If you notice periodic drops in utilization, correlate them with events. For example, a drop in utilization every 5 minutes might coincide with checkpoint saving. Such patterns point to optimization opportunities, such as staggering checkpoints and using asynchronous flushes. Utilize tools like DCGM or nvidia-smi in daemon mode to log these metrics over time.持续监控 GPU 利用率、SM 效率、显存带宽使用率等。如果发现利用率周期性下降，将其与相关事件关联起来。例如，每 5 分钟一次的利用率下降可能与检查点保存同时发生。这类模式指向优化机会，例如错开检查点保存时间、使用异步刷写。可使用 DCGM 或守护进程模式下的 nvidia-smi 等工具，随时间记录这些指标。

Use NVTX markers使用 NVTX 标记

Instrument your code with NVTX ranges or framework profiling APIs to label different phases, including data loading, forward pass, backward pass, etc. These markers show up in the Nsight Systems or Perfetto timeline and help you attribute GPU idle times or latencies to specific parts of the pipeline. This makes it easier to communicate to developers which part of the code needs attention. For PyTorch, you can use torch.profiler.record_function().用 NVTX 范围或框架的性能剖析 API 对代码进行插桩，为不同阶段打上标签，包括数据加载、前向传播、反向传播等。这些标记会显示在 Nsight Systems 或 Perfetto 时间线中，帮助你判断 GPU 空闲时间或时延归属于流水线的哪个具体环节。这样也更容易向开发者说明代码的哪一部分需要关注。对于 PyTorch，可以使用 torch.profiler.record_function()。

Utilize kernel profiling and analysis tools beyond just the PyTorch profiler使用 PyTorch 性能剖析器之外的内核剖析与分析工具

For performance-critical kernels, use Nsight Compute to examine kernel-level metrics like occupancy and throughput, or Nsight Systems to analyze GPU/CPU timelines and overlap. Check achieved occupancy, memory throughput, and instruction throughput. Look for signs of memory bottlenecks, such as memory bandwidth near the hardware maximum. This helps to identify memory-bound workloads. The profiler’s “Issues” section often directly suggests if a kernel is memory bound or compute bound and why. Use this feedback to guide code changes, such as improving memory coalescing if global load efficiency is low.对于性能关键的内核，使用 Nsight Compute 检查占用率、吞吐量等内核级指标，或使用 Nsight Systems 分析 GPU/CPU 时间线与重叠情况。检查实际达到的占用率、显存吞吐量和指令吞吐量。留意内存瓶颈的迹象，例如显存带宽接近硬件上限。这有助于识别受内存限制的工作负载。性能剖析器的“Issues”部分通常会直接提示某个内核是受内存限制还是受计算限制，以及原因。利用这些反馈指导代码修改，例如在全局加载效率偏低时改善内存合并访问。

Check for warp divergence检查线程束分化（warp divergence）

Use the profiler to see if warps are diverging, as it can show branch efficiency and divergent branch metrics. Divergence means some threads in a warp are inactive due to branching, which hurts throughput. If significant, revisit the kernel code to restructure conditionals or data assignments to minimize intrawarp divergence and ensure that each warp handles uniform work.使用性能剖析器查看线程束是否发生分化，它可以显示分支效率和分化分支指标。分化意味着线程束中部分线程因分支而不活跃，从而损害吞吐量。如果分化严重，应重新审视内核代码，重构条件判断或数据赋值，以尽量减少线程束内分化，并确保每个线程束处理一致的工作。

Verify load balancing验证负载均衡

In multi-GPU jobs, profile across ranks. Sometimes one GPU (rank 0) does extra work like aggregating stats and data gathering—and often becomes a bottleneck. Monitor each GPU’s timeline. If one GPU is consistently lagging, distribute that extra workload. For example, you can have the nonzero ranks share the I/O and logging responsibilities. Ensuring that all GPUs/ranks have similar workloads avoids the slowest rank dragging the rest.在多 GPU 作业中，跨 rank 进行性能剖析。有时某块 GPU（rank 0）承担额外工作，例如聚合统计信息和收集数据，往往因此成为瓶颈。监控每块 GPU 的时间线。如果某块 GPU 持续落后，就把那部分额外工作分摊出去。例如，可以让非零 rank 分担 I/O 和日志记录职责。确保所有 GPU/rank 的工作负载相近，可避免最慢的 rank 拖累其余部分。

Monitor memory usage监控显存使用情况

Track GPU memory allocation and usage over time. Ensure you are not near OOM, which can cause the framework to unexpectedly swap tensors to host, which will cause huge slowdowns. If memory usage climbs iteration by iteration, you have likely identified leaks. In this case, profile with tools like torch.cuda.memory_summary() and Nsight Systems’ GPU memory trace to analyze detailed allocations. On the CPU side, monitor for paging, as your process’s resident memory (RES) should not exceed physical RAM significantly. If you see paging, reduce dataset preload size or increase RAM.跟踪随时间变化的 GPU 显存分配与使用情况。确保未接近 OOM，否则框架可能意外地将张量换出到主机，从而导致严重变慢。如果显存使用量逐次迭代攀升，你很可能已经发现了显存泄漏。此时，使用 torch.cuda.memory_summary() 和 Nsight Systems 的 GPU 显存跟踪等工具进行性能剖析，以分析详细的分配情况。在 CPU 侧，监控是否发生分页，因为进程的常驻显存（RES）不应显著超过物理 RAM。如果出现分页，请减小数据集预加载大小或增加 RAM。

Monitor network and disk监控网络与磁盘

For distributed jobs, use OS tools to monitor network throughput and disk throughput. Ensure the actual throughput matches expectations. For example, on a 100 Gbps link, you should see 12.5 GB/s (12.5 GB/s = 100 Gb/s ÷ 8 bits per byte) if fully utilized. If not, the network might be a bottleneck or misconfigured. Similarly, monitor disk I/O on training nodes. If you see spikes of 100% disk utilization and GPU idle, you likely need to buffer or cache data better.对于分布式作业，使用操作系统工具监控网络吞吐量和磁盘吞吐量。确保实际吞吐量符合预期。例如，在 100 Gbps 链路上，如果链路被充分利用，你应该看到 12.5 GB/s（12.5 GB/s = 100 Gb/s ÷ 8 比特/字节）。如果不是，网络可能成为瓶颈或配置有误。同样，监控训练节点上的磁盘 I/O。如果你看到磁盘利用率飙升至 100% 而 GPU 空闲，你很可能需要更好地缓冲或缓存数据。

Set up alerts for anomalies为异常设置告警

In a production or long-running training context, set up automated alerts or logs for events like GPU errors, such as ECC errors, device overheating, etc. This will help identify abnormally slow iterations. For example, NVIDIA’s DCGM can watch health metrics, and you can trigger actions if a GPU starts throttling or encountering errors. This helps catch performance issues—like a cooling failure causing throttling—immediately rather than after the job finishes.在生产环境或长时间运行的训练场景中，为 GPU 错误（如 ECC 错误、设备过热等）之类的事件设置自动告警或日志。这有助于识别异常缓慢的迭代。例如，NVIDIA 的 DCGM 可以监视健康指标，你可以在 GPU 开始降频或遇到错误时触发相应操作。这有助于立即捕获性能问题——例如散热故障导致的降频——而不是等到作业结束后才发现。

Perform regression testing执行回归测试

Maintain a set of benchmark tasks to run whenever you change software, including CUDA drivers, CUDA versions, AI framework versions, or even your training code. Compare performance to previous runs to catch regressions early. It’s not uncommon for a driver update or code change to inadvertently reduce throughput—a quick profiling run on a standard workload will highlight this so you can investigate. For example, maybe a kernel is accidentally not using Tensor Cores anymore. This is something to look into for sure.维护一组基准任务，在每次更改软件时运行，包括 CUDA 驱动、CUDA 版本、AI 框架版本，甚至你的训练代码。将性能与之前的运行结果进行比较，以便尽早发现性能回退。驱动更新或代码更改无意中降低吞吐量并不罕见——对标准工作负载做一次快速的性能剖析就能凸显这一点，以便你进一步排查。例如，某个内核可能意外地不再使用张量核心。这绝对值得深入调查。

GPU 编程与 CUDA 调优优化

GPU Programming and CUDA Tuning Optimizations

Aligning kernels with the memory hierarchy and hardware features is where large, durable gains come from. Fusion, Tensor Cores, CUDA Graphs, and compiler paths (e.g., torch.compile and OpenAI’s Triton) convert launch overhead into useful math.让内核与内存层级和硬件特性对齐，才能获得大幅且持久的收益。融合、张量核心、CUDA 图和编译器路径（例如 torch.compile 和 OpenAI 的 Triton）可将启动开销转化为有用的数学计算。

Optimize for the memory hierarchy: coalesce global loads, tile into shared memory, manage registers/occupancy, and overlap transfers (e.g., cp.async/TMA) with compute. Prefer tuned libraries and CUDA Graphs, leverage torch.compile and Open-AI’s Triton for fusion, and validate scalability with roofline analysis and PTX/SASS inspection. The following are some GPU and CUDA programming optimization tips and techniques:针对内存层级进行优化：合并全局加载、分块载入共享内存、管理寄存器/占用率，并将传输（例如 cp.async/TMA）与计算重叠。优先使用经过调优的库和 CUDA 图，利用 torch.compile 和 OpenAI 的 Triton 进行融合，并通过屋顶线模型分析和 PTX/SASS 检查来验证可扩展性。以下是一些 GPU 与 CUDA 编程的优化技巧和方法：

Understand GPU memory hierarchy理解 GPU 显存层级

Keep in mind the tiered memory structure of GPUs—registers per thread, shared memory/L1 cache per block/SM, L2 cache across SM, and global HBM. Maximize data reuse in the higher tiers. For example, use registers and shared memory to reuse values and minimize accesses to slower global memory. A good kernel ensures the vast majority of data is either in registers or gets loaded from HBM efficiently using coalescing and caching.牢记 GPU 的分层显存结构——每线程的寄存器、每线程块/流多处理器的共享内存/L1 缓存、跨流多处理器的 L2 缓存，以及全局 HBM。在更高层级中最大化数据复用。例如，使用寄存器和共享内存来复用数值，并尽量减少对较慢全局内存的访问。优秀的内核能确保绝大多数数据要么位于寄存器中，要么通过合并访问和缓存高效地从 HBM 加载。

Coalesce global memory accesses合并全局内存访问

Ensure that threads in the same warp access contiguous memory addresses so that the hardware can service them in as few transactions as possible. Strided or scattered memory access by warp threads will result in multiple memory transactions per warp, effectively wasting bandwidth. Restructure data layouts or index calculations so that whenever a warp loads data, it’s doing so in a single, wide memory transaction.确保同一线程束中的线程访问连续的内存地址，以便硬件能以尽可能少的事务完成这些访问。线程束中的线程若以跨步或分散方式访问内存，将导致每个线程束产生多次内存事务，实际上浪费了带宽。重新组织数据布局或索引计算方式，使线程束在加载数据时始终以单次、宽幅的内存事务完成。

Use shared memory for data reuse使用共享内存实现数据复用

Shared memory is like a manually managed cache with very high bandwidth. Load frequently used data—such as tiles of matrices—into shared memory. And have threads operate on those tiles multiple times before moving on. This popular tiling technique greatly cuts down global memory traffic. Be cautious of shared-memory bank conflicts. Organize shared-memory access patterns or pad data to ensure threads aren’t contending for the same memory bank, which would serialize accesses and reduce performance.共享内存类似于一个手动管理、带宽极高的缓存。将频繁使用的数据——例如矩阵分块——加载到共享内存中，并让线程对这些分块进行多次操作后再继续后续工作。这种常用的分块（tiling）技术可大幅减少全局内存流量。注意共享内存的存储体冲突。合理组织共享内存的访问模式或对数据进行填充，确保线程不会争用同一存储体，否则访问将被串行化并降低性能。

Optimize memory alignment优化内存对齐

Align data structures to 128 bytes whenever possible, especially for bulk memory copies or vectorized loads. Misaligned accesses can force multiple transactions even if theoretically coalesced. Using vectorized types like float2 and float4 for global memory I/O can help load/store multiple values per instruction, but ensure your data pointer is properly aligned to the vector size.尽可能将数据结构对齐到 128 字节，尤其是在批量显存拷贝或向量化加载时。即使理论上可以合并访问，未对齐的访问也可能强制产生多次事务。使用 float2、float4 等向量化类型进行全局内存 I/O 有助于在单条指令中加载/存储多个值，但要确保数据指针按向量大小正确对齐。

Minimize memory transfers尽量减少内存传输

Only transfer data to the GPU when necessary and in large chunks. Consolidate many small transfers into one big transfer if you can. For example, if you have many small arrays to send each iteration, pack them into one buffer and send once. Small, frequent cudaMemcpy can become a bottleneck. If using Unified仅在必要时向 GPU 传输数据，且以大块方式进行。尽可能将多次小传输合并为一次大传输。例如，如果每次迭代都要发送许多小数组，可将它们打包进一个缓冲区并一次性发送。频繁的小规模 cudaMemcpy 会成为瓶颈。如果使用统一

Memory, use explicit prefetch (cudaMemPrefetchAsync) to stage data on GPU before it’s needed, avoiding on-demand page faults during critical compute sections.内存，则使用显式预取（cudaMemPrefetchAsync）在需要之前将数据暂存到 GPU，避免在关键计算阶段发生按需缺页。

Avoid excessive temporary allocations避免过多的临时分配

Frequent allocation and freeing of GPU memory can hurt performance. For example, frequently using cudaMalloc/cudaFree or device malloc in kernels will cause extra overhead. Instead, reuse memory buffers or use memory pools available within most DL frameworks, like PyTorch, that implement a GPU caching allocator. If writing custom CUDA code, consider using cudaMallocAsync with a memory pool or manage a pool of scratch memory yourself to avoid the overhead of repetitive alloc/free.频繁分配和释放 GPU 显存会损害性能。例如，频繁使用 cudaMalloc/cudaFree 或在内核中使用设备端 malloc 会带来额外开销。应改为复用显存缓冲区，或使用大多数深度学习框架（如 PyTorch）提供的、实现了 GPU 缓存分配器的显存池。如果编写自定义 CUDA 代码，可考虑使用带显存池的 cudaMallocAsync，或自行管理一块暂存显存池，以避免反复分配/释放带来的开销。

Balance threads and resource use平衡线程数与资源使用

Achieve a good occupancy-resource balance. Using more threads for higher occupancy helps hide memory latency, but if each thread uses too many registers or too much shared memory, occupancy drops. Tune your kernel launch parameters—including threads per block—to ensure you have enough warps in flight to cover latency, but not so many that each thread is starved of registers or shared memory. In kernels with high instruction-level parallelism (ILP), reducing register usage to boost occupancy might actually hurt performance. The optimal point is usually in the middle of the occupancy spectrum, as maximum occupancy is not always ideal. Use the NVIDIA Nsight Compute Occupancy Calculator to experiment with configurations.在占用率与资源之间取得良好平衡。使用更多线程以获得更高占用率有助于隐藏内存时延，但如果每个线程使用的寄存器或共享内存过多，占用率就会下降。调整内核启动参数——包括每个线程块的线程数——以确保有足够的线程束处于执行状态以覆盖时延，但又不会多到使每个线程的寄存器或共享内存不足。在指令级并行（ILP）程度较高的内核中，为提升占用率而减少寄存器使用实际上可能损害性能。最优点通常位于占用率区间的中部，因为最大占用率并不总是理想选择。可使用 NVIDIA Nsight Compute 的 Occupancy Calculator 试验各种配置。

Monitor register and shared-memory usage监控寄存器与共享内存使用情况

Continuously monitor per-thread register and shared-memory consumption using profiling tools like Nsight Compute. If the occupancy is observed to be below 25%, consider increasing the number of threads per block to better utilize available hardware resources. However, verify that this adjustment does not cause excessive register spilling by reviewing detailed occupancy reports and kernel execution metrics. Register spilling can lead to additional memory traffic and degrade overall performance.使用 Nsight Compute 等性能剖析工具持续监控每线程的寄存器与共享内存消耗。若观察到占用率低于 25%，可考虑增加每个线程块的线程数，以更好地利用可用硬件资源。但需通过查看详细的占用率报告和内核执行指标，确认该调整不会导致过度的寄存器溢出。寄存器溢出会带来额外的内存流量并降低整体性能。

Overlap memory transfers with computation将内存传输与计算重叠

Overlap memory transfers with computation whenever possible. Use cuda Mem cpyAsync in multiple CUDA streams to prefetch while kernels run. Prefer the Tensor Memory Accelerator for bulk movement to shared memory, and use cp.async for fine-grained staged copies and prefetch. These approaches effectively mask global memory latency by overlapping data transfers with computation, making sure the GPU cores remain fully utilized without waiting for memory operations to complete.尽可能将内存传输与计算重叠。在多个 CUDA 流中使用 cuda Mem cpyAsync，在内核运行的同时进行预取。对于向共享内存的批量搬运，优先使用张量内存加速器（Tensor Memory Accelerator）；对于细粒度的暂存拷贝与预取，使用 cp.async。这些方法通过将数据传输与计算重叠，有效掩盖全局内存时延，确保 GPU 核心保持充分利用，而无需等待内存操作完成。

Use bulk prefetching when possible尽可能使用批量预取

For predictable patterns, prefetch into L2 using the PTX cp.async.bulk.pre fetch.tensor.[1–5]d.L2.global* (or the prefetch.global.L2 family), and use TMA (e.g., cp.async.bulk.tensor) to stage blocks into shared memory. You can also use cp.async to stage global memory into shared memory asynchronously and overlap copy with compute. You can also explicitly load data into registers ahead of use. These proactive methods reduce the delay caused by global memory accesses and make sure that critical data is available in faster, lower-latency storage—such as registers or shared memory—right when it’s needed, thus minimizing execution stalls and improving overall kernel efficiency.对于可预测的访问模式，使用 PTX 的 cp.async.bulk.pre fetch.tensor.[1–5]d.L2.global*（或 prefetch.global.L2 系列）预取到 L2，并使用 TMA（例如 cp.async.bulk.tensor）将数据块暂存到共享内存。你也可以使用 cp.async 将全局内存异步暂存到共享内存，并将拷贝与计算重叠。你还可以在使用之前显式地将数据加载到寄存器中。这些主动的方法减少了全局内存访问带来的延迟，确保关键数据在需要时正好位于更快、更低时延的存储中——例如寄存器或共享内存——从而最大限度地减少执行停顿并提升内核的整体效率。

Utilize cooperative groups利用协作组

Utilize CUDA’s cooperative groups to achieve efficient, localized synchronization among a subset of threads rather than enforcing a full block-wide barrier. This technique enables finer-grained control over synchronization, reducing unnecessary waiting times and overhead. By grouping threads that share data or perform related computations, you can synchronize only those threads that require coordination, which can lead to a more efficient execution pattern and better overall throughput.利用 CUDA 的协作组，在一部分线程之间实现高效的局部同步，而不是强制执行整个线程块范围的屏障。这种技术可以对同步进行更细粒度的控制，减少不必要的等待时间和开销。通过将共享数据或执行相关计算的线程分组，你可以只同步那些需要协调的线程，从而获得更高效的执行模式与更好的整体吞吐量。

Optimize warp divergence优化线程束发散

Structure your code so that threads within a warp follow the same execution path as much as possible. Divergence can double the execution time for that warp— for example, half the warp (16 threads) taking one branch and half the warp (16 threads) taking another branch. If you have branches that some data rarely triggers, consider “sorting” or grouping data so warps handle uniform cases such that all are true or all are false. Use warp-level primitives like ballot and shuffle to create branchless solutions for certain problems. Treat a warp as the unit of work, and aim for all 32 threads to do identical work in lockstep for maximum efficiency.组织代码结构，使同一线程束内的线程尽可能遵循相同的执行路径。发散可能使该线程束的执行时间翻倍——例如，线程束的一半（16 个线程）走一个分支，另一半（16 个线程）走另一个分支。如果某些分支很少被数据触发，可以考虑对数据进行“排序”或分组，使线程束处理一致的情况，即全部为真或全部为假。对于某些问题，使用 ballot 和 shuffle 等线程束级原语来构造无分支的解决方案。把线程束视为工作单元，力求全部 32 个线程以锁步方式执行相同的工作，以获得最高效率。

Leverage warp-level operations利用线程束级操作

Use CUDA’s warp intrinsics to let threads communicate without going to shared memory when appropriate. For example, use __shfl_sync to broadcast a value to all threads in a warp or to do warp-level reductions—like summing registers across a warp—instead of each thread writing to shared memory. These intrinsics bypass slower memory and can speed up algorithms like reductions or scans that can be done within warps. By processing these tasks within a warp, you avoid the latency associated with shared memory and full-block synchronizations.在适当的时候，使用 CUDA 的线程束内建函数（warp intrinsics）让线程之间无需经过共享内存即可通信。例如，使用 __shfl_sync 将一个值广播给线程束中的所有线程，或执行线程束级归约——比如对一个线程束内的寄存器求和——而不是让每个线程都写入共享内存。这些内建函数绕过了较慢的内存，可以加速那些能够在线程束内完成的算法，如归约或扫描。通过在线程束内处理这些任务，你避免了与共享内存和全块同步相关的时延。

Use CUDA streams for concurrency使用 CUDA 流实现并发

Within a single process/GPU, launch independent kernels in different CUDA streams to overlap their execution if they don’t use all resources. Overlap computation with computation—e.g., one stream computing one part of the model在单个进程/GPU 内，将相互独立的内核启动到不同的 CUDA 流中，如果它们没有用尽全部资源，就可以重叠执行。让计算与计算重叠——例如，一个流计算模型的某一部分

while another stream launches an independent kernel like data preprocessing on GPU or asynchronous memcpy. Be mindful of dependencies and use CUDA events to synchronize when needed. Proper use of streams can increase GPU utilization by not leaving any resource idle—especially if you have some kernels that are light.而另一个流启动独立的内核，比如在 GPU 上进行数据预处理，或异步 memcpy。注意依赖关系，并在需要时使用 CUDA 事件进行同步。正确使用流可以避免任何资源闲置，从而提高 GPU 利用率——尤其是在你有一些轻量内核的情况下。

Prefer library functions优先使用库函数

Wherever possible, use NVIDIA’s optimized libraries, such as cuBLAS, cuDNN, Thrust, and NCCL, for core math and collective operations. For point-to-point GPU data movement in distributed inference, use NIXL where available. You can also use NVSHMEM when you need fine-grained GPU-initiated transfers. These are heavily optimized for each GPU architecture and often approach theoretical “speed of light” peaks. This will save you the trouble of reinventing them. For example, use cuBLAS GEMM for matrix multiplies rather than a custom kernel, unless you have a very special pattern. The libraries also handle new hardware features transparently. AI frameworks like PyTorch (and its compiler) use these optimized libraries under the hood.尽可能使用 NVIDIA 优化的库，例如 cuBLAS、cuDNN、Thrust 和 NCCL，来处理核心数学运算与集合通信操作。对于分布式推理中的 GPU 点对点数据移动，在可用时使用 NIXL。当你需要细粒度的 GPU 发起传输时，也可以使用 NVSHMEM。这些库针对每种 GPU 架构都做了大量优化，通常能接近理论上的“光速”峰值。这样可以省去你重新造轮子的麻烦。例如，除非你有非常特殊的模式，否则应使用 cuBLAS GEMM 做矩阵乘法，而不是自定义内核。这些库还能透明地处理新的硬件特性。像 PyTorch（及其编译器）这样的 AI 框架在底层就使用了这些优化库。

Use CUDA Graphs for repeated launches对重复启动使用 CUDA 图

If you have a static training loop that is launched thousands of times, consider using CUDA Graphs to capture and launch the sequence of operations as a graph. This can significantly reduce CPU launch overhead for each iteration, especially in multi-GPU scenarios where launching many kernels and memcpy’s can put extra pressure on the CPU and incur additional latency.如果你有一个被启动数千次的静态训练循环，可以考虑使用 CUDA 图将该操作序列捕获并作为一个图来启动。这可以显著降低每次迭代的 CPU 启动开销，尤其是在多 GPU 场景下——启动大量内核和 memcpy 会给 CPU 带来额外压力并引入额外时延。

Check for scalability limits检查可扩展性限制

As you optimize a kernel, periodically check how it scales with problem size and across architectures. A kernel might achieve great occupancy and performance on a small input but not scale well to larger inputs, as it may start thrashing L2 cache or running into memory-cache evictions. Use roofline analysis. Compare achieved FLOPS and bandwidth to hardware limits to ensure you’re not leaving performance on the table.在优化内核时，定期检查它随问题规模以及跨架构的扩展情况。一个内核可能在小输入上获得很好的占用率和性能，但无法很好地扩展到更大的输入，因为它可能开始反复冲刷 L2 缓存或遭遇内存缓存逐出。使用屋顶线模型分析。将实际达到的 FLOPS 和带宽与硬件上限进行比较，确保你没有把性能白白留在桌上。

Inspect PTX and SASS for advanced kernel analysis检查 PTX 和 SASS 以进行高级内核分析

For performance-critical custom CUDA kernels, use Nsight Compute to examine the generated PTX and SASS. This deep dive can reveal issues like memory bank conflicts or redundant computations, guiding you toward targeted low-level optimizations.对于性能关键的定制 CUDA 内核，使用 Nsight Compute 检查生成的 PTX 和 SASS。这种深入分析可以揭示存储体冲突或冗余计算等问题，引导你进行有针对性的底层优化。

Use the PyTorch compiler使用 PyTorch 编译器

Take advantage of PyTorch’s torch.compile to fuse Python-level operations into optimized kernels through TorchInductor. The compiler can also reduce launch overhead by integrating CUDA Graphs. Typical gains of about 10%–40% are利用 PyTorch 的 torch.compile，通过 TorchInductor 将 Python 层操作融合为优化后的内核。该编译器还可以通过集成 CUDA 图来降低启动开销。一旦优化预热完成，通常可获得约 10%–40% 的收益。

common once the optimizations are warmed up. This eliminates interpreter overhead and unlocks compiler-level optimizations.这消除了解释器开销，并释放出编译器级别的优化。

In practice, enabling torch.compile has produced substantial speedups (e.g., 20%–50% on many models) by automatically combining kernels and utilizing NVIDIA GPU hardware (e.g., Tensor Cores) more efficiently. Always test compiled mode on your model. While it can massively boost throughput, you should ensure compatibility and correctness before deploying. When graphs are stable, enable CUDA Graphs to reduce per-iteration CPU overhead. Keep static memory pools to satisfy pointer-stability constraints.在实践中，启用 torch.compile 通过自动合并内核并更高效地利用 NVIDIA GPU 硬件（例如张量核心），已带来可观的加速（例如在许多模型上达到 20%–50%）。始终在你的模型上测试编译模式。虽然它可以大幅提升吞吐量，但在部署前应确保兼容性与正确性。当图稳定时，启用 CUDA 图以降低每次迭代的 CPU 开销。保留静态显存池以满足指针稳定性约束。

Plan for dynamic shapes为动态形状做好规划

If your input sizes vary, use torch._dynamo.mark_dynamic() to annotate dynamic dimensions or export shape-polymorphic graphs with torch.export(), and then compile. Control recompilation behavior with torch.compiler.set_ stance() using "fail_on_recompile" and torch._dynamo.error_on_ graph_ break() to surface problematic shape churn in testing and CI. Use static shapes where possible to enable CUDA Graphs to reduce per-iteration CPU overhead.如果你的输入尺寸会变化，使用 torch._dynamo.mark_dynamic() 标注动态维度，或用 torch.export() 导出形状多态图，然后再编译。使用 "fail_on_recompile" 配合 torch.compiler.set_ stance() 控制重编译行为，并使用 torch._dynamo.error_on_ graph_ break() 在测试和 CI 中暴露有问题的形状抖动。尽可能使用静态形状，以便启用 CUDA 图来降低每次迭代的 CPU 开销。

Leverage Triton kernels using *torch.compile利用 Triton 内核，使用 *torch.compile

If PyTorch doesn’t fuse an operation well, consider writing a custom GPU kernel in Triton and integrating it. PyTorch makes it easy to register a custom GPU kernel with torch.library.triton_op.如果 PyTorch 无法很好地融合某个算子，可以考虑用 Triton 编写自定义 GPU 内核并将其集成进来。PyTorch 让通过 torch.library.triton_op 注册自定义 GPU 内核变得很简单。

Use autotuning when available在可用时使用自动调优

Enable library autotuning features to maximize low-level performance. For example, set torch.backends.cudnn.benchmark=True when input sizes are fixed. This lets NVIDIA’s cuDNN library try multiple convolution algorithms and pick the fastest one for your hardware. The one-time overhead leads to optimized kernels that can accelerate training and inference. If exact reproducibility isn’t required, allow nondeterministic algorithms by disabling cudnn.deterministic to unlock these faster implementations.启用库的自动调优功能，以最大化底层性能。例如，当输入尺寸固定时，设置 torch.backends.cudnn.benchmark=True。这会让 NVIDIA 的 cuDNN 库尝试多种卷积算法，并为你的硬件挑选最快的一种。一次性的开销换来的是优化后的内核，可加速训练与推理。如果不需要精确的可复现性，可通过禁用 cudnn.deterministic 来允许非确定性算法，从而解锁这些更快的实现。

Leverage the read-only path利用只读路径

Mark frequently used constants or coefficients as read-only so the GPU can cache them in the dedicated L1 read-only cache. In CUDA C++, you can use const __restrict__ pointers to hint that data is immutable. On modern GPU architectures, the compiler generates cached global loads for const __restrict__ qualified pointers. When using AI frameworks and libraries, make sure that lookup tables or static weights are on the device and treated as constant. This optimization reduces global memory traffic and latency for those values, as each SM can quickly fetch them from cache instead of repeatedly accessing slow DRAM.将频繁使用的常量或系数标记为只读，以便 GPU 将它们缓存到专用的 L1 只读缓存中。在 CUDA C++ 中，可以使用 const __restrict__ 指针来提示数据不可变。在现代 GPU 架构上，编译器会为带 const __restrict__ 限定的指针生成带缓存的全局加载。在使用 AI 框架和库时，确保查找表或静态权重位于设备上并按常量对待。该优化可减少这些值的全局内存流量与时延，因为每个 SM 都能快速从缓存中取用它们，而不必反复访问缓慢的 DRAM。

内核调度与执行优化

Kernel Scheduling and Execution Optimizations

Launch overhead and unnecessary syncs create idle gaps that crush throughput. Fusing small kernels and using persistent/dynamic strategies keeps the device busy and latency hidden.启动开销和不必要的同步会造成空闲间隙，严重削弱吞吐量。融合小内核并采用持久化/动态策略，可以让设备保持忙碌、把时延隐藏起来。

Keep the device busy by minimizing synchronizations, fusing small kernels, and using persistent kernels when launching the same work repeatedly. For irregular tasks, consider GPU dynamic parallelism—but use it judiciously to avoid adding overhead. The following are tips on improving kernel scheduling and execution:通过最小化同步、融合小内核，以及在反复启动相同工作时使用持久化内核，让设备保持忙碌。对于不规则任务，可以考虑 GPU 动态并行——但要审慎使用，以免引入额外开销。以下是改进内核调度与执行的建议：

Minimize GPU synchronization calls最小化 GPU 同步调用

Avoid unnecessary global synchronizations that stall GPU progress. Excessive use of cudaDeviceSynchronize() or blocking GPU operations (like synchronous memory copies) will insert idle gaps where neither the CPU nor GPU can do useful work. Synchronize only when absolutely needed. For instance, synchronize when transferring final results or when debugging. By letting asynchronous operations queue up, you keep the GPU busy and the CPU free to prepare further work. This leads to a more continuous execution pipeline.避免不必要的全局同步，它们会阻碍 GPU 推进。过度使用 cudaDeviceSynchronize() 或阻塞式 GPU 操作（如同步显存拷贝）会插入空闲间隙，此时 CPU 和 GPU 都无法做有用的工作。只在绝对必要时才同步。例如，在传输最终结果或调试时同步。让异步操作排队执行，可使 GPU 保持忙碌、CPU 腾出手来准备后续工作。这会带来更连续的执行流水线。

Fuse small kernels to amortize launch overhead融合小内核以摊薄启动开销

If you have many tiny GPU kernels launching back-to-back, consider merging their operations to run in a single kernel where possible. Every kernel launch has a fixed cost on the order of tens of microseconds, so combining operations through manual CUDA kernel fusion, XLA fusion, or tools like NVIDIA CUT-LASS/Triton for custom ops can improve throughput. Fused kernels spend more time doing actual work and less time in launch overhead or memory round trips. This is especially helpful in inference or preprocessing pipelines where chains of elementwise ops can be executed in one go. Try torch.compile(mode="reduce-overhead") first. The compiler can fuse operation chains and wrap steady regions in CUDA Graphs. This will reduce CPU launch overhead. For unfused hotspots, consider migrating them to Triton kernels and using asynchronous TMA and automatic warp specialization where applicable.如果你有许多连续启动的微小 GPU 内核，尽可能将其操作合并到单个内核中运行。每次内核启动都有数十微秒量级的固定开销，因此通过手工 CUDA 内核融合、XLA 融合，或使用 NVIDIA CUT-LASS/Triton 等工具编写自定义算子来合并操作，可以提升吞吐量。融合后的内核把更多时间花在实际工作上，更少花在启动开销或内存往返上。这在推理或预处理流水线中尤其有用，因为其中的一连串逐元素操作可以一次性执行完。先尝试 torch.compile(mode="reduce-overhead")。编译器可以融合操作链，并把稳定的区域包装进 CUDA 图。这将降低 CPU 启动开销。对于未融合的热点，考虑将其迁移到 Triton 内核，并在适用时使用异步 TMA 和自动线程束专用化。

Utilize GPU dynamic parallelism for intra-GPU work scheduling利用 GPU 动态并行进行 GPU 内部的工作调度

Utilize CUDA’s Dynamic Parallelism to let GPU kernels launch other kernels from the GPU without returning to the CPU. In scenarios with unpredictable or iterative work, such as an algorithm that needs to spawn additional tasks based on intermediate results, dynamic parallelism cuts latency by removing the CPU launch bottleneck. For example, a parent kernel can divide and launch child kernels for further processing directly on the device. This keeps the entire workflow on the GPU, avoiding CPU intervention and enabling better overlap and utilization. Use this judiciously, however, as it can introduce its own overhead if overused.利用 CUDA 的动态并行（Dynamic Parallelism），让 GPU 内核可以在不返回 CPU 的情况下从 GPU 上启动其他内核。在工作不可预测或需要迭代的场景中——例如算法需要根据中间结果派生额外任务——动态并行通过消除 CPU 启动瓶颈来降低时延。例如，父内核可以直接在设备上划分并启动子内核以进行后续处理。这样整个工作流都留在 GPU 上，避免 CPU 介入，从而实现更好的重叠与利用率。不过要审慎使用，因为过度使用会引入其自身的开销。

Use persistent kernels for repeated workloads对重复性工作负载使用持久化内核（persistent kernel）

Use a persistent kernel strategy when a workload involves launching identical kernels in rapid succession, such as processing a work queue or streaming batches with the same computation. A persistent kernel is launched once and remains active, reusing threads to handle many units of work in a loop, rather than launching a fresh kernel for each unit. This approach trades a more complex kernel design for significantly lower scheduling overhead. By keeping the kernel alive, you avoid repeated launch costs and can achieve higher sustained occupancy. High-performance distributed training and inference systems often employ this technique to maximize throughput and minimize latency for iterative tasks.当工作负载涉及快速连续启动相同的内核时——例如处理工作队列，或以相同计算流式处理多个批次——应采用持久化内核策略。持久化内核只启动一次并保持活跃，在循环中复用线程来处理大量工作单元，而不是为每个工作单元启动一个新内核。这种方法以更复杂的内核设计换取显著更低的调度开销。通过让内核保持存活，可以避免重复的启动开销，并获得更高的持续占用率。高性能分布式训练与推理系统经常采用这一技术，以最大化迭代任务的吞吐量并最小化时延。

Evaluate thread block clusters评估线程块集群（thread block cluster）

Thread block clusters to keep data close and reduce relaunch overheads. Up to 16 thread blocks can form a cluster on Blackwell (after increasing the non-portable limit). Use cluster-aware synchronization and shared-memory residency to improve locality in persistent-style designs. Profile occupancy vs. residency trade-offs with kernel-level profiling tools like Nsight Compute.线程块集群可将数据保持在近处，并减少重新启动的开销。在 Blackwell 上（提高非可移植上限之后），最多 16 个线程块可以组成一个集群。使用集群感知的同步与共享内存驻留，可在持久化风格的设计中改善局部性。使用 Nsight Compute 等内核级性能剖析工具，剖析占用率与驻留性之间的权衡。

算术优化与降精度/混合精度

Arithmetic Optimizations and Reduced/Mixed Precision

Lower precisions and sparsity let you trade bits for big speed and memory wins— often with negligible accuracy impact. Mixed precision, TF32/FP8/INT8, and fused scaling exploit hardware math paths to raise throughput per dollar.更低的精度与稀疏性让你可以用比特换取大幅的速度与内存收益——而且往往精度影响可以忽略。混合精度、TF32/FP8/INT8 以及融合缩放（fused scaling）利用硬件的数学运算路径，提高每美元吞吐量。

Specifically, use mixed precision (BF16/FP16) and Tensor Cores for big gains, adopt TF32 for easy FP32 speedups, and evaluate FP8/FP4 where quality allows. Exploit structured sparsity, lower-precision gradients/communications, and INT8/INT4 quantization for inference—fusing scales/activations to preserve accuracy. The following optimization techniques apply to improving the performance of arithmetic computations and utilizing reduced/mixed precision:具体而言，使用混合精度（BF16/FP16）与张量核心可获得大幅收益，采用 TF32 可轻松获得 FP32 加速，并在质量允许时评估 FP8/FP4。利用结构化稀疏性、低精度梯度/通信，以及面向推理的 INT8/INT4 量化——融合缩放因子/激活以保持精度。以下优化技术适用于提升算术计算的性能并利用降精度/混合精度：

Use mixed-precision training使用混合精度训练

Leverage FP16 or BF16 for training to speed up math operations and reduce memory usage. Modern GPUs have Tensor Cores that massively accelerate FP16/ BF16 matrix operations. Keep critical parts like the final accumulation or a copy of weights in FP32 for numerical stability, but run bulk computations in half-precision. This often gives about a 1.5–3.5× speedup (depending on the model and kernel mix, with larger gains on matmul-heavy workloads) with minimal accuracy loss and is now standard in most frameworks with automatic mixed precision (AMP).在训练中使用 FP16 或 BF16，以加速数学运算并降低显存占用。现代 GPU 配备张量核心，可大幅加速 FP16/BF16 矩阵运算。将最终累加或权重副本等关键部分保留为 FP32 以保证数值稳定性，但批量计算以半精度运行。这通常带来约 1.5–3.5× 的加速（取决于模型与内核组合，matmul 密集型工作负载收益更大），而精度损失极小，如今已成为大多数框架中自动混合精度（AMP）的标准做法。

Embrace gradient accumulation and activation checkpointing采用梯度累加与激活检查点

Detail the use of gradient accumulation to effectively increase the batch size without extra memory usage, and consider activation checkpointing to reduce memory footprint in very deep networks. These techniques are crucial when training models that approach or exceed GPU memory limits.详细说明如何使用梯度累加在不增加显存占用的情况下有效增大批大小，并考虑使用激活检查点来降低极深网络的显存占用。当训练的模型接近或超出 GPU 显存上限时，这些技术至关重要。

Favor BF16 instead of FP16 on newer hardware在较新硬件上优先使用 BF16 而非 FP16

If available, use BF16 instead of FP16, as it has a larger exponent range and doesn’t require loss scaling. Modern GPUs support BF16 Tensor Cores at the same speed as FP16. BF16 will simplify training by avoiding overflow/underflow issues while still gaining the performance benefits of half precision.如果可用，请使用 BF16 而非 FP16，因为 BF16 具有更大的指数范围，且不需要损失缩放。现代 GPU 以与 FP16 相同的速度支持 BF16 张量核心。BF16 通过避免溢出/下溢问题简化训练，同时仍能获得半精度的性能收益。

Exploit FP8, novel precisions, and scaling techniques充分利用 FP8、新型精度与缩放技术

On modern GPUs, FP8 Tensor Cores provide roughly double the math throughput of FP16 or BF16 on compute-bound kernels while, at the same time, reducing activation and weight bandwidth. Additionally, FP4 (NVFP4) Tensor Cores double the throughput of FP8 and are used for inference with micro tensor scaling (an error-correction technique to maintain accuracy) to raise token throughput. For training, use FP8 with the NVIDIA Transformer Engine and maintain FP16 or FP32 accumulators when required. For inference, evaluate FP8 first and adopt NVFP4 only after calibration shows acceptable quality for your task. It’s recommended to use hybrid FP8 (E4M3 for forward activations/weights and E5M2 for gradients) for training. Specifically, consider using E4M3 for the forward pass (e.g., activations and weights) and E5M2 for the backward pass (e.g., gradients). It’s often beneficial to use a delayed scaling window of 256–1024. For inference, consider NVFP4 after calibration. TE integrates with PyTorch and is supported by modern GPU hardware. Prefer framework TE kernels over ad-hoc FP8 custom operations. End-to-end speedup depends on kernel mix, memory bandwidth, and calibration, so validate accuracy and performance on your model and workload.在现代 GPU 上，对于计算受限的内核，FP8 张量核心提供的数学吞吐量约为 FP16 或 BF16 的两倍，同时还能降低激活值和权重的带宽。此外，FP4（NVFP4）张量核心的吞吐量是 FP8 的两倍，并配合微张量缩放（一种用于保持精度的误差校正技术）用于推理，以提升 token 吞吐量。对于训练，请使用 NVIDIA Transformer Engine（TE）配合 FP8，并在需要时保留 FP16 或 FP32 累加器。对于推理，先评估 FP8，只有在校准显示对你的任务质量可接受之后才采用 NVFP4。训练时建议使用混合 FP8（前向激活/权重使用 E4M3，梯度使用 E5M2）。具体而言，考虑在前向传播中使用 E4M3（例如激活值和权重），在反向传播中使用 E5M2（例如梯度）。使用 256–1024 的延迟缩放窗口通常是有益的。对于推理，可在校准后考虑 NVFP4。TE 与 PyTorch 集成，并受现代 GPU 硬件支持。优先使用框架提供的 TE 内核，而非临时的 FP8 自定义算子。端到端加速取决于内核组合、显存带宽和校准，因此请在你的模型和工作负载上验证精度与性能。

Leverage Tensor Cores and the Tensor Memory Accelerator (TMA)利用张量核心与张量内存加速器（TMA）

Make sure your custom CUDA kernels utilize Tensor Cores for matrix ops if possible. This might involve using CUTLASS templates for simplicity. By using Tensor Cores and TMA for asynchronous tensor movement to shared memory, you can achieve dramatic speedups for GEMM, convolutions, and other tensor operations—often reaching near-peak FLOPS of the GPU. Ensure your data is in FP16/BF16/TF32 as needed and aligned to Tensor Core tile dimensions, which are multiples of 8 or 16.确保你的自定义 CUDA 内核尽可能对矩阵运算使用张量核心。为简便起见，这可能涉及使用 CUTLASS 模板。通过使用张量核心和 TMA 将张量异步搬运到共享内存，你可以为 GEMM、卷积和其他张量运算实现显著的加速——通常可达到 GPU 接近峰值的 FLOPS。确保你的数据按需为 FP16/BF16/TF32，并对齐到张量核心的 tile 维度，即 8 或 16 的倍数。

Use TF32 for easy speedup使用 TF32 轻松获得加速

For 32-bit matrix multiplies, set对于 32 位矩阵乘法，设置

torch.set_float32_matmul_preci

sion("high") to enable TF32 (fast FP32) for operations that are numerically safe in PyTorch. Libraries like cuBLAS and cuDNN will automatically pick optimalsion("high") 以在 PyTorch 中对数值安全的运算启用 TF32（快速 FP32）。cuBLAS 和 cuDNN 等库会自动选择最优的

Tensor Core code paths on modern GPU hardware. If you force full-precision FP32 with “highest” (instead of “high”), make sure to understand the performance impact.现代 GPU 硬件上的张量核心代码路径。如果你用“highest”（而非“high”）强制使用全精度 FP32，请务必了解其性能影响。

Exploit structured sparsity利用结构化稀疏性

Modern NVIDIA GPUs support 2:4 structured sparsity in matrix multiply, which zeros out 50% of weights in a structured pattern. This allows the hardware to double its throughput. Leverage this by pruning your model. If you can prune weights to meet the 2:4 sparsity pattern, your GEMMs can run ~2× faster for those layers. Use NVIDIA’s SDK or library support to apply structured sparsity and ensure the sparse Tensor Core paths are used. This can give a free speed boost if your model can tolerate or be trained with that sparsity, which often requires retraining with sparsity regularization.现代 NVIDIA GPU 在矩阵乘法中支持 2:4 结构化稀疏，即按结构化模式将 50% 的权重置零。这使硬件能够将吞吐量翻倍。通过剪枝模型来利用这一点。如果你能将权修剪到满足 2:4 稀疏模式，这些层的 GEMM 就能运行得快约 2 倍。使用 NVIDIA 的 SDK 或库支持来应用结构化稀疏，并确保使用稀疏张量核心路径。如果你的模型能够容忍这种稀疏性，或用这种稀疏性进行训练，这可以带来免费的加速，而这通常需要用稀疏正则化重新训练。

Reduce precision for gradients and activations when possible在可能的情况下降低梯度和激活值的精度

Even if you keep weights at higher precision, consider compressing gradients or activations to lower precision. For instance, use FP16/BF16 or FP8 communication for gradients. Many frameworks support FP16 gradient all-reduce. Similarly, for activation checkpointing, storing activations in 16-bit instead of FP32 saves memory. Research continues on FP8 and FP4 optimizers and quantized gradients. These help maintain model quality while reducing memory and bandwidth costs. In bandwidth-limited environments, gradient compression in particular can be a game changer. DeepSeek demonstrated this by compressing gradients to train on constrained GPUs.即使权重保持较高精度，也可以考虑将梯度或激活压缩到较低精度。例如，对梯度使用 FP16/BF16 或 FP8 通信。许多框架支持 FP16 梯度全归约。同样，对于激活检查点，用 16 位而非 FP32 存储激活可以节省内存。围绕 FP8 和 FP4 优化器以及量化梯度的研究仍在继续。这些方法有助于在降低内存与带宽开销的同时保持模型质量。在带宽受限的环境中，梯度压缩尤其可能带来颠覆性改变。DeepSeek 通过在受限 GPU 上压缩梯度进行训练，展示了这一点。

Use custom quantization for inference为推理使用自定义量化

For deployment, use INT8 quantization wherever possible. INT8 inference on GPUs is extremely fast and memory-efficient. Use NVIDIA’s TensorRT or quantization tools to quantize models to INT8 and calibrate them. Many neural networks like transformers can run in INT8 with a negligible accuracy drop. The speedups can be 2–4× over FP16. On the newest GPUs, also explore and evaluate FP8 or INT4 for certain models to further boost throughput for inference.在部署时，尽可能使用 INT8 量化。GPU 上的 INT8 推理极快且内存高效。使用 NVIDIA 的 TensorRT 或量化工具将模型量化为 INT8 并对其进行校准。许多神经网络（如 Transformer）可以在 INT8 下运行，精度下降可忽略不计。相比 FP16 可获得 2–4× 的加速。在最新的 GPU 上，还可以针对某些模型探索并评估 FP8 或 INT4，以进一步提升推理吞吐量。

Fuse scaling and computing operations when possible尽可能融合缩放与计算操作

When using lower precision, remember to fuse operations to retain accuracy. For example, Blackwell’s FP4 “microscaling” suggests keeping a scale per group of values. Incorporate these fused operations by scaling and computing in one pass—rather than using separate passes, which could cause precision loss. Many of these are handled by existing libraries, so just use them rather than implementing them from scratch.使用较低精度时，请记住融合操作以保持精度。例如，Blackwell 的 FP4“微缩放（microscaling）”建议为每组数值保留一个缩放因子。通过在一趟中完成缩放与计算来融入这些融合操作——而不是使用可能造成精度损失的多趟独立处理。其中许多已由现有库处理，因此直接使用它们，而不要从头实现。

高级调优策略与算法技巧

Advanced Tuning Strategies and Algorithmic Tricks

Algorithmic shifts routinely beat hardware upgrades on ROI by reducing work rather than pushing it faster. Autotuning, FlashAttention, overlap of comm/compute, and sharding unlock scale while cutting waste.算法层面的变革通常能比硬件升级带来更高的投资回报率（ROI），因为它减少工作量，而不是把工作推得更快。自动调优、FlashAttention、通信/计算重叠以及分片可在削减浪费的同时释放扩展能力。

Specifically, autotune kernel and layer parameters, swap in fused/FlashAttention kernels, and overlap communication with computation in distributed training. Scale deep models with pipeline/tensor parallelism and ZeRO sharding, and consider asynchronous updates or pruning/sparsity to trade a little accuracy work for big throughput wins. The following are some advanced performance optimizations and algorithmic tricks:具体而言，自动调优内核与层参数，换用融合/FlashAttention 内核，并在分布式训练中将通信与计算重叠。使用流水线并行/张量并行与 ZeRO 分片来扩展深度模型，并考虑异步更新或剪枝/稀疏性，以少量精度代价换取大幅吞吐量收益。以下是一些高级性能优化与算法技巧：

Autotune kernel parameters自动调优内核参数

Autotune your custom CUDA kernels for the target GPU. Choosing the correct block size, tile size, unroll factors, etc., can affect performance, and the optimal settings often differ between GPUs’ generations, such as Ampere, Hopper, Blackwell, and beyond. Use autotuning scripts or frameworks like OpenAI Triton—or even brute-force search in a preprocessing step—to find the best launch config. This can easily yield 20%–30% improvements that you’d miss with static “reasonable” settings. Use Triton features in your autotuning loop—for instance, set num_warps and num_stages, enable automatic warp specialization, and test asynchronous TMA layouts. Prefer tensor map descriptor APIs for shared-memory staging. Re-benchmark tile shapes when migrating to different hardware, as optimal choices will differ across GPU generations.针对目标 GPU 自动调优你的自定义 CUDA 内核。选择正确的块大小、瓦片大小、展开因子等会影响性能，而最优设置在不同代际的 GPU（如 Ampere、Hopper、Blackwell 及以后）之间往往不同。使用自动调优脚本或 OpenAI Triton 之类的框架——甚至在预处理步骤中做暴力搜索——来找到最佳启动配置。这很容易带来 20%–30% 的提升，而静态的“合理”设置会错失这些提升。在自动调优循环中使用 Triton 的特性——例如设置 num_warps 和 num_stages，启用自动线程束专用化，并测试异步 TMA 布局。优先使用张量映射描述符 API 进行共享内存暂存。迁移到不同硬件时重新基准测试瓦片形状，因为最优选择会因 GPU 代际而异。

Use kernel fusion in ML workloads在机器学习工作负载中使用内核融合

Utilize fused kernels provided by deep learning libraries. For example, enabling fused optimizers will fuse elementwise ops like weight update, momentum, etc. This will also use fused multihead attention implementations and fused normalization kernels. NVIDIA’s libraries and some open source projects like Transformer Engine and FasterTransformer provide fused operations for common patterns, such as fused LayerNorm + dropout. These reduce launch overhead and use memory more efficiently.利用深度学习库提供的融合内核。例如，启用融合优化器会融合权重更新、动量等逐元素操作。这也会使用融合多头注意力实现和融合归一化内核。NVIDIA 的库以及 Transformer Engine、FasterTransformer 等一些开源项目为常见模式（如融合 LayerNorm + dropout）提供融合操作。这些可降低启动开销并更高效地使用内存。

Utilize memory-efficient attention like FlashAttention利用 FlashAttention 等内存高效的注意力机制

Integrate advanced algorithms like FlashAttention for transformer models. FlashAttention computes attention in a tiled, streaming fashion to avoid materi-alizing large intermediate matrices, drastically reducing memory usage and increasing speed—especially for long sequences. Replacing the standard attention with FlashAttention can improve both throughput and memory footprint, allowing larger batch sizes or sequence lengths on the same hardware.为 Transformer 模型集成 FlashAttention 等先进算法。FlashAttention 以分块、流式的方式计算注意力，避免将大型中间矩阵实际写入内存，从而大幅降低显存占用并提升速度——对长序列尤为明显。用 FlashAttention 替换标准注意力可同时改善吞吐量和显存占用，使同一硬件上能够支持更大的批大小或序列长度。

Overlap communication and computation重叠通信与计算

In distributed training, overlap network communication with GPU computation whenever possible. For example, with gradient all-reduce, launch the all-reduce asynchronously as soon as each layer’s gradients are ready, while the next layer is still computing the backward pass. This pipelining can hide all-reduce latency entirely if done right. Use asynchronous NCCL calls or framework libraries like PyTorch’s Distributed Data Parallel (DDP), which provide overlapping out of the box. This ensures the GPU isn’t idle waiting for the network.在分布式训练中，尽可能将网络通信与 GPU 计算重叠。例如，在梯度全归约中，一旦某一层的梯度就绪就异步启动全归约，而此时下一层仍在计算反向传播。若实施得当，这种流水线化可以完全隐藏全归约的时延。使用异步 NCCL 调用或 PyTorch 的 Distributed Data Parallel（DDP）等框架库，它们开箱即用地提供重叠能力。这样可确保 GPU 不会因等待网络而空闲。

Use pipeline parallelism for deep models对深层模型使用流水线并行

When model size forces you to pipeline across GPUs using tensor parallelism or pipeline parallelism, you can use enough microbatches to keep all pipeline stages busy. Exploit NVLink/NVSwitch to send activations quickly between stages. Overlap and reduce pipeline bubbles by using an interleaved schedule. Some frameworks automate this type of scheduling. The NVL72 fabric is especially helpful here, as even communication-heavy pipeline stages can exchange data at multiterabyte speeds, minimizing pipeline stalls.当模型规模迫使你使用张量并行或流水线并行跨 GPU 做流水线切分时，可以使用足够多的微批（microbatch）让所有流水线阶段保持忙碌。利用 NVLink/NVSwitch 在各阶段之间快速传输激活值。通过交错调度重叠并减少流水线气泡。一些框架可自动完成这类调度。NVL72 网络结构在此尤为有用，因为即便是通信密集的流水线阶段也能以数 TB/s 的速度交换数据，从而将流水线停顿降到最低。

Utilize distributed optimizer sharding利用分布式优化器分片

Use a memory-saving optimization strategy like Zero Redundancy Optimizer (ZeRO), which shards tensors like optimizer states and gradients across GPUs instead of replicating them. This allows scaling to extreme model sizes by distributing the memory and communication load. It improves throughput by reducing per-GPU memory pressure, avoiding swapping to CPU, and reducing communication volume if done in chunks. Many frameworks like DeepSpeed and Megatron-LM provide this type of sharding. Leverage it for large models to maintain high speed without running OOM or hitting slowdown from swapping.采用 Zero Redundancy Optimizer（ZeRO）这类节省内存的优化策略，它将优化器状态、梯度等张量在各 GPU 之间分片，而不是复制。通过分摊内存与通信负载，该方法可扩展到极端的模型规模。它降低单 GPU 的显存压力、避免向 CPU 交换，并在分块执行时减少通信量，从而提升吞吐量。DeepSpeed 和 Megatron-LM 等许多框架都提供这类分片。对大模型加以利用，可在不出现 OOM、也不因交换而变慢的前提下保持高速。

Train asynchronously when possible尽可能异步训练

If applicable, consider asynchronous updates. For example, you can use stale sto-chastic gradient descent (SGD) in which workers don’t always wait for one another to share updates. This approach can increase throughput, though it may require careful tuning to not impact convergence. Asynchronous training can provide large performance benefits if done properly.在适用的情况下，考虑异步更新。例如，可以使用陈旧梯度随机梯度下降（stale SGD），其中各工作节点不必总是相互等待来共享更新。这种做法可以提升吞吐量，但可能需要仔细调优，以免影响收敛性。若实施得当，异步训练可带来显著的性能收益。

Incorporate sparsity and pruning引入稀疏性与剪枝

Large models often have redundancy. Use pruning techniques during training to introduce sparsity, which you can exploit at inference—and partially during training if supported. Modern GPU hardware supports accelerated sparse matrix multiply (2:4), and future GPUs will likely extend this feature. Even if you leave training as dense and prune only for inference, a smaller model will run faster and use less memory. This increases cost-efficiency for model deployments. Explore the lottery ticket hypothesis, distillation, or structured pruning to maintain accuracy while trimming model size.大模型往往存在冗余。在训练期间使用剪枝技术引入稀疏性，你可以在推理时利用这种稀疏性——若硬件支持，也可在训练中部分利用。现代 GPU 硬件支持加速的稀疏矩阵乘法（2:4），未来的 GPU 很可能扩展这一特性。即使训练保持稠密、仅在推理时剪枝，更小的模型也会运行更快、占用内存更少。这提升了模型部署的成本效益。可以探索彩票假说（lottery ticket hypothesis）、蒸馏或结构化剪枝，在缩减模型规模的同时保持精度。

分布式训练与网络优化

Distributed Training and Network Optimization

At cluster scale, the network becomes the limiter. Untreated, the network can break linear scaling and inflate costs. RDMA/Jumbo frames, hierarchical collectives, affinity, and compression protect bandwidth and tame latency.在集群规模下，网络成为限制因素。若不加以处理，网络会破坏线性扩展并推高成本。RDMA/巨型帧（Jumbo frames）、分层集合通信、亲和性与压缩可保护带宽并抑制时延。

Use RDMA (InfiniBand/RoCE) when available; if on Ethernet, tune TCP buffers, enable jumbo frames, and select modern congestion control. Align NIC/CPU affinity, adjust NCCL threads/buffers (and SHARP/CollNet where supported), compress or accumulate gradients, and test the fabric to catch loss or misconfigurations. Follow this guidance to optimize your network for distributed environments such as multi-GPU and multinode model training:在可用时使用 RDMA（InfiniBand/RoCE）；若使用以太网，则调优 TCP 缓冲区、启用巨型帧，并选择现代的拥塞控制算法。对齐网卡/CPU 亲和性，调整 NCCL 线程/缓冲区（以及在支持时使用 SHARP/CollNet），压缩或累加梯度，并测试网络结构（fabric）以发现丢包或配置错误。遵循以下指导，为多 GPU 与多节点模型训练等分布式环境优化网络：

Use RDMA networking when available在可用时使用 RDMA 网络

Equip your multinode cluster with InfiniBand or RoCE for low latency and high throughput. Ensure NCCL and MPI are using RDMA for training. NCCL will autodetect InfiniBand and use GPUDirect RDMA if available. RDMA bypasses the kernel networking stack and can reduce latency significantly versus traditional TCP. If you only have Ethernet, enable RoCE on RDMA-capable NICs to get RDMA-like performance. On NVLink domain systems (NVL72, GB200/ GB300, etc.), keep collectives on-fabric when possible. Reserve host networking for inter-island links. Align NCCL topology hints with your NVLink/NVSwitch domains.为多节点集群配备 InfiniBand 或 RoCE，以获得低时延和高吞吐量。确保 NCCL 和 MPI 在训练中使用 RDMA。NCCL 会自动检测 InfiniBand，并在可用时使用 GPUDirect RDMA。RDMA 绕过内核网络栈，相比传统 TCP 可显著降低时延。如果只有以太网，请在支持 RDMA 的网卡上启用 RoCE，以获得接近 RDMA 的性能。在 NVLink 域系统（NVL72、GB200/GB300 等）上，尽可能让集合通信留在 fabric 内。将主机网络保留给岛间（inter-island）链路。使 NCCL 拓扑提示与你的 NVLink/NVSwitch 域保持一致。

Tune the TCP/IP stack if using Ethernet若使用以太网，则调优 TCP/IP 栈

For TCP-based clusters, increase network buffer sizes. Raise /proc/sys/net/ core/{r,w}mem_max and the autotuning limits (net.ipv4.tcp_{r,w}mem) to allow larger send/receive buffers. This helps saturate 10/40/100 GbE links. Enable jumbo frames (MTU 9000) on all nodes and switches to reduce overhead per packet, which improves throughput and reduces CPU usage. Also consider modern TCP congestion control like BBR for wide-area or congested networks.对于基于 TCP 的集群，增大网络缓冲区大小。提高 /proc/sys/net/ core/{r,w}mem_max 以及自动调优上限（net.ipv4.tcp_{r,w}mem），以允许更大的发送/接收缓冲区。这有助于打满 10/40/100 GbE 链路。在所有节点和交换机上启用巨型帧（MTU 9000），以减少每个数据包的开销，从而提升吞吐量并降低 CPU 使用率。对于广域网或拥塞网络，还可考虑使用 BBR 等现代 TCP 拥塞控制算法。

Assign CPU affinity for NICs为网卡分配 CPU 亲和性

Pin network interrupts and threads to the CPU core(s) on the same NUMA node as the NIC. This avoids cross-NUMA penalties for network traffic and keeps the networking stack’s memory accesses local. Check /proc/interrupts and use irqaffinity settings to ensure, for example, your NIC in NUMA node 0 is handled by a core in NUMA node 0. This can improve network performance and consistency, especially under high packet rates.将网络中断和线程绑定到与网卡处于同一 NUMA 节点的 CPU 核上。这可以避免网络流量跨 NUMA 带来的性能损失，并使网络栈的显存访问保持本地。检查 /proc/interrupts 并使用 irqaffinity 设置来确保，例如 NUMA 节点 0 中的网卡由 NUMA 节点 0 中的核处理。这可以提升网络性能与一致性，尤其在高包速率下。

Optimize NCCL environment variables for your environment针对你的环境优化 NCCL 环境变量

Experiment with NCCL parameters for large multinode jobs. For example, increase NCCL_NTHREADS, the number of CPU threads per GPU for NCCL, from the default 4 to 8 or 16 to drive higher bandwidth at the cost of more CPU usage.针对大规模多节点作业试验 NCCL 参数。例如，将 NCCL_NTHREADS（NCCL 为每个 GPU 使用的 CPU 线程数）从默认的 4 提高到 8 或 16，以换取更高带宽，代价是更多 CPU 占用。

Increase NCCL_BUFFSIZE, the buffer size per GPU, from the default 1 MB to 4 MB or more for better throughput on large messages. If your cluster uses SHARP-capable switches, install the NCCL SHARP plugin and enable CollNet by setting NCCL_COLLNET_ENABLE=1, then use the SHARP plugin variables such as SHARP_ COLL_LOCK_ON_COMM_INIT=1 and将 NCCL_BUFFSIZE（每个 GPU 的缓冲区大小）从默认的 1 MB 提高到 4 MB 或更大，以在大消息上获得更好的吞吐量。如果你的集群使用支持 SHARP 的交换机，请安装 NCCL SHARP 插件并通过设置 NCCL_COLLNET_ENABLE=1 启用 CollNet，然后按文档使用 SHARP 插件变量，例如 SHARP_ COLL_LOCK_ON_COMM_INIT=1 和

SHARP_COLL_NUM_COLL_GROUP_RESOURCE_

ALLOC_THRESHOLD=0 as documented. Expect speedups only when your reductions are large enough and the network fabric supports SHARP offload.ALLOC_THRESHOLD=0。只有当你的归约操作足够大且网络结构支持 SHARP 卸载时，才可期待加速效果。

Use gradient accumulation for slow networks为慢速网络使用梯度累加

If your network becomes the bottleneck because you are scaling too many nodes linked by a moderate-performance interconnect, use gradient accumulation to perform fewer, larger all-reduce operations. Accumulate gradients over a few minibatches before syncing so that you communicate once for N batches instead of every batch. This trades a bit of extra memory and some model accuracy tuning for significantly reduced network overhead. It’s especially helpful when adding more GPUs yields diminishing returns due to communication costs.如果你的网络因为扩展了过多由中等性能互连连接的节点而成为瓶颈，请使用梯度累加来执行更少、更大的全归约操作。在同步前跨若干小批次累加梯度，这样每 N 个批次才通信一次，而不是每个批次都通信。这以少量额外内存和一些模型精度调优为代价，显著降低网络开销。当增加更多 GPU 因通信成本而收益递减时，这一点尤其有用。

Optimize all-reduce topologies优化全归约拓扑

Ensure you’re using the optimal all-reduce algorithm for your cluster topology. NCCL will choose ring or tree algorithms automatically, but on mixed interconnects like GPUs connected by NVLink on each node and InfiniBand or Ethernet between nodes, hierarchical all-reduce can be beneficial. Hierarchical all-reduce will first perform the all-reduce operation within the node, then it will proceed across nodes. Most frameworks will perform NCCL-based hierarchical aggregations by default but verify by profiling. In traditional MPI setups, you may consider manually doing this same two-level reduction—first intranode and then internode.确保针对你的集群拓扑使用最优的全归约算法。NCCL 会自动选择环形或树形算法，但在混合互连环境下——例如每个节点内的 GPU 通过 NVLink 连接、节点之间通过 InfiniBand 或以太网连接——分层全归约可能更有利。分层全归约会先在节点内执行全归约操作，然后再跨节点进行。大多数框架默认会执行基于 NCCL 的分层聚合，但请通过性能剖析加以验证。在传统 MPI 环境中，你可以考虑手动执行同样的两级归约——先节点内，再节点间。

Avoid network oversubscription避免网络超额订阅

On multi-GPU servers, ensure the combined traffic of GPUs doesn’t oversubscribe the NIC. For example, eight GPUs can easily generate more than 200 Gbps of traffic during all-reduce, so having only a single 100 Gbps NIC will constrain you. Consider multiple NICs per node and 200/400 Gbps InfiniBand if scaling to many GPUs per node. Likewise, watch out for PCIe bandwidth limits if your NIC and GPUs share the same PCIe root complex.在多 GPU 服务器上，确保 GPU 的总流量不会使网卡（NIC）超额订阅。例如，八块 GPU 在全归约期间很容易产生超过 200 Gbps 的流量，因此仅有一块 100 Gbps 网卡就会成为限制。若要在单节点扩展到大量 GPU，可考虑每节点配置多块网卡以及 200/400 Gbps InfiniBand。同样，如果你的网卡与 GPU 共享同一个 PCIe 根复合体，要注意 PCIe 带宽限制。

Compress communication压缩通信

Just as with single-node memory, consider compressing data for network transfer. Techniques include 16-bit or 8-bit gradient compression, quantizing activations for cross-node pipeline transfers, or even more exotic methods like sketching. If your network is the slowest component, a slightly higher compute cost to compress/decompress data can be worth it. NVIDIA’s NCCL doesn’t natively compress, but you can integrate compression in frameworks (e.g., gradient compression in Horovod or custom AllReduce hooks in PyTorch). This与单节点内存的情况一样，可以考虑对网络传输的数据进行压缩。相关技术包括 16 位或 8 位梯度压缩、为跨节点流水线传输量化激活值，甚至更奇特的方法，如 sketching。如果网络是最慢的组件，那么为压缩/解压数据付出略高的计算开销是值得的。NVIDIA 的 NCCL 本身不做压缩，但你可以在框架中集成压缩（例如 Horovod 中的梯度压缩，或 PyTorch 中自定义的 AllReduce 钩子）。这

was one key to DeepSeek’s success—compressing gradients to cope with limited internode bandwidth.正是 DeepSeek 成功的关键之一——通过压缩梯度来应对有限的节点间带宽。

Monitor network health监控网络健康状况

Ensure no silent issues are hampering your distributed training. Check for packet loss (which would show up as retries or timeouts—on InfiniBand, use counters for resend, and on Ethernet, check for TCP retransmits). Even a small packet loss can severely degrade throughput due to congestion control kicking in. Use outof-band network tests (like iPerf or NCCL tests) to validate you’re getting expected bandwidth and latency. If not, investigate switch configurations, NIC firmware, or CPU affinity.确保没有隐蔽的问题在拖累你的分布式训练。检查丢包情况（丢包会表现为重试或超时——在 InfiniBand 上使用重传计数器，在以太网上检查 TCP 重传）。由于拥塞控制会被触发，即使很小的丢包也可能严重降低吞吐量。使用带外网络测试（如 iPerf 或 NCCL 测试）验证你是否获得了预期的带宽和时延。如果未达到，请排查交换机配置、网卡固件或 CPU 亲和性。

高效推理与服务

Efficient Inference and Serving

Serving is a cost-and-latency game—utilization rises through orchestration and batching, not just bigger GPUs. Specialized runtimes, KV cache strategies, and warm-ups keep throughput high without violating SLOs.服务是一场成本与时延的博弈——利用率的提升来自编排与批处理，而不只是更大的 GPU。专用运行时、KV 缓存策略和预热能在不违反 SLO 的前提下保持高吞吐量。

Orchestrate for demand with autoscaling, microservices, and dynamic/continuous batching to keep GPUs hot without violating latency SLOs. Use specialized runtimes (vLLM, SGLang, TensorRT-LLM), exploit NIXL and KV cache offloading for disaggregated serving, warm models, and isolate resources to control tail latency. Follow these techniques to improve model inference efficiency and performance:通过自动扩缩容、微服务和动态/连续批处理按需编排，让 GPU 保持高负载运行，同时不违反时延 SLO。使用专用运行时（vLLM、SGLang、TensorRT-LLM），利用 NIXL 和 KV 缓存卸载实现解耦式服务，预热模型，并隔离资源以控制尾时延。遵循以下技术来提升模型推理效率与性能：

Orchestrate dynamic resources efficiently高效编排动态资源

Integrate advanced container orchestration platforms, such as Kubernetes aug-mented with custom performance metrics. This enables dynamic scaling and balancing workloads based on live usage patterns and throughput targets.集成先进的容器编排平台，例如辅以自定义性能指标的 Kubernetes。这样可以根据实时使用模式和吞吐量目标动态扩缩容并平衡工作负载。

Embrace serverless architectures for inference拥抱面向推理的无服务器架构

Explore serverless architectures and microservice designs for inference workloads, which can handle bursty traffic efficiently and reduce idle resource overhead by scaling down when demand is low.探索面向推理工作负载的无服务器架构与微服务设计，这类架构能够高效处理突发流量，并在需求低迷时通过缩容来降低空闲资源开销。

Optimize batch and concurrency优化批处理与并发

For inference workloads, find the right batching strategy. For inference workloads, favor dynamic or continuous batching to automatically batch incoming requests. Larger batch sizes improve throughput by keeping the GPU busy, but too large can add latency. Also, run multiple inference streams in parallel if one stream doesn’t use all GPU resources—e.g., two concurrent inference batches to use both GPU SMs and Tensor Cores fully.对于推理工作负载，找到合适的批处理策略。对于推理工作负载，优先采用动态批处理或连续批处理，以自动对到来的请求进行批处理。更大的批大小通过让 GPU 保持忙碌来提升吞吐量，但批大小过大会增加时延。此外，如果单个流无法用满全部 GPU 资源，可并行运行多个推理流——例如两个并发的推理批次，以充分利用 GPU 的 SM 与张量核心。

Leverage NIXL for distributed inference利用 NIXL 实现分布式推理

When serving large models across GPUs or nodes, use the NVIDIA Inference Xfer Library to stream KV cache between prefill and decode workers over当跨 GPU 或跨节点服务大模型时，使用 NVIDIA Inference Xfer Library 在预填充与解码工作进程之间通过

RDMA. In the case of NIXL, the large transformer-based KV cache is transferred between nodes. NIXL provides a high-throughput, low-latency API for streaming the KV cache from a prefill GPU to a decode GPU in a disaggregated LLM inference cluster. It does this using GPUDirect RDMA and optimal paths—and without involving the CPU. This reduces tail latency for disaggregated prefill decode serving across nodes.RDMA 流式传输 KV 缓存。在 NIXL 的场景中，基于 Transformer 的大型 KV 缓存会在节点之间传输。NIXL 提供高吞吐量、低时延的 API，用于在解耦的 LLM 推理集群中将 KV 缓存从预填充 GPU 流式传输到解码 GPU。它通过 GPUDirect RDMA 与最优路径实现这一点——且无需 CPU 参与。这降低了跨节点解耦预填充/解码服务的尾部时延。

Offload KV cache if necessary如有必要，卸载 KV 缓存

If an LLM’s attention KV cache grows beyond GPU memory, use hierarchical offloading. NVIDIA Dynamo’s Distributed KV Cache Manager offloads less frequently accessed KV pages to CPU memory, SSD, or networked storage, while inference engines like TensorRT-LLM and vLLM support paged and quantized KV caches. Reuse caches to lower memory pressure and first-token latency. Validate end-to-end impact because offloaded misses introduce extra I/O latency. This allows inference on sequences that would otherwise exceed GPU memory— and with minimal performance hit thanks to fast NVMe and compute-I/O overlapping. Ensure your inference server is configured to use this if you expect very long prompts or chats. Offloading to disk is better than failing completely.如果 LLM 的注意力 KV 缓存增长到超出 GPU 显存，请使用分层卸载。NVIDIA Dynamo 的 Distributed KV Cache Manager 会将访问频率较低的 KV 页卸载到 CPU 内存、SSD 或网络存储，而 TensorRT-LLM 和 vLLM 等推理引擎支持分页与量化 KV 缓存。复用缓存以降低显存压力和首 token 时延。验证端到端影响，因为卸载未命中会引入额外的 I/O 时延。这使得原本会超出 GPU 显存的序列也能进行推理——并且得益于快速 NVMe 与计算-I/O 重叠，性能损失极小。如果你预计会有超长提示或对话，请确保推理服务器已配置使用该功能。卸载到磁盘总比彻底失败要好。

Serve models efficiently高效地服务模型

Use optimized model inference systems, such as vLLM, SGLang, NVIDIA Dynamo, and NVIDIA TensorRT-LLM for serving large models with low latency and high throughput. They should implement quantization, low-precision formats, fusion, highly optimized attention kernels, and other tricks to maximize GPU utilization during inference. These libraries should also handle tensor parallelism, pipeline parallelism, expert parallelism, context parallelism, speculative decoding, chunked prefill, disaggregated prefill/decode, and dynamic request batching—among many other high-performance features.使用经过优化的模型推理系统，例如 vLLM、SGLang、NVIDIA Dynamo 和 NVIDIA TensorRT-LLM，以低时延、高吞吐量服务大模型。它们应实现量化、低精度格式、算子融合、高度优化的注意力内核以及其他技巧，以在推理期间最大化 GPU 利用率。这些库还应处理张量并行、流水线并行、专家并行、上下文并行、投机解码、分块预填充、解耦预填充/解码以及动态请求批处理——以及许多其他高性能特性。

Monitor and tune for tail latency监控并调优尾部时延

In real-time services, both average latency and (long-)tail latency (99th percentile) matter. Profile the distribution of inference latencies. If the tail is high, identify outlier causes, such as unexpected CPU involvement, garbage-collection (GC) pauses, or excessive context switches. Pin your inference server process to specific cores, isolate it from noisy neighbors, and use real-time scheduling if necessary to get more consistent latency.在实时服务中，平均时延和（长）尾时延（99 百分位）都很重要。剖析推理时延的分布。如果尾部较高，找出异常值成因，例如意外的 CPU 参与、垃圾回收（GC）暂停或过多的上下文切换。将推理服务器进程绑定到特定核心，将其与嘈杂邻居隔离，必要时使用实时调度以获得更一致的时延。

Warm up to avoid cold-start latency预热以避免冷启动时延

Warm up the GPUs by loading the model into the GPU and running a few dummy inferences. This will avoid one-time, cold-start latency hits when the first real request comes into the inference server.通过将模型加载到 GPU 并运行若干次虚拟推理来预热 GPU。这样可以避免第一个真实请求进入推理服务器时遭遇一次性的冷启动时延。

Partition resources efficiently for quality of service (QoS)为服务质量（QoS）高效划分资源

If running mixed, heterogeneous workloads, such as training and inference—or models with different architectures—on the same infrastructure, consider partitioning resources to ensure the latency-sensitive inference gets priority. This could mean dedicating some GPUs entirely to inference or using MIG to give an inference service a guaranteed slice of a GPU if it doesn’t need a full GPU but requires predictable latency. Separate inference from training on different nodes if possible, as training can introduce jitter with heavy I/O or sudden bursts of communication.如果在同一套基础设施上运行混合的异构工作负载，例如训练与推理，或不同架构的模型，应考虑划分资源，以确保对时延敏感的推理获得优先权。这可能意味着将部分 GPU 完全专用于推理，或者在推理服务不需要整块 GPU 但要求时延可预测时，使用多实例 GPU（MIG）为其提供有保障的 GPU 切片。尽可能将推理与训练放在不同节点上，因为训练可能因繁重的 I/O 或突发的通信而引入抖动。

Utilize Grace CPU for inference preprocessing利用 Grace CPU 进行推理预处理

In Grace Blackwell systems, the server-class CPU can handle preprocessing— such as tokenization and batch collation—extremely fast in the same memory space as the GPU. Offload such tasks to the CPU and have it prepare data in the shared memory that the GPU can directly use. This reduces duplication of buffers and leverages the powerful CPU to handle parts of the inference pipeline, freeing the GPU to focus on more compute-intensive neural-network computations.在 Grace Blackwell 系统中，服务器级 CPU 可以在与 GPU 相同的内存空间中极快地完成预处理，例如分词和批次组装。将这类任务卸载到 CPU，让它在 GPU 可直接使用的共享内存中准备数据。这减少了缓冲区重复，并利用强大的 CPU 承担推理流水线的部分工作，使 GPU 得以专注于计算密集程度更高的神经网络计算。

Tune carefully for edge AI and latency-critical deployments针对边缘 AI 与延迟敏感型部署进行细致调优

Extend performance tuning to the edge by leveraging specialized edge accelerators and optimizing data transfer protocols between central servers and edge devices. This will help achieve ultralow latency for time-sensitive applications.将性能调优延伸到边缘：利用专用的边缘加速器，并优化中心服务器与边缘设备之间的数据传输协议。这将有助于对时间敏感的应用实现超低时延。

多节点推理与服务

Multinode Inference and Serving

Disaggregating prefill/decode and sharding models lets you handle bigger contexts and more users with higher occupancy. Continuous batching and hierarchical memory/offload maintain flow even under long prompts and heavy concurrency.将预填充/解码解耦并对模型进行分片，可以处理更大的上下文和更多用户，并获得更高的占用率。连续批处理与分层内存/卸载机制即使在长提示词和高并发下也能维持处理流程的顺畅。

Specifically, disaggregate prefill and decode across devices, continuously pool tokens across requests, and shard oversized models via tensor/pipeline parallelism. Add hierarchical memory/offload for very long contexts so you serve more without OOMs, trading small latency for much higher capacity. The following performance tips apply to multinode inference and serving:具体而言，跨设备解耦预填充与解码，跨请求持续池化 token，并通过张量并行/流水线并行对超大模型进行分片。为超长上下文增加分层内存/卸载机制，从而在不发生 OOM 的情况下服务更多请求，以少量时延换取高得多的容量。以下性能建议适用于多节点推理与服务：

Disaggregate inference pipelines解耦推理流水线

Separate the inference workflow into distinct phases, including the “prefill” phase that processes the input prompt through all model layers, and the iterative “decode” phase that generates outputs token by token. Allocate these phases to different resources to allow for independent scaling. This two-stage approach prevents faster tasks from being bottlenecked by slower ones. For large language models, one strategy is to run the full model to encode the prompt, then handle autoregressive decoding on a stage-wise basis, possibly with specialized workers将推理工作流拆分为不同阶段，包括让输入提示词经过所有模型层的“预填充”阶段，以及逐 token 生成输出的迭代式“解码”阶段。将这些阶段分配到不同资源上，以便各自独立扩展。这种两阶段方法可避免较快的任务被较慢的任务拖成瓶颈。对于大语言模型，一种策略是先运行完整模型对提示词进行编码，再以分阶段的方式处理自回归解码，可能还会使用专用的工作进程

for each phase. By disaggregating the pipeline, you ensure that GPUs continuously work on the portion of the task they’re most efficient at, avoiding head-ofline blocking, where one long generation stalls others behind it.来分别处理每个阶段。通过解耦流水线，你可以确保 GPU 持续处理它们最高效的那部分任务，避免队头阻塞，即一个长生成任务阻塞了排在它后面的其他任务。

Use continuous batch processing for LLMs为 LLM 使用连续批处理

Move beyond simple request batching and use continuous batching strategies to maximize throughput under heavy loads. Traditional dynamic batching groups incoming requests and processes them as a batch to improve GPU utilization. Continuous batching takes this further by dynamically merging and splitting sequences of tokens across requests in real time. Systems like vLLM implement token pooling, where as soon as any thread is ready to generate the next token, it gets grouped with other ready threads to form a new batch. This approach keeps the GPU at high occupancy at all times and drastically reduces idle periods. The result is significantly higher token throughput and better latency consistency, especially when serving many concurrent users with varying sequence lengths.不要停留在简单的请求批处理上，而应使用连续批处理策略，在高负载下最大化吞吐量。传统的动态批处理将到达的请求分组，作为一个批次处理，以提高 GPU 利用率。连续批处理则更进一步，实时地在不同请求之间动态合并与拆分 token 序列。vLLM 等系统实现了 token 池化：一旦某个线程准备好生成下一个 token，它就会与其他已就绪的线程组合成一个新批次。这种方法使 GPU 始终保持在较高占用率，并大幅减少空闲时段。其结果是显著更高的 token 吞吐量和更好的一致性时延，在服务大量序列长度各异的并发用户时尤为明显。

Shard models efficiently across GPUs and nodes跨 GPU 与节点高效分片模型

For models that are too large to fit into a single GPU’s memory, employ model-parallel inference techniques by partitioning the model across multiple GPUs or even multiple servers. This can be done with tensor parallelism, in which it splits each layer’s weights and computation across devices, or pipeline parallelism, which splits the model’s layers into segments hosted on different GPUs and streams the data through them sequentially. While model sharding introduces communication overhead and some added latency as data must flow between shards, it enables deployment of trillion-parameter models that would otherwise be impossible to serve. Ensure high-speed interconnects, such as NVLink or InfiniBand, between GPUs to make this feasible, and overlap communication with computation where possible. The key is to balance the load so all devices work in parallel and no single stage becomes a bottleneck.对于大到无法放入单块 GPU 显存的模型，可采用模型并行推理技术，将模型划分到多个 GPU 甚至多台服务器上。这可以通过张量并行实现——把每一层的权重和计算拆分到各设备上；也可以通过流水线并行实现——把模型的层切分为若干段，分别托管在不同的 GPU 上，并让数据依次流过这些段。模型分片会引入通信开销，并且由于数据必须在分片之间流动而带来一定的额外时延，但它使得万亿参数模型的部署成为可能，否则这些模型根本无法提供服务。确保 GPU 之间具备 NVLink 或 InfiniBand 等高速互连，以使方案可行，并尽可能将通信与计算重叠。关键在于均衡负载，使所有设备并行工作，且没有任何单个阶段成为瓶颈。

Offload memory for extended contexts为扩展上下文卸载内存

Use hierarchical memory strategies to support inference workloads that demand more memory than GPUs have available. Incorporate memory offloading when serving very large models or long sequence contexts, such as long multiturn conversations and large documents. Less frequently used data, such as old attention KV cache entries or infrequently accessed model weights, can be moved to CPU RAM or even NVMe storage when GPU memory gets tight. Modern inference frameworks can automatically swap out these tensors and bring them back on the fly when needed. While this introduces additional latency for cache misses, it prevents out-of-memory errors and allows you to handle extreme cases. By thoughtfully offloading and prefetching data, you trade a bit of speed for the ability to serve requests with large working sets, achieving a better overall throughput under memory constraints.使用分层内存策略来支撑显存需求超出 GPU 可用容量的推理负载。在服务超大模型或长序列上下文（例如长多轮对话和大文档）时，引入内存卸载。当 GPU 显存紧张时，可将不常使用的数据——例如旧的注意力 KV 缓存条目或不常访问的模型权重——迁移到 CPU 内存甚至 NVMe 存储。现代推理框架可以自动换出这些张量，并在需要时即时换回。虽然这会因缓存未命中而引入额外时延，但它可避免显存不足错误，让你能够处理极端情况。通过有策略地卸载与预取数据，你以少量速度换取服务大工作集请求的能力，从而在显存受限条件下获得更好的整体吞吐量。

功耗与散热管理

Power and Thermal Management

Performance per watt is a first-class metric—thermal or power throttling erases tuning gains and shortens hardware life. Power caps, efficient packing, and proactive cooling stabilize clocks while cutting energy spend.每瓦性能是一等指标——散热或功耗降频会抹平调优收益并缩短硬件寿命。功耗上限、高效装箱和主动散热可稳定时钟频率，同时降低能耗支出。

Track perf/watt and thermals alongside speed: cap power or underclock memory-bound workloads for better efficiency with minimal throughput loss. Proactively manage cooling, consolidate jobs to run GPUs near full, monitor per-GPU power draw, and schedule around energy price/renewables when it reduces cost. Here are some tips on managing your power and thermal characteristics of your AI systems:在关注速度的同时跟踪每瓦性能与温度：对显存受限负载设置功耗上限或降低显存频率，可在吞吐量损失极小的前提下获得更高能效。主动管理散热，整合作业使 GPU 接近满载运行，监控每块 GPU 的功耗，并在能降低成本时围绕电价/可再生能源进行调度。以下是管理 AI 系统功耗与散热特性的一些建议：

Utilize efficient and environmentally friendly energy when possible尽可能使用高效且环保的能源

Track and optimize energy consumption alongside performance. In addition to managing power and thermal limits, monitor energy usage metrics and consider techniques that improve both performance and sustainability. For example, by implementing dynamic power capping or workload shifting based on renewable energy availability, you can reduce operational costs and carbon footprint. This dual focus reduces operational costs and supports responsible, environmentally friendly AI deployments.在关注性能的同时跟踪并优化能耗。除了管理功耗与温度上限外，还要监控能耗指标，并考虑能同时提升性能与可持续性的技术。例如，通过实施动态功耗上限或根据可再生能源可用性迁移负载，你可以降低运营成本与碳足迹。这种双重关注既降低了运营成本，也支持负责任、环境友好的 AI 部署。

Monitor thermals and clocks监控温度与时钟频率

Keep an eye on GPU temperature and clock frequencies during runs. If GPUs approach thermal limits (85°C in some cases), they may start throttling clocks, which reduces performance. Use nvidia-smi dmon or telemetry to see if clocks drop from their max. If you detect throttling, improve cooling, increase fan speeds, improve airflow, or slightly reduce the power limit to keep within a stable thermal envelope. The goal is consistent performance without thermal-induced dips.运行期间密切关注 GPU 温度和时钟频率。如果 GPU 接近温度上限（某些情况下为 85°C），它们可能开始降频，从而降低性能。使用 nvidia-smi dmon 或遥测数据查看时钟频率是否从最大值下降。如果检测到降频，应改善散热、提高风扇转速、改善气流，或略微降低功耗上限，以保持在稳定的温度包络内。目标是获得一致的性能，避免因温度导致的性能下滑。

Use energy-aware dynamic power management使用能耗感知的动态功耗管理

Modern data centers are increasingly using energy-aware scheduling to adjust workloads based on real-time energy costs and renewable energy availability. Incorporating adaptive power capping and dynamic clock scaling can help optimize throughput per watt while reducing operational costs and carbon footprint.现代数据中心越来越多地采用能耗感知调度（energy-aware scheduling），根据实时能源成本和可再生能源可用性来调整工作负载。引入自适应功耗封顶（adaptive power capping）和动态时钟调节（dynamic clock scaling），有助于优化每瓦吞吐量，同时降低运营成本和碳足迹。

Optimize for perf/watt针对每瓦性能进行优化

In multi-GPU deployments where power budget is constrained (or energy cost is high), consider tuning for efficiency. Many workloads, especially memory-bound ones, can run at slightly reduced GPU clocks with negligible performance loss but noticeably lower power draw. For example, if a kernel is memory bound, locking the GPU at a lower clock can save power while not hurting runtime. This increases throughput per watt. Test a few power limits using nvidia-smi -pl to see if your throughput/watt improves. For some models, going from a 100% to 80% power limit yields nearly the same speed at 20% less power usage.在功耗预算受限（或能源成本较高）的多 GPU 部署中，可考虑针对能效进行调优。许多工作负载，尤其是显存受限型工作负载，可以在 GPU 时钟略有降低的情况下运行，性能损失可忽略不计，但功耗明显下降。例如，如果某个内核是显存受限的，将 GPU 锁定在较低时钟可以节省功耗而不影响运行时间。这会提高每瓦吞吐量。使用 nvidia-smi -pl 测试几个功耗上限，看看吞吐量/瓦是否改善。对某些模型而言，将功耗上限从 100% 降到 80%，速度几乎相同，而功耗降低 20%。

Use adaptive cooling strategies采用自适应冷却策略

If running in environments with variable cooling or energy availability, integrate with cluster management to adjust workloads. For instance, schedule heavy jobs during cooler times of the day or when renewable energy supply is high—if that’s a factor for cost. Some sites implement policies to queue nonurgent jobs to run at night when electricity is cheaper. This doesn’t change single-job performance but significantly cuts costs.如果在冷却能力或能源供应可变的环境中运行，应与集群管理集成以调整工作负载。例如，在一天中较凉爽的时段或可再生能源供应充足时调度重型作业——如果这对成本有影响的话。一些站点实施策略，将非紧急作业排队到夜间电价较低时运行。这不会改变单个作业的性能，但能显著降低成本。

Consolidate workloads整合工作负载

Run GPUs at high utilization rather than running many GPUs at low utilization. A busy GPU is more energy efficient in terms of work done per watt than an idle or lightly used GPU. This is because the baseline power is better amortized when the GPU is busy. It may be better to run one job after another on one GPU at 90% utilization than two GPUs at 45% each in parallel—unless you need to optimize for the smallest wall-clock time. Plan scheduling to turn off or idle whole nodes when not in use, rather than leaving lots of hardware running at low utilization.让 GPU 保持高利用率运行，而不是让许多 GPU 低利用率运行。就每瓦完成的工作量而言，繁忙的 GPU 比空闲或轻度使用的 GPU 更节能。这是因为 GPU 繁忙时基线功耗能得到更好的摊销。在一块 GPU 上以 90% 利用率依次运行作业，可能优于在两块 GPU 上各以 45% 利用率并行运行——除非你需要优化最短实际耗时。规划调度，在不用时关闭或闲置整个节点，而不是让大量硬件低利用率运行。

Configure cooling efficiently高效配置冷却

For air-cooled systems, consider setting GPU fans to a higher fixed speed during heavy runs to preemptively cool the GPUs. Some data centers always run fans at the maximum to improve consistency. Ensure inlet temps in the data center are within specifications. Check periodically for dust or obstructions in server GPUs. Clogged fins can greatly reduce cooling efficiency. For water-cooled, ensure flow rates are optimal and water temperature is controlled.对于风冷系统，可考虑在重负载运行期间将 GPU 风扇设为较高的固定转速，以预先冷却 GPU。一些数据中心始终以最高转速运行风扇，以提高一致性。确保数据中心进风温度在规格范围内。定期检查服务器 GPU 是否有灰尘或堵塞物。散热鳍片堵塞会大幅降低冷却效率。对于水冷系统，确保流量最优且水温受控。

Monitor power carefully仔细监控功耗

Use tools to monitor per-GPU power draw. nvidia-smi reports instantaneous draw, which helps in understanding the power profile of your workload. Spikes in power might correlate with certain phases. For example, the all-reduce phase might measure less compute load and less power, while dense layers will spike the load and power measurements. Knowing this, you can potentially sequence workloads to smooth power draw. This is important if operating the cluster on a constrained power circuit. In the power-constrained scenario, you may need to avoid running multiple power-spikey jobs simultaneously on the same node to avoid tripping power limits.使用工具监控每块 GPU 的功耗。nvidia-smi 报告瞬时功耗，有助于了解工作负载的功耗特征。功耗尖峰可能与某些阶段相关。例如，全归约阶段可能测得较低的计算负载和功耗，而密集层会使负载和功耗测量值出现尖峰。了解这一点后，你可以对工作负载排序以平滑功耗。如果集群运行在受限的供电回路上，这一点很重要。在功耗受限的场景中，你可能需要避免在同一节点上同时运行多个功耗尖峰型作业，以免触发功耗限制。

Improve job resilience for long-running jobs提升长时间运行作业的韧性

If you are running a months-long training job or 24-7 inference job, consider the impact of thermals on hardware longevity. Running at 100% power and thermal limit constantly can marginally increase failure risk over time. In practice, data center GPUs are built for this type of resiliency, but if you want to be extra safe, running at 90% power target can reduce component stress with minimal slowdown. It’s a trade-off of longer training runs versus less wear on the如果你正在运行长达数月的训练作业或 7×24 的推理作业，请考虑热量对硬件寿命的影响。持续以 100% 功耗和温度上限运行，会随着时间推移略微增加故障风险。实际上，数据中心 GPU 就是为这种韧性而设计的，但如果你想格外稳妥，以 90% 的功耗目标运行可以减轻组件压力，而性能下降极小。这是在更长的训练运行时间与更少的硬件磨损之间的权衡——尤其是当

hardware—especially if that hardware will be reused for multiple projects over a long period of time.该硬件将在很长一段时间内被多个项目复用时。

结语

Conclusion

Treat the checklist as a repeatable playbook: profile, tune the right bottleneck at the right layer, and verify gains before scaling out. By methodically applying these practices—from OS and kernels to distributed comms and serving—you’ll achieve fast, cost-efficient, and reliable AI systems at any size.把这份清单当作可重复执行的行动手册：做性能剖析，在正确的层次上调优正确的瓶颈，并在横向扩展之前验证收益。通过有条不紊地应用这些实践——从操作系统与内核到分布式通信与服务化——你将获得在任何规模下都快速、高性价比且可靠的 AI 系统。

This list, while comprehensive, is not exhaustive. The field of AI systems performance engineering will continue to grow as hardware, software, and algorithms evolve. And not every best practice listed here applies to every situation. But, collectively, they cover the breadth of performance engineering scenarios for AI systems. These tips encapsulate much of the practical wisdom accumulated over years of optimizing AI system performance.这份清单虽然全面，但并非穷尽。随着硬件、软件与算法的演进，AI 系统性能工程这一领域将持续发展。而且这里列出的每一条最佳实践并非都适用于所有情形。但总体而言，它们覆盖了 AI 系统性能工程场景的广度。这些技巧凝聚了多年来优化 AI 系统性能所积累的大量实践智慧。

When tuning your AI system, you should systematically go through each of the relevant categories listed in this chapter and run through each of the items in the checklist. For example, you should ensure the OS is tuned, confirm GPU kernels are efficient, check that you’re using libraries properly, monitor the data pipeline, optimize the training loop, tune the inference strategies, and scale out gracefully. By following these best practices, you can diagnose and resolve most performance issues and extract the maximum performance from your AI system.在调优 AI 系统时，你应当系统地过一遍本章列出的每个相关类别，并逐项检查清单中的每一条。例如，你应当确保操作系统已完成调优，确认 GPU 内核高效，检查库的使用是否正确，监控数据流水线，优化训练循环，调优推理策略，并平滑地横向扩展。遵循这些最佳实践，你就能诊断并解决大多数性能问题，从 AI 系统中榨取最大性能。

And remember that before you scale up your cluster drastically, you should profile on a smaller number of nodes and identify potential scale bottlenecks. For example, if you see an all-reduce collective operation already taking 20% of an iteration on 8 GPUs, it will only get worse at a larger scale—especially as you exceed the capacity of a single compute node or data center rack system, such as the Grace Blackwell GB200 and GB300 NVL72 and Vera Rubin VR200 and VR300 NVL systems.并且请记住，在大幅扩展集群规模之前，你应当在较少数量的节点上做性能剖析，找出潜在的规模瓶颈。例如，如果你发现在 8 块 GPU 上 all-reduce 集合通信操作已经占用了单次迭代 20% 的时间，那么规模更大时只会更糟——尤其是当你超出单个计算节点或数据中心机架系统的容量时，例如 Grace Blackwell GB200 与 GB300 NVL72，以及 Vera Rubin VR200 与 VR300 NVL 系统。

Keep this checklist handy and add to it as you discover new tricks. Combine these tips and best practices with the in-depth understanding from the earlier chapters, and you will design and run AI systems that are efficient, scalable, maintainable, cost-effective, and reliable.把这份清单放在手边，并在发现新技巧时不断补充。将这些技巧与最佳实践同前面各章的深入理解结合起来，你就能设计并运行高效、可扩展、可维护、高性价比且可靠的 AI 系统。

Now go forth and make your most ambitious ideas a reality. Happy optimizing!现在就去把你最雄心勃勃的想法变为现实吧。优化愉快！