前言

In the vibrant streets of San Francisco, where innovation is as common as autonomous vehicle traffic on US Route 101, we find ourselves surrounded by an amazing world of artificial intelligence. Rapid advancements in AI are redefining our daily lives in every aspect. Over the last 20 years, we’ve experienced recommendation engines (2000s), AI assistants (2010s), and fully autonomous vehicles (2020s). The 2030s are going to be even more exciting, as AI is progressing extremely quickly and with massive societal influence.在旧金山的活力街头，创新的普遍程度堪比 101 号美国国道上的自动驾驶车流，我们身处一个令人惊叹的人工智能世界之中。AI 的快速进步正在全方位地重新定义我们的日常生活。过去 20 年间，我们先后经历了推荐引擎（2000 年代）、AI 助手（2010 年代）和完全自动驾驶汽车（2020 年代）。2030 年代将更加令人振奋，因为 AI 正以极快的速度发展，并产生巨大的社会影响。

My personal journey into the fast-moving AI systems performance engineering field was driven by a curiosity to understand the delicate balance and codesign between cutting-edge hardware, highly optimized software, and clever algorithms that power such complex systems and impactful use cases. This realization inspired me to dive deep into the realm of “full-stack” AI performance engineering. I wanted to understand how multiple components like processors, memory architectures, network interconnects, operating systems, and software frameworks all work together in harmony. The complexity of these interactions presented the challenges—and opportunities—that fueled my desire to dive deep and explore this unique combination of technologies.我个人踏入这个快速演进的 AI 系统性能工程领域的旅程，源于一种好奇心：想理解支撑如此复杂系统和有影响力用例的前沿硬件、高度优化的软件与巧妙算法之间那种微妙的平衡与协同设计（codesign）。这一认识促使我深入“全栈”AI 性能工程领域。我想弄明白处理器、内存架构、网络互连、操作系统和软件框架等多个组件如何协同一致地工作。这些交互的复杂性带来了挑战——也带来了机遇——正是它们激发了我深入探索这一独特技术组合的渴望。

This book is a realization of my explorations throughout the years as a hands-on ML and AI performance engineer. I created this book for engineers, researchers, practitioners, and enthusiasts who are eager to understand the underpinnings of AI systems performance at all levels. Readers might be building AI applications, optimizing neural network training strategies, or designing and managing scalable inference servers, or they may simply be fascinated by the mechanics of modern AI systems. Overall, this book provides the insights that bridge theory and practice across multiple disciplines.本书是我多年来作为一线机器学习与 AI 性能工程师所做探索的结晶。我写这本书，是为了那些渴望在各个层面理解 AI 系统性能底层机理的工程师、研究人员、从业者和爱好者。读者可能正在构建 AI 应用、优化神经网络训练策略，或设计与管理可扩展的推理服务器，也可能只是对现代 AI 系统的运作机制着迷。总体而言，本书提供的洞见跨越多个学科，架起了理论与实践之间的桥梁。

The reader of this book likely has a foundational understanding of neural networks and a basic familiarity with Python and ML. However, even without these fundamentals, a curious reader can follow the multidimensional codesign performance narrative rooted in the first principles across hardware, software, and algorithms. I promise there is something in this book for every type of reader—and I guarantee every reader will learn a few new things in these pages.本书读者大概具备神经网络的基础知识，并对 Python 和机器学习有基本了解。不过，即使没有这些基础，充满好奇心的读者也能跟上这条植根于第一性原理、横跨硬件、软件与算法的多维协同设计性能叙事。我保证，本书对每一类读者都有可取之处——我也确信每位读者都能在书中收获一些新知识。

Throughout the chapters, we examine the evolution of hardware architectures, dive into the nuances of software optimization, and explore real-world case studies that highlight the patterns and best practices of building both high-performance and cost-efficient AI systems. Each section is designed to build upon the last, covering everything from foundational concepts to advanced applications.在各章中，我们考察硬件架构的演进，深入软件优化的细节，并探讨真实案例研究，以凸显构建高性能且高性价比 AI 系统的模式与最佳实践。每一节都在前一节的基础上展开，内容涵盖从基础概念到高级应用的方方面面。

本书使用的排版约定

Conventions Used in This Book

The following typographical conventions are used in this book:本书采用以下排版约定：

Italic斜体

Indicates new terms, URLs, email addresses, filenames, and file extensions.表示新术语、URL、电子邮件地址、文件名和文件扩展名。

Constant width

Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords.用于程序清单，也用于正文段落中指代程序元素，如变量名或函数名、数据库、数据类型、环境变量、语句和关键字。

Constant width bold

Shows commands or other text that should be typed literally by the user.表示应由用户原样键入的命令或其他文本。

Constant width italic

Shows text that should be replaced with user-supplied values or by values determined by context.表示应替换为用户提供的值或由上下文决定的值的文本。

使用代码示例

Using Code Examples

Supplemental material (code examples, exercises, etc.) is available for download at https://github.com/cfregly/ai-performance-engineering.补充材料（代码示例、练习等）可在 https://github.com/cfregly/ai-performance-engineering 下载。

If you have a technical question or a problem using the code examples, please send email to support@oreilly.com.如果你在使用代码示例时遇到技术问题或疑问，请发送电子邮件至 support@oreilly.com。

This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission.本书旨在帮助你完成工作。一般而言，如果本书附带了示例代码，你可以在自己的程序和文档中使用它们。除非你要复制大量代码，否则无需联系我们获取许可。例如，编写一个使用本书中若干段代码的程序不需要许可。销售或分发 O'Reilly 图书中的示例则需要许可。通过引用本书并摘录示例代码来回答问题不需要许可。将本书中大量示例代码纳入你产品的文档中则需要许可。

We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “AI Systems Performance Engineering by Chris Fregly (O’Reilly). Copyright 2026 Flux Capacitor, LLC, 979-8-341-62778-9.”我们感谢署名，但通常不要求署名。署名通常包括书名、作者、出版社和 ISBN。例如：“AI Systems Performance Engineering by Chris Fregly (O'Reilly). Copyright 2026 Flux Capacitor, LLC, 979-8-341-62778-9.”

If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com.如果你认为自己对代码示例的使用超出了合理使用或上述许可的范围，欢迎通过 permissions@oreilly.com 联系我们。

O'Reilly 在线学习

O’Reilly Online Learning

For more than 40 years, O’Reilly Media has provided technology and business training, knowledge, and insight to help companies succeed.40 多年来，O'Reilly Media 一直提供技术与商业培训、知识和洞见，帮助企业取得成功。

Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit https://oreilly.com.我们由专家和创新者组成的独特网络通过图书、文章和在线学习平台分享他们的知识与专长。O'Reilly 的在线学习平台让你按需访问实时培训课程、深度学习路径、交互式编程环境，以及来自 O'Reilly 和 200 多家出版商的庞大文本与视频资源。欲了解更多信息，请访问 https://oreilly.com。

如何联系我们

How to Contact Us

Please address comments and questions concerning this book to the publisher:请将有关本书的评论和问题发送给出版社：

O’Reilly Media, Inc. 141 Stony Circle, Suite 195 Santa Rosa, CA 95401 800-889-8969 (in the United States or Canada) 707-827-7019 (international or local) 707-829-0104 (fax) support@oreilly.com https://oreilly.com/about/contact.htmlO'Reilly Media, Inc. 141 Stony Circle, Suite 195 Santa Rosa, CA 95401 800-889-8969（美国或加拿大）707-827-7019（国际或本地）707-829-0104（传真）support@oreilly.com https://oreilly.com/about/contact.html

We have a web page for this book, where we list errata and any additional information. You can access this page at https://oreil.ly/AI-SysPerfEng.我们为本书设立了一个网页，其中列出了勘误和任何补充信息。你可以通过 https://oreil.ly/AI-SysPerfEng 访问该页面。

For news and information about our books and courses, visit https://oreilly.com.欲了解我们图书和课程的新闻与信息，请访问 https://oreilly.com。

Find us on LinkedIn: https://linkedin.com/company/oreilly-media.在 LinkedIn 上找到我们：https://linkedin.com/company/oreilly-media。

Watch us on YouTube: https://youtube.com/oreillymedia.在 YouTube 上观看我们：https://youtube.com/oreillymedia。

致谢

Acknowledgments

Although I’m the sole author of this book, this effort builds on the work of many practitioners and academics who have dedicated their lives to these topics. I am indebted to the awesome folks whose research and innovations have paved the way for the topics covered in this book. Their contributions helped shape the content and depth of this work. I’d like to thank the end-to-end reviewers (Adrian Cockroft, Antje Barth, Arpitha Moorthy, Chaim Rand, Fabrizio Milo, Simon Zamarin, Suneeta Mall, and Todd Bezenek) as well as many part-time reviewers who greatly improved the overall quality of this book. You challenged me and my understanding of these complex topics. And your insights are embedded in every page of this book.尽管我是本书的唯一作者，但这项工作建立在许多从业者和学者的成果之上，他们毕生致力于这些课题。我要感谢那些出色的同仁，他们的研究与创新为本书所涵盖的主题铺平了道路。他们的贡献帮助塑造了本书的内容与深度。我要感谢全篇审阅者（Adrian Cockroft、Antje Barth、Arpitha Moorthy、Chaim Rand、Fabrizio Milo、Simon Zamarin、Suneeta Mall 和 Todd Bezenek），以及许多部分章节的审阅者，他们极大地提升了本书的整体质量。你们挑战了我，也挑战了我对这些复杂课题的理解。你们的洞见已融入本书的每一页。

I’d especially like to thank Mark Saroufim from Meta. He and his successful GPU Mode community helped me realize the need for this type of deep technical content covering hardware-software co-design across PyTorch, CUDA, and GPUs. I know how much work goes into maintaining and growing a large community for advanced topics such as these.我要特别感谢来自 Meta 的 Mark Saroufim。他和他的 GPU Mode 社区非常成功，帮助我认识到市场需要这类深入的技术内容，涵盖跨 PyTorch、CUDA 和 GPU 的软硬件协同设计。我深知维护并发展一个面向此类高级课题的大型社区需要付出多少努力。

And to my close family (Ann Marie, Meredith, Tommy, and Conor), whose continuous support kept me writing through the night and into the early morning hours, I extend much heartfelt gratitude. Apologies if I wasn’t always “present” while visiting you over the past year...I was down in the basement researching, coding, and writing!还要感谢我的至亲家人（Ann Marie、Meredith、Tommy 和 Conor），你们持续的支持让我通宵达旦、直到凌晨仍在写作，我向你们致以由衷的感激。如果过去一年探望你们时我并不总是“在场”，我深表歉意……我当时正在地下室做研究、写代码和写作！

And in a somewhat peculiar way, I’d like to also thank my car, Little Red Whiting Hood (Figure P-1). You are a complex, multidisciplinary system of highly tuned hardware and software. I’ve had you since I started this San Francisco journey more than 15 years ago. Every time I drive you, you remind me why I became an engineer. In relative GPU generations, you would be the NVIDIA Ampere A100—a timeless classic built by multidisciplinary engineers using the highest-performance components available at that time. And while you were originally designed for air cooling (Luftkühlung), you understand that water cooling (Wasserkühlung) is required for maximum performance. Your creator has stubbornly and relentlessly pushed your physical engineering limits generation after generation. Your turbos are like Tensor Cores, which get both moody and hot when pushed too hard. I’ll never get rid of you.以一种略显奇特的方式，我还要感谢我的车——Little Red Whiting Hood（图 P-1）。你是一个由高度调校的硬件与软件构成的复杂多学科系统。从我 15 多年前开始旧金山之旅起，你就一直陪伴着我。每次驾驶你，你都会提醒我当初为何成为一名工程师。若以 GPU 世代类比，你相当于 NVIDIA Ampere A100——一款由多学科工程师采用当时可获得的最高性能组件打造的不朽经典。虽然你最初是为风冷（Luftkühlung）而设计，但你也明白要实现最大性能需要水冷（Wasserkühlung）。你的创造者一代又一代地固执而执着地推动你的物理工程极限。你的涡轮就像张量核心，被逼得太狠时既会闹脾气也会发热。我永远不会把你处理掉。

And as the lyrics in the classic Phish song, “Contact,” say:正如 Phish 乐队的经典歌曲《Contact》中的歌词所唱：

Thanks for encouraging me to embrace mechanical sympathy, Little Red!谢谢你鼓励我拥抱机械同理心，Little Red！

Figure P-1. Dashboard of a high-performance, mechanically sympathetic Porsche 911 Turbo named “Little Red Whiting Hood” (white exterior, red interior)图 P-1. 高性能、机械感十足的保时捷 911 Turbo 仪表盘，名为“Little Red Whiting Hood”（白色外观，红色内饰）

As we continue to push the frontiers of artificial intelligence and supercomputing, the knowledge in this book aims to inspire, educate, and empower the reader. The exploration through the complexities of AI Systems Performance Engineering is not just a technical exploration—it’s a reminder of human ingenuity, the need to understand our surroundings, and a desire to continuously improve through technology and innovation. There are few people in this world who understand the fundamentals of codesigning hardware, software, and algorithms for maximum performance and efficiency. After reading this book, you will be one of them!随着我们不断拓展人工智能与超级计算的边界，本书中的知识旨在启发、教育并赋能读者。对 AI Systems Performance Engineering 复杂性的探索不仅是一次技术探索——它提醒着人类的创造力、理解周围环境的必要性，以及通过技术与创新持续改进的愿望。世界上很少有人理解为最大化性能与效率而协同设计硬件、软件和算法的基础。读完本书后，你将成为其中之一！

Chris Fregly San Francisco, CaliforniaChris Fregly 加利福尼亚州旧金山