1 Introduction 前言

Rocket Mortgage wanted to scale Agile while still providing autonomy for engineering teams. Agile environments are critical to adapting to changing conditions, and during COVID-19, Agile business units outperformed their non-Agile counterparts by up to 94% [1]. Rocket Mortgage began formally adopting a consistent Agile methodology in Q3 of 2018 before COVID-19 by implementing a Scaled Agile framework. By Q4, the implementation resulted in the formation of 26 release trains and improved feature throughput by 101% by Q3 2019, as measured by the number of features completed in a program increment. Deployment continues to expand, and in 2021 there are 41 release trains of roughly 30-90 people each.

Rocket Mortgage贷款公司希望在为工程团队提供自主权的同时扩大敏捷的规模。敏捷的环境对于适应不断变化的条件而言是至关重要的，在COVID-19期间，采用敏捷的业务单元比传统业务单元的业绩高出94%[1]。在COVID-19爆发前，Rocket Mortgage于2018年第三季度开始在全公司的范围内实施SAFe。到了第四季度，已经形成了26个发布火车，并在2019年第三季度前将特性吞吐量提高了101%，该数据是由以一个项目群增量（Program Increment, PI）中完成的特性数量来衡量的。之后，敏捷实施还在继续扩大；到2021年有41列发布火车，每列火车大约有30-90人。

But MIT Sloan Business Review [2] reports only 17% of leading companies today will remain leaders in five years. Knowing this, the Rocket Mortgage Director of Engineering for the front end of the mortgage loan systems began evaluating the Scaled Scrum framework as a tool for improving their Scaled Agile implementation. This is significant because it is commonly assumed that Scaled Agile and Scaled Scrum are incompatible and are either/or options. Here we address the results when Scaled Scrum is added correctly to a highly successful Scaled Agile implementation.

但麻省理工学院斯隆商业评论[2]报告说，今天处于领先地位的公司中只有17%能在五年内继续保持领先。了解到这一点，Rocket Mortgage贷款系统前端的研发总监开始评估另一种规模化框架--Scrum@Scale，看看能不能在SAFe实施的基础上继续提高。这点值得敲一下重点：因为人们普遍认为SAFe和Scrum@Scale是不相容的，是非此即彼的选择。在这篇文章中，大家将会看到把Scrum@Scale正确地添加到一个非常成功的SAFe实施中取得的丰硕成果。

7 Countermeasures Taken 采取对策

Because Scaled Scrum and Scaled Agile are both major Agile frameworks, it is generally accepted that one or the other should be selected [30]. But while the rest of Product Engineering stopped at Scaled Agile, Client Marketing took the extra steps of putting Scaled Scrum on top of Scaled Agile. This allowed them to address all the issues found in their root cause analysis and reach their target conditions, essentially applying the A3 Process to Scaled Agile. As part of the process of adding Scaled Scrum, the countermeasures below were put into place.

因为Scrum@Scale和SAFe都是主要的规模化敏捷框架，人们普遍认为应该选择其中之一[30]。但是，当产品工程部的其他部门满足于SAFe实施时，客户营销发布火车采取了额外的措施，在SAFe之上继续应用Scrum@Scale的一些原则。这使得他们能够解决在根本原因分析中发现的所有问题，并达到他们的目标条件。这么做本质上是将A3流程应用于SAFe。作为Scrum@Scale过程的一部分，他们采取的对策如下：

7.1Scrum Reset 重置Scrum

In addition to the organization-wide Scaled-Agile transformation, Client Marketing employed four primary tactics to improve and scale its Scrum practice. First, Client Marketing senior leadership completed training with Jeff Sutherland on scaling Scrum. Second, Client Marketing was retrained in the three Scrum roles, the five events, and the three artifacts. Third, all of Client Marketing did a "hard reset," establishing a strict implementation of Scrum at the team level. While each team within the release train was already doing some variation of Scrum, leadership set the expectation that each team would strip away any complexity that had been layered on over the years and revert to following the Scrum Guide [9].

在公司整体实施SAFe的基础上，客户营销发布火车还采用了四个主要策略来改善和扩展Scrum实践。首先，客户营销发布火车高层领导与Jeff Sutherland一起完成了关于规模化Scrum的培训。第二，客户营销发布火车接受了三个Scrum角色、五个事件和三个工件的再培训。第三，所有的客户营销发布火车做了一次 "硬重置"，在团队层面建立了严格的Scrum实施。虽然每个团队在发布火车中已经在做一些Scrum的变体，但领导层设定了一个期望，即将每个团队从多年来层层设置的枷锁中剥离出来，并恢复到以遵循Scrum指南为准的状态[9]。

Before the reset, only one-third of Scrum Teams held Daily Scrums, Sprint Planning, Sprint Retrospectives, or had a Definition of Ready. None of the Scrum Teams had a Definition of Done. After the reset, all teams followed each ceremony as defined in the Scrum Guide [20] and had Definitions of Ready and Done. Finally, the Client Marketing leadership began to operate as a Scrum team as well.

在重置之前，只有三分之一的Scrum团队举行了每日Scrum、Sprint计划、Sprint回顾，或者有一个 "就绪的定义(DOR)"。没有一个Scrum团队有 "完成的定义（DoD）”。重置后，所有的团队都遵循了Scrum指南[20]中定义的每个仪式，并且有了 "DOR "和 "DOD "的定义。最后，客户营销发布火车的领导层也开始作为一个Scrum团队运作。

7.2 Scrum Roles Scrum角色

Consistent with the Scrum Guide [9], it was agreed that POs were primarily responsible for increasing the product's value, while SMs were accountable for accelerating delivery. All permutations of these roles were eliminated.

严格将Scrum的角色与Scrum指南[9]一致，即：PO主要负责提高产品的价值，而SM则负责加速交付。这些角色的所有衍变都被取消了。

7.3 MetaScrum 产品决策团队-- MetaScrum

The MetaScrum is a regular meeting (at least once a Sprint) with Senior Leadership, the Chief PO, and senior members of the PO and engineering teams [10]. The purpose is to align the organization with the Chief PO's backlog. This meeting was first implemented at Patient Keeper [31] in 2003 and was later formally defined by the Scrum Patterns Group [32] as essential to high-performing organizations with many Scrum teams. Client Marketing implemented a weekly MetaScrum to coordinate prioritization and dependencies and achieve ongoing alignment with stakeholders. The process was led by the Release Train Engineer and involved consolidating all prioritization, train-level work intake, feature refinement, and dependency coordination discussions down to a weekly meeting with all stakeholders, product managers, and leaders present. This allowed prioritization to be aligned through POs down to each team.

MetaScrum是由高级领导层、首席PO、PO和工程团队的高级成员参加的定期会议（至少每个Sprint一次）[10]。其目的是使组织与首席PO的待办列表保持一致。这个概念于2003年在Patient Keeper[31]首次实施，后来被Scrum模式小组[32]正式定义为MetaScrum，并作为拥有多个Scrum团队的高绩效组织的精髓对外推广。客户营销发布火车实施了每周一次的MetaScrum，以协调优先级和依赖，并与利益相关者持续保持一致。这个过程由发布火车工程师（Release Train Engineer-RTE，译者注：SAFe框架中的一个角色）主导，内容包括提前整合所有的优先级、火车级的工作项、特性精炼和依赖协调的讨论，并在每周的会议上与所有的利益相关者、产品经理和领导会面。这样可以将优先级通过PO传递到每个团队，从而使各团队对齐。

7.4 Scaled Daily Scrum 规模化的Daily Scrum（SDS）

As MetaScrums began, Client Marketing leadership started holding a Scaled Daily Scrum (SDS) with all the SMs in the Release Train. Two of the most significant changes of the SDS from the previous Leadership Standup were the time it occurred and its focus. The SDS occurred after each team's Daily Scrum, and the focus was purely on delivery. With a dashboard with Sprint burn down charts for each team prominently displayed on a screen, each SM would report on their team's progress towards the Sprint goal and any impediments.

随着MetaScrums的开始实施，客户营销发布火车的领导层开始与发布火车上的所有SM举行规模化的每日Scrum（SDS）。与以前的领导层会议相比，SDS的两个最重要的变化是它的时间和专注点。SDS发生在每个团队的Daily Scrum之后，其专注点是纯粹的交付。在屏幕上醒目地显示着每个团队的Sprint 燃尽图，每个SM将报告他们的团队在实现Sprint目标方面的进展和任何障碍。

If a team did not clearly show to be trending towards early Sprint completion, the SM was expected to bring one or more impediments to the SDS that they were focused on removing. If the SM could not eliminate an impediment within hours, it became the highest priority item the Release Train Leader would focus on. The SDS converted a low-value status report meeting about the schedules of SMs into a high-value event focused on taking immediate action to resolve challenges that same day that teams were facing to hit Sprint goals. This Scaled Scrum ceremony, along with the Daily Scrum, directly attack and resolve decision latency. The importance of this cannot be overstressed, as the Standish Group noted, “The value of the interval is greater than the quality of the decision” and “The root cause of poor performance in a software project is slow decision latency [33].”

如果一个团队没有明确显示出提前完成Sprint目标的趋势，SM就应该向SDS提出团队遇到的一个或多个障碍，他们要集中精力消除这些障碍。如果SM不能在几个小时内消除一个障碍，它就会成为发布火车领导们需要关注的最高优先级的事项。SDS将一个关于SM日程安排的低价值“状态报告会”转化为一个高价值的活动，专注于采取立即行动来解决团队当天面临的挑战，以达到Sprint目标。这个Scrum@Scale仪式和Daily Scrum一起，直接攻克和解决决策延迟的问题。这一点的重要性怎么强调都不为过，正如Standish Group所指出的，"决策的效率比决策是否完美更有价值"，"软件项目中表现不佳的根本原因是决策延迟过长[33]"。

7.5 Role Alignment 角色调整

The leadership team examined all roles and their responsibilities. Any role that was not playing a direct part in delivery, or was causing a bottleneck, was eliminated or repurposed. With SMs focused on removing constraints within the team's delivery process, it became increasingly clear that traditional quality assurance and specialized roles that only concentrate on a small part of the delivery lifecycle were causing bottlenecks. SMs worked with team members in these roles to either broaden their skillset to apply to any user story in the backlog or helped them find positions within the broader organization that still required their high degree of specialization.

领导团队检查了所有的角色和他们的责任。任何在交付过程中没有发挥直接作用的角色，或者造成瓶颈的角色，都被取消或重新安排。随着SM专注于消除团队交付过程中的制约因素，大家越来越明显地看到，传统的质量保证和只专注于交付生命周期的一小部分的专门角色正在造成瓶颈。SM与这些角色的团队成员一起工作，要么扩大他们的技能范围，使他们能够承担待办列表中的任何用户故事，要么帮助他们在更广泛的组织内找到仍然需要他们高度专业化的岗位。

For team members in the role of Quality Analyst (QAs), there was a six-month transition period during which they became Software Engineers (SEs). Those who were unable to do so were transferred to a specialty test group within the larger organization. Business Analysts (BAs) became Software Engineers, POs, or were assigned to the specialty test group.

对于担任质量分析师（QAs）的团队成员，有六个月的过渡期可以成为软件工程师（SE）。那些不能这样做的人被转移到上级部门的专业测试组。业务分析师们（BAs）要么成为软件工程师、PO或被分配到专业测试组。

7.6 Testing 测试

With the elimination of QA roles, the testing requirements increased for SEs. Because there was already a bottleneck in testing, the Client Marketing leadership team took corrective action. Automation testing was increased. This reduced the bottleneck and minimized the amount of manual testing. Unit, accessibility, and regression testing were all automated. The only testing still performed manually was visual testing on devices and browsers.

随着QA角色的取消，软件工程师的测试需求增加了。由于测试已经出现了瓶颈，客户营销领导团队采取了纠正措施，增加了自动化测试。这减少了瓶颈，并最大限度地减少了人工测试的数量。单元测试、可访问性测试和回归测试都被自动化了。唯一仍在手工进行的测试是设备和浏览器上的视觉测试。

7.7 Quality Gates 质量门禁

Quality gates were added to the pipeline to ensure the high standards of Rocket Mortgage were maintained. The quality gates verified that increased performance and shorter time to market were not detrimental to quality. Successfully passing the gates meant successfully passing performance, regression, and accessibility testing. The gates added were:

质量门禁被添加到流水线中，以确保Rocket Mortgage的高标准得到保持。质量门禁验证了提高性能、保证缩短上市时间的同时不会对质量造成损害。成功通过这些关卡意味着成功通过性能、回归和可访问性测试。增加的门禁包括：

Accessibility Testing

An automated testing tool that produces a score value which is then compared against a predefined minimum score to be considered passing.

可访问性测试

通过一个自动化测试工具，产生一个分数值，然后与预定的最低分数进行比较，如果大于这个最低分数就可以认为是通过了。

Secrets Scanner

The Secret Scanner looks for plain text secrets within source code. Anything that could grant knowledge or access to items that people should otherwise not have access would be considered a secret.

秘密扫描器

秘密扫描器寻找源代码中的纯文本秘密（译者注：比如密钥）。任何可能授予人们知识或人们不应该访问东西会被认为是秘密。

Security Scanner

Security Scanner is an automated scan that checks for any dependency vulnerabilities.

安全扫描器

安全扫描器是一种自动扫描，可以检查任何依赖性的漏洞。

Content Scanner

A content scanner looks for language-specific files and directories that should not exist within source control.

内容扫描器

内容扫描器寻找不应该存在于源代码控制中的特定语言的文件和目录。

Static Code Analysis

A Static Code Analysis tool measures code quality via unit-test coverage and reports information about code quality to CI tools.

静态代码分析

静态代码分析工具通过单元测试覆盖率度量代码质量，并向CI工具报告代码质量信息。

7.8 Deployments 部署

The ability for each team to autonomously deploy components on demand has been a game-changer in many high-performing organizations like Spotify [34]. Deployment rates were increased from the end of every Sprint to an on-demand Continuous Integration/Continuous Deployment pipeline (CI/CD). Dependencies on other release teams were eliminated to allow for on-demand independent releases by each team. Deployments could occur at any time due to downtime being eliminated through the use of Blue- Green Deployments [35].

每个团队按需自主部署组件的能力已经成了许多高绩效组织（比如Spotify）的制胜法宝 [34]。部署频率从每个Sprint结束时进行一次提高到利用CI/CD流水线按需持续集成和持续部署。客户营销部通过努力消除了发布团队间的依赖性，允许每个团队按需独立发布。由于通过使用蓝绿部署消除了停机时间，部署可以在任何时候发生[35]。

Previously, deployments happened at a maximum of once per week, with a typical cadence of once every two weeks. This was because code deployments involved an external team that manually deployed code to production environments, and this could only be done during late evening hours because of downtime that would occur. The deployments took multiple hours of planning and coordination. The actual code deployment would last 2-3 hours, frequently resulting in needing to roll back the deployment if changes caused unanticipated errors in production.

以前，部署工作最多每周一次，典型的节奏是每两周一次。这是因为代码部署涉及到一个外部团队，他们手动将代码部署到生产环境中，而这只能在晚间进行，因为可能会出现宕机的情况。每次部署都需要几个小时的计划和协调。实际的代码部署将持续2-3个小时，如果变更导致生产环境中出现意想不到的错误，需要频繁的回滚部署。

7.9 Infrastructure 基础设施

The current infrastructure dependencies caused bottlenecks and increased cycle times. All Scrum Teams were dependent on an external infrastructure team that would manage on-premises servers. The servers were a black box to the Software Engineers because only a Systems Engineer on the infrastructure team was allowed to access it directly. The Systems Engineers did not sufficiently understand the production code being deployed to these servers because that was the responsibility of the Software Engineer. This often led to hours spent debugging environment issues and constantly having to work towards keeping a consistent state across the testing, staging, and production environments. Software Engineers would develop locally but could not test their code until they promoted it to the test environment, which would lead to situations where code could not be deployed because it was waiting for a test server to become available. Because the test and staging environments were managed separately, there were often differences between the environments, causing additional issues in testing changes before releasing to production.

目前的基础设施依赖性造成了瓶颈并增加了周期时间。所有的Scrum团队都依赖于一个外部的基础设施团队来管理内部的服务器。这些服务器对软件工程师来说是一个黑匣子，因为只有基础设施团队的系统工程师被允许直接访问它。系统工程师并不充分了解部署在这些服务器上的生产代码，因为那是软件工程师的责任。这往往会导致花费数小时来调试环境问题，并不断努力保持测试、预发环境和生产环境的状态一致。软件工程师会在本地开发，但在提交到测试环境之前无法测试他们的代码，这将导致代码无法部署，因为它在等待可用的测试服务器。因为测试环境和预发环境是分开管理的，环境之间经常有差异，导致在发布到生产之前，当测试变化时会引起额外的问题。

To eliminate these dependencies, infrastructure was containerized, and a Docker environment was implemented, including automated migration and setup/termination as needed using terraform scripting. Additionally, a blue/green deployment model was implemented to test before moving traffic. This allowed each Scrum Team to work independently, free from being dependent on the external infrastructure team.

为了消除这些依赖性，基础设施被容器化，并实施了Docker环境，包括自动迁移、设置和终止，根据需要使用terraform 脚本。此外，采用蓝/绿部署模型，在用户使用系统产生流量之前进行测试。这使得每个Scrum团队可以独立工作，不需要依赖外部的基础设施团队。

7.10 OKRs OKR

Objectives and Key Results [36] were established for all Release Trains. The specific relevant Key Results were:

New-New Code Coverage

• • • A target coverage rate for automated testing of all new lines of code for either legacy or new applications was established.

Overall Code Coverage

• • • A target coverage rate        for automated testing overall (new and legacy code) was established.

Mean Time To Recovery (MTTR)

• • • The failure of any individual gate within the CI pipeline must be recovered within 24 hours.

所有的发布火车都建立了OKR[36]。具体的相关关键结果(KR)是：

新代码覆盖率

• 无论是遗留的的还是新的应用程序，对所有新的代码行确立了自动化测试的目标覆盖率。

总体代码覆盖率

• 对全部新代码和遗留代码，确立了自动化测试的总体覆盖率目标。

平均恢复时间（MTTR）

• 在CI流水线中的任何单个故障必须在24小时内恢复。

8 Results 结论

As one of the largest Scaled Agile implementations, Rocket Mortgage was more successful than most large implementations, reducing cycle time by 46.5%. The goal of implementing Scaled Scrum on top of Scaled Agile was further reduction in cycle time, higher quality, and improved customer and team satisfaction.

作为最大的SAFe实施案例之一，Rocket Mortgage比大多数规模化敏捷实施更成功，交付周期减少了46.5%。在 SAFe的基础上实施Scrum@Scale的目标是进一步缩短交付周期，提高质量，并提升客户和团队的满意度。

8.1 Cycle Time Reduction（缩短交付周期）

By leveraging the Scaled Scrum practices described above, in addition to the organizational Scaled Agile rollout, Client Marketing was able further to improve feature delivery cycle times, feature throughput, and planned work (commitment) completion rates. Table 1 shows that Client Marketing reduced the average feature cycle time by 75%, from 86 days to 21 days. The first year saw a 51% reduction, from 86 days to 42 days, with another 50% reduction to 21 days the second year. The result was that the implementation of Scaled Scrum brought down cycle time another 55.12% over Scaled Agile alone. Additionally, feature predictability improved by a factor of 3x with a cycle time standard deviation reduction from 46 days to 14 days. When normalized for 10-week releases, feature throughput also increased by 340% for the train. Commitment completion increased by 91% for Client Marketing, bringing it on par with the rest of the organization.

通过利用上述的Scrum@Scale实践，在原来组织层面SAFe推广的基础上，客户营销部能够进一步改善特性交付周期，特性吞吐量，以及承诺完成率。表1显示，客户营销部将平均特性周期时间减少了75%，从86天减少到21天。第一年减少了51%，从86天减少到42天，第二年又减少了50%，达到21天。结果表明，实施Scrum@Scale后，交付周期比单独实施SAFe又减少了55.12%。此外，需求的可预测性提高了3倍，交付周期的标准差从46天减少到14天。如果将10周的发布规范化，特性吞吐量增加了340%。客户营销部的承诺完成率提高了91%，使其与组织的其他部门持平。

Table 1. Pre- and Post-Scaled Scrum and Agile Metrics

表1. Scrum@Scale和SAFe执行前、后的和度量指标

1. Feature Cycle Times for Client Marketing were 83 days and11.6 days

2. Feature throughput for Client Marketing was 5

3. Commitment completion for Client Marketing was 46%

1. 客户营销部的交付周期为83天和11.6天

2. 客户营销部的特性吞吐量为5

3. 客户营销部的承诺完成率为46%。

8.2 Improved Administration Efficiency 提升管理效能

Through restructuring roles and modifying its approach to software quality, deployment operations, and product ownership, Client Marketing saw a change in its ratio of non-delivery roles (team leaders, business analysts, quality assurance) to production-focused roles (engineers and developers). The overall change went from ~2:1 in Q3 2017 to ~1:2 by Q3 2019, indicating an increased focus on delivering working software rather than managerial overhead. Figure 3 shows the correlation between cycle time reduction (top chart blue line) and the decrease of admin roles (bottom chart blue line). Section one reveals a negative trend where cycle times and non-delivery roles were both increasing. Section two represents the period of change from admin roles. Section three indicates the onboarding of additional teams and adding PO roles (considered non- delivery roles for these graphs, thus the rise in the bottom chart).

Figure 3. Rocket Mortgage Role Restructuring

图3 Rocket Mortgage 角色重组

通过重组角色和修改其对软件质量、部署操作和产品所有权的方法，客户营销部的非交付角色（团队领导、业务分析师、质量保证）与交付角色（工程师和开发人员）的比例发生了变化：总的来说从2017年第三季度Q3的大约2:1到2019年第三季度Q3的大约1:2. 这表明组织越来越注重交付可工作软件，而不是管理上的开销。图3显示了交付周期缩短（上图蓝线）和管理角色减少（下图蓝线）之间的相关性。第一部分显示了一个消极的趋势，周期时间和非交付角色都在增加。第二部分代表了管理角色的变化时期。第三部分显示了因为有更多团队的加入增加了PO角色（在这些图表中被视为非交付角色，因此底部图表中是上升的）。

8.3 Delivery Focus 聚焦交付

The Scrum Reset brought a new level of clarity to the SM and PO. The MetaScrum and SDS provided frequent cycles in which POs and SMs could receive real-time feedback on their performance, take corrective action, and compare results before the next increment. Through this increased visibility and clarity, SMs and POs typically either saw a dramatic improvement in their effectiveness or were able to determine that the role did not fit their strengths objectively. This led to Client Marketing having the right people in key leadership roles, resulting in a high-performance culture focused on delivering value. The result of having effective SMs and POs, along with delivery teams made up of team members with multiple skills, was a team structure and process that drove shorter cycle times for delivering value to clients.

Scrum重置给SM和PO带来了新的清晰度。MetaScrum和SDS提供了频繁的循环，PO和SM可以收到关于他们表现的实时反馈，采取纠正措施，并在下一次增量之前比较结果。通过这种增加的可视化和清晰度，SM和PO通常要么看到他们的效率有了极大的提高，要么能够客观地确定这个角色不符合他们的强项。这使得客户营销部在关键的领导岗位上有了合适的人选，从而形成了一种专注于交付价值的高绩效文化。拥有高水平的SM和PO，同时配备一个具有多种技能的执行团队，是最好的团队组合方式，它能以最短时间向客户提供有价值的服务。

8.4 OKRs OKR

The tracking of OKRs became a bellwether for Rocket Mortgage technical teams to be proactive in their work. Client Marketing’s code coverage data is presented in Table 2. Other OKRs either had limited or no pre-Agile data, or contained proprietary information.

对OKR的跟踪成为Rocket Mortgage技术团队积极主动工作的风向标。客户营销部的代码覆盖率数据如表2所示。其他的OKR要么是有限的或没有敏捷前的数据，要么包含专有信息。

Table 2. Client Marketing overall code coverage

表2 客户营销部的整体代码覆盖率

8.5 Deployments 部署

Deployment rates were improved through a careful process. After having all prerequisites in place, releases could be done on-demand using a CI/CD gated pipeline as previously mentioned instead of the previous once per Sprint limitation.

通过一个谨慎的过程，部署频率得到了提高。在具备所有的先决条件后，可以使用之前提到的CI/CD流水线按需发布，而不局限于每个Sprint发布一次。

8.6 Release Planning 发布计划

In addition to faster deployments, release planning frequency was also increased. A survey by Puppet and DORA showed that organizations with high-performing DevOps enjoy 22% less time spent on unplanned work, 3x lower change failure rates, and enhanced employee engagement [37]. Client Marketing incrementally increased their original release planning cadence of once per quarter to five times per year, then once per month, and then eliminated it with the implementation of continuous planning.

除了更快的部署之外，发布计划的频率也得到了提高。Puppet和DORA的一项调查显示，拥有高绩效的DevOps的组织在计划外工作上花费的时间减少了22%，变更失败率降低了3倍，员工的敬业度也有所提高[37]。客户营销部逐步将他们原来每季度一次的发布计划节奏增加到每年五次，然后是每月一次，然后随着持续计划的实施将发布计划取消。

10. References 参考文献

1. C. Handscomb, D. Mahadevan, L. Schor, M. Siebere,E. Naidoo, and S. Srinivasan. (2020, 21 May). An Operating Model for the Next Normal: Lessons from Agile Organizations in the Crisis. Available: https://www.mckinsey.com/business- functions/organization/our-insights/an-operating- model-for-the-next-normal-lessons-from-agile- organizations-in-the-crisis

2. M. Reeves, K. Whitaker, and T. Deegan, "Fighting the Gravity of Average Performance," MIT Sloan Management Review, 2020.

3. O. Mikhieieva and K. Stephan, "A Retrospective on Agile Transformations: Survey Results on Agility of German Organisations," in 2020 IEEE European Technology and Engineering Management Summit, 2020.

4. S. Wu. (2021, May 31). Cerno. Available: https://www.scaledagile.com/case_study/cerno/

5. R. Barnahor. (2021, May 31). SAFe Case Study: EdgeVerve Systems. Available: https://www.scaledagileframework.com/case-study- edgeverve-systems/

6. S. Jagadeesan. (2021, May 31). Case Study: Royal Philips. Available:https://www.scaledagileframework.com/royal-phillips- case-study/

7. L. Christopher and M. Vries, "Selecting a scaled Agile approach for a Fin Tech company," South African Journal of Industrial Engineering, vol. 31, pp. 196-208, 2020.

8. H. Takeuchi and I. Nonaka, "The New New Product Development Game," Harvard Business Review, 1986.

9. K. Schwaber and J. Sutherland, "The Scrum Guide - The Definitive Guide to Scrum: The Rules of the Game," Scrumguides.orgNovember 2020.

10. J. Sutherland and ScrumInc, "The Scrum At Scale® Guide: The Definitive Guide to the Scrum@Scale Framework Version 2.3," Scrum Inc., Cambridge, MA2021.

11. D. Rigby, J. Sutherland, and H. Takeuchi, "Embracing Agile," Harvard Business Review, May 2016.

12. J. Shook, "How to Change a Culture: Lessons From NUMMI," MIT Sloan Management Review, vol. 51, 2010.

13. D. K. I. Sobek and A. Smalley, Understaning A3 Thinking: A Critical Component of Toyota’s PDCA Management System. New York: Productivity Press, 2008.

14. J. Shook and Womack, Managing to Learn: Using the A3 Management Process to Solve Problems, Gain Agreement, Mentor and Lead. Cambridge MA: Lean Enterprises Institute Inc., 2008.

15. E. Elbourne, Fundamentals of Industrial Administration: An Introduction to Management. London: MacDonald & Evans, 1949.

16. E. Baker, The Symphony of Profound Knowledge. Bloomington IN: iUniverse, 2016.

17. T. Ohno, Toyota Production System: Beyond Large- Scale Production: Productivity Press, 1988.

18. M. Tromans, "DEPD-PKT Software Team Process Improvement (A3)," Toyota Kentucky2016.

19. A. Smalley, 4 Types of Problems. Cambridge MA: Lean Enterprise Institute, 2018.

20. ScaledAgile. (2021, June 1). SAFe DevOps Series. Available: https://www.scaledagileframework.com/devops/

21. G. Hermkes and L. Quintela, Scaling Done Right: How to Achieve Business Agility with Scrum@Scale and Make the Competition Irrelevant. Berlin: Behendigkeit Publishing, 2020.

22. J. O. Coplien and N. Harrison, Organizational patterns of agile software development. Upper Saddle River, NJ: Pearson Prentice Hall, 2005.

23. J. O. Coplien, "Borland Software Craftsmanship: A New Look at Process, Quality and Productivity," in 5th Annual Borland International Conference, Orlando, FL, 1994.

24. C. Verwijs and D. Russo, "A Theory of Scrum Team Effectiveness," IEEE Transactions on Software Engineering, May 26 2021.

25. S. Daukus, "Test and fix time inside vs outside a sprint.," Cambridge MA: Scrum Inc, 2016.

26. J. Humble and D. Farley, Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation: Addison Wesley, 2010.

27. ScaledAgile. (2021, June 3). Case Study: Telstra. Available: www.scaledagileframework.com/telstra- case-study/

28. F. Verbruggen, J. Sutherland, J. M. van der Werf, andS. Brinkkemper, "Process Efficiency – Adapting Flow to the Agile Improvement Effort," in 52nd Hawaii International Conference on System Sciences, Hawaii, 2019, pp. 6981-6987.

29. C. R. Jakobsen and J. Sutherland, "Scrum and CMMI Going from Good to Great," in Agile Conference, 2009. AGILE '09., 2009, pp. 333-337.

30. M. Venema. (2021, 29 August). 6 Scaled Agile Frameworks – Which One Is Right For You? Available: https://www.digite.com/blog/scaled-agile-frameworks/

31. J. Sutherland, "Future of Scrum: Parallel Pipelining of Sprints in Complex Projects," presented at the AGILE 2005 Conference, Denver, CO, 2005.

32. J. Sutherland and J. Coplien, The Scrum Book: The Spirit of the Game: Pragmatic Bookshelf, 2019.

33. J. Johnson, Chaos 2020: Beyond Infinity. Boston MA: Standish Group, 2020.

34. H. Kniberg. (2019, June 7). Spotify: A Scrum@Scale Case Study. Available: https://resources.scrumalliance.org/Article/spotify- scrum@scale-case-study

35. M. Fowler. (2010, 20 Sep). BlueGreenDeployment. Available: https://martinfowler.com/bliki/BlueGreenDeployment. html

36. J. Doerr, Measure What Matters: How Google, Bono, and the Gates Foundation Rock the World with OKRs: Portfolio, 2018.

37. Puppet and DORA, "State of Dev/Ops Report 2016," Puppet2016.

38. Red Hat. (2018, 29 August). What is CI/CD? Available: https://www.redhat.com/en/topics/devops/what-is-ci- cd

39. Cisco. (2021, 29 August). What is CI/CD? Available: https://www.cisco.com/c/en/us/solutions/data- center/data-center-networking/what-is-ci-cd.html

40. A. Crawford. (2019, 29 August). DevOps. Available: https://www.ibm.com/cloud/learn/devops-a-complete- guide

41. Amazon. (2021, 29 August). What is Dev/Ops? Available: https://aws.amazon.com/devops/what-is- devops/