When a company tests resilience it doesn't have: how a million euros a year vanishes into planned outages

The article criticizes the practice of a Czech technology company that shuts down its data center every two weeks for a resilience test, even though its internal systems (Git, CI/CD, wiki) are not redundant and crash during every test. The author shows that this costs the company over a million euros a year in lost productivity from 600 developers, and compares this approach with the chaos engineering principles of Netflix, Google, and Amazon – which first built redundant infrastructure and only then tested it.

Každé dva týdny na hodinu spadne Git → Every two weeks, for one hour, Git goes down, along with the build pipeline and the internal wiki. Management calls it "resilience testing." Developers call it something else.

On Thursday afternoon, Petr sets his coffee down next to the keyboard and opens a merge request he has been working on for two days. The code is finished, the tests passed locally, all that remains is the merge and deployment. He clicks the button — and gets a timeout. GitLab is unresponsive. Slack reports an outage. Jenkins is dead. Build artifacts unavailable. A moment later a laconic message appears on the noticeboard: a scheduled data center resilience test.

Petr closes his laptop. In an hour he will come back, restart the build chain, re-verify the state of the code repository, and rebuild his context. But an hour of unplanned downtime does not mean an hour of loss — the context switch, the sync-up with colleagues, and getting back into focused work cost him another forty minutes. Multiplied by the six hundred developers at the company.

Management does this in good faith. Every two weeks it shuts down one data center for an hour to verify that customer-facing services — search, email, news — survive the outage. And they do survive. The customer infrastructure is prepared for it, redundant, tested. But the internal systems — Git, continuous integration and deployment, the wiki, communication tools, the build farm — are not redundant. With every test, they simply go down. And six hundred people wait.

To the company it looks like a minor inconvenience in the name of a greater good. In reality it is the inversion of one of the most important engineering disciplines of the past fifteen years. And that inversion costs the company over a million euros a year.

The story of resilience testing through controlled failure begins around 2010 in Los Gatos, California, where Netflix, in the middle of migrating from its own data centers to Amazon's cloud, is grappling with a fundamental engineering problem. The migration, launched after a major outage in 2008, will take a total of seven years — it will not be completed until January 2016. But already in its first phase the engineers realize an uncomfortable truth: in the cloud you cannot rely on the infrastructure to keep working. Instances die, zones go down, the network drops packets. It cannot be predicted — but it can be prepared for.

Greg Orzell and his team build a tool they name Chaos Monkey. During working hours it randomly terminates production instances. Not to do harm, but to force engineers to design services so they survive the failure of individual components. A key detail that is often overlooked: Netflix first built a redundant architecture on Amazon's cloud — and only then began testing it with Chaos Monkey. The monkey was not terminating instances in a system without a safety net. It was terminating them in a system that had a safety net, to verify that the net held.

Out of Chaos Monkey grew an entire toolset called the Simian Army — Latency Monkey simulated network slowdowns, Chaos Gorilla took down whole availability zones, Chaos Kong simulated the failure of an entire region. But Netflix quickly discovered that even this toolset had a fundamental problem: some of the tools affected all services indiscriminately, causing unintended impacts on customers.

So in October 2014 Kolton Andrus and his team created FIT — Failure Injection Testing. A fundamentally different approach: instead of blanket chaos, precisely targeted injection of failure into a specific service, with gradual expansion. First a single test account, then one percent of traffic, and only after verification the full production. Not random destruction, but a controlled experiment with a measurable outcome.

Netflix built a formal resilience-testing team around 2015 under the leadership of Casey Rosenthal — five years after Chaos Monkey. Developing the discipline took years. And it began with redundancy, not with testing.

In 2017 Rosenthal and colleagues published the principles of resilience testing through controlled failure (the manifesto was further updated in 2019 at principlesofchaos.org). It defines the discipline through five principles. They are worth going through one by one — because most organizations that claim to "do chaos engineering" violate at least three of them.

Principle one: define the steady state. Before you start breaking the system, you have to know what its normal behavior looks like. That means metrics — latency, error rate, throughput, availability. Without measuring the steady state, you cannot know whether the experiment revealed anything. If a company has no service-level objectives for its internal systems, it has nothing to test.

Principle two: formulate a hypothesis. A resilience experiment is not "we'll turn it off and see what happens." It is a scientific experiment: "Hypothesis: during a failure of data center A, GitLab will remain available through data center B with latency under 200 ms." If you know in advance that GitLab will go down because it has no replica, you are not running an experiment — you are running a planned outage.

Principle three: simulate real-world events. Simulate genuine types of failure — network connectivity breakdown, CPU saturation, full disks, slow domain name resolution. A complete data center shutdown is a legitimate scenario, but it belongs at the top of the escalation pyramid, not at its base.

Principle four: run experiments in production. This is what companies like to hear. But the manifesto also says —

Principle five: minimize the blast radius. And this they are less keen to hear. It is the engineer's responsibility to ensure that the impact of experiments is contained. Start with one container, move on to one service, then one zone. The whole data center is the final level, not the starting point.

Google has run a program called DiRT — Disaster Recovery Testing — since roughly 2006. It was led by Kripa Krishnan and the program became one of the pillars of Google's site reliability engineering culture. DiRT tests include shutting down entire data center buildings — but with a discipline a Mars mission would envy.

Every DiRT test requires a formal test plan approved by management. There must be a rollback plan — exactly what we will do if the test reveals a critical problem. Real production incidents always take precedence; if a genuine outage arises during a test, the test is stopped immediately. The whole program has a control center that documents how it unfolds.

During one DiRT test, Google discovered that the alerting infrastructure — the system that sends warning notifications to engineers — was located in the very center they had just shut down. The alerts queued up and no one received them. Exactly the kind of lesson that resilience testing exists for. But a lesson that emerged from a controlled experiment with a prepared rollback to the original state, not from blanket destruction.

Amazon has a concept called the "game day" — GameDay — which Jesse Robbins created in 2004. His job title carried the nickname "Master of Disaster." A game day lasts two to four hours, begins with a health check of the environment, has clearly defined abort criteria, and ends with a blameless post-mortem. Werner Vogels, Amazon's CTO, summed it up in a sentence that became a mantra for the entire industry: "Everything fails all the time." But Amazon drew from this not "so let's switch it off and see," but rather "so let's design it to survive failure — and then verify that it survived."

The common pattern across all three companies is unambiguous: first build resilience, then test it. Netflix did not unleash Chaos Monkey on systems without a fallback. Google only tests systems with rollback plans. Amazon begins a game day with a health check of the environment.

Let us return to Petr and his company. What exactly happens when management shuts down a data center every two weeks?

Customer-facing services survive — they are redundant, distributed, designed for this scenario. The test confirms what everyone already knows. The internal tools go down — because they are not redundant. The test confirms what, again, everyone already knows. No one learns anything new. Six hundred people lose an hour of productive work plus forty minutes to rebuild context. And in two weeks, again.

Professor Gloria Mark of the University of California, Irvine, found in her research that returning to an interrupted task takes an average of 23 minutes — between the interruption and the return, a person typically passes through two other tasks that derail them further. With six hundred developers, every test means roughly 230 person-hours just to regain context — not counting the hour of downtime itself.

Let's do the math. The fully loaded hourly rate of a developer at a Czech technology company is around 40 euros (including payroll taxes, overhead, and tooling). With two tests a month and a two-hour effective impact (an hour of downtime plus rebuilding context), the company loses roughly 24,000 euros for a single hour of downtime in direct productivity, plus another 9,200 euros on context switching. Annually that comes to roughly 1.0 to 1.2 million euros.

A 2024 study by Splunk confirms the context: Global 2000 companies lose 400 billion dollars a year due to unplanned downtime. Stagnating developer productivity is among the most significant hidden costs.

Petr's employer is spending over a million euros a year to test a hypothesis whose outcome it knows in advance.

The solution is not to stop testing resilience — it is to stop testing non-existent resilience. The path from the current state to a mature resilience-testing program has four phases, and on container infrastructure it is technically well charted.

Phase zero — the safety net (weeks, minimal cost). Automated backups of critical services to the second data center. Mirroring of key code repositories via scheduled jobs. Reducing the time-to-live of domain name records to 60 seconds for faster failover. Runbooks for manually switching to the backup system. An internal status page. This costs two engineers for two weeks and dramatically reduces the impact of outages — during a data center failure there is at least read access to the repositories and the documentation.

Phase one — basic redundancy (months). Continuous PostgreSQL replication to the second data center. Object storage replication for build artifacts and attachments. GitLab Geo for replicating code repositories and build artifacts — the secondary node serves as a warm standby with read access and the ability to be promoted to primary during an outage. Infrastructure as code for all internal tools, so they can be restored in any data center.

Phase two — automatic failover (quarters). Build machines in both data centers. Communication tools in a highly available configuration — Mattermost with database replication, or Matrix/Synapse, which is architecturally decentralized and naturally continues in the second data center when one fails. The wiki as a stateless service with a read replica of the database. Domain-name-resolution-based switching using k8gb or ExternalDNS. At this stage, most internal services survive a data center failure with minimal impact.

Phase three — genuine resilience testing through controlled failure. Only now, with redundant infrastructure in place, does it make sense to deploy tools like Litmus Chaos or Chaos Mesh and begin formulating hypotheses. Gradual escalation: terminating a single container, a service failure, a zone outage, a full data center outage. Every experiment with a defined steady state, measurable metrics, and abort criteria. Quarterly game days inspired by Google's DiRT program — with a formal plan, a control center, and a blameless post-mortem.

The total implementation cost runs around 800,000 euros in the first year (infrastructure, licenses, engineering time), and from the second year roughly 470,000 euros a year to operate. Against annual productivity losses of over a million euros, the investment pays for itself within eighteen months and from the second year delivers a threefold return. And that does not count the hard-to-quantify but real factor: developer retention. A developer whose work environment goes down twice a month goes looking for a different work environment.

It would be convenient to tell the story in black and white: management does resilience testing wrong, so let it do it right. Reality has nuance.

First, the very decision to test the resilience of customer-facing services by shutting down a data center is legitimate and progressive. Most Czech technology companies do not even do that. A culture that treats planned outages as acceptable is healthier than a culture that pretends outages do not happen. The problem is not that testing takes place, but what is sacrificed during the test.

Second, full redundancy of internal systems is not trivial. Multi-data-center deployment on a container platform requires solving the network layer (Cilium Cluster Mesh, Submariner), the data layer (PostgreSQL replication, distributed databases), and the orchestration layer (ArgoCD, GitOps). The KubeFed project (Kubernetes Federation v2) has been archived since April 2023 — there is no "one-click" solution for multi-cluster setups. Implementation requires skilled site reliability engineers, who are in short supply on the Czech market.

Third, there is an argument that internal-system outages are "useful pressure" on developers to design their own tools more resiliently. This argument falls apart the moment developers have no control whatsoever over the architecture of the internal tools — they cannot make GitLab redundant themselves; that is a job for the platform team.

Even after accounting for these objections, the key point remains valid: testing a system that you know will fail is not an experiment — it is a planned outage with a known outcome. Resilience testing through controlled failure requires a hypothesis that can be refuted. If it never can be, we learn nothing new.

It is Thursday afternoon, two weeks after the last test. Petr opens a merge request, the code is finished, the tests passed. He knows that today is test day. He waits. A moment later a message appears on the noticeboard.

But this time it is different. GitLab fails over to the secondary node, read access is running within fifteen seconds. Build machines in the second data center take over the chain. Slack switches to a backup instance. Petr postpones the merge request by two minutes — not by two hours.

Management looks at the dashboard: the failover completed in 47 seconds, three services exceeded their latency target by 12 %, one internal service returned two error responses before it failed over. This is data. This is an experiment with a refutable hypothesis. These are findings worth fixing.

Netflix did not unleash Chaos Monkey on systems without a safety net. Google does not take down data centers without a rollback plan. Amazon does not begin a game day without a health check of the environment. All three first built resilience — and only then began breaking it.

Petr, meanwhile, merges his code. The build passes on the first try.

The author works as an infrastructure systems developer at a Czech technology company. The article is based on personal experience and publicly available sources on resilience testing through controlled failure.

Sources:

Transparency of authorship

The concept, structure, and editorial line of the article are the work of the author, who produced the content outline, established the key theses, and directed the entire creation process. Generative AI (Claude Opus 4.6, Anthropic) was used as a tool for research, fact-checking, and fleshing out the author's draft.

The author verified the key findings and approved the final wording. No part of the text was published without conscious authorial control. Factual data were verified against the publicly available sources cited in the text.

The procedure complies with the transparency principles of EU Regulation 2024/1689 (the AI Act). #poweredByAI