On Wednesday 21 October at 15:43, the security update for the major PHP flaw came out. By 16:56, the Dutch broadcaster RTL and the tech site Tweakers were back online, according to the NOS live blog. How fast you are safely back online after an update like that depends mostly on the route an update has to travel before it reaches production.
Patch management is the fixed process by which you know what software you run, pick up updates, test them and roll them out. With a CI/CD pipeline, a critical security update is an ordinary change: you bump a version number, the pipeline builds and tests, and after approval the update is live. Going back to the previous version is then a matter of redeploying the previous build.
What patch management is, and what the PHP flaw showed
Patch management is more than pressing a button. You need to know which versions are running, where your updates come from, how you test that everything still works and how the update reaches production. Drop any one of those four and every update becomes a gamble, and people put gambles off.
Where your updates come from mattered more in the PHP case than many teams expected. According to the PHP project's advisory, Linux distributions and long-term support vendors received the patches before publication. Anyone pulling PHP packages from the volunteer-run repository rpms.remirepo.net was advised to switch to distribution packages or build PHP themselves, because that repository would not have updates that day. Two servers on the same PHP version could therefore be hours apart in how fast they were safe.
Our post on the October PHP vulnerability covers what to do in the first hours of a flaw like that. This post is about the route you need to do it.
Why manual patching stalls
Teams that patch slowly usually get stuck on the way their servers and releases are set up:
- Servers that were once configured by hand drift apart over the years. Nobody knows exactly what is on them, so every update feels like a risk.
- Without automated tests, every update means someone clicking through the application by hand. That takes a day, so updates get saved up.
- Operations updates the server, and the developers only hear about it when something breaks. A PHP update that breaks the application gets rolled back and then forgotten.
The result is predictable: small updates pile up, and when a critical flaw arrives you have to catch up three versions at once under time pressure.
Patch management with CI/CD: how it works
With CI/CD, an update becomes a change like any other, through the same route:
- Versions live in code. The PHP version sits in the Dockerfile or in server configuration under version control, the package versions in
composer.lock. An update is a one-line change, for example fromphp:8.4.26-apachetophp:8.4.27-apache. - A bot reports new versions. Renovate or Dependabot automatically prepares a proposal as soon as a new version of an image or package is out.
- The pipeline builds and tests. Every change goes through the same steps: build, unit and integration tests, security scans and a rollout to a test environment.
- Rollout happens in stages. First a share of the servers or users, then the rest. If errors go up, the rollout stops by itself.
- You can always go back. The previous image is ready, so rolling back means redeploying the previous version.
In DevOps and CI/CD we build exactly that route: a pipeline that takes a change from commit to production, with building, testing, quality gates and rollout without manual steps. We capture the servers themselves as code, with Terraform for provisioning and Ansible for configuration, so every environment can be rebuilt. A server nobody dares to touch then simply does not exist.
Agreements to make in advance
A pipeline solves the technical part. Wednesday showed that decisions have to be made as well, and you would rather make them on a quiet day:
- A deadline per severity. For example: a critical flaw that can be exploited from outside goes live the same day; everything else goes out with the next regular release.
- Who decides on going offline. Someone has to be able to weigh a site going dark against the risk of a breach, also outside office hours.
- Who updates what. Agree with your supplier and hosting provider who updates the operating system, PHP, the application and the plugins, and how fast. What else to agree before a project starts is in our post on vendor lock-in with custom software.
- What your customers hear. A short notice on the site or in the portal prevents phone calls and rumours when you have to go offline for a while.
Where to start
You don't have to rebuild everything at once. Start with the application that carries the most risk: the one with a public login or with personal data. Put its versions in code, build a pipeline around it and make sure you can see whether a rollout went well. We set up monitoring so that logging, metrics and alerts show within seconds when error rates or performance change, with thresholds we agree with the operations team.
Then rehearse the route with an update that is not urgent. The first time a pipeline rolls out a PHP update should not be during a crisis. Once it works, move on to the next application.
Frequently asked questions
What is the difference between patch management and installing updates?
Installing updates is a single action. Patch management is the process around it: knowing what you run, noticing new versions, testing, rolling out and recording what happened. Without that process, installing depends on whoever happens to think of it that day.
Should updates go to production automatically?
Not necessarily. Building and testing run automatically, and the step to production can require a human approval. For small package updates with good tests, many teams do choose fully automatic.
Does this also work for WordPress or an off-the-shelf package?
Yes. A WordPress site can also run in a container with its PHP version in code, and plugin updates can go through a test environment first. For a package the supplier hosts, it is mostly about agreements on who updates what and when.
How fast does a critical update need to be live?
That depends on the flaw and on what runs on the system. A flaw that can be exploited from outside without a login, like the PHP one, is one you want closed the same day. Agree that deadline in advance, so nobody has to debate it in the moment.
Want to know how fast your applications can receive a critical update, or where the route gets stuck today? Get in touch for a short call. You will hear back within 24 hours.
