Nobody Is Patching MySQL 5.6

A database version reaching end of life is unlike any other deadline in operations, because nothing observable changes on the day. There is no error, no degradation, no alert. The software you are running on the eleventh of February is bit-for-bit the software you are running on the twelfth, and it will keep doing its job for years if you let it. What ended was not the software. What ended was the arrangement under which somebody else fixes it.

Oracle's lifetime support policy puts MySQL 5.6 at general availability in February 2013, Premier Support ending five years later in February 2018, and Extended Support ending this February. After that comes Sustaining Support, which is indefinite and which is worth reading carefully, because indefinite sounds generous and means something narrow: you may keep calling, and you get the fixes that already exist. Not new ones. A vulnerability discovered in 5.6 next month has no patch coming for 5.6, and that is not a lapse, it is the published terms.

So the exposure is not that the version stops working. It is that the version stops being maintained while continuing to work, which is a much worse shape, because the thing that would prompt action is exactly what has been removed. A server that broke would get fixed. A server that is fine and quietly uninsurable gets a ticket in a backlog, and it stays there until somebody outside the organization sets a date.

Which somebody usually does. If the database is running on a managed service, the provider is on the hook for patching an engine that no longer receives patches, and no provider carries that indefinitely. They publish their own end-of-support date for the major version, some months after the upstream one, and when it arrives instances still on that version get upgraded on the provider's schedule rather than yours, in a maintenance window, to a version the provider chooses. That is the deadline that actually governs, and it is the one nobody has written on a plan.

The vendor clock is the one everybody quotes. Premier, then Extended, then Sustaining, on a published schedule years in advance. This is the date that appears in the compliance spreadsheet and it is genuinely useful, because it is announced long enough ahead to plan against. It is also the least urgent of the five, because Sustaining Support means the phone still works and an organization can talk itself into treating that as coverage for another two budget cycles.

The platform clock is the one that moves you. A managed provider set its own launch date for the version - Amazon made MySQL 5.6 available on RDS in July 2013 - and it sets its own retirement date too. That date comes with an action attached rather than a status change: instances that have not moved get moved. Find the provider's date, not the vendor's, and treat the difference between the two as the entire planning window you actually have.

The client clock is the one nobody owns. Connectors, ORMs and drivers drop support for old servers on their own schedules, and they do it quietly, in a minor release, in a changelog. The bind is that the application's dependency updates and the database's version become coupled without anyone deciding they should be: a security patch to a driver arrives requiring a newer server, and now a database upgrade is on the critical path of an unrelated fix.

The audit clock is arbitrary and binding anyway. An unsupported database component is a finding in most frameworks, and a finding has a remediation date attached that bears no relationship to engineering capacity. This clock is the one people find infuriating and it is the one I have seen actually cause upgrades to happen, because it is the only one attached to a person whose job is to insist.

The mechanical upgrade is close to a solved problem and has been for years. On a managed service it is a console action, and Amazon shipped point-and-click major version upgrades from 5.6 to 5.7 back in April 2016. Self-managed, it is a package upgrade and a data dictionary rebuild. Either way the work is measured in a maintenance window, it is well documented, and thousands of people have done it. If the mechanics were the cost, this would be a scheduling problem and it would already be done.

The cost is that the new version does not behave like the old one. Not in the sense of bugs, but in the sense of deliberate, documented, correct changes to what the server does with input it used to accept and with queries it used to plan a particular way. Your application was written against the behavior, not against the manual, and the parts of the behavior it depends on were never enumerated because there was never a reason to enumerate them.

This is the estimating error that runs through every one of these projects. The plan is written against the mechanics because the mechanics are visible, documented and easy to put a number on. The actual work is a search problem over your own code and data, whose size is unknown until the search is under way, and which has no natural end other than running out of things to find. A project that takes an afternoon of engine work and six weeks of finding out what broke gets estimated at an afternoon, every time.

Which reframes what an upgrade project is. It is not an infrastructure change with an application impact. It is an application change that happens to be triggered by an infrastructure date, and the people who need to be assigned to it are the ones who know what the queries mean rather than the ones who know how to run the upgrade.

The character set was never what its name said. MySQL's utf8 is a three-byte encoding covering part of Unicode, and the manual now calls it a deprecated alias for utf8mb3. Full Unicode is utf8mb4, and 8.0 makes it the default. The conversion is not a settings change: it rewrites every affected table, it can move rows across the wire between application and server differently, and any place a length was measured in bytes rather than characters now measures something else.

And there is the index prefix limit that ambushes the conversion. InnoDB caps an index key prefix at 767 bytes under the older row formats and 3,072 bytes under the newer ones. A VARCHAR(255) indexed column under three-byte utf8 needs 765 bytes and fits. The same column under utf8mb4 needs 1,020 and does not. So a conversion that looks like a character set change turns into a row format change on some subset of tables, discovered one failed ALTER at a time, usually on the largest table you own.

Strict mode stops accepting what it used to swallow. From 5.7 the default sql_mode includes strict transactional tables and ONLY_FULL_GROUP_BY, so writes that used to be silently truncated or coerced now fail, and a GROUP BY selecting a column that is not grouped or aggregated is now an error rather than an arbitrary row. The second one is the expensive one. It is a legitimate defect the old server hid, it is scattered through reporting code written years apart, and each instance needs somebody to decide what the query was supposed to mean.

Optimizer plan drift produces no errors at all. Newer versions merge derived tables that used to be materialized, cost differently, and in 8.0 add histograms and invisible indexes. Every one of those is an improvement on average and a regression somewhere specific, and the failure mode is the nastiest in the set: correct results, arriving slower, on a query nobody changed. There is no exception to catch and no log line to grep. It shows up as load, days after the upgrade, when the causal link is no longer obvious.

Replication format and topology constrain the order. The default binary log format became row-based in 5.7, and replication is only supported from an older source to a newer replica, never the reverse. Those two facts together dictate the sequence of a rolling upgrade and remove most of the clever ordering people propose. They also mean the rollback plan cannot be "fail back to the old replica," because once the new version has written to it, there is no supported path home.

Authentication and reserved words are the small ones that block cutover. MySQL 8.0 changes the default authentication plugin, so older connectors that cannot negotiate the new one fail at connect time rather than at query time. It also adds reserved words, which turns an unquoted column named for a windowing function into a syntax error. Neither is hard. Both are the kind of thing found at two in the morning on cutover night if nobody ran the application against the new version first.

The path is fixed by the tooling: 5.6 to 5.7, then 5.7 to 8.0, one major version at a time, with no supported jump straight across. So the first decision is whether to stop at the waystation. Stopping is genuinely cheaper this year. The 5.7 move is smaller, the behavior changes are the sql_mode and character set ones rather than the whole set, the connectors already work, and a team under an audit date can be finished in a fraction of the time.

The argument against stopping is on the same page of the same support document. MySQL 5.7 left Premier Support in October 2020, seven months ago, so the destination is already on its Extended clock with a published end in October 2023. Landing there buys about two and a half years and guarantees a second project, run by whoever is here then, against a codebase that will have grown in the meantime. The behavior work does not go away either. It splits into two projects with the harder half deferred.

I would still take the two-step for the tests rather than the destination. Running production on 5.7 for a while is the only way to separate the sql_mode and character set problems from the optimizer problems, and debugging one class at a time is dramatically easier than debugging both while trying to work out which is which. What I would refuse is stopping there and calling it done, because the thing that makes 8.0 expensive is plan drift, and plan drift is not smaller for having waited.

One more constraint that shapes the whole plan: MySQL 8.0 has no downgrade. The data dictionary changes on the way up and there is no supported route back, which means the rollback is a restore from backup and the recovery objective for the upgrade is whatever your restore time actually is, measured on a full-sized dataset, not the number in the runbook. Anyone who has not timed that recently is planning around a figure they have not verified.

Capture real queries and replay them, not a test suite. The behavior differences live in the queries your application actually sends, including the ones generated by an ORM that nobody has read. A capture of production traffic replayed against the new version finds the strict-mode failures, the reserved word collisions and the plan regressions in one pass, and it finds them in proportion to how often they matter. A test suite finds the ones somebody thought to write a test for, which is a different and much smaller set.

Get the slow query log before you have anything to compare it to. Plan drift can only be detected against a baseline, and the baseline has to be captured on the old version, at representative load, before anything moves. This costs an afternoon in advance and is impossible to reconstruct afterward. Without it, the argument three weeks after cutover about whether a report was always this slow has no evidence in it and will be settled by whoever is most confident.

Time the restore, on real data, before you need the number. Since there is no downgrade, the rollback is a restore, and the restore duration is the actual blast radius of a failed upgrade. Measure it on a full-sized copy rather than a sample, because the relationship is not linear and the number people quote is usually from a test dataset. If the measured figure is unacceptable, that is not a reason to skip the test. It is the finding, and it changes the plan.

Put the provider date in the plan, in writing. Every one of these projects I have seen run late did so because the date being planned against was the vendor's rather than the platform's, and the platform's was months earlier. Write down which date is binding, which clock it comes from, and what happens automatically if it passes. A forced upgrade in somebody else's maintenance window, to a version somebody else chose, is a strictly worse version of the project you were already going to do.

I want to be careful not to make this sound like negligence. Staying on 5.6 was frequently the right call at the time, made by people who correctly judged that the upgrade had a large unknown cost and no user-visible benefit, and who had something in front of them with both. The failure is not the decision, which was defensible each year it was taken. The failure is that it was never converted into a dated plan, so a series of reasonable annual deferrals added up to a position nobody chose and nobody owns.

What stays with me about database end of life is that it is the rare deadline with no symptom attached. Almost everything else in operations announces itself: the disk fills, the certificate expires and the browser complains, the queue backs up and somebody notices. This one arrives as a paragraph in a support policy and changes nothing you can see, right up until the morning your provider upgrades you on its own schedule and you find out, in production, which of your queries were relying on the old behavior.