Moving 900+ GB of Prometheus Without a Cutover Weekend

The Prometheus I was responsible for had grown past 900 GB of time series data, and at that size it had every disease that comes with it. 

Queries timed out unless you narrowed them down to an hour. Cardinality kept growing, because teams added labels "for debugging" and then sometimes forgot to remove them. On top of that, we were shipping a copy of fractions of our metric data to an external SaaS vendor, just in case, and that bill grew every quarter. And the fleet under all of this ran latency-sensitive trading services, so monitoring itself was not allowed to eat an extra CPU core at a wrong moment.

We moved it all to VictoriaMetrics. I am not going to argue here about storage engines; there is plenty written about which one wins. What nobody warned me about is how much the order of the steps matters. The standard way to migrate is a cutover weekend: you stand up the new system, you pick a Saturday, you switch, and then you spend three weeks finding out what broke. I think this order is backward. What worked for us was to delete first, then dual-write for a long time, then rebuild the dashboards and compare them panel by panel, and move the alerts at the very end. And most of the alerts should not move at all, but about that below.

The Inventory Nobody Has

Before touching anything, I wanted to know what anyone actually reads out of those 900+ GB. It sounds like a trivial question, and it is not, because the data you have does not answer it directly. The answer lives in three places. The TSDB status endpoint knows which metrics are the biggest. The Grafana dashboards know what humans look at. And the rule files know what the machines query, because alerts and recording rules read plenty of metrics which no human ever opens.

The status endpoint reports on the in-memory head block, so strictly speaking it shows what is filling the disk going forward, and not the composition of the historical 900+ GB. As a proxy for the biggest offenders, it was good enough:

curl -s 'http://prometheus:9090/api/v1/status/tsdb?limit=15' \
  | jq '.data.seriesCountByMetricName'

For the second list, dashboards are JSON, so every expression can be pulled out with jq. I tokenized the expressions instead of parsing them, which means the output is polluted with PromQL function names and label names. The pollution disappears later, at the intersection step:

find dashboards/ -name '*.json' \
  -exec jq -r '.. | objects | .expr? // empty' {} + \
  | grep -oE '[a-zA-Z_:][a-zA-Z0-9_:]+' | sort -u > tokens_dashboards.txt

The same jq pass over the exported alerting and recording rules gave the third list, and it caught a family of recording rules that fed an SLO report that no dashboard mentioned anywhere. (There is also mimirtool analyze, which builds this inventory for you. I wanted the raw lists.) Then you intersect the TSDB metric names with the union of the other two lists, and what falls out at the bottom is the set which nobody reads. For us that set covered a large share of the 900+ GB, and most of it was high-cardinality out of habit: per-pod histograms for services whose one dashboard had a single aggregate panel.

Delete at the Source Before Anything Moves

The instinct is to migrate everything and clean up later. Later is a bad deal here. Dead series which you keep get dual-written for months; they get baked into the sizing of the new cluster, and they pollute every panel diff which you will run later in the project. All of that in service of data which nobody reads.

So the first change of the whole project landed on the old system, and not on the new one:

metric_relabel_configs:
  - source_labels: [__name__]
    regex: 'go_gc_duration_seconds.*|.*_bucket_experimental_.*'
    action: drop
  - source_labels: [__name__, pod]
    regex: 'legacy_worker_queue_depth;.+'
    action: drop

The obvious worry is that you will need a deleted metric later. That came up twice over the following weeks, and both times the fix was to remove one relabel rule and reload the config with a SIGHUP or a POST to /-/reload. Do not restart a 900 GB Prometheus for this; the WAL replay will leave you without monitoring for longer than you expect. And one caveat which you should agree on before deleting anything: the reversal only works going forward. The rule drops samples before ingestion, so the gap for the dropped period never backfills. In our case nobody ever missed those gaps.

The Dual-Write Months

With the estate slimmed down, the old Prometheus started writing into the new storage in parallel with its own:

remote_write:
  - url: http://vminsert.internal:8480/insert/0/prometheus/api/v1/write
    queue_config:
      max_samples_per_send: 10000
      capacity: 50000
      max_shards: 12

Those numbers were not the first ones we tried. With the defaults, the send queue was too shallow for our sample rate, and within the first hour the log was full of this:

ts=... caller=queue_manager.go level=warn component=remote
  msg="Remote storage falling behind" pending_samples=48231 shards=4

So max_samples_per_send went up, capacity followed at five times that, which is what the tuning guidance says, and shards were raised until the queue stopped falling behind at the evening peak. And because the trading fleet was the whole point of being careful, we measured what dual-writing costs the old server before trusting it. CPU rose by roughly a tenth. Memory rose by about what the queue capacity implies. Scrape latencies did not move. The abort criterion was written down in advance: if scrape or rule evaluation starts lagging, the remote_write block comes out that same day. It never triggered.

Then, deliberately, nothing else happened for roughly a quarter. The old system stayed the source of truth for every dashboard and every page, and the new one filled up in the background, until every window which people habitually query existed on both sides, from the last six hours out to the month-over-month capacity view. This is the whole point of dual-writing: at no moment did the project depend on one switch being flipped correctly on one Saturday.

The one incident which the phase produced was self-inflicted. A relabel change on the old side renamed a label on its way into the new storage only, and the affected series split in two. A panel diff caught it about a week later, and that is exactly the place where such things should get caught.

Dashboards Are the Test Suite

We rebuilt the dashboards against the new engine while the old ones kept running, and then compared them panel by panel over the same time ranges. Every discrepancy meant something, but not always that the new panel was wrong.

The classic one is increase(). Prometheus extrapolates it, so a counter which grew by exactly 300 over the window renders as 299.4 or 301.7, and everyone has learned to not see it anymore. MetricsQL does not extrapolate, so the same panel suddenly shows round integers:

increase(http_requests_total{service="order-gateway"}[1h])
# old panel: 299.4        new panel: 300

Some panels had been quietly misleading for years, and the diffs surfaced that. Other panels were wrong on my side: a rate window that did not match the scrape interval, or a sum that collapsed a label that the legend needed. The comparison caught both kinds. The work was tedious. It was also the cheapest reliability work of the whole project, because whatever you catch at the desk at this stage, nobody has to chase later in the middle of an incident.

Alerts Move Last, and Most Should Not Move

Alerting stayed on the old system until everything above was finished. A bad page wakes somebody up at night, so pages moved in the last phase, group by group, and every rule was re-run against the new engine and watched for a while before its old copy retired.

The stronger claim, and the one I get pushback on: a migration is the wrong moment to preserve your alert rules, because most of them do not deserve the trip. Ours had accumulated over years; they fired constantly, and they meant little. Copying them onto the new engine would carry the same noise across intact, just on faster storage. So the rules were rebuilt around a single test instead: does this page fire on something which a user or a dependent service can feel.

# Retired with the old system:
- alert: HighCPU
  expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
  for: 10m

# What replaced it:
- alert: OrderGatewayP99LatencyHigh
  expr: histogram_quantile(0.99,
          sum(rate(http_request_duration_seconds_bucket{service="order-gateway"}[5m])) by (le)
        ) > 0.4
  for: 5m

CPU sitting at 91 percent has never made anyone I know roll a deploy back. The rollbacks which I have actually watched were all triggered by latency which a client could feel, and that is what the replacement rule fires on. The rest — the CPU levels, the queue depths, the disk growth — went into dashboards and recording rules for capacity planning. And outside of a migration, deleting a single alert rule means an argument with whoever added it. The migration made bulk deletion politically possible for once, and roughly a hundred stale rules did not make the trip.

What Fell Out of It

After the source-side deletes, the estate which actually needed a new home was well under half of the original 900+ GB. The SaaS dependence went away, and the bill went away with it. Queries stopped timing out. And the pages dropped from dozens on a loud week to a handful, so the ones which did fire got taken seriously, and incident response got faster off the back of that. The migration ran for months under a live trading workload and produced no cutover incidents, because at every step the previous state was still there, and we could fall back to it.

If you are staring at your own 900 GB, the short version is this. Run the inventory and believe it, even when it embarrasses the dashboards which produced it. Delete at the source before you move a byte. Dual-write across the longest window that people actually query. And when the time comes to move the alerts, ask about every rule whether a user can feel the thing that it fires on, because a migration is the only moment when your organization will agree to delete alert rules in bulk. A cutover weekend takes two days, and this took months; that is true. But the weekend does not remove any of the work that is described above. It just moves the increase() mismatches and the stale rules into production, and there they surface one at a time, usually at night.