Blog

When the control plane is down and the data plane is up

2026-08-28

DigitalOcean's late-August API and control-panel incident left running Droplets up while creates and dashboard access failed. How to separate control-plane pain from host reachability without inventing calm green.

Your Droplets are still answering. Customer traffic looks fine. The dashboard will not load, and every create call returns 5xx. Is the vendor "down"? That question is too coarse. You need two labels: control plane and data plane.

A recent case: DigitalOcean, 24 to 25 August 2026

On 24 August 2026 at 17:14 UTC, DigitalOcean opened a critical incident for the Cloud Control Panel and public API. Their first update said users would see elevated 5xx errors for API requests, including creates, and that many people could not reach cloud.digitalocean.com. A mitigation landed at 19:14 UTC. The problem recurred at 22:27 UTC. In that update DigitalOcean stated the issue did not impact the data plane (running Droplets). Further mitigations followed overnight; the incident was marked resolved at 19:04 UTC on 25 August, with a promised postmortem on the status page.

That timeline is public on DigitalOcean's incident page. Independent day-status writeups, including isinternetup.com's August outage digest, framed the same split: management and API pain, existing workloads still running. We do not invent StackWitness probe outcomes for that past window here. StackWitness does not currently publish a DigitalOcean truth page; the lesson below still holds for any vendor where create/update paths fail while serving paths keep answering.

Why this shape fools people

Host reachability answers one question: did the service host answer an outside probe? A reachable Droplet IP, or a reachable product hostname, does not prove that the control API, the billing console, or "create a new resource" still works. Conversely, a broken control plane does not mean every customer workload is offline.

Teams that collapse both into one badge do expensive things:

• They page the wrong on-call (app owners instead of platform/automation owners).
• They tell customers "we are fully down" when serving still works, or "all systems operational" when nobody can scale or recover via API.
• They treat "homepage / droplet still pings" as proof that IaC, autoscaling, and emergency creates are healthy.

Three columns to keep separate

• Vendor control-plane status - dashboard, public API, creates, resizes, auth into the console. Read the component the vendor named, with timestamps.
• Data-plane / workload health - are existing instances, containers, or edge nodes still serving the paths your customers hit?
• Independent host reachability - did an outside probe of the relevant host answer? Reachable means the host answered. It is not a certificate that every management or product feature works. On StackWitness truth pages (for example /status/cloudflare or /status), that column sits next to vendor self-report on purpose.

When DigitalOcean said Droplets were unaffected while the API and panel were not, that was an honest control-plane vs data-plane split. Your customer update should keep the same split if your product only lost management paths.

Copy you can adapt

When creates and the vendor console fail but serving continues:

We cannot currently [create / scale / change] resources via [vendor] API or console as of [time UTC]. Existing [workloads / Droplets / nodes] continue to serve traffic. [Vendor] reports a control-plane / API incident; they state the data plane is not impacted. We will update this page when management paths recover or when serving fails.

When you are unsure which plane failed:

We are seeing [symptom] on [management path or customer path]. We have not yet confirmed whether this is limited to [vendor]'s control plane or also affects serving. We are checking vendor status component updates, our own error rates, and an outside reachability view of the host. We will not mark this operational until those signals agree.

What not to do

• Do not mark the whole product green because existing VMs still answer ICMP or HTTP.
• Do not declare a total blackout because the vendor dashboard is 500ing while customer traffic is fine.
• Do not invent independent probe results you did not collect. Attribute vendor status, your APM, and any outside reachability view separately.
• Do not skip the timestamp trail. Note when creates started failing, when the vendor named "data plane unaffected," and when management recovered.

Operator checklist

1. Name the broken path: create/update/console vs serve/read for customers.
2. Read the vendor status component timeline (API, control panel, Droplets, data plane), not only the top badge.
3. Confirm whether existing workloads still pass your own synthetic checks and customer SLIs.
4. If you have an outside reachability view of a relevant host, treat it as host answer, not product-health certification.
5. Write the customer update with the split explicit. If signals conflict, say they conflict.

StackWitness measures reachability and reads vendor self-reports so teams can attribute faster. We do not certify that a vendor met an SLA, and we do not issue a compliance verdict. Control plane down and data plane up is still an outage for anyone who needs to change the system. Name the plane you lost.