Status
Subscribe to OP3 status updates as an
RSS feed
2025-11-18: Cloudflare global outage (11-19 update)
- OP3 was impacted by a Cloudflare global outage, which affected all edge locations, caused by "a latent bug" (full Cloudflare post-mortem)
- Based on initial telemetry it looks like OP3 podcast redirects went down completely for a short period around 7am Eastern (12pm GMT), then down again at 8am (1pm GMT) for 90 minutes, and back to normal at 9.30am Eastern (2.30pm GMT)
- [2025-11-19]: Looking at all OP3-measured downloads for yesterday in a comparison with the same day last week, we see the same outage periods confirmed. Apart from those time periods, very similar download numbers on a daily, hourly, and 10-minutely basis.
2023-12-15: Reprocessing from mid-October (done 2023-12-21)
- We've just noticed that data from three large incoming colos (IAD, ATL, & MIA) have not been included in the processing pipeline for several weeks: IAD since 10/10, ATL since 10/14, MIA since 11/03. Nothing has been lost, but we now have a potentially large amount of unexpected data to reprocess and recompute.
- First, we'll need to process the 'hits' (redirect logs) coming in from each of the three colos. We're starting with MIA, since it's the least backed up, then we'll process IAD and ATL. This is likely to take several days, as it competes with continuous live data coming in at the same time from all colos. See the colo monitor below to track this process live. (This is now complete)
- Once all colos are up to date, we'll need to recompute downloads for all shows based on the underlying hits, starting from 10/10 up to the present. This will probably take several hours. (This is now complete)
- It's likely that download numbers currently shown for most shows are a bit lower than they should be, starting from the middle of October. The numbers will be up to date and comprehensive again once this process completes. No data has been lost, just delayed. (This is now complete)
- Sorry about the delay: OP3 will always report issues like this as soon as they come up since we are committed to transparency – not only in data and codebase, but also in operations and costs.
- Subscribe to this page (RSS link above) to follow the updates as we catch up.
- [2023-12-21]: Finished - all shows have been fully reprocessed.
Historical uptime
Colo Monitor
Here's a live look at the internal backlog of incoming hits coming in from each colo (if a backlog exists). We want to keep all colos within an hour behind.
(no backlog, everything up to date)