The Projects API is briefly offline.

A database migration has run past its window. Writes to projects fail with a 500 while we roll back to the last stable schema. Reads, sign-in and the dashboard are unaffected — and no data has been lost.

Projects API responding with errors

Started 14:02 UTC · 31 min · Next update in 6 min

500

Rollback across edge nodes8 / 10 nodes

Writes blockedReplay queue 1,284 eventsReads healthy

Check services Contact support

Writes are paused; retrying will not change your data.

Request trace

14:31:08 UTC

POST/api/v2/projects500

region: eu-west-1
request_id: req_01JBY7M4X2Q9K6T3
trace_id: tr_88f1c2e9a4d7
edge: edge-eu-07

Writes are blocked in eu-west-1. Reads and authentication keep working.

Include the request ID when contacting support.

What is still working

checked 14:33 UTC

Project readsAll regions
Sign-in & SSONo errors
BackupsVerified 13:55

No action is required from you. Queued writes will replay in order once the rollback completes.

Live incident

Monitoring

Write failures on the Projects API

INC-2491 · opened 14:02 UTC · 31 min

Peak impact: 38% of writes
Scope: eu-west-1
Data loss: None detected
Next update: In 6 min

Monitoring14:26 UTCLatest
Writes are succeeding again on 8 of 10 edge nodes. The error rate has fallen from 38% to 4.1%; we are holding the maintenance window open until the last two nodes report healthy.
Mitigating14:18 UTC
The migration has been rolled back to the last stable schema. Queued writes are being replayed and checksums verified before traffic is reopened.
Identified14:09 UTC
A column default in the migration conflicts with the current write path. Reads are unaffected; project writes fail with a 500 in eu-west-1.

Show 3 earlier updates

Investigating14:02 UTC
The Projects API is returning 500 responses in eu-west-1. We are checking the maintenance window that opened at 13:58 UTC.
Maintenance started13:58 UTC
The planned database migration began on schedule. Expected impact was a short write pause in eu-west-1 only.
Preflight checks13:55 UTC
Backups were verified, replication lag was under one second, and the migration job was queued.

Updates every 15 minGet notified

Get updates

One email when this incident changes, one when it resolves. No marketing, no account needed.

RSS feed @halcyonstatus

Service impact

4 services

2 healthy · 1 degraded · 1 down

Projects APIRead and write endpoints for workspace projectsMajor outageWrites blocked in eu-west-1
WebhooksOutbound event delivery and retriesDegradedDelivery lag up to 4 min
DashboardWeb console and read-only viewsOperationalLoading normally
AuthenticationSign-in, sessions and SSOOperationalNo elevated errors

Last checked 14:33 UTC · from 3 regionsRefresh status