Railway users report blocked deployments after environment secrets are compromised

• Customer secrets may have been exposed: Railway disclosed possible access to encrypted environment variables and some decrypted contents, while one user reported keys being drained.
• Replacement secrets could not reach running applications: theo said blocked deployments kept Lakebed, Postplan and Shoo offline while he waited to redeploy with rolled keys.
• Deployment access was still blocked 17 hours in, theo reported; Railway's status log later recorded an Enterprise-only restart.
• Leaving came with a substantial price gap: theo estimated that replicating his setup elsewhere would cost at least three times as much.

Railway's variable docs make secret changes wait for a deployment, a brutal recovery dependency. The company believes the attacks targeted AI inference providers. Its credential-reset guide contains another consequential distinction: removing GitHub registry tokens from Railway did not revoke them at GitHub.

Railway detected anomalous activity in systems storing customer environment variables on October 21. Its disclosure said an unauthorized third party may have accessed encrypted variables and obtained decrypted contents for some of them.

The potentially exposed credentials included:

• API keys.
• Database credentials.
• Private registry credentials.

Initial analysis suggested a subset of users was affected, but Railway asked all customers to replace stored secrets as a precaution. The notice did not identify an initial access vector or provide an affected-account count.

Railway also said it had contacted upstream AI inference providers to request proactive key revocation, describing the incident as part of a suspected series of attacks against platform companies.

Changing a value in Railway's dashboard does not immediately update a running application. The variable documentation says additions, edits and removals become staged changes that must be deployed before they apply.

Railway's zero-downtime rotation guide describes a credential-overlap sequence:

1. A replacement credential is issued while the old one remains valid.
2. Services are redeployed with the replacement.
3. The new deployment is verified.
4. The old credential is revoked.

The build freeze blocked the second step. Revoking a compromised credential could therefore leave the existing deployment without working access to its dependencies.

Lakebed, Postplan and Shoo would remain offline until replacement keys could be deployed, according to theo's outage report. He subsequently posted the dashboard after manually taking his Railway-hosted projects offline.

Railway's public incident log began at 07:24 UTC on October 21. By 09:28 UTC, the company was reporting unavailable dashboard, API and CLI access, paused builds in every region, and an affected support site.

The security disclosure was published at 20:44 UTC that day. Customers criticized the delay in communicating the exposure of secrets.

Theo reported 16 hours without deployment access, followed by 17 hours. In a subsequent reply, he said he still could not build.

The deployment outage and the shutdown of his applications were distinct events: Railway had paused the deployment pipeline, while Theo said he manually stopped his workloads.

Theo spent several hours evaluating alternatives before reporting that none offered a drop-in replacement for his setup.

His follow-up replies identified specific obstacles:

• Render pricing: approximately 2.6 times his Railway cost, he estimated for his setup.
• Cloudflare compatibility: two affected projects could not run there, and a third would be a rough fit, according to his reply about Cloudflare.
• Fly.io reliability: he questioned its stability when another user raised it as an alternative.

Railway's October 22 reset guide documented changes beyond customer-managed environment variables:

• Storage buckets: Railway invalidated and regenerated ACCESS_KEY_ID and SECRET_ACCESS_KEY for every bucket in every environment. Bucket names, endpoints and objects were unchanged.
• Running bucket clients: containers read their variables at startup, so services needed redeployment to receive the replacement credentials, including services using reference variables.
• GitHub account connections: Railway invalidated the connection tokens it held, requiring customers to reconnect GitHub.
• GitHub registry credentials: Railway removed stored registry tokens, but those tokens remained valid at GitHub until revoked there. Private ghcr.io image pulls required replacement credentials.

The incident log also recorded invalidation of all Railway API tokens, disabling existing integrations that used those tokens to call Railway's public API.

By the latest retrieved October 22 update, build access had resumed for Enterprise workspaces; other plans were still awaiting restoration. Recovery remained impaired by deployments spending longer than expected at individual stages, including “Running pre-deploy command.”