Railway's post-mortem: a build-sandbox escape, one over-trusted credential and $740,000 of fraudulent AI charges

Railway, the developer hosting platform, published its full incident report on November 10 for the breach it disclosed on October 21. The report is unusually specific about the chain of failures, and the number at the end of it, about $740,000 of unauthorised AI usage on customers' stolen keys, is the first hard figure on what a platform-scale credential theft costs once the keys reach the resale market.

The chain, step by step

The attacker created dummy accounts from October 18 and, on October 20, exploited a previously unknown vulnerability to partially escape filesystem isolation on a multi-tenant build virtual machine. That exposed a credential meant only for build work. An internal Temporal workflow service accepted it as a trusted client, so the attacker could read the stored histories of 353 other builds, including some build-time variables, and pull a copy of Railway's monorepo through the build cache. The source code revealed a wrongly defined OIDC binding, which on October 21 at 03:26 UTC yielded a short-lived cloud credential. With that and internal SSH access the attacker reached an orchestration machine holding a long-lived decryption credential, queried the database of encrypted environment variables and decrypted some through the key-management service between 04:42 and 07:15. Railway revoked the cloud and static credentials by 07:22 and evicted the attacker at 09:25, about 26 hours after the first exploit.

What was taken and what it was used for

At most 1.85% of workspaces, about 100,000, may have had variables decrypted; Railway stresses that is an upper bound counted by environment. Linked-account tokens for 1,852 users, including GitHub OAuth tokens and bring-your-own-key agent credentials, were reset. The attacker's database queries matched AI provider key names first and cloud, database and payment credentials second; no customer database was found to have been accessed. The keys then went to work. Customers saw coding-agent sessions and training-data jobs they never ran. Railway believes the credentials were sold for distillation, the pattern Google calls LLMjacking, and cannot yet say whether the intruder ran the resale service or supplied it. CSO Online, citing Wiz's analysis, adds the timing that matters for defenders: keys were tested within 30 minutes of theft and flooded with traffic four to seven days later, and one key restricted to a narrow set of model profiles throttled most of the thief's traffic.

Who pays

Railway is reimbursing customers, including for charges the providers refuse. AWS Bedrock and Google Gemini are refunding under existing fraud policies; OpenAI and SpaceXSI are reviewing case by case; Railway is covering Anthropic and OpenRouter costs directly. That split is now a known term sheet. A complaint has been filed with the FBI's IC3 and a forensics firm is engaged.

What Railway changed

Every build now runs on a dedicated ephemeral VM with a narrowly scoped credential. CI was rebuilt with fine-grained access, two-factor and biometric approval for critical areas and SIEM alerting. Service accounts were reviewed for least privilege and moved to short-lived, just-in-time credentials where possible. Every encrypt and decrypt operation is centrally logged, and only the secrets service can touch data-encryption keys. Railway also rotated everything twice, bootstrapped a new root CA, added mass revocation of app permissions, and committed to a fixed assessment window after which it notifies broadly regardless of what it knows.

The counter-argument

Railway's candour is being praised, and the report is a model of its kind, but the controls it added are standard practice at the cloud providers it competes with. The preliminary report took a day; the full one took three weeks; customers' keys were being drained on day four. Candour after the fact does not substitute for isolation before it.

What to do

Treat every secret stored on a third-party platform before October 21 as stolen until rotated. Put a hard spend cap and a model allow-list on every production AI key; that is the control Wiz saw working. Rename nothing, scope everything. Audit which internal services accept which credentials, and whether a build token can reach anything beyond the build system. Log decryption, not just access. And write the refund question into your inference contracts now, because the answer currently depends on which vendor's logo is on the invoice.