What Broke When We Moved to a Hosted File Upload API
Four breaks, fourteen days, and a verdict that survived all of them
For six hours after cutover the dashboard was entirely green, and I remember saying out loud that this had gone suspiciously well. Then someone noticed the analytics pipeline had stopped receiving upload events, and had stopped quietly, in the way that things do when nothing is technically failing. We migrated to a hosted file upload API in March. This is the honest changelog, breaks first.
“Moving to a hosted file upload API broke four things we did not expect: our tests (which mocked the old endpoint), our analytics (upload events changed shape), a compliance assumption about file location, and one webhook consumer. All four were fixed within two weeks, and none argued for going back.”
Migrations rarely fail at the seam you are watching. They fail at the three or four you had forgotten were seams at all.
Key Takeaways
- Test suites coupled to a mocked upload endpoint are hidden migration debt, and ours took the longest to repay.
- Upload event schemas change shape in a migration, so analytics consumers need a mapping layer rather than optimism.
- A data-residency requirement can survive a hosted pipeline when files land in your own bucket and region.
- One webhook consumer assumed a payload format, and contract tests would have caught it before a partner did.
- All four breaks were fixed inside two weeks, with no regressions after day fourteen.
Why We Moved, In One Paragraph
Why we moved is one paragraph, because the decision is the least interesting part of this story. Maintaining uploads was crowding out the roadmap, so we migrated to a hosted file upload API and braced for surprises. That is the entire answer to how best to handle file uploads without building your own infrastructure: somebody else runs the part that is not your product, and you spend the reclaimed time on the part that is. Everything after this paragraph is what that cost us in the fortnight nobody plans for.
Break 1, The Test Suite
CI went red within an hour of the migration branch existing, and the failures were in tests that had nothing to do with uploading. Order processing. User onboarding. A billing test from 2022.
The cause was mocks. Over three years we had stubbed our old upload endpoint in dozens of places, and every stub encoded the old response shape as a fact about the world. Any test that touched a flow involving a file, however incidentally, had one of these frozen inside it. Changing the endpoint invalidated all of them at once, and none of them were where anyone would think to look.
The fix took six days, longer than the other three combined, and most of that was reading tests to work out what each one had actually been asserting. We rebuilt the stubs as fixtures generated from real responses, so the next time the shape changes the fixtures change with it rather than lying convincingly. If you are planning a migration, start here, and start before the migration branch exists.
Break 2, Analytics Event Shapes
This is the one that stayed quiet. Uploads worked, files arrived, users were happy, and our upload analytics simply stopped populating, because the new events carried the same information under different names and a different nesting. Nothing errored. A consumer expecting one field found nothing where it looked, wrote nothing, and moved on.
We lost about six hours of upload analytics before anyone noticed, which was six hours of a dashboard nobody was looking at during a migration everyone was looking at. The fix was a mapping layer that translates the new event into the shape the downstream consumers already expected, three days of work, most of it spent finding every consumer.
The lesson is narrow and worth stating plainly. When your upload provider changes, your event schema changes, and silence is the default failure mode. Alert on the absence of events, not only on errors.
Break 3, The Compliance Assumption
Three weeks in, an enterprise security questionnaire asked a question we had been answering confidently for years: where are customer files physically stored? Before the migration the answer was a bucket we owned in a region we chose. After it, the honest answer was that we needed to check, and “we need to check” is a bad sentence to write in a questionnaire. Anything touching data sovereignty has a way of turning a technical detail into a contractual one.
The resolution took four days and no argument, because a hosted pipeline does not have to mean hosted storage. We configured uploads to be processed through the service and written into our own S3 bucket, in the region we had already committed to, so the answer to the questionnaire went back to being the answer it had been before. Setting up a file upload API that stores into your own bucket is a normal configuration rather than a special case, and I wish we had confirmed that during evaluation rather than during a compliance review.
Break 4, The Webhook Consumer
We found this one when a partner emailed to ask why their integration had stopped working. Not from monitoring. Not from a test. From a person, politely, three days after it broke.
Their consumer parsed our post-upload webhook and depended on a field that had moved. We had reviewed our own consumers and forgotten there was an external one, which is exactly the class of mistake consumer-driven contracts exist to prevent: each consumer states what it actually needs from the payload, and the provider validates against the union of those expectations before shipping anything.
Two days to fix, versioning the payload and adding contract tests on both sides. The fix was the cheapest of the four and the finding was the most uncomfortable, because a partner discovering your breakage before your monitoring does is a monitoring problem wearing a migration costume.
The 90-Day Scoreboard
Ninety days after cutover we compared the quarter before against the quarter after, on measures we were already tracking, because a migration that only improves the numbers you invented afterwards has not improved anything.
Upload success rate went from 97.8 to 99.4 percent, which is the number that decided the verdict. On-call pages attributable to uploads went from eleven in the quarter to zero, including through the migration weeks. The p95 duration for a 100 MB file on a mobile connection nearly halved, on the same networks, because chunked transfer does that.
We had asked which file upload service has the most reliable uptime and upload success rate, and I still think that question is unanswerable from outside. A status page reports the provider’s uptime, not yours. What we could answer is what our own success rate did on our own traffic, and it did this. Measure your own, on both sides of the change, or you have no basis for a verdict at all.
The last row of that table is the honest cost. We now have one external dependency in the upload path that we cannot fix ourselves at three in the morning. We took that on knowingly, and the eleven pages we no longer get are what we took it on for.
Breaks First, Verdict After
Four breaks: a test suite full of mocks, an analytics schema that changed underneath us, a compliance assumption that needed re-establishing, and one webhook consumer we forgot we had. Fourteen days from cutover to the last fix. Nothing has regressed since.
Would we do it again? Yes, and the reason is in what broke rather than in spite of it. Every one of the four was a coupling we had built and never noticed, exposed by a change rather than caused by one. The migration did not create those problems. It sent us an invoice for them, and the invoice came to two weeks.
So take the checklist. Grep for mocks of your upload endpoint before you write a line of migration code. Alert on the absence of upload events, not just on errors. Ask where the files will physically live before a customer asks you. And find your webhook consumers, including the ones outside your company, because one of them is depending on a field you are about to move.
Frequently Asked Questions
What breaks in a migration to hosted uploads?
In ours, four things: tests that mocked the old endpoint, analytics consumers expecting the old event shape, an assumption about where files are physically stored, and a webhook consumer parsing the old payload format.
Can hosted uploads satisfy data-residency requirements?
Often yes, by processing uploads through the hosted pipeline while the files themselves land in your own bucket, in the region you have committed to. That is how we resolved ours.
How long did the breakage last?
All four were fixed within two weeks of cutover, with the test suite taking six days and the webhook two. No regressions appeared after day fourteen.
What would you do differently?
Audit the test suite for mocked upload calls before starting, and inventory every webhook consumer including external partners. Those two account for eight of our fourteen days.
