How do you keep prompt rollbacks from splitting versions across services?
retained text
We tried to restore prompt version 181 across 22 services last night and ended with a split fleet. 15 services returned to 181. The other 7 stayed on version 184. Both versions still produced valid JSON but 184 emitted the newer field layout, so downstream parsers rejected 6.8% of structured responses. The rollback looked complete and the incident stayed open. The staging and production prompt copies were updated by hand. A rollback meant opening every service config, pasting 181, redeploying and checking logs. We didn’t have prompt lineage beyond a version label that operators could set independently of the text. The deployment record shows which service artifact shipped but it can’t show which prompt text a request saw. We found the seventh service after everyone had already called the rollback complete. So the immediate fix is to stop treating prompt text as service configuration. We need immutable versions instead of copied text. Environment promotion should move the same version from staging to production. One rollback target should resolve consistently across all 22 services at request time. Each response should carry 181 or 184 in request metadata so parser rejection can be tied to the exact text and schema contract. Manual reruns have hit their ceiling. What I’m still working through is failure behavior during partial promotion. A central pointer reduces version drift. Caches and regional rollout can still leave requests on different versions for a few minutes. We can fail closed when the parser contract changes or keep serving the older schema until every service confirms the new version. Neither option is free once requests are already in flight. How are you making prompt rollbacks atomic across services while keeping the exact version visible on each request
document fetched 2026-10-14 00:39 · canonical https://reddit.com/r/PromptEngineering/comments/1x4u0hy/how_do_you_keep_prompt_rollbacks_from_splitting