We had a small bad code-push that went onto the servers at 07:54 UTC and was resolved at 09:54 UTC; during this 2 hours interval, some amount of our "side effect" code was effectively disabled. I believe that this was a side-effect of this problem.
We have:
- fixed the affected code (released at 09:54 UTC)
- added multiple separate automated mechanisms to ensure that we can't hit the same problem again
- started reviewing for any side-effects we missed in this period that will have significant impact
We apologise for any adverse impact here.
