Being an engineer at Amazon has some plusses and minuses. One of the big minuses that you hear of often is the on-call nature of the place.
Amazon’s engineers are also expected to handle the operations for the team. In many ways this is a very good thing since it encourages good quality code since no one wants to get paged in the middle of the night for some problem that could have been prevented with a little more care.
The downside is the page in the middle of the night because something broke.
We now know a new failure mode for our system and we’re fixing it. (Expiring credentials if you really are curious…)
Our team isn’t nearly as bad as some of the problem teams at Amazon. I too have heard horror stories. We handle mostly dev support questions (second and third tier questions, not dealing with devs directly) and random small things. A real page happens once a year or so.
The problem is, quite obviously, getting up at 2 in the morning and fixing a problem and babysitting deploying and testing a fix. And paging someone else to help out with a different dependent system as well just makes me feel bad.
That can really screw up the next day — as you’re only a step above a drooling lump.
But, an added bonus for us is that we’re getting another three peeps on the rotation! Ever six weeks is a lot better than every three I tell you what.