Name: Pager fatigue: Making the invisible work visible
Uploaded: 2025-08-07T16:42:12.300Z
Duration: 26 min 8 s
Description: Pager fatigue: Making the invisible work visible

Transcript for "Pager fatigue: Making the invisible work visible": Hello, everyone. Welcome to this, prerecorded session of page fatigue, making the invisible work visible. I'm sorry I couldn't be there in person tonight, but I'll do my best to, walk you through everything about page fatigue and what we at Instant, have done to sort of mitigate, engineer burnout and just bring visibility onto that invisible work that people do in the middle of the night to keep production up and running. If you work in incident management yourself, you might have been on a really rough shift, or you might have worked on an incident that, lasted for several days, and you know how exhausting they can be. And pain jeopardy can be sort of an invisible killer in a team, but it's really hard for EMs and your teammates to know exactly how much work has been done as you come into the, into the office in the morning still a bit bleary eyed and really tired from last night's pages. It's something that we see time and time again and something that we have tried to mitigate as much as possible. And this is one of the ways that we try to do so by making something called the fatigue score. And it looks like this. And the reason I'm here to talk about it is because I am the data analyst that made it and maintain it in some ways. My name is Matilda Hultgren. I am a product data analyst at Incidentio, and this is the fatigue score. This is sent, every morning in each of our product team's Slack channels. Every engineer is giving a fatigue category that goes from unaffected all the way up to severely disrupted. So today, I'm going to talk about, well, what the fatigue score is, how one goes about building it, how one goes about trying to capture the very, ethereal sense of fatigue, and especially pain fatigue, And how do we use it at Incidentio? And then it is prerecorded, but I will try to get ahead of a few questions that I frequently get asked internally that I'm sure you also will have. But, if you have any further questions, please just put it in the q and a, and, the team and I will be able to review them after the session. So we're gonna go into a small history session about fatigue because, this concept is not new. It's entirely unoriginal. Incident management already steal a lot of concepts from emergency services, severity levels, first responders, triaging. If you work in health, you're already very familiar with these. So fatigue scoring is something that is used in a lot of industries. It is particularly used in industries where human error can have disastrous consequences for either the environment or for other humans. When I was doing research for this, I think I looked at an example where I think it was an oil rig, and they use reaction times to make sure that people are sort of alert and ready to go on shift. In aviation, there's a bunch of scores and shift management tools that have been created to make sure that no one goes on a shift when they're not ready to do so. And if they have been on too many shifts in a row, they have adequate rest in between. This is also used in a lot of health service services. And at least in the EU, truckers, have military rest time to make sure that they don't spend too long behind the wheel. So not a new concept, but one that we are taking from all of these industries, that already use them to make sure that human error remains as low as possible. Because while we're all human, and you can absolutely make a mistake when you're well rested, trust me, I've done several, but fatigue makes us more error prone. And rest allows us to make fewer of those mistakes, reduces the risk, so to speak. So in our case, more rested engineers means ideally fewer bugs. It means better productivity, which should result in a better product. But the real reason is that it's kind of nice to know how your team is doing. You cannot really decide when your prompt breaks or when you get paged. These are things that are outside your control. But there are things that you can control, and those things are taking someone off the pager when they've had a really, really rough night, or maybe you need someone else to go over as incident lead if an incident has been running for a really long time. But in order to do these things and actively make a choice about them, an engineering manager or VP or tech lead need to know the state of their team. So as I was saying earlier, this is something that we are familiar with. We use our own product. We declare a lot of instance, and it's especially relevant because we're a London based organization with a lot of international customers. That means that if something goes wrong on the West Coast, maybe as they're wrapping up, work at the end of the day, it's possible that one of our engineers in London get paged, and they may even have to wake up. So it's something that we're very, very familiar with. So in order to increase the visibility on how much work is happening in and out of hours, so on incidents, what pages, we are seeing, One of our tech leads asked me to look into developing some kind of fatigue score so that each morning when we start work, we're able to see how the last twenty four hours on call have been for the people that were on the pager. A way to both categorize and visualize the level of pager fatigue that our engineers were experiencing. So how does one go about making a fatigue score? Unfortunately, for all of my colleagues, it involves me bothering them, quite a lot. So I obviously had to go to ask the people who knew what it was like to be on the pager. What does a bad night on the pager? What does a bad shift on call look like? So I conducted a few user interviews with the engineers, and a few common themes started, coming up the conversation again and again. So being paged at night is bad, but it isn't the end of the world. It's possible that it's a false positive or maybe it's something that it can be paused and worked on in the morning. A one time page, you you've woken up, but you can go back to sleep. Being paged multiple times in the night feels worse because now your sleep schedule is quite disrupted, and your sleep cycle is completely thrown off. But getting up to work actively on an instant for an hour is the worst because not only are you getting more and more awake, you are actively losing sleep. And while, yes, you do need rest to, if you've had a bad, night on the pager or a bad shift on call, but you're usually back to normal after a day off and, like, a good night of sleep. So in other words, the time of day matters. Being paged in the evening, say that you're paged at 8PM, is obviously a little bit annoying, but it's very different bit from being paged at 3AM. Also, pages only tell half the story. Time spent on working on incidents, contributes just as much as being just woken up. And fatigue does recover, but you do need proper rest where no one's paging you and you're not actively working on instance in order to fully recover. So I started making a model for this. If you already use Incidentio, you know that this sort of data is available or insights product. It also means that it's available for us. We can see how long our engineers have been actively working on that incident, and we can see, when they got paged. So second part, in order to visualize this a little bit better, was that I made a bar chart, showing over a full week when our engineers got paged. And then I asked them to draw on top of that roughly how their fatigue felt across the week going from zero and four. And then I started adjusting my model until it started looking like the lines they had drawn. Sometimes, maybe fatigue had gone up higher than I expected, and I realized that my weights for, late night work, was a little bit too low, and I increased the weight for the score that was attributed to that kind of work. Sometimes I realized that our major Incidentio are, affecting fatigue more than minor Incidentio do, so that needs to be, added to calculations. And after a while, after a few going back and forth, coming back to the engineers, asking them, could you draw again? It's been a new week. Could you draw your fatigue again? Eventually, my model started lining up with what I was seeing in these sketches. So part three is that it's time to ask the engineers how they're feeling every day for a few weeks. I would run my model, and then I would each morning, I would check what is the score at 8AM this morning for the engineers that have been on the pager. And then I would message that engineer and say, out of curiosity, on a scale from zero to four, how fatigued do you feel today? And then I would compare their answer to the number that my model had given me. Now this doesn't need to be perfectly accurate, but it needed to be directionally accurate, and the magnitude needed to be correct. If someone is fatigued and telling me that they're at a four, three and a half, my one and a half is not going to cut it. But it needs to be in the same ballpark. So in order to make this a little bit more human and a little bit easier to understand, a two can mean a lot of things to different people. But everyone seemed to roughly agree that these are the categories that we're working with. The difference between a two and a 2.5 might be minor, but two being disrupted, I could really use a break right now, just seem to be roughly accurate. So if someone, was between a a zero and a one, basically don't feel like they've been on the pager. Maybe they haven't been paged at all. Maybe they've got one one page during working hours. They're doing alright. They're doing fine. One to two means, yeah, I've I've done some work, but I'm feeling pretty good. I'll I can probably do another day on the pager. Two to three, you could probably use a break. Maybe someone needs to pick up an incident that you're working on for a bit just so you can have a breather, go get some coffee. Three to four is I have not slept. I feel awful. Please get me off this page immediately. And that resulted in this thing, which I hope you can see on the screen is a rough report, of the fatigue score, which is in the middle column here. And the bars two of columns to the right is the data that goes into the fatigue score, so you can see the breakdown of why someone was put into the category, that they're in. You got in and out of hours pages, and you got in and out of hours workload. And on the left hand side here is the engineer that was on the pager. You can also see, whether someone has moved between categories, from yesterday or if they have remained in the same category as yesterday. If someone spends two days in severely disrupted, for example, it's probably time that they get off the pager. But this information is only really useful if people, one, look at it, and two, use it to make decisions. Luckily, I know that quite a few people look at it, and I know this because I will get feature requests about it. People ask people tweaks about in the report, or people want to see a graph of their fatigue score across time. But how do we use it to inform our schedule and make decisions? A couple of ways to do this. The first one is I I asked my engineering managers, how do you use a fatigue score in your day to day work? And they said that, well, it's useful to reach out proactively when someone has had a bad night of a paycheck. Otherwise, you may have to rely on someone telling you or you have to go look into the data. You don't really have a sense of, like, you can see that a lot of work has happened, but you're not quite sure how bad that is in context. But this allows you to get a full overview of how your entire team is doing. And if someone's is in a, disrupted or severely disrupted stage, you can reach out, ask if they need to arrange cover, or offer your support in some in some way. It is also useful to know each morning as they look at this report, how widespread an instant was. Are the people in the report who have, been paged or have instant workload that were not supposed to be on call? Was there a major instant that several people got pulled into that it would be useful for you to know about? And, overall, getting visibility on that instant workload, how your team have spent the past twenty four hours is useful, in the context of in hours incident workload. If so if your team has spent quite a bit of time in incidents, they might not have had time for project work, and you may need to adjust your daily planning. Those are some of the business reasons why you might look at this. But, you could also use a fatigue score to take care of your colleagues and friends. I work in data, which means that while I am engineering adjacent and I've worked with engineers, I might not know that all the technical aspects of what an incident entails, might not know fully how bad it is. I don't have, you know, the complete overview of what their day has looked like. But I do know what a big red flashing sign next to someone's name look like, and that allows me to check-in on them, offer some support, let them know that I'm doing a coffee run, ask them if they need anything. And it just allows you to, as a company, check-in on your product teams and your engineers often and see if they require your help or your support in some way. The third way that we can use a fatigue score is to use it to look at long term trends. This is an example of a weekly visualization of the team and the days that they was that they spent in different categories of fatigue. But you can also extend this to look at a full month, see if there's a person who has had a lot of red days recently, or maybe your entire, team has been seeing more red and orange days overall. Not only is this, you know, not great for patient fatigue, but maybe you are seeing more instance in general, or maybe you're getting more out of hours pages than you previously did. By visualizing these long term trends like this, it avoids the, boiling frog problem. We're sort of just sensing that something might be getting worse over time, but you're not quite sure. There's not really a way to prove it. And it also allows you to look at it the other way. If you have tried to mitigate page noise, alert noise, and just try to reduce the amount of reactive work that your team does, using something like this can make make you realize that, yes, things are getting better, and you're able to sort of visualize things getting less and less red over time. So those are the ways that we are using the fatigue school to take care of each other and just get more visibility on how much time, and work people are spending in instance and how often they get paged. So I'm going to move over to the frequently asked questions. So these are frequently asked, internally and, some of the questions I suspect that people may have. First one, is this fatigue score available externally? And, unfortunately, no. At this point in time, there's no plan to add the fatigue score to the actual product. And the reason for that is, well, it's modeled on our engineers, and their specific perception of page fatigue. And every engineering team, every organization is going to look very different. You're all going to have your own perception of what a bad day on the page it looks like, or you might have different encore rotors that don't fully work with this type of modeling. And even if I had somehow made the absolute perfect model, that could be used and applied to everyone in the world, it's very difficult to know that without bothering your engineers as well and getting a sort of feedback loop checking you are in this category today. Would you say that that accurately reflects your page fatigue today? So So for the time being, it is currently not in the product. But some of the things that you can do instead, if you are interested in this sort of stuff, is to look at the data that I used in the fatigue score. This is a if you use Incidentio, it is available in the incident dashboard under insights. And you if you aren't able to measure incident workload, I would look into getting insights from, just your general page noise or how long people are spending on tickets. Doesn't matter too much how you measure it, at least in my opinion, as long as you have some kind of measure that you feel comfortable with. As long as you make it visible and you make it visible often. I'm talking daily because this is something that you need to check-in with your team essentially before they burn out. And if you set up a daily or weekly report, in a place where you see it often, it adds a reminder to review the data and check-in regularly. And another question that, have been asked internally quite a lot, and I'm not surprised, have you thought about comparing it, correlating it with external sleep or health data? At least our engineers wear Apple watches, Oura rings. I myself wear a Garmin. So there's obviously a lot of curiosity whether the pager fatigue and the fatigue score correlates with some of the stress levels or sleep scores that you might be seeing in these devices. And I was thinking about doing this and looking at whether there was a correlation here for a while. But after talking to a few of the engineers, they all sort of seem to agree that pain fatigue is a very different concept from regular fatigue. You can be fatigued for a variety of reasons. And for that reason, the correlation between your pager fatigue and your regular stress levels of fatigue, would probably not be correlated at all. You can actually have a great night sleep while you're on the pager, because you've had to to stay home for the night because you're on the pager. And for that reason, I didn't want to bring it in and try to correlate the two. I also don't think that a fatigue score should have a final say. It shouldn't be considered the end all of the conversation. It is very separate from a person's full state of tiredness, stress, fatigue, how they slept. If you come in in the morning and one of your teammates is says that they have low disruption, but they feel really, really rough, you should probably believe them because there's probably more to it than just the pager fatigue. The pager fatigue and the fatigue score is that you start a conversation and remind you to check-in on your team. It shouldn't be the final project. And that is the end of the webinar. I'm really hoping that this was useful in some way that you have, thought of some ideas that you might check page fatigue in your company or in your team, yeah, and and learn something about the history of page fatigue today. Thank you very much.