What leading Outward Bound expeditions taught me about incident management

Like many, I took a non-traditional path to get to engineering. My degree is actually in adventure education. Before I was flipping bits, I was teaching students how to flip 14-foot boats. My incidents were less SLO-centric and more around leveraging EMS (emergency medical services). 

I worked as an instructor leading expeditions and managing logistics for Outward Bound for six seasons. My bread and butter during my instructing days was the 22-day, multi-element course out of central Oregon — it involved expeditionary rafting, mountaineering, and rock climbing. After an amazing six years living the instructor lifestyle, I was ready for a change in pace (and a modicum of stability).

Although the incidents and emergencies may be different, there are a number of lessons I learned leading weeks-long expeditionary courses that I have carried with me into my roles doing incident management for companies like Sendgrid and Twilio, and that I still use today at FireHydrant. In this blog post, I’ll share three of the most impactful.

Preparation is key

Outward Bound, the organization I worked with most, has been running programs in the United States since 1961. When I worked for the organization in 2005 (pre smart phones), every instructor was required to carry a fairly full, 5.5 x 8.5-inch three-ring binder. This instructor manual was packed with institutional knowledge that had been accumulated over its decades of operation.

Vintage 2005 Outward Bound zippered manual

While we did complain about carrying this small tome around (an extra 3 pounds is a lot when you're carrying everything you have up the side of a mountain), it was an invaluable resource. “Incidents” on these expeditions could span a range of issues — anything from environmental hazards, medical issues, or a pinned 16-foot oar rig. If the exact situation wasn’t in the manual, we’d at least find a framework for how to think about the problem at hand — like when and how to “phone home” (we carried a satellite phone) and what protocols to leverage or what to change if the severity of the incident increased.

On a similar note, we would spend weeks before our main season on training trips to ensure we had a grasp of the areas and could be evaluated for our technical skill in the mediums we would be working. While you can’t capture all the collective knowledge of 60 years of outdoor programs in a three-ring binder, you can provide models and real-world context to ensure everyone is as prepared as they can be for potentially complicated situations. 

Sounds familiar, right? When managing software incidents, outlining a consistent incident management plan and process is critical to having long-term success. Running “game days” using prior and common incidents (aka building up familiar context and sharing technical skills) is a great complement to having things well outlined. 

Chris Voss and Tahl Raz said it well in their book Never Split the Difference, “When the pressure is on, you don’t rise to the occasion — you fall to your highest level of preparation.”

Stay calm

The difference between a trip and an adventure is that the outcome of an adventure is unknown. Similarly, incident management can be captivating due to the unknowns. Regardless of the source, being confronted with unknown and stressful situations often comes with a rush of adrenaline that triggers our animalistic “fight, flight, or freeze” responses. This response bypasses our amazing brains, which are capable of all manner of problem solving. Rash decisions in emergent environments oftentimes end up doing more harm than good. The key in an incident of any type is to stay calm.

Just another day in the office on the Deschutes River.

While practice, preparation, and simulations can be helpful in keeping your nervous system in line during an incident, the best thing to do when faced with a novel emergency is often paradoxically to do nothing. Well, not nothing exactly, but pausing, taking a breath, and looking closely at the situation sure will feel like nothing — especially when your reptilian brain is strongly suggesting to bolt and hide behind a rock.

One of my search and rescue instructors in college would encourage us to stop and smoke a metaphorical cigarette when facing something new. Action without critical thought, direction, and good communication can be even more “dangerous” than doing nothing. I’ve seen this play out in the backcountry and the incident war room in equal measure.

Let others lead too

On the first day of an expeditionary Outward Bound rafting course, the instructors would do everything for the students. Set up camp, guide paddle rafts, cook, pack — it looked nearly indistinguishable from a guided comercial trip. On the second day, students would start learning how to captain paddle boats full of their peers. That evening, we’d teach them how to use camp stoves and set up their own tarps. Then they’d go on to learn how to read a river map and run rapids, and learn about river (and group) dynamics — all while learning to lead themselves. If things went to plan, by the end of a week, they’d be rafting solo while we instructors would retreat to our oar rig to hang out, cheer them on, and run safety as needed. 

Working hard while students perform a boat flipping exercise.

The goal of these expeditions was substantially different from a guided trip where clients are there to have a good time. Our aims were to empower students to have experiences they otherwise never could have had, and build important human and technical skills on the way. 

My coworker Malcolm’s recent blog post on how to begin dismantling hero culture in your engineering org drives home the same idea. If one person is doing everything, it’s challenging for others to learn much of anything. If you find yourself disproportionately driving your company’s incident practices, consider progressively handing off responsibility and moving into more of an incident commander role, giving others the chance to step up and learn through experiences.

There is something uniquely rewarding about enabling a team vs directly performing a task. Whether it’s in the context of surviving and thriving in the outdoors or helping your organization start the journey of moving from a reactive to proactive incident management program, leveling up others and letting someone else take the reins is worth the effort.

Conclusion

Whether you’re dealing with a sprained ankle in the backcountry or a down service affecting your product, at the core, you’re solving a problem. Although the problem itself may change, one thing remains constant — the more prepared, calm, and empowered you and your team are, the better outcome you can expect.