How FireHydrant handled the SVB banking crisis

On Thursday, March 9, 2023, something was afoot at our primary bank, SVB. By Friday, March 10, 2023, messages from our investors helped us quickly understand that FireHydrant needed to maneuver through a complex incident that was unfolding.

Operational incidents are incidents like every other. Negotiating a fast-moving, unexpected situation requires clear roles and responsibilities, a place for the team to gather and work the problem, and a system to document the tasks, activities and decisions that transpire along the way. Naturally, we turned to FireHydrant. 

Mar 10, 2023, 7:49 AM PST - Incident declared

The first action we took was assigning roles. 

1. Incident Commander - Robert Ross (CEO)

2. Planning - Chris (CTO, General Manager)

3. Responder - Courtney (Head of Finance), Mignon (VP of People), Brian (Finance partner)

We then emailed our entire board a link to follow along on a FireHydrant private status page after they gave us input on how to approach the situation that was unfolding rapidly.

FireHydrant automatically created a private Slack channel for us to assess the situation and understand what was and was not possible. We were also able to invite contractors and advisors to help us navigate the rapidly-changing landscape. Updates were coming in fast: 

Mar 10, 2023, 8:07 AM PST - Let’s move some money to a new account

Within 24 hours it became clear this was a full blown banking crisis.. The consistent advice from our circles was to move funds if you can. We used FireHydrant to create and assign tasks in Slack to track these.

Our Head of Finance Courtney was assigned and moved quickly to set up new accounts with other institutional banks. Then some big news broke.

3/10/2023, 8:53 AM PST - We learn that the FDIC has taken over SVB

Screenshot from our incident retrospective in FireHydrant

3/10/2023, 6:54 PM PST - We start collecting options

The SVB collapse was truly historic, and no one really knew what was going to happen. Once we accepted that we would very likely not have additional information before Monday, we settled on solving the most important problem next: Payroll.

The SVB collapse happened at quite possibly the worst moment for a company: days before a payroll run and right before the weekend. With the uncertainty of what Monday would bring, we lined up several options to fund payroll for all of our team membersHere’s what we considered:

1. A note from our investors who went above and beyond to make sure they could fund it

2. Payroll loan from a lender who we knew well already

3. A (new) credit card and shoulder the fee associated with it

3/11/2023, 7:42 AM PST - New bank account opened

We continued making sure we had other banks available to send money to ultimately point our payroll provider at the new account. Given it was a weekend, it was more difficult to push through and get a new account open. Fortunately, several bankers were working a long weekend to support us (and others in our situation)..

3/12/2023, 3:30 PM PDT - Treasury announcement is linked in Slack

https://www.federalreserve.gov/newsevents/pressreleases/monetary20230312b.htm

“After receiving a recommendation from the boards of the FDIC and the Federal Reserve, and consulting with the President, Secretary Yellen approved actions enabling the FDIC to complete its resolution of Silicon Valley Bank, Santa Clara, California, in a manner that fully protects all depositors. Depositors will have access to all of their money starting Monday, March 13. No losses associated with the resolution of Silicon Valley Bank will be borne by the taxpayer.”

At this point, Courtney had been on-hold with the FDIC for 3.5 hours when this announcement was made.

Relief isn’t enough of a word to describe the feeling we all had. But we still had one lingering question: what about payroll?

“Theoretically” is right.

3/13/2023, 7:31 AM PDT - SVB Opens Up

The way payroll runs is the payroll provider performs an ACH transfer from the bank to themself, but in this situation, we were running low on time to get our funds from our bank to the payroll provider. This is where wire transfers come in, as they are “instant.” In-fact, ADP forced our hand as the only way to fund payroll for the current cycle.

We had to login to ADP, find the two amounts we needed to send, create the wires that were very specific in the details, and send them on their way, all while SVB’s website was flapping between available and unavailable. It felt like running across a tightrope over alligators with an egg on a spoon.

After retrieving the correct amounts from Mignon, Brian then hit a brick wall with two-factor authentication with SVB failing over and over again, likely due to the thundering herd of people trying to do the same thing.

My email inbox every time poor Brian tried logging in

3/14/2023, 9:37 AM PDT - Ready, Aim, Fire

Our wire transfers are set up to have multiple approvals before they are sent. Evan (another finance partner of ours) had done the first approval, which meant I had to do the second before they’d be released. I approved the wires, and then we all waited with bated breath for the next morning when our payroll was expected to hit its normal schedule.

3/15/2023, 4:25 AM PDT - Confirmation

After five days of chaos, stress, and a level of anxiety very few should ever experience, payroll went through on March 15th without a hitch for all FireHydrant employees.

How did FireHydrant make a difference during this process?

FireHydrant wasn’t designed for an insolvent bank crisis, but it was designed to quickly get the right people together at the drop of a hat and automatically keep a record of the path through problem solving to fix. When the SVB crisis unfolded, it came naturally to type “/fh new” in Slack and get started mitigating.

FireHydrant also works well for creating a psychologically safe environment for people to quickly make decisions in an uncertain environment. It worked exceptionally well to give key updates to outside stakeholders via our private status pages that we automatically create, too. Further, the incident retrospective in our platform was used to recap the entire operational incident, and write this very blog post about our perspective of the SVB incident.

Special thanks

Thank you Courtney for jumping in with us on a truly historic situation in only your second week, and making a huge difference. As every software engineer knows, incidents are a great way to learn about a system, and I think you got the crash course with this one!

Thank you to our investors, Work-Bench, Menlo Ventures, Harmony Partners, and Salesforce Ventures who truly went above and beyond and did not wait for us to ask if we needed help.

Also, thank you to Mignon and Chris for helping communicate and coordinate during the chaos, and keeping the ever steady hand you consistently bring to the team.

Finally, thank you to the entire FireHydrant team for the trust you gave us during this surreal experience. That made getting through it much easier on the soul.

Cheers! And let's hope that something like this never happens again... but the software engineer in me knows that it probably will. However, we're more prepared than ever to take it on now.