Changelog

Required fields have been a hot topic at FireHydrant. Choose too many (or the wrong ones), and you unnecessarily annoy your team during an incident or encourage sloppy data entry that someone has to come back and clean up manually. Don't use them at all and risk insufficient data to efficiently propel an incident toward resolution. But despite the controversy, required fields for incident declaration was one of our most-requested features — and it's now available as an option for configuring your instance.

The debate within our product team centered around ensuring that we didn't build any impediments to people's first experience with the product, declaring incidents. We feel that 1) it should be as easy and intuitive as possible for anyone to declare an incident, and 2) teams must be able to capture the full scope of their incident from the moment something seems off. We considered whether required fields would force investigation outside of FireHydrant incidents. Or worse, push people to abandon the incident declaration process altogether.

But there's a bigger picture: driving progressive reliability practices inside organizations. And that means working toward capturing necessary and accurate data from the moment an incident is declared. Ultimately, we believe that requiring a field or two upon incident declaration can play an essential role in improving an organization's overall maturity in incident management. If you're considering required fields as part of your process, we've got a few tips for ensuring that you can capture consistent incident data while promoting confidence and efficiency among responders.

See more about how we're using required fields internally at FireHydrant by checking out our recent blog https://firehydrant.com/blog/how-to-use-required-fields-for-incident-delcaration/ and our documentation https://support.firehydrant.com/hc/en-us/articles/8499733491348-Required-Fields-for-incident-Declaration

We significantly improved usability of FireHydrant’s Terraform Provider by adding sample code and logging, and fixing bugs for the following:

Resources

• firehydrant_environment
• firehydrant_functionality
• firehydrant_runbook
• firehydrant_severity
• firehydrant_team

Data sources

• firehydrant_environment
• firehydrant_functionality
• firehydrant_runbook
• firehydrant_runbook_action
• firehydrant_service
• firehydrant_services

Note that several breaking changes were made, including the planned deprecation of the services attribute in the resource firehydrant_functionality. There are also a number of breaking changes for the resource firehydrant_runbook. The steps attribute is now required, the steps config attribute is now a JSON string, and the type and severities attribute have been removed. In order to upgrade to 0.3.0, you will need to destroy your existing runbooks and recreate them after changing your configuration.

Additionally, the following new provider resources and data sources were added:

Resources

• firehydrant_incident_role
• firehydrant_priority
• firehydrant_service_dependency
• firehydrant_task_list

Data sources

• firehydrant_incident_role
• firehydrant_priority
• firehydrant_severity
• firehydrant_task_list

For more details see the FireHydrant Terraform Provider repo changelog.

Onboarding Updates

• We’ve all experienced the frustration of opening a new app or one we haven’t used recently, and not knowing where to start. To help improve onboarding for new and infrequent FireHydrant users, we’ve added an overview video, module descriptions, and links to knowledgebase articles.

Display the last update to specific status pages

• We will now display the last posted update to a status page incident on the Status Page tab in our updated command center. This should allow your incident communicator to quickly assess when the last update went out and what the contents of that update were. Keep your eyes open for more communication related improvements over the coming weeks.

Role assignment details in Slack

• We now include specific roles that people are assigned in the status message for an incident. Get up to speed with who is doing what right when joining an incident channel.

New Contextual Menu for Functionalities and Services

• Users can now select from a list of actions for any functionality in their functionalities table to make a change. No need to open a detail page individually to edit it or navigate to an incident and edit it to include a functionality.

Functionalities endpoint performance improvements

• Our functionalities endpoint has been slimmed down with the lite param and the removal of N+1 queries. This makes the endpoint usable by companies with more robust data loads. For example, one large organization's requests were reduced in time from >40s to just 1.4s!

Link directly to the service attribute you want to change

• When clicking on a link to change an aspect of a service, you are directed to that point in the service form. Previously, a user would land at the top of a long form, then search for the input they wanted to change.

Redirect sends users to the top of a page

• Previously, after completing a form and loading the next page, users arrived at the new page, scrolled down somewhere near the bottom. Really odd and confusing.

Tooltips on Services and Functionalities index tables

• Make life a little easier by adding a brief explanation of some of our column headings. Unobtrusive, but there if you need it.

Helper text in services and functionality forms

• Similar to our table tooltips, we’re offering some support while filling out the forms with some descriptive help text for tier, owner and responding teams inputs.

Bugs and Fixes

• Bug: Starred events in the new command center will update in real time as they are starred in Slack, no page refresh required.
• API Addition: The ticketing endpoint will now return timestamp data.

• Command Center has a refreshed look.
  • A new, cleaner interface for managing incidents is available today for all users and can be accessed in the Command Center UI by enabling "New Command Center" in the banner at the top of the page.
  • More easily navigate the timeline of events in your incident while easily accessing all of the metadata associated with the incident.
  • Services, Functionalities, and Environments now link to their pages in the command center to quickly navigate to their homepages.
• Functionalities Overview got a refresh as well.
  • With the latest features added to Functionalities, we cleaned up the overview to better showcase their new owning teams, impact status, and when they were last updated. These new display fields can be filtered on via the overview page. In addition, you can set labels and filter on them across your functionalities.
  • On a Functionality’s details page, you can click any label assigned. This will redirect you to the overview page and filter on that key:value pair to display all functionalities using these same values.
• Prepopulate new accounts with default incident types
  • When new accounts are created in FireHydrant, they will now come with incident types defined by our customer base. We’ve started with the seven most common incident types we’ve seen here at FireHydrant to ensure you can get up and running with FireHydrant as quickly as possible.
• Updated suggestion messages in Slack
  • When a service is attached to an incident, we will now display associated teams and functionalities to that service directly in Slack to quickly get the right people and data associated with your incident. We’ve also applied the same treatment to functionalities. Based on the work to update functionalities with associated teams, we can now show team and service relationships to functionalities inline in Slack when they are added to your incident.
• Fixes and Enhancements
  • 🔥 🚀 🌟 Emojis 😱are now supported in the incident timeline from Slack events.
  • Bugfix The Slack channel incorrectly displayed incident tickets.
  • Bugfix When a user navigates away from the timeline tab and comes back, sometimes it fails to fetch.

Updates to the Service Catalog

• Improved Functionalities view inside Service Catalog: Our Functionalities feature inside Service Catalog has a whole new look with lots of new features. The Functionalities overview, as well as detail pages, have new incident information views to quickly see what is impacted. This release also unlocks the ability for functionalities to do the following: Assign Owning & Responding Teams, Link third party Services, Auto-Alerting, new Related Service Tab, Labels, External Links, and other meta-data.
• Managing Service Editing Permissions by Owning Team: You can now limit access to editing or deleting services to specific teams and their members. This is so you can have confidence in your service catalog definitions remaining protected. To activate this permission feature, assign an Owning Team to a Service. To learn more see our Editing Permissions documentation.

Updates to the Slack App

• New slack command: /fh page functionality: Linking a functionality to a third party integration service will allow you to page this functionality via slack. Head into slack and type in /fh page functionality. This will populate a dropdown list of all functionalities with integration links configured. Select the functionality you would like to page. FireHydrant’s slack bot will fire off a page to the escalation policy linked to this functionality.
• Updated the Slack message for assigning a task list: We updated the incident channel and DM messages for adding a task-list to an incident and assigning it to a responder.
• Updated the /fh add task-list command modal: This change adds to improving and unifying our task list experience. We have rebuilt the tasks list modal and added command summaries to other parts of our task list experience, we wanted to do the same here.
• Updated the Role Assignment DM to use BlockKit: This release replaces the message that a user receives when being assigned to a role in an incident to match our new design styling.
• Updated the team suggestion message in the incident channel: If you add an impacted service to an incident, and that service has a responding (not just owning) team, we send a suggestion message about assigning said team to the incident.

Introducing our new Zendesk integration To more easily capture customer impact FireHydrant Enterprise tier users can now integrate with Zendesk Support. Our initial release allows Zendesk ticket linking to FireHydrant incidents. To get started see our Zendesk integration documentation. Stay tuned for enhancements coming next quarter, including capturing impacted customers and automating status updates back to support tickets.

Jira integration admin improvements

• We’ve made several changes to clarify the difference between global Jira integration authorization and account linking. Account linking is used to associate individual users to their corresponding Jira account for Jira ticket creator attribution. Under User > My Account, the Linked Account section now uses the verbiage Link.
• Global Jira integration reauthorization is required to reauthorize the Jira integration connection when Atlassian tokens are rotated. Only the person/account who set up the Jira integration can reauthorize the global integration. This person/account is now displayed on the Jira integration settings. If that person is logged in, they can now also reauthorize the global integration connection from that screen.
• Lastly, we’ve added a new Jira troubleshooting document that details self-service actions users can take to diagnose and correct common Jira fault scenarios.

Starting today, FireHydrant has a wholly redesigned Task experience for incident response.

• You can now predefine Task Lists before an incident to be attached via Runbook automation or manually throughout the incident. We’ve been using Task Lists internally for deprecating playbooks on cluster failovers, compliance tasks, incident commander responsibilities, and Slack reviews. Task lists ensure that your process is as seamless to use as possible.
• Task creation has been rebuilt from the ground up in Slack. All our commands have a fresh coat of paint to make it straightforward to create new tasks, add task lists to incidents, and assign those tasks during the incident. Messages for Tasks are more compact, and we’ve made the assignment DMs easier to take action on for assignees.
• The most significant update has focused on Task management in Slack. Now running /fh tasks will open a project management modal for all of your incident tasks. You can filter based on assignment, state, and task list while quickly being able to update the assignee and Task state without leaving that screen.

Starting today, incident response in both Slack and the Web has been completely redesigned and simplified. Let’s talk through some of the updates.

Guided workflows to help responders ramp

• Simplified incident declaration - speed up declaration, reduce fear in new responders, and stay flexible for power users with our new multi-step incident declaration.
• Updated status message - Get new responders and interested parties up to speed quickly with an updated /fh status command that is also sent as an ephemeral message to all new channel joiners.
• Contextual buttons - The status and new incident messages are now milestone aware to help responders take actions quickly based on the state of the incident.
• Updated Milestones - First Action is gone, replaced with a much easier to understand Investigating milestone. Additionally, we’ve added IDentified between Investigating and Mitigated.
• Slack Link reminders - Whenever unlinked users join an incident channel, FIreHydrant will send them a reminder to link their accounts
• Not and empty conditions for runbooks - Ensure process adherence by running runbook steps when roles, impact, and other incident metadata is missing.
• Incident Update Runbook step - Enforce incident state with Automatic escalations, milestone changes, and description updates are now all possible through runbooks to allow for even more flexibility for our power users.
• Notify channel step actions - Ensure consistent process, Incident messages now can prompt users to take actions through buttons, making it easier for custom process during the course of an incident.
• Slack Bookmarks - Reduce Noise with Bookmarks. Bookmarks allow for custom links and FH created information to be added to an incident channel, reducing noise in the channel.

Slack commands are more powerful and straightforward

• Start retro command - /fh start retro - Users can start the retro preparation immediately after resolving and can start the retro with proper information while the incident is still fresh.
• Updated Resolve incident command - /fh resolve - Post resolution updates to status pages and ensure that accidental resolutions happen less
• Updated assign team command - /fh assign team - Understand who you’re bringing into an incident when assigning a team
• Contextual Help menu - /fh help - Improving the design of this menu should make it easier for new responders to be less overwhelmed by the breadth of functionality in FireHydrant.

Simplified design to eliminate noise

• Reduced message noisiness - Shortening FH messages allows the most important information, the chat between responders to be more prominently displayed
• Threaded updates - By including updates to an incident in a single place, Slack users can scroll back on incident updates without having to scroll through the whole incident channel.

A redesigned Command Center adds Incident Response redundancy

• Command Center V2 - A completely redesigned incident command center in the web to allow users to respond to incidents in both the FireHydrant application and in Slack. Protect against Slack outages with chatops functionality and give your responders the ability to work wherever they are comfortable.

See more at our blog post at firehydrant.com/blog/redesigned-and-redundant-incident-response/

Incident Data CSV Export To address complex reporting and data science use cases, bulk incident data CSV export is now available to Enterprise users. After enabling the integration, bulk incident data will be exported daily to CSV file format and accessible via the FireHydrant API. For more details on configuration and data format see the user documentation.

Labels inside services now a clickable asset for search We decided to boost the power of labels inside services so users can easily search for shared key:value pairs across their catalog. Once a user adds a label to a service, they can now click on that label and be taken to their catalog overview with it automatically filtered. The overview will display all services that share these labels with a pre-injected query into the filter view.

This past week the team took some time to run a bug bash. Here's a high level overview of what we fixed, we closed almost 100 bugs over the past week.

Runbook fixes - Notify steps with buttons now can be properly removed, closed milestone state is no longer displayed, search works correctly on current milestone dropdown

Incident command fixes - Fixed issue with unassigned user roles displaying time incorrectly, incident creation time no longer changes when the started milestone changes, fixed an issue with incident CSVs not returning all incidents, internal status pages visits now return a proper authorization error, fixed help menu in slack, fixed broken links in the command center, service condition labels are now present in the pdf, CSS fixes on the status page as reported by StackOverflow.

Improved Service API

As the usage for Service Catalog has increased so has the need to take time to improve the scalability for retrieving service information across our product. The Service Catalog Team, with lead from Patick Vilhena, has improved the latency for returning service information for large catalogs with hundreds of services.

Feature - Added New Runbook Audit Logs endpoint for paid plan users. Enterprise customers can reach out- to their Customer Success Manager to enable this feature. Feature - Added API endpoint to create chat event types Feature - Bulk update events posted to the incident status message as a reply

Feature - Paid plan users can now automate Webex bridge creation through a new Webex integration Feature - Improved API for faster lookup of services from your service catalog Feature - Managing Runbook Editing Permissions on Owning Team - Users can now add Owning Teams to Runbooks to prevent edits from members outside their teams. Users with an Owner role will still have global editing permissions to avoid lockout Feature - Operational Readiness Templates available to quickly create service standards

Feature - Slack Improvments

1. Contextual buttons added to the Slack message to easily kick off a FireHydrant incident
2. Severity Description - When you choose a severity inside of Slack, we will update the help text under the field to include the severity description.
3. Fallback text for notifications - no more “this content cannot be displayed”
4. ‘/fh assign team’ - Assign Team Command opens detailed modal
5. ‘/fh start retro` command - Running this command will open a modal that will allow for an update of the incident metadata while kicking off a retrospective.

Feature - Team Assignment Updates

1. Filter incidents by assigned team using the API
2. Metrics on a team basis - We can now allow for filtering analytics data based on a team being assigned to an incident

Bug Fixed - Updating an incident’s impact will no longer post empty messages to Slack when there were no changes

Notice Retrospective Layout Updated to display incident description and customer impact prominently at the top of the retro

Retrospective Changes The design of retrospectives didn't lead users to fill out their data as they completed a retrospective correctly. The retrospective PDF displays the incident description and customer impact at the top left of the page and is usually the first thing that someone will read when looking at it. However, to edit those fields in a retro, you had to click an overflow menu to get to the form fields. We've changed the layout to ensure that these two fields display prominently at the top of the retro.

Operational Readiness Templates When it comes to defining a Readiness Checklist, organizations are looking for recommendations on how to get started on building their first checklist.

FireHydrant has built two template checklists for users to get started: Production Readiness Checklist & Security Compliance Checklist. Each list is a starting point for organizations to apply service standards during creation.

Feature - Users can now automate retrospective data export to Google Docs. Feature - FireHydrant now supports the creation of the most common Jira Cloud and Server epic ticket types and linking of FireHydrant incident data to Jira custom fields. Learn more here.

When an incident happens, most organizations have a way of identifying all affected services. The trouble is, it’s often a human-centered process that depends on the knowledge of key individuals or manually updated documentation. There might be a version in your alerting tool, a version in your corporate Wiki, and a different version still in your team’s head. This means that during incidents, responders spend valuable time determining the full impact that could be spent remediating core issues. With the addition of service dependencies to FireHydrant Service Catalog, responders can immediately identify the full scope of service and customer impact across an incident with a comprehensive view of downstream and upstream dependencies. And since Service Catalog documents service ownership, they can quickly pull in the needed responding teams.

Learn more at our blog post here

Feature - Users can define task lists for incidents, assign them to an incident as part of a runbook step, and add them to an active incident in slack Feature - Updated /fh help experience in an incident channel to include guidance for tasks and task lists Feature - Runbook Ownership now available to lock down Runbook editing to Owning Teams

Notice - Tasks will not be assigned tickets even if there is a default ticketing integration in the organization

Feature - FireHydrant is now available in MS Teams Feature - New Service Details Overview for Service Catalog Feature - Terraform 0.2.0 released Feature - Now a verified Terraform Provider Feature - New Free Tier experience for small teams released Feature - Free Tier Update: Can now set up a single project for Jira cloud Feature - We now support integration with multiple Shortcut projects Feature - We released new guidance for automating Jira multi-project configuration via the API. Feature - Added improved error handling for errors associated with Jira multi-project configuration.

Bug - Better messaging for Runbook Step access for Free Tier users Bug - Fixed a bug with the Jira multi-project configuration settings to allow new tickets to have a default status of "In Progress" Bug - Fixed a bug where FireHydrant status updates for linked action items were not propagating to linked Jira tickets Bug - Reordered the multi-project configuration UI to be more logical in field mapping flow. Incident ticket field mappings are now grouped, as are action item ticket field mappings Bug - Fixed a bug related to Zoom user assignment for schedule maintenance runbooks Bug - Fixed pagination to better support users with long lists of Jira projects

Feature - New Account Overview to show feature usage within your account. Feature - Task management is now available in Slack Feature - Incident tags are now sorted alphabetically in Slack

Bug - Fixed bug that was preventing some users from being able to save changes to a statuspage Bug - Fixed bug that caused some runbook steps to fail. Bug - Fixed bug that kept some users from being able to edit change events in a retro

Feature - New Operational Readiness Checklist feature for Service Catalog. Learn more here. Feature - Users can now export incident data into Confluence for collaborative activities, like incident reports and retrospectives. Get started by configuring your Confluence integration. Feature - We now support integration with multiple Jira Cloud and Jira On-Prem projects. This allows users to create incident tickets and action items in team-specific Jira projects directly from FireHydrant. This release also introduces a new integration-specific create incident ticket Runbook step. We ask all customers using the Jira integration to transition to their Runbooks to the new step, as we plan to deprecate the generic step at the end of June. Feature - Two new template variables are now supported, incident.last_note.body and incident.last_note.created_at to capture the most recent incident update and timestamp regardless of whether the update was generated by a single action or by bulk update. We plan to deprecate incident.last_update at the end of June 2022 and request that customers migrate over to the newer incident.last_note.body. Feature - Added Mean Time to Mitigation (MTTM) to our weekly summary email

Bug - Fixed alert ingestion for when an integration link is established between PagerDuty and a FireHydrant Service. Bug - Fixed runbook rules to prevent customers from creating a runbook rule that would not work with time constraints beyond the maximum time allowed for rule execution. Bug - Fixed bug that was preventing some customers from exporting incident and service data into a CSV Bug - Fixed bug to maintain consistent formatting of incident duration in our Overview and Analytics pages Bug - Fixed bug that was causing some customers to view updates on their internal incident status page after the incident was resolved Bug - Help icon in runbook conditions routes to the correct help documentation Bug - Fixed bug that was causing issues in weekly summary emails.

Feature - Users now more easily edit updates to status pages in the FH UI. Feature - Terraform Provider v0.1.4 released Feature - Enabled extra validation to prevent duplicate service names on creation Feature - Added additional access controls for private incidents Feature - Improved design for /fh help in Slack incident channels Feature - New Integration Links section has been added to Services to easily link and update alerting integrations to services. To test this out: 1. Select a service from the Service Overview page 2. Click 'Edit details' 3. Scroll down to the section titled 'Integration links'

Bug - Fixed ‘/fh page service’ command so that hyperlinks go to direct service and not overall Service Catalog. Bug - When configuring runbook steps, the help icon now directs users to the correct documentation. Bug - Fixed [runbook condition, "time since last incident note added"](https://support.firehydrant.io/hc/en-us/articles/360061144571-Communicating-with-teams-using-Runbook-steps "runbook condition, "time since last incident note added"") so that it only fires with the Slack command /fh add note, and not with /fh add update. Bug - Fixed liquid formatting template variable issue with the Email Notification Runbook Step Bug - Fixed Slack bug for VictorOps alerts that caused alerts to not go to their intended channel Bug - Fixed bug that prevented incident milestone times from updating Bug - Fixed Slack /fh link command bug that prevented role assignment Bug - Fixed bug that prevented auto-assigning of a linked user in Slack Bug - Fixed UI padding and spacing issues on Retrospective Configuration page in FH web. Bug - In the FireHydrant UI, addressed inconsistent experience that occurred when trying to assign a Team on the Overview page and none of the team members were assigned default roles. There are also clearer error messages calling out potential issues in this area. Bug - Fixed bug that allowed users to create runbooks with the same name Bug - Fixed incorrect error that occurred when using filter "retrospective completed" on the Incidents page in FH web Bug - Fixed bug that caused an incident status page to be blank Bug - Fixed bugs in weekly summary email related to labeling and filtering of email recipients

Feature - You can now create private incidents (beta) in FireHydrant. This is a great solution for security and compliance teams who need to collaborate with their engineering counterparts to resolve incidents. Learn more here. Feature - Added the ability to edit service catalog when using the /fh update command. Feature - Runbooks now display the user name and time of when it was last modified. Feature - Service catalog details can now be edited from the slack command /fh update. Feature - Roles can now be assigned after an incident is resolved, either from Slack or the UI. This is especially useful if you want to assign incident retrospective roles. Feature - New Incident Runbook View. Now you can easily access all executed Runbooks and dive deeper on each step. Feature - Services now include the power of incident analytics. Service owners can now get a quick overview of the reliability of their services over the past month Bug - Use of "Internal Status Page" is now consistent in both Slack and the UI. Before this fix, Slack showed "Private Status Page". Bug - Action items will always keep assignee info- even after they are marked 'done'. Bug - The naming of all teams is now unique. Duplicate teams names will be validated and prevented when trying to save.

Feature - New Runbook step: Auto-add Services Related To Functionality. Feature - When linking services to VictorOps and GitHub, their respective routing keys appear with the external link name. Feature - For clarity, new incident Slack modals contain a Description field instead of a Summary field. The description field does not have a character limit. Feature - Private Slack Channels are available and can be added by using Runbook steps Bug - Added the option to turn off paging of the default VictorOps escalation policy if there are no impacted services in an incident. Bug - When selecting Jira projects in slack, we improved the dropdown to show your default project as well as other projects you can select and not show blank. Bug - Enforced limiting the ability to add changes to the 'new note' and 'edit incident' actions; changes can only be added to only active incidents in the Slack UI. (This prevents users from adding details or notes to inactive incidents.) Bug - Fixed bug that was creating duplicate incidents in Slack.

Feature - Our new Service catalog feature has been released! Feel free to check out our new product release and to learn how to manage your services with your incident response. Check out this resource to get started Feature - VictorOps integration capabilities have been updated. If you use VictorOps, you can now more easily create and manage incidents from Runbook steps and VictorOps alerts Feature - ServiceNow integration is available for users to create incidents in ServiceNow from FireHydrant Runbook steps Bug - Fixed bug causing duplicate weekly summary emails from being sent to users Bug - Form error information is now displayed for PagerDuty and Opsgenie during setup instead of a 500 error (when API token is missing) Bug - Fixed bug that prevented users from removing a channel from the Channel for alerts Slack configuration

Bug - Saved searches now appear in the dropdown on page load; multiple saved searches can be updated and/or deleted without navigating away and back Bug - The /fh action items command now includes action items that do not contain a description Bug - The Slack setup page now redirects correctly in the README page Bug - Slack commands work regardless of use of upper or lower case

Feature - The new Incident Header provides better usability on the overview page of an Incident. The header now shows “Elapsed Time” under the incident title, as well an Edit button that opens a modal where you can view and edit important incident details. Additionally, you can update incident milestones and severity with a simple dropdown. You can also edit milestone times from within the header. This streamlines the incident response process by giving you access to multiple important functions from one central location.

Bug - Fixed a bug that was causing Zoom bridges to duplicate. When you declare an incident that has multiple Runbooks attached (and a step to create a Zoom bridge is included in more than one Runbook), only one Zoom bridge will be established for the incident. Bug - Fixed an issue that prevented action items that exceeded Slack’s character limit from displaying correctly in Slack. Now action items will be truncated in Slack, and users can view the full action item in the FireHydrant UI. Bug - The "New Incident" and "Edit Incident" Slack modals now list environments, functionalities, and services (as they appear in dropdown menus) in alphabetical order.

Feature - New metric card shows important Retrospective data like percentage of retros completed, min time to retro, max time to retro, and mean retro gap Bug - Fixed a bug that was causing google meet links to fail Bug - Fixed a bug that was causing a NameError in production for Runbooks

Bug - Fixed an issue that prevented publication of an incident event when a new Slack message with an attachment is created Bug - Fixed an issue that was preventing Google Meet integration from completeing successfully Bug - Fixed an issue that prevented the 'Cancel' and 'Save' buttons on the Retrospective Configuration page from scrolling with the rest of the controls on the page Bug - Fixed an issue that prevented the value in the "label value" field to include whitespace characters

Feature - Added the ability to properly assign service ownership from the "Teams" page Feature - Added the ability to search/filter for projects in the FireHydrant UI Feature - Added the option to configure multiple connections for the same integration Feature - New ConnectionStatuses API endpoint provides connection statuses about integrations Feature - Link to Notification settings page is available under the User menu

Bug - Fixed an issue with table borders in Safari Bug - Fixed Analytics view to exclude user involvements for archived incidents Bug - Changed imports page to a paginated list of importable objects, improving page performance Bug - "Customer impacted" details are now displayed on the incidents page

Feature - New Integration, Checkly Feature - New Integration, Google Meet Feature - Display active incidents on individual service page Feature - New Slack scope available for pinning initial incident message Feature - Display initial incident message to user joining Slack channel for the first time Feature - Add Tiers to services and runbook conditions

Bug - Alphabetize lists in dropdowns across app Bug - Improve service command search results Bug - Display all available PagerDuty services and Escalation Policies in dropdowns

Feature - Allow for internal status pages to be turned off Feature - Update Integrations page to make installing new integrations easier Feature - Added ability to associate reporter of FireHydrant action item to Jira Cloud reporter Feature - Allow associating owning team to a service Feature - Updated API documentation at developers.firehydrant.io Feature - Add incident tags to analytics filtering Feature - Display better user meta information on teams page

Bug - Properly handle new lines in markdown fields Bug - Allow Jira OnPrem credentials to be modified Bug - Fixed a bug where selecting custom date ranges in analytics would occasionally break Bug - Allow users to enter Zoom room first if they were not the incident opener

Feature - Customizable Slack Incident Modal - The new customizable Slack incident modal enables users to easily configure which fields they would like to appear in their modal, if they have requirements that differ from the default settings. Feature - Incident Tags - Incident Tags empower users to store more metadata on incidents, making it easier to query and get actionable insights compared to traditional labels. Now, with Incident Tags, users can tag incidents, filter by tags, and even automate Runbooks based on tag conditions.
Feature - Slack Service Command - The service command allows you to quickly query the metadata of a service from within Slack. Run /fh service to see it in action.

Feature - Added ability to reopen incidents from the web UI Feature - Added buttons to run commands from Slack help menu

Bug - Fix validation issue with Zoom runbook steps

Feature - Allow for star reactions in Slack to work when the incident is already resolved Feature - Invite all users to a channel in Slack if they already are assigned a role Feature - Add a new field on an incident for Customer Impact Feature - Update graphs on the overview page and improve analytics algorithms Feature - Add ability to configure what emoji causes a message to be starred in the FireHydrant timeline Feature - Add a response emoji when FireHydrant has successfully starred a timeline event in Slack (defaults to :fire: but can be changed) Feature - Added assigned roles to incident messages posted by FireHydrant

Bug - Fixed a bug where labels had incorrect validation requirements Bug - Fixed a bug where not all starred events populated on the PDF timeline Bug - Fixed a bug where selecting custom date ranges in analytics would occasionally break Bug - Fixed a bug where action item updates in Slack were in title case Bug - Fixed a bug where incident resolution times were incorrectly displayed on the incident table

Feature - We updated the onboarding experience for new accounts Feature - Using incident updates will now always post to the internal status page for an incident whether on Slack or the UI Feature - Added the ability to search in all runbook step configuration dropdowns Feature - You can now export incident data based on the current filter view as a CSV Feature - We added a new runbook step to assign a role in your incident. This can be assigned from a FireHydrant user, a user from an Opegenie or PagerDuty schedule, or the person who opened the incident

Bug - Fixed a bug where the analytics filter didn't stay in the header on scrolling Bug - Fixed a bug where infrastructure was not available to search in the condition UI Bug - Fixed a bug where starred events were not populating fully on page load Bug - Fixed a bug where incidents were still showing on status pages after they were archived Bug - Fixed a bug where password complexity requirements were not shown on the accept invitation page Bug - Fixed a bug where services were not correctly attaching to an incident if they were included on a linked Opsgenie alert

Change - Moved the bot users page to the Organization tab in the navigation

Runbook templates: An easier way to automate your processes Starting out with runbooks can be daunting, we've built a way to implement our best practices into a runbook that can be implemented in a single click. On top of this, there's now even more ways to attach runbooks to your incidents and a much easier way to test out the runbook that you're currently working on. Check it all out here.

Dig in deeper with filtering in analytics While our analytics platform provided some initial ways to examine your incident data, we always wanted a way to surface even more relevant data from your historical incidents. Now you can! With analytic filtering, you can take the same conditional evaluation from runbooks and filter your analytics based on that. Check out some initial ideas and learn more here.

Use additional variables in Runbooks for more customization We're always looking to make Runbooks more valuable. This month, we have made additional variables available in runbooks.

incident.last_update gives you the last note that was posted to the incident

incident.report_id gives you the id of the retrospective that is created so you can link to our generated pdf from any runbook step that takes template variables

New Integrations: Opsgenie & VictorOps

Now available - we've just released a first class Opsgenie integration. Now FireHydrant can automatically associate incidents to alerts, assign team members to on-call schedules, open incidents in Slack in a single click from new alerts, and more. Customers who've leveraged our alerting integrations have seen their mean time to respond drop by 75%! Learn more about what this integration can do here.

Not only do we support Opsgenie, we also support VictorOps as well! Now enjoy the ability to manage your response in FireHydrant no matter which alerting provider you use. Learn more on how to configure VictorOps here.

Templatize your incident types

Templatize your incident types Do you consistently have similar types of incidents? Do you wish that the correct infrastructure was always attached to the incidents that you consistently spin up? Using incident types you can now template all of the details, including runbooks, to include on an incident with only one selection. Learn more about how to implement types and get your team to open incidents faster and more reliably.

Use /fh and /incident

Let's face it, typing out /firehydrant in Slack every time you want to run a command takes too long. We have introduced two new commands that behave in the exact same way as /firehydrant but shorter! Now you can use /fh or /incident in Slack to interact with the bot.

ICYMI: Try conditions in FireHydrant Runbooks

FireHydrant Runbooks have consistently been the most powerful and popular way of implementing your automated incident response process. However, after receiving a lot of great feedback, we extended the ability to create automation beyond the start of your incident. With conditions, you can now add automation to any point of your incident response lifecycle.

You can find our announcement post here, but try out these 2 easy ways for you to get started: how to build automated stakeholder email notifications and how to archive your Slack channel after an incident has ended.

Get reporting how you want it: CSV export of analytics data

FireHydrant captures some of the most full featured incident metrics, but we know that these metrics are extremely valuable in other places outside of the FireHydrant platform. Now you can quickly export your quarterly SEV1 incident data as CSV to populate charts for your next board meeting or capacity planning. Visit Analytics to export some data.

A few little things to take note

We’re always making small, but mighty, improvements to the platform to increase how easy things are to use. This month we added support for searching for incidents based off of their number on the incidents list, dynamic title tags for easier history browsing, updates to the responders algorithm to more correctly show who interacted on an incident, and displaying who opened an incident on the incidents list. Go to the platform to check them out!

1. A new and improved analytics experience

After a lot of time in the lab, we’re excited to announce a new analytics experience. We now offer incident data covering Severities, Responders, and Infrastructure. Each category will return data around total time impacted, number of incidents, and mean time statistics. Click here to see your new analytics experience.

2. The Damage Report you’ve always wanted

Getting caught up on a long-running incident can be frustrating. Try out the new Slack command /firehydrant status to get an incident overview without scrolling back or bothering your team and link to common actions in a message only visible to you.

3. FireHydrant’s new Slack App Home

If you’ve ever wondered what the current state of incidents are across your org or wanted a quick rundown of multiple fires you’re fighting simultaneously, we’ve released the perfect feature for you. With our new Slack App home you can easily see the current state of incidents. Check it out by going to the Apps section in Slack.

4. Now easily archive incident channels

You’ve asked, we’ve answered. We’ve built a runbook step so you can automate archiving your incident channel after the completion of your retrospective. To enable the functionality all you need to do is add an ‘Archive Incident Channel’ step into your runbook and we handle the rest, automatically archiving your incident channel after you complete your retrospective.

ICYMI: We’ve got a Zoom Integration

Creating a conference bridge for an incident is one of the most common requests we have heard. Our new Zoom integration will automatically spin up a bridge and make it even easier for your response team to get on the same page.

Quickly Know Who’s On Call

Knowing who is on call is already difficult and now, when you're remote, knowing who is on call can present a whole other set of challenges. To help you keep track, we’ve built a new Slack command to easily see on-calls and escalation policies for any given service in your organization, inside of the tool that you are using most frequently: Slack. Try it out by running /firehydrant on-call.

Webhook Runbook Step

Integrating your incident response with your other applications has now become significantly easier. You can now leverage the automation of Runbooks to send webhooks based on the state of your incident.

New Slack Commands

Remembering all of FireHydrant's Slack commands can be challenging in the heat of an incident. By running /firehydrant you can now see a list of available commands and run them with one click.

Fire hydrants usually have a firehose hooked up, and boy do we have a firehose of updates for you. We've been focused on making FireHydrant simpler to use and more deeply integrated with existing workflows to make managing your complex systems easier. Let's dive in!

Status pages

Status pages are a common tool used in incident communication, by closely coupling our incident response platform to our status pages we have made keeping stakeholders and customers updated on your remediation easier than ever. By running your incident response through FireHydrant the manual work of keeping communications up to date are automated letting you focus on remediation. Read more about the updates Status Page Announcement.

Incident Page Redesign

We have simplified and redesigned the incident page to make it easier than ever to discover information and interact with your incident through our Incident Command Center. Read more about the updates Incident Page Redesign.

PagerDuty Integration Updates

In an effort to make it even faster to spin up an incident in FireHydrant we have upgraded our Alerting integrations to go from detected alert to opened incident in a single click on Slack. Read more about it PagerDuty Integration Updates.

Ticketing - Clubhouse

We're excited to announce that we are adding support for Clubhouse to our ticketing system. Now you can create an incident ticket that all action items from your incident and retrospective are easily tracked and assigned to. With the Clubhouse integration, tickets created in FireHydrant are created and synced to tickets in Clubhouse. Read more about Ticketing here.

Additional Incident types

As companies continue to expand the scope of what constitutes an incident FireHydrant is following suit. You can now designate incidents in FireHydrant as Unplanned Downtime, Scheduled Maintenance, or Gameday. By running these three types of incidents on FireHydrant you can leverage the automation of runbooks to ensure that your process adheres to your plans every single time.

Today we're releasing a vastly improved system for how you track work that stems from incidents. With our new ticketing functionality you are able to create actionable items for incidents as tasks or follow ups.

With this improvement, we're also excited to announce our Jira integration (For both Cloud and Server) have been updated to support the new ticket workflows.

FireHydrant now includes release notes within the application itself. We know that not all users of FireHydrant are subscribed to our mailing list, so this allows your team members to keep up-to-date about changes to our product. We’ll continue to publish product announcements on our blog and to the mailing list as well.

Anyone in Slack can open incidents now

Sometimes there isn't an alert. Instead you have the town locals knocking on your door with pitchforks. The best detection you have sometimes is your own team, because of that, now anyone can run /firehydrant new in Slack to kick off an incident.

Let us handle the severity for you

Who has time to pick the severity when opening an incident? We're also happy to go into the matrix, severity matrix that is. Customize your severity levels and pick the conditions that set them when an incident is opened. Less time deciphering a severity, more time putting out a fire.

Jira

We're also announcing our Beta Jira integration today as well! Whenever you create incidents and action items in FireHydrant, we'll send them to Jira as well so you can track it in everyone's favorite cloud hosted todo list.

Announcing Webhooks

If you want to build tools to communicate updates for deployments, it can be a ride on the struggle bus if you don't have a single source of truth.

Using FireHydrant's change log solves the problem of "source of truth", but until today there was no way to receive a notification about a change being deployed.

With our new webhooks feature, we've solved just that. Receive a HTTP request anytime a change event is created or updated to one or multiple endpoints that you get to configure!

Get hooked up

Keep your frequent searches for later

Using our change events filtering, you can find deploys to production for a certain service in the last hour with ease. Today we're also happy to announce that you can save these searches for later with our new Saved Search functionality. This is available or all users today on the change events page.

Today we're happy to announce our single sign on support and updates to our postmortem functionality.

Single Sign On Is Available

If you are using an SSO provider, you might be happy to hear we now support logging in with it! We have so many post-it notes with our passwords scattered around our office we had a fire hazard (forced pun intended).

It's easy to get started with it (we promise), just head to our Wiki to learn how!

Generate Postmortem PDFs

Along with moving postmortems into a single view (who likes multiple steps?), we're also happy to announce postmortems can now generate a PDF of your report! Not only that, users that participated in the report will also receive a shiny new email informing them about the published report too.