HashiConf Digital Wrap-up, June 2020

The first HashiConf Digital event was held on June 22-24, online. The event was meant to be HashiConf Amsterdam, but the team pivoted and moved it online because of COVID-19. My employer FireHydrant was a sponsor, and I was happy to have a chance to attend. The event was very well organized, and that’s even more impressive given that the team had to shift it online.

One of the biggest surprises for me was that HashiCorp not only moved the event online but built a platform to host it. When I heard they had done that, I have to admit that I scratched my head a little. I tend to come up on the buy side of the build vs. buy arguments, and it wasn’t clear to me why a company that makes infra software would be spending engineering resources on an event platform. But as I thought about it more, it made more sense. First, the company closed a big round of funding in March, so it’s not hurting for resources. And I’m sure the folks making these decisions understood that events moving online isn’t going to end when there’s a COVID-19 vaccine. Virtual conferences are likely to be much more of a trend going forward, so building the tool you want to run them is a better investment than it might sound like.

And they did a great job with the platform.

Besides the schedule, the platform also featured a bunch of on-demand video content to help people prepare for the conference and a place to sign up to host and attend meetups during the event. It was all very well done. I made a few suggestions to the HashiConf Twitter account during the conference, and they seemed to be working to improve things in real time.

I’ll provide links to some of the videos that have been posted on HashiCorp’s web site. The rest will be getting posted to their YouTube account soon.

Day 1

The schedule was pretty painful for me. Since it was supposed to happen in Amsterdam, the talks were held in the afternoon CEST. That meant starting at 4 AM Pacific time where I’m located. I knew getting up at 3:30 AM for three straight days was going to be hard.

The next digital event in October will be held on a US schedule.

I made it up for the first day, though, and got pretty hype when I saw this tweet from James Governor of Redmonk:

James is exceptionally well informed, so I expected something big.

The opening keynote (view video here) started with Armon Dadgar, co-founder of HashiCorp. Armon mentioned that Hashiconf had pivoted from a regional event in Amsterdam to an online event with over 8000 registered. He announced the Terraform 0.13 beta and talked through some of the improvements to Terraform that are coming, like being able to use count and for_each across modules.

Armon also spent some time talking about Nomad, and emphasizing the company’s continued commitment to it. I haven’t used Nomad in production, but I’ve spoken with some folks from the Nomad team, and I think it has some strong selling points. It’s easier to manage than Kubernetes, and it’s integrated with the other HashiCorp tools. I’ve heard that one of the primary use cases for Nomad is running batch jobs, which makes a lot of sense.

Next up was HashiCorp co-founder and CTO Mitchell Hashimoto, who had the big news I was waiting for. Mitchell announced the HashiCorp Cloud Platform (HCP), a new platform for people building cloud infrastructure. He started off talking about how HashiCorp was working to enable cloud workflows and multi-cloud. This is messaging that was repeated some during the conference, so it seems like they feel it’s very important. The first HashiCorp product launching on HCP is Consul, but others will follow.

I’ve been following HashiCorp since the beginning (I was a Vagrant user before the company existed). I think it’s been a struggle for the company to put up SaaS versions of their software because the open source versions provide so much for free. Someone agreed with me.

I’ll be watching to see how this all progresses. I’m especially interested to see what happens with Terraform, as the company has put a lot of energy into Terraform Cloud.

The rest of the day was broken up into shorter talk slots on the main track, and an extra breakout track. The videos for the talks are available on the HashiConf Digital site still, but they will be moving to the company’s YouTube account soon. I’ll add links to this article when that happens.

The next talk I viewed was Getting Started with Managed Service Mesh on AWS with Alex Jauch and Cody De Arkland. This was the real first look into what HCP can do, and it has some pretty important features.

There was more focus on multi-cloud.

Much of this talk was a great demo by Cody. I recommend watching it if you’re interested in HCP Consul, or if you like an excellent technical demo. It was very well done. This was the point where I realized the talks for the conference were prerecorded, as Cody liked one of my live tweets during the demo. It did look like he had done the demo all in one take, though, and he confirmed that later.

The next presentation I watched was The Hitchhiker’s Guide to Terraform Your Infrastructure by Fernanda Martins from L1nda. The talk was a recap of a lot of the things she and her team learned when Terraforming their infra. It had a lot of great advice, like using modules from the community when you can because they likely have more tests. Fernanda also mentioned some tools her team had used.

This would be a great talk to watch if you are newer to Terraform or have had workflow struggles.

I attended the first panel of the day next. The panels were unscripted presentations where the panelists answered questions from the audience and the event’s MCs, @devopsdom and @devops_rob. Domi and Rob were great hosts and helped make the conference a lot of fun.

The HashiCorp folks have put a lot of effort for several years now into making their tools work with Kubernetes. One of the struggles people had was installing Vault Enterprise with Helm, as the module was written for the open source version.

The last talk of the day on the main track was one of the ones I was looking forward to the most, Kristin Laemmert talking about The Path to Terraform 1.0 (view video here). I wasn’t the only person who was stoked for it.

To be honest, I’m one of the folks who kind of shakes my head at the amount of time it takes some of the HashiCorp tools to make it to a 1.0 release. People have been using Terraform in production for years, and it’s pretty weird to me that it’s not well past 1.0 by now. Kristen mentioned that this is intentional on HashiCorp’s part, that one of the things they want to see in a tool before 1.0 is that it’s been used in production for some time.

I see a couple of problems with that point of view. First, some shops are pretty conservative, and people may run into political issues trying to get approval to use a pre 1.0 tool. Also, it feels to me like this practice lets the vendor off of the hook a bit too much. I’m not sure that’s HashiCorp’s intention doing it, but by delaying the 1.0 release, it seems like they send the signal that users are taking more risk onto themselves by choosing to use it. I’m a big HashiCorp fanboy in a lot of ways, but this one practice has bothered me for years. I wish they’d revisit it.

The good news is that the Terraform team is moving towards a 1.0 release, or at least figuring out what that would look like. It didn’t sound like it’s right around the corner (we know 0.13 is just going into beta), but the team is evaluating what needs to happen to get there. Some of the things they’d like to see in place are changes for the engineers.

Kristen is a fantastic speaker, something I saw people mentioning on Twitter. I enjoyed this talk.

That was the final talk for day one. I hosted a meetup after it to talk about incident response. The way the meetups worked in the HashiConf Digital platform is that you signed up and entered a URL for your video conferencing meeting. Our meeting was a small group, but we had a good discussion. I was glad to see the meetup option offered as an additional way for people to connect.

Day Two

There was not an opening keynote for day two. We jumped right into the main track and breakout talks. The first talk I caught was Managing Policy as Code With Terraform and Sentinel by Kyle Ruddy. If you’re not familiar with Sentinel, it’s HashiCorp’s framework for managing policies, and it’s available in the enterprise versions of their products. I was at HashiConf a few years ago when they announced Sentinel, and I thought that it was a great idea and something that would encourage teams to pay for their products. Audits can be challenging and time-consuming, and being able to show auditors that you have a tool to enforce your policies would be a huge help.

Kyle started off showing off Terraform Cloud, which I hadn’t seen for a while, and talked about using Sentinel with Terraform. One of the examples he had was using Sentinel to do cost checking on AWS instances, which I thought was very smart.

His other example used a Sentinel policy to prevent people from deploying on Fridays. That one made me chuckle as deploying on Fridays has been the topic of many flamey Twitter threads. My feeling is that those decisions are about what your team needs and can support, but the idea that you can enforce those decisions as policies is great.

As I was tweeting about Kyle’s talk, one of my Twitter friends mentioned that their org couldn’t afford Sentinel and uses another tool to do static Terraform code analysis instead:

I had heard of Checkov but haven’t used it. It looks pretty rad, though.

I also learned that you can use CIS benchmarks with Sentinel.

I really enjoyed Kyle’s talk. There were lots of demos, and it was all presented very well.

Next up on the main track was the Life of a Packet Through Consul Service Mesh with Christopher Puhl.

Christopher filled the talk with a lot of great technical details, but it moved pretty fast and I wasn’t able to live tweet it well. If you are interested in using service mesh with Consul, you should give it a watch, though.

I took a break during the next talk slot. I tweeted a bit about why I think that’s important to do if you need a break. I think it’s even more true with virtual events.

I really enjoyed the next session I caught, called “Oh No! I Deleted My Vault Secret!” by Lucy Davinheart. Lucy based the talk on a real situation that came up in their shop, and it was great to hear them talk through the process of getting the secret back, step by step. Lesson number one was not to rely on replication alone.

But restoring backup data is problematic too.

Lucy’s team ended up having to spin up a new Vault environment to restore the backup to, and they were able to get to the key eventually. Lucy left the audience with some lessons learned.

Up next was the Nomad Demystified panel with Nick Ethier and James Rasell. Nomad is probably the least known of the HashiCorp products, and I learned some things from this Q&A. The big discovery for me was that you can run workloads besides containers with Nomad. Someone on Twitter pointed me to the driver docs:

The last talk of the day was one of the ones I’d been looking forward to the most, a fireside chat with Kelsey Hightower (view video here), hosted by Anubhav Mishra and Nic Jackson. I’ve known Kelsey for a few years, and also consider Mishra and Nic friends, so I was excited to see them in more of a Q&A setting.

The conversation covered Kelsey’s early experiences with computing, and his first jobs outside of it.

Kelsey was a shift manager at a McDonald’s at fifteen, and he ran his own IT business.

Kelsey also managed comedians, which was something I didn’t know about. He was very self-motivated. He later went to work at a Google data center (in his first stint with Google), where he had a chance to learn more about things like hardware and networking.

At one point, Kelsey addressed something very important:

I’ve wondered about this a lot. As Kelsey mentioned in his KubeCon San Diego keynote, people don’t treat him like an underrepresented person in tech. His reputation in the tech space precludes that. But tech is a bubble, and whatever celebrity he has in it doesn’t protect him from having run-ins with police that could turn violent. Driving While Black is real, and I can’t imagine the cognitive load that black people in America carry around daily.

One of my favorite parts of the interview was when Kelsey talked about the need to keep learning.

I found the interview moving and I wasn’t alone there.

Thanks to everyone involved for making it happen.

Day Three

The final day of HashiConf Digital was a bit shorter than the first two, and I appreciated that. I was more than ready for a nap after three straight days of being up by 4 AM. I started the day off watching Andy Manoske speak about Protecting Workflows and Secrets. Andy is a Principal Product Manager for Cryptography and Security, and he dug into a lot of cool things about Vault.

Andy started off talking about the challenges security teams face. Andy’s team tries to think about things like adversaries do, and he pointed out that Defense in Depth means more sensitive data like credentials to store. And more business has moved online due to COVID-9.

Something I didn’t know about was Vault Enterprise’s Advanced Data Protection feature, which can be used to secure external secrets.

With Advanced Data Protection, the users and applications access tokens instead of the underlying data.

The idea is to push attackers towards solving hard mathematical problems.

I haven’t touched Vault Enterprise, and this was all news to me. It was a very cool talk if you are interested in Vault and cryptography.

Next up, I watched one of the customer talks, Protecting the Team and the Product With Vault by Chris Storey and Arun Jayanth. Both of the speakers work for the Department for Work and Pensions, which I believe is in the UK. Since they are dealing with people’s money, they are very concerned with security.

Their story was pretty familiar to me: they stored encrypted secrets in Git. I was on a team in the 2000s that did that as well, in a private Git repo we hosted, and it was pretty hard to deal with. The pattern is to encrypt files to everyone’s GPG keys on the team, but then everyone has access to all the data, and you have to decrypt and re-encrypt the files as people leave and join the team. It was pretty painful. Chris and Arun wanted to make it easier to manage secrets without manually running scripts to do these tasks.

By implementing Vault, they were able to give development teams ownership over their secrets, which is a much better pattern than having a centralized Ops/SRE team own them. By making managing secrets self-service, they improved the quality of life for the engineering teams.

I just caught a bit of Brendan Burns talking about HCS on Azure.

The final presentation of the day was the closing keynote, which featured a distributed demo from the HashiCorp Community team. I have many friends who work there, and it was great getting to see them on camera for a bit. The idea of having them all pitch in for a demo was clever and fun.

The result of the demo was generating a HashiCorp Coin, that you could use to vote on which charities HashiCorp would donate to.

I really enjoyed HashiConf Digital, and I’m looking forward to the next event in October, which will be in US time. The event was very well organized, and I should give one more shoutout to Domi and Rob for their work as MCs. They did a great job keeping the event moving.

After seeing the presentations, the two big things I’ll be keeping my eye on are: Terraform and HCP. I’m excited to see what happens with Terraform 0.13, and also the eventual 1.0 release. I’m also very interested to know how the HCP Consul launch goes, and what happens as they move their other services to HCP. This is a very exciting time for HashiCorp, and I expect the company to keep delivering very relevant and interesting infrastructure tools.