Testing Shell Commands with the Crystal CLI#testing-shell-commands-with-the-crystal-cli

FireHydrant uses aCrystal-based CLI for some developer actions, called fhd (FireHydrant developers). Previously, we might have distributed workflows among new developers by having them copy/paste or clone scripts down to their machines--but Crystal lets us encapsulate shared tooling in a compiled binary. This way, we have a CLI that developers can install quickly, and that works seamlessly with our other tools. While it’s not statically linked like Go's binary, Crystal provides a much more approachable syntax for a team working with Ruby day-in and day-out.

One of the big things we wanted to do was wrap some common kubectl commands. That would mean shelling out and using kubectl on the user's device, which involves a lot of methods that look something like this:

Unfortunately, this approach is difficult to test, as you're stuck running the commands directly. I've really grown accustomed to Ruby's ability to stub basically anything (which is both a blessing and a curse). Not so much in Crystal, with its types and static compilation requirements. I did some thinking and reflected on a fairly common practice I've seen in Elixir. It goes like this:

This effectively lets you swap in a different module, which you can stub out as needed at test time. This is also especially useful when you need to test things involving the current time. Crystal isn't as dynamic as Ruby, but it does let you type on mixin modules and open up modules and classes. I came up with the following module that could be mixed in to the built-in Process and, eventually, a test class called Fhd::MockProcessRunner:

This is an explicit copy-paste of the source for Process.run, including type signatures. They needed to match exactly for Crystal to understand that Process satisfies the interface without actually overwriting any of the functions or requiring new ones. We also use abstract functions to avoid overwriting the existing functions, and instead just tell the compiler "hey, the Process class meets my needs."

It's unique that we need to extend in Process but we can include for Fhd::MockRunnerInterface. This means that when testing, we use an instance of the class defined here, but in actual running we'll use the Process class itself. This results in the following code for the main program:

Actual Testing#actual-testing

Great! We've got some skeletons and something that works for our main program. Now let’s write some specs. The first step is to add a method to the spec/spec_helper.cr file that lets us pass in a process runner to be used for a specific test:

What’s more interesting is the body of the Fhd::MockProcessRunner class. I'll post it in its entirety, then address each bit and describe the purpose it serves.

The Expectation Structs#the-expectation-structs

We have a pair of structs, FindExecutableExpectation and RunExpectation. These objects may look hefty, but they really just contain an instance variable for each argument in the appropriate function definition, available to find_executable and run (respectively). These are set to the values you'd expect to pass in to them during your actual method calls, which are handled by MockProcessRunner in a method we'll see shortly. These structs are, for most purposes, hidden from the developers writing tests.

One unique point in the RunExpectation struct is:

This exists because we have some methods in the CLI that set output to an IO::Memory buffer and only print it if there's an error executing the command. However, that means a new instance is instantiated each time, which we can't match on. Instead, this mutates those to something like RSpec's instance_of(class) in our use case.

Setting Expectations#setting-expectations

Let’s use set_find_executable_expectation as an example (the run version is much longer, but it's the same idea):

The set_<method>_expectation methods match the function signature of their respective methods, with one exception. Their last argument is a return_value, typed to match what the actual method signature in Fhd::ProcessRunner (which comes from Process) dictates. So an example using find_executable might look something like this:

Because the methods have the same defaults as the real thing, a user needs to specify only the arguments which differ from the default -- just like using the actual Process.

Ensuring Only Stubbed Calls#ensuring-only-stubbed-calls

This section ensures that the runner is only called with objects it knows about. Currently, it builds an Expectation struct and uses that as a Hash key with the value being the return value. The downside is that a given set of Process.run arguments can only be used once in a spec, but so far we haven't had any problems with that.

The code builds an Expectation with the arguments given the same way they would be when you set up your tests. This takes advantage of Crystal structs getting a #hash method for free, calculated from all the instance variable values. If the arguments passed in don't match those previously given as part of set_run_expectation, we fail the spec and list out available argument options that existed.

Ensuring All Expectations Called#ensuring-all-expectations-called

The final piece ensures that all expected calls were hit. The Fhd::MockProcessRunner struct provides a method that confirms both expectation hashes are empty:

We call this method at the end of our helper function from spec_helper.cr, which handles setting Fhd.process_runner for us. This way, we can guarantee that all expected Process.run calls were made.

Summary#summary

We wanted to convert some of our Bash functions into a CLI with testing, but that proved more complicated than expected. This was our way of being able to stub out expectations of a specific class (Process), but the practice can probably be expanded. It may even be possible to generalize this using macros and then share it with other Crystal developers. It was also a really fun learning process: figuring out how to convince the type system that "no no, I promise this is what I want." I look forward to us building and growing this CLI even more!