Testing Shell Commands with the Crystal CLI

FireHydrant uses a Crystal-based CLI for some developer actions, called fhd (FireHydrant developers). Previously, we might have distributed workflows among new developers by having them copy/paste or clone scripts down to their machines--but Crystal lets us encapsulate shared tooling in a compiled binary. This way, we have a CLI that developers can install quickly, and that works seamlessly with our other tools. While it’s not statically linked like Go's binary, Crystal provides a much more approachable syntax for a team working with Ruby day-in and day-out.

One of the big things we wanted to do was wrap some common kubectl commands. That would mean shelling out and using kubectl on the user's device, which involves a lot of methods that look something like this:

def kubectl_exec(context : String, command : String)
  Process.run(
    command: "kubectl",
    args: [
      "--context", context,
      "--namespace", "laddertruck",
      command.split(" "),
    ]
  )
end

Unfortunately, this approach is difficult to test, as you're stuck running the commands directly. I've really grown accustomed to Ruby's ability to stub basically anything (which is both a blessing and a curse). Not so much in Crystal, with its types and static compilation requirements. I did some thinking and reflected on a fairly common practice I've seen in Elixir. It goes like this:

defmodule CoolThing do
  def find_user(name) do
    user_module.find(name)
  end

  defp user_module() do
    # Assume User module exists
    Application.get_env(:my_app, :user_module, User)
  end
end

# in config/test.exs

import Config

config :my_app, user_module: MockUser

This effectively lets you swap in a different module, which you can stub out as needed at test time. This is also especially useful when you need to test things involving the current time. Crystal isn't as dynamic as Ruby, but it does let you type on mixin modules and open up modules and classes. I came up with the following module that could be mixed in to the built-in Process and, eventually, a test class called Fhd::MockProcessRunner:

module Fhd::ProcessRunnerInterface
  abstract def run(
    command : String,
    args = nil,
    env : Env = nil,
    clear_env : Bool = false,
    shell : Bool = false,
    input : Stdio = Redirect::Close,
    output : Stdio = Redirect::Close,
    error : Stdio = Redirect::Close,
    chdir : String? = nil
  ) : Process::Status
  abstract def find_executable(name : Path | String, path : String? = ENV["PATH"]?, pwd : Path | String = Dir.current) : String?
end

This is an explicit copy-paste of the source for Process.run, including type signatures. They needed to match exactly for Crystal to understand that Process satisfies the interface without actually overwriting any of the functions or requiring new ones. We also use abstract functions to avoid overwriting the existing functions, and instead just tell the compiler "hey, the Process class meets my needs."

class Process
  extend Fhd::ProcessRunnerInterface
end

class Fhd::MockRunnerInterface
  include Fhd::ProcessRunnerInterface

  # We need these aliases to match the shortened form in the main Process class
  alias Stdio = Process::Redirect | IO
  alias Redirect = Process::Redirect
  alias Env = Nil | Hash(String, Nil) | Hash(String, String?) | Hash(String, String)

  # I'm not putting the method signatures here for brevity, but let's assume they match.
  def run(); end
  def find_executable(); end
end

It's unique that we need to extend in Process but we can include for Fhd::MockRunnerInterface. This means that when testing, we use an instance of the class defined here, but in actual running we'll use the Process class itself. This results in the following code for the main program:

module Fhd
  def self.process_runner : ProcessRunnerInterface
    @@process_runner ||= Process
  end

  def self.process_runner=(runner : ProcessRunnerInterface)
    @@process_runner = runner
  end
end

# and we change our kubectl_exec from above to be:

def kubectl_exec(context : String, command : String)
  Fhd.process_runner.run(
    command: "kubectl",
    args: [
      "--context", context,
      "--namespace", "laddertruck",
      command.split(“ “),
    ]
  )
end

Actual Testing

Great! We've got some skeletons and something that works for our main program. Now let’s  write some specs. The first step is to add a method to the spec/spec_helper.cr file that lets us pass in a process runner to be used for a specific test:

def with_mocked_process(new_runner : Fhd::MockProcessRunner, &block)
  old_runner = Fhd.process_runner
  Fhd.process_runner = new_runner

  Log.capture { |logs| yield logs }

  new_runner.ensure_all_called!
  Fhd.process_runner = old_runner
end

What’s more interesting is the body of the Fhd::MockProcessRunner class. I'll post it in its entirety, then address each bit and describe the purpose it serves.

class Fhd::MockProcessRunner
  include Fhd::ProcessRunnerInterface

  alias Stdio = Process::Redirect | IO
  alias Redirect = Process::Redirect
  alias Env = Nil | Hash(String, Nil) | Hash(String, String?) | Hash(String, String)

  struct RunExpectation
    property command, args, env, clear_env, shell, chdir

    property input : Stdio | String
    property output : Stdio | String
    property error : Stdio | String

    def initialize(@command : String, @args : Array(String)?, @env : Env, @clear_env : Bool, @shell : Bool, input : Stdio, output : Stdio, error : Stdio, @chdir : String?)
      @input = mutate_io_arg(input)
      @output = mutate_io_arg(output)
      @error = mutate_io_arg(error)
    end

    def mutate_io_arg(io_arg : Stdio)
      if io_arg.is_a?(Fhd::MockProcessRunner::Redirect)
        io_arg
      else
        "instance_of #{io_arg.class}"
      end
    end
  end

  struct FindExecutableExpectation
    property name, path, pwd

    def initialize(@name : Path | String, @path : String?, @pwd : Path | String)
      @called = false
    end
  end

  def initialize
    @run_expectations = Hash(RunExpectation, Process::Status).new
    @find_executable_expectations = Hash(FindExecutableExpectation, String?).new
  end

  def ensure_all_called!
    errors = Array(String).new

    if @run_expectations.any?
      errors << "\\n=== run calls:\\n#{@run_expectations.keys.join("\\n")}" \\

    end

    if @find_executable_expectations.any?
      errors << "\\n=== find_executable calls:\\n#{@find_executable_expectations.keys.join("\\n")}"
    end

    if errors.any?
      fail "Not all expectations hit, the following were missed:\\n#{errors.join("\\n")}"
    end
  end

  def set_run_expectation(command : String, args = nil, env : Env = nil, clear_env : Bool = false, shell : Bool = false, input : Stdio = Redirect::Close, output : Stdio = Redirect::Close, error : Stdio = Redirect::Close, chdir : String? = nil, return_value : Process::Status = Process::Status.new(0)) : Nil
    run_args = RunExpectation.new(
      command: command,
      args: args,
      env: env,
      clear_env: clear_env,
      shell: shell,
      input: input,
      output: output,
      error: error,
      chdir: chdir
    )
    @run_expectations[run_args] = return_value
  end

  def set_find_executable_expectation(name : Path | String, path : String? = ENV["PATH"]?, pwd : Path | String = Dir.current, return_value : String? = nil) : Nil
    executable_args = FindExecutableExpectation.new(name, path, pwd)
    @find_executable_expectations[executable_args] = return_value
  end

  #Meet the obligations of our module type
  def run(command : String, args = nil, env : Env = nil, clear_env : Bool = false, shell : Bool = false, input : Stdio = Redirect::Close, output : Stdio = Redirect::Close, error : Stdio = Redirect::Close, chdir : String | ::Nil = nil) : Process::Status
    run_args = RunExpectation.new(
      command: command,
      args: args,
      env: env,
      clear_env: clear_env,
      shell: shell,
      input: input,
      output: output,
      error: error,
      chdir: chdir
    )

    @run_expectations.delete(run_args) { fail "run called with unknown args: #{run_args}\\n\\nExpected one of: #{@run_expectations.keys.join(", ")}" }
  end

  #Meet the obligations of our module type
  def find_executable(name : Path | String, path : String? = ENV["PATH"]?, pwd : Path | String = Dir.current) : String?
    executable_args = FindExecutableExpectation.new(name, path, pwd)

    @find_executable_expectations.delete(executable_args) { fail "find_executable called with unknown args: #{executable_args}\\n\\nExpected one of: #{@find_executable_expectations.keys.join(", ")}" }
  end
end

The Expectation Structs

We have a pair of structs, FindExecutableExpectation and RunExpectation. These objects may look hefty, but they really just contain an instance variable for each argument in the appropriate function definition, available to find_executable and run (respectively). These are set to the values you'd expect to pass in to them during your actual method calls, which are handled by MockProcessRunner in a method we'll see shortly. These structs are, for most purposes, hidden from the developers writing tests.

One unique point in the RunExpectation struct is:

def mutate_io_arg(io_arg : Stdio)
  if io_arg.is_a?(Fhd::MockProcessRunner::Redirect)
    io_arg
  else
    "instance_of #{io_arg.class}"
  end
end

This exists because we have some methods in the CLI that set output to an IO::Memory buffer and only print it if there's an error executing the command. However, that means a new instance is instantiated each time, which we can't match on. Instead, this mutates those to something like RSpec's instance_of(class) in our use case.

Setting Expectations

Let’s use set_find_executable_expectation as an example (the run version is much longer, but it's the same idea):

def set_find_executable_expectation(name : Path | String, path : String? = ENV["PATH"]?, pwd : Path | String = Dir.current, return_value : String? = nil) : Nil
  executable_args = FindExecutableExpectation.new(name, path, pwd)
  @find_executable_expectations[executable_args] = return_value
end

The set_<method>_expectation methods match the function signature of their respective methods, with one exception. Their last argument is a return_value, typed to match what the actual method signature in Fhd::ProcessRunner (which comes from Process) dictates. So an example using find_executable might look something like this:

describe "#ensure_kubectl!" do
  it "returns true if the executable exists" do
    runner = Fhd::MockProcessRunner.new
    runner.set_find_executable_expectation("kubectl", return_value: "/usr/bin/kubectl")

    with_mocked_process(runner) do
      result = MyClass.new(["-e fake"]).ensure_kubectl!
      result.should eq(true)
    end
  end
end

Because the methods have the same defaults as the real thing, a user needs to specify only the arguments which differ from the default -- just like using the actual Process.

Ensuring Only Stubbed Calls

This section ensures that the runner is only called with objects it knows about. Currently, it builds an Expectation struct and uses that as a Hash key with the value being the return value. The downside is that a given set of Process.run arguments can only be used once in a spec, but so far we haven't had any problems with that.

def find_executable(name : Path | String, path : String? = ENV["PATH"]?, pwd : Path | String = Dir.current) : String?
  executable_args = FindExecutableExpectation.new(name, path, pwd)

  @find_executable_expectations.delete(executable_args) { fail "find_executable called with unknown args: #{executable_args}\\n\\nExpected one of: #{@find_executable_expectations.keys.join(", ")}" }
end

The code builds an Expectation with the arguments given the same way they would be when you set up your tests. This takes advantage of Crystal structs getting a #hash method for free, calculated from all the instance variable values. If the arguments passed in don't match those previously given as part of set_run_expectation, we fail the spec and list out available argument options that existed.

Ensuring All Expectations Called

The final piece ensures that all expected calls were hit. The Fhd::MockProcessRunner struct provides a method that confirms both expectation hashes are empty:

def ensure_all_called!
  errors = Array(String).new

  if @run_expectations.any?
    errors << "\\n=== run calls:\\n#{@run_expectations.keys.join("\\n")}" \\

  end

  if @find_executable_expectations.any?
    errors << "\\n=== find_executable calls:\\n#{@find_executable_expectations.keys.join("\\n")}"
  end

  if errors.any?
    fail "Not all expectations hit, the following were missed:\\n#{errors.join("\\n")}"
  end
end

We call this method at the end of our helper function from spec_helper.cr, which handles setting Fhd.process_runner for us. This way, we can guarantee that all expected Process.run calls were made.

Summary

We wanted to convert some of our Bash functions into a CLI with testing, but that proved more complicated than expected. This was our way of being able to stub out expectations of a specific class (Process), but the practice can probably be expanded. It may even be possible to generalize this using macros and then share it with other Crystal developers. It was also a really fun learning process: figuring out how to convince the type system that "no no, I promise this is what I want." I look forward to us building and growing this CLI even more!