Testing Shell Commands with the Crystal CLI#testing-shell-commands-with-the-crystal-cli

FireHydrant uses aCrystal-based CLI for some developer actions, called fhd (FireHydrant developers). Previously, we might have distributed workflows among new developers by having them copy/paste or clone scripts down to their machines--but Crystal lets us encapsulate shared tooling in a compiled binary. This way, we have a CLI that developers can install quickly, and that works seamlessly with our other tools. While it’s not statically linked like Go's binary, Crystal provides a much more approachable syntax for a team working with Ruby day-in and day-out.

One of the big things we wanted to do was wrap some common kubectl commands. That would mean shelling out and using kubectl on the user's device, which involves a lot of methods that look something like this:

def kubectl_exec(context : String, command : String)
	Process.run(
		command: "kubectl",
		args: [
			"--context", context,
			"--namespace", "laddertruck",
			command.split(" "),
		]
	)
end

Unfortunately, this approach is difficult to test, as you're stuck running the commands directly. I've really grown accustomed to Ruby's ability to stub basically anything (which is both a blessing and a curse). Not so much in Crystal, with its types and static compilation requirements. I did some thinking and reflected on a fairly common practice I've seen in Elixir. It goes like this:

defmodule CoolThing do
	def find_user(name) do
		user_module.find(name)
	end

	defp user_module() do
		# Assume User module exists
		Application.get_env(:my_app, :user_module, User)
	end
end

# in config/test.exs

import Config

config :my_app, user_module: MockUser

This effectively lets you swap in a different module, which you can stub out as needed at test time. This is also especially useful when you need to test things involving the current time. Crystal isn't as dynamic as Ruby, but it does let you type on mixin modules and open up modules and classes. I came up with the following module that could be mixed in to the built-in Process and, eventually, a test class called Fhd::MockProcessRunner:

module Fhd::ProcessRunnerInterface
	abstract def run(
		command : String,
		args = nil,
		env : Env = nil,
		clear_env : Bool = false,
		shell : Bool = false,
		input : Stdio = Redirect::Close,
		output : Stdio = Redirect::Close,
		error : Stdio = Redirect::Close,
		chdir : String? = nil
	) : Process::Status
	abstract def find_executable(name : Path | String, path : String? = ENV["PATH"]?, pwd : Path | String = Dir.current) : String?
end

This is an explicit copy-paste of the source for Process.run, including type signatures. They needed to match exactly for Crystal to understand that Process satisfies the interface without actually overwriting any of the functions or requiring new ones. We also use abstract functions to avoid overwriting the existing functions, and instead just tell the compiler "hey, the Process class meets my needs."

class Process
	extend Fhd::ProcessRunnerInterface
end

class Fhd::MockRunnerInterface
	include Fhd::ProcessRunnerInterface

	# We need these aliases to match the shortened form in the main Process class
	alias Stdio = Process::Redirect | IO
	alias Redirect = Process::Redirect
	alias Env = Nil | Hash(String, Nil) | Hash(String, String?) | Hash(String, String)

	# I'm not putting the method signatures here for brevity, but let's assume they match.
	def run(); end
	def find_executable(); end
end

It's unique that we need to extend in Process but we can include for Fhd::MockRunnerInterface. This means that when testing, we use an instance of the class defined here, but in actual running we'll use the Process class itself. This results in the following code for the main program:

module Fhd
	def self.process_runner : ProcessRunnerInterface
		@@process_runner ||= Process
	end

	def self.process_runner=(runner : ProcessRunnerInterface)
		@@process_runner = runner
	end
end

# and we change our kubectl_exec from above to be:

def kubectl_exec(context : String, command : String)
	Fhd.process_runner.run(
		command: "kubectl",
		args: [
			"--context", context,
			"--namespace", "laddertruck",
			command.split(“ “),
		]
	)
end

Actual Testing#actual-testing

Great! We've got some skeletons and something that works for our main program. Now let’s write some specs. The first step is to add a method to the spec/spec_helper.cr file that lets us pass in a process runner to be used for a specific test:

def with_mocked_process(new_runner : Fhd::MockProcessRunner, &block)
	old_runner = Fhd.process_runner
	Fhd.process_runner = new_runner

	Log.capture { |logs| yield logs }

	new_runner.ensure_all_called!
	Fhd.process_runner = old_runner
end

What’s more interesting is the body of the Fhd::MockProcessRunner class. I'll post it in its entirety, then address each bit and describe the purpose it serves.

class Fhd::MockProcessRunner
	include Fhd::ProcessRunnerInterface

	alias Stdio = Process::Redirect | IO
	alias Redirect = Process::Redirect
	alias Env = Nil | Hash(String, Nil) | Hash(String, String?) | Hash(String, String)

	struct RunExpectation
		property command, args, env, clear_env, shell, chdir

		property input : Stdio | String
		property output : Stdio | String
		property error : Stdio | String

		def initialize(@command : String, @args : Array(String)?, @env : Env, @clear_env : Bool, @shell : Bool, input : Stdio, output : Stdio, error : Stdio, @chdir : String?)
			@input = mutate_io_arg(input)
			@output = mutate_io_arg(output)
			@error = mutate_io_arg(error)
		end

		def mutate_io_arg(io_arg : Stdio)
			if io_arg.is_a?(Fhd::MockProcessRunner::Redirect)
				io_arg
			else
				"instance_of #{io_arg.class}"
			end
		end
	end

	struct FindExecutableExpectation
		property name, path, pwd

		def initialize(@name : Path | String, @path : String?, @pwd : Path | String)
			@called = false
		end
	end

	def initialize
		@run_expectations = Hash(RunExpectation, Process::Status).new
		@find_executable_expectations = Hash(FindExecutableExpectation, String?).new
	end

	def ensure_all_called!
		errors = Array(String).new

		if @run_expectations.any?
			errors << "\\n=== run calls:\\n#{@run_expectations.keys.join("\\n")}" \\

		end

		if @find_executable_expectations.any?
			errors << "\\n=== find_executable calls:\\n#{@find_executable_expectations.keys.join("\\n")}"
		end

		if errors.any?
			fail "Not all expectations hit, the following were missed:\\n#{errors.join("\\n")}"
		end
	end

	def set_run_expectation(command : String, args = nil, env : Env = nil, clear_env : Bool = false, shell : Bool = false, input : Stdio = Redirect::Close, output : Stdio = Redirect::Close, error : Stdio = Redirect::Close, chdir : String? = nil, return_value : Process::Status = Process::Status.new(0)) : Nil
		run_args = RunExpectation.new(
			command: command,
			args: args,
			env: env,
			clear_env: clear_env,
			shell: shell,
			input: input,
			output: output,
			error: error,
			chdir: chdir
		)
		@run_expectations[run_args] = return_value
	end

	def set_find_executable_expectation(name : Path | String, path : String? = ENV["PATH"]?, pwd : Path | String = Dir.current, return_value : String? = nil) : Nil
		executable_args = FindExecutableExpectation.new(name, path, pwd)
		@find_executable_expectations[executable_args] = return_value
	end

	#Meet the obligations of our module type
	def run(command : String, args = nil, env : Env = nil, clear_env : Bool = false, shell : Bool = false, input : Stdio = Redirect::Close, output : Stdio = Redirect::Close, error : Stdio = Redirect::Close, chdir : String | ::Nil = nil) : Process::Status
		run_args = RunExpectation.new(
			command: command,
			args: args,
			env: env,
			clear_env: clear_env,
			shell: shell,
			input: input,
			output: output,
			error: error,
			chdir: chdir
		)

		@run_expectations.delete(run_args) { fail "run called with unknown args: #{run_args}\\n\\nExpected one of: #{@run_expectations.keys.join(", ")}" }
	end

	#Meet the obligations of our module type
	def find_executable(name : Path | String, path : String? = ENV["PATH"]?, pwd : Path | String = Dir.current) : String?
		executable_args = FindExecutableExpectation.new(name, path, pwd)

		@find_executable_expectations.delete(executable_args) { fail "find_executable called with unknown args: #{executable_args}\\n\\nExpected one of: #{@find_executable_expectations.keys.join(", ")}" }
	end
end

The Expectation Structs#the-expectation-structs

We have a pair of structs, FindExecutableExpectation and RunExpectation. These objects may look hefty, but they really just contain an instance variable for each argument in the appropriate function definition, available to find_executable and run (respectively). These are set to the values you'd expect to pass in to them during your actual method calls, which are handled by MockProcessRunner in a method we'll see shortly. These structs are, for most purposes, hidden from the developers writing tests.

One unique point in the RunExpectation struct is:

def mutate_io_arg(io_arg : Stdio)
	if io_arg.is_a?(Fhd::MockProcessRunner::Redirect)
		io_arg
	else
		"instance_of #{io_arg.class}"
	end
end

This exists because we have some methods in the CLI that set output to an IO::Memory buffer and only print it if there's an error executing the command. However, that means a new instance is instantiated each time, which we can't match on. Instead, this mutates those to something like RSpec's instance_of(class) in our use case.

Setting Expectations#setting-expectations

Let’s use set_find_executable_expectation as an example (the run version is much longer, but it's the same idea):

def set_find_executable_expectation(name : Path | String, path : String? = ENV["PATH"]?, pwd : Path | String = Dir.current, return_value : String? = nil) : Nil
	executable_args = FindExecutableExpectation.new(name, path, pwd)
	@find_executable_expectations[executable_args] = return_value
end

The set_<method>_expectation methods match the function signature of their respective methods, with one exception. Their last argument is a return_value, typed to match what the actual method signature in Fhd::ProcessRunner (which comes from Process) dictates. So an example using find_executable might look something like this:

describe "#ensure_kubectl!" do
	it "returns true if the executable exists" do
		runner = Fhd::MockProcessRunner.new
		runner.set_find_executable_expectation("kubectl", return_value: "/usr/bin/kubectl")

		with_mocked_process(runner) do
			result = MyClass.new(["-e fake"]).ensure_kubectl!
			result.should eq(true)
		end
	end
end

Because the methods have the same defaults as the real thing, a user needs to specify only the arguments which differ from the default -- just like using the actual Process.

Ensuring Only Stubbed Calls#ensuring-only-stubbed-calls

This section ensures that the runner is only called with objects it knows about. Currently, it builds an Expectation struct and uses that as a Hash key with the value being the return value. The downside is that a given set of Process.run arguments can only be used once in a spec, but so far we haven't had any problems with that.

def find_executable(name : Path | String, path : String? = ENV["PATH"]?, pwd : Path | String = Dir.current) : String?
	executable_args = FindExecutableExpectation.new(name, path, pwd)

	@find_executable_expectations.delete(executable_args) { fail "find_executable called with unknown args: #{executable_args}\\n\\nExpected one of: #{@find_executable_expectations.keys.join(", ")}" }
end

The code builds an Expectation with the arguments given the same way they would be when you set up your tests. This takes advantage of Crystal structs getting a #hash method for free, calculated from all the instance variable values. If the arguments passed in don't match those previously given as part of set_run_expectation, we fail the spec and list out available argument options that existed.

Ensuring All Expectations Called#ensuring-all-expectations-called

The final piece ensures that all expected calls were hit. The Fhd::MockProcessRunner struct provides a method that confirms both expectation hashes are empty:

def ensure_all_called!
	errors = Array(String).new

	if @run_expectations.any?
		errors << "\\n=== run calls:\\n#{@run_expectations.keys.join("\\n")}" \\

	end

	if @find_executable_expectations.any?
		errors << "\\n=== find_executable calls:\\n#{@find_executable_expectations.keys.join("\\n")}"
	end

	if errors.any?
		fail "Not all expectations hit, the following were missed:\\n#{errors.join("\\n")}"
	end
end

We call this method at the end of our helper function from spec_helper.cr, which handles setting Fhd.process_runner for us. This way, we can guarantee that all expected Process.run calls were made.

Summary#summary

We wanted to convert some of our Bash functions into a CLI with testing, but that proved more complicated than expected. This was our way of being able to stub out expectations of a specific class (Process), but the practice can probably be expanded. It may even be possible to generalize this using macros and then share it with other Crystal developers. It was also a really fun learning process: figuring out how to convince the type system that "no no, I promise this is what I want." I look forward to us building and growing this CLI even more!