Pipelines as code (part 2)…the API

This is a continuation of Pipelines as code…not text

There’s an art to writing frameworks: you have to balance helpfulness with flexibility.
I recently wrote a test framework to help migrate a large, slow, unusable regression suite from Rational Tester to a more manageable form. There were thousands of test cases and they were going to be transposed by a team of QA’s with minimal application knowledge and relatively limited coding skills. The ask was to have the framework supply a library of Cucumber steps that could be assembled to provoke and verify the desired scenarios, a DSL if you like.

At first it was envisioned that the Cucumber steps would be super helpful and take care of all the logic for building a valid request but it was immediately obvious to me that this would cause problems:
1 – Duplicating business logic in the test code, a maintenance burden
2 – It wasn’t much use for creating invalid requests for error testing.

This would be too much helpfulness and not enough flexibility. I ended up making lots of small, generic Cucumber steps that could be combined to build whatever request you wanted. Sure, the resulting scenarios were fairly verbose, but that made it clearer as to what the test was doing, avoided an avalanche of over-specialised Cucumber steps and the step implementations themselves were clean and simple to maintain. It’s important to treat test code as well as you treat production code (modular, maintainable, re-usable etc), it’s part of the overall system that gets your business ideas into customer’s hands.

---

So with that balance of helpfulness and flexibility in mind what should the Conveyor API look like?
Here’s a first look at the fluent interface I’ve put together

conveyor("Pipeline",
	stage("Build",
		job("Application Build",
			gitClone(REPOSITORY_URI),
			maven("clean install"),
			dummyTask("upload build to artifactory")
		),
		job("Configuration packaging",
			dummyTask("get config"),
			dummyTask("Upload config to artifactory")
		)
	)
).start();

If I’ve done my job right, it’ll be fairly clear what’s going on here, it uses similar concepts to GoCD.
Your conveyor is a list of stages.
Each stage contains a list of jobs.
Each job contains a list of tasks.
Each task performs a concrete action.

Each level is initialised with a name and a vararg list of sub-items for the next level down.
Static imports and varargs make this very clean when it comes to adding a variable number of sub-items to a level.
It’s pretty straightforward, just fancy nested lists, each with a start() method that runs it’s own list until you get to the tasks and actually do something useful.
(Memo to self: I might change the interface from start() to go() to avoid confusion with Threads)

---

Let’s take a look at the levels from top to bottom

Conveyor: This is a simple top-level component that holds a list of Stages to run.
When you start the conveyor it goes through each of it’s Stages and calls start().

Stage: This is another simple component holding a list of jobs to run.
When the stage gets started, it goes through each of it’s Jobs and calls start().
I foresee it being useful to specify stages to run in parallel but that’s for later.

Job: Each job creates a temporary workspace then passes it into each of it’s Tasks to share.
I think these Tasks will always be run serially and share a workspace but too early to say and can always be changed.

Task: This is where the work actually happens. By implementing the task interface it should be possible for anyone to create a task to do whatever they want and call it from Conveyor.

I’ve implemented a few concrete tasks so far.
dummyTask – prints out it’s name
gitClone – is a wrapper for JGit clone
maven – is a wrapper for MavenCLI from the maven embedder

But with just these we’re able to check out a project and build it, and not a Jenkins server in sight.
Check out the code and try it for yourself if you like, let me know how you get on.

---

What’s next? I think some more task implementations to continue towards being able to do a full deploy.

• Artifactory task for uploading builds to a repository. I expect this is how jobs and stages will cooperate but it raises the question of how they should communicate versions or id’s. Currently thinking a global audit/manifest/report object that gets passed through each Task interface.
• JRuby task will let you run ruby scripts. There are a lot of chef scripts out there implemented in ruby. Would make it easier to migrate to Conveyor if you don’t have to re-implement all them. Hopefully this approach will allow you to inspect, debug and step-through the code.