Pipelines as code…not text

This is a follow on to my post on pipelines in Breaking down barriers

It also bugs me that the pipelines can only really run on a server. That makes some sense in terms of security, concurrency, resources and moat building but not when it comes to constructing or modifying the behaviour of the pipeline itself. I think there’s a tendency to delegate too much functionality to these build servers so you end up with pipelines dependent on a mess of plugins and difficult to understand

If the pipeline is genuinely code then you should be able to enjoy all the good stuff you can do with code that makes life easier in an IDE: check it out, run a copy, set break-points, inspect, step through, code-completion, maybe even… test it 🤩

The more I think about it, the more I really like the idea of being able to specify and run my entire pipeline as code. Jenkins and GoCD are okay but they’re more pipelines as text.

It’ll be a simple java executable and consequently there’s absolutely nothing stopping you running it on Jenkins as part of a secured CI/CD environment taking advantage of the things Jenkins does well: polling repo’s, queueing, concurrency, agents etc.

I’m really tempted to fork the GoCD project and see if I can add the ability to specify the pipelines in Java and run them locally. There’s already so much CI/CD functionality implemented in Java, it’s just hidden on a server and only configurable via text files. I also like how their concepts are laser focussed on CD.

But the simplest thing is probably to do an MVP from scratch. I’ll try writing a small proof-of-concept that lets you define and execute a pipeline as Java code. Let’s start with git and maven functionality and take it from there.

I’ll call it Conveyor because it’s like Pipeline except you have better visibility into what’s going on inside. Github project is here with just a few initial commits. I’ll post more as I update the framework.