Proposal - PoC Pipeline templating


#1

Hi all,

Recently I’ve been working on a pipeline templating tool, uav, for use with concourse. Issues like this have been raised in the past, https://github.com/concourse/concourse/issues/1994 and https://github.com/concourse/concourse/issues/595 both propose similar ideas to what we’ve implemented here.

We’re looking for some feedback on this tool and the idea’s we’ve implemented. Should this proposal be accepted, we’re happy to properly re-write it in go, with the aim of it being merged into mainline fly.

The source code for it is here:

Looking forward to hearing everyone’s feedback :slight_smile:

Thanks,


#2

Hi dancrn,

we already attempted the same for our CI/CD solution where we put job definitions in job.erb templates to render our complete pipeline. At some point we had the feeling that we describe the job in the pipeline twice, once in the job.erb and once again in the pipeline.yml.erb.

When it comes to the reuse of jobs across different pipelines, we saw a growing demand in adding more and more dynamic variables into our job.erb templates to satisfy all demands. Eventually that lead us to a much more complicated pipeline.yml.erb code than before without saving too many lines of code.

I also see another challenge in this approach. Concourse is not very opinionated in how it approaches CI/CD, leaving a ton of freedom for us on how to implement our pipelines but adding complexity to the ecosystem on the other side. We need to take care that we are not putting the entry barrier for new people to high by adding to many abstraction layers to the Concourse system natively (e.g. by supporting the in the fly cli).

I think there are some easy things that can already make pipeline code much easier to maintain and overview:

  • Splitting resources, groups, resource_types, jobs into separate files that could be rendered to a pipeline aka sprucing them together
  • Using a template engine instead of raw .yml code for your pipeline (in our case ERB templates)
  • Adding next to the secrets file that is invoked with the -l flag another config file that describes flags which can be used in the pipeline.yml.erb to make the pipeline flexible (invoking the config file using he ERB template logic).

But altough we make extensive use of things like yml anchor tags and erb logic in our pipelines, we still have the feeling that everything is a bit “hacky” and the ultimate solution is still not found.

We should have something like a community conference meeting e.g. every two weeks which gives us a platform for an ongoing dialogue on Concourse best practices and how we can make pipelines scale better.

best,
D


#3

Hi gdenn,

Thanks for having a look at this.

With regard to your first concern, I’ve been careful to only allow for templating and parameter substitution. I too would be concerned at adding more complex constructs, like looping or conditionals into the syntax. Ideas like this would suggest that something is trying to be solved with pipeline configuration that is probably suited elsewhere. Again, the primary aim of this proposal is to reduce cognitive load of configuring a pipeline.

I agree - the configuration of pipelines should be kept simple. Indeed, these additions shouldn’t affect any pipelines configured in today’s pipeline schema. Additionally, I’d argue that this lowers the barrier for entry - having tasks configured and scripts defined in a separate git repo, which needs to be pulled as a resource on every job, feels more complicated that just create a single, final, output pipeline file.

With uav, you can already do this - it can take multiple pipeline definitions, even without the new syntax, and merge them into a single pipeline file. It also does some rudimentary checking of correctness with regard to uniqueness of job names and task names.

I also agree that perhaps yaml isn’t the best fit for this sort of templating approach. Honestly, I’d prefer a completely seperate syntax for this sort of thing, with its own parser, which can generate the final yaml file. I might look into this in my own time!

Our devops team and I would love to be involved in any active forums regarding concourse. We think it’s great, and we just want to make it better!

Thanks again for your input.


#4

Hi dancrn,
Where I work our main apps pipeline has been growing and getting complex as well and we’ve been toying with different ideas to make pipeline development easier. I experimented with the “Splitting resources, groups, resource_types, jobs into separate files that could be rendered to a pipeline aka sprucing them together” approach suggested @gdenn in the past but so far we’ve only adopted some basic templating w/spruce but still around a single/big pipeline.yml. I’d really prefer a more modular way of doing things though, and hopefully something built into Concourse instead of relying on more outside tooling that requires maintenance. I came across the same github issues in my searches on this as you referenced in addition to another project (https://github.com/JulzDiverse/aviator) by way of this post https://medium.com/@julian.skupnjak/create-a-concourse-pipeline-for-your-cloud-foundry-apps-with-ease-98ceaa055be7 seems interesting as well.

Cheers,
David


#5

hi drennalls,

you are right, splitting just the resources, jobs and groups in different files and merging them with spruce will not help to make the pipeline code scalable.

Currently we do it like this:
I use the Concourse native task.yml files to place all my CI-Tasks in a separate Github Repository. For us thats a way to share CI/CD logic with other departments in our company, so we avoid the maintainance of multiple automation systems.

The tasks repo is structured like this:

- README.md
- task1/
--- README.md (short description of what it does, Input / Output / Params)
--- task.yml (standard Concourse task file)
--- task.rb (ruby script that is used in the task.yml)
- task2/
--- README.md (short description of what it does, Input / Output / Params)
--- task.yml
--- task.rb
.
.
.
- spec
--- spec_helper.rb
--- task1_spec.rb (contains unit tests for task1's Ruby script)
--- task2_spec.rb

On the other hand we have another repo for pipelines:

- pipeline1/
--- README.md
--- partials/
----- _deploy_service_job.yml.erb
--- resources.yml.erb
--- groups.yml.erb
--- jobs.yml.erb
.
.
.

To keep our pipeline code DRY, we follow a similar approach like Ruby on Rails in the rendering of HTML views using partials. Partials can contain any part of a pipeline e.g. one or multiple jobs and receive through a Ruby binding a config attribute from the caller. These partials can be rendered in resources.yml.erb, groups.yml.erb, jobs.yml.erb or even nested in other partials.

To partials are invoked by calling the render_partialmethod:
<%= render_partial(path: "_deploy_service_job.yml.erb", object: @config %>

  • path is always the path to the *.yml.erb file relative to the partials folder
  • @config is the root configuration which is passed to the resources.yml.erb, groups.yml.erb and jobs.yml.erb.

Since we work with many different infrastructures, our pipelines need to be adjustable. Therefore many of our pipelines have configuration.yml files. Each file holds pipeline specific configuration parameters like flags and are accessible in resources.yml.erb, groups.yml.erb and jobs.yml.erb through @config.

The rendering of the ERB templates in the Pipeline is currently done by a a Ruby build script, but i am currently working on a little CLI Gem which shall do this.

Using a scripting language with an expressive template engine like Rub / ERB gives us the opportunity to not only split our pipeline in different code junks, but also simple features like for or while loops that the Ruby language provides us.

I think our approach is still far from perfect and i am happy that we have an active discussion here.

What do you think about sheduling a hangouts conference for the week of the 15. - 21. of October? This could be a great starting point for an ongoing discussion.

Maybe some of you are at CF Summit next week in Basel as well?

best,
D