CI/CD

  8 minute read  

Some of these date back to older versions but efforts are made to keep the most important - sometimes :)

Comparision of CI/CD tools

Research of available solutions

After some research I found below tools. I group them by categories in columns:

name	paid	open source	self hosted	cloud hosted
jenkin-x	0	1	1	0
tekton	0	1	1	0
jenkins	0	1	1	0
gitlabCI	0	1	1	0
goCD	0	1	1	0
bazel	0	1	1	0
argoCD	0	1	1	0
spinnaker	0	1	1	0
buildBot	0	1	1	0
Travis	0	0	0	1
buddy	1	0	1	1
circleCI	1	0	1	1
TeamCity	1	0	1	1
CodeShip	1	0	0	1
azureDevOps	1	0	0	1
Bamboo	1	0	1	0

First for recognition goes only open source and free (at least in our usage model) tools.

Closer look on choosen tools

name	paid	open source	self hosted	cloud hosted	comment
jenkins-x	0	1	1	0
tekton	0	1	1	0
jenkins	0	1	1	0
gitlabCi	0	1	1	0	requires use GitLab
goCD	0	1	1	0
argoCD	0	1	1	0	CD tool requie other CI tool
bazel	0	1	1	0	this is build engine not a build server
spinnaker	0	1	1	0	mostly used for CD purposes
buildBot	0	1	1	0	looks worse then previous tools
Travis	0/1	0	0	1	In our usage model we will have to pay

After closer look I consider this tools:

• goCD
• jenkins-x
• tekton
• jenkins
• argoCD - this is CD tools so it's not compared in table below
• spinnaker - wasn't tested because it is CD tools and we need also CI tool

Comparision

Run server on kubernetes

gocd: easily installed by helm chart, requires to be accesible from outside cluster if we want to access UI. Can be run on Linux systems also

jenkins: can be easily started on any cluster

jenkins-x: hard to set up on running cluster. I created new kubernetes cluster by their tool which generally is ok - but in my vision it will be good to use it on LambdaStack cluster (eat your own dog food vs drink your own champane). Many (probably all) services works based on DNS names so also I have to use public domain (use mine personal)

tekton: easily started on LambdaStack cluster.

Accesses

gocd: , OAuth, LDAP or internal database

jenkins: OIDC, LDAP, internal, etc.

jenkins-x: Jenkins X uses Role-Based Access Control (RBAC) policies to control access to its various resources

tekton: For building purposes there is small service which webhooks can connect and there predined pipeline is starting. For browsing purposes dashboard has no restrictions - it's open for everybody - this could be restricted by HAProxy or nginx. Only things you can do in dashbord is re-run pipeline or remove historical builds. Nothing more can be done.

Pipeline as a Code

gocd: possible and looks ok, pipeline code can be in different repository

jenkins: possible and looks ok

jenkins-x: possible looks ok (Tekton)

tekton: pipelines are CRD so can be only as a code

Build in pods

gocd: Elastic agent concepts. Can create many groups (probably on different clusters - not tested yet) and assigned them to proper pipelines

jenkins: plugin for building in kubernetes

jenkins-x: building in pods in cluster jenkins-x is installed. Possible to install many jenkins-x servers (according to documentation per each team in different namespace). Able to run in multi cluster mode

tekton: building in cluster easily. Not possible to build on different server - but I didn't any sence in that use case. Possible to deploy on other kubernetes service.

Secrets

gocd: Plugins for secrets from: hashicorp vault, kubernetes secrets, file based

jenkins: plugins for many options: hashicorp vault, kubernetes secrets, internal secrets, etc

jenkins-x: Providers for secrets from: hashicorp vault, kubernetes secrets

tekton: Use secrets from kubernetes so everything what is inside kubernetes can be read

Environment varaibles:

gocd: multiple level of variables: environment, pipeline, stage, job

jenkins: environment variables can be overriden

jenkins-x: Didn't find any information but expect it will not be worst than in gocd

tekton: You can read env variables from any config map so this is kind of overriding.

Plugins

gocd: not big number of plugins (but is this really bad?) but very of them really usefull (LDAP, running in pods, vault, k8s secrets, docker registry, push to S3, slack notification, etc)

jenkins: many plugins. But if there is too much of them they start making serious issues. Each plugin has different quality and each can breake the server and has its own security issues so we have to be very careful with them.

jenkins-x: plugins are called app. There are few of them and this app are helm charts. Jenkins-x uses embeded nexus, chartmuseum and monocular services. I don't know if the is option to get rid of them.

tekton: tekton itself is kind of plugin for building. You can create whatever you want in different pod and get it.

Personal conclusion

gocd:

• This looks like really good CI/CD central server which can be use by many teams.
• Really mature application. Older release on github from Nov 2014. According to wiki first release in 2007.
• very intuitive
• Working really good in kubernetes
• Good granuality of permission.
• Good documentation
• Small amount of help in Internet (compare to jenkins)
• Small community

GoCD can be easily set up for our organizations. Adding new customers should not be big deal. Working with is very intuitive - old school concept of CICD.

jenkins:

• Production ready
• The most search CI/CD tool in google - so almost every case is describe somwhere
• Very simple
• Working very good in kubernetes
• After using it for some time pipelines are getting bigger and harder to maintain
• Good granuality of permission
• XML configuration for many plugins
• Big amount of information in Internet
• Big community

The most popular CI/CD tool. Small and simple. You can do everything as a code or by GUI - which is not good because it's temptation to fix it right now and then probably do not put it to repository. A lot of plugins which and each of them is single point of failure. Hard to configure some plugin as a code - but still possible.

jenkins-x:

• There is new sheriff in town - new way of maintainig CICD server
• New application still under heavy development (don't know what exactly but number of commits is really big)
• New concept of CICD, a lot of magic doing under the hood, GitOps and ChatOps
• Designed to work inside oif kubernetes
• Still don't know how to manage permissions
• Big community (CDFoundation is under Linux Foundation)

Jenkins-x is definetly new sheriff in town. But to enable it in big existing organization with new way of CICD process requires changing the way of thinking about all process. So it's really hot topic, but is it ok for us to pay that price.

tekton:

• New concept of CI - serverless.
• Tekton is young (first release 20 Feb 2019).
• Is a part of jenkins-x so it's simpler when you starting playing with it and still you can configure everything as in jenkins-x by yourself.
• Easy to install in LambdaStack cluster - kubernetes CRD
• Easy to install triggers which allow to build when request is comming.
• It should be separate namespace for every team. Builds will be running in one cluster using the same hosts.
• No permission to dashboard. It has to be resolve by properly configure HAProxy or nginx in front of dashboard. Dashboard is running as kubernetes service.
• Big community.
• Smal but good enough help regarding tekton itself. Under the hood it's kubernetes so you can configure it as you want.

Comparing it previous solutions jenkins-x is using tekton. So it has less features then jenkins-x - and thanks to that is simpler - but by deafult I was not able to configure really usefull feature building on push. There is such possibility by running tekton triggers which is realy simple. This project is under CDFoundation and has a big community which is really good. My personal choice.

Another concept CI and CD tool

Use separate tools for Continious Integration and Continious Deployment. In this concept I recognized Tekton for building and ArgoCD for delivery purposes.

ArgoCD

In ArgoCD you can easily deploy one of your applications described as kubernetes resources into one of your kubernetes clusters. In that case recommended option is to have two repos one for code and one for configuration. Thanks to that you can easily separate code from configuration. It also works with one repo where you keep code and configuration in one repo.

When Argo detect changes in configuration it runs new configuration on cluster. It's simple like that.

User management

Possible to use: local users, SSO with Bundled Dex OIDC provider, SSO with Existing OIDC provider

Secrets

• Bitnami Sealed Secrets
• Godaddy Kubernetes External Secrets
• Hashicorp Vault
• Banzai Cloud Bank-Vaults
• Helm Secrets
• Kustomize secret generator plugins
• aws-secret-operator
• KSOPS

Conclusion

ArgoCD looks very good if you have a really big number of clusters you are managing. Thanks to that you can deploy whatever you want wherever you need. But this is needed for really for big scale.