In 2019 I discovered a collection of 40+ microservices that all shared a similar Jenkinsfile. All the generic boilerplate to build and tag Docker images, upload artifacts to S3, and send notifications to Slack were being copy/pasted multiple times across all the microservices that were sharing this repo. You could manually change logic in one location (like changing the notification channel) and have to manually update every other location (in every microservice) where this logic was defined. This created an ongoing problem of drift and manual labour to keep everything in sync and working properly.
I removed the reusable parts of the Jenkinsfiles and created a Jenkins shared library Groovy pipeline. A shared library pipeline has made the maintenance of our shared library Groovy pipelines far less time-consuming because it takes what we previously had as 40+ microservice Jenkinsfiles and turns them into re-usable, easily testable steps that can be included by each team with a single line. If you’re still dealing with the inefficiency and repeatability of using declarative pipelines, here’s how I did it.
Quick Summary:
- Centralize all common pipeline steps into a single repository for the shared library
- Use
vars/for global custom step functions, andsrc/to store Groovy class files - Version your library and tag it, so you don’t break existing branches that are using previous versions of your library.
- Test your Groovy code with Jenkins-Spock before you push to production.
- Get your security approval for your script as early as possible to avoid mysterious pipeline failures.
Building a Modular CI/CD Pipeline Architecture
The Problem with Repetitive Jenkinsfiles
A common library approach for importing pipeline declarative Jenkinsfiles was not something we used back in our early days. All of our pipeline Jenkinsfiles started as one big fat Jenkinsfile with the same pipeline stages. As time went on, each project started to add their own unique variations — one team added retry logic while another team forgot to change the base image. Using copy/paste for this type of logic is not a scalable solution nor does it ensure consistency.A modular CI/CD pipeline architecture would have prevented issues stemming from the shared nature of the build logic and application code as well as the requirement to duplicate code across all repositories. The root cause was discovered when a security issue (CVE) forced an update to the Dockerfile on every repository. It was clear at this point that we needed to develop a centralized repository for the build logic to create a single source of truth.
How Global Shared Libraries Solve Scale
The global shared library is stored in its own Git repository, allowing Jenkins to pull from it as needed. Any pipeline that uses this library has access to its contents. Therefore, the extensive build process using a 200-line Docker file can now be replaced by a single line of code (e.g. myDockerBuild("service-name")). This streamlined process not only reduces the amount of typing, but it eliminates the possibility of breakage due to an exclusion from a build standard.
By moving the build logic into a shared library, you are able to build your CI/CD pipeline in a modular way that can scale without chaos as various teams are responsible for the Jenkinsfile that runs the pipeline while the shared library is controlled centrally, allowing you to update, version, and release enhancements to the shared library without needing to modify any application repositories.
My Final Decision: After testing out Jenkins Pipeline Templates (using extends) I concluded that the flexibility provided through shared libraries was much greater than what I was getting with Jenkins Pipeline Templates. Shared libraries allow you to create Groovy classes that interact with simple scripting steps and enable the use of unit testing. This was a major factor in my decision to pursue the use of shared libraries.
Prerequisites for a Jenkins Shared Library Groovy Pipeline
In order to use Jenkins Shared Libraries with your Groovy Pipeline, you will require the following: a Jenkins Controller running with a few plugins; a Git Repository containing your library’s code; and a little bit of Configuration to define additional options globally. Once this is completed, you should be able to reference your shared library automatically from any pipeline.
Required Plugins and Git Repository Setup
You need to install the plugin that provides this functionality to Jenkins.
Go to Manage Jenkins > Plugins > Available and search for the Pipeline: Shared Groovy Libraries plugin. Check the box next to it and click Install without restart. If you are running a very recent version of Jenkins, the shared libraries plugin is already included, but it’s always a good idea to check.
I normally create a git repository under our organization’s CI-tools folder. In the root directory of the repository you will need the following folder structure:
- vars/
- src/
- Optionally – resources/
You cannot add the folders above inside another folder; Jenkins expects the libraries to be in the root directory.
You will not need to create a webhook or a Jenkinsfile in this repository; you only need to push the initial files and you are ready to go.
Configuring the Global @Library Annotation Jenkins
You need to tell Jenkins where it can find this library. To do that, navigate to Manage Jenkins > Configure System. Next, scroll down to the Global Pipeline Libraries area.
Then, click the Add button. You will give your library a name – let’s say “my-shared-lib” and then set the default version (for a development library, you may use “master” or a specific tag like “v1.0”). In Retrieval Methods, select Modern SCM and choose Git. Then enter the repository URL into the field box. If the repository is private, you will need to provide Jenkins with the credentials for accessing the repository. Leave the Library Path field blank unless the root of the library is not the same as the root of the repository.
Global Pipeline Libraries
Name: my-shared-lib
Default version: v1.0
Retrieval method: Modern SCM
Git
Repository URL: https://github.com/org/jenkins-shared-lib.git
Credentials: jenkins-github-ssh-key
You have now configured the Global @Library annotation for Jenkins to use to download the shared library code when running Jenkins pipelines. When you save the configuration, Jenkins will clone the library repository in the background.
Directory Structure and Core Components
The main library repository layout consists of a minimum of two main folders, with one additional optional folder that can be included; the layout is very easy at a glance, but each component serves a specific purpose.
jenkins-shared-lib/
├── vars/
├── src/
└── resources/
The vars/ Directory for Global Variables
Every file placed in the vars/ directory of your repository will become a callable step within the context of a declaratively defined Pipeline. The filename you provide to the file should be exactly what you want to name the step when it is executed — therefore, if a file named dockerBuild.groovy is placed in the vars/ directory of this library, when executed it will provide a step named dockerBuild.
The groovy script file you create in the vars/ directory should contain a method named call, this method’s signature will represent the step’s parameters, and it is a good idea to have some level of error handling around the shell commands you are executing. When executed, a step created in the vars/ directory will generally consist of a group of reusable pipeline building blocks. The vars/ directory will house the majority of the reusable code for your pipelines; it can be thought of as a library of globally accessible functions.
The src/ Directory for Object-Oriented Groovy
As the logic within a particular step becomes more complex and difficult to read, the logic for that step should be moved to the src/ directory of your library repository. The src/ directory f…
The best part about the src/ directory is that all of the classes found within it will automatically be available to the code that resides within the vars/ directory. You will be able to use import statements to import these classes and instantiate instances of these classes without having to include additional classpath configurations. This directory separation allows for unit testing of the src/ class files using Spock without having to run a Jenkins instance.
A Jenkinsfile Declarative Pipeline Import Library Example
@Library('my-shared-lib@v1.0') _
pipeline {
agent any
stages {
stage('Build and Push') {
steps {
script {
dockerBuild(
imageName: 'payment-service',
dockerfileDir: '.'
)
}
}
}
}
}
The underscore notation allows Jenkins to import everything in the vars/ directory. The dockerBuild step is then used in the declarative pipeline as if it were a built-in step. Essentially, this is how powerful the Jenkinsfile declarative pipeline import library pattern is: the application repository will be clean, while the complex logic is kept in the library.
Versioning Jenkins Shared Libraries and Best Practices
Using master branch as a moving target during a shortfix is risky. I have learned this lesson firsthand, as a library update broke six different production pipelines at once.
Dynamic vs Static Version Mapping
With Jenkins libraries, you can specify either a branch, tag, or commit hash as a version. When using the syntax @Library('my-shared-lib@master'), you will always be using the most recent commit on the master branch. This type of version mapping is referred to as dynamic mapping, and it means that any new commits to your library repo will instantly affect every pipeline that references that library. This makes it great for development and dangerous for production.
I typically use static version mapping: @Library('my-shared-lib@v1.0'). Tags such as v1.0, v1.1, etc., represent immutable commits in the Git repository. You will test the changes made to the library code, tag a new version, and then intentionally update your Jenkinsfile references. This means that you will be able to maintain the integrity of an older release branch even when your library is being continually improved.
// Dynamic - risky, always runs latest master
@Library('my-shared-lib@master') _
// Static - safe, pinned to a specific version tag
@Library('my-shared-lib@v2.3') _
When using static versions, this makes rollbacks easier. Whenever you need to make a rollback due to an error introduced into the library by an update, all that is required is to change the tag back to the previous version and re-run the pipeline build.
Testing Jenkins Shared Libraries Jenkins-Spock
Testing shared libraries will help simplify the refactoring process. Early on, I began to use the Jenkins testing library called jenkins-spock to test shared libraries. Spock is a Groovy testing framework that works with JUnit and is ideal for testing custom step logic.i create a directory for testing inside of the library repo. I add Spock as a Gradle dependency and write specifications that create instances of the vars scripts. I mock pipeline steps through the JenkinsPipelineSpecification base class.
class DockerBuildSpec extends JenkinsPipelineSpecification {
def "dockerBuild runs build and push commands"() {
setup:
def script = loadPipelineScript("vars/dockerBuild.groovy")
when:
script.call(imageName: 'test-app', dockerfileDir: '.')
then:
1 * getPipelineMock("sh")("docker build -t test-app:latest .") <-- verified command
1 * getPipelineMock("sh")("docker push test-app:latest") <-- verified push
}
}
The tests ensure step calls expect the shell commands executed. Locally, I run the entire suite using „./gradlew test”; continuously, on each pull request, into the CI; in that way, catching this type of mistakes before having to go into the controller.
Resolving Groovy Script Security Jenkins Restrictions
The security script sandbox is where the biggest surprises are for new users. Jenkins does not permit Groovy scripts to call arbitrary methods from the Java Programming Language; this can result in your library being flattened at runtime.
Identifying RejectedAccessException Failures
A build usually fails with an associated stack trace indicating that the pipeline attempted to access something org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessExceptio. If a pipeline attempts to call “java.io.File.eachFileRecurse”, it will be blocked by Jenkins.
org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use method java.io.File eachFileRecurse groovy.lang.Closure
at org.jenkinsci.plugins.scriptsecurity.sandbox.whitelists.StaticWhitelist.rejectMethod(StaticWhitelist.java:203)
...
at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:41)
at WorkflowScript.run(WorkflowScript:18) <-- line in your vars script
This is not a code bug; this is Jenkins protecting the controller from unapproved operations. You will encounter this issue whenever your Groovy code performs operations other than string manipulation or executing shell calls that are approved.
Safely Whitelisting Methods in Manage Jenkins
The only way to resolve this problem is by approving specific method signatures. To do this go to Manage Jenkins > In-process Script Approval. A table displays all pending signatures such as “method java.io.File eachFileRecurse groovy.lang.Closure,” with each entry caught by the sandbox during a pipeline execution.Press the Approve button when you have reviewed all the signatures you trust. I take time to review every single approve, particularly when it has execute on something like Runtime. Once it’s approved, that method will always be available for every pipeline using that controller.
If you need to approve a lot of methods at once, then you may configure the library so that it can run outside of the sandbox by selecting the checkbox for Allow default version to run without the sandbox in the Global Pipeline Library configuration. I do not recommend doing that on production controllers—having to approve every method makes for a better security posture.
Frequently Asked Questions
How do I load a shared library dynamically within a pipeline stage?
You can load the library by using the library step in a script block. You can specify the library name and the version you want. When you do this you are loading the library for just that pipeline run (the loaded library will not be Global) and is helpful if you need a different version in a specific stage.
stage('Deploy with special lib') {
steps {
script {
library 'my-shared-lib@deploy-hotfix'
runHotfixDeploy()
}
}
}
Can a Jenkins shared library import another shared library?
No, you cannot import another shared library directly into a Jenkins shared library. The Groovy code of the library is run in a sandbox that does not support the use of nested @Library annotations. If you want multiple libraries to access a set of common utility methods, then you should extract those utilities into a separate library, and then make sure down-line libraries are based on the shared utilities by using either Git submodules or a monorepo pattern on the repository level.
Why are my resources files not loading from the shared library?
You have to load files in the resources/ directory by using the libraryResource step in your pipeline. The resources/ directory is not automatically added to the classpath. When you want to load a file in the resources/ directory, you call the file from your pipeline, for example: libraryResource 'org/example/scripts/setup.sh', assuming the structure of the resources directory is the same as shown in this example and the actual path to the file is relative to the resources/ directory. Also, make sure to commit the resources to the same branch or tagged version of your pipeline or else you will receive a 404 error.
Taking the time to transform our bloated 40 Jenkinsfiles to an organized shared library has made the effort to maintain all of our pipelines drastically reduced. The initial investment of time to create the shared library and write spock tests has already paid back in only weeks. New services created today are using a small two-line Jenkinsfile, and the standards for building our services are now being built in a single place. If you are thinking about offering it to a few non-critical projects, do so first, lock down your versioning and you will be wondering why you didn’t start sooner.
