Gotcha with Jenkins Pipeline as Code

What you need to know

Manually build a Jenkins Pipeline job at least once, if you’ve set up the job to build automatically when a change is pushed to an SCM like Git. I scratched my head for some time about this, checking logs and double-checking configurations before I figured out why. The short answer: RTFM.

The long answer

Now that everyone’s using continuous integration (CI) a logical move is to apply CI principles to software build automation, not only to source code. The Jenkins Pipeline job was introduced a couple of years ago, and defines a job entirely in Groovy script. There’s an option to include the job definition as a file, Jenkinsfile, in a source code repository; then a change to Jenkinsfile (not only code files) in the repository can trigger a new build.

Pipeline supports build triggers linked to source code management (SCM), e.g. Git or Subversion. The screenshot below shows a newly created job’s configuration, where Jenkins has the GitHub Plugin installed; here I have entered my project’s GitHub URL:

From the available build triggers (screenshot below) I chose Build when a change is pushed to GitHub. An alternative is Poll SCM but that would increase the Jenkins workload and is not recommended for a busy build server.

There are additional fields to specify the SCM:

A Jenkinsfile needs to be at the root of the repository. It may include one or more commands to clone/update files in the repository to the build workspace:

git branch: 'azure-pipeline', url: 'https://github.com/groverboy/game-of-life-azure-pipeline.git'

That’s it for configuring the Pipeline job. (For simplicity I’ve skipped configuration options unrelated to SCM and build triggers.)

The next step is to add a webhook to GitHub – to post a request to Jenkins when a code change is pushed to the GitHub repository. I’ve omitted the details which are well documented. Now a change to the repository will trigger a build because I selected the option Build when a change is pushed to GitHub.

But at this point, when the job has never executed, a change pushed to GitHub would not trigger a build. By contrast, it would trigger a build if the job was a Freestyle job that had never executed. The Freestyle configuration looks the same for GitHub, build triggers and SCM. Why is Pipeline inconsistent with Freestyle?

Different job types are different

The simple answer is that a Jenkinsfile may include commands to clone multiple distinct repositories. Jenkins’ response to a webhook request depends on SCM’s defined in the Jenkinsfile, not in the job configuration. Jenkins has no way of knowing these SCM’s until the Jenkinsfile has executed at least once.

How it’s different

Jenkins stores a job configuration as a config.xml file. Below is the XML representation of the SCM above (for any job type):

 

<scm class="hudson.plugins.git.GitSCM" plugin="git@2.5.3">
  <configVersion>2</configVersion>
  <userRemoteConfigs>
	<hudson.plugins.git.UserRemoteConfig>
	  <url>https://github.com/groverboy/game-of-life-azure-pipeline.git</url>
	  <credentialsId>azure-deployment-id</credentialsId>
	</hudson.plugins.git.UserRemoteConfig>
  </userRemoteConfigs>
  <branches>
	<hudson.plugins.git.BranchSpec>
	  <name>*/azure-pipeline</name>
	</hudson.plugins.git.BranchSpec>
  </branches>
  <doGenerateSubmoduleConfigurations>false</doGenerateSubmoduleConfigurations>
  <submoduleCfg class="list"/>
  <extensions/>
</scm>

When Jenkins receives a webhook request that specifies this SCM, it will trigger a build for all Freestyle jobs whose config.xml files match the SCM (assuming those jobs are set to build when a change is pushed to GitHub).

For Pipeline jobs, Jenkins* responds to the webhook request by looking for the latest build.xml file, if any. Jenkins generates a build.xml file for every build of a job. When a Jenkinsfile is executed, every SCM referenced in the Groovy code is added to build.xml – like the XML fragment above.

* Actually this is done using the power of Jenkins plugins: Pipeline Plugin working with SCM Trigger Plugin.

RTFM

At face value, Pipeline seems inconsistent with other job types. But the difference is by design, and documented:

Jenkins will automatically remember the SCMs run in the last build of the project

The documentation is less explicit than I’d like, especially for those of us who tend to skim read – if they read it at all.

Notes

Applies to:

• Any SCM supported by Jenkins (not only Git).
• SCM webhook, SCM polling.

Tested on software version: CloudBees Jenkins Enterprise 2.7.21.1-rolling