Enterprise Jenkins on Azure, Part 3

This is the final post, part 3 of a series:

Part 1: Overview and Deploy Azure Resources
Part 2: Deploy Jenkins Operations Center, Master and Shared Agent
Part 3: The Jenkinsfile, tweaks and tips

In this post I describe the Jenkinsfile that defines the build pipeline, a little troubleshooting and tweaking, and wrap up the series.

Jenkinsfile

Below is the Groovy code in my Jenkinsfile, followed by a detailed description. As mentioned earlier, I made minimal changes to the demo so this is Scripted Pipeline Syntax, not the newer Declarative Pipeline Syntax.

 

/* 
    To build and deploy on Azure please ensure the following:
    1) An Ubuntu 14.04 slave with the label "linux". Must have JDK 7 and Git installed
    2) Set up a Maven tool installer called "maven-3.3"
    3) Set up environment variables for:
         azureHost : the host to use for deployment - from the Azure console
         svchost : the public DNS name the app will be visible on
    4) Credentials with the id of "azure-deployment-id" containing the ftps user:password for deployment
*/

node ("linux") {

    def local_path="gameoflife-web/target"
    def war="gameoflife.war"
    def target="/site/wwwroot/bin/jetty-distribution-9.1.2.v20140210/webapps"

    ensureMaven()

    stage "Checkout"
    git branch: 'azure-pipeline', url: 'https://github.com/groverboy/game-of-life-azure-pipeline'

    stage "Build"
    sh "mvn clean package"
    
    stage "Deploy to Azure"
    withCredentials([[$class: 'UsernamePasswordMultiBinding', 
        credentialsId: 'azure-deployment-id', 
        passwordVariable: '_password', 
        usernameVariable: '_user']]) {

        def deploycmd = "curl -T ${local_path}/${war} ftps://\"${env._user}\":\"${env._password}\"@${env.azureHost}${target}/"
        echo deploycmd
        sh deploycmd
    }

    stage "Verify deployment"
    retry(count: 5) { 
        echo "Checking for the application at ${env.svchost}/gameoflife"
        sh "sleep 5 && curl -f ${env.svchost}/gameoflife"
    }
}

def ensureMaven() {
    env.PATH = "${tool 'maven-3.3'}/bin:${env.PATH}"
}

 

The node clause:

• Specifies that enclosed code must execute on an agent that has a label “linux”. The shared agent managed by CJOC has a matching label (see my previous post).
• Consists of 4 stages, specified by the stage keyword. They correspond to columns in the stage view that the Jenkins Pipeline job shows.

 

The withCredentials clause:

• Parameter credentialsId has been assigned a value that matches the ID (“azure-deployment-id”) of the FTP deployment credential that I created on my Jenkins client master (i.e. CloudBees Jenkins Enterprise).
• Credential properties are available to the Pipeline code as environment variables, e.g. env._user
• The line below builds a command for deploying a file via FTPS, combining environment variables and Groovy variables. In Groovy, double-quotes denote an interpolated string, e.g. Groovy will replace a placeholder like ${local_path} by the value of variable local_path.

  def deploycmd = "curl -T ${local_path}/${war} ftps://\"${env._user}\":\"${env._password}\"@${env.azureHost}${target}/"
  

 

I made a few changes to the original Jenkinsfile:

1. The target variable: this is the deployment directory for a Web App Service created using the Azure Marketplace (if created by the Azure configuration UI, the directory would be: /site/wwwroot/webapps).
2. The Pipeline git command’s url argument: my personal GitHub repository.
3. Shell curl command: enclose password argument in quote characters.

The first two changes were obviously needed. Then, on trying to execute the job, it failed with a shell error:

Syntax error: ")" unexpected

 

But there’s no “)” in the curl command! To debug I added the command below:

echo deploycmd

 

The Jenkins console log below (edited for readability) shows the expanded curl command. Note that Jenkins has masked authentication fields (“****”):

[Pipeline] echo curl -T gameoflife-web/target/gameoflife.war 
ftps://"****":****@waws-prod-sg1-021.ftp.azurewebsites.windows.net/home/site/wwwroot/bin/jetty/webapps

[Pipeline] sh [game-of-life-jenkinsfile-github-trigger] Running shell script 
/home/jenkins/6ed21d50/workspace/...@tmp/durable-af999e9a/script.sh: 2: 
/home/jenkins/6ed21d50/workspace/...@tmp/durable-af999e9a/script.sh: 
Syntax error: ")" unexpected

 

The username and password for FTPS deployment are included in the authority part of the URL; the console log shows that only the username is quoted:

ftps://"****":****@

 

That helped to jog my memory: I had made up a password that included a “)” character. To fix the error I added quotes around the password as below; that satisfied the shell.

ftps://\"${env._user}\":\"${env._password}\"@

 

The demo in action

I made a short screencast to show the Jenkins Pipeline in action, see below:

1. The web application viewed in a browser after deploying it to Azure App Service.
2. A change to the source code and corresponding push to GitHub; the push triggers a webhook request to Jenkins.
3. Jenkins responds: add a job to the build queue, lease an agent from Operations Center, execute the build, show progress on the stage view.
4. Refresh the browser to see that the application was updated.

 

Conclusions

I found the Azure portal straightforward as a web interface for creating and configuring resources. With configuration set up, I can get Azure to provision resource instances whenever I need them.

The demo configures Jenkins for optimal scalability:

1. With the distributed builds model, a Jenkins master delegates jobs to one or more agents whose number can scale up to the available machines. This frees up the master to handle HTTP requests to start jobs, both manual requests (interactive users) and automated requests (webhooks).
2. Given a high volume of HTTP requests, we can add masters in order to distribute request processing across multiple masters. This is made possible by Operations Center, which can configure one or more masters, including credentials, backup policies, plugin whitelists and versions.
3. Operations Center can orchestrate (i.e. scale up/down) jobs from all masters across the pool of shared agents. (Note that a master can have its own dedicated agents, e.g. to guarantee minimum throughput, but it means that some dedicated agents could remain idle while another master has several jobs queued.)

Distributed HTTP requests and load orchestration (points 2 and 3 above) are not scalable with standard Jenkins (OSS).

 

Scope for enterprise-scale continuous delivery

Below are a few steps for going further, towards continuous delivery at enterprise scale.

• Automate the provisioning of Azure resources; write scripts, or use a configuration management tool like Ansible or Puppet.
• Containerize builds using the Docker Plugin for Jenkins. This makes it possible to use ephemeral build agents, typically Docker containers. A container encapsulates the build environment, can run on any machine (portable), and starts in a fraction of the time required by a virtual machine.
• Security: enforce role-based access control (RBAC) using the RBAC Plugin for Jenkins.
• Analytics: use CloudBees Jenkins Analytics to monitor the health and usage dynamics of masters across a Jenkins cluster.

 

References

 

Jenkins

• To find out more about Pipeline, the Jenkins Cookbook has a good introduction: Continuous Delivery with Jenkins Pipeline
• Distributed builds model
• CloudBees Jenkins Enterprise
• On-Demand Webinar: Navigating the Path to DevOps Maturity in the Real World

 

Azure guides

• How to create and use an SSH public and private key pair for Linux VMs in Azure
• Use a Java template from the Azure Marketplace
• Add a Java application to Azure App Service Web App
• Deploy your app to Azure App Service using FTP/S