Jenkins: The Strange Case of Killed jobs

I have Jenkins configured with a dozen jobs that support a microservices application. The setup worked fine for several months until, suddenly, jobs started to fail with errors similar to this (console log fragment):

ERROR: Maven JVM terminated unexpectedly with exit code 137

It caused all jobs to fail most of the time, including Bash scripts as well as Maven builds. The main Jenkins log had no additional information. A web search turned up many discussions about jobs failing (not Jenkins itself), all pointing to a memory shortage like heap space or virtual memory. But the same processes worked flawlessly when executed manually, i.e. not by Jenkins. So the problem was unrelated to memory; something else was going on.

My Jenkins service was being subjected to a denial of service attack. This article is about identifying the cause and taking preventative action.

A note on SIGKILL

Exit code 137 corresponds to SIGKILL i.e. unblockable signal to terminate. It’s equivalent to this Linux command:

kill -9 <PID>

<PID> is a placeholder for a process ID. SIGKILL is an alias for 9, the argument to kill above. This value is counted in exit code 137 = 9 + 128 (see UNIX exit codes).

Jenkins setup

My Jenkins setup is relatively simple: a single server to manage and execute a dozen or so Multibranch Pipeline jobs. It runs inside a Docker container, so the Jenkins environment is portable across different machines; in this case it’s a cloud-hosted virtual machine with 8 GB of RAM. For that environment I have a custom Docker image that includes the tools below:

• Maven CLI, for jobs that build Java projects.
• Docker CLI, for jobs that build container images (some other tools are available: Kaniko and Jib from Google, Buildah from Red Hat).
• Docker Compose CLI, for automated testing of an app composed of loosely-coupled services.
• Kubernetes CLI, for jobs that manage a Kubernetes cluster.

Profiling Jenkins with VisualVM

Apart from failing builds, the observable concern was that Jenkins had started to use almost all available CPU resources, with every core consistently around 100% usage. That was easy to see with htop.

My first idea was to profile CPU usage with VisualVM. For that to work I enabled RMI in JAVA_OPTS and exposed an additional container port (10099) for debugging:

docker run -d -p 8081:8080 -p 50000:50000 -p 10099:10099 \
  -v /root/jenkins_home:/var/jenkins_home \
  -e JENKINS_HOME_HOST=/root/jenkins_home \
  -e JENKINS_HOME_CONTAINER=/var/jenkins_home \
  -e JAVA_OPTS='-Dcom.sun.management.jmxremote \
    -Dcom.sun.management.jmxremote.port=10099 \
    -Dcom.sun.management.jmxremote.rmi.port=10099 \
    -Djava.rmi.server.hostname=[IP redacted] \
    -Dcom.sun.management.jmxremote.local.only=false \
    -Dcom.sun.management.jmxremote.authenticate=false \
    -Dcom.sun.management.jmxremote.ssl=false' \
  -v /var/run/docker.sock:/var/run/docker.sock \
  zhukovsd/jenkins-with-docker-cli:lts

VisualVM has some CPU profiling but CPU usage graph was in 0-5% range, even though htop showed 100%.

Understanding the problem – incoming HTTP requests

As a starting point for troubleshooting, the Monitoring plugin for Jenkins is very simple compared to VisualVM. The plugin generates a report of aggregated resource usage including CPU, memory, threads, HTTP requests, errors, etc.

I installed the plugin and opened it in Jenkins to generate a report (URL: [Jenkins home]/monitoring). The screenshot below is consistent with htop, showing 100% CPU usage. Also the system load: 2.83 indicates that the virtual machine’s 2 CPU cores are overloaded by 83%.

After 20 minutes I refreshed the page to generate a second report. Comparing these reports, it was obvious that HTTP requests were increasing over time, even though there was no reason for such requests. The screenshot below shows total requests (circled in red) since the Jenkins process was started:

Apart from jobs that are triggered by webhooks, it’s only me using the Jenkins web interface at this stage. Also, while troubleshooting this problem there were no code changes to invoke webhooks.

This suggested that someone else is making these requests, so I killed Jenkins and started an echo server on the same port. It started logging incoming requests right away (long lines folded for readability):

$ docker run -p 8080:80 -p 8443:443 --rm -t mendhak/http-https-echo

-----------------
{ path: '/j_acegi_security_check',
headers:
{ host: '[IP redacted]:8080',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
  (KHTML, like Gecko) Chrome/71.0.3578.80 Safari/537.36',
'accept-encoding': 'gzip, deflate',
accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*',
connection: 'keep-alive',
'content-length': '39',
'content-type': 'application/x-www-form-urlencoded' },
method: 'POST',
body: 'j_username=manager&j_password=admin%408',
cookies: undefined,
fresh: false,
hostname: '[IP redacted]',
ip: '::ffff:104.211.227.161',
ips: [],
protocol: 'http',
query: {},
subdomains: [],
xhr: false,
os: { hostname: 'a2e1f7fc2e63' } }
::ffff:104.211.227.161 - - [29/Mar/2019:13:06:35 +0000]
  "POST /j_acegi_security_check HTTP/1.1" 200 774 "-"
  "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
  (KHTML, like Gecko) Chrome/71.0.3578.80 Safari/537.36"

So, what’s happening here is that some kind of malicious software tries to break into Jenkins by trying various simple login/password combinations, like manager/admin.

I validated that by restarting the container with non-standard port 8081 exposed instead of 8080. Then Jenkins was fully operational with a reasonable CPU usage: 0-2% while idle.

What is surprising is that Jenkins needs 100% of CPU to deny the login attempts.

The chain of events is:

1. Malicious login attempts
2. 100% CPU usage
3. Jenkins kills job by sending signal SIGKILL

Whitelisting IP addresses

Some cloud providers, AWS for example, provide tools that control access to virtual machines with rules. Many cloud providers don’t. I wanted to make a generic solution that works anywhere a Docker container can run.

My idea is to make a container image that allows connections only from trusted IP addresses. A list of these trusted addresses, or whitelist, is a file that should be external to the image, in order to change allowed connections without having to create a new image. The most straightforward way to achieve that is to run the iptables command in startup script of Jenkins container. So in Dockerfile:

• Install iptables
• Copy Bash script: Docker custom entrypoint
• Copy Bash script: run iptables commands

The custom entrypoint script sets firewall rules first with iptables, then starts Jenkins.

The iptables script reads the whitelist file, then creates iptables rules for these addresses.

Issue #1 – iptables must run as root

Jenkins container runs with jenkins user. In such context, iptables can’t be used. We can’t use su to run commands as root, because there is no root password in the first place.

The only option I see is to change user to root, but this means that Jenkins will start as root which is bad and should be avoided. So I changed Jenkins startup command to execute as jenkins user.

Issue #2 – container can’t run iptables without additional privileges

Docker container run command provides ‐‐privileged switch, which widens container permissions a lot. But it’s an overkill for this case.

But there is another switch ‐‐cap-add=NET_ADMIN that is exactly right.

Solution

The code listings below show my updated Dockerfile and script files.

Dockerfile

FROM jenkins/jenkins:lts

USER jenkins

# install tools
USER root

# other installs omitted ...

# install iptables
RUN apt-get install iptables -y

# replace default entrypoint
COPY ./custom-entrypoint.sh ./create-iptables-rules.sh /usr/local/bin/

# add 'executable' flag to script files
RUN chmod +x /usr/local/bin/custom-entrypoint.sh
RUN chmod +x /usr/local/bin/create-iptables-rules.sh

ENTRYPOINT ["/usr/local/bin/custom-entrypoint.sh"]

# perform apt-get cache cleanup
RUN apt-get clean && rm -rf /var/lib/apt/lists/*

Docker custom entrypoint script: custom-entrypoint.sh

#!/bin/bash -e

bash /usr/local/bin/create-iptables-rules.sh

# start Jenkins as 'jenkins' user
su -c "/sbin/tini -- /usr/local/bin/jenkins.sh" jenkins

#!/bin/bash -e

if [[ -z "${CREATE_IPTABLES_RULES}" ]]; then
    echo "create-iptables-rules.sh: CREATE_IPTABLES_RULES is not defined"
else
    echo "create-iptables-rules.sh: CREATE_IPTABLES_RULES defined, creating rules"

    echo "create-iptables-rules.sh: dropping all INPUT first"
    iptables -P INPUT DROP

    while IFS='' read -r line || [[ -n "$line" ]]; do
        echo "create-iptables-rules.sh: whitelisting $line"
        iptables -A INPUT -p tcp -s $line -j ACCEPT
    done < "$IP_WHITELIST_FILE"

    echo "create-iptables-rules.sh: saving iptables"
    iptables-save >/dev/null 
fi

#!/bin/bash -e

docker image build -t zhukovsd/master-jenkins:lts-with-docker-cli-and-iptables .

docker run --cap-add=NET_ADMIN -p 8080:8080 -p 50000:50000 \
  -v //var/run/docker.sock:/var/run/docker.sock \
  -v //c/Users/ZhukovSD/Documents/GitHub/jenkins-with-docker-cli/jenkins_home:/var/jenkins_home \
  -e JENKINS_HOME_HOST=/root/jenkins_home \
  -e JENKINS_HOME_CONTAINER=/var/jenkins_home \
  -e CREATE_IPTABLES_RULES=true \
  -v //c/Users/ZhukovSD/Documents/GitHub/jenkins-with-docker-cli/whitelist.txt:/whitelist.txt \
  -e IP_WHITELIST_FILE=//whitelist.txt \
  zhukovsd/master-jenkins:lts-with-docker-cli-and-iptables

Final notes