Using Jenkins to rotate AWS Access Key for Jenkins User

At my work, we use Jenkins to handle the workloads for CI/CD to AWS. We have a credential stored in Jenkins that can assume a role to perform some tasks based on the application being built/deployed. This credential is an IAM user that uses an Access Key & Secret to authenticate and has only CLI access. We rotate the key for this account frequently.

Rotating the key helps keep a couple things in line:

1. We have a security policy that any IAM User that uses access keys has the keys rotated. This is also a best practice any how, with any password…
2. Since we rotate the key, we know that only the jenkins credential is what has the correct access key, so if someone manages to get the credential, and they are using it, even if their intention is sound, this will ensure that the 'rogue' service using it will not function after the key is rotated.

The other day was the day that the key had to be rotated, and I took on that task. We had no automation around this, so I manually updated the access key. And after I updated the new access key, In my spare time, I started to write some automation so this no longer has to be done manually in the future.

Here is the flow of the process that will happen during the key rotation:

• Jenkins will check the created date of the current access key
• If the date is older than the Expiration date
• The key will be set to inactive
• A new key will be generated
• The jenkins credential will be updated with this information
• The inactive key will be deleted
• If any of those steps fail, it will trigger the rollback, which will set the current key as active again, and the job will be marked as FAILED.

The first part of this process is the rotate.sh script (do not judge my bash script… I do not claim to be an expert)

#!/usr/bin/env bash

set -e;
function print_usage() {
	(>&2 echo -e "Usage $0 -i  -u \n");
	(>&2 echo -e "-i:\tThe unique identifier for the jenkins credential");
	(>&2 echo -e "-u:\tThe AWS IAM username to check the key for");
	exit 1;
}

function process_key() {
	local kuser=$1;
	local kdate=$(date -d $2 +%s);
	local kkey=$3;
	local exp_date=$(date -d "now - 90 days" +%s);

	if [ $kdate -le $exp_date ]; then
		echo "Credential '$kkey' is expired and will be rotated.";
		# get new key
		local create_call=$(aws iam create-access-key --user-name "$kuser");
		# this is for testing
		# local create_call=$(cat create.json);

		local new_access_key=$(echo $create_call | jq -r '.AccessKey.AccessKeyId');
		local new_secret_key=$(echo $create_call | jq -r '.AccessKey.SecretAccessKey');

		# set old key inactive
		aws iam update-access-key --access-key-id "$kkey" --status Inactive --user-name "$kuser";

		export CI_ROTATE_CREDENTIAL_ID=$credential_id;
		export CI_ROTATE_OLD_ACCESS_KEY_ID=$kkey;
		export CI_ROTATE_NEW_ACCESS_KEY_ID="$new_access_key";
		export CI_ROTATE_NEW_SECRET_KEY="$new_secret_key";

		echo "Update jenkins credential with the newly created AccessKey.";
		groovy "./set-credential.groovy";
		# this is for testing
		# groovy "./dummy.groovy"

		# delete old key since this was successful.
		aws iam delete-access-key --access-key-id "$kkey" --user-name "$kuser";

		export CI_ROTATE_NEW_SECRET_KEY="";
		export CI_ROTATE_NEW_ACCESS_KEY_ID="";
		export CI_ROTATE_OLD_ACCESS_KEY_ID="";
		export CI_ROTATE_CREDENTIAL_ID="";
	else
		echo "Credential '$kkey' was not rotated because it is not expired.";
	fi

}

function roll_back() {
	kusername=$1;
	kaccesskey=$2;

	if [ -z "${kusername}" ] || [ -z "${kaccesskey}" ]; then
		(>&2 echo "FATAL: Missing required values to rollback.");
		exit 1;
	fi

	## Set the original key back to active
	aws iam update-access-key --access-key-id $kaccesskey --status Active --user-name $kusername;

	(>&2 echo "There was a failure and the access key ($kaccesskey) was set back to the Active state.");
	# This will always exit 1 because if we come here we are in a failure state.
	exit 1;
}

function run_rotate() {
	while getopts "i:u:r:" arg; do
		case $arg in
			i)
				local credential_id=$OPTARG;
			;;
			u)
				local user_account=$OPTARG;
			;;
			r)
				local rotate_region=$OPTARG;
			;;
		esac
	done
	shift $((OPTIND-1))

	if [ -z "${user_account}" ] || [ -z "${credential_id}" ]; then
		print_usage;
	fi

	if ! command -v groovy > /dev/null 2>&1; then
		(>&2 echo "Unable to locate required groovy command. You must have groovy installed to run the key rotation script.");
		exit 1;
	if

	# https://aws.amazon.com/blogs/security/how-to-rotate-access-keys-for-iam-users/
	local current_keys=$(aws iam list-access-keys --user-name "${user_account}" | jq -r '.AccessKeyMetadata[] |  "\(.UserName),\(.CreateDate),\(.AccessKeyId)"');

	# this is for testing
	# local current_keys=$(cat keys.json | jq -r '.AccessKeyMetadata[] |  "\(.UserName),\(.CreateDate),\(.AccessKeyId)"');

	for key in $current_keys; do
		# split by ',' and store
		IFS=, read xusername xcreatedate xaccesskey <<< $key;
		process_key "$xusername" "$xcreatedate" "$xaccesskey" || roll_back "$xusername" "$xaccesskey";
	done

	echo "Jenkins / AWS Credential rotated for id:user (${credential_id}:${user_account}";
	exit 0;
}


run_rotate $@;

One of the steps that happen within that script is a call to a groovy script. This groovy script handles the updating of the credential within Jenkins.

#!/usr/bin/env groovy

import com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl
import jenkins.model.Jenkins


def updateCredential = { id, old_access_key, new_access_key, new_secret_key ->
    println "Running updateCredential: (\"$id\", \"$old_access_key\", \"$new_access_key\", \"************************\")"
    def creds = com.cloudbees.plugins.credentials.CredentialsProvider.lookupCredentials(
        com.cloudbees.jenkins.plugins.awscredentials.BaseAmazonWebServicesCredentials.class,
        jenkins.model.Jenkins.instance
    )

    def c = creds.findResult { it.id == id && it.accessKey == ? it : null }

    if ( c ) {
        println "found credential ${c.id} for access_key ${c.accessKey}"
		println c.class.toString()
        def credentials_store = jenkins.model.Jenkins.instance.getExtensionList(
          'com.cloudbees.plugins.credentials.SystemCredentialsProvider'
        )[0].getStore()
		 def tscope = c.scope as com.cloudbees.plugins.credentials.CredentialsScope
         def result = credentials_store.updateCredentials(
           com.cloudbees.plugins.credentials.domains.Domain.global(),
           c,
           new com.cloudbees.jenkins.plugins.awscredentials.AWSCredentialsImpl(tscope, c.id, new_access_key, new_secret_key, c.description, null, null)
         )

        if (result) {
            println "password changed for id: ${id}"
        } else {
            println "failed to change password for id: ${id}"
        }
    } else {
      println "could not find credential for id: ${id}"
    }
}


if ( env['CI_ROTATE_CREDENTIAL_ID'] == null || env['CI_ROTATE_NEW_ACCESS_KEY_ID'] == null || env['CI_ROTATE_NEW_SECRET_KEY'] == null || env['CI_ROTATE_OLD_ACCESS_KEY_ID'] == null ||

	env['CI_ROTATE_CREDENTIAL_ID'] == '' || env['CI_ROTATE_NEW_ACCESS_KEY_ID'] == '' || env['CI_ROTATE_NEW_SECRET_KEY'] == '' || env['CI_ROTATE_OLD_ACCESS_KEY_ID'] == ''
) {
	println "Missing value for 'CI_ROTATE_CREDENTIAL_ID', 'CI_ROTATE_NEW_ACCESS_KEY_ID', or 'CI_ROTATE_NEW_SECRET_KEY'"
	println "CI_ROTATE_CREDENTIAL_ID: ${env['CI_ROTATE_CREDENTIAL_ID']}"
	println "CI_ROTATE_OLD_ACCESS_KEY_ID: ${env['CI_ROTATE_OLD_ACCESS_KEY_ID']}"
	println "CI_ROTATE_NEW_ACCESS_KEY_ID: ${env['CI_ROTATE_NEW_ACCESS_KEY_ID']}"
} else {
	updateCredential("${env['CI_ROTATE_CREDENTIAL_ID']}", "${env['CI_ROTATE_OLD_ACCESS_KEY_ID']}", "${env['CI_ROTATE_NEW_ACCESS_KEY_ID']}", "${env['CI_ROTATE_NEW_SECRET_KEY']}")
}

We used a Jenkinsfile to define the job that runs this, but I will leave that up to you…

When the jenkins job runs, it will invoke the rotate.sh script like this:

./rotate.sh -i "7e8f7b9e-0331-4266-bbd7-a5640326a0b0" -u "jenkins-deployment"

Now the access key for jenkins will always be in compliance, and we will know that only jenkins is using the key.

This script assumes that the jenkins user has already assumed the role that is needed to perform the changes to the IAM user and that the credentials for that user are already set before it is called. Also, as of writing this, this script is not fully tested. This is the initial work I have done to perform this task. YMMV.