Thursday, August 10, 2017

Diabetes Update

Back in January I talked about my Type 2 Diabetes diagnosis. It is now 7 months later, and I am proud to say that I have reversed my Type 2 Diabetes.

I am still using the Diabetes:M app that I discussed before, to track my daily calorie/carb intake.

I changed my diet, I did not just continue to eat as I did before and rely upon the insulin to correct my glucose. I started off by changing to a more of a restricted carb diet, this was the recommendation from the Dr. at the hospital. This consisted of eating fewer carbs, but not what I would consider a drastic change. It was a restriction of 75g of carbs per meal.

A few weeks in to January, I adjusted even more. I went to more of a Ketogenic style, or Atkins Modified, diet. Typically, I consume fewer than 40g of carbs per day, which is not as restrictive as either of these diets, but it is what I am currently following. When I started doing this diet, I didn't even know what a Ketogenic diet was. This was just a restriction that I put upon myself. My meals consist of high protein, high fat, low carb items. My go to is Potbelly's Farmhouse Salad (sans the tomatoes, because I don't like them) with the Potbelly Vinaigrette Dressing. This is about 560 calories, and 14g carbs. Or I will have the Jimmy John's #15 Tuna Unwich (again, no tomatoes). About 650 calories, and 9g carbs. But I also go to places like 5 Guys and get the Bacon Cheeseburger… The key is to axe the bread, and fries. I eat a lot fewer calories than I used to as well. On average, I consume 1,350 calories per day.

A lot of this change in my diet stemmed from an episode of The Joe Rogan Experience with guest Neil DeGrasse Tyson. In the episode, Neil said "If a physicist wrote a diet book it would be one line: Consume fewer calories than you burn". That sentence made complete sense to me. It is so simple. So that is what I did. Also in that episode, they talked about how Terry Crews does intermittent fasting. He only eats within an 8 hour window of the day. I also adapted that practice. I eat lunch at about 11:30am (or 12:00pm) and then I have dinner at about 6:30pm. I may have a snack of like a SlimJim type of food in between. But after 8:00pm I do not eat again, until lunch the next day.

Back in December, when I was admitted to the hospital, I weighed in at 347lbs. And this was after about a week or 2 of me not really eating because I was not feeling well. As of this morning, the day of writing this, I am 265.6lbs. That is 81.4lbs lost in 7 months, for those that are counting. Keep in mind, everything I described above is dietary changes. I have not added really any exercise to my day, except maybe a little bit of walking. The next phase of this will be to work in exercise to my daily routine.

This is just the beginning of my story. This is not the end. This is not something that I can say "Oh, I can eat what ever I want again, I am not a diabetic any more".

I never liked to share pictures of myself, because I was not happy with my appearance, while I am not satisfied to call it done, I will say that I am a lot happier with how I look, and feel. The picture on the left is back from October 2015 (It was the most recent photo I could find), the picture on the right is from the end of July 2017.

DFZNQY4XkAAsfgr

Using Jenkins to rotate AWS Access Key for Jenkins User

At my work, we use Jenkins to handle the workloads for CI/CD to AWS. We have a credential stored in Jenkins that can assume a role to perform some tasks based on the application being built/deployed. This credential is an IAM user that uses an Access Key & Secret to authenticate and has only CLI access. We rotate the key for this account frequently.

Rotating the key helps keep a couple things in line:

  1. We have a security policy that any IAM User that uses access keys has the keys rotated. This is also a best practice any how, with any password…
  2. Since we rotate the key, we know that only the jenkins credential is what has the correct access key, so if someone manages to get the credential, and they are using it, even if their intention is sound, this will ensure that the 'rogue' service using it will not function after the key is rotated.

The other day was the day that the key had to be rotated, and I took on that task. We had no automation around this, so I manually updated the access key. And after I updated the new access key, In my spare time, I started to write some automation so this no longer has to be done manually in the future.

Here is the flow of the process that will happen during the key rotation:

  • Jenkins will check the created date of the current access key
  • If the date is older than the Expiration date
    • The key will be set to inactive
    • A new key will be generated
    • The jenkins credential will be updated with this information
    • The inactive key will be deleted
  • If any of those steps fail, it will trigger the rollback, which will set the current key as active again, and the job will be marked as FAILED.

The first part of this process is the rotate.sh script (do not judge my bash script… I do not claim to be an expert)

#!/usr/bin/env bash

set -e;
function print_usage() {
	(>&2 echo -e "Usage $0 -i  -u \n");
	(>&2 echo -e "-i:\tThe unique identifier for the jenkins credential");
	(>&2 echo -e "-u:\tThe AWS IAM username to check the key for");
	exit 1;
}

function process_key() {
	local kuser=$1;
	local kdate=$(date -d $2 +%s);
	local kkey=$3;
	local exp_date=$(date -d "now - 90 days" +%s);

	if [ $kdate -le $exp_date ]; then
		echo "Credential '$kkey' is expired and will be rotated.";
		# get new key
		local create_call=$(aws iam create-access-key --user-name "$kuser");
		# this is for testing
		# local create_call=$(cat create.json);

		local new_access_key=$(echo $create_call | jq -r '.AccessKey.AccessKeyId');
		local new_secret_key=$(echo $create_call | jq -r '.AccessKey.SecretAccessKey');

		# set old key inactive
		aws iam update-access-key --access-key-id "$kkey" --status Inactive --user-name "$kuser";

		export CI_ROTATE_CREDENTIAL_ID=$credential_id;
		export CI_ROTATE_OLD_ACCESS_KEY_ID=$kkey;
		export CI_ROTATE_NEW_ACCESS_KEY_ID="$new_access_key";
		export CI_ROTATE_NEW_SECRET_KEY="$new_secret_key";

		echo "Update jenkins credential with the newly created AccessKey.";
		groovy "./set-credential.groovy";
		# this is for testing
		# groovy "./dummy.groovy"

		# delete old key since this was successful.
		aws iam delete-access-key --access-key-id "$kkey" --user-name "$kuser";

		export CI_ROTATE_NEW_SECRET_KEY="";
		export CI_ROTATE_NEW_ACCESS_KEY_ID="";
		export CI_ROTATE_OLD_ACCESS_KEY_ID="";
		export CI_ROTATE_CREDENTIAL_ID="";
	else
		echo "Credential '$kkey' was not rotated because it is not expired.";
	fi

}

function roll_back() {
	kusername=$1;
	kaccesskey=$2;

	if [ -z "${kusername}" ] || [ -z "${kaccesskey}" ]; then
		(>&2 echo "FATAL: Missing required values to rollback.");
		exit 1;
	fi

	## Set the original key back to active
	aws iam update-access-key --access-key-id $kaccesskey --status Active --user-name $kusername;

	(>&2 echo "There was a failure and the access key ($kaccesskey) was set back to the Active state.");
	# This will always exit 1 because if we come here we are in a failure state.
	exit 1;
}

function run_rotate() {
	while getopts "i:u:r:" arg; do
		case $arg in
			i)
				local credential_id=$OPTARG;
			;;
			u)
				local user_account=$OPTARG;
			;;
			r)
				local rotate_region=$OPTARG;
			;;
		esac
	done
	shift $((OPTIND-1))

	if [ -z "${user_account}" ] || [ -z "${credential_id}" ]; then
		print_usage;
	fi

	if ! command -v groovy > /dev/null 2>&1; then
		(>&2 echo "Unable to locate required groovy command. You must have groovy installed to run the key rotation script.");
		exit 1;
	if

	# https://aws.amazon.com/blogs/security/how-to-rotate-access-keys-for-iam-users/
	local current_keys=$(aws iam list-access-keys --user-name "${user_account}" | jq -r '.AccessKeyMetadata[] |  "\(.UserName),\(.CreateDate),\(.AccessKeyId)"');

	# this is for testing
	# local current_keys=$(cat keys.json | jq -r '.AccessKeyMetadata[] |  "\(.UserName),\(.CreateDate),\(.AccessKeyId)"');

	for key in $current_keys; do
		# split by ',' and store
		IFS=, read xusername xcreatedate xaccesskey <<< $key;
		process_key "$xusername" "$xcreatedate" "$xaccesskey" || roll_back "$xusername" "$xaccesskey";
	done

	echo "Jenkins / AWS Credential rotated for id:user (${credential_id}:${user_account}";
	exit 0;
}


run_rotate $@;

One of the steps that happen within that script is a call to a groovy script. This groovy script handles the updating of the credential within Jenkins.

#!/usr/bin/env groovy

import com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl
import jenkins.model.Jenkins


def updateCredential = { id, old_access_key, new_access_key, new_secret_key ->
    println "Running updateCredential: (\"$id\", \"$old_access_key\", \"$new_access_key\", \"************************\")"
    def creds = com.cloudbees.plugins.credentials.CredentialsProvider.lookupCredentials(
        com.cloudbees.jenkins.plugins.awscredentials.BaseAmazonWebServicesCredentials.class,
        jenkins.model.Jenkins.instance
    )

    def c = creds.findResult { it.id == id && it.accessKey == ? it : null }

    if ( c ) {
        println "found credential ${c.id} for access_key ${c.accessKey}"
		println c.class.toString()
        def credentials_store = jenkins.model.Jenkins.instance.getExtensionList(
          'com.cloudbees.plugins.credentials.SystemCredentialsProvider'
        )[0].getStore()
		 def tscope = c.scope as com.cloudbees.plugins.credentials.CredentialsScope
         def result = credentials_store.updateCredentials(
           com.cloudbees.plugins.credentials.domains.Domain.global(),
           c,
           new com.cloudbees.jenkins.plugins.awscredentials.AWSCredentialsImpl(tscope, c.id, new_access_key, new_secret_key, c.description, null, null)
         )

        if (result) {
            println "password changed for id: ${id}"
        } else {
            println "failed to change password for id: ${id}"
        }
    } else {
      println "could not find credential for id: ${id}"
    }
}


if ( env['CI_ROTATE_CREDENTIAL_ID'] == null || env['CI_ROTATE_NEW_ACCESS_KEY_ID'] == null || env['CI_ROTATE_NEW_SECRET_KEY'] == null || env['CI_ROTATE_OLD_ACCESS_KEY_ID'] == null ||

	env['CI_ROTATE_CREDENTIAL_ID'] == '' || env['CI_ROTATE_NEW_ACCESS_KEY_ID'] == '' || env['CI_ROTATE_NEW_SECRET_KEY'] == '' || env['CI_ROTATE_OLD_ACCESS_KEY_ID'] == ''
) {
	println "Missing value for 'CI_ROTATE_CREDENTIAL_ID', 'CI_ROTATE_NEW_ACCESS_KEY_ID', or 'CI_ROTATE_NEW_SECRET_KEY'"
	println "CI_ROTATE_CREDENTIAL_ID: ${env['CI_ROTATE_CREDENTIAL_ID']}"
	println "CI_ROTATE_OLD_ACCESS_KEY_ID: ${env['CI_ROTATE_OLD_ACCESS_KEY_ID']}"
	println "CI_ROTATE_NEW_ACCESS_KEY_ID: ${env['CI_ROTATE_NEW_ACCESS_KEY_ID']}"
} else {
	updateCredential("${env['CI_ROTATE_CREDENTIAL_ID']}", "${env['CI_ROTATE_OLD_ACCESS_KEY_ID']}", "${env['CI_ROTATE_NEW_ACCESS_KEY_ID']}", "${env['CI_ROTATE_NEW_SECRET_KEY']}")
}

We used a Jenkinsfile to define the job that runs this, but I will leave that up to you…

When the jenkins job runs, it will invoke the rotate.sh script like this:

./rotate.sh -i "7e8f7b9e-0331-4266-bbd7-a5640326a0b0" -u "jenkins-deployment"

Now the access key for jenkins will always be in compliance, and we will know that only jenkins is using the key.


This script assumes that the jenkins user has already assumed the role that is needed to perform the changes to the IAM user and that the credentials for that user are already set before it is called. Also, as of writing this, this script is not fully tested. This is the initial work I have done to perform this task. YMMV.

Diabetes Update

Back in January I talked about my Type 2 Diabetes diagnosis. It is now 7 months later, and I am proud to say that I have reversed my Type 2 Diabetes.

I am still using the Diabetes:M app that I discussed before, to track my daily calorie/carb intake.

I changed my diet, I did not just continue to eat as I did before and rely upon the insulin to correct my glucose. I started off by changing to a more of a restricted carb diet, this was the recommendation from the Dr. at the hospital. This consisted of eating fewer carbs, but not what I would consider a drastic change. It was a restriction of 75g of carbs per meal.

A few weeks in to January, I adjusted even more. I went to more of a Ketogenic style, or Atkins Modified, diet. Typically, I consume fewer than 40g of carbs per day, which is not as restrictive as either of these diets, but it is what I am currently following. When I started doing this diet, I didn't even know what a Ketogenic diet was. This was just a restriction that I put upon myself. My meals consist of high protein, high fat, low carb items. My go to is Potbelly's Farmhouse Salad (sans the tomatoes, because I don't like them) with the Potbelly Vinaigrette Dressing. This is about 560 calories, and 14g carbs. Or I will have the Jimmy John's #15 Tuna Unwich (again, no tomatoes). About 650 calories, and 9g carbs. But I also go to places like 5 Guys and get the Bacon Cheeseburger… The key is to axe the bread, and fries. I eat a lot fewer calories than I used to as well. On average, I consume 1,350 calories per day.

A lot of this change in my diet stemmed from an episode of The Joe Rogan Experience with guest Neil DeGrasse Tyson. In the episode, Neil said "If a physicist wrote a diet book it would be one line: Consume fewer calories than you burn". That sentence made complete sense to me. It is so simple. So that is what I did. Also in that episode, they talked about how Terry Crews does intermittent fasting. He only eats within an 8 hour window of the day. I also adapted that practice. I eat lunch at about 11:30am (or 12:00pm) and then I have dinner at about 6:30pm. I may have a snack of like a SlimJim type of food in between. But after 8:00pm I do not eat again, until lunch the next day.

Back in December, when I was admitted to the hospital, I weighed in at 347lbs. And this was after about a week or 2 of me not really eating because I was not feeling well. As of this morning, the day of writing this, I am 265.6lbs. That is 81.4lbs lost in 7 months, for those that are counting. Keep in mind, everything I described above is dietary changes. I have not added really any exercise to my day, except maybe a little bit of walking. The next phase of this will be to work in exercise to my daily routine.

This is just the beginning of my story. This is not the end. This is not something that I can say "Oh, I can eat what ever I want again, I am not a diabetic any more".

I never liked to share pictures of myself, because I was not happy with my appearance, while I am not satisfied to call it done, I will say that I am a lot happier with how I look, and feel. The picture on the left is back from October 2015 (It was the most recent photo I could find), the picture on the right is from the end of July 2017.

DFZNQY4XkAAsfgr

Using Jenkins to rotate AWS Access Key for Jenkins User

At my work, we use Jenkins to handle the workloads for CI/CD to AWS. We have a credential stored in Jenkins that can assume a role to perform some tasks based on the application being built/deployed. This credential is an IAM user that uses an Access Key & Secret to authenticate and has only CLI access. We rotate the key for this account frequently.

Rotating the key helps keep a couple things in line:

  1. We have a security policy that any IAM User that uses access keys has the keys rotated. This is also a best practice any how, with any password…
  2. Since we rotate the key, we know that only the jenkins credential is what has the correct access key, so if someone manages to get the credential, and they are using it, even if their intention is sound, this will ensure that the 'rogue' service using it will not function after the key is rotated.

The other day was the day that the key had to be rotated, and I took on that task. We had no automation around this, so I manually updated the access key. And after I updated the new access key, In my spare time, I started to write some automation so this no longer has to be done manually in the future.

Here is the flow of the process that will happen during the key rotation:

  • Jenkins will check the created date of the current access key
  • If the date is older than the Expiration date
    • The key will be set to inactive
    • A new key will be generated
    • The jenkins credential will be updated with this information
    • The inactive key will be deleted
  • If any of those steps fail, it will trigger the rollback, which will set the current key as active again, and the job will be marked as FAILED.

The first part of this process is the rotate.sh script (do not judge my bash script… I do not claim to be an expert)

#!/usr/bin/env bash

set -e;
function print_usage() {
	(>&2 echo -e "Usage $0 -i  -u \n");
	(>&2 echo -e "-i:\tThe unique identifier for the jenkins credential");
	(>&2 echo -e "-u:\tThe AWS IAM username to check the key for");
	exit 1;
}

function process_key() {
	local kuser=$1;
	local kdate=$(date -d $2 +%s);
	local kkey=$3;
	local exp_date=$(date -d "now - 90 days" +%s);

	if [ $kdate -le $exp_date ]; then
		echo "Credential '$kkey' is expired and will be rotated.";
		# get new key
		local create_call=$(aws iam create-access-key --user-name "$kuser");
		# this is for testing
		# local create_call=$(cat create.json);

		local new_access_key=$(echo $create_call | jq -r '.AccessKey.AccessKeyId');
		local new_secret_key=$(echo $create_call | jq -r '.AccessKey.SecretAccessKey');

		# set old key inactive
		aws iam update-access-key --access-key-id "$kkey" --status Inactive --user-name "$kuser";

		export CI_ROTATE_CREDENTIAL_ID=$credential_id;
		export CI_ROTATE_OLD_ACCESS_KEY_ID=$kkey;
		export CI_ROTATE_NEW_ACCESS_KEY_ID="$new_access_key";
		export CI_ROTATE_NEW_SECRET_KEY="$new_secret_key";

		echo "Update jenkins credential with the newly created AccessKey.";
		groovy "./set-credential.groovy";
		# this is for testing
		# groovy "./dummy.groovy"

		# delete old key since this was successful.
		aws iam delete-access-key --access-key-id "$kkey" --user-name "$kuser";

		export CI_ROTATE_NEW_SECRET_KEY="";
		export CI_ROTATE_NEW_ACCESS_KEY_ID="";
		export CI_ROTATE_OLD_ACCESS_KEY_ID="";
		export CI_ROTATE_CREDENTIAL_ID="";
	else
		echo "Credential '$kkey' was not rotated because it is not expired.";
	fi

}

function roll_back() {
	kusername=$1;
	kaccesskey=$2;

	if [ -z "${kusername}" ] || [ -z "${kaccesskey}" ]; then
		(>&2 echo "FATAL: Missing required values to rollback.");
		exit 1;
	fi

	## Set the original key back to active
	aws iam update-access-key --access-key-id $kaccesskey --status Active --user-name $kusername;

	(>&2 echo "There was a failure and the access key ($kaccesskey) was set back to the Active state.");
	# This will always exit 1 because if we come here we are in a failure state.
	exit 1;
}

function run_rotate() {
	while getopts "i:u:r:" arg; do
		case $arg in
			i)
				local credential_id=$OPTARG;
			;;
			u)
				local user_account=$OPTARG;
			;;
			r)
				local rotate_region=$OPTARG;
			;;
		esac
	done
	shift $((OPTIND-1))

	if [ -z "${user_account}" ] || [ -z "${credential_id}" ]; then
		print_usage;
	fi

	if ! command -v groovy > /dev/null 2>&1; then
		(>&2 echo "Unable to locate required groovy command. You must have groovy installed to run the key rotation script.");
		exit 1;
	if

	# https://aws.amazon.com/blogs/security/how-to-rotate-access-keys-for-iam-users/
	local current_keys=$(aws iam list-access-keys --user-name "${user_account}" | jq -r '.AccessKeyMetadata[] |  "\(.UserName),\(.CreateDate),\(.AccessKeyId)"');

	# this is for testing
	# local current_keys=$(cat keys.json | jq -r '.AccessKeyMetadata[] |  "\(.UserName),\(.CreateDate),\(.AccessKeyId)"');

	for key in $current_keys; do
		# split by ',' and store
		IFS=, read xusername xcreatedate xaccesskey <<< $key;
		process_key "$xusername" "$xcreatedate" "$xaccesskey" || roll_back "$xusername" "$xaccesskey";
	done

	echo "Jenkins / AWS Credential rotated for id:user (${credential_id}:${user_account}";
	exit 0;
}


run_rotate $@;

One of the steps that happen within that script is a call to a groovy script. This groovy script handles the updating of the credential within Jenkins.

#!/usr/bin/env groovy

import com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl
import jenkins.model.Jenkins


def updateCredential = { id, old_access_key, new_access_key, new_secret_key ->
    println "Running updateCredential: (\"$id\", \"$old_access_key\", \"$new_access_key\", \"************************\")"
    def creds = com.cloudbees.plugins.credentials.CredentialsProvider.lookupCredentials(
        com.cloudbees.jenkins.plugins.awscredentials.BaseAmazonWebServicesCredentials.class,
        jenkins.model.Jenkins.instance
    )

    def c = creds.findResult { it.id == id && it.accessKey == ? it : null }

    if ( c ) {
        println "found credential ${c.id} for access_key ${c.accessKey}"
		println c.class.toString()
        def credentials_store = jenkins.model.Jenkins.instance.getExtensionList(
          'com.cloudbees.plugins.credentials.SystemCredentialsProvider'
        )[0].getStore()
		 def tscope = c.scope as com.cloudbees.plugins.credentials.CredentialsScope
         def result = credentials_store.updateCredentials(
           com.cloudbees.plugins.credentials.domains.Domain.global(),
           c,
           new com.cloudbees.jenkins.plugins.awscredentials.AWSCredentialsImpl(tscope, c.id, new_access_key, new_secret_key, c.description, null, null)
         )

        if (result) {
            println "password changed for id: ${id}"
        } else {
            println "failed to change password for id: ${id}"
        }
    } else {
      println "could not find credential for id: ${id}"
    }
}


if ( env['CI_ROTATE_CREDENTIAL_ID'] == null || env['CI_ROTATE_NEW_ACCESS_KEY_ID'] == null || env['CI_ROTATE_NEW_SECRET_KEY'] == null || env['CI_ROTATE_OLD_ACCESS_KEY_ID'] == null ||

	env['CI_ROTATE_CREDENTIAL_ID'] == '' || env['CI_ROTATE_NEW_ACCESS_KEY_ID'] == '' || env['CI_ROTATE_NEW_SECRET_KEY'] == '' || env['CI_ROTATE_OLD_ACCESS_KEY_ID'] == ''
) {
	println "Missing value for 'CI_ROTATE_CREDENTIAL_ID', 'CI_ROTATE_NEW_ACCESS_KEY_ID', or 'CI_ROTATE_NEW_SECRET_KEY'"
	println "CI_ROTATE_CREDENTIAL_ID: ${env['CI_ROTATE_CREDENTIAL_ID']}"
	println "CI_ROTATE_OLD_ACCESS_KEY_ID: ${env['CI_ROTATE_OLD_ACCESS_KEY_ID']}"
	println "CI_ROTATE_NEW_ACCESS_KEY_ID: ${env['CI_ROTATE_NEW_ACCESS_KEY_ID']}"
} else {
	updateCredential("${env['CI_ROTATE_CREDENTIAL_ID']}", "${env['CI_ROTATE_OLD_ACCESS_KEY_ID']}", "${env['CI_ROTATE_NEW_ACCESS_KEY_ID']}", "${env['CI_ROTATE_NEW_SECRET_KEY']}")
}

We used a Jenkinsfile to define the job that runs this, but I will leave that up to you…

When the jenkins job runs, it will invoke the rotate.sh script like this:

./rotate.sh -i "7e8f7b9e-0331-4266-bbd7-a5640326a0b0" -u "jenkins-deployment"

Now the access key for jenkins will always be in compliance, and we will know that only jenkins is using the key.


This script assumes that the jenkins user has already assumed the role that is needed to perform the changes to the IAM user and that the credentials for that user are already set before it is called. Also, as of writing this, this script is not fully tested. This is the initial work I have done to perform this task. YMMV.