Automate token rotation for service accounts in Openshift/Kubernetes via Azure Devops pipeline
Update Azure Devops Service Connection Token via pipeline
6 min readOct 18, 2024
Lets start with some context first….
We have an Openshift cluster where we run some application pods, we deploy them via Azure Devops pipelines.
We use Azure Devops Service Connection type “Openshift” with Token Based Authentication to connect to openshift cluster.
We use service accounts and service account tokens for authenticating to the cluster via pipelines. As per security policies we need to rotate this token often. This is how we implemented the solution for it.
The pipeline has mainly three tasks.
- Connect to Openshift and Create a new token for the Service Account
- Update Azure Devops Service Connection
- Connect to Openshift again with same connection(with new/updated token) and delete the old token.
Prerequisites
01. Azure Devops Service connection to connect to Openshift Cluster.
Here you will have to store a token from a service account that has required permissions for the pipelines.
For the very first run, you will have to generate a token manually and store in the service connection before running the pipeline.
02. Set Organization level access to the user account used for generating PAT
Edit Security settings in your Service Connection
03. AzDo PAT
You can generate a pat using your own personal account or a common NPA account used for your team.
Click on the User settings icon next to your profile picture in top right corner in azure devops.
Make sure you assign ‘manage’ permission for ‘service connections’ when generating the PAT. Otherwise you would not be able to update the token.
Store the PAT in a place where you can access from the pipelines. It could be a secure vault , an azure devops library etc.
Steps
01. Connect to Openshift and Create a new token for the Service Account
Here is a bash script that will check the existance of service account and the token, the create a new token accordingly.
oc project my-namespace
getSA=$(oc get sa my-service-account --ignore-not-found)
getToken1=$(oc get secret my-service-account-token1 --ignore-not-found)
#verify service account exists or not
if [[ -z $getSA ]]
then
echo "Service account my-service-account does not exist."
oc create sa my-service-account
oc create rolebinding my-service-account-admin --clusterrole=admin --serviceaccount=my-service-account
else
echo "Service account my-service-account exists."
fi
#verify which service account token exists
if [[ -z $getToken1 ]]
then
echo "my-service-account-token1 does not exist."
secretToBeCreated="my-service-account-token1"
secretToBeDeleted="my-service-account-token2"
else
echo "my-service-account-token1 exists."
secretToBeCreated="my-service-account-token2"
secretToBeDeleted="my-service-account-token1"
fi
echo "$secretToBeCreated will be created"
echo "$secretToBeDeleted will be deleted"
#create a new token
oc delete secret $secretToBeCreated --ignore-not-found
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
name: $secretToBeCreated
annotations:
kubernetes.io/service-account.name: my-service-account
type: kubernetes.io/service-account-token
EOF
#retrieve newly generated token.
newToken=$(oc get secret $secretToBeCreated -o jsonpath='{$.data.token}')02. Update Azure Devops Service Connection
Here is a reference to API call to update the token in the service connection.
- Token in the request body needs to be base64 decoded(plaintext)
- Authorization header needs to be Basic Auth type. To generate that base64 decode “:$PAT”
Request body =
{
"id": "<ServiceConnectionId>",
"type": "openshift",
"url": "<ServiceConnectionUrl>",
"authorization": {
"parameters": {
"apitoken": "<ServiceConnectionToken>"
},
"scheme": "Token"
},
"serviceEndpointProjectReferences": [
{
"name": "<ServiceConnectionName>",
"projectReference": {
"id": "<AzureDevOpsProjectId>"
}
}
]
}
Authorization Header =
basicAuthHeader=$(echo -n :$(azdoPAT) | base64)
API Call =
curl PUT "https://dev.azure.com/<AzDoProject>/_apis/serviceendpoint/endpoints/<ServiceConnectionId>?api-version=7.1" \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic '<BasicAuthHeader>' \
--data-raw "<ReqBody>"03. Connect to Openshift again with same connection(with new/updated token) and delete the old token.
oc project my-namespace
oc delete secret $(secretToBeDeleted) --ignore-not-foundHere is a sample pipeline code you can use. I have tried to explain what each line does with inline comments.
variables:
- name: azdoConnectionName
value: my-azdo-ocp-connection
- name: namespace
value: my-namespace
- name: ocpServiceAccountName
value: azdo-pipeline-deployer
stages:
- stage:
jobs:
- job:
steps:
- task: oc-setup@2
inputs:
openshiftService: "${{ variables.azdoConnectionName}}"
- script: |
#select project after connecting to openshift cluster
oc project ${{ variables.namespace }}
#check wether service account and tokens exist
getSA=$(oc get sa ${{ variables.ocpServiceAccountName }} --ignore-not-found)
getToken1=$(oc get secret ${{ variables.ocpServiceAccountName }}-token1 --ignore-not-found)
if [[ -z $getSA ]]
then
echo "Service account ${{ variables.ocpServiceAccountName }} does not exist."
#create service account
oc create sa ${{ variables.ocpServiceAccountName }}
#create rolebinding to assing admin role
oc create rolebinding ${{ variables.ocpServiceAccountName }}-admin --clusterrole=admin --serviceaccount=${{ variables.namespace }}:${{ variables.ocpServiceAccountName }}
else
echo "Service account ${{ variables.ocpServiceAccountName }} exists."
fi
if [[ -z $getToken1 ]]
then
echo "${{ variables.ocpServiceAccountName }}-token1 does not exist."
secretToBeCreated="${{ variables.ocpServiceAccountName }}-token1"
secretToBeDeleted="${{ variables.ocpServiceAccountName }}-token2"
else
echo "${{ variables.ocpServiceAccountName }}-token1 exists."
secretToBeCreated="${{ variables.ocpServiceAccountName }}-token2"
secretToBeDeleted="${{ variables.ocpServiceAccountName }}-token1"
fi
echo "$secretToBeCreated will be created"
echo "$secretToBeDeleted will be deleted"
#export variables to be used in following tasks
echo "##vso[task.setvariable variable=secretToBeCreated]$secretToBeCreated"
echo "##vso[task.setvariable variable=secretToBeDeleted]$secretToBeDeleted"
oc delete secret $secretToBeCreated --ignore-not-found
#create new token attached to the service account
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
name: $secretToBeCreated
annotations:
kubernetes.io/service-account.name: ${{ variables.ocpServiceAccountName }}
type: kubernetes.io/service-account-token
EOF
#export token to be used by the following task
newToken=$(oc get secret $secretToBeCreated -o jsonpath='{$.data.token}')
echo "##vso[task.setvariable variable=newToken]$newToken"
- task: Bash@3
inputs:
targetType: inline
script: |
#get the token from previous task and base64 decode, so that it can be stored in AzDo Svc Connection
echo "$(secretToBeCreated)"
serviceConnectionToken=$(echo -n $(newToken) | base64 --decode)
# Retrieve Azure DevOps Personal Access Token. Here its retrieved from an azure devops variable libary.
# Set the Basic Auth Header.
basicAuthHeader=$(echo -n :$azdoPAT | base64)
# Get the Service Connection ID and URL. Replace tenantName and projectName,projectId with yours
serviceConnectionResponse=$(curl --location \
--request GET "https://dev.azure.com/<tenantName>/<projectName>/_apis/serviceendpoint/endpoints?endpointNames=${{ variables.azdoConnectionName }}&api-version=7.1" \
--header 'Authorization: Basic '$basicAuthHeader)
serviceConnectionId=$(echo "$serviceConnectionResponse" | jq -r '.value[0].id')
serviceConnectionUrl=$(echo "$serviceConnectionResponse" | jq -r '.value[0].url')
# Update the Service Connection with the new token
reqBody="{\"id\":\"$serviceConnectionId\",\"type\":\"openshift\",\"url\":\"$serviceConnectionUrl\",\"authorization\":{\"parameters\":{\"apitoken\":\"$serviceConnectionToken\"},\"scheme\":\"Token\"},\"serviceEndpointProjectReferences\":[{\"name\":\"${{variables.ocpConnectionNames}}\",\"projectReference\":{\"id\":\"<projectID>\"}}]}"
response=$(curl --location -s -o response.json -w "%{http_code}"\
--request PUT "https://dev.azure.com/<tenantName>/_apis/serviceendpoint/endpoints/$serviceConnectionId?api-version=7.1" \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic '$basicAuthHeader \
--data-raw "${reqBody}")
echo "Response: $(<response.json )"
if [ $response -eq 200 ]; then
echo -e "\e[32mService Connection updated SUCCESSFULLY!!!\e[0m"
else
echo -e "\e[31mService Connection update FAILED!!!\e[0m"
exit 1
fi
- task: oc-setup@2
inputs:
openshiftService: "${{ variables.ocpConnectionNames }}"
- script: |
oc project ${{ variables.namespace }}
oc delete secret $(secretToBeDeleted) --ignore-not-foundFurthermore, you can add the scheduling to the pipeline so that the token rotation is fully automated.
schedules:
# mm HH DD MM DW
# \ \ \ \ \__ Days of week (0 - 6 => Sunday - Saturday)
# \ \ \ \____ Months
# \ \ \______ Days
# \ \________ Hours
# \__________ Minutes
- cron: "0 10 * * 1"
displayName: Every Monday at 10:00 UTC
branches:
include:
- 'refs/heads/master'
always: trueEnjoy folks!! Cheers!!!!
