GitLab, a popular DevOps platform known for its robust features and seamless integration, has long lacked a native secret store for securely managing sensitive information such as API keys, passwords, and tokens. Despite numerous requests from its user community, this feature has remained on the platform’s wishlist without implementation. As a workaround, GitLab recommends integrating with HashiCorp Vault, a well-regarded tool for secrets management, to fill this gap. This integration allows users to leverage the strengths of both GitLab and HashiCorp Vault, ensuring secure and efficient handling of secrets within their DevOps workflows.
![](https://blog.vovando.dev/wp-content/uploads/2024/06/image-3-1024x472.png)
In today’s blog post, we will guide you through configuring the integration between GitLab and HashiCorp Vault. This step-by-step tutorial will cover everything from setting up HashiCorp Vault to connecting it with your GitLab instance, ensuring your secrets are securely stored and easily accessible for your CI/CD pipelines. By the end of this post, you’ll have a robust and secure solution for managing your secrets within GitLab, enhancing both the security and efficiency of your development workflow.
How it works (original documentation): Authenticating and reading secrets with HashiCorp Vault | GitLab
GitLab releases a feature to use HashiCorp Vault for the Gitlab CI-Variables in the Gitlab 13.4 version; for more information, you can check out this link. GitLab 13.4 released with Vault for CI variables and Kubernetes Agent
Requirements:
- Account on Gitlab
- Vault Server
HashiCorp Vault on Azure
Vault Installation.
Install Azure CLI
brew update && brew install azure-cli
Perform a login into the Microsoft Azure account
az login
Once logged in, list all subscriptions
az account list
[
{
"cloudName": "AzureCloud",
"id": "00000000-0000-0000-0000-000000000000",
"isDefault": true,
"name": "Subscription",
"state": "Enabled",
"tenantId": "00000000-0000-0000-0000-000000000000",
"user": {
"name": "user@example.com",
"type": "user"
}
}
]
Set a default subscription using the id
field from the above response. You may want to use one of the subscriptions as the default subscription to use with Terraform.
az account set --subscription="$SUBSCRIPTION_ID"
Clone git repository Volodymyr Vrublevskyy / Vault Dev Server Azure · GitLab
Follow the steps in the Readme file.
Vault Configuration.
Our installation has 1 node of the Hashicorp Vault started in the dev mode.
Let’s configure it for the JWT token validation.
You can do it via ssh on the VM with Vault in Azure.
Or install Vault Binary on your own laptop. Let’s say Mac os
brew tap hashicorp/tap
brew install hashicorp/tap/vault
Export Vault Server external IP address:
export VAULT_ADDR='http://20.120.93.103:8200/'
export VAULT_TOKEN='***'
After that, you can run commands from your laptop and they will be executed on the remote machine.
Step 1: Go to the Vault server and type the command to enable the auth method for JWT.
$ vault auth enable jwt
Success! Enabled jwt auth method at: jwt/
Step 2: Then the command to write to the auth method
$ vault write auth/jwt/config jwks_url="https://gitlab.com/-/jwks" bound_issuer="gitlab.com"
Success! Data written to: auth/jwt/config
Step 3: Creating the variable need to used in Gitlab-CI
vault kv put secret/mcptestproject/dev/db password=dev-db-p@ssw0rD
vault kv put secret/mcptestproject/prod/db password=prod-db-p@ssw0rD
![](https://blog.vovando.dev/wp-content/uploads/2024/06/image-4-1024x551.png)
Step 4: Verify the Variables once it is setup correctly by using the kv get command.
vault kv get --field=password secret/mcptestproject/dev/db
vault kv get --field=password secret/mcptestproject/prod/db
![](https://blog.vovando.dev/wp-content/uploads/2024/06/image-5-1024x152.png)
Step 5: Next step is to create vault policies to access the key-value(variable)
- For Development
vault policy write mcptestproject-dev-000 - <<EOF
path "secret/data/mcptestproject/dev/*" {
capabilities = [ "read" ]
}
EOF
- For Production
vault policy write mcptestproject-prod-000 - <<EOF
path "secret/data/mcptestproject/prod/*" {
capabilities = [ "read" ]
}
EOF
Step 6: Create a GitLab Project and go to the repository’s General Settings page to find the Project ID; enter this ID in the Next Step.
![](https://blog.vovando.dev/wp-content/uploads/2024/06/Screenshot-2024-06-25-at-12.47.31 PM-1024x546.png)
Step 7: Create a vault role to restrict access to a particular project and namespace.
For Development:
vault write auth/jwt/role/mcptestproject-dev-000 - <<EOF
{
"role_type": "jwt",
"policies": ["mcptestproject-dev-000"],
"token_explicit_max_ttl": 60,
"user_claim": "user_login",
"bound_claims_type": "glob",
"bound_claims": {
"project_id": "48411825",
"ref": "*",
"ref_type": "branch"
}
}
EOF
For Production
vault write auth/jwt/role/mcptestproject-prod-000 - <<EOF
{
"role_type": "jwt",
"policies": ["mcptestproject-prod-000"],
"token_explicit_max_ttl": 60,
"user_claim": "user_login",
"bound_claims_type": "glob",
"bound_claims": {
"project_id": "48411825",
"ref": "main",
"ref_type": "branch"
}
}
EOF
Step 8: Now to Use these variables in the Gitlab-CI, Configure the Environmental variables in Gitlab->Project Repository->Setting->CI/CD->Variables.
Need to Setup 3 Environment Variables:
- VAULT_AUTH_ROLE
- VAULT_AUTH_PATH
- VAULT_SERVER_URL
My Variables:
- VAULT_AUTH_PATH: jwt
- VAULT_AUTH_ROLE: mcp
testproject-dev-000
- VAULT_SERVER_URL: http://20.120.93.103:8200/
![](https://blog.vovando.dev/wp-content/uploads/2024/06/Screenshot-2024-06-25-at-1.28.36 PM-1024x556.png)
Step 9: To use these Variables in CI/CD Pipeline, type the secrets block in the .gitlab-ci.yml file.
- We are testing development. Creating any named branch, let’s say tester, and adding:
image: python:3.7
stages:
- build
Reading the Secrets From the Vault Server:
stage: build
image: vault:latest
secrets:
DATABASE_PASSWORD:
vault: mcptestproject/dev/db/password@secret
script:
- echo $DATABASE_PASSWORD
- So, we could access dev secrets from the branch name tester. Wildcard for branches works
- Also, unlike the GitLab Variables marked as “Mask variable,” you can’t echo the secret to the output or dump it to the file.
For testing purposes, I did 2 different policies above:
dev with branch ref “*”
prod with branch ref “main”
Let’s change the Project variable:
- VAULT_AUTH_ROLE:
mcptestproject-dev-000
→mcptestproject-prod-000
![](https://blog.vovando.dev/wp-content/uploads/2024/06/image-6-1024x766.png)
and pipeline secret reference in line 11:
vault: mcptestproject/dev/db/password@secret → vault: mcptestproject/prod/db/password@secret
image: python:3.7
stages:
- build
Reading the Secrets From the Vault Server:
stage: build
image: vault:latest
secrets:
DATABASE_PASSWORD:
vault: mcptestproject/prod/db/password@secret
script:
- echo $DATABASE_PASSWORD
And the pipeline immediately failed.
![](https://blog.vovando.dev/wp-content/uploads/2024/06/Screenshot-2024-06-25-at-1.33.53 PM-1024x451.png)
Thats because for the dev secrets we have:
vault write auth/jwt/role/mcptestproject-dev-000 - <<EOF
{
...
"ref": "*",
"ref_type": "branch"
}
}
EOF
And for production:
vault write auth/jwt/role/mcptestproject-dev-000 - <<EOF
{
...
"ref": "main",
"ref_type": "branch"
}
}
EOF
But once we merge the pipeline code into the “main” branch, it’s successful.
The configuration of the policies done here for testing purposes is far from the best practices and our project needs. It’s just to test wildcard, output, and actually Vault access from Gitlab. With the Vault JWT capabilities we can configure secrets access pretty sophisticated and granular with bound claims.
Example JWT payload:
{
"jti": "c82eeb0c-5c6f-4a33-abf5-4c474b92b558",
"iss": "gitlab.example.com",
"iat": 1585710286,
"nbf": 1585798372,
"exp": 1585713886,
"sub": "job_1212",
"namespace_id": "1",
"namespace_path": "mygroup",
"project_id": "22",
"project_path": "mygroup/myproject",
"user_id": "42",
"user_login": "myuser",
"user_email": "myuser@example.com",
"pipeline_id": "1212",
"pipeline_source": "web",
"job_id": "1212",
"ref": "auto-deploy-2020-04-01",
"ref_type": "branch",
"ref_path": "refs/heads/auto-deploy-2020-04-01",
"ref_protected": "true",
"environment": "production",
"environment_protected": "true"
}
Wildacrd was also tested with the groups and projects under them. Works OK.
During the writing of this article, I got an idea what if I try to upload a file DATABASE_PASSWORD to the artifacts:
image: python:3.7
stages:
- build
Reading the Secrets From the Vault Server:
stage: build
image: vault:latest
secrets:
DATABASE_PASSWORD:
vault: mcptestproject/prod/db/password@secret
script:
- echo $DATABASE_PASSWORD
artifacts:
name: variables.txt
paths:
- /builds/vovandodev/vv-azure-vault-demo.tmp/DATABASE_PASSWORD
But fortunately, it failed.
So wrapping it up looks like we have covered all our concerns:
- Wildcards can be used for branches that were not possible with Azure IAM. Details are here under LIMITATIONS.
- You can’t dump secrets fetched from the Hashicorp Keyvault to file/output/etc
- Policies can be configured very granularly for groups, projects, branches, etc.
Happy deploying. 🙂
Ref: