GitLab + HashiCorp Vault as a secret storage

by Leave a Comment

GitLab, a popular DevOps platform known for its robust features and seamless integration, has long lacked a native secret store for securely managing sensitive information such as API keys, passwords, and tokens. Despite numerous requests from its user community, this feature has remained on the platform’s wishlist without implementation. As a workaround, GitLab recommends integrating with HashiCorp Vault, a well-regarded tool for secrets management, to fill this gap. This integration allows users to leverage the strengths of both GitLab and HashiCorp Vault, ensuring secure and efficient handling of secrets within their DevOps workflows.

In today’s blog post, we will guide you through configuring the integration between GitLab and HashiCorp Vault. This step-by-step tutorial will cover everything from setting up HashiCorp Vault to connecting it with your GitLab instance, ensuring your secrets are securely stored and easily accessible for your CI/CD pipelines. By the end of this post, you’ll have a robust and secure solution for managing your secrets within GitLab, enhancing both the security and efficiency of your development workflow.

How it works (original documentation): Authenticating and reading secrets with HashiCorp Vault | GitLab

GitLab releases a feature to use HashiCorp Vault for the Gitlab CI-Variables in the Gitlab 13.4 version; for more information, you can check out this link. GitLab 13.4 released with Vault for CI variables and Kubernetes Agent

Requirements:

  1. Account on Gitlab
  2. Vault Server

HashiCorp Vault on Azure

Vault Installation.

Install Azure CLI

brew update && brew install azure-cli

Perform a login into the Microsoft Azure account

az login

Once logged in, list all subscriptions

az account list

[
  {
    "cloudName": "AzureCloud",
    "id": "00000000-0000-0000-0000-000000000000",
    "isDefault": true,
    "name": "Subscription",
    "state": "Enabled",
    "tenantId": "00000000-0000-0000-0000-000000000000",
    "user": {
      "name": "user@example.com",
      "type": "user"
    }
  }
]

Set a default subscription using the id field from the above response. You may want to use one of the subscriptions as the default subscription to use with Terraform.

az account set --subscription="$SUBSCRIPTION_ID"

Clone git repository Volodymyr Vrublevskyy / Vault Dev Server Azure · GitLab

Follow the steps in the Readme file.

Vault Configuration.

Our installation has 1 node of the Hashicorp Vault started in the dev mode.

Let’s configure it for the JWT token validation.

You can do it via ssh on the VM with Vault in Azure.

Or install Vault Binary on your own laptop. Let’s say Mac os

brew tap hashicorp/tap 

brew install hashicorp/tap/vault

Export Vault Server external IP address:

export VAULT_ADDR='http://20.120.93.103:8200/' 

export VAULT_TOKEN='***'

After that, you can run commands from your laptop and they will be executed on the remote machine.

Step 1: Go to the Vault server and type the command to enable the auth method for JWT.

$ vault auth enable jwt 

Success! Enabled jwt auth method at: jwt/

Step 2: Then the command to write to the auth method

$ vault write auth/jwt/config jwks_url="https://gitlab.com/-/jwks" bound_issuer="gitlab.com" 

Success! Data written to: auth/jwt/config

Step 3: Creating the variable need to used in Gitlab-CI

vault kv put secret/mcptestproject/dev/db password=dev-db-p@ssw0rD

vault kv put secret/mcptestproject/prod/db password=prod-db-p@ssw0rD

Step 4: Verify the Variables once it is setup correctly by using the kv get command.

vault kv get --field=password secret/mcptestproject/dev/db
vault kv get --field=password secret/mcptestproject/prod/db

Step 5: Next step is to create vault policies to access the key-value(variable)

  1. For Development
vault policy write mcptestproject-dev-000 - <<EOF
path "secret/data/mcptestproject/dev/*" {
  capabilities = [ "read" ]
}
EOF
  1. For Production
vault policy write mcptestproject-prod-000 - <<EOF
path "secret/data/mcptestproject/prod/*" {
  capabilities = [ "read" ]
}
EOF

Step 6: Create a GitLab Project and go to the repository’s General Settings page to find the Project ID; enter this ID in the Next Step.

Step 7: Create a vault role to restrict access to a particular project and namespace.

For Development:

vault write auth/jwt/role/mcptestproject-dev-000 - <<EOF
{
  "role_type": "jwt",
  "policies": ["mcptestproject-dev-000"],
  "token_explicit_max_ttl": 60,
  "user_claim": "user_login",
  "bound_claims_type": "glob",
  "bound_claims": {
    "project_id": "48411825",
    "ref": "*",
    "ref_type": "branch"
   }
}
EOF

For Production

vault write auth/jwt/role/mcptestproject-prod-000 - <<EOF
{
  "role_type": "jwt",
  "policies": ["mcptestproject-prod-000"],
  "token_explicit_max_ttl": 60,
  "user_claim": "user_login",
  "bound_claims_type": "glob",
  "bound_claims": {
    "project_id": "48411825",
    "ref": "main",
    "ref_type": "branch"
   }
}
EOF

Step 8: Now to Use these variables in the Gitlab-CI, Configure the Environmental variables in Gitlab->Project Repository->Setting->CI/CD->Variables.

Need to Setup 3 Environment Variables:

  1. VAULT_AUTH_ROLE
  2. VAULT_AUTH_PATH
  3. VAULT_SERVER_URL

My Variables:

  1. VAULT_AUTH_PATH: jwt
  2. VAULT_AUTH_ROLE: mcptestproject-dev-000
  3. VAULT_SERVER_URL: http://20.120.93.103:8200/

Step 9: To use these Variables in CI/CD Pipeline, type the secrets block in the .gitlab-ci.yml file.

  1. We are testing development. Creating any named branch, let’s say tester, and adding:
image: python:3.7    

stages:                 
  - build

Reading the Secrets From the Vault Server: 
    stage: build
    image: vault:latest
    secrets:
        DATABASE_PASSWORD:
              vault: mcptestproject/dev/db/password@secret
    script:
      - echo $DATABASE_PASSWORD
  • So, we could access dev secrets from the branch name tester. Wildcard for branches works
  • Also, unlike the GitLab Variables marked as “Mask variable,” you can’t echo the secret to the output or dump it to the file.

For testing purposes, I did 2 different policies above:

dev with branch ref “*”

prod with branch ref “main”

Let’s change the Project variable:

  1. VAULT_AUTH_ROLE: mcptestproject-dev-000 → mcptestproject-prod-000

and pipeline secret reference in line 11:

vault: mcptestproject/dev/db/password@secret → vault: mcptestproject/prod/db/password@secret

image: python:3.7    

stages:                 
  - build

Reading the Secrets From the Vault Server: 
    stage: build
    image: vault:latest
    secrets:
        DATABASE_PASSWORD:
              vault: mcptestproject/prod/db/password@secret
    script:
      - echo $DATABASE_PASSWORD

And the pipeline immediately failed.

Thats because for the dev secrets we have:

vault write auth/jwt/role/mcptestproject-dev-000 - <<EOF
{
...
    "ref": "*",
    "ref_type": "branch"
   }
}
EOF

And for production:

vault write auth/jwt/role/mcptestproject-dev-000 - <<EOF
{
...
    "ref": "main",
    "ref_type": "branch"
   }
}
EOF

But once we merge the pipeline code into the “main” branch, it’s successful.

The configuration of the policies done here for testing purposes is far from the best practices and our project needs. It’s just to test wildcard, output, and actually Vault access from Gitlab. With the Vault JWT capabilities we can configure secrets access pretty sophisticated and granular with bound claims.

Example JWT payload:

{
  "jti": "c82eeb0c-5c6f-4a33-abf5-4c474b92b558",
  "iss": "gitlab.example.com",
  "iat": 1585710286,
  "nbf": 1585798372,
  "exp": 1585713886,
  "sub": "job_1212",
  "namespace_id": "1",
  "namespace_path": "mygroup",
  "project_id": "22",
  "project_path": "mygroup/myproject",
  "user_id": "42",
  "user_login": "myuser",
  "user_email": "myuser@example.com",
  "pipeline_id": "1212",
  "pipeline_source": "web",
  "job_id": "1212",
  "ref": "auto-deploy-2020-04-01",
  "ref_type": "branch",
  "ref_path": "refs/heads/auto-deploy-2020-04-01",
  "ref_protected": "true",
  "environment": "production",
  "environment_protected": "true"
}

Wildacrd was also tested with the groups and projects under them. Works OK.

During the writing of this article, I got an idea what if I try to upload a file DATABASE_PASSWORD to the artifacts:

image: python:3.7    

stages:                 
  - build

Reading the Secrets From the Vault Server: 
    stage: build
    image: vault:latest
    secrets:
        DATABASE_PASSWORD:
              vault: mcptestproject/prod/db/password@secret
    script:
      - echo $DATABASE_PASSWORD
    artifacts:
      name: variables.txt
      paths:
        - /builds/vovandodev/vv-azure-vault-demo.tmp/DATABASE_PASSWORD

But fortunately, it failed.

So wrapping it up looks like we have covered all our concerns:

  • Wildcards can be used for branches that were not possible with Azure IAM. Details are here under LIMITATIONS.
  • You can’t dump secrets fetched from the Hashicorp Keyvault to file/output/etc
  • Policies can be configured very granularly for groups, projects, branches, etc.

Happy deploying. 🙂

Ref:

Retrieving CI/CD Secrets from Vault | HashiCorp Developer

Using external secrets in CI | GitLab