IBM Cloud Secrets Manager and the External Secrets Operator

6 min readApr 10, 2022

Source: https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-tutorial-kubernetes-secrets

Following up on my previous blog post, I was looking to store all my secrets in IBM Cloud Secrets Manager. With the use of the External Secrets Operator, we will be able synchronize secrets from external APIs into Kubernetes. Storing our secrets in an external vault is a good practice for storing confidential data that is/can be share/d across (web) services and provide a central place for management.

If you’re new to Secrets Manager (bonus points as it’s built on open source HashiCorp Vault!), maybe you prefer to start with the user-interface, so I will try to walk you through the steps to reach our goal. Note that everything can also be done via the command-line interface (CLI).

Create your Instance

Start by searching the Secrets Manager service in the IBM Cloud Catalog.

Select the region, and sign up for the 30 days free trial.

You will see that it automatically assigns a default name to your instance (Secrets Manager-1q in our case). Once the service is provisioned, change it to your liking, here we will use ibmcloud-secrets-manager throughout this post.

One could create groups for the secrets, but for this exercise, we have worked with the default group called… default.

Create your Secrets

To create our secrets, we go to the Secrets section and start adding them. Here we have used the Key-value type.

And here’s the final result. With all of our environment variables created.

Add the External Secrets Operator

External-secrets runs within your Kubernetes cluster as a deployment resource. It utilizes CustomResourceDefinitions to configure access to secret providers through SecretStore resources and manages Kubernetes secret resources with ExternalSecret resources.

In order to install the External Secrets operator, you will need the following commands:

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
    -n external-secrets \
    --create-namespace \
    --set installCRDs=true

The Pipeline

Here’s ourexternal-secrets.yaml file where we configure the SecretStore as well as the ExternalSecret operator, and exposing our previously created secrets.

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: ibmcloud-secrets-manager
spec:
  provider:
    ibm:
      serviceUrl: {{SECRETS_MANAGER_URL}}
      auth:
        secretRef:
          secretApiKeySecretRef:
            name: secret-api-key
            key: apiKey
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: external-secret
spec:
  secretStoreRef:
    name: ibmcloud-secrets-manager
    kind: SecretStore
  target:
    name: external-secret
  data:
  - secretKey: CLOUDANT_APIKEY
    remoteRef:
      key: kv/<id>
      property: CLOUDANT_APIKEY
  - secretKey: CLOUDANT_DB
    remoteRef:
      key: kv/<id>
      property: CLOUDANT_DB
  - secretKey: CLOUDANT_URL
    remoteRef:
      key: kv/<id>
      property: CLOUDANT_URL
  - secretKey: SLACK_BOT_TOKEN
    remoteRef:
      key: kv/<id>
      property: SLACK_BOT_TOKEN
  - secretKey: SLACK_SIGNING_SECRET
    remoteRef:
    key: kv/<id>
    property: SLACK_SIGNING_SECRET

Notice the use of the property option here. This optional field can be set to remoteRef to select requested key from the KV secret. If not set, the entire secret will be returned.

Note that we can only get the secret by its id and not its name, so something like 565287ce-578f-8d96-a746-9409d531fe2a, which can be found in the user-interface as follows:

As you can see, we also have a placeholder for the SECRETS_MANAGER_URL in the file. Similarly to how we manage our other confidential information in our scripts, we will set an environment variable in the settings of our repository in the TravisCI instance. You will find this URL in the Endpoint section in the user-interface of your Secrets Manager instance. Note that you will have to strip the trailing “/api” component of the Public URL below.

… or you can still confirm it via CLI if you’re more comfortable there ;)

$ export SECRETS_MANAGER_URL=`ibmcloud resource service-instance ibmcloud-secrets-manager — output json | jq -r ‘.[].dashboard_url | .[0:-3]’`; echo $SECRETS_MANAGER_URL

… and the output

https://<id>.<region>.secrets-manager.appdomain.cloud

We will then add the following line to to our deployment script, to get the above applied:

sed "s|{{SECRETS_MANAGER_URL}}|${SECRETS_MANAGER}|" \
    "./external-secrets.yaml" | kubectl apply -f -

The Authentication

At the time or writing, the External Secrets Operator only supports API key authentication for IBM Secrets Manager. Let’s go to our Identity and Access Management (IAM) user-interface to create a service ID and the key.

Now we need to assign the service ID permissions to read secrets from Secrets Manager. By assigning SecretsReader service access, the External Secrets controller has the correct level of access to read secrets from Secrets Manager and populate them in a Kubernetes cluster.

Select Secrets Manager as the service we want to assign access to.

Scroll down, select the SecretsReader service access, and click the Add button.

Now let’s create an API key for our service ID.

We will be using the name of secret-api-key, which was set/used in our external-secrets.yaml file shared earlier in this post.

IMPORTANT: copy or save it because keys can’t be displayed or downloaded twice.

Then create a secret with your newly created API key:

kubectl create secret generic secret-api-key \
    --from-literal=apiKey='API_KEY_VALUE'

Validation

Finally, validate the status of the stores (see the .yaml file metadata sections for names used below) using the following command:

$ kubectl describe SecretStores ibmcloud-secrets-manager
[...]
Events:
  Type    Reason  Age                      From          Message
  ----    ------  ----                     ----          -------
  Normal  Valid   3m31s (x770 over 2d21h)  secret-store  store validated
$ kubectl describe ExternalSecrets external-secret
[...]
Events:
  Type    Reason   Age                   From              Message
  ----    ------   ----                  ----              -------
  Normal  Updated  2m32s (x5 over 132m)  external-secrets  Updated Secret

… and that we are able to retrieve our secrets:

$ kubectl get secret external-secret -o json | jq '.data'
{
  "CLOUDANT_APIKEY": <base64 encoded data>,
  "CLOUDANT_DB": <base64 encoded data>,
  "CLOUDANT_URL": <base64 encoded data>,
  "SLACK_BOT_TOKEN": <base64 encoded data>,
  "SLACK_SIGNING_SECRET": <base64 encoded data>
}

Note that, by default, Kubernetes stores secrets as base64 encoded strings.

Now that we have the secrets synchronised, we will have to update our deployment.yaml file, from our previous post, to reflect these changes:

Before:

    env:
      - name: CLOUDANT_APIKEY
        valueFrom:
          secretKeyRef:
            name: cloudant-apikey
            key: CLOUDANT_APIKEY
      - name: CLOUDANT_DB
        valueFrom:
          secretKeyRef:
            name: cloudant-db
            key: CLOUDANT_DB
      - name: CLOUDANT_URL
        valueFrom:
          secretKeyRef:
            name: cloudant-url
            key: CLOUDANT_URL
      - name: SLACK_BOT_TOKEN
        valueFrom:
          secretKeyRef:
            name: slackbot-token
            key: SLACK_BOT_TOKEN
      - name: SLACK_SIGNING_SECRET
        valueFrom:
          secretKeyRef:
            name: slack-signing-secret
            key: SLACK_SIGNING_SECRET

After:

env:
      - name: CLOUDANT_APIKEY
        valueFrom:
          secretKeyRef:
            name: external-secret
            key: CLOUDANT_APIKEY
      - name: CLOUDANT_DB
        valueFrom:
          secretKeyRef:
            name: external-secret
            key: CLOUDANT_DB
      - name: CLOUDANT_URL
        valueFrom:
          secretKeyRef:
            name: external-secret
            key: CLOUDANT_URL
      - name: SLACK_BOT_TOKEN
        valueFrom:
          secretKeyRef:
            name: external-secret
            key: SLACK_BOT_TOKEN
      - name: SLACK_SIGNING_SECRET
        valueFrom:
          secretKeyRef:
          name: external-secret
          key: SLACK_SIGNING_SECRET

Now, at application run time, the secrets retrieved from IBM Secrets Manager are converted to Kubernetes secrets that can be used by our cluster. Hooray!

Conclusion

Thanks for reading. I will leave here couple of interesting links if you want to go above and beyond with IBM Secrets Manager and the External Secrets Operator:

The list of secrets types from IBM Secrets Manager supported by External Secrets Operator
The IBM Secrets Manager API documentation

Feel free to reach out and provide feedback. Until then, stay safe, and keep on learning new things!