But isn’t Single Sign On already fun and easy?

All joking aside, SSO can be tricky to implement securely. Below, I’ll take you through the process I used to enable SSO for Prometheus with TLS from the ground up, on all internal and external facing services.

I leveraged Pomerium, a context and identity aware gateway similar to Google’s IAP tooling but free and open source. Additionally, Pomerium offers an Enterprise setup with advanced features for self-service, access controls, audit logs, and more.

I used Google as our identity provider to be able to apply Google Groups to our configuration—although this setup does not specifically make use of it. This process is almost entirely executed in Infrastructure as Code onto a Kubernetes cluster. I also used Github Actions to automate the process, which is documented here.

I started with an existing cluster hosted on Elastic Kubernetes Service (EKS) by AWS (cluster creation is outside the scope of this post).

Prerequisites:

• An existing EKS cluster
• A top level domain for which to create records
• A Google IDP. For specific instructions, reference this helpful doc. 
• EKS cluster - for the purposes of this demo you can use AdministratorAccess. For more info see https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage-attach-detach.html 
• An AWS user with the ability to assume the role Administrator, whose credentials must be added to the Github secrets for the repository as dev_AWS_ACCESS_KEY_ID or prod_AWS_ACCESS_KEY_ID and dev_AWS_SECRET_ACCESS_KEY or prod_AWS_SECRET_ACCESS_KEY. The Administrator role must have the following Trust Relationship

{
    "Sid": "",
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::$ACCOUNT:user/$USERNAME"
    },
    "Action": [
        "sts:AssumeRole",
        "sts:TagSession"
    ]
}

I also created three SSM parameters out of band with the following keys:

• /cluster/${var.env}/POMERIUM_GOOGLE_CLIENT_ID
• /cluster/${var.env}/POMERIUM_GOOGLE_CLIENT_SECRET
• /cluster/${var.env}/POMERIUM_GOOGLE_SERVICE_ACCOUNT
• For details on how to do this, check out the AWS documentation on creating a Systems Manager parameter.  

The action is executed on workflow dispatch—and looks like this:

Certificates and Issuer yamls

I could’ve optimized this by adding the cert-manager resources into Terraform, as there’s a cert-manager provider. However, I preferred to keep the certificates out of the Terraform state. This is because if there were any issues with the certificate order not getting fulfilled, it wouldn’t break my entire state. Ideally, that wouldn’t be an issue.

In the following example, change “admin@example.com” to the email address you want to use to register with Let’s Encrypt.

I used the following Certificates and Issuer yamls:

issuer.yaml

certificates.yaml

cert-manager-clusterissuer.yaml

Terraform time

IIt’s time for the Terraform. As you’ll see below, I did not include my tfvars files, as you’ll need to supply values for all the variables that relate to your environment in a dev.tfvars or prod.tfvars file.  An incomplete dev.tfvars example would look like the following:

dev.tfvars

Fill in the variables in dev.tfvars as the variables.tf file requires.

backend-dev.conf

provider.tf

variables.tf

Prometheus

The following file is the values file that was supplied to Prometheus. Make sure to swap in your IDP email domain into the allowed policy for the Pomerium Ingress Controller. If the email you use to log into Google is myemail@exampledomain.com, you would swap in “exampledomain.com” instead of “<< your IDP email domain here >>”.

The main.tf file provisions Route53 records with subdomains ops.yourdomain.com and ops.dev.yourdomain.com .  Adjust these in the “locals” value as you see fit.

prometheus-stack-1.yaml

main.tf

Fun! Easy!

Notice that the Terraform points to the authenticate DNS record, as well as my custom DNS record for Prometheus to the Pomerium Proxy Load Balancer hostname.  

That’s it! Once this is all applied, you should have a custom link to your Prometheus installation that prompts you for a Google account before redirecting you to the metrics landing page.

Have questions, thoughts, or feedback? Join the conversation in the Pomerium community.