Cluster Status
The Status page of the Zero Console displays notifications and alerts regarding the health of a cluster. If you see an alert in the Status page, it means something may be wrong with your cluster and you should take steps to resolve it.
We've addressed all the cluster status alerts below. Refer to the relevant alert to learn how to fix it.
Certificates are expired or invalid
This cluster status alert inspects a cluster's certificates to see if they expire within 10 days. It runs daily, and every time the cluster configuration is updated.
What causes this alert:
- If your cluster uses managed certificates provisioned by Pomerium, then this error is a bug. Please contact support.
- If you manually uploaded your own custom certificate, then you must upload a new certificate.
Steps to resolve: Upload a valid certificate, or allow Pomerium to automate certificate management for you by adding a Custom Domain.
Hostnames do not have matching certificates
This cluster status alert ensures that all the hostnames available in a cluster's configuration have matching certificates. It runs every time the cluster configuration is updated.
What causes this alert: Either the From URL in a route block is incorrect, or the certificate is missing.
Steps to resolve: Make sure all From URLs are correct, and that the hostnames have matching certificates.
For more information on certificates in Pomerium, see the following pages:
Cluster configuration is not current
This cluster status alert makes sure the cluster is running the latest changeset. It runs daily to check for the latest configuration, or every time the cluster configuration is updated.
What causes this alert: This cluster status alert may indicate cluster connectivity issues. Specifically, Pomerium Zero can't connect to the cluster to apply the latest changeset to its configuration.
Steps to resolve: Restart the cluster. If that doesn't resolve the issue, please contact support.
Cluster version is not current
This cluster status alert makes sure the cluster replica is running the latest release. It runs daily.
What causes this alert: The replica is not running the latest version of Pomerium Zero.
Steps to resolve: Update your Pomerium Zero cluster configuration to run the latest version of Pomerium.
Cluster using an in-memory storage backend
This cluster status alert throws an error if it detects a cluster isn't connected to a persistent storage backend. It runs every time the cluster configuration is updated.
What causes this alert:
By default, a cluster uses an in-memory storage backend to synchronize replicas. This does not persist your configuration. For production environments, we recommend connecting each cluster instance to its own dedicated PostgreSQL database.
Steps to resolve: See the Databroker Storage Connection String reference page to learn how to connect to a database instance in Pomerium.
Incompatible configuration
This cluster status alert looks for configuration incompatibilities in a replica. It runs every time the cluster configuration is updated.
What causes this alert: This is caused by a version incompatibility in the cluster configuration. For example, you somehow apply a configuration change that isn't compatible with your version of Pomerium Zero.
Steps to resolve: If you see this warning, please contact support.
Cluster can't communicate with the storage backend
This cluster status alert monitors the connection to a PostgreSQL database. It runs every time the cluster configuration is updated.
What causes this alert: Pomerium can't connect to your PostgreSQL database.
Steps to resolve: Check the parameters of your Databroker Storage Connection String.
There's an issue with cluster configuration
This cluster status alert tells you if Envoy accepted the cluster configuration or not after a configuration update.
What causes this alert: If you see this warning, please contact support.
There's an issue with the network listener
This cluster status alert makes sure the network listener is configured correctly. It runs every time the cluster configuration is updated.
What causes this alert: Typically, this error is caused when Pomerium's network listener can't bind to the cluster's port. The error may also be the result of a non-root process with insufficient permissions attempting to bind to a privileged port.
Steps to resolve:
- If this is a port binding error, you may need to kill a process running on that port or specify a port to bind to.
- If this is a permissions issue, you may need to add the
CAP_NET_BIND_SERVICEcapability to the process.
There's an issue with the route configuration
This cluster status alert verifies the route configuration. It runs every time the cluster configuration is updated.
What causes this alert: A route misconfiguration in a cluster.
Steps to resolve: If you see this warning, it's likely a problem with Pomerium. Please contact support.
There's an issue with the configuration
This cluster status alert monitors xDS issues in a replica. It runs every time the cluster configuration is updated.
What causes this alert: If you see this warning, it's likely a problem with Pomerium. Please contact support.
Pomerium can't write some of its cache files
This cluster status alert makes sure Pomerium can persist a replica's bootstrap configuration. It runs when you start a cluster, or every time the cluster configuration is updated.
What causes this alert: Pomerium doesn't have sufficient permissions to read from a replica's configuration file.
Steps to resolve: Check your file permissions to make sure Pomerium has read access.
Cluster connectivity issues with the cloud
This cluster status alert monitors the connection between the Pomerium Zero cloud and a cluster. It maintains a consistent streaming connection.
What causes this alert: Connectivity issues between Pomerium Zero and a cluster.
Steps to resolve: You may experience this issue if you're accessing Pomerium Zero behind a firewall. Check your firewall settings to make sure it allows HTTP and HTTPS traffic.