Most frequently Asked Prometheus Interview Questions

1. What is Prometheus?
2. How to persist data in Prometheus running in a Docker container?
3. How can we join two metrics in a Prometheus query?
4. How to calculate containers' cpu usage in kubernetes with prometheus as monitoring?
5. How can we group labels in a Prometheus query?
6. How to use the selected period of time in a query?
7. How can we increase Prometheus storage retention?
8. How can we gracefully avoid divide by zero in Prometheus?
9. How to add extra label to Prometheus metrics?

What is Prometheus?

Prometheus is developed at soundcloud and integrated into the CNCF, it has the ability for creating user specific customs alerts and notifications which is based on the data through metrics.
Prometheus is also used for running ad-hoc queries and minor issues during debugging, its aspect is utilized when integrating with visualization backends.

How to persist data in Prometheus running in a Docker container?

For using this line instead of what you have in your command:

...
--volume a-new-volume:/prometheus \
...

How can we join two metrics in a Prometheus query?

We can use the consul exporter for ingesting the health and status of our service in Prometheus and the fire alerts when the status of services and nodes in Consul is critical.
We can join metrics by using the following command:

(
  max(consul_health_service_status{status="critical"})
  by (service_name,status,node) == 1
)
   + on(service_name,node) group_left(env)
(
   0 * consul_service_tags
)

How to calculate containers' cpu usage in kubernetes with prometheus as monitoring?

We can get the CPU usage at cluster level by:

sum (rate (container_cpu_usage_seconds_total{id="/"}[1m])) / sum (machine_cpu_cores) * 100

We can track the CPU usage by using:

sum (rate (container_cpu_usage_seconds_total{image!=""}[1m])) by (pod_name)

How can we group labels in a Prometheus query?

We can use the label replace to group all the misc together:

sum by (new_group) (
  label_replace(
    label_replace(my_metric, "new_group", "$1", "group", ".+"),
    "new_group", "misc", "group", "misc group.+"
  )
)

How to use the selected period of time in a query?

We can use the $__interval variable like this:

use the $__interval variable like this:

How can we increase Prometheus storage retention?

We can config the retention by using the following code:

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries \
    --web.external-url=http://34.89.26.156:9090 \
    --storage.tsdb.retention.time=1y
[Install]
WantedBy=multi-user.target

How can we gracefully avoid divide by zero in Prometheus?

rate({__name__="hystrix_command_latency_total_seconds_sum"}[60s])
rate({__name__="hystrix_command_latency_total_seconds_count"}[60s])

How to add extra label to Prometheus metrics?

We can use the following command for adding the extra label:

mongodb_exporter_last_scrape_duration_seconds{instance="127.0.0.1:9216",job="mongo"}

If we need to chande the label we can use the following command:

mongodb_exporter_last_scrape_duration_seconds{cluster="stage", instance="127.0.0.1:9216",job="mongo"}