1.6 Container Monitoring

1. Container Monitoring Tools

Container monitoring tools are specialized software designed to track, analyze, and report on the performance, health, and resource usage of containers across an environment. These tools collect metrics like CPU, memory, network usage, and storage I/O, helping you to understand the state of your applications and infrastructure. They often provide alerts, dashboards, and logs to assist with troubleshooting and performance optimization.

Examples of container monitoring tools include:

• Prometheus: Collects and stores metrics with powerful querying capabilities.
• Grafana: Visualizes data from various sources, including Prometheus.
• cAdvisor: Monitors resource usage and performance characteristics of containers.
• Datadog, New Relic, Zabbix: Provide comprehensive monitoring, including custom metrics, logs, and tracing.

1.1 Monitoring Tools Details

1. Prometheus: An open-source monitoring and alerting toolkit designed for reliability. Prometheus scrapes metrics from containers via HTTP endpoints and supports alerting and data visualization through Grafana.

2. cAdvisor: Container Advisor (cAdvisor) is a Google-developed tool that provides insights into resource usage and performance characteristics of running containers. It can integrate with Prometheus to export metrics.

3. Grafana: While primarily a visualization tool, Grafana works with Prometheus to visualize metrics from Docker containers in dashboards. It supports multiple data sources and is commonly paired with Prometheus.

4. Datadog: A commercial monitoring and analytics tool that provides detailed insights into container performance, logs, and infrastructure. It offers integrations for Docker and Kubernetes with automated dashboards.

5. Zabbix: An open-source monitoring solution that supports Docker monitoring via its agent. It provides alerts, dashboards, and analytics.

6. New Relic: A commercial monitoring platform that provides real-time performance monitoring for Docker containers, offering insights into application performance, infrastructure, and logs.

These tools help monitor the health, performance, and resource usage of Docker containers, ensuring that your applications run smoothly in a containerized environment.

1.2 Difference from Docker Desktop

Docker Desktop is an application that provides a development environment for building, shipping, and running Docker containers on a local machine. It includes Docker Engine, Docker CLI, Docker Compose, and Kubernetes, enabling developers to run and manage containers on their desktops.

Key differences:

1. Purpose:

• Container Monitoring Tools focus on monitoring, alerting, and analyzing container performance across environments (production, staging, etc.).
• Docker Desktop is for local development, allowing you to build, run, and manage containers on your computer.

2. Scope:

• Monitoring Tools work across large-scale, distributed systems, monitoring many containers, often in cloud or on-premise environments.
• Docker Desktop is limited to managing and running containers locally, with some basic stats and management via its UI.

3. Usage:

• Monitoring Tools are essential for production environments where container performance, uptime, and health are critical.
• Docker Desktop is used for development, testing, and debugging applications before they are deployed to production.

TIP

container monitoring tools are designed to manage and monitor containerized applications in large-scale environments, while Docker Desktop is a local development tool for working with containers on your machine.

1. cAdvisor

docker run --volume=/:/rootfs:ro --volume=/var/run:/var/run:ro --volume=/sys:/sys:ro  --volume=/var/lib/docker:/var/lib/docker:ro --publish=8080:8080 --detach --name=cadvisor gcr.io/cadvisor/cadvisor

The command will work on Docker for Windows if you’re using Docker Desktop with the WSL 2 backend or Hyper-V. Docker Desktop for Windows allows container paths such as /var/run to work by using a virtualized Linux environment that maps these paths from the Linux file system within the virtual machine (VM) running the containers.

Here’s how it works:

1. WSL 2 (Windows Subsystem for Linux): Docker uses the WSL 2 backend to run containers in a lightweight VM. The paths like /var/run exist in the Linux VM, so when you mount them, Docker can interpret and manage these paths inside the Linux environment, not directly on the Windows file system.

2. Volume mounting: When you use --volume, Docker mounts the host file system into the container’s file system. Since Docker Desktop uses Linux VM for containers, it treats paths like /var/run as part of the Linux environment. Windows itself doesn’t have these paths, but Docker bridges this gap by providing a Linux kernel in the VM.

Tip

Paths like /var/run and /sys work because the containers run inside a virtualized Linux environment that contains those directories, making it transparent for the user running Docker on Windows.

2. Prometheus

Prometheus can scrape metrics from other containers by using service discovery mechanisms or static configurations.

1. Service Discovery: Prometheus can automatically discover targets using various service discovery methods like Kubernetes, Docker Swarm, Consul, etc. In the context of containers, if you’re using Kubernetes, Prometheus can discover pods, services, and endpoints automatically by labeling and annotating them. For Docker, Prometheus can discover containers by using the Docker daemon API.

2. Static Configuration: If service discovery is not available, you can manually define the IP addresses and ports of the containers you want to monitor. This is less dynamic but straightforward for small environments.

3. Prometheus Exporters: Containers typically expose their metrics via HTTP endpoints in a format that Prometheus understands (e.g., /metrics). You can either:

   • Run applications with built-in Prometheus metrics endpoints.
   • Use Prometheus exporters like node-exporter, cadvisor, etc., to expose container metrics.

Example Configuration:

For Kubernetes, the Prometheus prometheus.yml might look like this:

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        action: keep
        regex: myapp

In this setup, Prometheus will discover all pods labeled with app=myapp and scrape their metrics.

For Docker, you might configure Prometheus to scrape based on static IPs and ports:

scrape_configs:
  - job_name: 'my-docker-container'
    static_configs:
      - targets: ['container_ip:port']

These configurations allow Prometheus to monitor and collect metrics from other containers.

2.1 Simple Example

Play

Walkthrough

1. Create a Configuration File

Create a file named prometheus.yml with the following content:

global:
  scrape_interval: 15s  # How often to scrape targets

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

This configuration tells Prometheus to scrape metrics from itself (running on localhost:9090).

2. Run Prometheus Using Docker

Start Prometheus with Docker:

docker run -d \
  --name=prometheus \
  -p 9090:9090 \
  -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus

This command runs Prometheus, mapping port 9090 on your machine to the container, and uses the configuration file you created.

3. Access Prometheus

Open your web browser and go to http://localhost:9090. You’ll see the Prometheus dashboard.

4. Query Metrics

In the Prometheus UI, you can run queries to see metrics. For example, try:

up

This query shows if the Prometheus server itself is up and running.

TIP

This example sets up Prometheus using Docker, monitors itself, and provides a web UI for querying metrics. You can extend this setup by adding more targets or exporters to monitor other services.

2.2 Docker Daemon Metrics

2.2.1 Create /etc/docker/daemon.json

In the Docker daemon configuration, the setting:

{
  "metrics-addr": "127.0.0.1:9323"
}

enables Docker to expose internal metrics on the specified address (127.0.0.1:9323) in a format that Prometheus can scrape. Here’s a breakdown of what it means:

• metrics-addr: This option tells Docker to start an HTTP server at the given address and port that exposes metrics about the Docker daemon.

• 127.0.0.1:9323: This is the address where the Docker daemon will serve metrics.

  • 127.0.0.1: This limits the metrics endpoint to only be accessible from the local machine (localhost).
  • 9323: This is the default port where Docker metrics will be available for Prometheus scraping.

Once enabled, Prometheus can be configured to scrape this endpoint (e.g., localhost:9323) to collect metrics related to Docker, such as container states, resource usage, and other performance metrics related to the Docker daemon.

Example Prometheus scrape_configs to include Docker metrics:

scrape_configs:
  - job_name: 'docker'
    static_configs:
      - targets: ['localhost:9323']

This allows Prometheus to gather information about the Docker daemon’s internal metrics.

2.2.2 Create prometheus.yml

global:
  scrape_interval: 15s  # How often to scrape targets

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: docker
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    static_configs:
    - targets: ["host.docker.internal:9323"]

2.2.3 Docker command

docker run -d --name=prometheus -p 9090:9090 -v ${pwd}/prometheus.yml:/etc/prometheus/prometheus.yml --add-host=host.docker.internal:host-gateway prom/prometheus

The host-gateway is a special DNS name introduced in Docker that resolves to the IP address of the host machine (the system running Docker) from inside the Docker container. It allows containers to communicate with services running on the host machine without needing to hard-code the host’s IP address, which can change depending on the network configuration.

How to Configure host-gateway

You can use host-gateway in your Docker run command as part of the --add-host flag to map the host.docker.internal domain to the gateway IP of the host system.

Here’s how you can configure it:

1. Use the --add-host flag: Add the host.docker.internal mapping to the host-gateway using this format:

   docker run -d --name=prometheus -p 9090:9090 \
   -v C:/Users/chath/docker_workspace/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml \
   --add-host=host.docker.internal:host-gateway \
   prom/prometheus

2. Ensure Docker Daemon supports host-gateway: This feature was introduced in Docker 20.10. If you are using an older version, you might need to update Docker to access this functionality.

3. No manual configuration needed: Docker internally resolves host-gateway to the host’s gateway IP. The gateway IP is automatically determined by Docker, so you don’t need to manually configure it.

After this, you can access the host system from within the container by using the hostname host.docker.internal.

2.3 Another Example

Prometheus to monitor Docker containers using Docker Compose:

This setup will monitor the Docker containers using cAdvisor and node_exporter and provide the data to Prometheus.

docker-compose.yml:

version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker:/var/lib/docker:ro
    ports:
      - "8080:8080"

  node_exporter:
    image: prom/node-exporter:latest
    container_name: node_exporter
    ports:
      - "9100:9100"

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"

prometheus.yml (basic Prometheus configuration):

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']

  - job_name: 'node_exporter'
    static_configs:
      - targets: ['node_exporter:9100']

Steps:

1. Create files:

   • Create docker-compose.yml and prometheus.yml files in the same directory.

2. Run the services:

   • In the same directory as the docker-compose.yml, run the following command to start the services:

     docker-compose up -d

3. Access Prometheus:

   • Open a browser and go to http://localhost:9090 to access Prometheus.

4. Grafana Dashboard:

   • Grafana is available at http://localhost:3000. You can log in with the default credentials (admin/admin), add Prometheus as a data source, and create dashboards.

Direct Monitoring of Containers without Cadvisor

You can monitor Docker containers directly in Prometheus without cAdvisor by using the Docker daemon’s built-in metrics, which can be exposed via the Docker API. However, this provides only basic metrics about the Docker daemon and containers (such as memory, CPU usage, etc.). cAdvisor gives more detailed container-level metrics, including disk I/O, network, and per-container statistics.

Steps to Directly Monitor Containers with Prometheus via Docker API:

1. Enable Docker Metrics: Ensure the Docker daemon is configured to expose metrics. Modify /etc/docker/daemon.json (or create it if it doesn’t exist) with the following configuration:

   {
     "metrics-addr": "127.0.0.1:9323",
     "experimental": true
   }

   • metrics-addr: Exposes metrics at 127.0.0.1:9323.
   • experimental: Enables Docker’s experimental features, including metric reporting.

2. Restart Docker: After modifying the configuration, restart the Docker daemon to apply changes:

   sudo systemctl restart docker

3. Configure Prometheus: Add the Docker metrics endpoint to the Prometheus configuration (prometheus.yml):

   scrape_configs:
     - job_name: 'docker'
       static_configs:
         - targets: ['localhost:9323']

4. Run Prometheus: Start or reload Prometheus. It will now scrape Docker’s metrics exposed at localhost:9323.

Limitations:

• Docker API: The Docker metrics exposed here are more focused on the daemon’s health and resource usage, rather than detailed per-container metrics (like individual container I/O, network stats, etc.).
• cAdvisor: Provides far more detailed, container-specific metrics such as:
  • CPU, memory, and disk usage for each container
  • Per-container network stats
  • Historical data on resource usage

If you need detailed per-container metrics, it’s recommended to use cAdvisor alongside Prometheus.