Как добавить конфиг в prometheus?

Question

[shell_guy]

У меня подняты несколько докер конейнеров Grafana + Prometheus + Alertmanager и пр.

Задача заключается в том, чтобы добавить проверку ICMP и SNMP в один файл конфигурации.
Пытаюсь добавить, но докер контейнер с Prometheus постоянно рестартится. Фейлиться именно когда добавляю блок job snmp.

Вот конфиг прометеус:

# my global config
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # By default, scrape targets every 15 seconds.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'my-project'

# Load and evaluate rules in this file every 'evaluation_interval' seconds.
rule_files:
  - 'alert.rules'
  # - "first.rules"
  # - "second.rules"

# alert
alerting:
  alertmanagers:
  - scheme: http
    static_configs:
    - targets:
      - "alertmanager:9093"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
  - job_name: 'node'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
         - targets:
           - localhost:9090
           - cadvisor:8080
           - node-exporter:9100
           
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]  # Look for a HTTP 200 response.
    static_configs:
      - targets:
        - https://www.google.com
       
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: "blackbox_exporter:9115"  # Blackbox exporter.
      - target_label: region
        replacement: "local"

    scrape_configs:
      - job_name: 'snmp'
    static_configs:
      - targets:
        - 192.168.0.42  # SNMP device.
        - switch.local # SNMP device.
    metrics_path: /snmp
    params:
      module: [if_mib]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9116  # The SNMP exporter's real hostname:port.

Что нужно поправить в конфиге, чтобы он заработал?

Comments

[Александр Карабанов]

Конфиг Prometheus на валидность можно проверять так:

docker exec -it <prometheus_container_name> promtool check config /etc/prometheus/prometheus.yml

С Aertmanager так же, только утилита называется amtool, ну и имя конфига иное, соответственно.

Answer 1

[unseriously]

Как именно фейлится, что говорит?
docker logs <container-id> что выдает?
Контейнер с блэкбоксом работает?
У меня крутится контейнер с промиком, рядом с ним крутится контейнер с блэкбоксом.

Comments

[shell_guy]

prom/blackbox-exporter:v0.10.0 работает,
rонфиг поправил нужно было просто убрать scape_configs , но вылезла новая проблема, пытаюсь добавить ICMP

[33;1mprometheus|[0m level=error ts=2021-10-04T13:03:39.538Z caller=main.go:360 msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" err="parsing YAML file /etc/prometheus/prometheus.yml: yaml: line 75: did not find expected key"

Актуальный конфиг:

global:
scrape_interval: 15s # By default, scrape targets every 15 seconds.
evaluation_interval: 15s # By default, scrape targets every 15 seconds.

# Attach these labels to any time series or alerts when communicating with
# external systems (federation, remote storage, Alertmanager).
external_labels:
monitor: 'my-project'

# Load and evaluate rules in this file every 'evaluation_interval' seconds.
rule_files:
- 'alert.rules'
# - "first.rules"
# - "second.rules"

# alert
alerting:
alertmanagers:
- scheme: http
static_configs:
- targets:
- "alertmanager:9093"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=` to any timeseries scraped from this config.
- job_name: 'prometheus'
- job_name: 'node'

# Override the global default and scrape targets from this job every 5 seconds.
scrape_interval: 5s

# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.

static_configs:
- targets:
- localhost:9090
- cadvisor:8080
- node-exporter:9100
- job_name: 'blackbox'
metrics_path: /probe
params:
module: [http_2xx] # Look for a HTTP 200 response.
static_configs:
- targets:
- https://www.google.com

relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: "blackbox_exporter:9115" # Blackbox exporter.
- target_label: region
replacement: "local"

- job_name: 'snmp'
metrics_path: /snmp
params:
module: [if_mib]
static_configs:
- targets:
- 192.168.0.42 # SNMP device - add your IPs here
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: 192.168.3.219:9116 # SNMP exporter

- job_name: 'blackbox-icmp-general'
scrape_interval: 30s
metrics_path: /probe
params:
module: [icmp-general]
static_configs:
- targets:
- 192.168.1.1 # server1
- 192.168.1.2 # server2
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: blackbox-exporter:9115

[Александр Карабанов]

err="parsing YAML
Не валидный синтаксис. Используй тектстовый редактор с подстветкой синтаксиса YAML.
Пользуйся линтером yamllint для проверки синтаксиса YAML.

[unseriously]

у прометеуса жеж есть тулза для проверки конфигов:

/usr/local/bin/promtool check config /etc/prometheus/prometheus.yml