Clever Geek Handbook

如何使用grok导出器从非结构化日志生成Prometheus指标

将有2个关于grok出口商的文章的翻译。



第一译：如何使用grok导出器从非结构化日志中生成Prometheus指标



让我们谈谈希腊出口商。在本文中，我将解释如何使用grok导出器从非结构化日志中生成Prometheus度量。





Grok ELK (ElasticSearch, Logstash, Kibana) Fabian Stäber grok exporter.



grok exporter => https://github.com/fstab/grok_exporter



1: Grok exporter



zip grok exporter https://github.com/fstab/grok_exporter/releases.



  1. ​ (releases) ( v0.2.7).
  2. ​ zip-, . — 64- Linux. .


wget https://github.com/fstab/grok_exporter/releases/download/v0.2.7/grok_exporter-0.2.7.linux-amd64.zip


  1. ​ .
  2. ​ , grok exporter.


[root@localhost grok_exporter-0.2.7.linux-amd64]# ./grok_exporter -config ./config.yml
Starting server on http://localhost.localdomain:9144/metrics


http://localhost.localdomain:9144/metrics.



2:



Grok exporter. , .



30.07.2016 04:33:03 10.3.4.1 user=Nijil message="logged in"
30.07.2016 06:47:03 10.3.4.2 user=Alex message="logged failed"
30.07.2016 06:55:03 10.3.4.2 user=Alex message="logged in"
30.07.2016 07:03:03 10.3.4.3 user=Alan message="logged in"
30.07.2016 07:37:03 10.3.4.1 user=Nijil message="logged out"
30.07.2016 08:47:03 10.3.4.2 user=Alex message="logged out"
30.07.2016 14:34:03 10.3.4.3 user=Alan message="logged out"


, . Prometheus .



1 , , config.xml, grok exporter. .



global:
    config_version: 2
input:
    type: file
    path: ./example/nijil.log  # Specify the location of the your log
    readall: true              # This should be True if you want to read whole log and False if you want to read only new lines.
grok:
    patterns_dir: ./patterns    
metrics:
    - type: counter
      name: user_activity
      help: Counter metric example with labels.
      match: "%{DATE} %{TIME} %{HOSTNAME:instance} user=%{USER:user} message=\"%{GREEDYDATA:data}\""
      labels:
          user    : '{{.user}}'

server:
    port: 9144


.



global:
    # Config version
input:
    # How to read log lines (file or stdin).
grok:
    # Available Grok patterns.
metrics:
    # How to map Grok fields to Prometheus metrics.
server:
    # How to expose the metrics via HTTP(S).


3: Grok exporter



, , Prometheus.



metrics:
    - type: counter
      name: user_activity
      help: Counter metric example with labels.
      match: "%{DATE} %{TIME} %{HOSTNAME:instance} user=%{USER:user} message=\"%{GREEDYDATA:data}\""
      labels:
          user    : '{{.user}}'


grok – %{SYNTAX:SEMANTIC}, SYNTAX — , , SEMANTIC – . %{HOSTNAME:instance}, HOSTNAME – grok, IP- , IP- ( ), . , SYNTAX , , IP- . , DATE, TIME, HOSTNAME, USER GREEDYDATA , , " " .



, , . , . , (SEMANTIC of the SYNTAX) . . . . (Counter), grok exporter , .



grok exporter ./grok_exporter -config ./config.yml . , user_activity, .



# TYPE user_activity counter
user_activity{user="Alan"} 2
user_activity{user="Alex"} 3
user_activity{user="Nijil"} 2


Prometheus . , prometheus, Prometheus .



: Apache grok exporter



https://www.robustperception.io/getting-metrics-from-apache-logs-using-the-grok-exporter



, .



, , , , . grok. , Apache, access.log:



x.x.x.x - - [20/Jan/2020:06:25:24 +0000] "GET / HTTP/1.1" 200 62316 "http://178.62.121.216" "Go-http-client/1.1"
x.x.x.x - - [20/Jan/2020:06:25:25 +0000] "GET / HTTP/1.1" 200 16061 "-" "Go-http-client/1.1"
x.x.x.x - - [20/Jan/2020:06:25:25 +0000] "GET / HTTP/1.1" 200 16064 "-" "Go-http-client/1.1"
x.x.x.x - - [20/Jan/2020:06:25:25 +0000] "GET /blog/rss HTTP/1.1" 301 3478 "-" "Tiny Tiny RSS/19.2 (adc2a51) (http://tt-rss.org/)"
x.x.x.x - - [20/Jan/2020:06:25:26 +0000] "GET / HTTP/1.1" 200 16065 "-" "Go-http-client/1.1"
x.x.x.x - - [20/Jan/2020:06:25:26 +0000] "GET /blog/feed HTTP/1.1" 200 3413 "-" "Tiny Tiny RSS/19.2 (adc2a51) (http://tt-rss.org/)"
x.x.x.x - - [20/Jan/2020:06:25:27 +0000] "GET /feed HTTP/1.1" 200 6496 "-" "Emacs Elfeed 3.2.0"
x.x.x.x - - [20/Jan/2020:06:25:27 +0000] "GET / HTTP/1.1" 200 62316 "http://178.62.121.216" "Go-http-client/1.1"


:



wget https://github.com/fstab/grok_exporter/releases/download/v1.0.0.RC2/grok_exporter-1.0.0.RC2.linux-amd64.zip




unzip grok_exporter-*.zip
cd grok_exporter*amd64


:



cat << 'EOF' > config.yml
global:
    config_version: 2
input:
    type: file
    path: access.log
    readall: true
grok:
    patterns_dir: ./patterns
metrics:
    - type: counter
      name: apache_http_response_codes_total
      help: HTTP requests to Apache
      match: '%{COMBINEDAPACHELOG}'
      labels:
          method: '{{.verb}}'
          path: '{{.request}}'
          code: '{{.response}}'
server:
    port: 9144
EOF


grok exporter:



./grok_exporter -config config.yml


http://localhost:9144/metrics :



# HELP apache_http_response_codes_total HTTP requests to Apache
# TYPE apache_http_response_codes_total counter
apache_http_response_codes_total{code="200",method="GET",path="/"} 5
apache_http_response_codes_total{code="200",method="GET",path="/blog/feed"} 1
apache_http_response_codes_total{code="200",method="GET",path="/feed"} 1
apache_http_response_codes_total{code="301",method="GET",path="/blog/rss"} 1


readall, , .



Grok – , Logstash (Logstash — L ELK). , , Apache. Grok , , . COMMMONAPACHELOG , , 



COMMONAPACHELOG %{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)


. , . Go ( Prometheus alerting notification) .



, Grok , :



    - type: summary
      name: apache_http_response_bytes
      help: Size of HTTP responses
      match: '%{COMMONAPACHELOG}'
      value: '{{.bytes}}'


, , :



    - type: gauge 
      name: apache_http_last_request_seconds
      help: Timestamp of the last HTTP request
      match: '%{COMMONAPACHELOG}'
      value: '{{timestamp "02/Jan/2006:15:04:05 -0700" .timestamp}}'


(timestamp) grok exporter, time.Parse Golang. (divide), .



, Grok. , , , .





More articles:


All Articles