Logstash-支持的输入

📌 相关文章

📜 Logstash-支持的输入

📅 最后修改于: 2020-11-01 04:22:03 🧑 作者: Mango

Logstash支持来自不同来源的大量日志。它将与著名资源合作，如下所述。

从指标收集日志

系统事件和其他时间活动记录在指标中。 Logstash可以从系统指标访问日志，并使用过滤器对其进行处理。这有助于以定制的方式向用户显示事件的实时提要。根据指标过滤器的flush_interval设置刷新指标，默认情况下；它设置为5秒。

通过收集和分析通过Logstash运行的事件并在命令提示符下显示实时提要，我们正在跟踪Logstash生成的测试指标。

logstash.conf

此配置包含一个生成器插件，Logstash提供了该插件以用于测试指标，并将类型设置设置为“ generated”以进行解析。在过滤阶段，我们仅使用’if’语句处理生成类型的行。然后，指标插件对仪表设置中指定的字段进行计数。指标插件会在flush_interval中指定的每5秒刷新一次计数。

最后，使用编解码器插件将过滤器事件输出到标准输出(如命令提示符)，以进行格式化。编解码器插件正在使用[ events ] [ rate_1m ]值在1分钟的滑动窗口中输出每秒的事件。

input {
   generator {
         type => "generated"
   }
}
filter {
   if [type] == "generated" {
      metrics {
         meter => "events"
         add_tag => "metric"
      }
   }
}
output {
   # only emit events with the 'metric' tag
   if "metric" in [tags] {
      stdout {
         codec => line { format => "rate: %{[events][rate_1m]}"
      }
   }
}

运行Logstash

我们可以使用以下命令运行Logstash。

>logsaths –f logstash.conf

标准输出(命令提示符)

rate: 1308.4
rate: 1308.4
rate: 1368.654529135342
rate: 1416.4796003951449
rate: 1464.974293984808
rate: 1523.3119444107458
rate: 1564.1602979542715
rate: 1610.6496496890895
rate: 1645.2184750334154
rate: 1688.7768007612485
rate: 1714.652283095914
rate: 1752.5150680019278
rate: 1785.9432934744932
rate: 1806.912181962126
rate: 1836.0070454626025
rate: 1849.5669494173826
rate: 1871.3814756851832
rate: 1883.3443123790712
rate: 1906.4879113216743
rate: 1925.9420717997118
rate: 1934.166137658981
rate: 1954.3176526556897
rate: 1957.0107444542625

从Web服务器收集日志

Web服务器会生成大量有关用户访问和错误的日志。 Logstash有助于使用输入插件从不同的服务器提取日志并将其存储在集中位置。

我们正在从本地Apache Tomcat服务器的stderr日志中提取数据，并将其存储在output.log中。

logstash.conf

该Logstash配置文件指示Logstash读取apache错误日志并添加名为“ apache-error”的标签。我们可以简单地使用文件输出插件将其发送到output.log。

input {
   file {
      path => "C:/Program Files/Apache Software Foundation/Tomcat 7.0 /logs/*stderr*"
      type => "apache-error"  
   }
} 
output {
   file {
      path => "C:/tpwork/logstash/bin/log/output.log"
   }
}

运行Logstash

我们可以使用以下命令运行Logstash。

>Logstash –f Logstash.conf

输入日志样本

这是示例stderr日志，该日志在Apache Tomcat中发生服务器事件时生成。

C：\ Program Files \ Apache Software Foundation \ Tomcat 7.0 \ logs \ tomcat7-stderr.2016-12-25.log

Dec 25, 2016 7:05:14 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-9999"]
Dec 25, 2016 7:05:14 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["ajp-bio-8009"]
Dec 25, 2016 7:05:14 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 823 ms

输出日志

{
   "path":"C:/Program Files/Apache Software Foundation/Tomcat 7.0/logs/
   tomcat7-stderr.2016-12-25.log","@timestamp":"2016-12-25T11:05:27.045Z",
   "@version":"1","host":"Dell-PC",
   "message":"Dec 25, 2016 7:05:14 PM org.apache.coyote.AbstractProtocol start\r",
   "type":"apache-error","tags":[]
}
{
   "path":"C:/Program Files/Apache Software Foundation/Tomcat 7.0/logs/
   tomcat7-stderr.2016-12-25.log","@timestamp":"2016-12-25T11:05:27.045Z",
   "@version":"1","host":"Dell-PC",
   "message":"INFO: Starting ProtocolHandler [
      \"ajp-bio-8009\"]\r","type":"apache-error","tags":[]
}
{
   "path":"C:/Program Files/Apache Software Foundation/Tomcat 7.0/logs/
   tomcat7-stderr.2016-12-25.log","@timestamp":"2016-12-25T11:05:27.045Z",
   "@version":"1","host":"Dell-PC",
   "message":"Dec 25, 2016 7:05:14 PM org.apache.catalina.startup.Catalina start\r",
   "type":"apache-error","tags":[]
}
{
   "path":"C:/Program Files/Apache Software Foundation/Tomcat 7.0/logs/
   tomcat7-stderr.2016-12-25.log","@timestamp":"2016-12-25T11:05:27.045Z",
   "@version":"1","host":"Dell-PC",
   "message":"INFO: Server startup in 823 ms\r","type":"apache-error","tags":[]
}

从数据源收集日志

首先，让我们了解如何配置MySQL进行日志记录。在[mysqld]下的MySQL数据库服务器的my.ini文件中添加以下行。

在Windows中，它位于MySQL的安装目录中，该目录位于-

C:\wamp\bin\mysql\mysql5.7.11

在UNIX中，可以在/etc/mysql/my.cnf中找到它。

general_log_file   = "C:/wamp/logs/queries.log"
general_log = 1

logstash.conf

在此配置文件中，文件插件用于读取MySQL日志并将其写入ouput.log。

input {
   file {
      path => "C:/wamp/logs/queries.log"
   }
}
output {
   file {
      path => "C:/tpwork/logstash/bin/log/output.log"
   }
}

query.log

这是在MySQL数据库中执行的查询生成的日志。

2016-12-25T13:05:36.854619Z   2 Query        select * from test1_users
2016-12-25T13:05:51.822475Z    2 Query    select count(*) from users
2016-12-25T13:05:59.998942Z    2 Query         select count(*) from test1_users

输出日志

{
   "path":"C:/wamp/logs/queries.log","@timestamp":"2016-12-25T13:05:37.905Z",
   "@version":"1","host":"Dell-PC",
   "message":"2016-12-25T13:05:36.854619Z    2 Query\tselect * from test1_users",
   "tags":[]
}
{
   "path":"C:/wamp/logs/queries.log","@timestamp":"2016-12-25T13:05:51.938Z",
   "@version":"1","host":"Dell-PC",
   "message":"2016-12-25T13:05:51.822475Z    2 Query\tselect count(*) from users",
   "tags":[]
}
{
   "path":"C:/wamp/logs/queries.log","@timestamp":"2016-12-25T13:06:00.950Z",
   "@version":"1","host":"Dell-PC",
   "message":"2016-12-25T13:05:59.998942Z    2 Query\tselect count(*) from test1_users",
   "tags":[]
}