Elastic Beat定制开发FastDFS集群监控 - FastDFS收集器代码-技术教程-四时宝库

Beat 使用go mod源码模式，编译打包make-like build tool使用 mage，mage编译构建不依赖外部工具完全基于go语音标准库实现。

Beat 6.8与7.x 版本差异

Beat 6.x : Beat生成使用 python script/generate.py ,源码管理使用gopath

Beat 7.x : Beat生成使用 mage 命令更简洁不依赖python环境,Index template 中使用 constant_keyword

Mage安装

执行go run会自动生成二进制文件，自动拷贝到 GOPATH/bin 路径下

git clone https://github.com/magefile/mage
cd mage
go run bootstrap.go

?  mage git:(master) go run bootstrap.go
Running target: Install
exec: go env GOBIN
exec: go env GOPATH
exec: git rev-parse --short HEAD
exec: git describe --tags
exec: go build -o /usr/local/bin/go/bin/mage -ldflags=-X "github.com/magefile/mage/mage.timestamp=2021-06-07T10:41:15+08:00" -X "github.com/magefile/mage/mage.commitHash=4cf3cfc" -X "github.com/magefile/mage/mage.gitTag=v1.11.0-2-g4cf3cfc" github.com/magefile/mage

GOPATH默认在 $HOME/go 路径下面，可以更改但不能设置为go安装目录。

If you would like to work in a different location, you will need to set GOPATH to the path to that directory. (Another common setup is to set GOPATH=$HOME.) Note that GOPATH must not be the same path as your Go installation.

mage 未生效则需配置 bin

vi /etc/profile
export PATH=$(go env GOPATH)/bin:$PATH

生成自定义 Beat

Beat 使用 mage 工具生成自定义 Beat 代码

mage generateCustomBeat

mage generateCustomBeat
Enter the beat name [examplebeat]: fastdfsbeat
Enter your github name [your-github-name]: yusys-cloud
Enter the beat path [github.com/yusys-cloud/fastdfsbeat]:
Enter your full name [Firstname Lastname]: y zq
Enter the beat type [beat]:
Enter the github.com/elastic/beats revision [master]:
go: creating new go.mod: module github.com/yusys-cloud/fastdfsbeat
go: to add module requirements and sums:
	go mod tidy
  ...
Generated fields.yml for fastdfsbeat to /root/go/src/github.com/yusys-cloud/fastdfsbeat/fields.yml
=======================
Your custom beat is now available as /root/go/src/github.com/yusys-cloud/fastdfsbeat
=======================

也可以在 metricbeat 库之上生成自定义 beat ，使用metricbeat生成 beat

cd ~/tmp/beat/beats-7.13.1
NEWBEAT_TYPE=metricbeat mage generateCustomBeat

编译运行

使用mage命令构建二进制文件

mage build

编译依赖gcc环境，gcc版本过低会提示如下错误：

/usr/local/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/usr/bin/ld: /tmp/go-link-973005765/000010.o: unrecognized relocation (0x2a) in section `.text'
/usr/bin/ld: final link failed: 错误的值
collect2: 错误：ld 返回 1

需执行升级gcc安装升级

yum install centos-release-scl -y
yum install devtoolset-7 -y
scl enable devtoolset-7 bash

FastDFS Beat 编译后可使用help命令查看包含 Easticsearch index 创建等命令

2021-06-08T14:23:08.127+0800	INFO	[template_loader]	template/load.go:143	Try loading template fastdfsbeat-8.0.0 to Elasticsearch
2021-06-08T14:23:08.233+0800	ERROR	instance/beat.go:971	Exiting: error loading template: could not load template. Elasticsearch returned: couldn't load template: 400 Bad Request: {"error":{"root_cause":[{"type":"mapper_parsing_exc
eption","reason":"No handler for type [constant_keyword] declared on field [namespace]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [doc]: No handler for type [constant_keyword] declared on field [namespace]","ca
used_by":{"type":"mapper_parsing_exception","reason":"No handler for type [constant_keyword] declared on field [namespace]"}},"status":400}. Response body: {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"No handler fo
r type [constant_keyword] declared on field [namespace]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [doc]: No handler for type [constant_keyword] declared on field [namespace]","caused_by":{"type":"mapper_parsin
g_exception","reason":"No handler for type [constant_keyword] declared on field [namespace]"}},"status":400}. Template is: {
  "index_patterns": [
    "fastdfsbeat-8.0.0-*"
  ],
  "mappings": {
    "doc": {
      "_meta": {
        "beat": "fastdfsbeat",

fastdfsbeat索引模版初始化

./fastdfsbeat setup

默认索引模版会使用 Elastic Common Schema (ECS)

2021-06-08T14:23:08.127+0800	INFO	[template_loader]	template/load.go:143	Try loading template fastdfsbeat-8.0.0 to Elasticsearch
2021-06-08T14:23:08.233+0800	ERROR	instance/beat.go:971	Exiting: error loading template: could not load template. Elasticsearch returned: couldn't load template: 400 Bad Request: {"error":{"root_cause":[{"type":"mapper_parsing_exc
eption","reason":"No handler for type [constant_keyword] declared on field [namespace]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [doc]: No handler for type [constant_keyword] declared on field [namespace]","ca
used_by":{"type":"mapper_parsing_exception","reason":"No handler for type [constant_keyword] declared on field [namespace]"}},"status":400}. Response body: {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"No handler fo
r type [constant_keyword] declared on field [namespace]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [doc]: No handler for type [constant_keyword] declared on field [namespace]","caused_by":{"type":"mapper_parsin
g_exception","reason":"No handler for type [constant_keyword] declared on field [namespace]"}},"status":400}. Template is: {
  "index_patterns": [
    "fastdfsbeat-8.0.0-*"
  ],
  "mappings": {
    "doc": {
      "_meta": {
        "beat": "fastdfsbeat",

手动导出默认的 index template

./fastdfsbeat export template > fastdfs.template.json

再修改为Beat 7.x template

{
  "index_patterns": [
    "fastdfsbeat-8.0.0-*"
  ],
  "mappings": {
    "_meta": {
      "beat": "fastdfsbeat",
      "version": "8.0.0"
    },
    "date_detection": false,
    "dynamic_templates": [
      {
        "fields": {
          "mapping": {
            "type": "keyword"
          },
          "match_mapping_type": "string",
          "path_match": "fields.*"
        }
      },
      {
        "strings_as_keyword": {
          "mapping": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "match_mapping_type": "string"
        }
      }
    ],
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "counter": {
        "type": "long"
      }
    }
  },
  "order": 1,
  "settings": {
    "index": {
      "lifecycle": {
        "name": "fastdfsbeat",
        "rollover_alias": "fastdfsbeat-8.0.0"
      },
      "mapping": {
        "total_fields": {
          "limit": 10000
        }
      },
      "max_docvalue_fields_search": 200,
      "query": {
        "default_field": [
          "message",
          "fields.*"
        ]
      },
      "refresh_interval": "5s",
      "number_of_replicas" : 0,
      "number_of_shards": 1
    }
  }
}

Beater代码接口

type Beater interface {
	// The main event loop. This method should block until signalled to stop by an
	// invocation of the Stop() method.
	Run(b *Beat) error

	// Stop is invoked to signal that the Run method should finish its execution.
	// It will be invoked at most once.
	Stop()
}

New

Creates the Beat object

Run 实现fastdfs监控代码

Contains the main application loop that captures data and sends it to the defined output using the publisher

Stop

Contains logic that is called when the Beat is signaled to stop

创建 Index templates

新建 index 时自动使用 templates 的 settings/mappings 设置

Elasticsearch 7.x template 语法差异省去了 Elasticsearch 6.x template 中的"_doc"

使用 Elasticsearch 7.x

docker run --name es -d -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.5.2

执行 fastdfs.template-7.5.json 索引模版

curl -XPUT -H 'Content-Type: application/json' http://localhost:9200/_template/fastdfsbeat-8.0.0 -d@fastdfs.template-7.5.json

启动fastdfsbeat

mage build
./fastdfsbeat -e -d "*"
./fastdfsbeat -e -c fastdfsbeat.yml

可查询到 fastdfsbeat 收集的自定义 counter 数据

源码地址: https://github.com/yusys-cloud/fastdfsbeat

四时宝库

程序员的知识宝库

Elastic Beat定制开发FastDFS集群监控 - FastDFS收集器代码