centos7.6下的python3.6.9虚拟环境安装elastalert，虚拟环境python,centos7.6安

centos7.6下的python3.6.9虚拟环境安装elastalert，虚拟环境python,centos7.6安

centos7.6安装python3.6.9+elastalert1.编译安装python3.6.9环境# 安装依赖yum -y install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel# 获取编译安装python3.6.9mkdir -p /usr/local/python3wget https://www.python.org/ftp/python/3.6.9/Python-3.6.9.tgztar xf Python-3.6.9.tgzcd Python-3.6.9./configure --prefix=/usr/local/python-3.6.9make && make installln -s /usr/local/python3/bin/python-3.6.9 /usr/bin/python3ln -s /usr/local/python3/bin/pip3 /usr/bin/pip32.安装virtualenv虚拟环境pip3 install virtualenv# 创建存放虚拟环境的目录mkdir -p  /usr/local/venv_py3.6_elastalert-0.2.1# 创建纯净的虚拟环境cd /usr/localgit clone https://github.com/Yelp/elastalert.gitcd /usr/local/elastalert/usr/local/python-3.6.9/bin/virtualenv --no-site-packages --python=/usr/local/python-3.6.9/bin/python3.6 /usr/local/venv_py3.6_elastalert-0.2.1[[email protected]-kibana-elastalert-01:/usr/local/venv_py3.6_elastalert-0.2.1]# source bin/activate(venv_py3.6_elastalert-0.2.1) [[email protected]01:/usr/local/venv_py3.6_elastalert-0.2.1]#3.在虚拟的python3.6环境中安装alasticalert# 指定库，安装依赖，否则可能安装失败(venv_py3.6_elastalert-0.2.1) [[email protected]01:/usr/local/elastalert]# pip install -r requirements.txt -i https://pypi.python.org/simple# 安装主程序，否则无法使用 elastalert-create-index 命令(venv_py3.6_elastalert-0.2.1) [[email protected]01:/usr/local/elastalert]# python setup.py install# 运行 elastalert-create-index 配置(venv_py3.6_elastalert-0.2.1) [[email protected]01:/usr/local/elastalert]# elastalert-create-indexEnter Elasticsearch host: 172.30.0.62Enter Elasticsearch port: 19200Use SSL? t/f: fEnter optional basic-auth username (or leave blank): Enter optional basic-auth password (or leave blank): Enter optional Elasticsearch URL prefix (prepends a string to the URL of every request): New index name? (Default elastalert_status) New alias name? (Default elastalert_alerts) Name of existing index to copy? (Default None) Elastic Version: 7.3.0Reading Elastic 6 index mappings:Reading index mapping ‘es_mappings/6/silence.json‘Reading index mapping ‘es_mappings/6/elastalert_status.json‘Reading index mapping ‘es_mappings/6/elastalert.json‘Reading index mapping ‘es_mappings/6/past_elastalert.json‘Reading index mapping ‘es_mappings/6/elastalert_error.json‘New index elastalert_status createdDone!(venv_py3.6_elastalert-0.2.1) [[email protected]01:/usr/local/elastalert]# # 报错(venv_py3.6_elastalert-0.2.1) [[email protected]01:/usr/local/elastalert]# elastalert-test-rule example_rules/my_rule.yml  File "/usr/local/venv_py3.6_elastalert-0.2.1/lib/python3.6/site-packages/tzlocal/unix.py", line 90, in _get_localzone    utils.assert_tz_offset(tz)  File "/usr/local/venv_py3.6_elastalert-0.2.1/lib/python3.6/site-packages/tzlocal/utils.py", line 38, in assert_tz_offset    raise ValueError(msg)ValueError: Timezone offset does not match system offset: 28800 != -25200. Please, check your config files.# 代码和系统时区不匹配，重新设置为上海时区(venv_py3.6_elastalert-0.2.1) [[email protected]01:/usr/local/elastalert]# timedatectl set-timezone Asia/Shanghai(venv_py3.6_elastalert-0.2.1) [[email protected]01:/usr/local/elastalert]# elastalert-test-rule example_rules/my_rule.ymlINFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.            To send them but remain verbose, use --verbose instead.WARNING:elasticsearch:GET http://172.30.0.62:19200/logstash-*/_search?ignore_unavailable=true&size=1 [status:400 request:0.004s]Error running your filter:RequestError(400, ‘parsing_exception‘, {‘error‘: {‘root_cause‘: [{‘type‘: ‘parsing_exception‘, ‘reason‘: ‘[term] query malformed, no start_object after query name‘, ‘line‘: 1, ‘col‘: 151}], ‘type‘: ‘parsing_exception‘, ‘reason‘: ‘[term] query malformed, no start_object after query name‘, ‘line‘: 1, ‘col‘: 151}, ‘status‘: 400})INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.                To send them but remain verbose, use --verbose instead.1 rules loadedINFO:apscheduler.scheduler:Adding job tentatively -- it will be properly scheduled when the scheduler startsWARNING:elasticsearch:GET http://172.30.0.62:19200/logstash-*/_search?_source_includes=%2A%2C%40timestamp&ignore_unavailable=true&scroll=30s&size=10000 [status:400 request:0.003s]ERROR:root:Error running query: RequestError(400, ‘parsing_exception‘, ‘[term] query malformed, no start_object after query name‘)Would have written the following documents to writeback index (default is elastalert_status):elastalert_error - {‘message‘: "Error running query: RequestError(400, ‘parsing_exception‘, ‘[term] query malformed, no start_object after query name‘)", ‘traceback‘: [‘Traceback (most recent call last):‘, ‘  File "/usr/local/venv_py3.6_elastalert-0.2.1/lib/python3.6/site-packages/elastalert-0.2.1-py3.6.egg/elastalert/elastalert.py", line 384, in get_hits‘, ‘    **extra_args‘, ‘  File "/usr/local/venv_py3.6_elastalert-0.2.1/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped‘, ‘    return func(*args, params=params, **kwargs)‘, ‘  File "/usr/local/venv_py3.6_elastalert-0.2.1/lib/python3.6/site-packages/elasticsearch/client/__init__.py", line 819, in search‘, ‘    "GET", _make_path(index, "_search"), params=params, body=body‘, ‘  File "/usr/local/venv_py3.6_elastalert-0.2.1/lib/python3.6/site-packages/elasticsearch/transport.py", line 350, in perform_request‘, ‘    timeout=timeout,‘, ‘  File "/usr/local/venv_py3.6_elastalert-0.2.1/lib/python3.6/site-packages/elasticsearch/connection/http_requests.py", line 156, in perform_request‘, ‘    self._raise_error(response.status_code, raw_data)‘, ‘  File "/usr/local/venv_py3.6_elastalert-0.2.1/lib/python3.6/site-packages/elasticsearch/connection/base.py", line 181, in _raise_error‘, ‘    status_code, error_message, additional_info‘, "elasticsearch.exceptions.RequestError: RequestError(400, ‘parsing_exception‘, ‘[term] query malformed, no start_object after query name‘)"], ‘data‘: {‘rule‘: ‘eus-log-elasticsearch-cluster-alert‘, ‘query‘: {‘query‘: {‘bool‘: {‘filter‘: {‘bool‘: {‘must‘: [{‘range‘: {‘@timestamp‘: {‘gt‘: ‘2019-09-17T05:06:25.831477Z‘, ‘lte‘: ‘2019-09-17T05:21:25.831477Z‘}}}, {‘term‘: None}, {‘query_string‘: {‘query‘: ‘message: error‘}}]}}}}, ‘sort‘: [{‘@timestamp‘: {‘order‘: ‘asc‘}}]}}}4.配置elastalert##############全局配置[root:/usr/local/elastalert#cp  config.yaml.example config.yaml#存放elastalert 规则的文件夹，你的elastalert 放到哪里就放到哪里就行了rules_folder: /usr/local/elastalert/example_rules #Elastalert 多久去查询一下根据定义的规则去elasticsearch 查询是否有符合规则的字段，如果有就会触发报警，如果没有就等待下一次时间再检查，时间定义的单位从周到秒都可以，具体定义方法如下。run_every: #seconds：1  minutes: 1  #hours：1  #days：1  #weeks：1 #当查询开始一直到结束，最大的缓存时间。buffer_time:  minutes: 15 #你的Elasticsearch ip地址es_host: 172.30.0.52 #Elasticsearch 的端口es_port: 9200 #这个是elastalert 在es里边写的index# The index on es_host which is used for metadata storage# This can be a unmapped index, but it is recommended that you run# elastalert-create-index to set a mappingwriteback_index: elastalert_status #如果alert当时没有发出去重试多久之后放弃发送；alert_time_limit:  days: 2[[email protected]-kibana-elastalert-01:/usr/local/elastalert]# egrep -v ‘^#|^$‘ config.yamlrules_folder: example_rulesrun_every:  minutes: 1buffer_time:  minutes: 15es_host: 172.30.0.62es_port: 19200writeback_index: elastalert_statuswriteback_alias: elastalert_alertsalert_time_limit:  days: 2##############rules 的定义[[email protected]-elk-cluster01:/usr/local/elastalert]#cp example_frequency.yaml my_rule.yamlvi my_rule.yaml# Alert when the rate of events exceeds a threshold#Elasticsearch  机器es_host: 192.168.115.65 #Elasticsearch  端口es_port: 9200 #如果elasticsearch 有认证，填写用户名和密码的地方# (Optional) basic-auth username and password for Elasticsearch#es_username: someusername#es_password: somepassword #rule name 必须是独一的，不然会报错，这个定义完成之后，会成为报警邮件的标题# (Required)# Rule name, must be uniquename: ws-elk-cluster-alert #配置一种数据验证的方式，有 any，blacklist，whitelist，change，frequency，spike，flatline，new_term，cardinalityany：只要有匹配就报警；blacklist：compare_key字段的内容匹配上 blacklist数组里任意内容；whitelist：compare_key字段的内容一个都没能匹配上whitelist数组里内容；change：在相同query_key条件下，compare_key字段的内容，在 timeframe范围内 发送变化；frequency：在相同 query_key条件下，timeframe 范围内有num_events个被过滤出 来的异常；spike：在相同query_key条件下，前后两个timeframe范围内数据量相差比例超过spike_height。其中可以通过spike_type设置具体涨跌方向是- up,down,both 。还可以通过threshold_ref设置要求上一个周期数据量的下限，threshold_cur设置要求当前周期数据量的下限，如果数据量不到下限，也不触发；flatline：timeframe 范围内，数据量小于threshold 阈值；new_term：fields字段新出现之前terms_window_size(默认30天)范围内最多的terms_size (默认50)个结果以外的数据；cardinality：在相同 query_key条件下，timeframe范围内cardinality_field的值超过 max_cardinality 或者低于min_cardinality # (Required)# Type of alert.# the frequency rule type alerts when num_events events occur with timeframe time#我配置的是frequency，这个需要两个条件满足，在相同 query_key条件下，timeframe 范围内有num_events个被过滤出来的异常type: frequency #这个index 是指再kibana 里边的index，支持正则匹配，支持多个index，同时如果嫌麻烦直接* 也可以。index: customer*#index: es-nginx*,winlogbeat* #时间出发的次数num_events: 5 #和上边的参数关联，也就是说在4分钟内出发5次会报警timeframe:  minutes: 4 #这个还是非常关键的地方，就是你希望程序的message里边出现了什么样的关键字就报警，这个其实就是elasticsearch 的query语句，支持 AND&OR等。filter:- query:    query_string:      query: "message: 错误  OR Error" #一但需要报警用那种方式报警，支持如下的方式，同时官方支持自定义，我用常规的邮件方式作为报警方式。alert:- "email"#在邮件正文会显示你定义的alert_textalert_text: "Ref Log http://192.168.254.194"#报警邮箱的smtp serversmtp_host: mail.chinasoft.cn#报警邮箱的smtp 端口smtp_port: 25#需要把认证信息写到额外配置文件里，需要user和password两个属性smtp_auth_file: /usr/local/elastalert/example_rules/smtp_auth_file.yamlemail_reply_to:[email protected]163.comfrom_addr: [email protected]163.com #接受报警邮箱的地址,可以写多个，当然后边搞个邮件组最好了。# (required, email specific)# a list of email addresses to send alerts toemail:- "[email protected]"[[email protected]-kibana-elastalert-01:/usr/local/elastalert/example_rules]# egrep -v ‘^#|^$‘ my_rule.yml es_host: 172.30.0.62es_port: 19200name: eus-log-elasticsearch-cluster-alerttype: frequencyindex: filebeats-log*num_events: 5timeframe:  hours: 4filter:- term:- query:    query_string:      query: "message: error"alert:- "email"email:- "[email protected]"alert_text: "Ref Log http://172.30.0.62"smtp_host: mail.chinasoft.cnsmtp_port: 25smtp_auth_file: /usr/local/elastalert/example_rules/smtp_auth_file.yamlemail_reply_to: [email protected]from_addr: [email protected] ######################smtp认证文件[[email protected]-elk-cluster01:/usr/local/elastalert]#vi smtp_auth_file.yamluser: "jack"password: "jack123"#通过elastalert-test-rule 测试一下我们写的rule 是否有问题[[email protected]-elk-cluster01:/usr/local/elastalert/example_rules]# elastalert-test-rule my_rule.yaml#配置检查成功之后，我们就可以把程序跑起来了，把所有的日志直接打在前端，这样方便验证/usr/local/venv_py3.6_elastalert-0.2.1/bin/python3.6 -m elastalert.elastalert --verbose --rule /usr/local/elastalert/example_rules/my_rule.yaml

centos7.6下的python3.6.9虚拟环境安装elastalert