Document guides the installation, configuration and index setup of the GrayLog service.
We are supporting GrayLog versions 2.4.7 - 3.1.2 which works with the specific version of Elasticsearch (5th version).
This section shows how to configure GrayLog for log collection from the platform services, as well as from the flow steps and one-time executions.
We recommend setting up different streams for the platform services and the integration flow executions to enable different retention policies for the technical logs produced by the platform and the execution logs produced by the integration flow steps.
Fluentd collects logs from the Kubernetes containers
and sends logs to GrayLog in the GELF format. The plan is to configure GrayLog
input to receive messages via the following two ports:
12201 for the platform services (eio-platform)12202 for the flow step and one-time executions (eio-exec)We use this input to collect all logs from the platform services on the port
12201. The following are the configurations in the GrayLog UI.
GELF UDP in the select box,Platform input,12201,eio-platform,{
"extractors": [
{
"title": "Truncate message",
"extractor_type": "regex_replace",
"converters": [],
"order": 0,
"cursor_strategy": "copy",
"source_field": "message",
"target_field": "message",
"extractor_config": {
"replacement": "$1",
"regex": "^(.{0,524288}).*$"
},
"condition_type": "none",
"condition_value": ""
},
{
"title": "JSON",
"extractor_type": "json",
"converters": [],
"order": 1,
"cursor_strategy": "copy",
"source_field": "message",
"target_field": "message",
"extractor_config": {
"flatten": false,
"key_separator": ".",
"list_separator": ", ",
"kv_separator": "="
},
"condition_type": "string",
"condition_value": "level"
},
{
"title": "K8S Fluentd",
"extractor_type": "json",
"converters": [],
"order": 2,
"cursor_strategy": "copy",
"source_field": "log",
"target_field": "",
"extractor_config": {
"flatten": true,
"list_separator": ", ",
"kv_separator": "=",
"key_prefix": "",
"key_separator": "_",
"replace_key_whitespace": false,
"key_whitespace_replacement": "_"
},
"condition_type": "none",
"condition_value": ""
}
],
"version": "2.4.7"
}
We use this input to collect all logs from the integration flow step and one-time
executions on the port 12202. The following are the configurations in the GrayLog UI.
GELF UDP in the select box,Exec-input,12202,eio-exec,{
"extractors": [
{
"title": "Extract threadId",
"extractor_type": "regex",
"converters": [],
"order": 0,
"cursor_strategy": "copy",
"source_field": "message",
"target_field": "threadId",
"extractor_config": {
"regex_value": "\"threadId\" ?: ?\"([a-z0-9-]+)\""
},
"condition_type": "none",
"condition_value": ""
},
{
"title": "Extract level",
"extractor_type": "regex",
"converters": [
{
"type": "numeric",
"config": {}
}
],
"order": 1,
"cursor_strategy": "copy",
"source_field": "message",
"target_field": "level",
"extractor_config": {
"regex_value": "\"level\" ?: ?([0-9-]+)"
},
"condition_type": "none",
"condition_value": ""
},
{
"title": "Copy message to msg",
"extractor_type": "copy_input",
"converters": [],
"order": 2,
"cursor_strategy": "copy",
"source_field": "message",
"target_field": "msg",
"extractor_config": {},
"condition_type": "none",
"condition_value": ""
},
{
"title": "Try to parse JSON from message to msg",
"extractor_type": "regex",
"converters": [],
"order": 3,
"cursor_strategy": "copy",
"source_field": "message",
"target_field": "msg",
"extractor_config": {
"regex_value": "\"msg\" ?: ?\"(.*?)(?<!\\\\)\""
},
"condition_type": "none",
"condition_value": ""
}
],
"version": "2.4.7"
}
A GrayLog stream writes messages to an index set, which has configuration for retention, sharding, and replication of the stored data. By configuring index sets, you could, for example, have different retention times for certain streams.
This index set is used to configure rotation and retention policy for logs from the platform services. Add index set following the instructions:
Platform index set,Platform logs,platform,curl -X PUT -H 'Content-Type: application/json' 'http://{GRAYLOG_HOST}:9200/_template/platform-custom-mapping?pretty' -d '
{
"template": "platform_*",
"settings": {
"index": {
"analysis": {
"analyzer": {
"analyzer_keyword": {
"filter": "lowercase",
"tokenizer": "keyword"
}
}
}
}
},
"mappings": {
"message": {
"properties": {
"msg": {
"type": "text",
"analyzer": "analyzer_keyword"
}
}
}
}
}
'
This index set is used to configure rotation and retention policy for logs from the integration flow steps and one-time executions. Add index set following the instructions:
Exec index set,Flow steps and one-time execs logs,exec,curl -X PUT -H 'Content-Type: application/json' 'http://{GRAYLOG_HOST}:9200/_template/exec-custom-mapping?pretty' -d '
{
"order": 0,
"index_patterns": [
"exec_*"
],
"mappings": {
"properties": {
"msg": {
"analyzer": "simple",
"type": "text"
}
}
}
}
'
The Graylog streams are a mechanism to route messages into categories in realtime while they are processed. You define rules that instruct Graylog which message to route into which streams.
Every stream is assigned to an index set which controls how messages routed into that stream are being stored into the Elasticsearch.
Two streams must be configured:
This stream is used to route all platform logs to Platform index set.
Platform stream,Platform logs,Platform index set,source,match exactly,eio-platform,This stream is used to route all flow steps & one-time execution logs to Exec index set.
Exec stream,Flow steps & one-time execs logs,Exec index set,source,match exactly,eio-exec,