[Question] how to setup tlog to log to elasticsearch

[haiwu]

The current documentation is not clear to me.

Not going to use sssd, only to use tlog-rec-session.

By default /etc/tlog/tlog-rec-session.conf uses journal. If switching to use file, then the whole logging to elasticsearch from rsyslog would not work.

After switching /etc/tlog/tlog-rec-session.conf to use syslog, I could see /var/log/tlog.log sometimes would have tlog entries, most time it would log NO MATCH every second, if there's no tlog activity. Is this expected?

Also it seems rsyslog auto created an index in elasticsearch, but there's no docs in this index, so apparently it is not working at all. How to get this to work?

[gstorme]

You could use the file writer and use logstash to ship the file to elasticsearch.
The basic config below works well:

input {
  file {
    path => "/var/log/tlog/tlog.log"
    codec => "json"
  }
}

output {
  elasticsearch {
    hosts => [ "" ]
    index => ""
    user => ""
    password => ""
  }
}

[haiwu]

Thanks @gstorme!

Does the above require creating index mapping and creating new index on Elasticsearch side first?
Could the same be done via Filebeat -> Elasticsearch?

My understanding is that Logstash might need more compute resources on client side than Filebeat..

[gstorme]

If you grant the required permissions to the configured user within elasticsearch, logstash will create the index.
And filebeat -> ES should also work, it's indeed more lightweight compared to logstash.

[haiwu]

@gstorme: Since I have not got this to work yet, I am still wondering about index mappings that Logstash/Filebeat/Rsyslog could auto create by default. In order for tlog-play -r es CLI to work with tlog-rsyslog index, what kind of index mappings are to be expected? Will it create each field from each log line in /var/log/tlog/tlog.log file, such as ver, term, in_bin, out_txt?

[gstorme]

I don't know how rsyslog handles this.
With logstash you don't need to create a mapping yourself. It will import the JSON fields from the logfile automatically.
I assume filebeat can also do this, if your input config is correct.

[haiwu]

@gstorme: I hit the same issue as this one with Filebeat: #257 (no solution listed in this issue), where Filebeat would ensure almost the same as this one:
"host": {
"name": "my"
},

I managed to use some Filebeat directives to update the above to be:
"host": "my"

But Elasticsearch keeps refusing to create index due to the above host not in its expected "host": { "name": format.

What does your "host" field look like?

[sowabdoul]

Hello @gstorme @haiwu, Help me please!
How to configure Tlog to submit sessions to elasticsearch (SSL and Xpack are enabled). Can it send directly, or should I use Filebeat or Logstash? Is it possible to view sessions via the elasticsearch machine?

[macgahe]

Hello @gstorme @haiwu, Help me please! How to configure Tlog to submit sessions to elasticsearch (SSL and Xpack are enabled). Can it send directly, or should I use Filebeat or Logstash? Is it possible to view sessions via the elasticsearch machine?

usually I capture de logs from Tlog via any log agent, if you are using ES, I'd use Filebeat which encrypts the traffic till your ES Cluster. Regards your 2nd question you can retrieve the data from ES cluster using tlog and reproduce that session, but TLOG is the one which does the work, ES is just purely the storage to retrieve the data..

[sowabdoul]

@macgahe, Thank you so much!, I used tlog --> rsyslog --> elasticsearch
How to see all sessions from the ES machine (in one place)

[macgahe]

@macgahe, Thank you so much!, I used tlog --> rsyslog --> elasticsearch How to see all sessions from the ES machine (in one place)

even using rsyslog you can encrypt the traffic with certificates..

to play the session and retrieve it from ES.. I'd suggest you to look into tlog-play command

[sowabdoul]

Thank you for your attention, @macgahe

[macgahe]

@macgahe, Thank you so much!, I used tlog --> rsyslog --> elasticsearch How to see all sessions from the ES machine (in one place)

if you allow me the suggestion I would change rsyslog to filebeat ( ElasticSearch Native Agent ) mostly because parsing the data will be much more easier and more productive.. unless you move the Rsyslog into another server and there you process the logs through filebeat or elastic agent depending on your ES version

[sowabdoul]

I considered configuring tlog-rec-session to record sessions in a file and then using Filebeat to send them to Elasticsearch. However, a concern is that the files might be deleted before the data is sent to Elasticsearch or there could be a longer wait time.
In comparison, with Rsyslog, I use UDP and sessions are logged as journal files.

[sowabdoul]

Hello @macgahe, please. In addition to having numerous recordings per session, I have this error.

Message ID is out of order
Failed reading the source at message #1

tlog-rec-session.conf

{
    "shell": "/bin/bash",
    "notice": "",
    "latency": 10,
    "payload": 16384,
    "log": {
        "input": true,
        "output": true,
        "window": false
    },
    "limit": {
        "rate": 16384,
        "burst": 32768,
        "action": "pass"
    },
    "file": {
        "path": ""
    },
    "syslog": {
        "facility": "authpriv",
        "priority": "info"
    },
    "journal": {
        "priority": "info",
        "augment": true
    },
    "writer": "journal"
}