Feature Discussion - Log Noise reduction, regexp precompilation, and age-out for History Store

[bryannolen]

Discussion for the proposed Pull Request 182.

The main features proposed are:

1. Adding a command line option to allow tuning of the GO GC.
   Initally it was thought that the GC was not running frequeuently enough, leading to memory consumption growing unnecessarily. This was discussed in Issue 184. The root cause was a logic bug, but having a small memory limit and proving that the binary can run well with a small envelope (<=100MiB) does mean that the need for large instances or hardware is reduced.

2. Regexp Precomputation & Load Time Validation.
   One problem with thre previous menthod of just-in-time regexp computation (at comparison time in the strategy) is that invalid regexes would lead to runtime errors, potentially some time after startup (hours+). This is solved by precompiling all the regexp in the supplied configuration files at startup, and then operating on the precomputed regex matchers. This means that errors are identified fast, and that pattern matching is faster too.

3. Age-out for HistoryStore
   The initial implementaton of HistoryStore was a simple thread-safe map. The updated version adds a new value to the store for each key (LastSeen) and an automated cleanup process HistoryCleaner() - a goroutine that will run though the stored entries on an interval (default 1 minute) and remove any that are older than MaxHistoryAge (default 1 hour).

4. Fix History Exponental Growth Bug.
   In the SSH Strategy, a logic bug was introduced in 3.3.10 that means that every time an interactive command is ran, the entire previous cache of messages would be re-appended to the store, meaning the amount of heap needed (and volume of messages sent to the LLM!) would grow exponentially. This has been addressed.

5. Reduce the volume of logs, and clarify the meaning of some messages.
   The "End of Raw Command Session" message is removed, and the "msg" value of other messages changed to more accuratly represent the phase of the interaction. No other content of the events is affected.

[bryannolen]

Oh, and this one too
6. Added an new element to the Command configuration stanza, "name". This allows you to track which Command handled the request. This is implemented in both HTTP and SSH strategies.

Example of named command:

  - regex: "^whoami$"
    handler: "root"
    name: "IAmRoot"

Will add the field Handler: "IAmRoot" to the logged event.

[bryannolen]

Released as https://github.com/mariocandela/beelzebub/releases/tag/v3.3.11