LogJuicer Extracts Anomalies From Log Files

Based on baseline logs, LogJuicer highlights useful texts in target logs. The goal is to save time in finding failures’ root causes.

§ How it works

LogJuicer implements a custom diffing process to compare logs:

• A tokenizer removes random words.
• Lines are converted into feature vectors using the hashing trick.
• The logs are compared using cosine similarity.

§ Install

Install the logjuicer command line by running:

$ cargo install --git https://github.com/logjuicer/logjuicer logjuicer-cli

If you don’t have cargo, see this install rust documentation.

Or grab the latest release assets logjuicer-x86_64-linux.tar.bz2 from https://github.com/logjuicer/logjuicer/releases

§ Use

Analyze a local file:

$ logjuicer file /var/log/zuul/scheduler.log

Analyze a remote url:

$ logjuicer url https://zuul/build/uuid

Compare two inputs (when baseline discovery doesn’t work):

$ logjuicer diff https://zuul/build/success-build https://zuul/build/failed-build

Save and re-use trained model using the --model file-path argument.

A command line to extract anomaly from log files.

Usage: logjuicer [OPTIONS] <COMMAND>

Commands:
  diff         Compare targets
  path         Analyze a path
  url          Analyze a url
  train        Train a model
  check-model  Check a pre-built model
  test         Evaluate datasets

Options:
      --config <FILE>    LogJuicer configuration
      --report <REPORT>  Save the report
      --model <FILE>     Load or save the model
  -h, --help             Print help
  -V, --version          Print version

§ Configure

LogJuicer supports the ant’s fileset configuration to filter the processed files:

• includes: list of files regex that must be included. Defaults to all files.
• excludes: list of files regex that must be excluded. Defaults to default excludes or none if default_excludes is false.
• default_excludes: indicates whether default excludes should be used or not.

§ Learn

To read more about the project:

• Initial presentation blog post
• The command line specification: ./doc/adr/0001-architecture-cli.md
• How the tokenizer works: Improving LogJuicer Tokenizer
• How the nearest neighbor works: Implementing LogJuicer Nearest Neighbors
• How the log file iterator works: Introducing the BytesLines Iterator
• Completing the first release of LogJuicer
• How the web interface works: WASM based web interface
• The report file format: Leveraging Cap’n Proto For LogJuicer Reports

§ Contribute

Clone the project and run tests:

git clone https://github.com/logjuicer/logjuicer && cd logjuicer
cargo test && cargo fmt && cargo clippy

Run the project:

cargo run -p logjuicer-cli -- --help

Activate tracing debug:

export LOGJUICER_LOG="logjuicer_model=debug,logjuicer_cli=debug"
# Create a chrome trace that can be viewed in web browser with `chrome://tracing`
export LOGJUICER_TRACE=./chrome.trace

Join the project Matrix room: #logjuicer:matrix.org.