AMD: LogsLop — open-source tool compresses large log files to 11% and saves 78% of tokens

What is LogsLop and how does it work?

Jaccard similarity — a measure of the overlap between two sets, defined as the ratio of their intersection to their union — is the core of the algorithm that powers LogsLop. AMD’s new open-source tool uses it to group similar log lines: it first normalizes all numbers in lines (IP addresses, PIDs, timestamps), then computes Jaccard similarity between lines and incrementally clusters them into groups.

The result: instead of thousands of nearly identical lines, the LLM receives one representative line per group.

What results does it achieve on real data?

On a dataset of 73 real log files with a total of 256,000 lines, LogsLop reduces the content to 11% of the original line count and 7.5% of the original byte count. For comparison, naive log truncation (first N lines) loses critical information from the middle and end of the file — LogsLop retains representative samples from all parts.

With the Llama 3.3 70B model, the compressed log uses 78% fewer tokens and requires 57% less processing time than full-file analysis.

How to use it and where to get it

LogsLop is a Python CLI tool available through the standard package manager: pip install logslop. It requires Python 3.7 or later, and the source code is published under amd/logslop on GitHub. The tool is intended for DevOps teams and SRE engineers who feed long log files to LLMs for diagnosing errors or anomalies in production systems.