A release that adds support for viewing JSON search results in context by opening the archive that contains them using the log viewer. This release also includes some features that improve usability.

A release with support for viewing text search results in context by opening the file that contains

The CLP release includes two tars:

• clp-json for compressing and searching JSON logs
• clp-text for compressing and searching unstructured text logs

Docs

The docs for this release are available here.

New features

• Support for viewing JSON search results in context by opening the archive that contains them using the log viewer. (#569, #584, #596, #600, #615)
  • Note that the log viewer doesn’t open the archive directly but rather CLP decompresses the archive into chunks of JSONL files that the log viewer opens. In a future release, these chunks will be IR files to lower resource usage.
• clp-json: Support for querying fields whose keys contain periods by escaping them with a backslash. (#560, #617)
• Support for deleting archives that are entirely within a time range. (#594)

Bug fixes & improvements

• Homebrew path detection for mariadb-connector-c to fix macOS build failures. (#582)

View the full changelog for more details.

Thanks to @AVMatthews, @Bill-hbrhbr, @LinZhihao-723, @anlowee, @gibber9809, @haiqi96, @junhaoliao, @kirkrodrigues, and @wraymo for their contributions.

A release that uses the rewritten log viewer and includes new features for clp-s, bug fixes, and improvements.

NOTE: We are bumping the minor version due to a breaking format change in clp-json/clp-s’ archive format. If this affects you, reach out and we can help with version migration.

The CLP release includes two tars:

• clp-json for compressing and searching JSON logs
• clp-text for compressing and searching unstructured text logs

Docs

The docs for this release are available here.

New features

• An upgrade to the rewritten log viewer. (#562, #565)
  • The rewrite is based around the clp-ffi-js library (which uses WebAssembly) and improves performance for switching pages by 2.77-4.67x, and log-level filtering by 1.64-4.84x.
  • The rewrite also includes a UI refresh, more fine-grained log-level filtering, support for exporting the decompressed logs, and support for opening larger log files.
  • NOTE: The rewrite also includes support for formatting and viewing JSON logs which will be integrated with CLP in the next release.
• Support for building on CentOS Stream 9. (#521)
  • NOTE: We have dropped support for building on CentOS 7.4.
• clp-s:
  • Support for projecting a subset of columns during search. (#510)
    • Support for projection from the package will be integrated in a future release.
  • Packing of small tables into a single compression stream which can improve compression ratio as well as compression and search performance. (#466)

Bug fixes & improvements

• Fixes to allow CLP’s core to build with shared linking again. (#526)
• Upgrades for vulnerable dependencies. (#536)

View the full changelog for more details.

Thanks to @anlowee, @Henry8192, @LinZhihao-723, @diy1, @gibber9809, @junhaoliao, @kirkrodrigues, and @zechenericduan for their contributions.

A bug-fix release to improve error reporting, improve the UI, and fix some macOS-specific issues.

The CLP release includes two tars:

• clp-json for compressing and searching JSON logs
• clp-text for compressing and searching unstructured text logs

Docs

The docs for this release are available here.

Bug fixes & improvements

• clp-s: Report exactly where parsing error occurs when parsing JSON (fixes #514). (#503)
• webui: Improve UI of links for viewing search results in context. (#515)
• core: Add support for retrieving CURL error messages, handle unexpected CURL return code on macOS, and log such codes in tests (fixes #519). (#517)
• core: Mark spdlog specialization functions as const (fixes #524). (#525)

View the full changelog for more details.

Thanks to @AVMatthews, @Bill-hbrhbr, @gibber9809, @haiqi96, @junhaoliao, @kirkrodrigues, and @LinZhihao-723 for their contributions.

A release with support for viewing text search results in context by opening the file that contains them using using the log viewer. In a future release, we will also add support for viewing JSON search results in context. This release also includes a handful of bug fixes.

The CLP release includes two tars:

• clp-json for compressing and searching JSON logs
• clp-text for compressing and searching unstructured text logs

Docs

The docs for this release are available here.

Features/Changes

Support for viewing text search results in context by opening the file that contains them using the log viewer. (#420, #424, #435, #437, #440, #442, #449, #451, #452, #458, #460, #468, #470, #472, #474, #476, #481, #485, #489, #490)

Bug-fixes

• clp-package: Add support for running package as root (fixes #500). (#464)
• clp-s: Ensure the option to print archive stats remains set when splitting archives. (#462)
• clp-s: Correctly report uncompressed size of archives when splitting archives (fixes #469). (#463)
• core-clp: Add class to encapsulate libcurl's global resource management to fix a race condition. (#461)
• core-clp: Add a more complete UTF-8 validation function to filter out binary data. (#477)
• core-clp: Fix ArchiveMetadata deserialization by deserializing previously missed field. (#488)
• webui: Optimize Meteor bundle size (fixes #498). (#499)

View the full changelog for more details.

Thanks to @Bill-hbrhbr, @gibber9809, @haiqi96, @Henry8192, @junhaoliao, @kirkrodrigues, @LinZhihao-723, and @wraymo for their contributions.

A release with a few new features and bug fixes.

The CLP release includes two tars:

• clp-json for compressing and searching JSON logs
• clp-text for compressing and searching unstructured text logs

Docs

The docs for this release are available here.

Features/Changes

• A new way of storing arrays found in JSON log events. (#355, #401, #408, #413, #429)
  • Try it using clp-s's --structurize-arrays option.
• clp-json no longer adds implicit wildcards (’*’) at the beginning or end of a query. (#406)
• More frequent progress updates (on the command line) for compression jobs. (#388)

Bug fixes

• clp-json: Better handling for wildcards in the middle of a token. (#407)
• Improved robustness when cancelling search jobs. (#425)
• Reduced CPU usage when queries from the webui complete. (#418)

View the full changelog for more details.

Thanks to @davemarco, @gibber9809, @haiqi96, @Henry8192, @kirkrodrigues, @LinZhihao-723, and @wraymo for their contributions.

A release containing several new features including a new subsystem for compressing JSON logs, a web interface, a server for aggregations, and more!

Note that the release now includes two tarballs:

• clp-json for compressing and searching JSON logs
• clp-text for compressing and searching unstructured text logs

Docs

The docs for this release are available here.

Highlights

• Added a subsystem for compressing and searching JSON logs. (#217)
• Added a web interface for searching logs and viewing compression statistics. (#250)
• Added support for ingesting CLP IR streams generated by our logging library plugins. (#152)
• Added a reducer worker to support aggregation jobs like count. (#221)
• Added support for tagging archives and filtering archives by tags and/or timestamps. (#138, #285)
• Replaced the scheduler with separate compression and search schedulers for improved robustness and performance. (#238, #240)

This release also includes many other improvements and fixes. View the full changelog for more details.

Thanks to @abvarun226, @davemarco, @davidlion, @diy1, @gibber9809, @haiqi96, @junhaoliao, @kirkrodrigues, @LinZhihao-723, @OliverSM95, @SharafMohamed, and @wraymo for their contributions!

A release containing all of CLP's open-source features and bug-fixes to date.

Docs

The package contains a README or you can view it here.

Highlights

This release features several usability improvements, bug-fixes, and performance improvements. Some highlights include:

• Support for compression/search using user-specified variable schemas
• A custom encoding for floating-point variables that improves performance and provides better compression
• Reduced memory usage during compression
• Search using multiple workers

Features

• Use a custom float encoding for improved performance and better compression (#38)
• Reduce memory usage during compression by keeping only the necessary parts of various data structures (including the dictionaries) in memory (#40, #47, #48)
• Separate components into different containers and allow them to be started separately (#26, #65)
• Use the host's network rather than Docker's networking to eliminate the difference between single and multi-node deployments (#65)
• Use an instance ID to name all components to allow multiple deployments on a single node (#65)
• Expose the complete config file to the user rather than a simplified one (#65)
• Don't generate a config file but instead use default settings unless modified by a config file (#65)
• Add support for search with multiple nodes (#68)
• Add initial support for compression/search using user-defined variable schemas (#74)
• Update variables schema and encoding methods to match those of the existing logging libraries (#92, #115)
• Add support for building CLP core on macOS (#105, #106, #107)
• Switch to C++ 17's from_chars for better performance (#85)

Bug Fixes

• Use the current user's UID/GID (rather than root) when reading/writing files within mounted directories (#10)
• Recombine large files split across archives in their original order (#37)
• Fixes to enable compression on macOS and within WSL (#45)
• Update celery to 5.2.2 to mitigate CVE-2021-23727 (#46)
• Resolve input paths for compression before using them in the containers (#49)
• Use the archive output directory specified in the config file (#65)
• Cancel decompression/search when the user sends a keyboard interrupt (#65)
• Wait for the database to initialize before trying to use it (#65)
• Mount input logs at a dedicated path in the container that doesn't overlap with files in the container (#65)
• Allow filtering for files which overlap (rather than are encompassed by) the specified time range (#70)
• Allow an empty logtype for messages that only include a timestamp (#76)
• Remove ad-hoc rules used on top of CLP's default variables schema (#42)
• Compress integers less than -2^63 losslessly (#85)
• Add support for compressing zstd-compressed log files (#114)

Thanks to @All-less, @davidlion, @haiqi96, @kirkrodrigues, @LinZhihao-723, @NamanGulati, @SharafMohamed, and @thepegasos for their contributions.

A patch release to correct some archive metadata.

Documentation

The package contains a README.md or you can view it here.

Features & Fixes

• Fix for #15: The global metadata database now stores the correct uncompressed size of each archive
• Add the table prefix option for users of the global mysql metadata database

CLP's first release, one of many to come!

Documentation

The package contains a README.md or you can view it here.

Features

• Parallel compression via a push-button cluster of Docker containers or through a multi-node deployment
• Decompression
• Search

Thanks to @davidlion @diy1 @jackluo923 @kirkrodrigues @NamanGulati @rishikeshdevsot @robinlee09201 @wraymo for their code and testing contributions.