Ported to rust

[2bndy5]

I did it again... I ported the pure python cpp-linter package to the rust language.

This repo is essentially a cargo virtual workspace (cargo is the package manager for rust). It has 3 separate packages:

• cpp-linter-lib: The main codebase as a library. This is used as the executables' backend.
• cpp-linter-cli: This package wraps the cpp-linter-lib package into a native executable binary. The built binary does not require python nor rust installed to run.
• cpp-linter-py: This package wraps the cpp-linter-lib package into a pip-installable python package. See below for more detail.

cpp-linter-py

We can still deploy this port to pypi because there is a python package/binding located in cpp-linter-py directory. The python binding only exposes the run::main() function and is specified as an entry point script in pyproject.toml. There is no other rust API exposed in python.

How to build/run the python package?

1. Install the recommended rust-to-python package builder named maturin (similar to the setuptools build backend)

   python3 -m pip install maturin
   

2. Install the development build (aka "editable install")

   cd cpp-linter-py
   maturin develop
   

   If you get an error about openssl headers, add the argument --features openssl-vendored (which should build openssl from source)
3. Run the cpp-linter entry point like we do in the cpp-linter-action repo's action.yml step

   cpp-linter -h
   

cpp-linter-lib

Differences vs pure python package

Most of the functionality from the pure python codebase has been accurately ported over to rust. There is 1 difference that I plan on back-porting to the pure python pkg:

The --verbosity argument only accepts the values info or debug. Normally, this arg's default value is info, but if the user re-runs a cpp-linter step in CI with debugging logs enabled, then the arg's value changes to debug.

Testing and code coverage

The testing is not complete (yet), but I got code coverage up to 59%. Rust separates unit tests from integration tests:

• unit tests go into the code in the cpp-linter-lib/src/ folder, in the same *.rs file that has the code being tested.
• integration tests go into a cpp-linter-lib/tests folder. These are more like what we are doing with the more complicated pytest functions.

Code coverage in rust seems to have been an afterthought. It does not seem well matured when compared to python's testing capabilities (no way to easily parametrize in rust).

cpp-linter-cli

Note

UPDATE: The cpp-linter-cli crate was consolidated into the cpp-linter-lib crate as a standalone binary executable for the cpp-linter-lib.

I have setup the CI to build (& deploy as release assets) several supported platforms:
https://github.com/cpp-linter/cpp_linter_rs/blob/33a22b5c78c8df4bfa1324870776b3f34ce08689/.github/workflows/binary-builds.yml#L174-L183
This should cover most use cases for CI workflows. It even covers ARM64 platforms on Linux (GNU C lib only) and Apple.

Other toolchains are supported by the rustc compiler, but cross-compiling them in CI is rather painful since openssl is required for the libgit2 rust binding (package named git2). I have to devise a better CI workflow that can make use of different docker images specifically tailored to cross-compiling rust and openssl.

How to build/run the native binary exe?

From the repo root, simply run

cargo run --bin cpp-linter-cli -- -h

This will build the exe in debug mode. To make compiler optimize the code, add --release after the run command.

The -- -h means anything after the -- is passed to the built executable. So, you should see the help menu.

To only build the exe, use the build command instead:

cargo build --bin cpp-linter-cli

And you'll find the binary located at target/debug/cpp-linter-cli.

Installing binary exe from release assets

There are a few ways to download the built binary executable from release assets.

Note

None of these ways will take advantage of dependabot updates though.

This simplest way would be with cargo-binstall (a cargo aftermarket sub-command). But cargo-binstall uses Github's REST API (implying rate limits) and requires our cpp-linter-cli and cpp-linter-lib packages be published to crates.io.

Another way would be using the same method we get static clang-tools binaries from GitHub release assets (in clang-tools-pip).

We could also just use cargo install, but this will compile the binary from source every time.

[2bndy5]

Initial benchmarks using hyperfine:

• Benchmark 1 was the latest release of pure python cpp-linter
• Benchmark 2 was the release-optimized rust native binary

Looks like my efforts are fruitful here (about 1 second faster than pure python). I expect that the rust port would scale better when the test case is a much more complex scenario (like a 5000 line file full of clang-tidy and clang-format violations). The above scenario was just using the this repo's root folder.

[shenxianpeng]

I'm impressed with how quickly you ported to rust (previous is dart). I have a stupid question does the port to rust benefit from any other improvements except a second speed faster (I apologize if missed something you already mentioned), it's already pretty good by the time you're done refactoring the pure python cpp-linter.

Rust is a language worth trying, and if clang-format, or even clang-tidy (especially existing build issue of clang-tidy on Linux and MacOSX)., were rewritten with rust, go, or dart maybe, this would be a big thing for the open source community.

[2bndy5]

Python's memory consumption does not scale well. Most of our issues (where users reported actual problems) involve a very large list of changes in PRs. The more memory we consume, the slower we can run. Our unit tests are bite-size problems, so they aren't a good measure for performance.

Simply put, python is great for data science, but only when used in small doses. I expect that rust will perform better when scanning multiple files consisting of thousands of lines and hundreds of clang-diagnostic concerns.

Also, using a rust native binary executable will avoid requirements about what version of python is installed. Just download and run it.

---

There is a way to dynamically link to libclang from rust at runtime: see clang-sys crate ("crate" is like a rust "package"). I haven't explored this much because we would still need to make sure libclang is installed.

Or we could compile (statically link) libclang into the rust native binary. But the compile size would be pretty large and we'd have to distribute multiple version of the binary to support multiple versions of libclang...

Alternatively, there is libclang binding in python... But not all version distributed to pypi are in working order.

[2bndy5]

I looked at building a binary with the clang-rs (libclang rust binding). I found that it doesn't support clang v17 (yet) and has some other issues.

To build a cpp-linter binary that uses libclang would have to be done in C++. That way we wouldn't have to worry about any third party binding. I could try doing that but not in the near future.

[2bndy5]

Release Candidate 1 lives! 🎉

I just published v2.0.0-rc1 to test-pypi (as a trusted publisher with attestations) and crates.io.

Take a peek at lib.rs for package info at a glance (dependencies used, download frequency, etc).

I finally got the feature parity up to par with the python codebase (plus async HTTP requests).

How to install

We have some more options for installing cpp-linter v2.x:

Using crates.io

Note

These options skip installing from pre-releases by default.
We can drop the --version 2.0.0-rc1 after I publish a stable release (like v2.0.0).

Using cargo

The cargo install command will compile the executable from sources uploaded to crates.io.

cargo install cpp-linter --version 2.0.0-rc1

Using cargo-binstall

To install a pre-compiled executable, we can use cargo-binstall. This will get the package metadata from crates.io and download the release asset from the corresponding GitHub release.

cargo install cargo-binstall
cargo binstall -y cpp-linter --version 2.0.0-rc1

Tip

Cargo-binstall also ships as a GitHub Action:

- uses: cargo-bins/[email protected]
- run: cargo binstall -y cpp-linter --version 2.0.0-rc1

Using pip

Note

Any pre-release will always be published to test-pypi. Any stable release (like v2.0.0) will be published to pypi.

pip install -i https://test.pypi.org/simple/ cpp-linter

Once a stable release is published the cpp-linter-action will use the pypi releases (after a dependabot update of course).

About dependencies

Naturally, we still need to make sure clang-tidy and clang-format are installed to the system.

However, using a pre-compiled executable will relieve us of the need for a python interpreter (and venv). 😉

I've been having an itch to port clang-tools-pip to rust so we can completely remove the dependence on python in cpp-linter-action.

About docs

The user-facing docs (hosted by GitHub Pages -- see here) are separated from the rust API docs. This is mostly because any release published to crates.io will automatically have the API docs published at docs.rs.

Important

There is no python API anymore. The pyo3 library (used to build python bindings from rust code) does not support async (yet). I'm currently working around that, but it would be nice to have a full fledged python API exposed from the rust code someday.