Benchmarking

Run Benchmarks

Eradiate’s benchmarking suite is based on the Airspeed Velocity (ASV) package which enables tracking a project’s performance over time. To run the benchmark suite:

conda activate <env name>
cd benchmarks
python -m eradiate.test_tools.benchmark.cli

By default, it will run the benchmark using the currently active environment. For dev environments, this means that you need an operational version of Eradiate. Note that environments that use production versions prior to v0.28.0 cannot be benchmarked as they do no include changes required for the benchmark.

If you want to archive results to a separate asv database, add the --archive-dir=<dir> argument. The path should point to the root of the database, where the asv.conf.json file exists.

python -m eradiate.test_tools.benchmark.cli --git-ref=main^!

The benchmark results are stored in the results folder. Those can be publish to an html page and viewed by running:

# publish results to html page
asv publish
# start a server to view results
asv preview

ASV commands are also available, for custom use of ASV, please refer to the package documentation.

Write Benchmarks

Benchmarks are in the benchmarks folder. They follow the ASV syntax. The test cases that are benchmarks are often also used for regression test. For this reason, they are stored in src/eradiate/test_tools/test_cases/.