This reference covers `hspZ`, `run`, `index`, `benchmark`, `compare`, and `hits-estimate`. CI regenerates it from the release binary and fails if the snapshot drifts.
$ hspZ --help
GPU-accelerated high-scoring ungapped alignment pair backend
Usage: hspZ <COMMAND>
Commands:
run Seed, extend and filter one reference/query pair
index Write one reference's per-bin seed tables to an on-disk index
benchmark Cold/warm benchmark: timed runs against wall clock
compare Run the C++ CUDA reference and this implementation back to back and compare runtime and HSP output
hits-estimate Exact per-unit seed-hit estimates before any GPU work (CPU-only)
help Print this message or the help of the given subcommand(s)
Options:
-h, --help Print help
-V, --version Print version
------------------------------------------------------------------------------
$ hspZ run --help
Seed, extend and filter one reference/query pair
Usage: hspZ run [OPTIONS]
Options:
-r, --reference <REFERENCE>
Reference FASTA. Optional with `--index`: the index then supplies the records, and a FASTA given here is verified against the index in the background; no output is written until it matches
-q, --query <QUERY>
--query-list <QUERY_LIST>
One query FASTA path per line (blank lines and `#` comments ignored). Batch: one reference × many queries, spread over `--gpus` workers by the same unit partition as single queries (a slot position never becomes a query-bin id). All-or-nothing: a failing job aborts the batch, so partial per-job outputs (and a truncated `-Z` archive) may remain on disk
-o, --output <OUTPUT>
Directory to write `tmp<n>.block<q>.r<r>.{plus,minus}.segments` into [default: .]
-S, --strand <STRAND>
plus / minus / both [default: both]
-s, --seed <SEED>
12of19, 14of22, or an arbitrary pattern of 1s, 0s and Ts [default: 12of19]
-e, --step <STEP>
[default: 1]
-n, --notransition
Don't allow one transition in a seed hit
-x, --xdrop <XDROP>
[default: 910]
-H, --hspthresh <HSPTHRESH>
[default: 3000]
-E, --noentropy
Don't apply the entropy correction to low-scoring segment pairs
-a, --ambiguous <AMBIGUOUS>
Ambiguous nucleotide handling: `n`, `iupac`, or `<field>,<reward>,<penalty>` [default: ""]
-c, --scoring <SCORING>
Substitution matrix in LASTZ score-set format
-T, --target-prefix <TARGET_PREFIX>
[default: ""]
-Q, --query-prefix <QUERY_PREFIX>
[default: ""]
-C, --wga-chunk-size <WGA_CHUNK_SIZE>
Query positions per seed batch; must be positive [default: 250000]
-I, --lastz-interval-size <LASTZ_INTERVAL_SIZE>
LASTZ interval size on the query; must be positive [default: 10000000]
-B, --seq-block-size <SEQ_BLOCK_SIZE>
Target bin size in bases (not a hard cut; chromosomes stay atomic). Default 500 Mbp is the KegAlign-matched digest, and it stays the multi-GPU default: with `--gpus W` the unit partition spreads its work units over the workers, while coarser layouts (down to one bin per worker) measured 17-22% slower on a 2-GPU test host. Any other target is a *different* HSP set. Never overwritten from `--gpus`; `-B 0` runs a printed automatic layout (opt-in, for per-machine measurement). See <https://hillerlab.github.io/hspZ/docs/gpus/> [default: 500000000]
--query-block-size <QUERY_BLOCK_SIZE>
Bin target for the *query* side; defaults to `--seq-block-size`. Splitting the reference is what balances workers — splitting the query only multiplies work units (`R × Q`), and each unit costs a query swap and its own `MAX_HITS` chunk boundary on every worker. Raise this above `-B` to pick a reference layout without paying for a query one. Changes the plan, so it changes the HSP set: keep a digest per layout
-m, --max-hits <MAX_HITS>
Semantic hits per GPU chunk (target H); 0 derives KegAlign's `4194304 * GiB` default. Physical capacity C>=H is derived from free VRAM and only affects success/failure, never successful output bytes [default: 0]
-t, --threads <THREADS>
Worker threads for seed generation; 0 uses available parallelism [default: 0]
-b, --hsp-blocks <HSP_BLOCKS>
N8: `find_hsps` grid blocks; 0 uses the compiled-in optimum (16384) [default: 0]
-P, --no-pinned-seeds
Stage seeds in pageable memory. The default stages them in pinned host memory, which measured -0.5..-0.8% on an L4 and is what makes the H->D copy a DMA; backends without `cuMemHostAlloc` fall back automatically
--no-persistent-seed-buffers
Reallocate `d_seeds` / `d_hit_num` per batch. The default reuses them, worth -3.5% (A) / -3.8% (B) on an L4
--kegalign-bins
Bin records the way KegAlign does — sequential fill in input order, closing a block once it exceeds `--seq-block-size` — instead of the balanced planner. For matched-granularity benchmarking only: it makes block membership, and so the dedup scope, identical on both sides. The planner will not shrink the block size to fit the GPU in this mode; it errors instead, because shrinking would unmatch the granularity
--dump-plan <DUMP_PLAN>
Write the plan's bin membership to this file and continue: one `side<TAB>bin<TAB>record_name<TAB>bp` line per record. Matched-granularity benchmarking compares it against KegAlign's `{ref,query}_block*.name`, which is the only way to *show* both tools binned the input the same way
--dump-manifest <DUMP_MANIFEST>
Write a frozen executable plan (GPU `run` only, same binary required for replay). A second node given `--from-manifest` will *validate fit and fail*, not replan
--from-manifest <FROM_MANIFEST>
Replay a frozen plan from `--dump-manifest` (GPU `run` only). Requires the same binary, inputs, resolved scoring matrix, strand and record prefixes. The planner will not shrink bins or the hit cap
-G, --gpus <GPUS>
GPUs to run on. With one worker per device and every device the same class (SM count, L2, clocks within 10%), the plan's work units are split across workers by count quotas; otherwise each worker owns whole reference bins by deterministic LPT (`HSPZ_UNIT_PARTITION=0|1` forces either). Output follows `WorkUnit.ordinal`, so it does not depend on which GPU finished first. More workers than devices time-slices one GPU: a correctness configuration, not a performance one. Batch mode (`--query-list`) uses the same policy over `(job, unit)` slots [default: 1]
--no-async-stages
Wait for the GPU after every stage instead of enqueueing the whole per-batch chain. The default enqueues and waits only where the host needs a device result; on its own that was neutral (round 29), but together with the overlapped seed upload it is worth -1.51% on an L4 with disjoint ranges over six paired rounds (round 30)
--no-async-seed-copy
Upload each batch's seeds with a blocking copy at the point of use. The default uploads on a second stream one batch ahead, so the DMA overlaps the previous batch's kernels: exposed `H->D seeds` 5,734 -> 72 ms on an L4. The overlap needs pinned staging and the persistent seed buffers, and turns itself off when either is missing (ZLUDA has no `cuMemHostAlloc`, and an async copy from pageable memory blocks anyway)
--no-ref-prefetch
Build each reference bin only when its turn comes. The default builds the next bin's pack + seed table on a worker thread while the current bin's work units run on the GPU; that build is the largest host cost the GPU cannot otherwise hide (7.1% of an L4 multi5 run)
--index <INDEX>
Load per-bin reference tables from an on-disk index written by `hspZ index` instead of rebuilding them. The index MANIFEST supplies the reference records for planning, so `-r` is optional; a reference given with `-r` is verified against the index in the background before any output is written. The planned reference bins must equal the index's, or it errors naming the mismatch
-y, --time
Report the full wall-time accounting
-d, --hit-stats
Report the hits-per-seed distribution
-u, --cpu-only
Stop after CPU preprocessing and report seed/hit counts. Needs no GPU
--dump-raw <DUMP_RAW>
Write every pre-dedup HSP to this file as `strand ref_start query_start len score`
-D, --diagonal-partition
Split each output file along its diagonal, KegAlign-style, emitting `*.split1.segments` onward instead of one file. Partitions the HSP structs directly — no unsplit file is written first
-Z, --tarball [<TARBALL>]
Write output into one `.tar.gz` instead of a directory. Without a path, `<output>.tar.gz` is used. Archived directly from the formatted bytes. In batch mode (`--query-list`) one archive per job is written (`OUT/000001.tar.gz` …) and a custom path value is ignored (a note is printed)
-h, --help
Print help
------------------------------------------------------------------------------
$ hspZ index --help
Write one reference's per-bin seed tables to an on-disk index
Usage: hspZ index [OPTIONS] --reference <REFERENCE> --index <INDEX>
Options:
-r, --reference <REFERENCE>
Reference FASTA (or FASTA.gz / 2bit) to index
--index <INDEX>
Directory to write the index into. Required; must not exist yet (no --force: delete it or pick another path)
-s, --seed <SEED>
12of19, 14of22, or an arbitrary pattern of 1s, 0s and Ts [default: 12of19]
-e, --step <STEP>
[default: 1]
-B, --seq-block-size <SEQ_BLOCK_SIZE>
Target bin size in bases, as in `run` (`-B 0` rejected: the automatic layout is device-dependent and cannot be frozen into an index) [default: 500000000]
--kegalign-bins
Bin records the way KegAlign does — sequential fill in input order — instead of the balanced planner. Must match the `run` side
-t, --threads <THREADS>
Worker threads for seed generation; 0 uses available parallelism [default: 0]
-y, --time
Report per-bin build/write timings and payload sizes
-h, --help
Print help
------------------------------------------------------------------------------
$ hspZ benchmark --help
Cold/warm benchmark: timed runs against wall clock
Usage: hspZ benchmark [OPTIONS] --reference <REFERENCE>
Options:
-r, --reference <REFERENCE>
-q, --query <QUERY>
--query-list <QUERY_LIST>
One query FASTA path per line (blank lines and `#` comments ignored). Batch: one reference × many queries, spread over `--gpus` workers by the same unit partition as single queries (a slot position never becomes a query-bin id). All-or-nothing: a failing job aborts the batch, so partial per-job outputs (and a truncated `-Z` archive) may remain on disk
-o, --output <OUTPUT>
Directory to write `tmp<n>.block<q>.r<r>.{plus,minus}.segments` into [default: .]
-S, --strand <STRAND>
plus / minus / both [default: both]
-s, --seed <SEED>
12of19, 14of22, or an arbitrary pattern of 1s, 0s and Ts [default: 12of19]
-e, --step <STEP>
[default: 1]
-n, --notransition
Don't allow one transition in a seed hit
-x, --xdrop <XDROP>
[default: 910]
-H, --hspthresh <HSPTHRESH>
[default: 3000]
-E, --noentropy
Don't apply the entropy correction to low-scoring segment pairs
-a, --ambiguous <AMBIGUOUS>
Ambiguous nucleotide handling: `n`, `iupac`, or `<field>,<reward>,<penalty>` [default: ""]
-c, --scoring <SCORING>
Substitution matrix in LASTZ score-set format
-T, --target-prefix <TARGET_PREFIX>
[default: ""]
-Q, --query-prefix <QUERY_PREFIX>
[default: ""]
-C, --wga-chunk-size <WGA_CHUNK_SIZE>
Query positions per seed batch; must be positive [default: 250000]
-I, --lastz-interval-size <LASTZ_INTERVAL_SIZE>
LASTZ interval size on the query; must be positive [default: 10000000]
-B, --seq-block-size <SEQ_BLOCK_SIZE>
Target bin size in bases (not a hard cut; chromosomes stay atomic). Default 500 Mbp is the KegAlign-matched digest, and it stays the multi-GPU default: with `--gpus W` the unit partition spreads its work units over the workers, while coarser layouts (down to one bin per worker) measured 17-22% slower on a 2-GPU test host. Any other target is a *different* HSP set. Never overwritten from `--gpus`; `-B 0` runs a printed automatic layout (opt-in, for per-machine measurement). See <https://hillerlab.github.io/hspZ/docs/gpus/> [default: 500000000]
--query-block-size <QUERY_BLOCK_SIZE>
Bin target for the *query* side; defaults to `--seq-block-size`. Splitting the reference is what balances workers — splitting the query only multiplies work units (`R × Q`), and each unit costs a query swap and its own `MAX_HITS` chunk boundary on every worker. Raise this above `-B` to pick a reference layout without paying for a query one. Changes the plan, so it changes the HSP set: keep a digest per layout
-m, --max-hits <MAX_HITS>
Semantic hits per GPU chunk (target H); 0 derives KegAlign's `4194304 * GiB` default. Physical capacity C>=H is derived from free VRAM and only affects success/failure, never successful output bytes [default: 0]
-t, --threads <THREADS>
Worker threads for seed generation; 0 uses available parallelism [default: 0]
-b, --hsp-blocks <HSP_BLOCKS>
N8: `find_hsps` grid blocks; 0 uses the compiled-in optimum (16384) [default: 0]
-P, --no-pinned-seeds
Stage seeds in pageable memory. The default stages them in pinned host memory, which measured -0.5..-0.8% on an L4 and is what makes the H->D copy a DMA; backends without `cuMemHostAlloc` fall back automatically
--no-persistent-seed-buffers
Reallocate `d_seeds` / `d_hit_num` per batch. The default reuses them, worth -3.5% (A) / -3.8% (B) on an L4
--kegalign-bins
Bin records the way KegAlign does — sequential fill in input order, closing a block once it exceeds `--seq-block-size` — instead of the balanced planner. For matched-granularity benchmarking only: it makes block membership, and so the dedup scope, identical on both sides. The planner will not shrink the block size to fit the GPU in this mode; it errors instead, because shrinking would unmatch the granularity
--dump-plan <DUMP_PLAN>
Write the plan's bin membership to this file and continue: one `side<TAB>bin<TAB>record_name<TAB>bp` line per record. Matched-granularity benchmarking compares it against KegAlign's `{ref,query}_block*.name`, which is the only way to *show* both tools binned the input the same way
--dump-manifest <DUMP_MANIFEST>
Write a frozen executable plan (GPU `run` only, same binary required for replay). A second node given `--from-manifest` will *validate fit and fail*, not replan
--from-manifest <FROM_MANIFEST>
Replay a frozen plan from `--dump-manifest` (GPU `run` only). Requires the same binary, inputs, resolved scoring matrix, strand and record prefixes. The planner will not shrink bins or the hit cap
-G, --gpus <GPUS>
GPUs to run on. With one worker per device and every device the same class (SM count, L2, clocks within 10%), the plan's work units are split across workers by count quotas; otherwise each worker owns whole reference bins by deterministic LPT (`HSPZ_UNIT_PARTITION=0|1` forces either). Output follows `WorkUnit.ordinal`, so it does not depend on which GPU finished first. More workers than devices time-slices one GPU: a correctness configuration, not a performance one. Batch mode (`--query-list`) uses the same policy over `(job, unit)` slots [default: 1]
--no-async-stages
Wait for the GPU after every stage instead of enqueueing the whole per-batch chain. The default enqueues and waits only where the host needs a device result; on its own that was neutral (round 29), but together with the overlapped seed upload it is worth -1.51% on an L4 with disjoint ranges over six paired rounds (round 30)
--no-async-seed-copy
Upload each batch's seeds with a blocking copy at the point of use. The default uploads on a second stream one batch ahead, so the DMA overlaps the previous batch's kernels: exposed `H->D seeds` 5,734 -> 72 ms on an L4. The overlap needs pinned staging and the persistent seed buffers, and turns itself off when either is missing (ZLUDA has no `cuMemHostAlloc`, and an async copy from pageable memory blocks anyway)
--no-ref-prefetch
Build each reference bin only when its turn comes. The default builds the next bin's pack + seed table on a worker thread while the current bin's work units run on the GPU; that build is the largest host cost the GPU cannot otherwise hide (7.1% of an L4 multi5 run)
--index <INDEX>
Load per-bin reference tables from an on-disk index written by `hspZ index` instead of rebuilding them. The run still parses the reference and plans as usual, then requires the planned reference bins to equal the index's, or it errors naming the mismatch
-y, --time
Report the full wall-time accounting
-d, --hit-stats
Report the hits-per-seed distribution
-u, --cpu-only
Stop after CPU preprocessing and report seed/hit counts. Needs no GPU
--dump-raw <DUMP_RAW>
Write every pre-dedup HSP to this file as `strand ref_start query_start len score`
-D, --diagonal-partition
Split each output file along its diagonal, KegAlign-style, emitting `*.split1.segments` onward instead of one file. Partitions the HSP structs directly — no unsplit file is written first
-Z, --tarball [<TARBALL>]
Write output into one `.tar.gz` instead of a directory. Without a path, `<output>.tar.gz` is used. Archived directly from the formatted bytes. In batch mode (`--query-list`) one archive per job is written (`OUT/000001.tar.gz` …) and a custom path value is ignored (a note is printed)
-i, --iterations <ITERATIONS>
Timed Seed + Filter iterations after the warmup [default: 20]
-w, --warmup <WARMUP>
Untimed iterations first [default: 3]
-l, --launch-probe <LAUNCH_PROBE>
Launches used to price ZLUDA's per-launch overhead [default: 200]
-j, --json <JSON>
Append one JSON record per warm iteration to this file
-v, --variant <VARIANT>
Variant label recorded in each JSON record (e.g. `nvidia-segment-align16`) [default: baseline]
-k, --workload <WORKLOAD>
Workload label recorded in each JSON record (e.g. `A`) [default: unnamed]
-f, --env-json <ENV_JSON>
A JSON object merged into each record's `environment`. The runner fills this with the things a shell knows and this binary should not shell out for: git commit, rustc/cargo versions, GPU name, Slurm ids [default: {}]
-h, --help
Print help
------------------------------------------------------------------------------
$ hspZ compare --help
Run the C++ CUDA reference and this implementation back to back and compare runtime and HSP output
Usage: hspZ compare [OPTIONS] --reference <REFERENCE> --query <QUERY>
Options:
-r, --reference <REFERENCE>
-q, --query <QUERY>
-k, --kegalign <KEGALIGN>
The C++ CUDA oracle [default: /tmp/kegalign/build/kegalign]
-L, --ld-preload <LD_PRELOAD>
`LD_PRELOAD` for the oracle process. Empty (the default) inherits this process's environment. Running the Thrust/CUB reference under ZLUDA needs a legacy-stream shim here, so a ZLUDA host must pass its own path [default: ""]
-l, --ld-library-path <LD_LIBRARY_PATH>
`LD_LIBRARY_PATH` for the oracle process. Empty (the default) inherits this process's environment rather than clearing it [default: ""]
-w, --workdir <WORKDIR>
Where to keep both runs' output. Defaults to a fresh temp directory
-s, --seed <SEED>
[default: 12of19]
-x, --xdrop <XDROP>
[default: 910]
-H, --hspthresh <HSPTHRESH>
[default: 3000]
-c, --scoring <SCORING>
-h, --help
Print help
------------------------------------------------------------------------------
$ hspZ hits-estimate --help
Exact per-unit seed-hit estimates before any GPU work (CPU-only)
Usage: hspZ hits-estimate [OPTIONS] --reference <REFERENCE> --query <QUERY>
Options:
-r, --reference <REFERENCE>
-q, --query <QUERY>
-B, --seq-block-size <SEQ_BLOCK_SIZE>
Target bin size in bases, as in `run` (`-B 0` resolves to the default) [default: 500000000]
--query-block-size <QUERY_BLOCK_SIZE>
Bin target for the query side; defaults to `--seq-block-size`
--kegalign-bins
Bin records the way KegAlign does (sequential fill, not LPT)
--seed <SEED>
12of19, 14of22, or an arbitrary pattern of 1s, 0s and Ts (k<=12) [default: 12of19]
-e, --step <STEP>
[default: 1]
--notransition
Don't allow one transition in a seed hit
-S, --strand <STRAND>
plus / minus / both [default: both]
-t, --threads <THREADS>
Worker threads for histogram passes; 0 uses available parallelism [default: 0]
-I, --lastz-interval-size <LASTZ_INTERVAL_SIZE>
LASTZ interval size, as in `run` (`-I`): interval boundaries decide which shared window starts the pipeline seeds once and which twice [default: 10000000]
-C, --wga-chunk-size <WGA_CHUNK_SIZE>
Query seed chunk size, as in `run` (`-C`): the interval schedule (`seed::chunks`) is part of the count, so this must match `run` [default: 250000]
--stride <STRIDE>
Count every S-th query window start and scale by S (approximation) [default: 1]
--dump-plan <DUMP_PLAN>
Write the plan's bin membership to this file and continue
-h, --help
Print help