Walk the app with dua-core in place of jwalk

[dduugg]

Summary

• pks walks the app with dua-core in place of jwalk. jwalk 0.9's changelog says it is no longer maintained and points to dua-core, the walker its author now develops in dua-cli.
• pks still skips what it skipped. dua-core asks before it reads each directory, so pks still skips hidden, gitignored and excluded directories without reading them. jwalk skipped hidden entries by default, and pks now does that itself, by the same rule: a name that starts with a dot.
• Files are matched to their package.yml after the walk. Each entry dua-core yields names the directory it's in, and a file can come before its directory's package.yml. So the walk records the directories and the package.yml files it finds, and once it's done, each included file gets the package.yml nearest above it. pks no longer checks each directory for a package.yml with a stat.
• A thread drains the walk ahead of pks's loop. dua-core's workers stop once a few batches of entries wait for the loop, which happens each time the loop parses a package.yml. I opened perf: let walk workers queue more batches ahead of the iterator Byron/dua-cli#413 to let them queue more, and it merged as Options::queued_batches_per_worker, keeping the default of two batches. Once a release has it, pks can set it and drop the thread.
• The walk uses as many threads as rayon's pool, so RAYON_NUM_THREADS still limits it, as it limited jwalk, which ran on that pool.

Behavior change: symlinked directories

dua-core never follows symlinks, while jwalk did, as pks set follow_links(true). So pks now does what packwerk's Dir.glob("**/*.{rb,rake,erb}") does with a symlinked directory:

	packwerk	main	This branch
Symlink to a file	Listed	Listed	Listed
Broken symlink	Listed	Skipped	Skipped
Symlinked directory	Not descended into	Followed	Not descended into
packs/foo/loop -> ..	Not descended into	Followed until the OS stops resolving the links, about 32 levels deep on macOS and 40 on Linux, listing the same files and packs dozens of times	Not descended into

So an app with a symlinked pack, or files under a symlinked directory, loses them. The CHANGELOG says so.

One smaller difference: files in a directory named package.yml belonged to no pack on main, so check stopped with "Could not find pack for referencing file path". They now belong to the pack above it.

Equivalence

I built main and this branch and compared list-included-files, list-packs and list-references (sorted, with --no-cache) on every fixture and on four Rails apps, the largest with about 54,000 included files, 772 packs and 201,000 references. list-references names each reference's pack, so it covers the package.yml each file belongs to.

• All 120 comparisons match, but for one: list-references on app_with_monkey_patches panics on both, naming its two duplicate definitions in an order that varies from run to run on main too.
• Peak memory on the largest app is unchanged: 101.5 to 102.8 MB against 102.2 to 105.0 MB, over 3 runs each.

Performance

The time between the walk's "Beginning directory walk" and "Finished directory walk" debug logs, medians of 15 runs alternating between the binaries. The machine was busy throughout (load average 10 to 18), so the times are rough.

App	jwalk	dua-core	dua-core with the thread
Largest	433 ms	511 ms	414 ms
About 12,500 included files	93 ms	103 ms	99 ms

• Without the thread, pks's loop spent longer waiting on dua-core. In the largest app, its loop took about as long to handle the entries either way, but it waited on the walk for 134 to 184 ms per run, against 20 to 86 ms for jwalk.
• With room for 128 batches per worker in dua-core, the wait fell to 11 ms, and the loop took as long as jwalk's. #413 has the measurements and a repro that doesn't use pks.
• The two walkers are equally fast on their own. Pruning only hidden entries and pks's default excluded directories, and doing nothing else, both walk the largest app's 125,212 files and 41,408 directories in 820 to 850 ms on average.
• Matching files to their package.yml after the walk costs little. Collecting the matches into a map of the right size takes about 27 ms on 118,000 files, where growing the map took about 80.
• Before the follow-up commit, the code, rerun on all four apps at a load average of 18, was within noise of jwalk: 707 against 679 ms, 723 against 942 ms, 88 against 89 ms, and 20 against 20 ms.

Test plan

• cargo test (371 tests), with cargo fmt --check and cargo clippy --all-targets --all-features -- -Dwarnings.
• New tests in tests/list_included_files_test.rs:
  • hidden files and directories are skipped (passes on main too);
  • a symlink to a file is listed, and a broken symlink isn't (passes on main too);
  • a symlinked directory, one with a name like a Ruby file's, and loop -> .. aren't descended into (fails on main);
  • --project-root naming a file ends in "No root pack found", as on main, rather than a panic.
• A new unit test gives the package.yml resolution directories before their parents. Without the sort by depth it panics. Until now, removing the sort failed no test.
• Mutation checks: with the pruning in read_directory removed, test_list_included_files_excludes_gitignored and test_check_ignores_violations_in_gitignored_files fail, and with exists() in place of is_file() for a symlink, the symlinked-directory test fails.
• The Build jobs build and run pks on every release target.