Skip to content

[FIX] --sentencecap, --capfile, --kf, --profanity-file miss most words (word lists never sorted) - #2397

Open
Deepak-negi11 wants to merge 1 commit into
CCExtractor:masterfrom
Deepak-negi11:fix/sentencecap-sort
Open

Deepak-negi11 wants to merge 1 commit into
CCExtractor:masterfrom
Deepak-negi11:fix/sentencecap-sort

Conversation

@Deepak-negi11

@Deepak-negi11 Deepak-negi11 commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

In raising this pull request, I confirm the following (please check boxes):

Reason for this PR:

  • This PR adds new functionality.
  • This PR fixes a bug that I have personally experienced or that a real user has reported and for which a sample exists.
  • This PR is porting code from C to Rust.

Sanity check:

  • I have read and understood the contributors guide.
  • I have checked that another pull request for this purpose does not exist.
  • If the PR adds new functionality, I've added it to the changelog. If it's just a bug fix, I have NOT added it to the changelog.
  • I am NOT adding new C code unless it's to fix an existing, reproducible bug.

Repro instructions:

Sample SCC file with these four captions: "I LIKE JUNE AND MONDAY.", "SEE YOU IN JANUARY.", "WHAT THE SHIT IS THIS.", "HELLO WORLD, ZONK BLURG." (I can attach it). words.txt contains WoRlD and HeLLo; bad.txt contains zonk and blurg.

Command master this PR
--sentencecap I like june and monday. I like June and Monday.
--capfile words.txt Hello WoRlD / See you in january. HeLLo WoRlD / See you in January.
--profanity-file bad.txt HELLO WORLD, ZONK BLURG. HELLO WORLD, **** *****.

Problem

The C encoders look words up with bsearch() and strcasecmp (ccx_encoders_helpers.c), which needs the lists sorted the same way. The old C parser sorted them with ccx_encoders_helpers_perform_shellsort_words(). The Rust parser builds the lists (built-in words, then the user's file) but never sorts them, and nothing calls that function anymore. So most words are missed, depending on where they land in the list.

Fix

Sort both lists in parse_parameters() right after they're built, in strcasecmp order (byte-wise, ASCII-lowercased).

Testing

  • New test: the sort order matches strcasecmp (including _, which sorts differently with lowercase vs uppercase comparison).
  • New test: every word of the real --sentencecap and --kf lists is found by binary search, as bsearch does. Fails on master.
  • cargo test (428 passed), cargo fmt --check, cargo clippy -- -D warnings.
  • Built on macOS and ran the table above.

Copilot AI balanced review requested due to automatic review settings October 18, 2026 11:47

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@Deepak-negi11 Deepak-negi11 changed the title [PROPOSAL] The C-vs-Rust dual-build comparison from #2207 never runs; the C-only build drops 708 and HEVC captions [FIX] --sentencecap, --capfile, --kf, --profanity-file miss most words (word lists never sorted) Oct 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants