fix(wc): respect C/POSIX locale for character counting by naoNao89 · Pull Request #11006 · uutils/coreutils

naoNao89 · 2026-02-18T04:27:10Z

Fix wc -m to count bytes as characters in C/POSIX locales, matching GNU coreutils.

Apply locale-aware counting in both fast and decoded paths, including invalid UTF-8 bytes.
Reuse uucore locale encoding detection.
Preserve UTF-8 counting on WASI when no locale is available.

tests:

Add regression tests for C/POSIX and UTF-8 locales, covering both counting paths (-m and -wm) and invalid byte sequences.

Fixes #9712
Refs #5831

github-actions · 2026-02-18T04:38:12Z

GNU testsuite comparison:

GNU test failed: tests/rm/isatty. tests/rm/isatty is passing on 'main'. Maybe you have to rebase?
GNU test failed: tests/tail/retry. tests/tail/retry is passing on 'main'. Maybe you have to rebase?
Congrats! The gnu test tests/date/date-locale-hour is no longer failing!
Congrats! The gnu test tests/cp/link-heap is now passing!

codspeed · 2026-02-18T04:50:59Z

Merging this PR will not alter performance

1 warning

⚠️ 9 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

✅ 22 untouched benchmarks
⏩ 450 skipped benchmarks¹

_{Comparing naoNao89:fix-wc-locale-chars (49cd7ea) with main (4939cd4)²}

450 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
No successful run was found on main (2f8b194) during the generation of this report, so 4939cd4 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

sylvestre · 2026-02-18T09:08:09Z

As no human would write such duplication of code, i guess it is LLM generated ...
Please review the changes before submitting them for review...

cakebaker · 2026-02-18T10:51:48Z

In the newly added tests you always use -m or -cm. However, this means only your changes in count_fast.rs are tested due to the logic in word_count_from_reader in wc.rs. To test your changes in wc.rs you also have to provide -w or -L.

naoNao89 · 2026-02-18T12:44:25Z

sr, refactored 💀 ~~dup is_c_or_posix_locale()~~

sylvestre · 2026-02-18T12:54:49Z

@@ -616,11 +617,21 @@ fn process_chunk<
            total.lines += 1;
        }
        if SHOW_CHARS {
-            total.chars += 1;
+            if chars_are_bytes {


seriously ?!
please review your patches before substitutions ...

i thought clippy had the ability to check for empty if :v

github-actions · 2026-02-18T13:00:30Z

GNU testsuite comparison:

GNU test failed: tests/date/date-locale-hour. tests/date/date-locale-hour is passing on 'main'. Maybe you have to rebase?
Skipping an intermittent issue tests/pr/bounded-memory (passes in this run but fails in the 'main' branch)
Congrats! The gnu test tests/cut/bounded-memory is no longer failing!

github-actions · 2026-02-18T13:11:07Z

GNU testsuite comparison:

GNU test failed: tests/rm/isatty. tests/rm/isatty is passing on 'main'. Maybe you have to rebase?
Note: The gnu test tests/rm/many-dir-entries-vs-OOM is now being skipped but was previously passing.

github-actions · 2026-02-18T14:08:13Z

GNU testsuite comparison:

Congrats! The gnu test tests/cut/bounded-memory is no longer failing!

github-actions · 2026-05-25T06:21:27Z

GNU testsuite comparison:

Congrats! The gnu test tests/wc/wc is no longer failing!

sylvestre · 2026-05-25T08:47:51Z


+/// Check if the current locale is C or POSIX (where characters == bytes).
+/// This follows GNU coreutils behavior where MB_CUR_MAX == 1 in these locales.
+pub(crate) fn is_c_or_posix_locale() -> bool {


We have two functions already doing this:

src/uucore/src/lib/features/i18n/charmap.rs:33 - inside get_encoding(), uses the same ["LC_ALL", "LC_CTYPE", "LANG"] lookup and the same "C"/"POSIX" short-circuit. Not exposed as a
public helper.

src/uu/uniq/src/uniq.rs:189 - Uniq::is_c_locale(), a private method with the identical env
lookup, used for the exact same purpose (chars==bytes in C/POSIX).

maybe dedup this?

naoNao89 · 2026-05-26T07:39:37Z

tks, i wasn't sure how broad you wanted the dedup to be

The latest commit adds a shared helper in uucore and reuses it from both wc and uniq, since both had the same C/POSIX locale check. If you prefer keeping this PR narrower, I can limit the change to wc only

sylvestre · 2026-05-26T07:41:29Z

+///
+/// Empty values are ignored, matching the locale precedence used for character
+/// map detection.
+pub fn get_effective_ctype_locale() -> Option<String> {


does it need to be pub ?

this doesn't need to be pub, only the boolean helper is used outside this module, so will make the raw locale lookup private

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.

sylvestre · 2026-10-21T07:07:38Z

                if COUNT_CHARS {
-                    total.chars += if simd_allowed {
-                        bytecount::num_chars(&buf[..n])
+                    if chars_are_bytes {


could be a single if chars_are_bytes { n } else if simd_allowed { .. } else { .. }, and drop the duplicated comment

sylvestre · 2026-10-21T07:07:48Z

    let mut in_word = false;
    let mut current_len = 0;
    let is_posixly_correct = *IS_POSIXLY_CORRECT;
+    let chars_are_bytes = SHOW_CHARS && locale_encoding() == Encoding::SingleByte;


chars_are_bytes already includes SHOW_CHARS, so do we still need the new generic on handle_error?

sylvestre · 2026-10-21T07:07:57Z


 #[test]
 fn test_utf8() {
+    // Requires UTF-8 locale for character counting


same comment 11 times, please remove them :)

sylvestre · 2026-10-21T07:08:03Z

 }
+
+#[test]
+#[cfg_attr(wasi_runner, ignore = "WASI has no native C/POSIX locale")]


why ignore on wasi? the runner forwards LC_ALL via --env, so this should pass there too

i missed the --env forwarding, iwll remove the ignore and let WASI CI verify

sylvestre · 2026-10-21T07:08:09Z

+#[cfg_attr(wasi_runner, ignore = "WASI has no native C/POSIX locale")]
+fn test_wc_chars_default_locale() {
+    // When no locale is set (empty LC_ALL), it defaults to POSIX (chars == bytes)
+    // This ensures backward compatibility


this isn't backward compatibility, it changes the default behavior

sylvestre · 2026-10-21T07:08:15Z

+
+#[test]
+#[cfg_attr(wasi_runner, ignore = "WASI has no native C/POSIX locale")]
+fn test_wc_multibyte_c_locale() {


mostly duplicates test_wc_chars_c_locale, could you merge them into one loop over inputs?

sylvestre · 2026-10-21T10:17:25Z

thanks for the update!
Mostly cleanup left: the redundant SHOW_CHARS generic on handle_error, the repeated test comments and the wasi ignores, see inline.
Also, the commit "keep WASI default locale UTF-8 compatible" only adds test ignores. Please fix its message or squash it.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.

- Count bytes instead of UTF-8 characters for wc -m in C/POSIX (single-byte) locales, matching GNU coreutils. - Cover both the fast path (count_fast.rs) and the decoder path (wc.rs), including invalid byte sequences in single-byte locales. - Leave UTF-8 invalid-byte handling and wc -L unchanged. - Add byte-exact regression tests with explicit locales for both paths. Fixes uutils#9712. Refs uutils#5831.

cakebaker reviewed Feb 18, 2026

View reviewed changes

Comment thread src/uu/wc/src/wc.rs Outdated

cakebaker reviewed Feb 18, 2026

View reviewed changes

Comment thread src/uu/wc/src/count_fast.rs Outdated

cakebaker reviewed Feb 18, 2026

View reviewed changes

naoNao89 force-pushed the fix-wc-locale-chars branch 3 times, most recently from d906c13 to 1ed5ccd Compare February 18, 2026 12:51

sylvestre reviewed Feb 18, 2026

View reviewed changes

naoNao89 force-pushed the fix-wc-locale-chars branch from 1ed5ccd to bf04096 Compare February 18, 2026 13:02

naoNao89 force-pushed the fix-wc-locale-chars branch from 224212a to 3eb470f Compare May 25, 2026 06:05

sylvestre reviewed May 25, 2026

View reviewed changes

sylvestre reviewed May 26, 2026

View reviewed changes

naoNao89 force-pushed the fix-wc-locale-chars branch 4 times, most recently from 32cccc9 to 3effba3 Compare May 26, 2026 09:14

Copilot AI balanced review requested due to automatic review settings October 4, 2026 18:13

sylvestre force-pushed the fix-wc-locale-chars branch from 3effba3 to 2b823e4 Compare October 4, 2026 18:13

Copilot AI reviewed Oct 4, 2026

naoNao89 marked this pull request as draft October 20, 2026 21:30

naoNao89 force-pushed the fix-wc-locale-chars branch from 88c7571 to 2e332a9 Compare October 21, 2026 03:21

naoNao89 marked this pull request as ready for review October 21, 2026 03:21

Copilot AI balanced review requested due to automatic review settings October 21, 2026 03:21

Copilot AI reviewed Oct 21, 2026

sylvestre reviewed Oct 21, 2026

View reviewed changes

Copilot AI balanced review requested due to automatic review settings October 21, 2026 11:27

naoNao89 force-pushed the fix-wc-locale-chars branch from 2e332a9 to d4fce06 Compare October 21, 2026 11:27

Copilot AI reviewed Oct 21, 2026

Copilot AI balanced review requested due to automatic review settings October 21, 2026 11:32

naoNao89 force-pushed the fix-wc-locale-chars branch from d4fce06 to 8cab248 Compare October 21, 2026 11:32

Copilot AI reviewed Oct 21, 2026

sylvestre reviewed Oct 21, 2026

View reviewed changes

Comment thread src/uu/wc/src/count_fast.rs Outdated

sylvestre reviewed Oct 21, 2026

View reviewed changes

Comment thread src/uu/wc/src/wc.rs Outdated

sylvestre reviewed Oct 21, 2026

View reviewed changes

Comment thread tests/by-util/test_wc.rs Outdated

Copilot AI balanced review requested due to automatic review settings October 21, 2026 19:49

naoNao89 force-pushed the fix-wc-locale-chars branch from 8cab248 to a40113f Compare October 21, 2026 19:49

Copilot AI reviewed Oct 21, 2026

naoNao89 force-pushed the fix-wc-locale-chars branch from a40113f to 49cd7ea Compare October 21, 2026 20:35

Conversation

naoNao89 commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Feb 18, 2026

Uh oh!

codspeed Bot commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

sylvestre commented Feb 18, 2026

Uh oh!

Uh oh!

Uh oh!

cakebaker Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

naoNao89 commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Feb 18, 2026

Uh oh!

github-actions Bot commented Feb 18, 2026

Uh oh!

github-actions Bot commented Feb 18, 2026

Uh oh!

github-actions Bot commented May 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

naoNao89 commented May 26, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sylvestre commented Oct 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

naoNao89 commented Feb 18, 2026 •

edited

Loading

codspeed Bot commented Feb 18, 2026 •

edited

Loading

cakebaker Feb 18, 2026 •

edited

Loading

naoNao89 commented Feb 18, 2026 •

edited

Loading

github-actions Bot commented May 25, 2026 •

edited

Loading