Skip to content

16-bit float segments round half up instead of to nearest even #2447

Open

Description

@bettio

Found while reviewing #2357; pre-existing (same code on release-0.7 at 18b3587, #2357 only moves it). Building a :16/float segment from a double that is not representable in half precision does not round the way IEEE 754 and BEAM do, so the bytes and the value read back differ:

F = id(6546.0),   <<F:16/float>>   %% BEAM <<16#6E,16#64>> reads back 6544.0   AtomVM <<16#6E,16#65>> reads back 6548.0
F = id(-4474.0),  <<F:16/float>>   %% BEAM <<236,94>> -> -4472.0               AtomVM <<236,95>> -> -4476.0
F = id(1.0e-6),   <<F:16/float>>   %% BEAM <<0,17>> -> 1.0132790e-6            AtomVM <<0,16>> -> 9.5367432e-7 (subnormal)

6546.0 and -4474.0 sit exactly halfway between two representable values (the spacing is 4 in [4096, 8192)); BEAM rounds the tie to the even mantissa, AtomVM always rounds it away from zero. Subnormals are not rounded at all.

Cause: in bitstring_insert_f16 (src/libAtomVM/bitstring.c:764), the normal path (:808) computes (mantissa + 0x1000) >> 13, which is round-half-up despite the "round to nearest even" comment, and the subnormal path (:801-802) shifts the mantissa right and truncates. Extraction (bitstring_extract_f16) is exact, so the error is only on construction.

Fix: round the discarded bits to nearest, ties to even, in both paths, for instance (mantissa + 0xFFF + ((mantissa >> 13) & 1)) >> 13 for normal numbers and the same scheme with the subnormal shift amount, keeping the existing carry handling into the exponent. Tests should compare bytes with BEAM for a tie (6546.0), a value just below and above a tie, and a subnormal.

Activity

  1. added 5 commits that reference this issue on Nov 6, 2026
    c75a2fe
    0ec7398
    9f2422c
    d2e76b5
    cce39f2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions