Fix page hash size limit for large PE files

[francoisgirinon-netizen]

Pull Request Type

• Bug fix

Related Issue

Issue number: N/A

Current Behavior

Signing a large PE file with -ph fails with:

Page hash size exceeds limit: 32683176
Failed to calculate page hash
Failed to get spcIndirectDataContent
Unable to prepare new signature

pe_page_hash_calc() sizes its buffer from the size of the whole file
(fileend / pagesize) and rejects anything above 16 MB. Data that is not part
of a section, such as an overlay appended by an installer, is not hashed, but it
was still counted. Reproduced on a real 3.7 GB Inno Setup installer (SectionAlignment 4096):
the estimate was 31 MB while the actual page hash is about 4.5 KB. Without -ph
the same file signs fine.

With SHA-256 and 4 KB pages the limit was reached at about 1.9 GB (SHA-1: 2.9 GB),
even for files whose sections are small.

New Behavior

The number of pages is computed from the section table, so only hashed data counts.
The bounds of each section are checked before its size is used, and the estimate
never exceeds the previous one based on the file size, so a crafted section table
cannot cause a larger allocation than before. The sanity limit is raised to 512 MB. A PE file with a certificate table is limited
to 4 GB, and with 512 byte pages (the minimum FileAlignment) the page hash cannot be
larger than that.

The same installer now signs with -ph and verify reports the signature, the
message digest and the page hash as valid. For files that already worked the output
is unchanged: signing tests/files/unsigned.exe with -ph -time 1556668800 gives a
byte-identical file before and after the change, with both SHA-1 and SHA-256.

Scope of Changes

• pe.c: pe_page_hash_calc() computes the size of the page hash buffer from
  the SizeOfRawData of each section instead of the file size.
• osslsigncode.h: add SIZE_512M.
• cmake/CMakeTest.cmake, tests/make_large_pe.py: optional tests, enabled with
  -DLARGE_FILE_TESTS=ON (off by default: they need about 16 GB of disk space and a
  64-bit build).
  They sign and verify a 2.6 GB PE with -ph, with the data in a section and as an overlay.
  A third test signs a PE with a section of 0xFFFFF200 bytes (about 4 GB, so about
  16 GB of disk space in total). It is not verified: the signed file is over 4 GB and
  cannot be opened again.
• pe.c: the page offset l in pe_page_hash_calc() is uint64_t. With the higher
  limit, l += pagesize could wrap to zero for a section close to 4 GB.

Testing

• Existing tests

• New tests added

• Manual testing

• ctest: 381/381 passed on Linux x86-64 (OpenSSL 3.5), with -DLARGE_FILE_TESTS=ON.

• Both new tests fail without the change with Page hash size exceeds limit: 22851684.

• Manual: sign -h sha256 -ph then verify on a 3.7 GB installer.

• A crafted 8 KB PE with a section size of 0xFFFFFE00 is rejected with the same
  "invalid bounds" error as before, without a large allocation.

• Not tested: Windows, macOS, 32-bit builds.

Additional Notes

• The limit of 4 GB for a signed PE file comes from the format (32-bit offset and
  size of the certificate table) and is not changed here.

License Declaration

• I hereby agree to license my contribution under the project's license.

[mtrojnar]

lgtm

[olszomal]

The fix looks good overall. Thank you for your contribution!

[olszomal]

Increasing the page hash limit from 16 MB to 512 MB makes an existing uint32_t overflow in the page hashing loop reachable.

For a section close to 4 GB (e.g. SizeOfRawData = 0xFFFFF200 with 4096-byte pages), l += pagesize wraps to zero, causing the loop to hash the section again until the page hash buffer limit is reached.

Previously, such files were rejected by the 16 MB limit before reaching this loop.

I suggest changing l to uint64_t as part of this PR.

[francoisgirinon-netizen]

Good catch, thanks. Fixed in 00e425e: l is now uint64_t (the PUT_UINT32_LE argument is cast back, it cannot exceed the section bounds already checked against the file size).

I added a test in e87d27c (large_wrap_*, under LARGE_FILE_TESTS): a PE with a 0xFFFFF200 byte section. Without the fix, sign -ph fails with "Failed to calculate page hash"; with it, signing succeeds.

Note that the signed output is over 4 GB and cannot be opened again (32-bit file size limit in get_file_size()), so that test only signs and does not verify. That limit is unrelated to this PR.