Skip to content

[FIX] Keep the next file's first 16 MB with --bufferinput and several inputs - #2404

Open
kaihere14 wants to merge 1 commit into
CCExtractor:masterfrom
kaihere14:fix/bufferinput-concat-drops-next-file
Open

kaihere14 wants to merge 1 commit into
CCExtractor:masterfrom
kaihere14:fix/bufferinput-concat-drops-next-file

Conversation

@kaihere14

@kaihere14 kaihere14 commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

In raising this pull request, I confirm the following (please check boxes):

Reason for this PR:

  • This PR adds new functionality.
  • This PR fixes a bug that I have personally experienced or that a real user has reported and for which a sample exists.
  • This PR is porting code from C to Rust.

Sanity check:

  • I have read and understood the contributors guide.
  • I have checked that another pull request for this purpose does not exist.
  • If the PR adds new functionality, I've added it to the changelog. If it's just a bug fix, I have NOT added it to the changelog.
  • I am NOT adding new C code unless it's to fix an existing, reproducible bug.

Problem

With --bufferinput and more than one input file (binary concatenation, the default), every input after the first loses its first 16 MB. An input smaller than 16 MB is skipped entirely. The captions in that part are missing from the output, the captions after it are mistimed, and the log says the file "ended prematurely".

This is a regression from #2394. That PR made init_file_buffer() reset bytesinbuffer, which fixed the default (unbuffered) mode but exposed this in buffered mode.

Cause

When the current file runs out, buffered_read_opt() calls switch_to_next_file(), which opens the next file. open() runs stream detection. With --bufferinput, detection reads up to 16 MB of the new file into filebuffer and hands it back with return_to_buffer(). Back in buffered_read_opt(), the code then does:

ctx->filebuffer_pos = keep;
ctx->bytesinbuffer = (int)i + keep; // i == 0 here

This throws away everything detection just buffered. The next read() starts where detection left the file descriptor: 16 MB into the new file, or at its end.

Before #2394 nobody noticed. bytesinbuffer was not reset, so detection replayed the previous file's leftover bytes instead of reading the new file, and this reset happened to discard those bytes.

Fix

After a successful switch_to_next_file(), continue the loop and read from the buffer as open() left it. When the input format is forced (e.g. -in=ts), open() skips detection and leaves the buffer empty, so the loop just reads the new file from its start.

The Rust port of buffered_read_opt() has the same code and gets the same change. Today it is only called by stream detection, which turns binary concatenation off, so on the Rust side this only keeps the port in sync with the C code.

Repro

Sample: VH1.ts, an ATSC MPEG transport stream with CEA-608 captions (212,140,032 bytes, sha256 f9879aff48295d8ef9055289521ff8162e671a7a10eeac931fd4b3570fabb96a). Any captioned TS longer than about 32 MB shows the same thing.

Cut the first 56.4 MB into parts and compare against the uncut file:

head -c 56400000 VH1.ts > whole.ts
head -c 28200000 whole.ts > p1.ts; tail -c +28200001 whole.ts > p2.ts   # two 28.2 MB halves
head -c 47000000 whole.ts > u1.ts; tail -c +47000001 whole.ts > u2.ts   # second part is 9.4 MB

ccextractor whole.ts -o whole.srt
ccextractor --bufferinput p1.ts p2.ts -o split.srt
ccextractor --bufferinput u1.ts u2.ts -o small.srt
cmp whole.srt split.srt
cmp whole.srt small.srt

whole.srt has 57 cues. On master (d7eb132):

  • --bufferinput p1.ts p2.ts gives 43 cues. Cues 22 to 35 (00:01:46 to 00:02:34) are missing, and every cue after them is about 60 s early (cue 36 comes out at 00:01:35,027 instead of 00:02:34,989). The log shows exactly 16 MiB missing: 28200000 − 11422784 = 16777216 = FILEBUFFERSIZE.
    Processing of p2.ts 1 ended prematurely 11422784 < 28200000, please send bug report.
    
  • --bufferinput u1.ts u2.ts gives 38 cues. Nothing from u2.ts is read:
    Processing of u2.ts 1 ended prematurely 0 < 9400000, please send bug report.
    

With this PR, both outputs are byte-identical to whole.srt.

Testing

Built with mac/build.command. Each output is compared with cmp against whole.srt:

Input master this PR
p1.ts p2.ts (default mode) identical identical
--bufferinput p1.ts p2.ts (cut on a packet boundary) 43/57 cues identical
--bufferinput two halves cut mid-packet 43/57 cues identical
--bufferinput u1.ts u2.ts (second part < 16 MB) 38/57 cues identical
--bufferinput three parts cut mid-packet 24/57 cues identical
--bufferinput -in=ts p1.ts p2.ts (no detection) identical identical
--bufferinput whole.ts (single file) identical identical

With --bufferinput and several input files (binary concatenation),
buffered_read_opt() calls switch_to_next_file() when the current file
runs out. That opens the next file, and stream detection reads the
file's first bytes into filebuffer and hands them back with
return_to_buffer(). buffered_read_opt() then reset filebuffer_pos and
bytesinbuffer to the few bytes it keeps from the old file, discarding
what detection had buffered, and went on reading from the file
descriptor, which is already past those bytes.

Since CCExtractor#2394 made init_file_buffer() reset bytesinbuffer, detection
really reads the new file, so this drops the first 16 MB
(FILEBUFFERSIZE) of every later input, or all of it when it is
smaller: its captions are lost and the following ones are mistimed.
Before CCExtractor#2394, detection replayed the previous file's leftover bytes
instead, and the reset happened to discard those.

After a successful switch, continue the loop and read from the buffer
as open() left it. Same change in the Rust port.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant