Repository navigation
Conversation
With --bufferinput and several input files (binary concatenation), buffered_read_opt() calls switch_to_next_file() when the current file runs out. That opens the next file, and stream detection reads the file's first bytes into filebuffer and hands them back with return_to_buffer(). buffered_read_opt() then reset filebuffer_pos and bytesinbuffer to the few bytes it keeps from the old file, discarding what detection had buffered, and went on reading from the file descriptor, which is already past those bytes. Since CCExtractor#2394 made init_file_buffer() reset bytesinbuffer, detection really reads the new file, so this drops the first 16 MB (FILEBUFFERSIZE) of every later input, or all of it when it is smaller: its captions are lost and the following ones are mistimed. Before CCExtractor#2394, detection replayed the previous file's leftover bytes instead, and the reset happened to discard those. After a successful switch, continue the loop and read from the buffer as open() left it. Same change in the Rust port.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In raising this pull request, I confirm the following (please check boxes):
Reason for this PR:
Sanity check:
Problem
With
--bufferinputand more than one input file (binary concatenation, the default), every input after the first loses its first 16 MB. An input smaller than 16 MB is skipped entirely. The captions in that part are missing from the output, the captions after it are mistimed, and the log says the file "ended prematurely".This is a regression from #2394. That PR made
init_file_buffer()resetbytesinbuffer, which fixed the default (unbuffered) mode but exposed this in buffered mode.Cause
When the current file runs out,
buffered_read_opt()callsswitch_to_next_file(), which opens the next file.open()runs stream detection. With--bufferinput, detection reads up to 16 MB of the new file intofilebufferand hands it back withreturn_to_buffer(). Back inbuffered_read_opt(), the code then does:This throws away everything detection just buffered. The next
read()starts where detection left the file descriptor: 16 MB into the new file, or at its end.Before #2394 nobody noticed.
bytesinbufferwas not reset, so detection replayed the previous file's leftover bytes instead of reading the new file, and this reset happened to discard those bytes.Fix
After a successful
switch_to_next_file(),continuethe loop and read from the buffer asopen()left it. When the input format is forced (e.g.-in=ts),open()skips detection and leaves the buffer empty, so the loop just reads the new file from its start.The Rust port of
buffered_read_opt()has the same code and gets the same change. Today it is only called by stream detection, which turns binary concatenation off, so on the Rust side this only keeps the port in sync with the C code.Repro
Sample:
VH1.ts, an ATSC MPEG transport stream with CEA-608 captions (212,140,032 bytes, sha256f9879aff48295d8ef9055289521ff8162e671a7a10eeac931fd4b3570fabb96a). Any captioned TS longer than about 32 MB shows the same thing.Cut the first 56.4 MB into parts and compare against the uncut file:
whole.srthas 57 cues. On master (d7eb132):--bufferinput p1.ts p2.tsgives 43 cues. Cues 22 to 35 (00:01:46 to 00:02:34) are missing, and every cue after them is about 60 s early (cue 36 comes out at 00:01:35,027 instead of 00:02:34,989). The log shows exactly 16 MiB missing: 28200000 − 11422784 = 16777216 =FILEBUFFERSIZE.--bufferinput u1.ts u2.tsgives 38 cues. Nothing fromu2.tsis read:With this PR, both outputs are byte-identical to
whole.srt.Testing
Built with
mac/build.command. Each output is compared withcmpagainstwhole.srt:p1.ts p2.ts(default mode)--bufferinput p1.ts p2.ts(cut on a packet boundary)--bufferinputtwo halves cut mid-packet--bufferinput u1.ts u2.ts(second part < 16 MB)--bufferinputthree parts cut mid-packet--bufferinput -in=ts p1.ts p2.ts(no detection)--bufferinput whole.ts(single file)--bufferinput p1.ts p2.tsandu1.ts u2.tsare identical towhole.srtagain (and the default-mode bug [FIX] Drop the previous file's bytes when opening a file #2394 fixed comes back). So [FIX] Drop the previous file's bytes when opening a file #2394 is what exposed this.cargo test: 426 passed.cargo fmt --checkis clean.