Repository navigation
Conversation
The maxval, poll, and currval deprecation warnings used stacklevel=1, so they named progressbar's own source instead of the line that used the old name. A fixed stacklevel=2 would only be right for currval: _apply_deprecated_aliases is reached through ProgressBar.__init__ and any subclass __init__ chain above it, so the depth varies per caller. _caller_stacklevel() walks the frames from the warning site outward until the code is no longer inside the progressbar package and returns that depth. Five tests record the warning and assert its filename and line are the caller's, for a plain bar, a DataTransferBar, a user subclass with its own __init__, and the currval property.
WrappingIO.write iterated the shared listener set while start_capturing and stop_capturing on another thread added or removed bars, which on 3.14t and 3.15t raised "Set changed size during iteration" tens of thousands of times per run and dropped bar updates. The listener set is now guarded by a small dedicated StreamWrapper.listeners_lock that every WrappingIO shares, held only around the add, remove and the snapshot in write(), never while calling out, so it cannot take part in a lock-order cycle with a logging handler or a locking target stream. The wider StreamWrapper.lock (reentrant) guards wrap, unwrap and capture transitions. Both classes pickle and deepcopy again by dropping the locks in __getstate__ and recreating them on load. StreamWrapper has class-level default locks so a subclass that skips super().__init__() keeps working. Unwrapping delivers the dropped wrapper's buffered text to the original stream and never raises for a stream that cannot take it (logged at DEBUG). What working code can notice: WrappingIO gains an optional lock= keyword (api_surface_snapshot regenerated), its default lock is a plain threading.Lock, and text that was buffered in a wrapper at unwrap time now reaches the original stream instead of being lost.
The new-style widget formatter resolved any attribute path a template
named, so a template taken from user input could walk {variables.x.__class__}
and further into the interpreter. Rendering goes through a string.Formatter
subclass that refuses attribute segments starting with two underscores
and raises ValueError; single-underscore names and everything else render
exactly as in 4.6.0. Non-str templates with a callable format() (lazy
translation proxies) are rendered through str(); None, lists and numbers
raise TypeError as before.
ProgressBar(desc=...) is rendered literally as f'{desc}: ' and is never
a format template; a prefix assigned afterwards is a template as before.
PROGRESSBAR_MINIMUM_UPDATE_INTERVAL that is not a non-negative finite
number is ignored with one WARNING per distinct value instead of raising
at construction.
What working code can notice: templates addressing __dunder__ paths now
raise ValueError, a desc containing braces renders them literally, and a
bad interval variable logs instead of raising.
Validate every size option through size_to_bytes with a minimum and a parser error (status 2) instead of a traceback, reject a missing input, a directory input, a directory output and an output path that is also an input before the output is truncated, and close every input after its copy. An early-closing reader (`... | head`) ends the copy quietly with status 0: the BrokenPipeError is caught and, when the broken stream is the process's real stdout (fd 1), the descriptor is pointed at os.devnull so the interpreter's exit flush cannot raise again. A replacement sys.stdout that owns another descriptor is left alone. Ctrl-C exits 130. main() is split into small helpers (_parse_size_option, _collect_inputs, _check_output_path, _open_input, _copy_stream) with no behaviour change beyond the above. What working code can notice: invalid -s/-B/-L values exit 2 with a message instead of a traceback, `-o` onto an input file is refused, and `... | head` exits 0 where 4.6.0 exited 120 with "Exception ignored".
…play
Batch item labels are data, not terminal markup: every ESC-introduced
sequence and every C0/C1 control is stripped before the label becomes a
MultiBar key, so an item cannot move the cursor, retitle the terminal or
hide output from the other rows. Labels that are not str are rendered
through str() first. A finished task's bar is finished (end='') before
it is dropped so its stream listeners, capture refcounts and resize
handler return to baseline instead of leaking per task.
In the shell runner the {} and {item} placeholders are substituted in a
single regex pass, so placeholder text inside an item is never expanded
again, and shell=True with a list or callable template is a TypeError:
subprocess would run only the first element through the shell and drop
the item.
The ANSI detection called the console-mode probe on Windows for every stream, including pipes and files, where it reports on the console the process happens to own rather than on the stream being asked about. The probe now runs only when the stream is a tty, matching the other platforms.
The publish job runs only for a v* tag whose name matches __version__ and whose commit is the checked-out commit, so a manual dispatch on a branch or a mistyped tag cannot publish. The built sdist and wheel are listed and refused when they contain sync-conflict, .orig, .rej or .bak files. The workflow token is limited to contents: read. Local secret and environment files are ignored by git.
Two adversarial reviews of the previous commit found that the
transition lock was still held while logging ("wrapping excepthook")
and while flushing captured text into the original streams. A logging
handler that writes into a wrapper, or a target stream with its own
lock, could then wait on a thread that was waiting for the transition
lock: a deadlock 4.6.0 could not have, because it held no lock at all.
Moving the call-outs outside the lock naively made two overlapping
stop_capturing() calls deliver the same text twice.
Now every transition takes the buffered text under the lock
(WrappingIO._take) and writes it to the original stream after
releasing it (WrappingIO._deliver), so nothing blocks while the lock is
held and no text is delivered twice. The excepthook installation logs
after the lock is released.
The locks live in a small shared box, _SharedLock, that a StreamWrapper
and the wrappers it installs hold in common. The box keeps its identity
through pickle and deepcopy whichever object is the root, so restored
wrappers share the restored lock, subclass slots survive (default
pickling again, no __getstate__), subclasses that define __eq__ without
__hash__ construct, and a forked child gets fresh unlocked locks for
every live box through os.register_at_fork, standalone wrappers
included. Lock annotations evaluate on 3.10 to 3.12, where
threading.Lock is a factory function.
What working code can notice: StreamWrapper.lock and listeners_lock
are read-only properties (both were introduced on this branch, 4.6.0
had neither), and get_type_hints(WrappingIO.__init__) works again on
3.10 to 3.12.
ProgressBar(desc=...) rendered the literal text by inserting the plain
str into the widget list, so bar.widgets[0] was a str where 4.6.0 (and
prefix=) had a FormatLabel; a subclass that rewrites that label after
start() silently missed it. The desc-derived prefix is now escaped and
wrapped in a FormatLabel when the widgets are built, while bar.prefix
keeps the raw f'{desc}: ' value that the fast path renders verbatim.
The safe formatter also refuses the non-dunder attributes that lead
from a generator, coroutine, traceback or frame stored as a bar
variable to a frame's globals (f_globals, f_locals, f_builtins,
f_back, gi_frame, cr_frame, ag_frame, tb_frame).
The once-per-value warning for a bad PROGRESSBAR_MINIMUM_UPDATE_INTERVAL
records the value under a lock before logging, so concurrent first
constructions produce one record, remembers every distinct value, and
its lock is reset in a forked child.
Refuse an -o path that is the file stdin is open on, an unreadable input and a closed stdin before the output is opened, so the output is never truncated for an invocation that then fails. Reject a size option above sys.maxsize instead of raising OverflowError from read(). Flush the output inside the guarded block, so a broken pipe that is only noticed when a small copy's buffer reaches the pipe exits 0 like a large one.
The render thread snapshots the bars and a worker may delete one (task_finished) before the finished-bar path runs. That path now returns early for a bar that is no longer registered, undoes its timestamp if the bar vanished while it was stamped, and ignores a concurrent removal when expiring it, instead of leaking the entry or raising KeyError from the render thread.
…ames The version check accepted only `__version__ = '...'`; the annotated spelling `__version__: str = '...'` failed the guard although the package built. The stray-file pattern matched any path containing .orig, .rej or .bak; it now matches those as file extensions only.
The test relied on 30 ms sleep differences between five workers and failed twice on a loaded machine. Only one item can now complete until a gate opens, so the first yielded pair is deterministic.
test_wrapping_io_and_a_redirecting_bar_deepcopy_and_pickle deep-copied a started, redirecting bar and a StreamWrapper holding one, and dropped the copies. A copy of a started bar is itself a started bar, so its __del__ ran finish() whenever the gc got to it, usually inside a later test, where it dropped the singleton's capture count and restored sys.stdout under that test's feet. That is what made test_redirected_stderr_lines_are_flushed_above_bar fail now and then under coverage. The copies are marked finished before they go.
A flush takes the buffered text of both wrapped streams and then writes it to the two original streams. When the stdout write raised, the exception left the loop before the stderr text was written, and that text was already gone from the buffer, so a broken stdout cost the stderr lines of the same round. Every taken text is now written before the first exception is raised. Delivery stays as 4.6.0 did it: each caller writes what it took, at once, with no lock held during the writes. A single ordered delivery queue was tried first and taken back again, in both a non-blocking shape (a caller returned with its own text still queued behind another thread, so text written right after a stop landed above it, and a flush during a render thread's delivery wrote nothing) and a blocking shape (a lock-order cycle with a logging handler whose stream is the wrapper, when the original stream itself logs). Working code notices: an exception from the stdout target no longer drops the stderr text flushed in the same call.
The shared resize registry saved the previous handler when its set of bars was empty and restored it from the last uninstall(). Three ways out of a bar skipped that restore: a bar collected without finish() left the registry's handler installed for the rest of the process, a finish() off the main thread hit the ValueError from signal.signal() with the bar already discarded, after which the next bar saved the registry's own handler as the "previous" one and the real one was lost for good, and a handler installed from C (getsignal() returns None) made the restore a TypeError. One bar whose _handle_resize() raised also skipped the other bars and propagated out of the signal handler into unrelated code. Whether the shared handler is installed is now read from getsignal() instead of inferred from the set, so a handler that other code set over ours is never clobbered. The saved handler is cleared only after a restore succeeds, a restore that fails off the main thread is retried at the next registration change or the next resize signal, None restores SIG_DFL (SIGWINCH is ignored by default), every registered bar gets a weakref finalizer that restores once no live bar is left, and a failing bar is logged at WARNING on the progressbar.bar logger and skipped. What working code can notice: a resize handler that raised now logs instead of raising, and the previous handler comes back after bars that were dropped without finish() or finished from a worker thread.
WrappingIO.write() appended to the StringIO buffer without a lock while _take() read it, then truncated it. A write landing between the read and the truncate was gone: deterministic with a buffer that pauses in truncate(), and a handful of tokens per run in a six-thread stress on the free-threaded builds. 4.6.0 has the same race. Each WrappingIO now holds a reentrant lock in a shared-lock box that write() and _take() take around the StringIO write, getvalue, seek and truncate calls only. It is never held while a listener, the target stream or the buffer's own flush() is called (a review found that a replacement buffer whose flush() logs through a handler on the wrapper deadlocked when the flush ran inside the lock), so it cannot take part in a lock-order cycle with a logging handler. The box survives pickle, deepcopy and fork like the listeners lock does, and the class-level default keeps a subclass that skips super().__init__() working.
`progressbar FILE -o /dev/stdout | head -c1` exited 1. The file opened for -o is not sys.stdout, so the quiet exit for a reader that went away did not apply, and the interpreter reported the failing flush at exit. _output_is_stdout() now also accepts a file whose fstat() reports the same device and inode as sys.stdout, and _discard_broken_stdout() points both descriptor 1 and the file's own descriptor at os.devnull so neither the exit flush nor the exit stack's close raises again. Nothing changes unless descriptor 1 is involved: a replacement stdout on another descriptor, a regular output file and a FIFO given by path are left alone.
python-utils before 4.1.4 scaled a value between 0 and 1/1024 to a
negative power, and DataSize and format_unit_value() used that power as
an index into their prefix tuple, so a fraction of a byte rendered with
the largest prefix ('0.1 YiB', '0.1 PiB'). 4.1.4 returns power 0 for
those values. The floor moves from 3.8.1 to 4.1.4, the development lock
follows, and two tests pin the rendering of a value below one byte.
What working code can notice: installing progressbar2 upgrades a
python-utils older than 4.1.4. Both packages already require Python
3.10 or newer.
…ests The two tests compare a FormatLabel called directly with plain text. On develop every widget colours its output by default (624ebf2), so the direct call returns ANSI codes and the merge of #330 went red on every tox job. Compare through utils.no_color(), as the FormatLabel doctest does. (cherry picked from commit 772a2ea914e6d1dbb0c9d810c93c0452a9456620)
…330 (cherry picked from commit b81f026c0db3541a8dbd64e29a401c47e6cdfa21)
… after #330 The Windows tox job of #330 failed in pyright: the stubs declare os.register_at_fork only off Windows, so the hasattr() guard did not narrow it. A sys.platform check does, and every platform that can fork has the hook. (cherry picked from commit 4dde4e55a2c29f8926df23ecfa9c76af5b52cf39)
A new-style template went through string.Formatter.vformat on every
render so that get_field() could refuse dunder and frame attributes.
The parse, the field split and the attribute check depend only on the
template text, so they ran again for every update with the same
result. _safe_format() now validates each distinct text once, keeps it
in a bounded dict, and renders a plain str with str.format_map(). A
specialised template keeps its own format() method and its text is
read again on every call, so a proxy that switches language is
checked for the text it is about to render.
_safe_format on '{value} of {max_value} {percentage:3.0f}%':
1.52 us before, 0.26 us after (str.format itself: 0.28 us).
FormatLabel with the default colours: 5.75 us before, 4.42 us after,
against 4.38 us on develop without this check.
…ate tests develop colours widgets by default, so the rendered text carries the ANSI prefix. The Markup type assertion stays on the raw result.
Two per-call costs crept into the hot paths with #330 and the fast label fix. Every captured write() ran _native_buffer(): a type test, vars() and a five-name generator, under a re-entrant lock. Every bar redraw called streams.flush(), which took the transition lock even with nothing wrapped, and the fast line measured its width through len_color() for text that never carries an escape code. write() now inlines the native test as one type check and one dict truth test, and the buffer lock is a plain Lock: it is only ever held around StringIO calls, never while calling out, so re-entrancy was never needed. flush() returns before the lock when neither stream is wrapped. The fast formatter uses len() when the default measurer applies and the text has no escape code, and still calls a custom measurer. Measured on CPython 3.13 against v4.6.0 with benchmarks/bench.py sizes: captured write 0.373 to 0.133 us (4.6.0: 0.041, unlocked and losing writes during a drain), forced fast render 5.89 to 4.96 us (4.6.0: 4.92), function calls per fast update back to the 4.6.0 count. Two tests pin the flush shortcut and the lock rule for a buffer with an unrelated instance attribute.
str.format_map refuses a positional field with ValueError, where 4.6.0's str.format(**data) raised IndexError. The validation step now raises that IndexError itself, with the 4.6.0 message, so callers that caught it keep working.
Colouring widgets by default made a forced redraw 1.5 times slower than
4.6.0 and a bare Timer() call 4.5 times slower. Every coloured widget
resolved its gradient through python_utils remap twice (type promotion,
range checks and typing.cast on each call), built three namedtuples for
the interpolated Color, created an SGRColor and rendered its escapes
through two str.format calls.
The gradient now does the remap arithmetic inline for a float value,
which is bit for bit what remap computed for a 0..1 input range, and a
plain ColorGradient with the default blend renders its SGR parameter
straight from the channel arithmetic. Color.ansi, RGB.to_ansi_16 and
RGB.to_ansi_256 share module functions with that shortcut, so there is
one rule for the encoding. apply_colors writes the escape sequence
directly instead of going through Color.fg and Color.bg, a Color
subclass keeps its own fg and bg, and a gradient with a custom blend, a
subclass or a non float value takes the old path. The FormatLabel alias
loop uses try/except instead of a contextlib.suppress per call,
len_color skips the no_color frame, and _format_widgets tests for a
plain str before the abstract isinstance checks and only converts a
widget result that is not already a str.
tests/test_color_fast_path.py compares apply_colors and get_color with
the plain composition of the public pieces for every ColorSupport level
over a dense percentage grid, fixed and unregistered colours, two to
eight stop gradients, snap and custom blends and a HueGradient.
Quick benchmark on this machine, CPython 3.13, real pty:
forced redraw, default widgets 37.6 us -> 21.0 us (4.6.0: 24.7 us)
Timer() call 4.06 us -> 1.55 us (4.6.0: 0.90 us)
FormatLabel() call 6.45 us -> 3.1 us (4.6.0: 1.18 us)
The remaining FormatLabel gap is the string.Formatter template path,
and the remaining Timer gap is the colour itself.
4.6.0's update_capturing() flushed through the public method once the refcount reached zero, so a StreamWrapper subclass that overrides flush() saw every capture end. #330 moved the drain into the locked _apply_capturing() path and the hook went silent. The drain stays where it is, and update_capturing() calls flush() afterwards, which restores the 4.6.0 call sequence and costs one early return when no stream is wrapped.
The bump from 3.8.1 to 4.1.4 is the first thing an install with a python-utils<4 pin notices, so it belongs in the Unreleased section.
A coloured widget call still cost 0.75 us more than its uncoloured
twin: the gradient arithmetic, the SGR parameter and two f-strings ran
on every call although the default widgets of one bar all resolve the
same gradient at the same percentage on every redraw, and the escapes
of a fixed Color never change for a given colour support.
apply_colors now remembers the rendered (prefix, suffix) pair per plain
ColorGradient by (start code, value) in a module table tagged with the
colour support it was rendered for, and per plain Color by (colour,
start code). Assigning a gradient's colors or interpolate drops its
entry, a support change invalidates on the next call, the per gradient
table stops at 1024 values and the gradient table at 64 gradients, and
a custom blend is never memoised. The hot path for a gradient fg is one
table lookup inside apply_colors with no helper call in between.
The wrap is a concatenation, prefix + text + suffix, as Color.fg and
Color.bg always produced it: a str subclass such as markupsafe.Markup
dispatches to its own __radd__ and comes back as that subclass, where
the f-string of the previous commit returned a plain str.
Quick benchmark on this machine, CPython 3.13, real pty, merged tree:
forced redraw, default widgets 20.9 us -> 18.7 us (4.6.0: 24.7 us)
Timer() call 1.47 us -> 1.04 us (4.6.0: 0.90 us)
FormatLabel() call 1.75 us -> 1.25 us (4.6.0: 1.18 us)
An uncoloured Timer() costs 0.77 to 0.88 us on this tree, so the
remaining gap to 4.6.0 is the colour step itself: one dict lookup and
one concatenation.
…bels The #331 label width fix measured prefix and suffix on every redraw, also for the default bar that has neither, which cost 0.1 us of the 1.6 us formatter. Without labels the width is len(left) + len(right) as in 4.6.0. With labels the str() conversion, the escape check and the custom measurer run as before, and two tests pin a coloured label and a str subclass label.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
CodeQL flagged five test asserts whose expression pops or writes, which python -O would skip, and the module global that tracked the colour support of the fixed-colour escape table. The asserts read a variable now, and the support lives in a one-cell list next to the table.
The CI terminal has no colour support, so the bar stripped the escapes from the label before the assertion looked for them.
| self._finished_at.clear() | ||
| self._labeled.clear() | ||
| # Release the last references after leaving the membership guard. | ||
| del removed |
| 'operation', ['replace', 'pop', 'popitem', 'clear', 'finish'] | ||
| ) | ||
| def test_membership_tracking_never_calls_bar_hash(operation: str) -> None: | ||
| class HashBar(progressbar.ProgressBar): |
| import pytest | ||
| from test_stream import reset_wrapped_streams | ||
|
|
||
| import progressbar |
| self.capturing = max(self.capturing - 1, 0) | ||
| try: | ||
| pending = self._apply_capturing() | ||
| except BaseException as exception: # noqa: BLE001 |
| # Never flushes here: the count just went above zero | ||
| try: | ||
| self._apply_capturing() | ||
| except BaseException as exception: # noqa: BLE001 |
| return None | ||
| try: | ||
| return self._take() | ||
| except BaseException as exception: # noqa: BLE001 |
| @typing.overload | ||
| def setdefault( | ||
| self, key: str, default: None = None | ||
| ) -> bar.ProgressBar | None: ... |
| @typing.overload | ||
| def setdefault( | ||
| self, key: str, default: bar.ProgressBar | ||
| ) -> bar.ProgressBar: ... |
The two tests check that an instance whose constructor never reached the parent's still wraps and writes. Creating it with __new__ and setting the attributes in the test keeps that scenario and no longer reads as a subclass that forgot super().__init__().
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Develop plus #330 rendered a forced redraw 1.5 times slower than 4.6.0
and a single widget call four times slower. This branch merges #330 and
brings the README benchmarks back to 4.6.0 speed or better, without
changing any public name, signature, default, return type or exception
type, and without undoing the #330 fixes.
Measured on one machine (Apple Silicon, CPython 3.13, real pty) with
benchmarks/bench.pysizes, 4.6.0 and this branch interleaved, minimumof several runs:
[fast]FastProgressBar[fast]FormatLabelcall, coloured since developTimercall, coloured since developprint()writeWhere the time went and what changed:
through
remap,interpolateand aColorround trip on everyredraw. The gradient arithmetic is inlined for the default blend, the
rendered escapes are remembered per gradient and per fixed colour
(bounded tables, invalidated when the colour support,
colorsorinterpolatechanges), andapply_colorsconcatenates so astrsubclass such as
Markupkeeps its type. Output is byte identical:tests/test_color_fast_path.pycompares against referenceimplementations for every
ColorSupportlevel on a dense grid.string.Formattersubclass on every call. A template's safety depends on its text only,
so each distinct text is validated once and then rendered with
str.format_map. Dunder and frame attributes are still refused, and apositional field still raises the
IndexError4.6.0 raised.generator on every captured
print(). The check is one type test andone dict test, the buffer lock is a plain
Lock, andstreams.flush()returns before the lock when nothing is wrapped. A captured write still
costs three times 4.6.0's 0.04 us: 4.6.0 took no lock and lost text
written during a drain, the lock is what fixes that.
redraw, also for the default bar that has neither. Without labels the
width is
len(left) + len(right)as in 4.6.0.update_capturing()callsflush()again once the last capture stops,so a
StreamWrappersubclass overridingflush()sees it as in 4.6.0.the default colours before comparing, the fork hook is guarded by
platform so pyright passes on Windows, the CLI docs state the exit
statuses, the tqdm page says
descis literal text, and the changelogmentions the python-utils floor.
Checks on this branch: pytest with coverage on 3.10 and 3.14t (100%),
ruff 0.15.20 check and format, codespell, pyright on Linux and Windows
platform, the 4.6.0 test files (290 pass, the 3
test_widgetscolourtests fail on develop already), and the #330 compatibility probes
(no new difference against develop plus #330 apart from the restored
IndexError message and the restored
flush()hook).Known and unchanged: widgets called directly return ANSI codes by
default since develop's colour change, so a single
Timer()call costsabout 0.1 us more than the uncoloured 4.6.0 call. Merging this branch
also lands #330.