diff --git a/.github/ISSUE_TEMPLATE/documentation.yml b/.github/ISSUE_TEMPLATE/documentation.yml index d720cf9c4de91e2..d21e2a88604087f 100644 --- a/.github/ISSUE_TEMPLATE/documentation.yml +++ b/.github/ISSUE_TEMPLATE/documentation.yml @@ -7,10 +7,16 @@ body: value: | > [!NOTE] > Trivial changes (for example typos) don’t require an issue before opening a PR. + - type: input + id: url + attributes: + label: "Documentation URL" + description: "A link to the relevant documentation page, if applicable." + placeholder: "https://docs.python.org/3/..." - type: textarea id: description attributes: label: "Documentation" - description: "A clear and concise description of the issue. Include a link to the page." + description: "A clear and concise description of the issue." validations: required: true diff --git a/Doc/builtins/functions.rst b/Doc/builtins/functions.rst index 717efe50c130744..63a359e8667060a 100644 --- a/Doc/builtins/functions.rst +++ b/Doc/builtins/functions.rst @@ -1181,6 +1181,11 @@ are always available. They are listed here in alphabetical order. a :exc:`TypeError` exception is raised. :exc:`TypeError` may not be raised for an invalid type if an earlier check succeeds. + If *object* has a ``__class__`` attribute that differs from + :func:`type(object) ` (for example a proxy, or a :mod:`unittest.mock` + object created with a *spec*), ``True`` is also returned when that class + is a subclass of *classinfo*. + .. versionchanged:: 3.10 *classinfo* can be a :ref:`types-union`. @@ -1742,6 +1747,10 @@ are always available. They are listed here in alphabetical order. .. versionchanged:: 3.3 Added the *flush* keyword argument. + .. versionchanged:: 3.16 + The output is now written in a single ``write()`` call whenever possible, + instead of writing each argument, separator, and terminator separately. + .. class:: property(fget=None, fset=None, fdel=None, doc=None) diff --git a/Doc/c-api/dict.rst b/Doc/c-api/dict.rst index e986c8b6456b749..cdb3d8fba5af80c 100644 --- a/Doc/c-api/dict.rst +++ b/Doc/c-api/dict.rst @@ -526,6 +526,8 @@ Dictionary objects as an argument, prefer :c:func:`PyFrozenDict_New` to leave the source intact. + Does not work for dicts with embedded values. + .. impl-detail:: Works with O(1) complexity. diff --git a/Doc/data/refcounts.dat b/Doc/data/refcounts.dat index c19a8a3083a4302..b488237b31b642e 100644 --- a/Doc/data/refcounts.dat +++ b/Doc/data/refcounts.dat @@ -2417,6 +2417,9 @@ PyType_GenericNew:PyObject*:type:0: PyType_GenericNew:PyObject*:args:0: PyType_GenericNew:PyObject*:kwds:0: +PyType_GetDict:PyObject*::+1: +PyType_GetDict:PyTypeObject*:type:0: + PyType_GetFlags:unsigned long::: PyType_GetFlags:PyTypeObject*:type:0: diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst index c105620fc3c04aa..d8a177020b921bf 100644 --- a/Doc/howto/logging-cookbook.rst +++ b/Doc/howto/logging-cookbook.rst @@ -330,11 +330,10 @@ Suppose you configure logging with the following JSON: } } -This configuration does *almost* what we want, except that ``sys.stdout`` would show messages -of severity ``ERROR`` and only events of this severity and higher will be tracked -as well as ``INFO`` and ``WARNING`` messages. To prevent this, we can set up a filter which -excludes those messages and add it to the relevant handler. This can be configured by -adding a ``filters`` section parallel to ``formatters`` and ``handlers``: +This configuration does *almost* what we want, except that ``sys.stdout`` would +also show messages of severity ``ERROR`` and above. To exclude those messages, +we can add a filter to the stdout handler. This can be configured by adding a +``filters`` section parallel to ``formatters`` and ``handlers``: .. code-block:: json @@ -562,9 +561,9 @@ One solution is to use a two-part approach. For the first part, attach only a :class:`QueueHandler` to those loggers which are accessed from performance-critical threads. They simply write to their queue, which can be sized to a large enough capacity or initialized with no upper bound to their -size. The write to the queue will typically be accepted quickly, though you -will probably need to catch the :exc:`queue.Full` exception as a precaution -in your code. If you are a library developer who has performance-critical +size. The write to the queue will typically be accepted quickly. If a bounded +queue is full, :class:`QueueHandler` calls :meth:`~logging.Handler.handleError` +with the record. If you are a library developer who has performance-critical threads in their code, be sure to document this (together with a suggestion to attach only ``QueueHandlers`` to your loggers) for the benefit of other developers who will use your code. @@ -1608,8 +1607,8 @@ application: The most current file is always :file:`logging_rotatingfile_example.out`, and each time it reaches the size limit it is renamed with the suffix -``.1``. Each of the existing backup files is renamed to increment the suffix -(``.1`` becomes ``.2``, etc.) and the ``.6`` file is erased. +``.1``. Before this happens, the oldest backup (``.5``), if present, is +deleted and the remaining backups are renamed (``.4`` becomes ``.5``, etc.). Obviously this example sets the log length much too small as an extreme example. You would want to set *maxBytes* to an appropriate value. @@ -1801,7 +1800,7 @@ Before Python 3.2, there were only two places where this creation was done: * :func:`makeLogRecord`, which is called with a dictionary containing attributes to be added to the LogRecord. This is typically invoked when a suitable dictionary has been received over the network (e.g. in pickle form - via a :class:`~handlers.SocketHandler`, or in JSON form via an + via a :class:`~handlers.SocketHandler`, or as URL-encoded data via an :class:`~handlers.HTTPHandler`). This has usually meant that if you need to do anything special with a @@ -1827,11 +1826,7 @@ do simply by adding new packages or modules and doing :: logger = logging.getLogger(__name__) -at module level). It's probably one too many things to think about. Developers -could also add the filter to a :class:`~logging.NullHandler` attached to their -top-level logger, but this would not be invoked if an application developer -attached a handler to a lower-level library logger --- so output from that -handler would not reflect the intentions of the library developer. +at module level). It's probably one too many things to think about. In Python 3.2 and later, :class:`~logging.LogRecord` creation is done through a factory, which you can specify. The factory is just a callable you can set with @@ -1913,15 +1908,15 @@ You can also subclass :class:`QueueListener` to get messages from other kinds of queues, for example a ZeroMQ 'subscribe' socket. Here's an example:: class ZeroMQSocketListener(QueueListener): - def __init__(self, uri, /, *handlers, **kwargs): - self.ctx = kwargs.get('ctx') or zmq.Context() + def __init__(self, uri, /, *handlers, ctx=None, **kwargs): + self.ctx = ctx or zmq.Context() socket = zmq.Socket(self.ctx, zmq.SUB) socket.setsockopt_string(zmq.SUBSCRIBE, '') # subscribe to everything socket.connect(uri) super().__init__(socket, *handlers, **kwargs) - def dequeue(self): - msg = self.queue.recv_json() + def dequeue(self, block): + msg = self.queue.recv_json(flags=0 if block else zmq.NOBLOCK) return logging.makeLogRecord(msg) .. _pynng-handlers: @@ -2245,7 +2240,7 @@ works:: import logging.config import logging.handlers from multiprocessing import Process, Queue, Event, current_process - import os + from multiprocessing import get_start_method import random import time @@ -2281,13 +2276,8 @@ works:: logging.config.dictConfig(config) listener = logging.handlers.QueueListener(q, MyHandler()) listener.start() - if os.name == 'posix': - # On POSIX, the setup logger will have been configured in the - # parent process, but should have been disabled following the - # dictConfig call. - # On Windows, since fork isn't used, the setup logger won't - # exist in the child, so it would be created and the message - # would appear - hence the "if posix" clause. + if get_start_method() == 'fork': + # dictConfig disables the setup logger inherited from the parent. logger = logging.getLogger('setup') logger.critical('Should not appear, because of disabled logger ...') stop_event.wait() @@ -2312,13 +2302,8 @@ works:: logging.CRITICAL] loggers = ['foo', 'foo.bar', 'foo.bar.baz', 'spam', 'spam.ham', 'spam.ham.eggs'] - if os.name == 'posix': - # On POSIX, the setup logger will have been configured in the - # parent process, but should have been disabled following the - # dictConfig call. - # On Windows, since fork isn't used, the setup logger won't - # exist in the child, so it would be created and the message - # would appear - hence the "if posix" clause. + if get_start_method() == 'fork': + # dictConfig disables the setup logger inherited from the parent. logger = logging.getLogger('setup') logger.critical('Should not appear, because of disabled logger ...') for i in range(100): @@ -2345,9 +2330,7 @@ works:: } # The worker process configuration is just a QueueHandler attached to the # root logger, which allows all messages to be sent to the queue. - # We disable existing loggers to disable the "setup" logger used in the - # parent process. This is needed on POSIX because the logger will - # be there in the child following a fork(). + # Disable the parent's "setup" logger if it was inherited via fork. config_worker = { 'version': 1, 'disable_existing_loggers': True, @@ -2365,9 +2348,7 @@ works:: # The listener process configuration shows that the full flexibility of # logging configuration is available to dispatch events to handlers however # you want. - # We disable existing loggers to disable the "setup" logger used in the - # parent process. This is needed on POSIX because the logger will - # be there in the child following a fork(). + # Disable the parent's "setup" logger if it was inherited via fork. config_listener = { 'version': 1, 'disable_existing_loggers': True, @@ -4077,11 +4058,9 @@ How to uniformly handle newlines in logging output Usually, messages that are logged (say to console or file) consist of a single line of text. However, sometimes there is a need to handle messages with -multiple lines - whether because a logging format string contains newlines, or -logged data contains newlines. If you want to handle such messages uniformly, so -that each line in the logged message appears uniformly formatted as if it was -logged separately, you can do this using a handler mixin, as in the following -snippet: +multiple lines because the message or its arguments contain newlines. +To format each line as if it were logged separately, you can use a handler +mixin, as in the following snippet: .. code-block:: python diff --git a/Doc/howto/logging.rst b/Doc/howto/logging.rst index c8ce0df9e937f8c..584f0901b8a69fc 100644 --- a/Doc/howto/logging.rst +++ b/Doc/howto/logging.rst @@ -146,13 +146,6 @@ interpreter, and don't just continue from the session described above:: logger.warning('And this, too') logger.error('And non-ASCII stuff, too, like Øresund and Malmö') -.. versionchanged:: 3.9 - The *encoding* argument was added. In earlier Python versions, or if not - specified, the encoding used is the default value used by :func:`open`. While - not shown in the above example, an *errors* argument can also now be passed, - which determines how encoding errors are handled. For available values and - the default, see the documentation for :func:`open`. - And now if we open the file and look at what we have, we should find the log messages: @@ -465,10 +458,9 @@ With the logger object configured, the following methods create log messages: a message and a level that corresponds to their respective method names. The message is actually a format string, which may contain the standard string substitution syntax of ``%s``, ``%d``, ``%f``, and so on. The - rest of their arguments is a list of objects that correspond with the - substitution fields in the message. With regard to ``**kwargs``, the - logging methods care only about a keyword of ``exc_info`` and use it to - determine whether to log exception information. + remaining positional arguments correspond to the substitution fields in the + message. These methods also accept the keyword arguments *exc_info*, *extra*, + *stack_info*, and *stacklevel*, described in :meth:`Logger.debug`. * :meth:`Logger.exception` creates a log message similar to :meth:`Logger.error`. The difference is that :meth:`Logger.exception` dumps a @@ -1006,9 +998,9 @@ Logged messages are formatted for presentation through instances of the use with the % operator and a dictionary. For formatting multiple messages in a batch, instances of -:class:`BufferingFormatter` can be used. In addition to the format -string (which is applied to each message in the batch), there is provision for -header and trailer format strings. +:class:`BufferingFormatter` can be used. A line formatter formats each record +in the batch. To add a header or footer, override +:meth:`~BufferingFormatter.formatHeader` or :meth:`~BufferingFormatter.formatFooter`. When filtering based on logger level and/or handler level is not enough, instances of :class:`Filter` can be added to both :class:`Logger` and @@ -1059,10 +1051,7 @@ In the preceding sections and examples, it has been assumed that the message passed when logging the event is a string. However, this is not the only possibility. You can pass an arbitrary object as a message, and its :meth:`~object.__str__` method will be called when the logging system needs to -convert it to a string representation. In fact, if you want to, you can avoid -computing a string representation altogether - for example, the -:class:`~handlers.SocketHandler` emits an event by pickling it and sending it -over the wire. +convert it to a string representation. Optimization diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst index 16c2dab5b8adba1..1b5f0c92faf02c7 100644 --- a/Doc/library/dis.rst +++ b/Doc/library/dis.rst @@ -817,7 +817,8 @@ not have to be) the original ``STACK[-2]``. .. opcode:: GET_AWAITABLE (where) - Implements ``STACK[-1] = get_awaitable(STACK[-1])``, where ``get_awaitable(o)`` + Replaces ``STACK[-1]`` with ``get_awaitable(STACK[-1])``, ``NULL`` and + ``None``, ready for :opcode:`SEND`. ``get_awaitable(o)`` returns ``o`` if ``o`` is a coroutine object or a generator object with the :data:`~inspect.CO_ITERABLE_COROUTINE` flag, or resolves ``o.__await__``. @@ -833,6 +834,10 @@ not have to be) the original ``STACK[-2]``. .. versionchanged:: 3.11 Previously, this instruction did not have an oparg. + .. versionchanged:: next + Also pushes ``NULL`` and ``None``, which were previously pushed by + separate instructions. + .. opcode:: GET_AITER @@ -846,11 +851,16 @@ not have to be) the original ``STACK[-2]``. .. opcode:: GET_ANEXT - Implement ``STACK.append(get_awaitable(STACK[-1].__anext__()))`` to the stack. + Implement ``STACK.append(get_awaitable(STACK[-1].__anext__()))`` to the stack, + then push ``NULL`` and ``None``, ready for :opcode:`SEND`. See ``GET_AWAITABLE`` for details about ``get_awaitable``. .. versionadded:: 3.5 + .. versionchanged:: next + Also pushes ``NULL`` and ``None``, which were previously pushed by + separate instructions. + .. opcode:: END_ASYNC_FOR diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 7f4b698eac38cd8..9cf4a4efe5c0ee9 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -6466,9 +6466,12 @@ Random numbers random bytes in non-blocking mode (using the :data:`GRND_NONBLOCK` flag) or to poll until the system urandom entropy pool is initialized. - On a Unix-like system, random bytes are read from the ``/dev/urandom`` - device. If the ``/dev/urandom`` device is not available or not readable, the - :exc:`NotImplementedError` exception is raised. + On other Unix-like systems, the C ``getrandom()`` function is used if it is + available, otherwise the C ``getentropy()`` function. + + If none of these functions can be used, random bytes are read from the + ``/dev/urandom`` device. If the ``/dev/urandom`` device is not available or + not readable, the :exc:`NotImplementedError` exception is raised. On Windows, it will use ``BCryptGenRandom()``. diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst index 868ef0667597435..80a1a05eb6e5e32 100644 --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -1965,7 +1965,7 @@ to speed up repeated connections from the same clients. Set the allowed ciphers for sockets created with this context when connecting using TLS 1.2 and earlier. The *ciphers* argument should be a string in the `OpenSSL cipher list format - `_. + `_. To set allowed TLS 1.3 ciphers, use :meth:`SSLContext.set_ciphersuites`. If no cipher can be selected (because compile-time options or other diff --git a/Doc/library/this.rst b/Doc/library/this.rst new file mode 100644 index 000000000000000..2893d5584addc4b --- /dev/null +++ b/Doc/library/this.rst @@ -0,0 +1,42 @@ +:orphan: + +:mod:`!this` --- The Zen of Python +================================== + +.. module:: this + :synopsis: The Zen of Python + +.. + This easter egg module is deliberately missing from the table of contents. + +:pep:`20` -- The Zen of Python, by Tim Peters (RIP). + + | Beautiful is better than ugly. + | Explicit is better than implicit. + | Simple is better than complex. + | Complex is better than complicated. + | Flat is better than nested. + | Sparse is better than dense. + | Readability counts. + | Special cases aren't special enough to break the rules. + | Although practicality beats purity. + | Errors should never pass silently. + | Unless explicitly silenced. + | In the face of ambiguity, refuse the temptation to guess. + | There should be one-\- and preferably only one -\-obvious way to do it. + | Although that way may not be obvious at first unless you're Dutch. + | Now is better than never. + | Although never is often better than *right* now. + | If the implementation is hard to explain, it's a bad idea. + | If the implementation is easy to explain, it may be a good idea. + | Namespaces are one honking great idea -\- let's do more of those! + +`In Memoriam: Tim Peters 1952-2026 `__. + + +Usage +----- + +.. code-block:: shell-session + + $ python3 -m this diff --git a/Doc/library/traceback.rst b/Doc/library/traceback.rst index 9e1be62c568ea14..8baf4b7d34905af 100644 --- a/Doc/library/traceback.rst +++ b/Doc/library/traceback.rst @@ -333,8 +333,8 @@ the module-level functions described above. The :attr:`~BaseException.__notes__` value from the original exception, or ``None`` - if the exception does not have any notes. If it is not ``None`` - is it formatted in the traceback after the exception string. + if the exception does not have any notes. If it is not ``None``, + it is formatted in the traceback after the exception string. .. versionadded:: 3.11 diff --git a/Doc/library/types.rst b/Doc/library/types.rst index 9e928e6dd3637b0..7a6e7aa3fae6a57 100644 --- a/Doc/library/types.rst +++ b/Doc/library/types.rst @@ -515,14 +515,14 @@ Additional Utility Classes and Functions .. versionchanged:: 3.13 Added support for an optional positional argument. -.. function:: DynamicClassAttribute(fget=None, fset=None, fdel=None, doc=None) +.. class:: DynamicClassAttribute(fget=None, fset=None, fdel=None, doc=None) - Route attribute access on a class to __getattr__. + Route attribute access on a class to :meth:`~object.__getattr__`. This is a descriptor, used to define attributes that act differently when accessed through an instance and through a class. Instance access remains normal, but access to an attribute through a class will be routed to the - class's __getattr__ method; this is done by raising AttributeError. + class's ``__getattr__`` method; this is done by raising :exc:`AttributeError`. This allows one to have properties active on an instance, and have virtual attributes on the class with the same name (see :class:`enum.Enum` for an example). diff --git a/Doc/library/unittest.mock-examples.rst b/Doc/library/unittest.mock-examples.rst index da57f05edcf78fa..7b736b5aecb6412 100644 --- a/Doc/library/unittest.mock-examples.rst +++ b/Doc/library/unittest.mock-examples.rst @@ -642,7 +642,7 @@ Mocking a generator method ~~~~~~~~~~~~~~~~~~~~~~~~~~ A Python generator is a function or method that uses the :keyword:`yield` statement -to return a series of values when iterated over [#]_. +to return a series of values when iterated over. A generator method / function is called to return the generator object. It is the generator object that is then iterated over. The protocol method for @@ -671,12 +671,6 @@ To configure the values returned from the iteration (implicit in the call to >>> list(mock_foo.iter()) [1, 2, 3] -.. [#] There are also generator expressions and more `advanced uses - `_ of generators, but we aren't - concerned about them here. A very good introduction to generators and how - powerful they are is: `Generator Tricks for Systems Programmers - `_. - Applying the same patch to every test method ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/Doc/using/ios.rst b/Doc/using/ios.rst index d7cf699d5bc3bfb..4dbb6e77459c480 100644 --- a/Doc/using/ios.rst +++ b/Doc/using/ios.rst @@ -226,7 +226,7 @@ To add Python to an iOS Xcode project: .. code-block:: bash set -e - source $PROJECT_DIR/Python.xcframework/build/build_utils.sh + source "$PROJECT_DIR/Python.xcframework/build/utils.sh" install_python Python.xcframework app If you have placed your XCframework somewhere other than the root of your diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst index 2b824fd70e7d34c..a53e3040ab3bdcc 100644 --- a/Doc/using/windows.rst +++ b/Doc/using/windows.rst @@ -169,7 +169,8 @@ It should be separated from the tag by a slash (either ``/`` or ``\``), and may be shortened to any prefix of its full value. Specifying the company is optional when it is ``PythonCore``, and specifying the tag is optional (but not the slash) when you want the latest release from a -specific company. +specific company. For more details about how tags are matched, see +:ref:`pymanager-tag-matching`. .. code:: @@ -998,6 +999,145 @@ Credentials may be embedded in either setting, however, when ``HTTPS_PROXY`` has credentials embedded they will be used in preference to other credentials. Only one set of credentials will be used for both types of proxy. +.. _pymanager-tag-matching: + +Tags and tag matching +--------------------- + +.. versionchanged:: 26.5 + Previously the behavior was very similar to what is described here, but had + many more inconsistencies. We have streamlined it and ensured the behavior is + well-defined, which may result in some small differences from previous + versions. + +Tags identify Python runtimes that may be installed, modified, or launched by +the Python install manager. They are based on :pep:`514`, with matching and +ordering rules specific to the install manager. These rules do not change +PEP 514's specification. + +A tag has three components: Company, Tag, and Platform. Each part is +case-insensitive, but the exact comparison semantics vary. + +.. code-block:: text + + [/][-] + +The Company is optional and defaults to ``PythonCore``. It may contain slashes, +while the rest may not, and so the last forward slash or backslash separates it +from the Tag. When omitting Company, also omit any slash. In general, Company is +matched as a prefix. + +The optional Platform suffix is one of the known platform suffixes, currently +``-32``, ``-64``, or ``-arm64``. Broadly speaking, the Platform suffix behaves +as part of the Tag, but it may be implied from the current configuration or +system in a way that is different from the rest of the tag. It also does not +parse or compare as a number. + +Runtimes are defined as having a primary tag, and offer several more tags for +installation (``install-for``) and launching (``run-for``). Whether the install +tags or the launch tags are considered depends on context, but both sets are +used to filter to matching runtimes, from which the best option is selected +using more complex logic. If a runtime doesn't have specific tags, its primary +tag may be used for installation and uninstallation (but not launch - it will be +unlaunchable without any ``run-for`` tags). + +You can observe this behaviour using ``py list`` with and without the ``--one`` +option (to show the "best" or all candidates) and with and without the +``--online``/``--source`` options (to show install matches or launch matches). + +Tags are not version numbers, though numeric sequences are treated as version +numbers. Hyphens divide the Tag into segments, where each segment is either text +or a version number with a text suffix. The version number part includes those +ending in ``a``, ``b``, or ``rc`` followed by a serial number, but any other +text is considered a separate suffix. Without a serial number, an ``a``, ``b`` +or ``rc`` suffix is also treated as text. + +Broadly, numbers are compared as numerical versions, so ``3.10`` is greater than +``3.1``, ``3.14`` is greater than ``3.14a1``, and ``3`` is equal to both ``3.0`` +and ``3.0.0``. Text is compared case-insensitively, and must match, so ``2024b`` +is not equivalent to ``2024.0`` because ``'b' != ''``. + +.. note:: + + Do not interpret the depth of detail of the following few paragraphs about + constraints as representing their importance. Constraints are a necessary + evil, and we would prefer nobody rely on them instead of specific tags, which + are much simpler to reason about for publishers and end users. Unfortunately, + we are obliged to explain how our implementation handles constraints, and + this is the place for that explanation. + +Constraints can select multiple tags. A matching tag will start with the same +segments as the constraining tag, where text segments are equal and numeric +segments are above, below, or equal, according to the constraint. The first +non-equal segment determines the result: non-equal text results in failure, +while a non-equal version either succeeds or fails based on the constraint. +Remaining segments in the matching tag are ignored. + +When applying a constraint check to a version segment that was shorter than the +constraint (e.g. ``3`` has one field and so is shorter than ``==3.14``), it is +interpreted as representing any possible version. This will result in many +failed constraint checks that will usually be overridden by a more specific one. +For example ``3`` could represent ``3.99`` and so it is not equal to ``3.14``, +but if the install is offering tags ``['3', '3.14']`` then the later tag will +result in the constraint matching. + +Here are some example cases showing the different applications of constraints: + +* ``>=3.13`` matches ``3.13.4`` and ``3.14`` +* ``>3.13`` matches ``3.14``, but not ``3.13.4`` +* ``>=3.13t`` matches ``3.14t``, while ``>=3.13`` does not +* ``>=3.13`` matches ``3.14-abc`` but not ``3.14abc`` +* ``>3.13-abc-123`` matches ``3.14`` (number is higher) and ``3.13-abc-124`` + (number ``124`` is higher) but not ``3.13-def-123`` (text mismatch), + ``3.14def`` (text ``def`` is not in the constraint), + ``3.13-abc-123`` (number is not higher) or + ``3.13-abc`` (not enough segments) +* ``>=3.0`` matches ``3``, but ``>3.0`` does not +* ``<4`` matches ``3``, but ``<3.12`` and ``<=3.12`` do not +* ``==3.0`` does not match ``3``, while ``==3.10`` still matches ``3.10.1`` +* ``!=3.13`` rejects ``3`` (because it could be ``3.13``), but ``!=3.13t`` + accepts ``3`` (different suffix) + +The difference between hyphen-separated segments and adjacent text and numbers +is subtle, but should not be relevant to most users. Publishers will choose and +advertise the tags for their packages, and so should take care to choose between +adjacent text or separated text. For example, the core python\.org releases use +``3.14t`` for free-threaded builds, which should not be confused with original +builds and so ``3.14t`` should not match ``3.14``. However, ``3.14`` should be +allowed to match with ``3.14-64`` (recalling that known platform tags are +treated at text), and so the hyphenated convention is chosen instead. End users +get intuitive behavior because the publisher considered the impact of their +choices. + +Note that constraints are not complementary, that is, the ``>=`` constraint does +not necessarily include every tag rejected by the ``<`` constraint. +For example, ``3.14t`` satisfies neither ``>=3.14`` nor ``<3.14``. + +Publishers should provide tags for each way they want users to select a +runtime, commonly from most specific to least specific. For example, a +free-threaded runtime might offer ``3.14.4t``, ``3.14t``, and ``3t``. Include +a generic tag such as ``3.14`` only if that runtime should be eligible for +generic requests. Exact matches of user requested tags are preferred over +partial matches, so include the tags you expect your users to use. + +The primary tag should be orderable against other runtimes from the same +publisher, but should not change with in-place updates. Also be aware that +operations reading an online feed assume that the feed is already ordered, and +will select the first runtime that matches requested tags. + +After filtering, runtimes are ordered by their primary tag and then the best is +selected. "Best" is very context sensitive, and may be influenced by user +configuration (such as the default tag), active virtual environments, prerelease +marks on a runtime's sortable version, and whether a windowless interpreter is +desired (such as launching through ``pyw.exe`` instead of ``py.exe``). + +Most notably, a request containing the text field ``dev``, such as ``3-dev``, +causes prereleases to be considered on equal footing with stable releases. +Typically stable releases are preferred unless filtering excludes all viable +options. However, this behavior only takes place after filtering, and so a +runtime that does not offer a tag containing ``dev`` will not be selected as a +candidate. + .. _pymanager-troubleshoot: Troubleshooting diff --git a/Doc/whatsnew/3.16.rst b/Doc/whatsnew/3.16.rst index 33328b2ee005219..e54556dcc46a6a3 100644 --- a/Doc/whatsnew/3.16.rst +++ b/Doc/whatsnew/3.16.rst @@ -109,6 +109,16 @@ Improved error messages Other language changes ====================== +* :func:`print` now collects its output and writes it with a single ``write()`` + call whenever possible. This reduces interleaving between concurrent calls + and the overhead of streams with expensive ``write()`` methods, such as + IDLE's output stream. If combining the output raises :exc:`MemoryError`, + :func:`print` attempts to write each part separately before propagating the + exception. + If converting an argument to a string fails, the output collected so far is + written without appending *end*, and the exception is propagated. + (Contributed by Inada Naoki in :gh:`161934`.) + * The :func:`iter` function now accepts the *stop_exception* parameter. The created iterator stops when the callable raises the specified exception. The second parameter is now named *stop_value* and can be passed by keyword. diff --git a/Include/internal/pycore_fileutils.h b/Include/internal/pycore_fileutils.h index 60204a69dd093cf..1b74e93b9850c32 100644 --- a/Include/internal/pycore_fileutils.h +++ b/Include/internal/pycore_fileutils.h @@ -314,6 +314,11 @@ PyAPI_FUNC(char*) _Py_UniversalNewlineFgetsWithSize(char *, int, FILE*, PyObject extern int _PyFile_Flush(PyObject *); +// Write a private list of strings to an exact TextIOWrapper. +// Return 1 on success, 0 to use write() instead, or a negative value on error. +// -2 means preparing the output failed before any write was attempted. +extern int _PyTextIOWrapper_WriteList(PyObject *file, PyObject *strlist); + #ifndef MS_WINDOWS extern int _Py_GetTicksPerSecond(long *ticks_per_second); #endif diff --git a/Include/internal/pycore_magic_number.h b/Include/internal/pycore_magic_number.h index 15d3ad6f3a336b7..041256df56b4938 100644 --- a/Include/internal/pycore_magic_number.h +++ b/Include/internal/pycore_magic_number.h @@ -304,6 +304,8 @@ Known values: Python 3.16a1 3704 (Replace DELETE_ATTR with PUSH_NULL; STORE_ATTR) Python 3.16a1 3705 (Add INTRINSIC_ADD_CONDITIONAL_ANNOTATION) Python 3.16a1 3706 (Add INTRINSIC_MATCH_CLASS_ISINSTANCE) + Python 3.16a1 3707 (GET_AWAITABLE also pushes NULL and None for SEND) + Python 3.16a1 3708 (GET_ANEXT also pushes NULL and None for SEND) Python 3.17 will start with 3750 @@ -313,7 +315,7 @@ Known values: */ -#define PYC_MAGIC_NUMBER 3706 +#define PYC_MAGIC_NUMBER 3708 /* This is equivalent to converting PYC_MAGIC_NUMBER to 2 bytes (little-endian) and then appending b'\r\n'. */ #define PYC_MAGIC_NUMBER_TOKEN \ diff --git a/Include/internal/pycore_opcode_metadata.h b/Include/internal/pycore_opcode_metadata.h index 04b826ed09144e7..f1e6f1beae644a5 100644 --- a/Include/internal/pycore_opcode_metadata.h +++ b/Include/internal/pycore_opcode_metadata.h @@ -714,9 +714,9 @@ int _PyOpcode_num_pushed(int opcode, int oparg) { case GET_AITER: return 1; case GET_ANEXT: - return 2; + return 4; case GET_AWAITABLE: - return 1; + return 3; case GET_ITER: return 2; case GET_ITER_SELF: diff --git a/Include/internal/pycore_uop_ids.h b/Include/internal/pycore_uop_ids.h index ba6e0252024a6df..31a8e60259f4baf 100644 --- a/Include/internal/pycore_uop_ids.h +++ b/Include/internal/pycore_uop_ids.h @@ -689,8 +689,8 @@ enum { _FOR_ITER_VIRTUAL_r23, _FOR_ITER_VIRTUAL_TIER_TWO_r23, _GET_AITER_r11, - _GET_ANEXT_r12, - _GET_AWAITABLE_r11, + _GET_ANEXT_r13, + _GET_AWAITABLE_r13, _GET_ITER_r12, _GET_ITER_TRAD_r12, _GET_LEN_r12, diff --git a/Include/internal/pycore_uop_metadata.h b/Include/internal/pycore_uop_metadata.h index 0cd82c53aa932ae..a264ad3f7848b76 100644 --- a/Include/internal/pycore_uop_metadata.h +++ b/Include/internal/pycore_uop_metadata.h @@ -1630,7 +1630,7 @@ const _PyUopCachingInfo _PyUop_Caching[MAX_UOP_ID+1] = { .best = { 1, 1, 1, 1 }, .entries = { { -1, -1, -1 }, - { 2, 1, _GET_ANEXT_r12 }, + { 3, 1, _GET_ANEXT_r13 }, { -1, -1, -1 }, { -1, -1, -1 }, }, @@ -1639,7 +1639,7 @@ const _PyUopCachingInfo _PyUop_Caching[MAX_UOP_ID+1] = { .best = { 1, 1, 1, 1 }, .entries = { { -1, -1, -1 }, - { 1, 1, _GET_AWAITABLE_r11 }, + { 3, 1, _GET_AWAITABLE_r13 }, { -1, -1, -1 }, { -1, -1, -1 }, }, @@ -4283,8 +4283,8 @@ const uint16_t _PyUop_Uncached[MAX_UOP_REGS_ID+1] = { [_MAKE_HEAP_SAFE_r33] = _MAKE_HEAP_SAFE, [_RETURN_VALUE_r11] = _RETURN_VALUE, [_GET_AITER_r11] = _GET_AITER, - [_GET_ANEXT_r12] = _GET_ANEXT, - [_GET_AWAITABLE_r11] = _GET_AWAITABLE, + [_GET_ANEXT_r13] = _GET_ANEXT, + [_GET_AWAITABLE_r13] = _GET_AWAITABLE, [_SEND_GEN_FRAME_r33] = _SEND_GEN_FRAME, [_GUARD_TOS_IS_NONE_r01] = _GUARD_TOS_IS_NONE, [_GUARD_TOS_IS_NONE_r11] = _GUARD_TOS_IS_NONE, @@ -5236,9 +5236,9 @@ const char *const _PyOpcode_uop_name[MAX_UOP_REGS_ID+1] = { [_GET_AITER] = "_GET_AITER", [_GET_AITER_r11] = "_GET_AITER_r11", [_GET_ANEXT] = "_GET_ANEXT", - [_GET_ANEXT_r12] = "_GET_ANEXT_r12", + [_GET_ANEXT_r13] = "_GET_ANEXT_r13", [_GET_AWAITABLE] = "_GET_AWAITABLE", - [_GET_AWAITABLE_r11] = "_GET_AWAITABLE_r11", + [_GET_AWAITABLE_r13] = "_GET_AWAITABLE_r13", [_GET_ITER] = "_GET_ITER", [_GET_ITER_r12] = "_GET_ITER_r12", [_GET_ITER_TRAD] = "_GET_ITER_TRAD", diff --git a/InternalDocs/profiling_binary_format.md b/InternalDocs/profiling_binary_format.md index a3d86b1a4ab5693..b467514ca228e3d 100644 --- a/InternalDocs/profiling_binary_format.md +++ b/InternalDocs/profiling_binary_format.md @@ -41,7 +41,7 @@ The file consists of five required sections and one optional extension: | Header | 64 bytes (fixed) +------------------+ Offset 64 | | -| Sample Data | Variable size (optionally compressed) +| Sample Data | Blocks of columns (optionally compressed) | | +------------------+ string_table_offset | String Table | Variable size @@ -60,8 +60,8 @@ so these tables must be built incrementally and written at the end. The header comes first so readers can quickly validate the file and locate the metadata tables. The sample data follows immediately, allowing the writer -to stream samples directly to disk (or through a compression stream) without -buffering the entire dataset in memory. +to write each block of samples as soon as it is full without buffering the +entire dataset in memory. The string and frame tables are placed after sample data because they grow as new unique entries are discovered during profiling. By deferring their @@ -120,86 +120,141 @@ network streams. ## Sample Data Sample data begins at offset 64 and extends to `string_table_offset`. Samples -use delta compression to minimize redundancy when consecutive samples from the -same thread have identical or similar call stacks. +are grouped into blocks of at most 1 MiB of decoded data. Within a block, +samples use delta compression to minimize redundancy when consecutive samples +from the same thread have identical or similar call stacks. Each block can be +decoded on its own once the string and frame tables are loaded, so a reader +only needs to decode one block at a time. + +### Block Format + +A block stores values of the same kind together, in six columns: + +``` ++------------------------+ +| Block header | 21 to 88 bytes ++------------------------+ +| STREAM_TABLE | (thread_id, interpreter_id) pairs ++------------------------+ +| SAMPLE_STREAM_IDS | One stream index per sample ++------------------------+ +| TIMESTAMP_RESIDUALS | One timestamp residual per sample ++------------------------+ +| STATUS_BITMAPS | One bitmap per varying status flag ++------------------------+ +| STACK_INSTRUCTIONS | Stack encodings ++------------------------+ +| FRAME_IDS | Frame table indices ++------------------------+ +``` + +**Block header:** +``` ++---------------------+-----------+----------------------------------------+ +| magic | 4 bytes | "TBLK" | +| sample_count | varint | Number of samples in this block | +| base_timestamp | varint | Timestamp of the first sample | +| frame_id_width | 1 byte | Bytes per frame index (1 to 4) | +| common_status_flags | 1 byte | Status bits set in every sample | +| status_bitmap_flags | 1 byte | Status bits stored as bitmaps | +| columns | varies | One column descriptor per column | ++---------------------+-----------+----------------------------------------+ +``` + +**Column descriptor:** +``` ++---------------------+-----------+----------------------------------------+ +| decoded_size | varint | Size of the decoded column | +| encoding | 1 byte | 0=raw, 1=zstd, 2=constant | +| stored_size | varint | Compressed size (zstd only) | ++---------------------+-----------+----------------------------------------+ +``` + +A raw column is stored as is, and a constant column is a single byte repeated +`decoded_size` times, so neither needs a stored size. The number of streams +and frame indices in a block follows from the decoded sizes of +`STREAM_TABLE` and `FRAME_IDS`. + +A block holds at most 4,096 streams. There is no separate limit on the number +of samples: the stream index and timestamp columns need at least one byte per +sample, so the 1 MiB limit already bounds it. ### Stack Encoding Types -Each sample record begins with thread identification, then an encoding byte: +Each thread's stack is reconstructed from the previous stack of the same +thread in the block. The first sample of a thread in a block starts from an +empty stack. `STACK_INSTRUCTIONS` holds one of these encodings for every +sample not covered by a `REPEAT`: | Code | Name | Description | |------|------|-------------| | 0x00 | REPEAT | RLE: identical stack repeated N times | -| 0x01 | FULL | Complete stack (first sample or no match) | -| 0x02 | SUFFIX | Shares N frames from bottom of previous stack | -| 0x03 | POP_PUSH | Remove M frames from top, add N new frames | +| 0x01 | REPLACE_SUFFIX | Keeps N outer frames of the previous stack, adds M new frames | +| 0x02 | REUSE | Complete stack stored earlier in the block | + +Stacks hold at most 256 frames. The writer keeps the innermost 256 frames of +deeper stacks. -### Record Formats +### Stack Encoding Formats + +All fields are varints. Stacks are written outermost frame first. **REPEAT (0x00) - Run-Length Encoded Identical Stacks:** ``` +-----------------+-----------+----------------------------------------+ -| thread_id | 8 bytes | Thread identifier (uint64, fixed) | -| interpreter_id | 4 bytes | Interpreter ID (uint32, fixed) | -| encoding | 1 byte | 0x00 (REPEAT) | +| encoding | varint | 0x00 (REPEAT) | | count | varint | Number of samples in this RLE group | -| samples | varies | Interleaved: [delta: varint, status: 1]| -| | | repeated count times | +-----------------+-----------+----------------------------------------+ ``` -The stack is inherited from this thread's previous sample. Each sample in the -group gets its own timestamp delta and status byte, stored as interleaved pairs -(delta1, status1, delta2, status2, ...) rather than separate arrays. +The stack is inherited from this thread's previous sample, for this sample +and the thread's next `count - 1` samples. Samples from other threads can +appear in between: the reader keeps a remaining count for each thread and +reads no encoding for that thread until the count reaches zero. Each sample +in the group keeps its own timestamp and status in the other columns. -**FULL (0x01) - Complete Stack:** +**REPLACE_SUFFIX (0x01) - Shared Outer Frames:** ``` +-----------------+-----------+----------------------------------------+ -| thread_id | 8 bytes | Thread identifier (uint64, fixed) | -| interpreter_id | 4 bytes | Interpreter ID (uint32, fixed) | -| encoding | 1 byte | 0x01 (FULL) | -| timestamp_delta | varint | Microseconds since thread's last sample| -| status | 1 byte | Thread state flags | -| stack_depth | varint | Number of frames in call stack | -| frame_indices | varint[] | Array of frame table indices | +| encoding | varint | 0x01 (REPLACE_SUFFIX) | +| keep_count | varint | Outer frames kept from previous stack | +| new_count | varint | Inner frames read from FRAME_IDS | +-----------------+-----------+----------------------------------------+ ``` -Used for the first sample from a thread, or when delta encoding would not -provide savings. +Used when a call or return changed the inner part of the stack: the outer +frames are kept, and `new_count` new frames are read from `FRAME_IDS`. A +complete stack, such as the first sample of a thread, uses `keep_count` 0. -**SUFFIX (0x02) - Shared Suffix Match:** +**REUSE (0x02) - Repeated Complete Stack:** ``` +-----------------+-----------+----------------------------------------+ -| thread_id | 8 bytes | Thread identifier (uint64, fixed) | -| interpreter_id | 4 bytes | Interpreter ID (uint32, fixed) | -| encoding | 1 byte | 0x02 (SUFFIX) | -| timestamp_delta | varint | Microseconds since thread's last sample| -| status | 1 byte | Thread state flags | -| shared_count | varint | Frames shared from bottom of prev stack| -| new_count | varint | New frames at top of stack | -| new_frames | varint[] | Array of new_count frame indices | +| encoding | varint | 0x02 (REUSE) | +| offset | varint | Index of the first frame in FRAME_IDS | +| count | varint | Number of frames in the stack | +-----------------+-----------+----------------------------------------+ ``` -Used when a function call added frames to the top of the stack. The shared -frames from the previous stack are kept, and new frames are prepended. +Used when a complete stack was already written earlier in the same block, +for example by another thread running the same code. The frames are copied +from `FRAME_IDS` instead of being stored again. -**POP_PUSH (0x03) - Pop and Push:** -``` -+-----------------+-----------+----------------------------------------+ -| thread_id | 8 bytes | Thread identifier (uint64, fixed) | -| interpreter_id | 4 bytes | Interpreter ID (uint32, fixed) | -| encoding | 1 byte | 0x03 (POP_PUSH) | -| timestamp_delta | varint | Microseconds since thread's last sample| -| status | 1 byte | Thread state flags | -| pop_count | varint | Frames to remove from top of prev stack| -| push_count | varint | New frames to add at top | -| new_frames | varint[] | Array of push_count frame indices | -+-----------------+-----------+----------------------------------------+ -``` -Used when the code path changed: some frames were popped (function returns) -and new frames were pushed (different function calls). +For example, two threads produce four samples: + +| Sample | Thread | Stack | Encoding | +|--------|--------|----------------|----------------------| +| 0 | A | `[4, 21, 37]` | `REPLACE_SUFFIX 0 3` | +| 1 | B | `[4, 21, 37]` | `REUSE 0 3` | +| 2 | A | `[4, 21, 52]` | `REPLACE_SUFFIX 2 1` | +| 3 | A | `[4, 21, 52]` | `REPEAT 1` | + +`FRAME_IDS` holds only `[4, 21, 37, 52]`. ### Thread and Interpreter Identification +Each `(thread_id, interpreter_id)` pair used in a block is stored once in +`STREAM_TABLE`, as an 8-byte thread ID followed by a 4-byte interpreter ID. +`SAMPLE_STREAM_IDS` then holds one varint per sample: the index of its pair +in the table. Samples are kept in arrival order, so samples from different +threads are interleaved. + Thread IDs are 64-bit values that can be large (memory addresses on some platforms) and vary unpredictably. Using a fixed 8-byte encoding avoids the overhead of varint encoding for large values and simplifies parsing @@ -226,23 +281,30 @@ Multiple flags can be set simultaneously (e.g., a thread can hold the GIL while also running on CPU). Analysis tools use these to filter samples or visualize thread states over time. +Most flags rarely change within a block, so the status bytes are stored as +bit planes. Bits set in every sample are recorded once in +`common_status_flags`. Each bit in `status_bitmap_flags` has a bitmap of +`ceil(sample_count / 8)` bytes in `STATUS_BITMAPS`, in increasing bit order, +with sample `i` in bit `i % 8` of byte `i / 8`. Bits that never occur take no +space. The reader rebuilds each status byte from `common_status_flags` and +the sample's bit in each bitmap. + ### Timestamp Delta Encoding Timestamps use delta encoding rather than absolute values. Absolute timestamps in microseconds require 8 bytes each, but consecutive samples from the same thread are typically separated by the sampling interval -(e.g., 1000 microseconds), so the delta between them is small and fits -in 1-2 varint bytes. The writer tracks the previous timestamp for each -thread separately. The first sample from a thread encodes its delta from -the profiling start time; subsequent samples encode the delta from that -thread's previous sample. This per-thread tracking is necessary because -samples are interleaved across threads in arrival order, not grouped by -thread. - -For REPEAT (RLE) records, timestamp deltas and status bytes are stored as -interleaved pairs (delta, status, delta, status, ...) - one pair per -repeated sample - allowing efficient batching while preserving the exact -timing and state of each sample. +(e.g., 1000 microseconds), so each timestamp is expected to be the thread's +previous timestamp plus the interval, and only the difference from that +expected timestamp is stored. The writer tracks the previous timestamp for +each thread separately. The expected timestamp for a thread's first sample +in a block is the block's `base_timestamp`. This per-thread tracking is +necessary because samples are interleaved across threads in arrival order, +not grouped by thread. + +Each residual is a signed 64-bit zigzag varint. A sample taken exactly on +schedule has a residual of 0 and takes one byte, while timestamps are still +preserved exactly. ### Frame Indexing @@ -251,15 +313,25 @@ rather than inline data. This provides massive space savings because call stacks are highly repetitive: the same function appears in many samples (hot functions), call stacks often share common prefixes (main -> app -> handler -> ...), and recursive functions create repeated frame sequences. -A frame index is typically 1-2 varint bytes. Inline frame data would be -20-200+ bytes (two strings plus a line number). For a profile with 100,000 -samples averaging 30 frames each, this reduces frame data from potentially +A frame index is typically 1-2 bytes. Inline frame data would be 20-200+ +bytes (two strings plus a line number). For a profile with 100,000 samples +averaging 30 frames each, this reduces frame data from potentially gigabytes to tens of megabytes. -Frame indices are written innermost-first (the currently executing frame -has index 0 in the array). This ordering works well with delta compression: -function calls typically add frames at the top (index 0), while shared -frames remain at the bottom. +`FRAME_IDS` stores the indices read by `REPLACE_SUFFIX`, using the smallest +byte width that fits the largest index in the block. The bytes are split into +planes, low bytes first, so similar bytes sit together for compression: + +``` +indices: 0x1234 0x5678 0x9abc +low-byte plane: 0x34 0x78 0xbc +high-byte plane: 0x12 0x56 0x9a +``` + +Frame indices are written outermost-first (the outermost frame has index 0 +in the array). This ordering works well with delta compression: function +calls typically change frames at the end of the array, while shared frames +remain at the start. ## String Table @@ -453,29 +525,34 @@ bytes. Zigzag encoding interleaves positive and negative values: ``` This ensures small-magnitude values (whether positive or negative) encode -in few bytes. +in few bytes. Frame table deltas use 32-bit zigzag values and timestamp +residuals use 64-bit zigzag values. ## Compression -When compression is enabled, the sample data region contains a zstd stream. +The header's compression type controls whether zstd may be used. Each column +of each block is encoded separately, using whichever representation is +smallest: + +- **constant**, when every byte of the column is the same, such as the stream + IDs of a single-threaded block or residuals of on-time samples; +- **zstd**, when compression is enabled and the compressed column plus its + stored-size varint is smaller than the raw column; +- **raw** otherwise. + The string table, frame table, and footer remain uncompressed so readers can access metadata without decompressing the entire file. A tool that only needs to report "this file contains 50,000 samples of 3 threads" can read the header -and footer without touching the compressed sample data. This also simplifies -the format: the header's offset fields point directly to the tables rather -than to positions within a decompressed stream. +and footer without touching the compressed sample data. Zstd provides an excellent balance of compression ratio and speed. Profiling data compresses very well (often 5-10x) due to repetitive patterns: the same -small set of frame indices appears repeatedly, and delta-encoded timestamps -cluster around the sampling interval. Zstd's streaming API allows compression -without buffering the entire dataset. The writer feeds sample data through -the compressor incrementally, flushing compressed chunks to disk as they -become available. +small set of frame indices appears repeatedly, and timestamp residuals +cluster around zero. -Level 5 compression is used as a default. Lower levels (1-3) are faster but -compress less; higher levels (6+) compress more but slow down writing. Level -5 provides good compression with minimal impact on profiling overhead. +Level 3 compression is used as a default. It gives good compression without +slowing down each block flush. Compressing columns separately keeps blocks +independent, at the cost of restarting compression history at every block. ## Reading and Writing @@ -486,9 +563,9 @@ compress less; higher levels (6+) compress more but slow down writing. Level 3. For each sample: - Intern any new strings, assigning sequential indices - Intern any new frames, assigning sequential indices - - Encode the sample record and write to the buffer - - Flush the buffer through compression (if enabled) when full -4. Flush remaining buffered data and finalize compression + - Append the sample to the current block's columns + - Flush the block when it reaches its size or stream limit +4. Flush the last block 5. Write the string table (length-prefixed strings in index order) 6. Write the frame table (varint-encoded entries in index order) 7. Write measured profile statistics, when available @@ -497,7 +574,13 @@ compress less; higher levels (6+) compress more but slow down writing. Level The writer maintains two dictionaries: one mapping strings to indices, one mapping (filename_idx, funcname_idx, lineno) tuples to frame indices. These -enable O(1) lookup during interning. +enable O(1) lookup during interning. To find `REUSE` candidates, it also keeps +a small per-block cache of complete stacks, confirmed by a full comparison. + +When a block is flushed, the status and frame ID columns are transposed into +bit and byte planes, the `REPEAT` counts are inserted into the instruction +column, and each column is encoded as described in +[Compression](#compression). ### Reading @@ -510,13 +593,32 @@ enable O(1) lookup during interning. 6. Parse the string table, populating the array 7. Allocate the frame array 8. Parse the frame table, populating the array -9. If compressed, decompress the sample data region -10. Iterate through samples, resolving indices to strings/frames - (byte-swapping thread_id and interpreter_id if needed) +9. For each block, decode its columns into a reusable buffer and replay its + samples in order, resolving indices to strings/frames (byte-swapping the + stream table if needed) The reader builds lookup arrays rather than dictionaries since it only needs index-to-value mapping, not value-to-index. +Consecutive samples of one stream with the same stack and status are passed +to the collector in a single call, with up to 8,192 timestamps. + +The block header and columns are validated before any sample of the block is +replayed. Stack instructions are checked as they are replayed, so a block +with a bad instruction may already have passed earlier samples to the +collector before the error is raised. The reader rejects a block when: + +- a size, encoding, or stream or frame reference is out of range, or a + column has trailing data; +- the stream table lists a thread twice or has a stream with no samples; +- a stream's first instruction in the block is `REPEAT`, or a `REPEAT` + count is zero or runs past the block; +- a stack would be deeper than 256 frames, or `REUSE` refers to frame IDs + that have not been read yet; +- not every frame ID is used; +- status bitmap padding bits are not zero; +- it uses zstd in an uncompressed file. + ## Platform Considerations ### Byte Ordering and Cross-Platform Portability @@ -540,7 +642,7 @@ zeros instead of the actual value. The reader tracks whether byte-swapping is needed via a `needs_swap` flag set during header parsing. All fixed-width fields in the header, footer, and -sample data are conditionally byte-swapped using Python's internal byte-swap +stream tables are conditionally byte-swapped using Python's internal byte-swap functions (`_Py_bswap32`, `_Py_bswap64` from `pycore_bitutils.h`). Variable-length integers (varints) are byte-order independent since they @@ -568,10 +670,8 @@ On Windows, the implementation falls back to standard file I/O with full file buffering. Profiling data files are typically small enough (tens to hundreds of megabytes) that this is acceptable. -The writer uses a 512 KB buffer to batch small writes. Each sample record -is typically tens of bytes; writing these individually would incur excessive -syscall overhead. The buffer accumulates data until full, then flushes in -one write() call (or feeds through the compression stream). +The writer buffers samples in the current block and writes each completed +block with a single call, so individual samples never cause a write. ## Future Considerations diff --git a/Lib/_pyio.py b/Lib/_pyio.py index 77239f026efbb17..ec1b20d04267688 100644 --- a/Lib/_pyio.py +++ b/Lib/_pyio.py @@ -733,6 +733,8 @@ def _readinto(self, b, read1): data = self.read1(len(b)) else: data = self.read(len(b)) + if data is None: + return None n = len(data) b[:n] = data @@ -1222,14 +1224,22 @@ def _readinto(self, buf, read1): if len(buf) - written > self.buffer_size: n = self.raw.readinto(buf[written:]) if not n: + if n is None and not written: + return None # would block break # eof written += n # Otherwise refill internal buffer - unless we're # in read1 mode and already got some data elif not (read1 and written): - if not self._peek_unlocked(1): + # The internal buffer is empty here. + chunk = self.raw.read(self.buffer_size) + if not chunk: + if chunk is None and not written: + return None # would block break # eof + self._read_buf = chunk + self._read_pos = 0 # In readinto1 mode, return as soon as we have some data if read1 and written: @@ -1404,6 +1414,15 @@ def peek(self, size=0): def read1(self, size=-1): return self.reader.read1(size) + def readline(self, size=-1): + r"""Read and return a line from the stream. + + If size is specified, at most size bytes will be read. + + The line terminator is always b'\n'. + """ + return self.reader.readline(size) + def readinto1(self, b): return self.reader.readinto1(b) diff --git a/Lib/getopt.py b/Lib/getopt.py index 25f3e2439b3e352..272a4bf7d0c2820 100644 --- a/Lib/getopt.py +++ b/Lib/getopt.py @@ -47,11 +47,10 @@ def __str__(self): error = GetoptError # backward compatibility def getopt(args, shortopts, longopts = []): - """getopt(args, options[, long_options]) -> opts, args + """Parse command line options and parameter list. - Parses command line options and parameter list. args is the - argument list to be parsed, without the leading reference to the - running program. Typically, this means "sys.argv[1:]". shortopts + args is the argument list to be parsed, without the leading reference + to the running program. Typically, this means "sys.argv[1:]". shortopts is the string of option letters that the script wants to recognize, with options that require an argument followed by a colon and options that accept an optional argument followed by @@ -72,7 +71,6 @@ def getopt(args, shortopts, longopts = []): string if the option has no argument. The options occur in the list in the same order in which they were found, thus allowing multiple occurrences. Long and short options may be mixed. - """ opts = [] @@ -92,7 +90,7 @@ def getopt(args, shortopts, longopts = []): return opts, args def gnu_getopt(args, shortopts, longopts = []): - """getopt(args, options[, long_options]) -> opts, args + """Parse command line options using GNU style scanning. This function works like getopt(), except that GNU style scanning mode is used by default. This means that option and non-option @@ -103,7 +101,6 @@ def gnu_getopt(args, shortopts, longopts = []): If the first character of the option string is '+', or if the environment variable POSIXLY_CORRECT is set, then option processing stops as soon as a non-option argument is encountered. - """ opts = [] diff --git a/Lib/json/decoder.py b/Lib/json/decoder.py index ee0464318458117..31f410632d4f87e 100644 --- a/Lib/json/decoder.py +++ b/Lib/json/decoder.py @@ -112,7 +112,7 @@ def py_scanstring(s, end, strict=True, char = _b[esc] except KeyError: msg = "Invalid \\escape: {0!r}".format(esc) - raise JSONDecodeError(msg, s, end) + raise JSONDecodeError(msg, s, end - 1) end += 1 else: uni = _decode_uXXXX(s, end) diff --git a/Lib/profiling/sampling/binary_collector.py b/Lib/profiling/sampling/binary_collector.py index 3d6d988077cfa2a..21f58daf76f3dd4 100644 --- a/Lib/profiling/sampling/binary_collector.py +++ b/Lib/profiling/sampling/binary_collector.py @@ -146,7 +146,7 @@ def get_stats(self): """Get encoding statistics. Returns: - Dict with encoding statistics including repeat/full/suffix/pop-push + Dict with encoding statistics including repeat/full/suffix/reuse record counts, frames written/saved, and compression ratio. """ return self._writer.get_stats() diff --git a/Lib/profiling/sampling/collector.py b/Lib/profiling/sampling/collector.py index 1dc3656e0ebe976..3e2baa0f1a6f91d 100644 --- a/Lib/profiling/sampling/collector.py +++ b/Lib/profiling/sampling/collector.py @@ -20,16 +20,13 @@ def normalize_location(location): """Normalize location to a 4-tuple format. Args: - location: tuple (lineno, end_lineno, col_offset, end_col_offset), - an integer line number, or None + location: tuple (lineno, end_lineno, col_offset, end_col_offset) or None Returns: tuple: (lineno, end_lineno, col_offset, end_col_offset) """ if location is None: return DEFAULT_LOCATION - if isinstance(location, int): - return (location, location, -1, -1) return location @@ -37,16 +34,13 @@ def extract_lineno(location): """Extract lineno from location. Args: - location: tuple (lineno, end_lineno, col_offset, end_col_offset), - an integer line number, or None + location: tuple (lineno, end_lineno, col_offset, end_col_offset) or None Returns: int: The line number (0 for synthetic frames) """ if location is None: return 0 - if isinstance(location, int): - return location return location[0] def _is_internal_frame(frame): diff --git a/Lib/profiling/sampling/gecko_collector.py b/Lib/profiling/sampling/gecko_collector.py index 2de8cce387e7f26..90092ac7d86fe72 100644 --- a/Lib/profiling/sampling/gecko_collector.py +++ b/Lib/profiling/sampling/gecko_collector.py @@ -368,11 +368,10 @@ def _record_stack_sample(self, thread_data, frames, tid, times, first_time): if self.opcodes_enabled and frames: leaf_frame = frames[0] filename, location, funcname, opcode = leaf_frame - if isinstance(location, tuple): - lineno, _, col_offset, _ = location + if location is None: + lineno, col_offset = None, -1 else: - lineno = location - col_offset = -1 + lineno, _, col_offset, _ = location current_state = (opcode, lineno, col_offset, funcname, filename) @@ -541,15 +540,11 @@ def _process_stack(self, thread_data, frames): for frame_tuple in reversed(frames): # frame_tuple is (filename, location, funcname, opcode) - # location is (lineno, end_lineno, col_offset, end_col_offset) or just lineno filename, location, funcname, opcode = frame_tuple - if isinstance(location, tuple): - lineno, end_lineno, col_offset, end_col_offset = location + if location is None: + lineno, col_offset = None, -1 else: - # Legacy format: location is just lineno - lineno = location - col_offset = -1 - end_col_offset = -1 + lineno, _, col_offset, _ = location # Get or create function func_idx = self._get_or_create_func( diff --git a/Lib/profiling/sampling/live_collector/collector.py b/Lib/profiling/sampling/live_collector/collector.py index 553c856c6f8c562..a69351e4c795c4a 100644 --- a/Lib/profiling/sampling/live_collector/collector.py +++ b/Lib/profiling/sampling/live_collector/collector.py @@ -313,7 +313,6 @@ def process_frames(self, frames, thread_id=None): self._seen_locations.clear() # Process each frame in the stack to track cumulative calls - # frame.location is (lineno, end_lineno, col_offset, end_col_offset), int, or None for frame in frames: lineno = extract_lineno(frame.location) location = (frame.filename, lineno, frame.funcname) diff --git a/Lib/profiling/sampling/pstats_collector.py b/Lib/profiling/sampling/pstats_collector.py index 0f31d93582b83e7..66144f9fdbf192a 100644 --- a/Lib/profiling/sampling/pstats_collector.py +++ b/Lib/profiling/sampling/pstats_collector.py @@ -30,7 +30,6 @@ def _process_frames(self, frames, weight=1): self._seen_locations.clear() # Process each frame in the stack to track cumulative calls - # frame.location is int, tuple (lineno, end_lineno, col_offset, end_col_offset), or None for frame in frames: lineno = extract_lineno(frame.location) location = (frame.filename, lineno, frame.funcname) diff --git a/Lib/profiling/sampling/sample.py b/Lib/profiling/sampling/sample.py index 1dff6529ade0408..e50af2528e93bf0 100644 --- a/Lib/profiling/sampling/sample.py +++ b/Lib/profiling/sampling/sample.py @@ -411,22 +411,22 @@ def _print_binary_stats(self, collector): repeat_samples = stats.get('repeat_samples', 0) full_records = stats.get('full_records', 0) suffix_records = stats.get('suffix_records', 0) - pop_push_records = stats.get('pop_push_records', 0) + reuse_records = stats.get('reuse_records', 0) total_records = stats.get('total_records', 0) if total_records > 0: repeat_pct = repeat_records / total_records * 100 full_pct = full_records / total_records * 100 suffix_pct = suffix_records / total_records * 100 - pop_push_pct = pop_push_records / total_records * 100 + reuse_pct = reuse_records / total_records * 100 else: - repeat_pct = full_pct = suffix_pct = pop_push_pct = 0 + repeat_pct = full_pct = suffix_pct = reuse_pct = 0 print(f" Records: {total_records:,}") print(f" RLE repeat: {repeat_records:,} ({ANSIColors.GREEN}{repeat_pct:.1f}%{ANSIColors.RESET}) [{repeat_samples:,} samples]") print(f" Full stack: {full_records:,} ({full_pct:.1f}%)") print(f" Suffix match: {suffix_records:,} ({suffix_pct:.1f}%)") - print(f" Pop-push: {pop_push_records:,} ({pop_push_pct:.1f}%)") + print(f" Stack reuse: {reuse_records:,} ({reuse_pct:.1f}%)") frames_written = stats.get('total_frames_written', 0) frames_saved = stats.get('frames_saved', 0) diff --git a/Lib/tarfile.py b/Lib/tarfile.py index b052ca0dd407292..ca6aea8396aaa1e 100644 --- a/Lib/tarfile.py +++ b/Lib/tarfile.py @@ -2554,7 +2554,7 @@ def extractall(self, path=".", members=None, *, numeric_owner=False, continue dirpath = os.path.join(path, tarinfo.name) try: - lstat = os.lstat(dirpath) + lstat = os.lstat(dirpath.replace(r'/', os.sep)) except FileNotFoundError: self._log_no_directory_fixup(tarinfo, 'missing') continue diff --git a/Lib/tempfile.py b/Lib/tempfile.py index 82c1ee353117e94..53fea2873d2bae8 100644 --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -987,10 +987,13 @@ def __init__(self, suffix=None, prefix=None, dir=None, @classmethod def _rmtree(cls, name, ignore_errors=False, repeated=False, dir_fd=None, fullname=None): - if fullname is None: - fullname = name + + # work with str paths throughout + name = _os.fsdecode(name) + fullname = name if fullname is None else _os.fsdecode(fullname) def onexc(func, path, exc, direntry=None, dir_fd=None): + path = _os.fsdecode(path) # On DragonFly BSD, UF_NOUNLINK removal fails with EISDIR, not EPERM. if isinstance(exc, (PermissionError, IsADirectoryError)): if repeated and path == name: diff --git a/Lib/test/lazy_imports_all_exclude.txt b/Lib/test/lazy_imports_all_exclude.txt index e5519323e88e38a..f85ea5e4f301c04 100644 --- a/Lib/test/lazy_imports_all_exclude.txt +++ b/Lib/test/lazy_imports_all_exclude.txt @@ -20,10 +20,8 @@ test.test_inspect.test_inspect test___all__ test__interpreters test_crossinterp -test_datetime test_import test_importlib -test_json test_pkg test_profile test_profiling diff --git a/Lib/test/libregrtest/main.py b/Lib/test/libregrtest/main.py index 032cb94223a7e65..547ef232c204940 100644 --- a/Lib/test/libregrtest/main.py +++ b/Lib/test/libregrtest/main.py @@ -210,9 +210,11 @@ def find_tests(self, tests: TestList | None = None) -> tuple[TestTuple, TestList if self.tsan: setup_tsan_tests(self.cmdline_args) + self.want_header = True if self.tsan_parallel: setup_tsan_parallel_tests(self.cmdline_args) + self.want_header = True alltests = findtests(testdir=self.test_dir, exclude=exclude_tests) diff --git a/Lib/test/libregrtest/result.py b/Lib/test/libregrtest/result.py index f8aa256be1ecfe8..ba4ba4d13ea1d7d 100644 --- a/Lib/test/libregrtest/result.py +++ b/Lib/test/libregrtest/result.py @@ -226,6 +226,7 @@ def get_rerun_match_tests(self) -> FilterTuple | None: def write_json_into(self, file) -> None: json.dump(self, file, cls=_EncodeTestResult) + file.flush() @staticmethod def from_json(worker_json: StrJSON) -> 'TestResult': diff --git a/Lib/test/libregrtest/results.py b/Lib/test/libregrtest/results.py index eac8f4c40ebca17..c57b82e05a5b21d 100644 --- a/Lib/test/libregrtest/results.py +++ b/Lib/test/libregrtest/results.py @@ -168,6 +168,7 @@ def prepare_rerun(self, *, clear: bool = True) -> tuple[TestTuple, FilterDict]: self.bad.clear() self.env_changed.clear() self.rerun_results.clear() + self.worker_bug = False return (tuple(tests), match_tests_dict) diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py index f69734037a8fe57..2cfe4b7f20e1015 100644 --- a/Lib/test/support/__init__.py +++ b/Lib/test/support/__init__.py @@ -1407,7 +1407,13 @@ def internal(*args, **kwargs): import_module('_testcapi') return test(*args, **kwargs) - return cpython_only(internal) + # gh-162144: set_nomemory() and call_with_nomemory() of _testcapi are not + # thread safe and emit warnings when using a thread sanitizer (TSan). + use_tsan = check_sanitizer(thread=True) + reason ='not working with thread sanitizer (gh-157415)' + skip_if_tsan = unittest.skipIf(use_tsan, reason) + + return cpython_only(skip_if_tsan(internal)) def bigaddrspacetest(f): """Decorator for tests that fill the address space.""" diff --git a/Lib/test/support/import_helper.py b/Lib/test/support/import_helper.py index 962f65d8dcd14a2..26790c6995e7810 100644 --- a/Lib/test/support/import_helper.py +++ b/Lib/test/support/import_helper.py @@ -156,6 +156,7 @@ def multi_interp_extensions_check(enabled=True): def import_fresh_module(name, fresh=(), blocked=(), *, deprecated=False, usefrozen=False, + eager_imports=True, ): """Import and return a module, deliberately bypassing sys.modules. @@ -187,7 +188,12 @@ def import_fresh_module(name, fresh=(), blocked=(), *, """ # NOTE: test_heapq, test_json and test_warnings include extra sanity checks # to make sure that this utility function is working as expected - with _ignore_deprecated_imports(deprecated): + eager_imports_context = ( + force_eager_imports + if eager_imports + else contextlib.nullcontext + ) + with eager_imports_context(), _ignore_deprecated_imports(deprecated): # Keep track of modules saved for later restoration as well # as those which just need a blocking entry removed fresh = list(fresh) diff --git a/Lib/test/test_build_details.py b/Lib/test/test_build_details.py index 12b7664d83aae16..40bf1bef11a8885 100644 --- a/Lib/test/test_build_details.py +++ b/Lib/test/test_build_details.py @@ -5,11 +5,12 @@ import sys import sysconfig import string +import textwrap import unittest from pathlib import Path from unittest import mock -from test.support import is_android, is_apple_mobile, is_wasm32 +from test.support import is_android, is_apple_mobile, is_wasm32, os_helper BASE_PATH = Path( __file__, # Lib/test/test_build_details.py @@ -142,6 +143,17 @@ def contents(self): def test_location(self): self.assertTrue(os.path.isfile(self.location)) + def test_abi(self): + self.assertEqual(self.key('abi.flags'), list(sys.abiflags)) + self.assertEqual(self.key('abi.extension_suffix'), + sysconfig.get_config_var('EXT_SUFFIX')) + self.assertIn(self.key('abi.stable_abi_suffix'), + importlib.machinery.EXTENSION_SUFFIXES) + + def test_extension_suffixes(self): + self.assertEqual(self.key('suffixes.extensions'), + importlib.machinery.EXTENSION_SUFFIXES) + # Override generic format tests with tests for our specific implemenation. @needs_installed_python @@ -289,6 +301,100 @@ def check_relative_paths(self, data_abs_path, data_rel_path): self.assertEqual(data_abs_path[part][key], abs_rel_path) +def generate_data(**config_vars): + """Generate build-details data with overridden config vars.""" + get_config_var = sysconfig.get_config_var + def patched_get_config_var(name): + if name in config_vars: + return config_vars[name] + return get_config_var(name) + with mock.patch('sysconfig.get_config_var', patched_get_config_var): + data = generate_build_details.generate_data(schema_version='1.0') + return json.loads(json.dumps(data)) + + +@unittest.skipIf( + generate_build_details is None, + "Failed to import generate-build-details" +) +@unittest.skipIf(os.name != 'posix', 'Feature only implemented on POSIX') +class BuildDetailsTargetTests(unittest.TestCase): + """Values that must come from the target configuration (gh-136267).""" + + def test_native_extension_suffixes(self): + self.assertEqual(generate_build_details.get_extension_suffixes(), + importlib.machinery.EXTENSION_SUFFIXES) + + def test_free_threaded_debug(self): + # A GIL-enabled release build Python generating for this target + # used to report its own flags and suffixes. + data = generate_data( + ABIFLAGS='td', MACHDEP='linux', DYNLOADFILE='dynload_shlib.o', + SOABI_PLATFORM='aarch64-linux-gnu', + SOABI='cpython-3XXtd-aarch64-linux-gnu', + ALT_SOABI='cpython-3XXt-aarch64-linux-gnu', + EXT_SUFFIX='.cpython-3XXtd-aarch64-linux-gnu.so', + Py_GIL_DISABLED=1) + self.assertEqual(data['abi']['flags'], ['t', 'd']) + self.assertEqual(data['abi']['extension_suffix'], + '.cpython-3XXtd-aarch64-linux-gnu.so') + self.assertEqual(data['abi']['stable_abi_suffix'], + '.abi3t-aarch64-linux-gnu.so') + self.assertEqual(data['suffixes']['extensions'], [ + '.cpython-3XXtd-aarch64-linux-gnu.so', + '.cpython-3XXt-aarch64-linux-gnu.so', + '.abi3t-aarch64-linux-gnu.so', + '.abi3t.so', + '.so', + ]) + + def test_no_dynamic_loading(self): + data = generate_data(MACHDEP='wasi', DYNLOADFILE='dynload_stub.o') + self.assertEqual(data['suffixes']['extensions'], []) + self.assertNotIn('extension_suffix', data['abi']) + self.assertNotIn('stable_abi_suffix', data['abi']) + self.assertFalse( + data['implementation']['supports_isolated_interpreters']) + + +@unittest.skipIf( + generate_build_details is None, + "Failed to import generate-build-details" +) +class BuildDetailsVersionTests(unittest.TestCase): + def test_native(self): + version_info = generate_build_details.get_version_info() + self.assertEqual(version_info, + generate_build_details.version_info_to_dict( + sys.version_info)) + self.assertEqual(generate_build_details.hexversion(version_info), + sys.hexversion) + + def test_patchlevel(self): + patchlevel_h = os.path.join(os_helper.TESTFN, 'patchlevel.h') + self.addCleanup(os_helper.rmtree, os_helper.TESTFN) + os.mkdir(os_helper.TESTFN) + with open(patchlevel_h, 'w', encoding='utf-8') as f: + f.write(textwrap.dedent('''\ + #define PY_RELEASE_LEVEL_GAMMA 0xC /* For release candidates */ + #define PY_MAJOR_VERSION 3 + #define PY_MINOR_VERSION 99 + #define PY_MICRO_VERSION 7 + #define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_GAMMA + #define PY_RELEASE_SERIAL 2 + #define PY_VERSION "3.99.7rc2" + ''')) + with mock.patch.object(generate_build_details, 'PATCHLEVEL_H', + patchlevel_h): + version_info = generate_build_details.get_version_info() + self.assertEqual(version_info, { + 'major': 3, 'minor': 99, 'micro': 7, + 'releaselevel': 'candidate', 'serial': 2, + }) + self.assertEqual(generate_build_details.hexversion(version_info), + 0x036307c2) + + @unittest.skipIf( generate_build_details is None, "Failed to import generate-build-details" @@ -299,26 +405,25 @@ class BuildDetailsLibpythonTests(unittest.TestCase): LIBDIR = '/prefix/lib' LIBPL = '/prefix/lib/python3.X/config-3.X' + def generate(self, **config_vars): + return generate_data(LIBDIR=self.LIBDIR, LIBPL=self.LIBPL, + LIBRARY='libpython3.X.a', **config_vars) + def generate_libpython(self, **config_vars): """Generate the libpython section with overridden config vars.""" - config_vars = dict(LIBDIR=self.LIBDIR, LIBPL=self.LIBPL, - LIBRARY='libpython3.X.a', **config_vars) - get_config_var = sysconfig.get_config_var - def patched_get_config_var(name): - if name in config_vars: - return config_vars[name] - return get_config_var(name) - with mock.patch('sysconfig.get_config_var', patched_get_config_var): - data = generate_build_details.generate_data(schema_version='1.0') - return json.loads(json.dumps(data)).get('libpython', {}) + return self.generate(**config_vars).get('libpython', {}) def test_static_only(self): - libpython = self.generate_libpython( + data = self.generate( LDLIBRARY='libpython3.X.a', STATIC_LIBPYTHON=1, PYTHONFRAMEWORK='') + libpython = data['libpython'] self.assertNotIn('dynamic', libpython) self.assertEqual(libpython['static'], os.path.join(self.LIBDIR, 'libpython3.X.a')) + # Extension modules are supported without a shared libpython. + self.assertIn('extension_suffix', data['abi']) + self.assertIn('stable_abi_suffix', data['abi']) def test_shared_and_static(self): # The static library is only installed into LIBPL, not LIBDIR. diff --git a/Lib/test/test_capi/test_dict.py b/Lib/test/test_capi/test_dict.py index 9a2e79d4da7e715..4028b56a9cc22c0 100644 --- a/Lib/test/test_capi/test_dict.py +++ b/Lib/test/test_capi/test_dict.py @@ -635,11 +635,47 @@ class DictSubtype(dict): ... self.assertIs(type(d), DictSubtype) self.assertEqual(d, DictSubtype({})) + # non str keys: + d = {1: 'a', 2: 'b', 3: 'c', None: 'f'} + f = as_frozendict(d) + self.assertIs(type(f), frozendict) + self.assertEqual(f, frozendict({1: 'a', 2: 'b', 3: 'c', None: 'f'})) + self.assertIs(type(d), dict) + self.assertEqual(d, {}) + + d = DictSubtype({1: 'a', 2: 'b', 3: 'c', None: 'f'}) + f = as_frozendict(d) + self.assertIs(type(f), frozendict) + self.assertEqual(f, frozendict({1: 'a', 2: 'b', 3: 'c', None: 'f'})) + self.assertIs(type(d), DictSubtype) + self.assertEqual(d, DictSubtype({})) + + # wrong inputs: for wrong_input in (frozendict(), [], None): with self.subTest(wrong_input=wrong_input): with self.assertRaises(SystemError): as_frozendict(wrong_input) + # gh-161767: check type with inline values and dict with embedded values + class Regular: ... + + for klass in (Regular, _testcapi.ManagedDictType): + with self.subTest(klass=klass): + instance = klass() # empty `.__dict__` + with self.assertRaisesRegex(ValueError, 'dict with embedded values'): + as_frozendict(instance.__dict__) + self.assertEqual(instance.__dict__, {}) + + instance = klass() # non-empty `.__dict__` + instance.key = 'value' + with self.assertRaisesRegex(ValueError, 'dict with embedded values'): + as_frozendict(instance.__dict__) + + self.assertEqual(instance.__dict__, {'key': 'value'}) + # Both of these actions were needed to cause a crash: + instance.other = 'new' + del instance # must not crash + def test_frozendict_check(self): # Test PyFrozenDict_Check() check = _testcapi.frozendict_check diff --git a/Lib/test/test_capi/test_unicode.py b/Lib/test/test_capi/test_unicode.py index 84303bc379cf130..fe4e8e3b5046e62 100644 --- a/Lib/test/test_capi/test_unicode.py +++ b/Lib/test/test_capi/test_unicode.py @@ -1883,7 +1883,9 @@ def test_check_interned(self): def fresh_string(text): # Create a new string object by decoding a bytes string. - return text.encode().decode() + # Add a suffix to make sure that it's not interned, even if + # the string is interned in the PYC file (Free Threading). + return (text + ' suffix').encode().decode() s = fresh_string('hello') self.assertEqual(check_interned(s), SSTATE_NOT_INTERNED) diff --git a/Lib/test/test_code_module.py b/Lib/test/test_code_module.py index 944e0e5655e0854..03217561858d0cd 100644 --- a/Lib/test/test_code_module.py +++ b/Lib/test/test_code_module.py @@ -225,7 +225,7 @@ def test_sysexcepthook_crashing_doesnt_close_repl(self): self.infunc.side_effect = ["1/0", "a = 123", "print(a)", EOFError('Finished')] self.sysmod.excepthook = 1 self.console.interact() - self.assertEqual(['write', ('123', ), {}], self.stdout.method_calls[0]) + self.stdout.write.assert_called_once_with('123\n') error = "".join(call.args[0] for call in self.stderr.method_calls if call[0] == 'write') self.assertIn("Error in sys.excepthook:", error) self.assertEqual(error.count("'int' object is not callable"), 1) @@ -239,7 +239,7 @@ def raise_base(*args, **kwargs): raise BaseException(s) self.sysmod.excepthook = raise_base self.console.interact() - self.assertEqual(['write', ('123', ), {}], self.stdout.method_calls[0]) + self.stdout.write.assert_called_once_with('123\n') error = "".join(call.args[0] for call in self.stderr.method_calls if call[0] == 'write') self.assertIn("Error in sys.excepthook:", error) self.assertEqual(error.count("not so fast"), 1) diff --git a/Lib/test/test_ctypes/test_arrays.py b/Lib/test/test_ctypes/test_arrays.py index 7f1f6cf58402c9f..5b7708afdaa1249 100644 --- a/Lib/test/test_ctypes/test_arrays.py +++ b/Lib/test/test_ctypes/test_arrays.py @@ -1,11 +1,14 @@ import ctypes import sys +import threading import unittest +from concurrent.futures import ThreadPoolExecutor from ctypes import (Structure, Array, ARRAY, sizeof, addressof, create_string_buffer, create_unicode_buffer, c_char, c_wchar, c_byte, c_ubyte, c_short, c_ushort, c_int, c_uint, c_long, c_ulonglong, c_float, c_double, c_longdouble) -from test.support import bigmemtest, _2G, threading_helper, Py_GIL_DISABLED +from test.support import (bigmemtest, _2G, threading_helper, + Py_GIL_DISABLED, SHORT_TIMEOUT) from ._support import (_CData, PyCArrayType, Py_TPFLAGS_DISALLOW_INSTANTIATION, Py_TPFLAGS_IMMUTABLETYPE) @@ -288,5 +291,49 @@ def run(): raise cm.exc_value +class ArrayTypeCacheTest(unittest.TestCase): + # gh-160978: the array type cache could hand out distinct type objects for + # the same (item type, length) pair when the array type was created from + # more than one thread at a time. + + @threading_helper.requires_working_threading() + def test_concurrent_creation_with_set_name(self): + # Creating the array type runs Python code -- ``__set_name__`` here -- + # which lets a second thread reach the cache before the first one has + # stored its type. Without the fix this fails on the default build as + # well, so it guards the fix even where threads cannot interleave + # freely. + barrier = threading.Barrier(2) + + class Meta(type(c_int)): + def __set_name__(cls, owner, name): + barrier.wait(timeout=SHORT_TIMEOUT) + + class Item(c_int, metaclass=Meta): + pass + + with ThreadPoolExecutor(2) as pool: + first, second = pool.map(lambda _: Item * 100, range(2)) + + self.assertIs(first, second) + # Raises TypeError if the two types are distinct. + first.from_param(second()) + + @threading_helper.requires_working_threading() + def test_concurrent_creation_returns_identical_types(self): + nthreads = 8 + barrier = threading.Barrier(nthreads) + + def make_type(length): + barrier.wait(timeout=SHORT_TIMEOUT) + return c_int * length + + with ThreadPoolExecutor(nthreads) as pool: + for length in range(1, 51): + types = list(pool.map(make_type, [length] * nthreads)) + barrier.reset() + self.assertEqual(len(set(types)), 1) + + if __name__ == '__main__': unittest.main() diff --git a/Lib/test/test_dis.py b/Lib/test/test_dis.py index 709fa374704f1f5..2c3972c7394d01b 100644 --- a/Lib/test/test_dis.py +++ b/Lib/test/test_dis.py @@ -615,8 +615,6 @@ async def _asyncwith(c): LOAD_SPECIAL 2 (__aenter__) CALL 0 GET_AWAITABLE 1 - PUSH_NULL - LOAD_COMMON_CONSTANT 7 (None) L2: SEND 4 (to L5) L3: YIELD_VALUE 1 L4: RESUME 3 @@ -632,8 +630,6 @@ async def _asyncwith(c): LOAD_COMMON_CONSTANT 7 (None) CALL 3 GET_AWAITABLE 2 - PUSH_NULL - LOAD_COMMON_CONSTANT 7 (None) L8: SEND 4 (to L11) L9: YIELD_VALUE 1 L10: RESUME 3 @@ -647,14 +643,12 @@ async def _asyncwith(c): RETURN_VALUE %4d L12: CLEANUP_THROW - L13: JUMP_BACKWARD_NO_INTERRUPT 28 (to L5) + L13: JUMP_BACKWARD_NO_INTERRUPT 26 (to L5) L14: CLEANUP_THROW L15: JUMP_BACKWARD_NO_INTERRUPT 10 (to L11) L16: PUSH_EXC_INFO WITH_EXCEPT_START GET_AWAITABLE 2 - PUSH_NULL - LOAD_COMMON_CONSTANT 7 (None) L17: SEND 5 (to L21) L18: YIELD_VALUE 1 L19: RESUME 3 diff --git a/Lib/test/test_external_inspection.py b/Lib/test/test_external_inspection.py index 104efaebd85c16f..5aebe1fb6b18e95 100644 --- a/Lib/test/test_external_inspection.py +++ b/Lib/test/test_external_inspection.py @@ -55,7 +55,7 @@ try: from _remote_debugging import PROCESS_VM_READV_SUPPORTED from _remote_debugging import RemoteUnwinder - from _remote_debugging import FrameInfo, CoroInfo, TaskInfo + from _remote_debugging import FrameInfo, CoroInfo, TaskInfo, AwaitedInfo except ImportError: raise unittest.SkipTest( "Test only runs when _remote_debugging is available" @@ -487,6 +487,23 @@ def _extract_coroutine_funcnames(self, trace): for task in info.awaited_by } + @skip_if_not_supported + def test_async_awaited_info(self): + async def main(): + task = asyncio.current_task() + unwinder = RemoteUnwinder(os.getpid()) + for method in (unwinder.get_async_stack_trace, + unwinder.get_all_awaited_by): + with self.subTest(method=method.__name__): + info, = (info for info in method() + if info.thread_id == threading.get_native_id()) + self.assertIsInstance(info, AwaitedInfo) + task_info, = info.awaited_by + self.assertEqual(task_info.task_id, id(task)) + self.assertEqual(task_info.task_name, task.get_name()) + + asyncio.run(main()) + @skip_if_not_supported def test_long_task_name_is_truncated(self): # gh-157788 @@ -520,6 +537,22 @@ async def rec(n): self.assertEqual(asyncio.run(rec(3)), ["rec"] * 4) + @skip_if_not_supported + def test_future_waiter_has_empty_stack(self): + async def main(): + task = asyncio.current_task() + future = asyncio.get_running_loop().create_future() + asyncio.future_add_to_awaited_by(task, future) + try: + trace = RemoteUnwinder(os.getpid()).get_all_awaited_by() + task_info = self._get_task_id_map(trace)[id(task)] + self.assertEqual(task_info.awaited_by, [([], id(future))]) + self.assertTrue(task_info.coroutine_stack[0].call_stack) + finally: + asyncio.future_discard_from_awaited_by(task, future) + + asyncio.run(main()) + @skip_if_not_supported def test_async_generator_stack_is_not_truncated(self): # gh-159620 @@ -663,6 +696,17 @@ async def main(): stacks = self._extract_coroutine_funcnames(asyncio.run(main())) self.assertEqual(stacks["consumer"], ["consumer"]) + @skip_if_not_supported + def test_self_trace_with_empty_linetable(self): + def sample(): + return RemoteUnwinder(os.getpid()).get_stack_trace() + + sample.__code__ = sample.__code__.replace(co_linetable=b"") + frame = self._find_frame_in_trace( + sample(), lambda f: f.funcname == sample.__qualname__ + ) + self.assertEqual(frame.location, (-1, -1, -1, -1)) + @skip_if_not_supported @unittest.skipIf( sys.platform == "linux" and not PROCESS_VM_READV_SUPPORTED, diff --git a/Lib/test/test_free_threading/test_iteration.py b/Lib/test/test_free_threading/test_iteration.py index 540e8edcc676285..bf42304b41e8b96 100644 --- a/Lib/test/test_free_threading/test_iteration.py +++ b/Lib/test/test_free_threading/test_iteration.py @@ -1,3 +1,4 @@ +import itertools import sys import threading import unittest @@ -170,6 +171,31 @@ def drain(it, barrier, results): self.assertEqual(sys.getrefcount(seq), refcount_before) +class ContendedCallIterExhaustionTest(unittest.TestCase): + """Test exhausting a shared iter(callable, sentinel) iterator. + + The values seen by each thread are unspecified, but exhausting the + iterator from several threads at once must not crash. + """ + + def test_shared_iterator_exhaustion(self): + niters = 2000 if support.check_sanitizer(thread=True) else 20000 + + def make_iter(): + counter = itertools.count().__next__ + return iter(lambda: counter(), 3) + + shared = [make_iter()] + + def worker(): + for _ in range(niters): + it = shared[0] + if next(it, None) is None: + shared[0] = make_iter() + + threading_helper.run_concurrently(worker, nthreads=NUMTHREADS) + + class ContendedRangeIterationTest(ContendedTupleIterationTest): def make_testdata(self, n): return range(n) diff --git a/Lib/test/test_interpreters/test_api.py b/Lib/test/test_interpreters/test_api.py index a9e296891bb8705..e9a64915a1b4a87 100644 --- a/Lib/test/test_interpreters/test_api.py +++ b/Lib/test/test_interpreters/test_api.py @@ -947,6 +947,21 @@ def test_failure(self): with self.assertRaises(ExecutionFailed): interp.exec('raise Exception') + def test_failure_with_broken_exception_str(self): + interp = interpreters.create() + for raised, expected in [ + ('ValueError', interpreters.InterpreterError), + ('MemoryError', MemoryError), + ]: + with self.subTest(raised=raised): + with self.assertRaises(expected): + interp.exec(dedent(f""" + class BrokenError(Exception): + def __str__(self): + raise {raised} + raise BrokenError + """)) + @force_not_colorized def test_display_preserved_exception(self): tempdir = self.temp_dir() diff --git a/Lib/test/test_io/test_bufferedio.py b/Lib/test/test_io/test_bufferedio.py index e83dd0d4e28d006..64789821afe59c2 100644 --- a/Lib/test/test_io/test_bufferedio.py +++ b/Lib/test/test_io/test_bufferedio.py @@ -306,12 +306,18 @@ def test_readinto(self): self.assertEqual(b, b"gf") self.assertEqual(bufio.readinto(b), 0) self.assertEqual(b, b"gf") - rawio = self.MockRawIO((b"abc", None)) - bufio = self.tp(rawio) + rawio = self.MockRawIO((b"abc", None, None, None, None, None)) + bufio = self.tp(rawio, buffer_size=4) self.assertEqual(bufio.readinto(b), 2) self.assertEqual(b, b"ab") self.assertEqual(bufio.readinto(b), 1) self.assertEqual(b, b"cb") + self.assertIsNone(bufio.readinto(b)) + self.assertIsNone(bufio.readinto1(b)) + # Larger than buffer_size: read directly into b + b = bytearray(5) + self.assertIsNone(bufio.readinto(b)) + self.assertIsNone(bufio.readinto1(b)) def test_readinto1(self): buffer_size = 10 @@ -1057,6 +1063,20 @@ def test_read(self): pair = self.tp(self.BytesIO(b"abc"), self.MockRawIO()) self.assertEqual(pair.read(None), b"abc") + def test_readline(self): + pair = self.tp(self.BytesIO(b"ab\ncd"), self.MockRawIO()) + + self.assertEqual(pair.readline(1), b"a") + self.assertEqual(pair.readline(5), b"b\n") + self.assertEqual(pair.readline(0), b"") + self.assertEqual(pair.readline(None), b"cd") + + def test_readline_nonblocking(self): + raw = self.MockRawIO([b"abc", None, None]) + pair = self.tp(raw, self.MockRawIO()) + + self.assertEqual(pair.readline(), b"abc") + def test_readlines(self): pair = lambda: self.tp(self.BytesIO(b"abc\ndef\nh"), self.MockRawIO()) self.assertEqual(pair().readlines(), [b"abc\n", b"def\n", b"h"]) diff --git a/Lib/test/test_io/test_general.py b/Lib/test/test_io/test_general.py index c4d5a80086718b9..883144817dddc33 100644 --- a/Lib/test/test_io/test_general.py +++ b/Lib/test/test_io/test_general.py @@ -713,15 +713,18 @@ def test_invalid_newline(self): def test_buffered_readinto_mixin(self): # Test the implementation provided by BufferedIOBase class Stream(self.BufferedIOBase): + def __init__(self, data): + self.data = data def read(self, size): - return b"12345" + return self.data read1 = read - stream = Stream() for method in ("readinto", "readinto1"): with self.subTest(method): buffer = byteslike(5) - self.assertEqual(getattr(stream, method)(buffer), 5) + self.assertEqual(getattr(Stream(b"12345"), method)(buffer), 5) self.assertEqual(bytes(buffer), b"12345") + # Non-blocking stream with no data + self.assertIsNone(getattr(Stream(None), method)(buffer)) def test_fspath_support(self): def check_path_succeeds(path): @@ -895,6 +898,41 @@ def test_immutable_types(self): with self.assertRaisesRegex(TypeError, "immutable"): tp.foo = "bar" + def test_managed_dict(self): + # The _io types use Py_TPFLAGS_MANAGED_DICT: there is no explicit + # dict slot and instance attributes are stored as inline values. + _testinternalcapi = import_helper.import_module("_testinternalcapi") + _io = self._io + no_dict = {_io.IncrementalNewlineDecoder, _io._BytesIOBuffer} + for tp in self.types: + with self.subTest(tp=tp): + if tp in no_dict: + self.assertEqual(tp.__dictoffset__, 0) + else: + self.assertEqual(tp.__dictoffset__, -1) + + # Use fresh subclasses: each class has its own shared keys, which + # may already be exhausted on the base types by earlier tests. + class BytesIO(_io.BytesIO): pass + class StringIO(_io.StringIO): pass + class BufferedReader(_io.BufferedReader): pass + class TextIOWrapper(_io.TextIOWrapper): pass + class RawIO(_io._RawIOBase): + def __init__(self): + self.attr = 1 + instances = [ + BytesIO(b"data"), + StringIO("data"), + BufferedReader(_io.BytesIO(b"data")), + TextIOWrapper(_io.BytesIO()), + RawIO(), + ] + for obj in instances: + with self.subTest(obj=type(obj).__name__): + obj.attr = 1 + self.assertTrue(_testinternalcapi.has_inline_values(obj)) + self.assertEqual(obj.__dict__["attr"], 1) + def test_class_hierarchy(self): def check_subs(types, base): for tp in types: diff --git a/Lib/test/test_io/test_memoryio.py b/Lib/test/test_io/test_memoryio.py index 89b0e4c9463ac0a..20c2777b6c3ce07 100644 --- a/Lib/test/test_io/test_memoryio.py +++ b/Lib/test/test_io/test_memoryio.py @@ -10,6 +10,7 @@ import io import _pyio as pyio import pickle +import struct import sys import weakref @@ -1037,11 +1038,13 @@ def test_setstate(self): @support.cpython_only def test_sizeof(self): if support.Py_GIL_DISABLED: - basesize = support.calcobjsize('P2n2Pni') + basesize = support.calcobjsize('P2nni') else: - basesize = support.calcobjsize('P2n2Pn') + basesize = support.calcobjsize('P2nn') check = self.check_sizeof self.assertEqual(object.__sizeof__(io.BytesIO()), basesize) + # sys.getsizeof() includes the managed dict and weakref pre-header + basesize += 2 * struct.calcsize('P') check(io.BytesIO(), basesize ) n = 1000 # use a variable to prevent constant folding check(io.BytesIO(b'a' * n), basesize + sys.getsizeof(b'a' * n)) diff --git a/Lib/test/test_io/test_winconsoleio.py b/Lib/test/test_io/test_winconsoleio.py index 1bae884ed9ae3ea..b2474007da98935 100644 --- a/Lib/test/test_io/test_winconsoleio.py +++ b/Lib/test/test_io/test_winconsoleio.py @@ -160,7 +160,7 @@ def test_write(self): testcases.append(data + b'\xff') # incomplete multibyte sequence with self.subTest(data=data): - self.assertEqual(f.write(data), len(a)) + self.assertEqual(f.write(data), len(a or data)) for data in testcases: with self.subTest(data=data): self.assertEqual(f.write(data), len(data)) diff --git a/Lib/test/test_iter.py b/Lib/test/test_iter.py index fe8617309da29d0..6ebde4848a5a491 100644 --- a/Lib/test/test_iter.py +++ b/Lib/test/test_iter.py @@ -532,6 +532,23 @@ def spam(): with self.assertRaises(StopIteration): next(spam.iterator) + def test_iter_function_sentinel_replaced_during_compare(self): + # gh-158031: the sentinel's __eq__ replaces the sentinel, which must + # stay alive until the comparison is finished. + class Sentinel: + def __eq__(self, other): + it.__setstate__(((), StopIteration)) + return NotImplemented + + class Result: + def __eq__(self, other): + self.other_type = type(other) + return NotImplemented + + it = iter(Result, Sentinel()) + result = next(it) + self.assertIs(result.other_type, Sentinel) + # Test exception propagation through function iterator def test_exception_function(self): def spam(state=[0]): diff --git a/Lib/test/test_json/test_fail.py b/Lib/test/test_json/test_fail.py index 7f99a6331d36251..76ab6f9ced89016 100644 --- a/Lib/test/test_json/test_fail.py +++ b/Lib/test/test_json/test_fail.py @@ -201,6 +201,24 @@ def test_unexpected_data(self): '%s: line 1 column %d (char %d)' % (msg, idx + 1, idx)) + def test_invalid_escape_position(self): + # The position points to the backslash of the invalid escape. + test_cases = [ + ('"\\a"', 1), + ('"ab\\x"', 3), + ('["\\\'"]', 2), + ('{"k": "v\\ "}', 8), + ] + for data, idx in test_cases: + with self.subTest(data=data): + with self.assertRaises(self.JSONDecodeError) as cm: + self.loads(data) + err = cm.exception + self.assertStartsWith(err.msg, 'Invalid \\escape') + self.assertEqual(err.pos, idx) + self.assertEqual(err.lineno, 1) + self.assertEqual(err.colno, idx + 1) + def test_extra_data(self): test_cases = [ ('[]]', 'Extra data', 2), diff --git a/Lib/test/test_long.py b/Lib/test/test_long.py index c16bf1b1fe020e9..ce72c256774d77a 100644 --- a/Lib/test/test_long.py +++ b/Lib/test/test_long.py @@ -1734,6 +1734,20 @@ class MyInt(int): # GH-117195 -- This shouldn't crash object.__sizeof__(1) + def test_long_add_overallocate(self): + # see gh-100687 + x = (MASK//2) * (MASK+1) + x2 = (MASK//2 + 1) * (MASK+1) + self.assertEqual(x + x2, MASK * (MASK + 1)) + + def test_karatsuba_single_digit_parts(self): + # gh-100687: k_mul() adds the halves of its operands with x_add(), + # and those halves can be single digits. + a = 1 + (1 << (SHIFT * 70)) + b = 1 << (SHIFT * 139) + self.assertEqual(a * b, (1 << (SHIFT * 139)) + (1 << (SHIFT * 209))) + self.assertEqual(b * a, a * b) + def test_hash(self): # gh-136599 self.assertEqual(hash(-1), -2) diff --git a/Lib/test/test_pathlib/test_pathlib.py b/Lib/test/test_pathlib/test_pathlib.py index 10e3ad6d92daec8..8013b0df26ec81c 100644 --- a/Lib/test/test_pathlib/test_pathlib.py +++ b/Lib/test/test_pathlib/test_pathlib.py @@ -3065,12 +3065,7 @@ def test_glob_dotdot(self): else: # ".." segments are normalized first on Windows, so this path is stat()able. self.assertEqual(set(p.glob("xyzzy/..")), { P(self.base, "xyzzy", "..") }) - if sys.platform == "emscripten": - # Emscripten will return ELOOP if there are 49 or more ..'s. - # Can remove when https://github.com/emscripten-core/emscripten/pull/24591 is merged. - NDOTDOTS = 48 - else: - NDOTDOTS = 50 + NDOTDOTS = 50 self.assertEqual(set(p.glob("/".join([".."] * NDOTDOTS))), { P(self.base, *[".."] * NDOTDOTS)}) def test_glob_inaccessible(self): diff --git a/Lib/test/test_print.py b/Lib/test/test_print.py index 12256b3b5626375..eeb0ce15a51e7df 100644 --- a/Lib/test/test_print.py +++ b/Lib/test/test_print.py @@ -1,8 +1,12 @@ -import unittest +import codecs +import io import sys +import unittest from io import StringIO +from unittest.mock import Mock, call from test import support +from test.support import import_helper NotDefined = object() @@ -100,6 +104,120 @@ def x(expected, args, sep=NotDefined, end=NotDefined): self.assertRaises(TypeError, print, '', end=3) self.assertRaises(AttributeError, print, '', file='') + def test_print_single_write(self): + class Str(str): + def __str__(self): + raise AssertionError('unexpected conversion') + + file = Mock() + print(ClassWith__str__(Str('value')), 2, sep=Str('|'), end=Str('!'), + file=file) + file.write.assert_called_once_with('value|2!') + + def test_print_str_error(self): + file = Mock() + with self.assertRaises(TypeError): + print('first', ClassWith__str__(42), file=file, flush=True) + self.assertEqual(file.method_calls, + [call.write('first '), call.flush()]) + + file.write.side_effect = OSError + with self.assertRaises(OSError) as cm: + print('first', ClassWith__str__(42), file=file) + self.assertIsInstance(cm.exception.__context__, TypeError) + + def test_print_str_error_read_ahead(self): + with io.TextIOWrapper(io.BytesIO(b'abcdef'), encoding='utf-8') as file: + self.assertEqual(file.read(1), 'a') + file.flush = Mock(wraps=file.flush) + with self.assertRaises(TypeError): + print(ClassWith__str__(42), file=file, flush=True) + file.flush.assert_not_called() + self.assertEqual(file.read(), 'bcdef') + + @support.nomemtest + def test_print_allocation_error(self): + import _testcapi + + class Value: + def __str__(self): + # Fail the first allocation after collecting the parts. + _testcapi.set_nomemory(0, 1) + return 'second' + + file = Mock() + raw = io.BytesIO() + with io.TextIOWrapper(raw, encoding='utf-8') as text: + for dest in (file, text): + try: + with self.assertRaises(MemoryError): + print('first', Value(), file=dest, flush=True) + finally: + _testcapi.remove_mem_hooks() + self.assertEqual(file.method_calls, [ + call.write('first'), call.write(' '), call.write('second'), + call.write('\n'), call.flush(), + ]) + self.assertEqual(raw.getvalue(), b'first second\n') + + def test_print_textio_write_error(self): + raw = io.BytesIO() + raw.write = Mock(side_effect=MemoryError) + with io.TextIOWrapper(raw, encoding='utf-8', + line_buffering=True) as file: + with self.assertRaises(MemoryError): + print('first', 'second', file=file) + # Retrying individual parts could duplicate output. + raw.write.assert_called_once_with(b'first second\n') + + def test_print_textio_newline(self): + for encoding in ('utf-8', 'latin-1', 'utf-16'): + with self.subTest(encoding=encoding): + raw = io.BytesIO() + with io.TextIOWrapper(raw, encoding=encoding, newline='\r\n', + line_buffering=True) as file: + print('café', 2, sep='\n', end='', file=file) + self.assertEqual(raw.getvalue(), + 'café\r\n2'.encode(encoding)) + + def test_print_textio_write_override(self): + with io.TextIOWrapper(io.BytesIO(), encoding='utf-8') as file: + file.write = Mock() + print('first', 'second', file=file) + file.write.assert_called_once_with('first second\n') + + @support.cpython_only + def test_print_textio_utf8_cache(self): + _testcapi = import_helper.import_module('_testcapi') + data = 'a\0€'.encode() + cached, uncached = data.decode(), data.decode() + _testcapi.unicode_asutf8(cached, len(data)) + size = sys.getsizeof(uncached) + raw = io.BytesIO() + with io.TextIOWrapper(raw, encoding='utf-8') as file: + print(cached, uncached, file=file, flush=True) + self.assertEqual(raw.getvalue(), data + b' ' + data + b'\n') + self.assertEqual(sys.getsizeof(uncached), size) + + @support.thread_unsafe("modifies the global 'strict' error handler") + def test_print_textio_strict_error_handler(self): + def handler(error): + self.assertEqual(error.object, 'first \ud800\r\n') + file.reconfigure(newline='\r') + return '!', error.end + + self.addCleanup(codecs.register_error, 'strict', + codecs.lookup_error('strict')) + codecs.register_error('strict', handler) + for errors, expected in [('replace', b'first ?\r\n'), + ('strict', b'first !\r\n')]: + with self.subTest(errors=errors): + raw = io.BytesIO() + with io.TextIOWrapper(raw, encoding='utf-8', errors=errors, + newline='\r\n') as file: + print('first', '\ud800', file=file, flush=True) + self.assertEqual(raw.getvalue(), expected) + def test_print_flush(self): # operation of the flush flag class filelike: diff --git a/Lib/test/test_profiling/test_sampling_profiler/test_binary_format.py b/Lib/test/test_profiling/test_sampling_profiler/test_binary_format.py index 66a9c68a61c47dc..67b2708129b7970 100644 --- a/Lib/test/test_profiling/test_sampling_profiler/test_binary_format.py +++ b/Lib/test/test_profiling/test_sampling_profiler/test_binary_format.py @@ -10,7 +10,7 @@ from unittest import mock from collections import defaultdict -from test.support import captured_stderr +from test.support import captured_stderr, import_helper try: import _remote_debugging @@ -63,29 +63,16 @@ def make_interpreter(interp_id, threads): return InterpreterInfo((interp_id, threads)) -def extract_lineno(location): - """Extract line number from location (tuple or int or None).""" - if location is None: - return 0 # Treat None as 0 - if isinstance(location, tuple): - return location[0] if location[0] is not None else 0 - return location - - def extract_location(location): """Extract full location info as dict from location tuple or None.""" if location is None: return {"lineno": 0, "end_lineno": 0, "column": 0, "end_column": 0} - if isinstance(location, tuple) and len(location) >= 4: - return { - "lineno": location[0] if location[0] is not None else 0, - "end_lineno": location[1] if location[1] is not None else 0, - "column": location[2] if location[2] is not None else 0, - "end_column": location[3] if location[3] is not None else 0, - } - # Fallback for old-style location - lineno = location[0] if isinstance(location, tuple) else location - return {"lineno": lineno or 0, "end_lineno": lineno or 0, "column": 0, "end_column": 0} + return { + "lineno": location[0], + "end_lineno": location[1], + "column": location[2], + "end_column": location[3], + } def frame_to_dict(frame): @@ -127,6 +114,28 @@ def export(self, filename): pass +def encode_varint(value): + """Encode an unsigned varint.""" + out = bytearray() + while value >= 0x80: + out.append((value & 0x7F) | 0x80) + value >>= 7 + out.append(value) + return bytes(out) + + +def decode_varint(data, offset): + """Decode an unsigned varint, returning (value, next offset).""" + value = shift = 0 + while True: + byte = data[offset] + offset += 1 + value |= (byte & 0x7F) << shift + shift += 7 + if byte < 0x80: + return value, offset + + def samples_to_by_thread(samples): """Convert input samples to by-thread format for comparison.""" by_thread = defaultdict(list) @@ -147,6 +156,20 @@ def samples_to_by_thread(samples): class BinaryFormatTestBase(unittest.TestCase): """Base class with common setup/teardown for binary format tests.""" + HDR_OFF_STR_TABLE = 40 + FILE_HEADER_PLACEHOLDER_SIZE = 64 + + # Sample block layout (see binary_io.h). + BLOCK_MAGIC = b"TBLK" + COLUMN_COUNT = 6 + COLUMN_STREAM_IDS = 1 + COLUMN_TIMESTAMP_RESIDUALS = 2 + COLUMN_STATUS_BITMAPS = 3 + COLUMN_STACK_INSTRUCTIONS = 4 + COLUMN_FRAME_IDS = 5 + COLUMN_ZSTD = 1 + COLUMN_CONSTANT = 2 + def setUp(self): self.temp_files = [] @@ -155,6 +178,13 @@ def tearDown(self): if os.path.exists(f): os.unlink(f) + def temp_filename(self): + """Return a new temporary file name that is removed on cleanup.""" + with tempfile.NamedTemporaryFile(suffix=".bin", delete=False) as f: + filename = f.name + self.temp_files.append(filename) + return filename + def create_binary_file(self, samples, interval=1000, compression="none", mode=None, capture_config=None): """Create a test binary file and track it for cleanup.""" @@ -166,9 +196,7 @@ def create_binary_file(self, samples, interval=1000, compression="none", def write_binary_file(self, samples, interval=1000, compression="none", mode=None, capture_config=None): """Like create_binary_file but also returns the writer collector.""" - with tempfile.NamedTemporaryFile(suffix=".bin", delete=False) as f: - filename = f.name - self.temp_files.append(filename) + filename = self.temp_filename() collector = BinaryCollector( filename, interval, compression=compression, mode=mode, @@ -179,6 +207,45 @@ def write_binary_file(self, samples, interval=1000, compression="none", collector.export(None) return filename, collector + def read_block_columns(self, filename): + """Return the header flags and (decoded size, encoding) per column. + + Each block is a dict with frame_width, common_status_flags, + status_bitmap_flags and columns. + """ + with open(filename, "rb") as raw: + data = raw.read() + sample_end = struct.unpack_from("=Q", data, self.HDR_OFF_STR_TABLE)[0] + offset = self.FILE_HEADER_PLACEHOLDER_SIZE + blocks = [] + while offset < sample_end: + magic_end = offset + len(self.BLOCK_MAGIC) + self.assertEqual(data[offset:magic_end], self.BLOCK_MAGIC) + _, offset = decode_varint(data, magic_end) # sample count + _, offset = decode_varint(data, offset) # base timestamp + block = {"frame_width": data[offset], + "common_status_flags": data[offset + 1], + "status_bitmap_flags": data[offset + 2]} + offset += 3 + columns = [] + stored_total = 0 + for _ in range(self.COLUMN_COUNT): + decoded_size, offset = decode_varint(data, offset) + encoding = data[offset] + offset += 1 + if encoding == self.COLUMN_ZSTD: + stored_size, offset = decode_varint(data, offset) + elif encoding == self.COLUMN_CONSTANT: + stored_size = 1 + else: + stored_size = decoded_size + columns.append((decoded_size, encoding)) + stored_total += stored_size + offset += stored_total + block["columns"] = columns + blocks.append(block) + return blocks + def roundtrip(self, samples, interval=1000, compression="none"): """Write samples to binary and read back.""" filename = self.create_binary_file(samples, interval, compression) @@ -393,6 +460,29 @@ def test_binary_replay_preserves_main_thread_for_gecko(self): self.assertTrue(threads[1]["isMainThread"]) self.assertFalse(threads[2]["isMainThread"]) + def test_binary_replay_starts_gecko_at_earliest_sample(self): + """Binary replay uses the earliest sample as Gecko's time origin.""" + filename = self.temp_filename() + collector = BinaryCollector(filename, 1000, compression="none") + for thread_id, name, timestamp in [(1, "first", 1000), + (2, "second", 2000), + (2, "changed", 3000)]: + frame = make_frame(f"{name}.py", 10, name) + collector.collect( + [make_interpreter(0, [make_thread(thread_id, [frame])])], + timestamp_us=timestamp, + ) + collector.export(None) + + gecko = GeckoCollector(1000) + with BinaryReader(filename) as reader: + self.assertEqual(reader.replay_samples(gecko), 3) + + profile = gecko._build_profile() + threads = {thread["tid"]: thread for thread in profile["threads"]} + self.assertEqual(threads[1]["samples"]["time"], [0.0]) + self.assertEqual(threads[2]["samples"]["time"], [1.0, 2.0]) + def test_multiple_threads_per_sample(self): """Multiple threads in one sample roundtrip exactly.""" threads = [ @@ -897,6 +987,277 @@ def test_rejects_non_pathlike(self): with self.assertRaises(TypeError): _remote_debugging.BinaryReader(123) + def test_writer_rejects_invalid_compression(self): + """BinaryWriter rejects an unknown compression type.""" + with self.assertRaisesRegex(ValueError, "compression"): + _remote_debugging.BinaryWriter( + self.temp_filename(), 1000, 0, compression=2 + ) + + def test_writer_rejects_status_outside_byte_range(self): + """A status above 255 raises OverflowError and drops that sample.""" + filename = self.temp_filename() + writer = _remote_debugging.BinaryWriter(filename, 1000, 0) + valid = [make_interpreter(0, [make_thread(1, [], status=1)])] + invalid = [make_interpreter(0, [make_thread(1, [], status=256)])] + writer.write_sample(valid, 1000) + with self.assertRaisesRegex(OverflowError, "thread status"): + writer.write_sample(invalid, 2000) + self.assertTrue(writer.limit_reached) + writer.write_sample(valid, 3000) + writer.finalize() + self.assertEqual(writer.total_samples, 2) + + with BinaryReader(filename) as reader: + self.assertEqual(reader.replay_samples(RawCollector()), 2) + + def test_writer_reports_stack_reuse(self): + """A stack seen earlier in the block is stored as STACK_REUSE.""" + filename = self.temp_filename() + frames = [make_frame("shared.py", line, "shared") for line in range(3)] + sample = [make_interpreter(0, [ + make_thread(1, frames), + make_thread(2, frames), + ])] + writer = _remote_debugging.BinaryWriter(filename, 1000, 0) + writer.write_sample(sample, 1000) + stats = writer.get_stats() + self.assertEqual(stats["reuse_records"], 1) + self.assertEqual(stats["total_records"], 2) + self.assertEqual(stats["total_samples"], 2) + writer.close() + + def test_writer_buffered_bytes_match_flushed_bytes(self): + """bytes_written before finalize() matches the flushed size.""" + filename = self.temp_filename() + frames = [make_frame("a.py", line, "f") for line in range(300)] + writer = _remote_debugging.BinaryWriter(filename, 1000, 0) + for i in range(100): + sample = [make_interpreter(0, [ + make_thread(1, frames[i:i + 5], status=i % 3), + make_thread(2, frames[:3]), + ])] + writer.write_sample(sample, 1000 * (i + 1)) + buffered = writer.get_stats()["bytes_written"] + writer.finalize() + + [block] = self.read_block_columns(filename) + self.assertEqual(buffered, + sum(size for size, _ in block["columns"])) + + def test_samples_span_multiple_blocks(self): + """Replay preserves samples and progress across block boundaries.""" + # Stream 4097 starts the second block. Thread 1 then returns. + frame = make_frame("many_threads.py", 1, "work") + samples = [ + [make_interpreter(0, [make_thread(i + 1, [frame])])] + for i in range(4097) + ] + samples.append(samples[0]) + filename = self.create_binary_file(samples) + self.assertEqual(len(self.read_block_columns(filename)), 2) + + collector = RawCollector() + progress = mock.Mock() + with BinaryReader(filename) as reader: + self.assertEqual(reader.replay_samples(collector, progress), + len(samples)) + self.assert_samples_equal(samples, collector) + total = len(samples) + self.assertEqual(progress.call_args_list, + [mock.call(current, total) + for current in (0, 1000, 2000, 3000, 4000, total)]) + + def test_block_size_limit_starts_new_block(self): + """A thread whose samples exceed 1 MiB is split across blocks.""" + # Each sample writes a new 256-frame stack, about 1 KiB while + # buffered, so 1,100 samples need more than one block. + frames = [make_frame("deep.py", line, "f") for line in range(1356)] + samples = [ + [make_interpreter(0, [make_thread(1, frames[i:i + 256])])] + for i in range(1100) + ] + filename = self.create_binary_file(samples) + blocks = self.read_block_columns(filename) + self.assertGreaterEqual(len(blocks), 2) + for block in blocks: + self.assertLessEqual(sum(size for size, _ in block["columns"]), + 1024 * 1024) + with BinaryReader(filename) as reader: + self.assertEqual(reader.replay_samples(RawCollector()), + len(samples)) + + def test_writer_uses_constant_columns_and_narrow_frame_ids(self): + """Repeated bytes are stored once and frame IDs use the least width.""" + frame = make_frame("steady.py", 1, "steady") + sample = [make_interpreter(0, [make_thread(1, [frame])])] + filename = self.temp_filename() + writer = _remote_debugging.BinaryWriter(filename, 1000, 0) + for i in range(100): + writer.write_sample(sample, 1000 * (i + 1)) + writer.finalize() + # One stream and on-time samples: every stream ID and residual is 0 + [block] = self.read_block_columns(filename) + self.assertEqual(block["frame_width"], 1) + self.assertEqual(block["columns"][self.COLUMN_STREAM_IDS], + (100, self.COLUMN_CONSTANT)) + self.assertEqual(block["columns"][self.COLUMN_TIMESTAMP_RESIDUALS], + (100, self.COLUMN_CONSTANT)) + + # 300 distinct frames need two bytes per frame ID + frames = [make_frame("wide.py", line, "f") for line in range(300)] + samples = [ + [make_interpreter(0, [make_thread(1, frames[:150])])], + [make_interpreter(0, [make_thread(1, frames[150:])])], + ] + filename = self.create_binary_file(samples) + [block] = self.read_block_columns(filename) + self.assertEqual(block["frame_width"], 2) + self.assertEqual(block["columns"][self.COLUMN_FRAME_IDS][0], 300 * 2) + + def test_writer_stores_common_status_bits_once(self): + """Status bits set in every sample go in the header, not a bitmap.""" + frame = make_frame("status.py", 1, "f") + filename = self.temp_filename() + writer = _remote_debugging.BinaryWriter(filename, 1000, 0) + for i in range(20): + # HAS_GIL in every sample; ON_CPU in every other one + status = THREAD_STATUS_HAS_GIL | (THREAD_STATUS_ON_CPU * (i % 2)) + sample = [make_interpreter(0, [make_thread(1, [frame], status)])] + writer.write_sample(sample, 1000 * (i + 1)) + writer.finalize() + [block] = self.read_block_columns(filename) + self.assertEqual(block["common_status_flags"], THREAD_STATUS_HAS_GIL) + self.assertEqual(block["status_bitmap_flags"], THREAD_STATUS_ON_CPU) + # One bitmap of ceil(20 / 8) bytes, for ON_CPU only + self.assertEqual(block["columns"][self.COLUMN_STATUS_BITMAPS][0], 3) + + @unittest.skipUnless(ZSTD_AVAILABLE, "zstd compression not available") + def test_writer_compresses_repetitive_columns(self): + """Compressed files store repetitive columns with zstd.""" + root = [make_frame("root.py", line, "root") for line in range(20)] + leaves = [make_frame("leaf.py", 1, "even"), + make_frame("leaf.py", 2, "odd")] + samples = [ + [make_interpreter(0, [make_thread(1, [leaves[i % 2]] + root)])] + for i in range(2000) + ] + filename = self.create_binary_file(samples, compression="zstd") + [block] = self.read_block_columns(filename) + columns = block["columns"] + self.assertEqual(columns[self.COLUMN_STACK_INSTRUCTIONS][1], + self.COLUMN_ZSTD) + self.assertEqual(columns[self.COLUMN_FRAME_IDS][1], self.COLUMN_ZSTD) + + def test_replay_byte_swapped_file(self): + """A file written with the other byte order replays the same.""" + samples = [[make_interpreter(0, [ + make_thread(1, [make_frame("a.py", 1, "f")]), + make_thread(2, [make_frame("b.py", 2, "g")]), + ])] for _ in range(3)] + filename = self.temp_filename() + writer = _remote_debugging.BinaryWriter(filename, 1000, 0) + for i, sample in enumerate(samples): + writer.write_sample(sample, 1000 * (i + 1)) + writer.finalize() + + data = bytearray(pathlib.Path(filename).read_bytes()) + # Header fields: magic, version, start time, interval, sample + # count, thread count, table offsets, compression and config + fields = [(0, 4), (4, 4), (12, 8), (20, 8), (28, 8), (36, 4), + (40, 8), (48, 8), (56, 4), (60, 4)] + # Footer string and frame counts + fields += [(len(data) - 32, 4), (len(data) - 28, 4)] + # The block's stream table entries follow its header + offset = self.FILE_HEADER_PLACEHOLDER_SIZE + len(self.BLOCK_MAGIC) + _, offset = decode_varint(data, offset) # sample count + _, offset = decode_varint(data, offset) # base timestamp + offset += 3 + for _ in range(self.COLUMN_COUNT): + _, offset = decode_varint(data, offset) + offset += 1 # raw encoding, no stored size + for entry in range(2): + fields += [(offset + entry * 12, 8), (offset + entry * 12 + 8, 4)] + for start, size in fields: + data[start:start + size] = data[start:start + size][::-1] + pathlib.Path(filename).write_bytes(data) + + collector = RawCollector() + with BinaryReader(filename) as reader: + self.assertEqual(reader.replay_samples(collector), 6) + self.assert_samples_equal(samples, collector) + + def test_reader_replays_twice(self): + """Replaying twice gives the same samples and statistics.""" + class RetainingCollector(RawCollector): + def __init__(self): + super().__init__() + self.frame_lists = [] + + def collect(self, stack_frames, timestamps_us): + self.frame_lists.extend( + thread.frame_info + for interp in stack_frames + for thread in interp.threads + ) + super().collect(stack_frames, timestamps_us) + + samples = [ + [make_interpreter(0, [ + make_thread(1, [make_frame("a.py", i % 3, "f")]), + make_thread(2, [make_frame("b.py", 1, "g")]), + ])] + for i in range(10) + ] + filename = self.create_binary_file(samples) + with BinaryReader(filename) as reader: + first = RetainingCollector() + second = RetainingCollector() + self.assertEqual(reader.replay_samples(first), 20) + self.assertEqual(reader.replay_samples(second), 20) + self.assertEqual(reader.get_stats()["total_samples"], 20) + self.assertEqual(first.by_thread, second.by_thread) + for first_frames, second_frames in zip( + first.frame_lists, second.frame_lists, strict=True + ): + self.assertIsNot(first_frames, second_frames) + self.assertEqual(len(first_frames), len(second_frames)) + for first_frame, second_frame in zip( + first_frames, second_frames, strict=True + ): + self.assertIs(first_frame, second_frame) + + def test_writer_repeat_run_spans_interleaved_streams(self): + """A thread's repeat run continues across other threads' samples.""" + filename = self.temp_filename() + idle = [make_frame("idle.py", 1, "wait")] + busy = [[make_frame("busy.py", 1, "even")], + [make_frame("busy.py", 2, "odd")]] + writer = _remote_debugging.BinaryWriter(filename, 1000, 0) + for i in range(10): + sample = [make_interpreter(0, [ + make_thread(1, idle), + make_thread(2, busy[i % 2]), + ])] + writer.write_sample(sample, 1000 * (i + 1)) + stats = writer.get_stats() + self.assertEqual(stats["repeat_records"], 1) + self.assertEqual(stats["repeat_samples"], 9) + writer.finalize() + + collector = RawCollector() + with BinaryReader(filename) as reader: + self.assertEqual(reader.replay_samples(collector), 20) + self.assertEqual(reader.get_stats()["repeat_samples"], 9) + idle_frames = [frame_to_dict(frame) for frame in idle] + self.assertEqual( + [sample["frames"] for sample in collector.by_thread[(0, 1)]], + [idle_frames] * 10) + self.assertEqual( + [sample["frames"][0]["funcname"] + for sample in collector.by_thread[(0, 2)]], + ["even", "odd"] * 5) + def test_invalid_path_error_preserves_pathlib(self): """Missing path: OSError carries the original path object, not a string.""" missing = pathlib.Path("/i/do/not/exist") @@ -907,13 +1268,9 @@ def test_invalid_path_error_preserves_pathlib(self): def test_writer_handles_empty_stack_first_sample(self): """BinaryWriter.write_sample tolerates an empty stack on a fresh thread. - Regression test for the C-level RLE bug in process_thread_sample: a - freshly-created ThreadEntry has prev_stack_depth == 0, so an empty - curr_stack compares as STACK_REPEAT against the zero-initialized - previous stack. Before the fix, this fell through the - `&& !is_new_thread` guard into write_sample_with_encoding, which had - no handler for STACK_REPEAT and raised - RuntimeError("Invalid stack encoding type"). Goes through + Regression test for gh-149342: an empty first stack equals the empty + stack a new thread starts from, but it must still be written as a + stack (STACK_REPLACE_SUFFIX 0 0) rather than a repeat. Goes through BinaryWriter.write_sample directly so the test cannot be masked by any Python-level filtering. """ @@ -931,8 +1288,7 @@ def test_writer_handles_empty_stack_first_sample(self): # scenario that exposes the bug. writer.write_sample(empty_sample, 1000) writer.write_sample(empty_sample, 2000) - # Mix in a real sample to exercise the transition out of the - # empty-stack RLE buffer. + # Mix in a real sample after the repeated empty stack. real_sample = [ make_interpreter(0, [make_thread(1, [make_frame("a.py", 1, "f")])]) ] @@ -942,8 +1298,8 @@ def test_writer_handles_empty_stack_first_sample(self): reader_collector = RawCollector() with BinaryReader(filename) as reader: count = reader.replay_samples(reader_collector) - # Empty-stack samples are recorded as STACK_REPEAT records with - # depth-0 stacks; the file must replay all three samples. + # The empty stack is written once and then repeated; the file must + # replay all three samples. self.assertEqual(count, 3) def test_writer_handles_mixed_empty_and_real_first_sample(self): @@ -977,11 +1333,9 @@ def test_writer_handles_mixed_empty_and_real_first_sample(self): def test_writer_total_samples_after_finalize_matches_reader(self): """BinaryWriter.total_samples after finalize() matches the reader's count.""" - # Five IDENTICAL samples force every sample beyond the first into the - # per-thread RLE buffer. Regression for the cached_total_samples - # ordering bug: capturing the cache BEFORE binary_writer_finalize() - # missed the buffered samples that flush_pending_rle() counts. Keep - # the samples identical to preserve coverage. See gh-149342. + # Five IDENTICAL samples stay in the open block, as one repeat, until + # finalize() flushes it; total_samples must still count all of them. + # Keep the samples identical to preserve coverage. See gh-149342. samples = [ [make_interpreter(0, [make_thread(1, [make_frame("a.py", 1, "f")])])] ] * 5 @@ -992,8 +1346,8 @@ def test_writer_total_samples_after_finalize_matches_reader(self): self.assertEqual(writer_collector.total_samples, len(samples)) self.assertEqual(writer_collector.total_samples, replayed) - def test_rle_buffer_flushes_before_finalize(self): - """RLE buffer flushes before the writer is destroyed during finalize().""" + def test_open_block_flushes_on_finalize(self): + """finalize() writes the open block before the writer is destroyed.""" with tempfile.NamedTemporaryFile(suffix=".bin", delete=False) as f: filename = f.name self.temp_files.append(filename) @@ -1194,11 +1548,10 @@ def test_finalize_failure_breaks_writer(self): class TestBinaryFormatValidation(BinaryFormatTestBase): """Tests for malformed binary files.""" + HDR_OFF_VERSION = 4 HDR_OFF_SAMPLES = 28 HDR_OFF_THREADS = 36 - HDR_OFF_STR_TABLE = 40 HDR_OFF_FRAME_TABLE = 48 - FILE_HEADER_PLACEHOLDER_SIZE = 64 FILE_FOOTER_SIZE = 32 FTR_OFF_STRINGS = 0 FTR_OFF_FRAMES = 4 @@ -1261,7 +1614,8 @@ def test_replay_rejects_trailing_partial_sample_header(self): with BinaryReader(filename) as reader: with self.assertRaises(ValueError) as cm: reader.replay_samples(RawCollector()) - self.assertEqual(str(cm.exception), "Truncated sample data: 1 trailing bytes") + self.assertEqual(str(cm.exception), + "Invalid sample block: truncated header") # Minimum on-disk size of one table entry (see binary_io.h). MIN_STRING_ENTRY_SIZE = 1 @@ -1345,6 +1699,21 @@ def test_open_accepts_frame_count_at_capacity_boundary(self): f"possible {max_frames}", ) + def test_open_rejects_older_format_version(self): + """Open rejects version 1 files and reports version 2.""" + filename = self.create_binary_file([], compression="none") + with BinaryReader(filename) as reader: + self.assertEqual(reader.get_info()["version"], 2) + with open(filename, "r+b") as raw: + raw.seek(self.HDR_OFF_VERSION) + raw.write(struct.pack("=I", 1)) + with self.assertRaisesRegex( + ValueError, + r"Unsupported format version 1 \(this reader supports " + r"version 2\)"): + with BinaryReader(filename): + pass + def test_sample_count_reads_full_64_bits(self): """sample_count values requiring the upper 32 bits decode correctly.""" filename = self.create_binary_file([], compression="none") @@ -1370,12 +1739,331 @@ def test_sample_count_boundary_values(self): with BinaryReader(filename) as reader: self.assertEqual(reader.get_info()["sample_count"], value) + # Block column indices and encodings (see binary_io.h). + MAX_BLOCK_SIZE = 1024 * 1024 + + def _write_single_block(self, num_samples=1, instructions=b"\x01\x00\x00", + frame_ids=b"", *, header_samples=None, + file_samples=None, stream_count=1, + stream_table=None, stream_ids=None, + residuals=None, statuses=b"", status_flags=(0, 0), + frame_width=1, descriptors=None, magic=b"TBLK", + truncate=None, frame_count=1): + """Write a file whose only block is built from the given parts. + + By default every sample belongs to stream 0, has status 0 and is + exactly one sample interval after the previous one, and the only + stack is empty (STACK_REPLACE_SUFFIX 0 0). The frame table holds + *frame_count* frames: frame 0 is in "a.py" and frame N in "N.py". + *descriptors* maps a column index to (encoding, decoded size) to + replace its raw descriptor. + """ + frames = [make_frame("a.py", 1, "f")] + frames += [make_frame(f"{index}.py", 1, "f") + for index in range(1, frame_count)] + samples = [[make_interpreter(0, [make_thread(1, frames[i:i + 256])])] + for i in range(0, frame_count, 256)] + filename = self.create_binary_file(samples, compression="none") + data = bytearray(pathlib.Path(filename).read_bytes()) + sample_end = struct.unpack_from("=Q", data, self.HDR_OFF_STR_TABLE)[0] + if stream_table is None: + stream_table = b"".join(struct.pack("=QI", thread, 0) + for thread in range(1, stream_count + 1)) + columns = ( + stream_table, + b"\x00" * num_samples if stream_ids is None else stream_ids, + b"\x00" * num_samples if residuals is None else residuals, + statuses, + instructions, + frame_ids, + ) + if header_samples is None: + header_samples = num_samples + block = (magic + encode_varint(header_samples) + + b"\x01" # base timestamp + + bytes([frame_width, *status_flags])) + payload = b"" + for index, column in enumerate(columns): + encoding, decoded_size = (descriptors or {}).get( + index, (0, len(column))) + block += encode_varint(decoded_size) + bytes([encoding]) + if encoding == self.COLUMN_ZSTD: + block += encode_varint(len(column)) + if encoding == self.COLUMN_CONSTANT: + column = column[:1] + payload += column + block = (block + payload)[:truncate] + data[self.FILE_HEADER_PLACEHOLDER_SIZE:sample_end] = block + shift = len(block) - (sample_end - self.FILE_HEADER_PLACEHOLDER_SIZE) + if file_samples is None: + file_samples = num_samples + struct.pack_into("=Q", data, self.HDR_OFF_SAMPLES, file_samples) + struct.pack_into("=I", data, self.HDR_OFF_THREADS, + max(len(stream_table) // 12, 1)) + for offset in (self.HDR_OFF_STR_TABLE, self.HDR_OFF_FRAME_TABLE): + old_offset = struct.unpack_from("=Q", data, offset)[0] + struct.pack_into("=Q", data, offset, old_offset + shift) + struct.pack_into("=Q", data, len(data) - 24, len(data)) + pathlib.Path(filename).write_bytes(data) + return filename + + def assert_block_rejected(self, filename, message): + with BinaryReader(filename) as reader: + with self.assertRaises(ValueError) as cm: + reader.replay_samples(RawCollector()) + self.assertEqual(str(cm.exception), f"Invalid sample block: {message}") + + def test_replay_accepts_hand_built_block(self): + """Replay accepts a block built by hand from its columns.""" + # STACK_REPLACE_SUFFIX 0 0, then STACK_REPEAT 2. + filename = self._write_single_block(3, b"\x01\x00\x00\x00\x02") + collector = RawCollector() + with BinaryReader(filename) as reader: + self.assertEqual(reader.replay_samples(collector), 3) + self.assertEqual(len(collector.by_thread[(0, 1)]), 3) + + def test_replay_decodes_every_frame_id_width(self): + """Frame IDs decode correctly at every byte width.""" + # STACK_REPLACE_SUFFIX 0 1 reads one frame ID, stored in one byte + # plane per byte of the width, low byte first. + for width, frame_id in ((1, 43), (2, 299), (3, 299), (4, 299)): + with self.subTest(width=width): + filename = self._write_single_block( + 1, b"\x01\x00\x01", frame_id.to_bytes(width, "little"), + frame_width=width, frame_count=300) + collector = RawCollector() + with BinaryReader(filename) as reader: + self.assertEqual(reader.replay_samples(collector), 1) + frames = collector.by_thread[(0, 1)][0]["frames"] + self.assertEqual([frame["filename"] for frame in frames], + [f"{frame_id}.py"]) + + def test_replay_reads_constant_stream_and_residual_columns(self): + """Constant stream index and residual columns decode every sample.""" + # Every sample is in stream 0 and one microsecond late (zigzag 2). + filename = self._write_single_block( + 3, b"\x01\x00\x00\x00\x02", residuals=b"\x02" * 3, + descriptors={self.COLUMN_STREAM_IDS: (self.COLUMN_CONSTANT, 3), + self.COLUMN_TIMESTAMP_RESIDUALS: + (self.COLUMN_CONSTANT, 3)}) + collector = mock.Mock() + with BinaryReader(filename) as reader: + interval = reader.get_info()["sample_interval_us"] + self.assertEqual(reader.replay_samples(collector), 3) + timestamps = [ts for call in collector.collect.call_args_list + for ts in call.args[1]] + self.assertEqual(timestamps, + [2 + i * (interval + 1) for i in range(3)]) + + def test_replay_rejects_unterminated_constant_varint(self): + """A constant byte that never ends a varint is rejected.""" + for column, part in ((self.COLUMN_STREAM_IDS, "stream_ids"), + (self.COLUMN_TIMESTAMP_RESIDUALS, "residuals")): + with self.subTest(part): + filename = self._write_single_block( + **{part: b"\x80"}, + descriptors={column: (self.COLUMN_CONSTANT, 1)}) + with BinaryReader(filename) as reader: + with self.assertRaisesRegex(ValueError, + "incomplete varint"): + reader.replay_samples(RawCollector()) + + def test_replay_rejects_reentrant_replay(self): + """Replaying from inside the same reader's collector raises.""" + filename = self._write_single_block( + 2, b"\x01\x00\x00\x00\x01") # REPEAT 1 + + with BinaryReader(filename) as reader: + class ReentrantCollector: + def collect(self, stack_frames, timestamps_us): + reader.replay_samples(RawCollector()) + + with self.assertRaisesRegex(RuntimeError, "already replaying"): + reader.replay_samples(ReentrantCollector()) + self.assertEqual(reader.replay_samples(RawCollector()), 2) + + def test_replay_rejects_close_during_replay(self): + """Closing the reader from inside its collector raises.""" + filename = self._write_single_block( + 2, b"\x01\x00\x00\x00\x01") # REPEAT 1 + closers = { + "close": lambda reader: reader._reader.close(), + "__exit__": lambda reader: reader._reader.__exit__( + None, None, None), + "__init__": lambda reader: reader._reader.__init__(filename), + } + for name, close in closers.items(): + with self.subTest(name): + with BinaryReader(filename) as reader: + class ClosingCollector: + def collect(self, stack_frames, timestamps_us): + close(reader) + + with self.assertRaisesRegex(RuntimeError, "replaying"): + reader.replay_samples(ClosingCollector()) + self.assertEqual(reader.replay_samples(RawCollector()), 2) + + def test_replay_rejects_close_from_initial_progress_callback(self): + """Closing the reader from its initial progress callback raises.""" + filename = self._write_single_block( + 2, b"\x01\x00\x00\x00\x01") # REPEAT 1 + + with BinaryReader(filename) as reader: + def close_reader(current, total): + reader._reader.close() + + with self.assertRaisesRegex(RuntimeError, "replaying"): + reader.replay_samples(RawCollector(), close_reader) + self.assertEqual(reader.replay_samples(RawCollector()), 2) + + def test_replay_rejects_invalid_block(self): + """Replay rejects every kind of malformed block.""" + frames = self.COLUMN_FRAME_IDS + cases = [ + ("wrong magic", dict(magic=b"XBLK")), + # Magic, sample count and base timestamp, then 1 of 3 flag bytes + ("truncated header", dict(truncate=7)), + ("no samples", dict(header_samples=0)), + ("frame ID width out of range", dict(frame_width=0)), + ("frame ID width out of range", dict(frame_width=5)), + ("common and varying status flags overlap", + dict(status_flags=(1, 1))), + # Magic, sample count, base timestamp, 3 flag bytes and the + # first decoded size + ("truncated column descriptor", dict(truncate=10)), + ("unknown column encoding", dict(descriptors={frames: (3, 0)})), + ("zstd column in an uncompressed file", + dict(descriptors={self.COLUMN_STACK_INSTRUCTIONS: + (self.COLUMN_ZSTD, 3)})), + ("encoded column is empty", + dict(descriptors={frames: (self.COLUMN_CONSTANT, 0)})), + ("decoded columns exceed the block limit", + dict(frame_ids=b"\x00", + descriptors={frames: (self.COLUMN_CONSTANT, + self.MAX_BLOCK_SIZE)})), + # Rejected before anything is allocated for the samples + ("more samples than stored stream indices", + dict(header_samples=0xFFFFFFFF)), + ("column extends past the sample section", dict(truncate=-1)), + ("stream table has a partial entry", dict(stream_table=b"0" * 11)), + ("stream count out of range", dict(stream_table=b"")), + ("frame ID column has a partial value", + dict(frame_width=2, frame_ids=b"\x00")), + ("status bitmaps have the wrong size", dict(status_flags=(0, 1))), + ("status bitmap padding is not zero", + dict(status_flags=(0, 1), statuses=b"\x02")), + ("stream table repeats a thread", + dict(stream_table=struct.pack("=QI", 1, 0) * 2)), + ("sample refers to an unknown stream", dict(stream_ids=b"\x01")), + ("sample refers to an unknown stream", + dict(num_samples=2, stream_ids=b"\x01", + descriptors={self.COLUMN_STREAM_IDS: + (self.COLUMN_CONSTANT, 2)})), + ("stream index column has trailing data", + dict(num_samples=2, header_samples=1)), + # A constant column longer than the sample count is expanded + # and then rejected like any other trailing data + ("stream index column has trailing data", + dict(descriptors={self.COLUMN_STREAM_IDS: + (self.COLUMN_CONSTANT, 2)})), + ("timestamp column has trailing data", + dict(residuals=b"\x00\x00")), + ("timestamp column has trailing data", + dict(descriptors={self.COLUMN_TIMESTAMP_RESIDUALS: + (self.COLUMN_CONSTANT, 2)})), + ("frame index out of range", + dict(instructions=b"\x01\x00\x01", frame_ids=b"\x05")), + # Frame IDs 65536 and 2**24 only have a byte in the third or + # fourth plane + ("frame index out of range", + dict(instructions=b"\x01\x00\x01", frame_width=3, + frame_ids=(1 << 16).to_bytes(3, "little"), + frame_count=300)), + ("frame index out of range", + dict(instructions=b"\x01\x00\x01", frame_width=4, + frame_ids=(1 << 24).to_bytes(4, "little"), + frame_count=300)), + ("unknown stack instruction", dict(instructions=b"\x03")), + ("first stack of a stream cannot repeat", + dict(instructions=b"\x00\x01")), + ("repeat count out of range", + dict(num_samples=2, instructions=b"\x01\x00\x00\x00\x00")), + ("stack prefix out of range", dict(instructions=b"\x01\x01\x00")), + ("stack uses more frame IDs than stored", + dict(instructions=b"\x01\x00\x02", frame_ids=b"\x00")), + # STACK_REUSE 0 257 + ("stack is too deep", dict(instructions=b"\x02\x00\x81\x02")), + ("reused stack is not stored earlier", + dict(instructions=b"\x02\x00\x01", frame_ids=b"\x00")), + ("stack instruction column has trailing data", + dict(instructions=b"\x01\x00\x00\x01")), + ("frame ID column has unused values", dict(frame_ids=b"\x00")), + ("stream table has an unused entry", dict(stream_count=2)), + ("repeat count extends past the block", + dict(num_samples=2, instructions=b"\x01\x00\x00\x00\x02")), + ("more samples than the file header declares", + dict(file_samples=0)), + ] + for message, kwargs in cases: + with self.subTest(message, **kwargs): + filename = self._write_single_block(**kwargs) + self.assert_block_rejected(filename, message) + + @unittest.skipUnless(ZSTD_AVAILABLE, "zstd compression not available") + def test_replay_rejects_invalid_zstd_column(self): + """Replay rejects corrupt and mismatched zstd columns.""" + zstd = import_helper.import_module("compression.zstd") + instructions = b"\x01\x00\x00" + frame = zstd.compress( + instructions, + options={zstd.CompressionParameter.checksum_flag: 1}) + bad_checksum = frame[:-1] + bytes([frame[-1] ^ 1]) + cases = [ + ("Corrupted zstd frame header", instructions, len(instructions)), + ("zstd frame size does not match the decoded size", + frame, len(instructions) + 1), + ("zstd decompression error: ", bad_checksum, len(instructions)), + ] + for message, stored, decoded_size in cases: + with self.subTest(message): + filename = self._write_single_block( + instructions=stored, + descriptors={self.COLUMN_STACK_INSTRUCTIONS: + (self.COLUMN_ZSTD, decoded_size)}) + with open(filename, "r+b") as raw: + raw.seek(56) # compression type + raw.write(struct.pack("=I", 1)) + with BinaryReader(filename) as reader: + with self.assertRaises(ValueError) as cm: + reader.replay_samples(RawCollector()) + self.assertStartsWith(str(cm.exception), message) + + def test_replay_propagates_callback_errors(self): + """Errors raised by the collector or progress callback propagate.""" + filename = self._write_single_block( + 2000, b"\x01\x00\x00\x00\xcf\x0f") # REPEAT 1999 + + class FailingCollector: + def collect(self, stack_frames, timestamps_us): + raise ZeroDivisionError + + def failing_progress(current, total): + if current: + raise ZeroDivisionError + + with BinaryReader(filename) as reader: + with self.assertRaises(ZeroDivisionError): + reader.replay_samples(FailingCollector()) + with self.assertRaises(ZeroDivisionError): + reader.replay_samples(RawCollector(), failing_progress) + self.assertEqual(reader.replay_samples(RawCollector()), 2000) + class TestBinaryEncodings(BinaryFormatTestBase): """Tests specifically targeting different stack encodings.""" - def test_stack_full_encoding(self): - """First sample uses STACK_FULL encoding and roundtrips.""" + def test_stack_first_sample_pattern(self): + """First sample stores its complete stack and roundtrips.""" frames = [make_frame(f"f{i}.py", i + 1, f"func{i}") for i in range(5)] samples = [[make_interpreter(0, [make_thread(1, frames)])]] collector, count = self.roundtrip(samples) @@ -1393,8 +2081,8 @@ def test_stack_repeat_encoding(self): self.assertEqual(count, 1000) self.assert_samples_equal(samples, collector) - def test_stack_suffix_encoding(self): - """Samples sharing suffix use STACK_SUFFIX and roundtrip.""" + def test_stack_shared_outer_frames_pattern(self): + """Samples keeping outer frames roundtrip (STACK_REPLACE_SUFFIX).""" samples = [] for i in range(10): frames = [make_frame(f"new{i}.py", i + 1, f"new{i}")] @@ -1409,8 +2097,8 @@ def test_stack_suffix_encoding(self): self.assertEqual(count, 10) self.assert_samples_equal(samples, collector) - def test_stack_pop_push_encoding(self): - """Samples with pop+push pattern roundtrip.""" + def test_stack_pop_push_pattern(self): + """Samples that pop frames and push new ones roundtrip.""" samples = [] base_frames = [make_frame("base.py", 10, "base")] @@ -1549,15 +2237,29 @@ def export(self, filename): def test_rle_long_run_splits_batches(self): - # Construct a single repeat record larger than the writer's buffer. - num_samples = 8193 + # Construct one block whose repeat run is longer than a replay batch. + num_samples = 8194 filename = self.create_binary_file([], compression="none") data = bytearray(pathlib.Path(filename).read_bytes()) - record = (struct.pack("=QIB", 1, 0, 0) # STACK_REPEAT - + b"\x81\x40" # 8193 as a varint - + b"\x01\x00" * num_samples) # delta=1, status=0 + instructions = (b"\x01\x00\x00" # STACK_REPLACE_SUFFIX 0 0 + + b"\x00\x81\x40") # STACK_REPEAT 8193 + record = (b"TBLK" + + b"\x82\x40" # 8194 samples as a varint + + b"\x01" # base timestamp + + b"\x01\x00\x00" # frame width, status flags + + b"\x0c\x00" # stream table: 12 raw bytes + + b"\x82\x40\x02" # stream indices: constant + + b"\x82\x40\x02" # timestamp residuals: constant + + b"\x00\x00" # status bitmaps: empty + + bytes([len(instructions), 0]) # raw instructions + + b"\x00\x00" # frame IDs: empty + + struct.pack("=QI", 1, 0) + + b"\x00" # every sample is in stream 0 + + b"\x00" # every residual is zero + + instructions) data[64:64] = record struct.pack_into("=Q", data, 12, 0) # start timestamp + struct.pack_into("=Q", data, 20, 1) # sample interval struct.pack_into("=Q", data, 28, num_samples) struct.pack_into("=I", data, 36, 1) # thread count for offset in (40, 48): # string and frame table offsets @@ -1571,7 +2273,7 @@ def test_rle_long_run_splits_batches(self): count = reader.replay_samples(collector) batches = [call.args[1] for call in collector.collect.call_args_list] self.assertEqual(count, num_samples) - self.assertEqual([len(batch) for batch in batches], [8192, 1]) + self.assertEqual([len(batch) for batch in batches], [1, 8192, 1]) self.assertEqual([ts for batch in batches for ts in batch], list(range(1, num_samples + 1))) @@ -1701,7 +2403,7 @@ def test_encoding_transitions_stress(self): # Repeat previous (will use RLE if identical) frames = base_frames[: random.randint(1, 5)] elif choice == 2: - # Add frames on top (suffix encoding) + # Add frames on top (keeps the outer frames) extra = random.randint(1, 3) frames = [ make_frame(f"top{i}_{j}.py", j, f"top{j}") @@ -1709,7 +2411,7 @@ def test_encoding_transitions_stress(self): ] frames.extend(base_frames[: random.randint(2, 4)]) else: - # Pop and push (pop-push encoding) + # Pop some frames and push new ones keep = random.randint(1, 3) push = random.randint(0, 2) frames = [ @@ -1893,6 +2595,25 @@ def test_replay_includes_persisted_stats(self): self.assertIn('"error_rate": 2.5', content) self.assertIn('"missed_samples": 1.5', content) + @unittest.skipUnless(ZSTD_AVAILABLE, "zstd compression not available") + def test_measured_stats_with_zstd(self): + """Measured statistics are read back when zstd is enabled.""" + frames = [make_frame(f"pkg/module_{i:03}.py", i + 1, f"func_{i:03}") + for i in range(200)] + sample = [make_interpreter(0, [make_thread(1, frames)])] + filename = self.temp_filename() + collector = BinaryCollector(filename, 2000, compression="zstd") + collector.collect(sample, timestamp_us=1000) + collector.set_stats(2000, 1.25, 4.0, error_rate=2.5, + missed_samples=1.5) + collector.export(None) + + with BinaryReader(filename) as reader: + info = reader.get_info() + self.assertEqual(info["duration_sec"], 1.25) + self.assertEqual(info["missed_samples"], 1.5) + self.assertEqual(reader.replay_samples(RawCollector()), 1) + def test_legacy_binary_has_no_measured_stats(self): frame = make_frame("hot.py", 99, "hot_func") bin_path = self.create_binary_file([ diff --git a/Lib/test/test_profiling/test_sampling_profiler/test_cli.py b/Lib/test/test_profiling/test_sampling_profiler/test_cli.py index e5dcd72c27ef5c2..789cc3408d5a856 100644 --- a/Lib/test/test_profiling/test_sampling_profiler/test_cli.py +++ b/Lib/test/test_profiling/test_sampling_profiler/test_cli.py @@ -954,7 +954,7 @@ def test_cli_replay_reader_errors_exit_cleanly(self): mock.patch("sys.argv", ["profiling.sampling.cli", "replay", profile]), mock.patch( "profiling.sampling.cli.BinaryReader", - side_effect=ValueError("Unsupported format version 2"), + side_effect=ValueError("Unsupported format version 3"), ), ): with self.assertRaises(SystemExit) as cm: @@ -962,7 +962,7 @@ def test_cli_replay_reader_errors_exit_cleanly(self): self.assertEqual( str(cm.exception), - "Error: Unsupported format version 2", + "Error: Unsupported format version 3", ) def test_cli_replay_propagates_recorded_mode(self): diff --git a/Lib/test/test_profiling/test_sampling_profiler/test_collectors.py b/Lib/test/test_profiling/test_sampling_profiler/test_collectors.py index d4ba2df5044559f..dcac6f3e7dfd9e7 100644 --- a/Lib/test/test_profiling/test_sampling_profiler/test_collectors.py +++ b/Lib/test/test_profiling/test_sampling_profiler/test_collectors.py @@ -2957,20 +2957,6 @@ def test_extract_lineno_from_none(self): """Test extracting lineno from None (synthetic frames).""" self.assertEqual(extract_lineno(None), 0) - def test_extract_lineno_from_int(self): - """Test extracting lineno from a bare integer line number. - - Mirrors normalize_location's int contract so callers like the - collapsed/flamegraph collectors do not crash on a bare-int location. - """ - self.assertEqual(extract_lineno(42), 42) - self.assertEqual(extract_lineno(0), 0) - - def test_normalize_location_with_int(self): - """Test normalize_location expands a legacy integer line number.""" - result = normalize_location(42) - self.assertEqual(result, (42, 42, -1, -1)) - def test_normalize_location_with_location_info(self): """Test normalize_location passes through LocationInfo.""" loc = LocationInfo(10, 15, 0, 5) diff --git a/Lib/test/test_readline.py b/Lib/test/test_readline.py index f999ea6cc1affdd..074faf94d2d792d 100644 --- a/Lib/test/test_readline.py +++ b/Lib/test/test_readline.py @@ -471,7 +471,7 @@ def test_write_read_limited_history(self): # So, we've only tested that the read did not fail. # See TestHistoryManipulation for the full test. - @unittest.skipUnless(sysconfig.get_config_var("HAVE_RL_CHANGE_ENVIRONMENT"), + @unittest.skipUnless(sysconfig.get_config_var("_Py_HAVE_RL_CHANGE_ENVIRONMENT"), "readline can modify the environment") def test_environment_is_not_modified(self): # os.environ contains environment at the time "os" module was loaded, so diff --git a/Lib/test/test_shutil.py b/Lib/test/test_shutil.py index 1104552215a44f1..47221d1bf46fa18 100644 --- a/Lib/test/test_shutil.py +++ b/Lib/test/test_shutil.py @@ -566,6 +566,63 @@ def onexc(*args): self.assertTrue(onexc_called) self.assertFalse(onerror_called) + @unittest.skipUnless(shutil.rmtree.avoids_symlink_attacks, + 'requires the fd based implementation of rmtree()') + def test_onexc_kwargs_dir_fd(self): + os.mkdir(TESTFN) + self.addCleanup(os_helper.rmtree, TESTFN) + top_name = os.path.join(TESTFN, 'top') + child_name = os.path.join(TESTFN, 'top', 'child') + os.mkdir(top_name) + os.mkdir(child_name) + parent_fd = os.open(TESTFN, os.O_RDONLY) + self.addCleanup(os.close, parent_fd) + + errors = [] + def onexc(func, path, err, direntry=None, dir_fd=None): + errors.append((func, path.replace(os.path.sep, '/'))) + name = path if direntry is None else direntry.name + self.assertEqual(name, os.path.basename(path)) + self.assertTrue(os.path.samestat( + os.stat(os.path.join(TESTFN, path)), + os.stat(name, dir_fd=dir_fd))) + + def failing_rmdir(path, *, dir_fd=None): + raise PermissionError(errno.EACCES, 'simulated') + with unittest.mock.patch('os.rmdir', failing_rmdir): + shutil.rmtree('top', dir_fd=parent_fd, onexc=onexc, + _onexc_kwargs=True) + self.assertEqual( + errors, + [(failing_rmdir, 'top/child'), (failing_rmdir, 'top')]) + + child_stat = os.stat(child_name) + scandir = os.scandir + + # Fail for the top-level directory + errors.clear() + def failing_scandir(fd): + raise PermissionError(errno.EACCES, 'simulated') + with unittest.mock.patch('os.scandir', failing_scandir): + shutil.rmtree('top', dir_fd=parent_fd, onexc=onexc, + _onexc_kwargs=True) + self.assertEqual( + errors, + [(failing_scandir, 'top'), (os.rmdir, 'top')]) + self.assertTrue(os.path.exists(top_name)) + + # Fail for the child (which is empty, so the rmdir succeeds) + errors.clear() + def failing_scandir_2(fd): + if not os.path.samestat(os.fstat(fd), child_stat): + return scandir(fd) + raise PermissionError(errno.EACCES, 'simulated') + with unittest.mock.patch('os.scandir', failing_scandir_2): + shutil.rmtree('top', dir_fd=parent_fd, onexc=onexc, + _onexc_kwargs=True) + self.assertEqual(errors, [(failing_scandir_2, 'top/child')]) + self.assertFalse(os.path.exists(top_name)) + def test_rmtree_does_not_choke_on_failing_lstat(self): try: orig_lstat = os.lstat diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py index 2f7b8798daa8c95..3e87ccfc436d121 100644 --- a/Lib/test/test_tempfile.py +++ b/Lib/test/test_tempfile.py @@ -1653,10 +1653,14 @@ def __exit__(self, *exc_info): class TestTemporaryDirectory(BaseTestCase): """Test TemporaryDirectory().""" - def do_create(self, dir=None, pre="", suf="", recurse=1, dirs=1, files=1, + def do_create(self, dir=None, pre=None, suf=None, recurse=1, dirs=1, files=1, ignore_cleanup_errors=False): if dir is None: dir = tempfile.gettempdir() + if pre is None: + pre = type(dir)() + if suf is None: + suf = type(dir)() tmp = tempfile.TemporaryDirectory( dir=dir, prefix=pre, suffix=suf, ignore_cleanup_errors=ignore_cleanup_errors) @@ -1666,13 +1670,17 @@ def do_create(self, dir=None, pre="", suf="", recurse=1, dirs=1, files=1, def do_create2(self, path, recurse=1, dirs=1, files=1): # Create subdirectories and some files + def join(name): + if isinstance(path, bytes): + name = os.fsencode(name) + return os.path.join(path, name) if recurse: for i in range(dirs): - name = os.path.join(path, "dir%d" % i) + name = join("dir%d" % i) os.mkdir(name) self.do_create2(name, recurse-1, dirs, files) for i in range(files): - with open(os.path.join(path, "test%d.txt" % i), "wb") as f: + with open(join("test%d.txt" % i), "wb") as f: f.write(b"Hello world!") def test_mkdtemp_failure(self): @@ -1698,6 +1706,15 @@ def test_explicit_cleanup(self): finally: os.rmdir(dir) + def test_explicit_cleanup_bytes(self): + # A TemporaryDirectory with a bytes name is deleted when cleaned up + with tempfile.TemporaryDirectory() as working_dir: + d = self.do_create(dir=os.fsencode(working_dir), recurse=2) + self.assertIsInstance(d.name, bytes) + self.assertTrue(os.path.exists(d.name)) + d.cleanup() + self.assertFalse(os.path.exists(d.name)) + def test_explicit_cleanup_ignore_errors(self): """Test that cleanup doesn't return an error when ignoring them.""" with tempfile.TemporaryDirectory() as working_dir: @@ -1910,6 +1927,87 @@ def hook(path, *, dir_fd=None): self.assertTrue(os.path.exists(target_file)) self.assertEqual(os.stat(target_file).st_mode, target_mode) + def _make_unopenable_subdir(self): + # Create the following structure: + # work/ - top-level + # / - tempdir to clean up (random name) + # sub/ - an unopenable dir + # subsub/ + # file + # Return: + # - `work` name (str) + # - `d` (TemporaryDirectory) + # - `subsub` stat result + work = self.enterContext(os_helper.temp_dir()) + d = tempfile.TemporaryDirectory(dir=work) + sub = os.path.join(d.name, 'sub') + subsub = os.path.join(sub, 'subsub') + os.mkdir(sub) + os.mkdir(subsub) + subsub_stat = os.lstat(subsub) + with open(os.path.join(subsub, 'file'), 'w') as f: + f.write("rm me pls") + os.chmod(sub, 0) + # If the test fails, make sure everything can still be removed. + self.addCleanup(d.cleanup) + @self.addCleanup + def _clean(): + if os.path.isdir(sub): + os.chmod(sub, 0o700) + return work, d, subsub_stat + + @os_helper.skip_unless_working_chmod + @support.requires_non_root_user + @unittest.skipUnless(tempfile._rmtree_use_dir_fd + and shutil.rmtree.avoids_symlink_attacks, + 'requires the fd based cleanup') + def test_cleanup_nested_toplevel_error(self): + # rmtree() reports errors relative to the correct parent directory + work, d, subsub_stat = self._make_unopenable_subdir() + # An unrelated directory with the same name; should not be touched + unrelated = os.path.join(work, 'sub') + os.mkdir(unrelated) + + rmdir = os.rmdir + failed_path = None + def failing_rmdir(path, *, dir_fd=None): + nonlocal failed_path + if path == 'sub' and dir_fd is not None and failed_path is None: + failed_path = path + raise PermissionError(errno.EPERM, 'simulated', path) + return rmdir(path, dir_fd=dir_fd) + with os_helper.change_cwd(work), mock.patch('os.rmdir', failing_rmdir): + d.cleanup() + self.assertEqual(failed_path, 'sub') + self.assertFalse(os.path.exists(d.name)) + self.assertTrue(os.path.exists(unrelated)) + + @os_helper.skip_unless_working_chmod + @support.requires_non_root_user + @unittest.skipUnless(tempfile._rmtree_use_dir_fd + and shutil.rmtree.avoids_symlink_attacks, + 'requires the fd based cleanup') + def test_cleanup_nested_scandir_error(self): + # A scandir() error is reported relative to the parent directory. + work, d, subsub_stat = self._make_unopenable_subdir() + + scandir = os.scandir + failed = False + def failing_scandir(path='.'): + nonlocal failed + if ( + not failed + and isinstance(path, int) + and os.path.samestat(os.fstat(path), subsub_stat) + ): + failed = True + raise PermissionError(errno.EACCES, 'simulated') + return scandir(path) + with os_helper.change_cwd(work), mock.patch('os.scandir', failing_scandir): + d.cleanup() + self.assertTrue(failed) + self.assertFalse(os.path.exists(d.name)) + @support.cpython_only def test_del_on_collection(self): # A TemporaryDirectory is deleted when garbage collected @@ -2068,11 +2166,17 @@ def test_context_manager(self): self.assertEqual(name, d.name) self.assertFalse(os.path.exists(name)) - def test_modes(self): + @support.subTests('bytes_name', (False, True)) + def test_modes(self, bytes_name): for mode in range(8): mode <<= 6 with self.subTest(mode=format(mode, '03o')): - d = self.do_create(recurse=3, dirs=2, files=2) + if bytes_name: + d = self.do_create(recurse=3, dirs=2, files=2, + dir=tempfile.gettempdirb()) + else: + d = self.do_create(recurse=3, dirs=2, files=2) + self.assertIsInstance(d.name, bytes if bytes_name else str) with d: # Change files and directories mode recursively. for root, dirs, files in os.walk(d.name, topdown=False): @@ -2083,13 +2187,17 @@ def test_modes(self): self.assertFalse(os.path.exists(d.name)) @support.subTests('ignore_errors', (True, False)) - def test_parent_mode_preserved(self, ignore_errors): + @support.subTests('bytes_name', (False, True)) + def test_parent_mode_preserved(self, ignore_errors, bytes_name): # Test that cleanup does not touch the parent directory, # even if that prevents removal. for mode in range(8): mode <<= 6 with self.subTest(mode=format(mode, '03o')): - outer = self.do_create() + if bytes_name: + outer = self.do_create(dir=tempfile.gettempdirb()) + else: + outer = self.do_create() with outer: d = self.do_create(dir=outer.name, dirs=2, files=2, ignore_cleanup_errors=ignore_errors) diff --git a/Lib/test/test_type_annotations.py b/Lib/test/test_type_annotations.py index 879a571d06b4a03..02f3e8b9caddb4c 100644 --- a/Lib/test/test_type_annotations.py +++ b/Lib/test/test_type_annotations.py @@ -153,7 +153,11 @@ class D(metaclass=C): self.assertEqual(D.__annotations__, {}) def test_partially_executed_module(self): - partialexe = import_helper.import_fresh_module("test.typinganndata.partialexecution") + # Results are different with and without lazy imports, we need both: + partialexe = import_helper.import_fresh_module( + "test.typinganndata.partialexecution", + eager_imports=False, + ) self.assertEqual( partialexe.a.__annotations__, {"v1": int, "v2": int}, diff --git a/Makefile.pre.in b/Makefile.pre.in index 1d109dac77fa906..650838d68286c73 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -2485,7 +2485,7 @@ sharedinstall: all # Install the interpreter with $(VERSION) affixed # This goes into $(exec_prefix) .PHONY: altbininstall -altbininstall: $(BUILDPYTHON) @FRAMEWORKPYTHONW@ +altbininstall: all @FRAMEWORKPYTHONW@ @for i in $(BINDIR) $(LIBDIR); \ do \ if test ! -d $(DESTDIR)$$i; then \ @@ -3526,7 +3526,7 @@ MODULE__CTYPES_MALLOC_CLOSURE=@MODULE__CTYPES_MALLOC_CLOSURE@ MODULE__ELEMENTTREE_DEPS=$(srcdir)/Modules/pyexpat.c @LIBEXPAT_INTERNAL@ MODULE__HASHLIB_DEPS=$(srcdir)/Modules/hashlib.h $(srcdir)/Modules/_openssl_mem.h MODULE__IO_DEPS=$(srcdir)/Modules/_io/_iomodule.h -MODULE__REMOTE_DEBUGGING_DEPS=$(srcdir)/Modules/_remote_debugging/_remote_debugging.h $(srcdir)/Modules/_remote_debugging/gc_stats.h +MODULE__REMOTE_DEBUGGING_DEPS=$(srcdir)/Modules/_remote_debugging/_remote_debugging.h $(srcdir)/Modules/_remote_debugging/binary_io.h $(srcdir)/Modules/_remote_debugging/gc_stats.h # HACL*-based cryptographic primitives MODULE__MD5_DEPS=$(srcdir)/Modules/hashlib.h $(LIBHACL_MD5_HEADERS) $(LIBHACL_MD5_LIB_@LIBHACL_LDEPS_LIBTYPE@) diff --git a/Misc/NEWS.d/next/Build/2026-11-06-12-00-01.gh-issue-136267.Qv7mRs.rst b/Misc/NEWS.d/next/Build/2026-11-06-12-00-01.gh-issue-136267.Qv7mRs.rst new file mode 100644 index 000000000000000..43cc054a04ac7a2 --- /dev/null +++ b/Misc/NEWS.d/next/Build/2026-11-06-12-00-01.gh-issue-136267.Qv7mRs.rst @@ -0,0 +1,8 @@ +``build-details.json`` now describes the interpreter being built rather than +the interpreter running :program:`generate-build-details.py`: ``abi.flags``, +``suffixes.extensions``, ``abi.stable_abi_suffix`` and +``implementation.supports_isolated_interpreters`` are derived from the +target's configuration, which fixes cross builds and builds that use +:option:`--with-build-python`. ``abi.extension_suffix`` and +``abi.stable_abi_suffix`` are now also present when libpython is only built +as a static library. diff --git a/Misc/NEWS.d/next/Build/2026-11-06-12-00-02.gh-issue-136267.Vt4pLm.rst b/Misc/NEWS.d/next/Build/2026-11-06-12-00-02.gh-issue-136267.Vt4pLm.rst new file mode 100644 index 000000000000000..d99ae1c42133402 --- /dev/null +++ b/Misc/NEWS.d/next/Build/2026-11-06-12-00-02.gh-issue-136267.Vt4pLm.rst @@ -0,0 +1,4 @@ +``build-details.json`` now takes ``language.version_info``, +``implementation.version`` and ``implementation.hexversion`` from +:file:`Include/patchlevel.h` instead of the interpreter running the generator, +which can be a different patch release under :option:`--with-build-python`. diff --git a/Misc/NEWS.d/next/Build/2026-11-17-20-12-03.gh-issue-162170.0SpcRZ.rst b/Misc/NEWS.d/next/Build/2026-11-17-20-12-03.gh-issue-162170.0SpcRZ.rst new file mode 100644 index 000000000000000..caa4b13ce303395 --- /dev/null +++ b/Misc/NEWS.d/next/Build/2026-11-17-20-12-03.gh-issue-162170.0SpcRZ.rst @@ -0,0 +1,3 @@ +``make install`` now depends on the ``all`` target, so running it directly +after ``./configure --enable-optimizations`` installs the PGO-optimized +interpreter rather than an unoptimized build. diff --git a/Misc/NEWS.d/next/Core_and_Builtins/2023-01-02-14-54-12.gh-issue-100687.yenIag.rst b/Misc/NEWS.d/next/Core_and_Builtins/2023-01-02-14-54-12.gh-issue-100687.yenIag.rst new file mode 100644 index 000000000000000..f4f63584b1659ea --- /dev/null +++ b/Misc/NEWS.d/next/Core_and_Builtins/2023-01-02-14-54-12.gh-issue-100687.yenIag.rst @@ -0,0 +1,2 @@ +Reduce the memory used by the result of adding two large :class:`int` +values. Such additions are also slightly faster for most sizes. diff --git a/Misc/NEWS.d/next/Core_and_Builtins/2026-10-23-08-01-14.gh-issue-159580.lEl21_.rst b/Misc/NEWS.d/next/Core_and_Builtins/2026-10-23-08-01-14.gh-issue-159580.lEl21_.rst new file mode 100644 index 000000000000000..5acc2d3283d5ad2 --- /dev/null +++ b/Misc/NEWS.d/next/Core_and_Builtins/2026-10-23-08-01-14.gh-issue-159580.lEl21_.rst @@ -0,0 +1 @@ +Improve performance of :func:`isinstance` when the check fails. diff --git a/Misc/NEWS.d/next/Core_and_Builtins/2026-11-11-04-58-51.gh-issue-161792.42EcBI8x.rst b/Misc/NEWS.d/next/Core_and_Builtins/2026-11-11-04-58-51.gh-issue-161792.42EcBI8x.rst new file mode 100644 index 000000000000000..a4b685d96bc8967 --- /dev/null +++ b/Misc/NEWS.d/next/Core_and_Builtins/2026-11-11-04-58-51.gh-issue-161792.42EcBI8x.rst @@ -0,0 +1,4 @@ +Fix crashes in the callable iterator returned by :func:`iter` with a +sentinel, when :meth:`!__setstate__` is called while the iterator is in use, +or when the iterator is shared between threads in the :term:`free-threaded +build`. diff --git a/Misc/NEWS.d/next/Core_and_Builtins/2026-11-12-14-37-21.gh-issue-161875.W-nGSw.rst b/Misc/NEWS.d/next/Core_and_Builtins/2026-11-12-14-37-21.gh-issue-161875.W-nGSw.rst new file mode 100644 index 000000000000000..693f39a2bef9177 --- /dev/null +++ b/Misc/NEWS.d/next/Core_and_Builtins/2026-11-12-14-37-21.gh-issue-161875.W-nGSw.rst @@ -0,0 +1,3 @@ +The :opcode:`GET_AWAITABLE` instruction now also pushes the ``NULL`` and +``None`` that start the :opcode:`SEND` loop, so ``await`` no longer needs +separate :opcode:`PUSH_NULL` and ``LOAD_CONST None`` instructions. diff --git a/Misc/NEWS.d/next/Core_and_Builtins/2026-11-12-14-46-10.gh-issue-161875.oOjR9w.rst b/Misc/NEWS.d/next/Core_and_Builtins/2026-11-12-14-46-10.gh-issue-161875.oOjR9w.rst new file mode 100644 index 000000000000000..9afa682ea8b6ffb --- /dev/null +++ b/Misc/NEWS.d/next/Core_and_Builtins/2026-11-12-14-46-10.gh-issue-161875.oOjR9w.rst @@ -0,0 +1,4 @@ +The :opcode:`GET_ANEXT` instruction now also pushes the ``NULL`` and +``None`` that start the :opcode:`SEND` loop, so each :keyword:`async for` +iteration no longer needs separate :opcode:`PUSH_NULL` and ``LOAD_CONST None`` +instructions. diff --git a/Misc/NEWS.d/next/Core_and_Builtins/2026-11-13-13-40-44.gh-issue-161934.MdYQcK.rst b/Misc/NEWS.d/next/Core_and_Builtins/2026-11-13-13-40-44.gh-issue-161934.MdYQcK.rst new file mode 100644 index 000000000000000..746bc662c782e03 --- /dev/null +++ b/Misc/NEWS.d/next/Core_and_Builtins/2026-11-13-13-40-44.gh-issue-161934.MdYQcK.rst @@ -0,0 +1,4 @@ +:func:`print` now writes its output in a single ``write()`` call whenever +possible, reducing interleaving between concurrent calls and the overhead of +repeated writes. If combining the output raises :exc:`MemoryError`, +:func:`print` may make multiple ``write()`` calls. Patch by Inada Naoki. diff --git a/Misc/NEWS.d/next/Core_and_Builtins/2026-11-14-23-17-18.gh-issue-161751.-silGa.rst b/Misc/NEWS.d/next/Core_and_Builtins/2026-11-14-23-17-18.gh-issue-161751.-silGa.rst new file mode 100644 index 000000000000000..0c39e484386dc58 --- /dev/null +++ b/Misc/NEWS.d/next/Core_and_Builtins/2026-11-14-23-17-18.gh-issue-161751.-silGa.rst @@ -0,0 +1,2 @@ +Fix race in mimalloc abandoned segment visit in free-threaded GC. Patch by +Donghee Na. diff --git a/Misc/NEWS.d/next/Library/2026-01-10-17-39-09.gh-issue-143663.9LEFAf.rst b/Misc/NEWS.d/next/Library/2026-01-10-17-39-09.gh-issue-143663.9LEFAf.rst new file mode 100644 index 000000000000000..8428770d51d734b --- /dev/null +++ b/Misc/NEWS.d/next/Library/2026-01-10-17-39-09.gh-issue-143663.9LEFAf.rst @@ -0,0 +1,2 @@ +:meth:`tarfile.TarFile.extractall` now uses OS-specific separators to call +lstat. diff --git a/Misc/NEWS.d/next/Library/2026-10-10-13-35-08.gh-issue-149584.Yq2bF7.rst b/Misc/NEWS.d/next/Library/2026-10-10-13-35-08.gh-issue-149584.Yq2bF7.rst new file mode 100644 index 000000000000000..7f92a1b19877cd4 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2026-10-10-13-35-08.gh-issue-149584.Yq2bF7.rst @@ -0,0 +1,2 @@ +Improve the performance of the Tachyon profiler by avoiding unnecessary remote +reads of the base frame sentinel. Patch by Maurycy Pawłowski-Wieroński. diff --git a/Misc/NEWS.d/next/Library/2026-10-20-12-49-32.gh-issue-159207.Lccezg.rst b/Misc/NEWS.d/next/Library/2026-10-20-12-49-32.gh-issue-159207.Lccezg.rst new file mode 100644 index 000000000000000..a96098fd211abc6 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2026-10-20-12-49-32.gh-issue-159207.Lccezg.rst @@ -0,0 +1,3 @@ +The :mod:`profiling.sampling` binary output format now stores samples in +independent blocks. This is a breaking format change: Python 3.16 does not +support reading profiles written in the previous format. diff --git a/Misc/NEWS.d/next/Library/2026-11-03-08-21-06.gh-issue-161184.PIhcD9.rst b/Misc/NEWS.d/next/Library/2026-11-03-08-21-06.gh-issue-161184.PIhcD9.rst new file mode 100644 index 000000000000000..3b15925773b99ec --- /dev/null +++ b/Misc/NEWS.d/next/Library/2026-11-03-08-21-06.gh-issue-161184.PIhcD9.rst @@ -0,0 +1,6 @@ +Speed up attribute access on instances of :mod:`io` classes and their +subclasses by storing instance attributes and weak references in the object +pre-header. The ``__dictoffset__`` and ``__weakrefoffset__`` attributes +of these classes now report the managed offsets, and C extension types that +subclass an :mod:`io` class can no longer declare their own ``__dictoffset__`` +or ``__weaklistoffset__`` member. diff --git a/Misc/NEWS.d/next/Library/2026-11-06-12-00-00.gh-issue-160978.Kx7mQa.rst b/Misc/NEWS.d/next/Library/2026-11-06-12-00-00.gh-issue-160978.Kx7mQa.rst new file mode 100644 index 000000000000000..be5cc945f4eb61d --- /dev/null +++ b/Misc/NEWS.d/next/Library/2026-11-06-12-00-00.gh-issue-160978.Kx7mQa.rst @@ -0,0 +1,3 @@ +Fix :mod:`ctypes` handing out distinct array types for the same item type +and length when the array type is created concurrently from several +threads. diff --git a/Misc/NEWS.d/next/Library/2026-11-14-21-04-56.gh-issue-161996.35uL2d.rst b/Misc/NEWS.d/next/Library/2026-11-14-21-04-56.gh-issue-161996.35uL2d.rst new file mode 100644 index 000000000000000..561bb62d6537073 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2026-11-14-21-04-56.gh-issue-161996.35uL2d.rst @@ -0,0 +1,2 @@ +Fix a crash in :meth:`concurrent.interpreters.Interpreter.exec` when the +uncaught exception's ``__str__`` raises. diff --git a/Misc/NEWS.d/next/Library/2026-11-15-02-06-21.gh-issue-162028.jAnDAg.rst b/Misc/NEWS.d/next/Library/2026-11-15-02-06-21.gh-issue-162028.jAnDAg.rst new file mode 100644 index 000000000000000..a52a970039d3df0 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2026-11-15-02-06-21.gh-issue-162028.jAnDAg.rst @@ -0,0 +1,2 @@ +Fix the position reported by the pure Python :mod:`json` decoder for an +invalid escape. It now points to the backslash, like the C decoder. diff --git a/Misc/NEWS.d/next/Library/2026-11-15-03-10-31.gh-issue-162031.2k-3CG.rst b/Misc/NEWS.d/next/Library/2026-11-15-03-10-31.gh-issue-162031.2k-3CG.rst new file mode 100644 index 000000000000000..01f1311c735d9b1 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2026-11-15-03-10-31.gh-issue-162031.2k-3CG.rst @@ -0,0 +1,3 @@ +Fix :meth:`io.BufferedIOBase.readinto` and :meth:`~io.BufferedIOBase.readinto1` +raising :exc:`TypeError` instead of returning ``None`` when called on a +non-blocking stream with no data available. diff --git a/Misc/NEWS.d/next/Library/2026-11-16-16-10-42.gh-issue-157579.6choQ_.rst b/Misc/NEWS.d/next/Library/2026-11-16-16-10-42.gh-issue-157579.6choQ_.rst new file mode 100644 index 000000000000000..1bd89a77b5995c1 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2026-11-16-16-10-42.gh-issue-157579.6choQ_.rst @@ -0,0 +1,3 @@ +The error handler :class:`~tempfile.TemporaryDirectory` uses to clean up +unreadable and inaccessible files no longer fails for temporary directories +created using :class:`bytes` paths. diff --git a/Misc/NEWS.d/next/Library/2026-11-18-12-18-53.gh-issue-149584.rDchNk.rst b/Misc/NEWS.d/next/Library/2026-11-18-12-18-53.gh-issue-149584.rDchNk.rst new file mode 100644 index 000000000000000..2d0e9d203cf6b3e --- /dev/null +++ b/Misc/NEWS.d/next/Library/2026-11-18-12-18-53.gh-issue-149584.rDchNk.rst @@ -0,0 +1,3 @@ +Read stack chunks directly in :mod:`profiling.sampling`, without the page +cache. This removes a redundant retry on macOS that can read a replacement +chunk after the target process reuses the same address. diff --git a/Misc/NEWS.d/next/Library/2026-11-18-18-28-37.gh-issue-162230.ezR86S.rst b/Misc/NEWS.d/next/Library/2026-11-18-18-28-37.gh-issue-162230.ezR86S.rst new file mode 100644 index 000000000000000..00b96c972ece6f6 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2026-11-18-18-28-37.gh-issue-162230.ezR86S.rst @@ -0,0 +1,3 @@ +Speed up :meth:`!io.BufferedRWPair.readline`. If a non-blocking read would +block, return the bytes read so far (possibly ``b''``). Previously, it raised +:exc:`OSError`. diff --git a/Misc/NEWS.d/next/Tests/2026-11-15-09-43-15.gh-issue-162034.egx5QV.rst b/Misc/NEWS.d/next/Tests/2026-11-15-09-43-15.gh-issue-162034.egx5QV.rst new file mode 100644 index 000000000000000..1b7d2c4ad5f27c6 --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2026-11-15-09-43-15.gh-issue-162034.egx5QV.rst @@ -0,0 +1,2 @@ +The Apple testbed projects now build when the path of the project contains a +space. diff --git a/Misc/NEWS.d/next/Tests/2026-11-17-00-39-51.gh-issue-162034.FmiuNi.rst b/Misc/NEWS.d/next/Tests/2026-11-17-00-39-51.gh-issue-162034.FmiuNi.rst new file mode 100644 index 000000000000000..1764fa2a458e599 --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2026-11-17-00-39-51.gh-issue-162034.FmiuNi.rst @@ -0,0 +1,2 @@ +Fix the Apple testbed's extension module processing for an XCframework or +package path with a space, or a build path with brackets or a backslash. diff --git a/Misc/NEWS.d/next/Tests/2026-11-17-15-57-41.gh-issue-162144.pL5k29.rst b/Misc/NEWS.d/next/Tests/2026-11-17-15-57-41.gh-issue-162144.pL5k29.rst new file mode 100644 index 000000000000000..bbd4a9cf54752e0 --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2026-11-17-15-57-41.gh-issue-162144.pL5k29.rst @@ -0,0 +1,3 @@ +Skip ``@support.nomemtest`` tests when using a thread sanitizer (TSan). +``set_nomemory()`` and ``call_with_nomemory()`` functions of the +:mod:`!_testcapi` extension are not thread safe. Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/Windows/2026-09-30-01-12-58.gh-issue-158414.738yJa.rst b/Misc/NEWS.d/next/Windows/2026-09-30-01-12-58.gh-issue-158414.738yJa.rst new file mode 100644 index 000000000000000..dba12665deddd93 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2026-09-30-01-12-58.gh-issue-158414.738yJa.rst @@ -0,0 +1 @@ +Fix a hang when writing an incomplete UTF-8 sequence to the Windows console. diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c index 217b5cfedf2adb0..14614dca1c02cee 100644 --- a/Modules/_ctypes/_ctypes.c +++ b/Modules/_ctypes/_ctypes.c @@ -186,7 +186,23 @@ _DictRemover_call(PyObject *myself, PyObject *args, PyObject *kw) { DictRemoverObject *self = _DictRemoverObject_CAST(myself); if (self->key && self->dict) { - if (-1 == PyDict_DelItem(self->dict, self->key)) { + /* Only remove the entry if it is still the proxy that died: the + key may have been reused for a new object in the meantime. */ + PyObject *proxy = PyTuple_GET_SIZE(args) == 1 + ? PyTuple_GET_ITEM(args, 0) : NULL; + PyObject *current; + int rc; + Py_BEGIN_CRITICAL_SECTION(self->dict); + rc = PyDict_GetItemRef(self->dict, self->key, ¤t); + if (rc == 1) { + rc = 0; + if (current == proxy) { + rc = PyDict_DelItem(self->dict, self->key); + } + Py_DECREF(current); + } + Py_END_CRITICAL_SECTION(); + if (rc < 0) { PyErr_FormatUnraisable("Exception ignored while " "calling _ctypes.DictRemover"); } @@ -215,32 +231,72 @@ static PyType_Spec dictremover_spec = { .slots = dictremover_slots, }; -int -PyDict_SetItemProxy(ctypes_state *st, PyObject *dict, PyObject *key, PyObject *item) +/* Cache `item` in `dict` under `key` through a weakref proxy, unless a live + entry already exists. On success *presult is a new reference to whatever + is cached for `key` afterwards: `item` itself, or the object another + thread cached first. + Returns 0 on success, -1 with an exception set on failure. */ +static int +_PyDict_SetDefaultItemProxy(ctypes_state *st, PyObject *dict, PyObject *key, + PyObject *item, PyObject **presult) { - PyObject *obj; - DictRemoverObject *remover; - PyObject *proxy; - int result; - - obj = _PyObject_CallNoArgs((PyObject *)st->DictRemover_Type); - if (obj == NULL) + PyObject *obj = _PyObject_CallNoArgs((PyObject *)st->DictRemover_Type); + if (obj == NULL) { return -1; + } - remover = (DictRemoverObject *)obj; + DictRemoverObject *remover = (DictRemoverObject *)obj; assert(remover->key == NULL); assert(remover->dict == NULL); remover->key = Py_NewRef(key); remover->dict = Py_NewRef(dict); - proxy = PyWeakref_NewProxy(item, obj); + PyObject *proxy = PyWeakref_NewProxy(item, obj); Py_DECREF(obj); - if (proxy == NULL) + if (proxy == NULL) { + return -1; + } + + PyObject *stored; + int present = PyDict_SetDefaultRef(dict, key, proxy, &stored); + if (present < 0) { + Py_DECREF(proxy); return -1; + } + if (present == 0) { + /* Our proxy was inserted; `item` is the canonical object. */ + Py_DECREF(stored); + Py_DECREF(proxy); + *presult = Py_NewRef(item); + return 0; + } - result = PyDict_SetItem(dict, key, proxy); + /* Another thread cached an object for this key first. Prefer it, so that + every caller gets the same object. */ + PyObject *existing = NULL; + int rc = 0; + if (PyWeakref_CheckProxy(stored)) { + rc = PyWeakref_GetRef(stored, &existing); + } + Py_DECREF(stored); + if (rc < 0) { + Py_DECREF(proxy); + return -1; + } + if (existing != NULL) { + Py_DECREF(proxy); + *presult = existing; + return 0; + } + + /* The cached entry had already been cleared; install ours instead. */ + rc = PyDict_SetItem(dict, key, proxy); Py_DECREF(proxy); - return result; + if (rc < 0) { + return -1; + } + *presult = Py_NewRef(item); + return 0; } static int @@ -5154,7 +5210,6 @@ Array_subscript_impl(PyObject *myself, PyObject *item) if (iteminfo->getfunc == _ctypes_get_fielddesc("c")->getfunc) { char *ptr = (char *)self->b_ptr; - char *dest; if (slicelen <= 0) return Py_GetConstant(Py_CONSTANT_EMPTY_BYTES); @@ -5164,19 +5219,17 @@ Array_subscript_impl(PyObject *myself, PyObject *item) slicelen); return res; } - dest = (char *)PyMem_Malloc(slicelen); - - if (dest == NULL) - return PyErr_NoMemory(); - + PyBytesWriter *writer = PyBytesWriter_Create(slicelen); + if (writer == NULL) { + return NULL; + } + char *dest = PyBytesWriter_GetData(writer); for (cur = start, i = 0; i < slicelen; cur += step, i++) { dest[i] = ptr[cur]; } - np = PyBytes_FromStringAndSize(dest, slicelen); - PyMem_Free(dest); - return np; + return PyBytesWriter_Finish(writer); } if (iteminfo->getfunc == _ctypes_get_fielddesc("u")->getfunc) { wchar_t *ptr = (wchar_t *)self->b_ptr; @@ -5437,13 +5490,20 @@ PyCArrayType_from_ctype(ctypes_state *st, PyObject *itemtype, Py_ssize_t length) Py_DECREF(key); return NULL; } - if (PyDict_SetItemProxy(st, st->array_cache, key, result) < 0) { + PyObject *cached; + int rc; + Py_BEGIN_CRITICAL_SECTION(st->array_cache); + rc = _PyDict_SetDefaultItemProxy(st, st->array_cache, key, result, + &cached); + Py_END_CRITICAL_SECTION(); + if (rc < 0) { Py_DECREF(key); Py_DECREF(result); return NULL; } Py_DECREF(key); - return result; + Py_DECREF(result); + return cached; } @@ -5964,8 +6024,6 @@ Pointer_subscript(PyObject *myself, PyObject *item) } assert(iteminfo); if (iteminfo->getfunc == _ctypes_get_fielddesc("c")->getfunc) { - char *dest; - if (len <= 0) return Py_GetConstant(Py_CONSTANT_EMPTY_BYTES); if (step == 1) { @@ -5977,18 +6035,18 @@ Pointer_subscript(PyObject *myself, PyObject *item) Py_END_CRITICAL_SECTION(); return res; } - dest = (char *)PyMem_Malloc(len); - if (dest == NULL) - return PyErr_NoMemory(); + PyBytesWriter *writer = PyBytesWriter_Create(len); + if (writer == NULL) { + return NULL; + } + char *dest = PyBytesWriter_GetData(writer); Py_BEGIN_CRITICAL_SECTION(self); char *ptr = *(void **)self->b_ptr; for (cur = start, i = 0; i < len; cur += step, i++) { dest[i] = ptr[cur]; } Py_END_CRITICAL_SECTION(); - np = PyBytes_FromStringAndSize(dest, len); - PyMem_Free(dest); - return np; + return PyBytesWriter_Finish(writer); } if (iteminfo->getfunc == _ctypes_get_fielddesc("u")->getfunc) { wchar_t *dest; diff --git a/Modules/_ctypes/ctypes.h b/Modules/_ctypes/ctypes.h index 248559aa364a198..98d8b660850b0d9 100644 --- a/Modules/_ctypes/ctypes.h +++ b/Modules/_ctypes/ctypes.h @@ -24,7 +24,7 @@ // Do we support C99 complex types in ffi? // For Apple's libffi, this must be determined at runtime (see gh-128156). #if defined(_Py_FFI_SUPPORT_C_COMPLEX) -# if USING_APPLE_OS_LIBFFI && defined(__has_builtin) +# if defined(USING_APPLE_OS_LIBFFI) && defined(__has_builtin) # if __has_builtin(__builtin_available) # define Py_FFI_COMPLEX_AVAILABLE __builtin_available(macOS 10.15, *) # else diff --git a/Modules/_cursesmodule.c b/Modules/_cursesmodule.c index 52b87e7cc0e8b98..e82c78691c7f6a5 100644 --- a/Modules/_cursesmodule.c +++ b/Modules/_cursesmodule.c @@ -1936,10 +1936,10 @@ Window_NoArgTrueFalseFunction(is_scrollok) Window_NoArgTrueFalseFunction(is_subwin) Window_NoArgTrueFalseFunction(is_syncok) #endif -#if defined(HAVE_CURSES_IS_KEYPAD) || defined(PDCURSES) +#if defined(_Py__Py_HAVE_CURSES_IS_KEYPAD) || defined(PDCURSES) Window_NoArgTrueFalseFunction(is_keypad) #endif -#if defined(HAVE_CURSES_IS_LEAVEOK) || defined(PDCURSES) +#if defined(_Py_HAVE_CURSES_IS_LEAVEOK) || defined(PDCURSES) Window_NoArgTrueFalseFunction(is_leaveok) #endif #if defined(HAVE_CURSES_IS_PAD) || defined(PDCURSES) @@ -2199,7 +2199,7 @@ static int curses_wattr_save(PyCursesWindowObject *self, attr_t *attrs, int *pair, const char *funcname) { -#if defined(HAVE_CURSES_WATTR_GET) && defined(HAVE_CURSES_WATTR_SET) +#if defined(_Py_HAVE_CURSES_WATTR_GET) && defined(_Py_HAVE_CURSES_WATTR_SET) int rtn; #if _NCURSES_EXTENDED_COLOR_FUNCS short legacy_pair; @@ -2222,7 +2222,7 @@ curses_wattr_save(PyCursesWindowObject *self, attr_t *attrs, int *pair, /* Put the rendition back. The name of the curses function used is _CURSES_WATTR_RESTORE_FUNC, for the caller to name it in an error. */ -#if defined(HAVE_CURSES_WATTR_GET) && defined(HAVE_CURSES_WATTR_SET) +#if defined(_Py_HAVE_CURSES_WATTR_GET) && defined(_Py_HAVE_CURSES_WATTR_SET) #define _CURSES_WATTR_RESTORE_FUNC "wattr_set" #else #define _CURSES_WATTR_RESTORE_FUNC "wattrset" @@ -2231,7 +2231,7 @@ curses_wattr_save(PyCursesWindowObject *self, attr_t *attrs, int *pair, static int curses_wattr_restore(PyCursesWindowObject *self, attr_t attrs, int pair) { -#if defined(HAVE_CURSES_WATTR_GET) && defined(HAVE_CURSES_WATTR_SET) +#if defined(_Py_HAVE_CURSES_WATTR_GET) && defined(_Py_HAVE_CURSES_WATTR_SET) #if _NCURSES_EXTENDED_COLOR_FUNCS return wattr_set(self->win, attrs, 0, &pair); #else @@ -2558,7 +2558,7 @@ _curses_window_attrset_impl(PyCursesWindowObject *self, attr_t attr) return curses_window_check_err(self, rtn, "wattrset", "attrset"); } -#ifdef HAVE_CURSES_WATTR_GET +#ifdef _Py_HAVE_CURSES_WATTR_GET /*[clinic input] _curses.window.attr_get @@ -2584,9 +2584,9 @@ _curses_window_attr_get_impl(PyCursesWindowObject *self) } return Py_BuildValue("(ki)", (unsigned long)attrs, (int)pair); } -#endif /* HAVE_CURSES_WATTR_GET */ +#endif /* _Py_HAVE_CURSES_WATTR_GET */ -#ifdef HAVE_CURSES_WATTR_SET +#ifdef _Py_HAVE_CURSES_WATTR_SET /*[clinic input] _curses.window.attr_set @@ -2610,9 +2610,9 @@ _curses_window_attr_set_impl(PyCursesWindowObject *self, attr_t attr, #endif return curses_window_check_err(self, rtn, "wattr_set", "attr_set"); } -#endif /* HAVE_CURSES_WATTR_SET */ +#endif /* _Py_HAVE_CURSES_WATTR_SET */ -#ifdef HAVE_CURSES_WATTR_ON +#ifdef _Py_HAVE_CURSES_WATTR_ON /*[clinic input] _curses.window.attr_on @@ -2629,9 +2629,9 @@ _curses_window_attr_on_impl(PyCursesWindowObject *self, attr_t attr) int rtn = wattr_on(self->win, attr, NULL); return curses_window_check_err(self, rtn, "wattr_on", "attr_on"); } -#endif /* HAVE_CURSES_WATTR_ON */ +#endif /* _Py_HAVE_CURSES_WATTR_ON */ -#ifdef HAVE_CURSES_WATTR_OFF +#ifdef _Py_HAVE_CURSES_WATTR_OFF /*[clinic input] _curses.window.attr_off @@ -2648,9 +2648,9 @@ _curses_window_attr_off_impl(PyCursesWindowObject *self, attr_t attr) int rtn = wattr_off(self->win, attr, NULL); return curses_window_check_err(self, rtn, "wattr_off", "attr_off"); } -#endif /* HAVE_CURSES_WATTR_OFF */ +#endif /* _Py_HAVE_CURSES_WATTR_OFF */ -#ifdef HAVE_CURSES_WCOLOR_SET +#ifdef _Py_HAVE_CURSES_WCOLOR_SET /*[clinic input] _curses.window.color_set @@ -2672,7 +2672,7 @@ _curses_window_color_set_impl(PyCursesWindowObject *self, int pair) #endif return curses_window_check_err(self, rtn, "wcolor_set", "color_set"); } -#endif /* HAVE_CURSES_WCOLOR_SET */ +#endif /* _Py_HAVE_CURSES_WCOLOR_SET */ /*[clinic input] _curses.window.getattrs @@ -5035,7 +5035,7 @@ static PyType_Spec PyCursesComplexStr_Type_spec = { }; #undef clinic_state -#if defined(HAVE_CURSES_USE_SCREEN) || defined(HAVE_CURSES_USE_WINDOW) +#if defined(_Py_HAVE_CURSES_USE_SCREEN) || defined(_Py_HAVE_CURSES_USE_WINDOW) /* Shared trampoline for window.use()/screen.use(): call func(obj, *extra, **kwargs) and store the result (NULL on exception) in data->result. */ @@ -5085,7 +5085,7 @@ curses_use_parse(PyObject *args, PyObject **func, PyObject **extra) } #endif -#ifdef HAVE_CURSES_USE_WINDOW +#ifdef _Py_HAVE_CURSES_USE_WINDOW static int curses_use_window_cb(WINDOW *Py_UNUSED(win), void *data) { @@ -5111,7 +5111,7 @@ PyCursesWindow_use(PyObject *self, PyObject *args, PyObject *kwargs) Py_DECREF(extra); return data.result; } -#endif /* HAVE_CURSES_USE_WINDOW */ +#endif /* _Py_HAVE_CURSES_USE_WINDOW */ static PyMethodDef PyCursesWindow_methods[] = { _CURSES_WINDOW_ADDCH_METHODDEF @@ -5250,12 +5250,12 @@ static PyMethodDef PyCursesWindow_methods[] = { "is_syncok($self, /)\n--\n\n" "Return the current value set by syncok()."}, #endif -#if defined(HAVE_CURSES_IS_KEYPAD) || defined(PDCURSES) +#if defined(_Py__Py_HAVE_CURSES_IS_KEYPAD) || defined(PDCURSES) {"is_keypad", PyCursesWindow_is_keypad, METH_NOARGS, "is_keypad($self, /)\n--\n\n" "Return the current value set by keypad()."}, #endif -#if defined(HAVE_CURSES_IS_LEAVEOK) || defined(PDCURSES) +#if defined(_Py_HAVE_CURSES_IS_LEAVEOK) || defined(PDCURSES) {"is_leaveok", PyCursesWindow_is_leaveok, METH_NOARGS, "is_leaveok($self, /)\n--\n\n" "Return the current value set by leaveok()."}, @@ -5336,7 +5336,7 @@ static PyMethodDef PyCursesWindow_methods[] = { "untouchwin($self, /)\n--\n\n" "Mark all lines in the window as unchanged since last refresh()."}, _CURSES_WINDOW_VLINE_METHODDEF -#ifdef HAVE_CURSES_USE_WINDOW +#ifdef _Py_HAVE_CURSES_USE_WINDOW {"use", _PyCFunction_CAST(PyCursesWindow_use), METH_VARARGS | METH_KEYWORDS, PyCursesWindow_use__doc__}, #endif @@ -5478,7 +5478,7 @@ static PyGetSetDef PyCursesScreen_getsets[] = { {NULL, NULL, NULL, NULL, NULL} /* sentinel */ }; -#ifdef HAVE_CURSES_USE_SCREEN +#ifdef _Py_HAVE_CURSES_USE_SCREEN static int curses_use_screen_cb(SCREEN *Py_UNUSED(sp), void *data) { @@ -5533,7 +5533,7 @@ PyCursesScreen_use(PyObject *self, PyObject *args, PyObject *kwargs) } return data.result; } -#endif /* HAVE_CURSES_USE_SCREEN */ +#endif /* _Py_HAVE_CURSES_USE_SCREEN */ PyDoc_STRVAR(PyCursesScreen_close__doc__, "close($self, /)\n--\n\n" @@ -5552,7 +5552,7 @@ PyCursesScreen_close(PyObject *self, PyObject *Py_UNUSED(ignored)) static PyMethodDef PyCursesScreen_methods[] = { {"close", PyCursesScreen_close, METH_NOARGS, PyCursesScreen_close__doc__}, -#ifdef HAVE_CURSES_USE_SCREEN +#ifdef _Py_HAVE_CURSES_USE_SCREEN {"use", _PyCFunction_CAST(PyCursesScreen_use), METH_VARARGS | METH_KEYWORDS, PyCursesScreen_use__doc__}, #endif @@ -5704,7 +5704,7 @@ _curses_filter_impl(PyObject *module) } #endif -#ifdef HAVE_CURSES_NOFILTER +#ifdef _Py_HAVE_CURSES_NOFILTER /*[clinic input] _curses.nofilter @@ -6269,7 +6269,7 @@ _curses_getwin(PyObject *module, PyObject *file) return res; } -#ifdef HAVE_CURSES_SCR_DUMP +#ifdef _Py_HAVE_CURSES_SCR_DUMP /*[clinic input] _curses.scr_dump @@ -6324,7 +6324,7 @@ _curses_scr_init(PyObject *module, PyObject *filename) /*[clinic end generated code: output=2e861d381d710419 input=81c45e4702124ef6]*/ ScreenDumpFunctionBody(scr_init) -#ifdef HAVE_CURSES_SCR_SET +#ifdef _Py_HAVE_CURSES_SCR_SET /*[clinic input] _curses.scr_set @@ -6341,8 +6341,8 @@ static PyObject * _curses_scr_set(PyObject *module, PyObject *filename) /*[clinic end generated code: output=6056fdec12c5935f input=d248c20543cc289b]*/ ScreenDumpFunctionBody(scr_set) -#endif /* HAVE_CURSES_SCR_SET */ -#endif /* HAVE_CURSES_SCR_DUMP */ +#endif /* _Py_HAVE_CURSES_SCR_SET */ +#endif /* _Py_HAVE_CURSES_SCR_DUMP */ /*[clinic input] _curses.halfdelay @@ -6421,7 +6421,7 @@ _curses_has_key_impl(PyObject *module, int key) } #endif -#ifdef HAVE_CURSES_DEFINE_KEY +#ifdef _Py_HAVE_CURSES_DEFINE_KEY /*[clinic input] _curses.define_key @@ -6447,9 +6447,9 @@ _curses_define_key_impl(PyObject *module, const char *definition, return curses_check_err(module, define_key(definition, keycode), "define_key", NULL); } -#endif /* HAVE_CURSES_DEFINE_KEY */ +#endif /* _Py_HAVE_CURSES_DEFINE_KEY */ -#ifdef HAVE_CURSES_KEY_DEFINED +#ifdef _Py_HAVE_CURSES_KEY_DEFINED /*[clinic input] _curses.key_defined @@ -6471,9 +6471,9 @@ _curses_key_defined_impl(PyObject *module, const char *definition) return PyLong_FromLong(key_defined(definition)); } -#endif /* HAVE_CURSES_KEY_DEFINED */ +#endif /* _Py_HAVE_CURSES_KEY_DEFINED */ -#ifdef HAVE_CURSES_KEYOK +#ifdef _Py_HAVE_CURSES_KEYOK /*[clinic input] _curses.keyok @@ -6494,7 +6494,7 @@ _curses_keyok_impl(PyObject *module, int keycode, int enable) return curses_check_err(module, keyok(keycode, enable), "keyok", NULL); } -#endif /* HAVE_CURSES_KEYOK */ +#endif /* _Py_HAVE_CURSES_KEYOK */ /*[clinic input] _curses.init_color @@ -7283,7 +7283,7 @@ _curses_set_term(PyObject *module, PyObject *screen) return prev != NULL ? prev : Py_NewRef(Py_None); } -#ifdef HAVE_CURSES_NEW_PRESCR +#ifdef _Py_HAVE_CURSES_NEW_PRESCR /*[clinic input] _curses.new_prescr @@ -7317,7 +7317,7 @@ _curses_new_prescr_impl(PyObject *module) state->prescreen = Py_NewRef(screenobj); return screenobj; } -#endif /* HAVE_CURSES_NEW_PRESCR */ +#endif /* _Py_HAVE_CURSES_NEW_PRESCR */ #ifdef HAVE_CURSES_ESCDELAY /*[clinic input] @@ -7573,7 +7573,7 @@ _curses_meta_impl(PyObject *module, int yes) } #if defined(HAVE_CURSES_GETMOUSE) || defined(PDCURSES) -#if defined(HAVE_CURSES_HAS_MOUSE) || defined(PDCURSES) +#if defined(_Py_HAVE_CURSES_HAS_MOUSE) || defined(PDCURSES) /*[clinic input] _curses.has_mouse @@ -7588,7 +7588,7 @@ _curses_has_mouse_impl(PyObject *module) return PyBool_FromLong(has_mouse()); } -#endif /* HAVE_CURSES_HAS_MOUSE || PDCURSES */ +#endif /* _Py_HAVE_CURSES_HAS_MOUSE || PDCURSES */ /*[clinic input] _curses.mouseinterval @@ -8272,7 +8272,7 @@ _curses_termattrs_impl(PyObject *module) return PyLong_FromUnsignedLong((unsigned long)(chtype)termattrs()); } -#ifdef HAVE_CURSES_TERM_ATTRS +#ifdef _Py_HAVE_CURSES_TERM_ATTRS /*[clinic input] _curses.term_attrs @@ -8290,7 +8290,7 @@ _curses_term_attrs_impl(PyObject *module) return PyLong_FromUnsignedLongLong(term_attrs()); } -#endif /* HAVE_CURSES_TERM_ATTRS */ +#endif /* _Py_HAVE_CURSES_TERM_ATTRS */ /*[clinic input] @permit_long_summary @@ -8877,7 +8877,7 @@ _curses_slk_attr_impl(PyObject *module) } #endif -#ifdef HAVE_CURSES_SLK_ATTR_ON +#ifdef _Py_HAVE_CURSES_SLK_ATTR_ON /*[clinic input] _curses.slk_attr_on @@ -8895,9 +8895,9 @@ _curses_slk_attr_on_impl(PyObject *module, attr_t attr) return curses_check_err(module, slk_attr_on(attr, NULL), "slk_attr_on", NULL); } -#endif /* HAVE_CURSES_SLK_ATTR_ON */ +#endif /* _Py_HAVE_CURSES_SLK_ATTR_ON */ -#ifdef HAVE_CURSES_SLK_ATTR_OFF +#ifdef _Py_HAVE_CURSES_SLK_ATTR_OFF /*[clinic input] _curses.slk_attr_off @@ -8915,9 +8915,9 @@ _curses_slk_attr_off_impl(PyObject *module, attr_t attr) return curses_check_err(module, slk_attr_off(attr, NULL), "slk_attr_off", NULL); } -#endif /* HAVE_CURSES_SLK_ATTR_OFF */ +#endif /* _Py_HAVE_CURSES_SLK_ATTR_OFF */ -#ifdef HAVE_CURSES_SLK_ATTR_SET +#ifdef _Py_HAVE_CURSES_SLK_ATTR_SET /*[clinic input] _curses.slk_attr_set @@ -8941,9 +8941,9 @@ _curses_slk_attr_set_impl(PyObject *module, attr_t attr, int pair) #endif return curses_check_err(module, rtn, "slk_attr_set", NULL); } -#endif /* HAVE_CURSES_SLK_ATTR_SET */ +#endif /* _Py_HAVE_CURSES_SLK_ATTR_SET */ -#ifdef HAVE_CURSES_SLK_COLOR +#ifdef _Py_HAVE_CURSES_SLK_COLOR /*[clinic input] _curses.slk_color @@ -8966,7 +8966,7 @@ _curses_slk_color_impl(PyObject *module, int pair) #endif return curses_check_err(module, rtn, "slk_color", NULL); } -#endif /* HAVE_CURSES_SLK_COLOR */ +#endif /* _Py_HAVE_CURSES_SLK_COLOR */ #ifdef HAVE_CURSES_USE_ENV /*[clinic input] @@ -9178,7 +9178,7 @@ static PyMethodDef cursesmodule_methods[] = { _CURSES_NEWPAD_METHODDEF _CURSES_NEWTERM_METHODDEF _CURSES_NEWWIN_METHODDEF -#ifdef HAVE_CURSES_NEW_PRESCR +#ifdef _Py_HAVE_CURSES_NEW_PRESCR _CURSES_NEW_PRESCR_METHODDEF #endif _CURSES_NL_METHODDEF diff --git a/Modules/_io/bufferedio.c b/Modules/_io/bufferedio.c index 44c3f56caee18b6..1e900084b549f4f 100644 --- a/Modules/_io/bufferedio.c +++ b/Modules/_io/bufferedio.c @@ -9,11 +9,11 @@ #include "Python.h" #include "pycore_call.h" // _PyObject_CallNoArgs() +#include "pycore_critical_section.h" // _Py_CRITICAL_SECTION_ASSERT_OBJECT_LOCKED() #include "pycore_fileutils.h" // _PyFile_Flush #include "pycore_object.h" // _PyObject_GC_UNTRACK() #include "pycore_pyerrors.h" // _Py_FatalErrorFormat() #include "pycore_pylifecycle.h" // _Py_IsInterpreterFinalizing() -#include "pycore_weakref.h" // FT_CLEAR_WEAKREFS() #include "_iomodule.h" @@ -60,6 +60,11 @@ _bufferediobase_readinto_generic(PyObject *self, Py_buffer *buffer, char readint if (data == NULL) return NULL; + /* Non-blocking stream with no data available */ + if (data == Py_None) { + return data; + } + if (!PyBytes_Check(data)) { Py_DECREF(data); PyErr_SetString(PyExc_TypeError, "read() should return bytes"); @@ -260,8 +265,6 @@ typedef struct { Py_ssize_t buffer_size; Py_ssize_t buffer_mask; - PyObject *dict; - PyObject *weakreflist; } buffered; #define buffered_CAST(op) ((buffered *)(op)) @@ -417,7 +420,7 @@ buffered_clear(PyObject *op) buffered *self = buffered_CAST(op); self->ok = 0; Py_CLEAR(self->raw); - Py_CLEAR(self->dict); + PyObject_ClearManagedDict(op); return 0; } @@ -431,7 +434,7 @@ buffered_dealloc(PyObject *op) return; _PyObject_GC_UNTRACK(self); self->ok = 0; - FT_CLEAR_WEAKREFS(op, self->weakreflist); + PyObject_ClearWeakRefs(op); if (self->buffer) { PyMem_Free(self->buffer); self->buffer = NULL; @@ -467,8 +470,7 @@ buffered_traverse(PyObject *op, visitproc visit, void *arg) buffered *self = buffered_CAST(op); Py_VISIT(Py_TYPE(self)); Py_VISIT(self->raw); - Py_VISIT(self->dict); - return 0; + return PyObject_VisitManagedDict(op, visit, arg); } /* Because this can call arbitrary code, it shouldn't be called when @@ -2250,8 +2252,6 @@ typedef struct { PyObject_HEAD buffered *reader; buffered *writer; - PyObject *dict; - PyObject *weakreflist; } rwpair; #define rwpair_CAST(op) ((rwpair *)(op)) @@ -2309,10 +2309,9 @@ bufferedrwpair_traverse(PyObject *op, visitproc visit, void *arg) { rwpair *self = rwpair_CAST(op); Py_VISIT(Py_TYPE(self)); - Py_VISIT(self->dict); Py_VISIT(self->reader); Py_VISIT(self->writer); - return 0; + return PyObject_VisitManagedDict(op, visit, arg); } static int @@ -2321,7 +2320,7 @@ bufferedrwpair_clear(PyObject *op) rwpair *self = rwpair_CAST(op); Py_CLEAR(self->reader); Py_CLEAR(self->writer); - Py_CLEAR(self->dict); + PyObject_ClearManagedDict(op); return 0; } @@ -2331,7 +2330,7 @@ bufferedrwpair_dealloc(PyObject *op) rwpair *self = rwpair_CAST(op); PyTypeObject *tp = Py_TYPE(self); _PyObject_GC_UNTRACK(self); - FT_CLEAR_WEAKREFS(op, self->weakreflist); + PyObject_ClearWeakRefs(op); (void)bufferedrwpair_clear(op); tp->tp_free(self); Py_DECREF(tp); @@ -2394,6 +2393,25 @@ _io_BufferedRWPair_read1_impl(rwpair *self, PyObject *size) return _forward_call(self->reader, &_Py_ID(read1), size); } +/*[clinic input] +_io.BufferedRWPair.readline + size: object(c_default="NULL") = -1 + / + +Read and return a line from the stream. + +If size is specified, at most size bytes will be read. + +The line terminator is always b'\n'. +[clinic start generated code]*/ + +static PyObject * +_io_BufferedRWPair_readline_impl(rwpair *self, PyObject *size) +/*[clinic end generated code: output=ee5bb2364231e6e1 input=1c552272a69db1ad]*/ +{ + return _forward_call(self->reader, &_Py_ID(readline), size); +} + /*[clinic input] _io.BufferedRWPair.readinto buffer: object @@ -2638,8 +2656,6 @@ static PyMethodDef bufferedreader_methods[] = { static PyMemberDef bufferedreader_members[] = { {"raw", _Py_T_OBJECT, offsetof(buffered, raw), Py_READONLY}, {"_finalizing", Py_T_BOOL, offsetof(buffered, finalizing), 0}, - {"__weaklistoffset__", Py_T_PYSSIZET, offsetof(buffered, weakreflist), Py_READONLY}, - {"__dictoffset__", Py_T_PYSSIZET, offsetof(buffered, dict), Py_READONLY}, {NULL} }; @@ -2697,8 +2713,6 @@ static PyMethodDef bufferedwriter_methods[] = { static PyMemberDef bufferedwriter_members[] = { {"raw", _Py_T_OBJECT, offsetof(buffered, raw), Py_READONLY}, {"_finalizing", Py_T_BOOL, offsetof(buffered, finalizing), 0}, - {"__weaklistoffset__", Py_T_PYSSIZET, offsetof(buffered, weakreflist), Py_READONLY}, - {"__dictoffset__", Py_T_PYSSIZET, offsetof(buffered, dict), Py_READONLY}, {NULL} }; @@ -2735,6 +2749,7 @@ static PyMethodDef bufferedrwpair_methods[] = { _IO_BUFFEREDRWPAIR_READ_METHODDEF _IO_BUFFEREDRWPAIR_PEEK_METHODDEF _IO_BUFFEREDRWPAIR_READ1_METHODDEF + _IO_BUFFEREDRWPAIR_READLINE_METHODDEF _IO_BUFFEREDRWPAIR_READINTO_METHODDEF _IO_BUFFEREDRWPAIR_READINTO1_METHODDEF @@ -2750,12 +2765,6 @@ static PyMethodDef bufferedrwpair_methods[] = { {NULL, NULL} }; -static PyMemberDef bufferedrwpair_members[] = { - {"__weaklistoffset__", Py_T_PYSSIZET, offsetof(rwpair, weakreflist), Py_READONLY}, - {"__dictoffset__", Py_T_PYSSIZET, offsetof(rwpair, dict), Py_READONLY}, - {NULL} -}; - static PyGetSetDef bufferedrwpair_getset[] = { _IO_BUFFEREDRWPAIR_CLOSED_GETSETDEF {NULL} @@ -2767,7 +2776,6 @@ static PyType_Slot bufferedrwpair_slots[] = { {Py_tp_traverse, bufferedrwpair_traverse}, {Py_tp_clear, bufferedrwpair_clear}, {Py_tp_methods, bufferedrwpair_methods}, - {Py_tp_members, bufferedrwpair_members}, {Py_tp_getset, bufferedrwpair_getset}, {Py_tp_init, _io_BufferedRWPair___init__}, {0, NULL}, @@ -2814,8 +2822,6 @@ static PyMethodDef bufferedrandom_methods[] = { static PyMemberDef bufferedrandom_members[] = { {"raw", _Py_T_OBJECT, offsetof(buffered, raw), Py_READONLY}, {"_finalizing", Py_T_BOOL, offsetof(buffered, finalizing), 0}, - {"__weaklistoffset__", Py_T_PYSSIZET, offsetof(buffered, weakreflist), Py_READONLY}, - {"__dictoffset__", Py_T_PYSSIZET, offsetof(buffered, dict), Py_READONLY}, {NULL} }; diff --git a/Modules/_io/bytesio.c b/Modules/_io/bytesio.c index 9bec70acebb3598..fa828f218928990 100644 --- a/Modules/_io/bytesio.c +++ b/Modules/_io/bytesio.c @@ -5,7 +5,6 @@ #include "pycore_object.h" #include "pycore_pyatomic_ft_wrappers.h" #include "pycore_sysmodule.h" // _PySys_GetSizeOf() -#include "pycore_weakref.h" // FT_CLEAR_WEAKREFS() #include // offsetof() #include "_iomodule.h" @@ -21,8 +20,6 @@ typedef struct { PyObject *buf; Py_ssize_t pos; Py_ssize_t string_size; - PyObject *dict; - PyObject *weakreflist; Py_ssize_t exports; #ifdef Py_GIL_DISABLED int buf_shared; @@ -974,11 +971,16 @@ _io_BytesIO___getstate___impl(bytesio *self) if (initvalue == NULL) return NULL; - if (self->dict == NULL) { + if (_PyObject_IsInstanceDictEmpty((PyObject *)self)) { dict = Py_NewRef(Py_None); } else { - dict = PyDict_Copy(self->dict); + dict = PyObject_GenericGetDict((PyObject *)self, NULL); + if (dict == NULL) { + Py_DECREF(initvalue); + return NULL; + } + Py_SETREF(dict, PyDict_Copy(dict)); if (dict == NULL) { Py_DECREF(initvalue); return NULL; @@ -1061,16 +1063,16 @@ _io_BytesIO___setstate___impl(bytesio *self, PyObject *state) Py_TYPE(dict)->tp_name); return NULL; } - if (self->dict) { - /* Alternatively, we could replace the internal dictionary - completely. However, it seems more practical to just update it. */ - if (PyDict_Update(self->dict, dict) < 0) - return NULL; + /* Alternatively, we could replace the internal dictionary + completely. However, it seems more practical to just update it. */ + PyObject *self_dict = PyObject_GenericGetDict((PyObject *)self, NULL); + if (self_dict == NULL) { + return NULL; } - else { - /* The LOAD_ATTR specializations read the dict slot lock-free - with an acquire load, so pair it with a release store. */ - FT_ATOMIC_STORE_PTR_RELEASE(self->dict, Py_NewRef(dict)); + int rc = PyDict_Update(self_dict, dict); + Py_DECREF(self_dict); + if (rc < 0) { + return NULL; } } @@ -1089,8 +1091,8 @@ bytesio_dealloc(PyObject *op) PyErr_Print(); } Py_CLEAR(self->buf); - Py_CLEAR(self->dict); - FT_CLEAR_WEAKREFS(op, self->weakreflist); + PyObject_ClearManagedDict(op); + PyObject_ClearWeakRefs(op); tp->tp_free(self); Py_DECREF(tp); } @@ -1187,16 +1189,15 @@ bytesio_traverse(PyObject *op, visitproc visit, void *arg) { bytesio *self = bytesio_CAST(op); Py_VISIT(Py_TYPE(self)); - Py_VISIT(self->dict); Py_VISIT(self->buf); - return 0; + return PyObject_VisitManagedDict(op, visit, arg); } static int bytesio_clear(PyObject *op) { bytesio *self = bytesio_CAST(op); - Py_CLEAR(self->dict); + PyObject_ClearManagedDict(op); if (FT_ATOMIC_LOAD_SSIZE_RELAXED(self->exports) == 0) { Py_CLEAR(self->buf); } @@ -1239,12 +1240,6 @@ static struct PyMethodDef bytesio_methods[] = { {NULL, NULL} /* sentinel */ }; -static PyMemberDef bytesio_members[] = { - {"__weaklistoffset__", Py_T_PYSSIZET, offsetof(bytesio, weakreflist), Py_READONLY}, - {"__dictoffset__", Py_T_PYSSIZET, offsetof(bytesio, dict), Py_READONLY}, - {NULL} -}; - static PyType_Slot bytesio_slots[] = { {Py_tp_dealloc, bytesio_dealloc}, {Py_tp_doc, (void *)_io_BytesIO___init____doc__}, @@ -1253,7 +1248,6 @@ static PyType_Slot bytesio_slots[] = { {Py_tp_iter, PyObject_SelfIter}, {Py_tp_iternext, bytesio_iternext}, {Py_tp_methods, bytesio_methods}, - {Py_tp_members, bytesio_members}, {Py_tp_getset, bytesio_getsetlist}, {Py_tp_init, _io_BytesIO___init__}, {Py_tp_new, bytesio_new}, diff --git a/Modules/_io/clinic/bufferedio.c.h b/Modules/_io/clinic/bufferedio.c.h index a85dd6a20bc2c1d..0cedf4a9948e9da 100644 --- a/Modules/_io/clinic/bufferedio.c.h +++ b/Modules/_io/clinic/bufferedio.c.h @@ -1197,6 +1197,42 @@ _io_BufferedRWPair_read1(PyObject *self, PyObject *const *args, Py_ssize_t nargs return return_value; } +PyDoc_STRVAR(_io_BufferedRWPair_readline__doc__, +"readline($self, size=-1, /)\n" +"--\n" +"\n" +"Read and return a line from the stream.\n" +"\n" +"If size is specified, at most size bytes will be read.\n" +"\n" +"The line terminator is always b\'\\n\'."); + +#define _IO_BUFFEREDRWPAIR_READLINE_METHODDEF \ + {"readline", _PyCFunction_CAST(_io_BufferedRWPair_readline), METH_FASTCALL, _io_BufferedRWPair_readline__doc__}, + +static PyObject * +_io_BufferedRWPair_readline_impl(rwpair *self, PyObject *size); + +static PyObject * +_io_BufferedRWPair_readline(PyObject *self, PyObject *const *args, Py_ssize_t nargs) +{ + PyObject *return_value = NULL; + PyObject *size = NULL; + + if (!_PyArg_CheckPositional("readline", nargs, 0, 1)) { + goto exit; + } + if (nargs < 1) { + goto skip_optional; + } + size = args[0]; +skip_optional: + return_value = _io_BufferedRWPair_readline_impl((rwpair *)self, size); + +exit: + return return_value; +} + PyDoc_STRVAR(_io_BufferedRWPair_readinto__doc__, "readinto($self, buffer, /)\n" "--\n" @@ -1441,4 +1477,4 @@ _io_BufferedRandom___init__(PyObject *self, PyObject *args, PyObject *kwargs) #define _IO_BUFFEREDRWPAIR_CLOSED_GETSETDEF {"closed", (getter)_io_BufferedRWPair_closed_get, (setter)NULL, NULL}, -/*[clinic end generated code: output=d9e1eb0efa4a2431 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=1596bb353f00a1ea input=a9049054013a1b77]*/ diff --git a/Modules/_io/fileio.c b/Modules/_io/fileio.c index 732eb28e51ba8c1..a954bc5cd407d27 100644 --- a/Modules/_io/fileio.c +++ b/Modules/_io/fileio.c @@ -5,7 +5,6 @@ #include "pycore_fileutils.h" // _Py_BEGIN_SUPPRESS_IPH #include "pycore_object.h" // _PyObject_GC_UNTRACK() #include "pycore_pyerrors.h" // _PyErr_ChainExceptions1() -#include "pycore_weakref.h" // FT_CLEAR_WEAKREFS() #include // bool #ifdef HAVE_UNISTD_H @@ -81,8 +80,6 @@ typedef struct { modified outside of the fileio object / Python (ex. gh-90102, GH-121941, gh-109523). */ struct _Py_stat_struct *stat_atopen; - PyObject *weakreflist; - PyObject *dict; } fileio; #define PyFileIO_Check(state, op) (PyObject_TypeCheck((op), state->PyFileIO_Type)) @@ -221,7 +218,6 @@ fileio_new(PyTypeObject *type, PyObject *args, PyObject *kwds) self->truncate = 0; self->stat_atopen = NULL; self->closefd = 1; - self->weakreflist = NULL; return (PyObject *) self; } @@ -562,15 +558,13 @@ fileio_traverse(PyObject *op, visitproc visit, void *arg) { fileio *self = PyFileIO_CAST(op); Py_VISIT(Py_TYPE(self)); - Py_VISIT(self->dict); - return 0; + return PyObject_VisitManagedDict(op, visit, arg); } static int fileio_clear(PyObject *op) { - fileio *self = PyFileIO_CAST(op); - Py_CLEAR(self->dict); + PyObject_ClearManagedDict(op); return 0; } @@ -588,7 +582,7 @@ fileio_dealloc(PyObject *op) PyMem_Free(self->stat_atopen); self->stat_atopen = NULL; } - FT_CLEAR_WEAKREFS(op, self->weakreflist); + PyObject_ClearWeakRefs(op); (void)fileio_clear(op); PyTypeObject *tp = Py_TYPE(op); @@ -1370,8 +1364,6 @@ static PyGetSetDef fileio_getsetlist[] = { static PyMemberDef fileio_members[] = { {"_finalizing", Py_T_BOOL, offsetof(fileio, finalizing), 0}, - {"__weaklistoffset__", Py_T_PYSSIZET, offsetof(fileio, weakreflist), Py_READONLY}, - {"__dictoffset__", Py_T_PYSSIZET, offsetof(fileio, dict), Py_READONLY}, {NULL} }; diff --git a/Modules/_io/iobase.c b/Modules/_io/iobase.c index 2b482687e85c5bc..13de04d3b9ce772 100644 --- a/Modules/_io/iobase.c +++ b/Modules/_io/iobase.c @@ -14,7 +14,6 @@ #include "pycore_long.h" // _PyLong_GetOne() #include "pycore_object.h" // _PyType_HasFeature() #include "pycore_pyerrors.h" // _PyErr_ChainExceptions1() -#include "pycore_weakref.h" // FT_CLEAR_WEAKREFS() #include // offsetof() #include "_iomodule.h" @@ -33,8 +32,6 @@ class _io._RawIOBase "PyObject *" "clinic_state()->PyRawIOBase_Type" typedef struct { PyObject_HEAD - PyObject *dict; - PyObject *weakreflist; } iobase; #define iobase_CAST(op) ((iobase *)(op)) @@ -383,15 +380,13 @@ iobase_traverse(PyObject *op, visitproc visit, void *arg) { iobase *self = iobase_CAST(op); Py_VISIT(Py_TYPE(self)); - Py_VISIT(self->dict); - return 0; + return PyObject_VisitManagedDict(op, visit, arg); } static int iobase_clear(PyObject *op) { - iobase *self = iobase_CAST(op); - Py_CLEAR(self->dict); + PyObject_ClearManagedDict(op); return 0; } @@ -400,8 +395,8 @@ iobase_clear(PyObject *op) static void iobase_dealloc(PyObject *op) { - /* NOTE: since IOBaseObject has its own dict, Python-defined attributes - are still available here for close() to use. + /* NOTE: since the managed dict is only cleared below, Python-defined + attributes are still available here for close() to use. However, if the derived class declares a __slots__, those slots are already gone. */ @@ -416,8 +411,8 @@ iobase_dealloc(PyObject *op) } PyTypeObject *tp = Py_TYPE(self); _PyObject_GC_UNTRACK(self); - FT_CLEAR_WEAKREFS(op, self->weakreflist); - Py_CLEAR(self->dict); + PyObject_ClearWeakRefs(op); + PyObject_ClearManagedDict(op); tp->tp_free(self); Py_DECREF(tp); } @@ -907,12 +902,6 @@ static PyGetSetDef iobase_getset[] = { {NULL} }; -static struct PyMemberDef iobase_members[] = { - {"__weaklistoffset__", Py_T_PYSSIZET, offsetof(iobase, weakreflist), Py_READONLY}, - {"__dictoffset__", Py_T_PYSSIZET, offsetof(iobase, dict), Py_READONLY}, - {NULL}, -}; - static PyType_Slot iobase_slots[] = { {Py_tp_dealloc, iobase_dealloc}, @@ -922,7 +911,6 @@ static PyType_Slot iobase_slots[] = { {Py_tp_iter, iobase_iter}, {Py_tp_iternext, iobase_iternext}, {Py_tp_methods, iobase_methods}, - {Py_tp_members, iobase_members}, {Py_tp_getset, iobase_getset}, {Py_tp_finalize, iobase_finalize}, {0, NULL}, @@ -932,7 +920,8 @@ PyType_Spec _Py_iobase_spec = { .name = "_io._IOBase", .basicsize = sizeof(iobase), .flags = (Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC | - Py_TPFLAGS_IMMUTABLETYPE), + Py_TPFLAGS_IMMUTABLETYPE | Py_TPFLAGS_MANAGED_DICT | + Py_TPFLAGS_MANAGED_WEAKREF), .slots = iobase_slots, }; diff --git a/Modules/_io/stringio.c b/Modules/_io/stringio.c index 25ebe8909278487..a5687207d65e089 100644 --- a/Modules/_io/stringio.c +++ b/Modules/_io/stringio.c @@ -1,8 +1,8 @@ #include "Python.h" #include // offsetof() +#include "pycore_critical_section.h" // _Py_CRITICAL_SECTION_ASSERT_OBJECT_LOCKED() #include "pycore_object.h" #include "pycore_unicodeobject.h" // _PyUnicode_ResizeCompact() -#include "pycore_weakref.h" // FT_CLEAR_WEAKREFS() #include "_iomodule.h" /* Implementation note: the buffer is always at least one character longer @@ -42,8 +42,6 @@ typedef struct { PyObject *readnl; PyObject *writenl; - PyObject *dict; - PyObject *weakreflist; _PyIO_State *module_state; } stringio; @@ -675,8 +673,7 @@ stringio_traverse(PyObject *op, visitproc visit, void *arg) Py_VISIT(self->readnl); Py_VISIT(self->writenl); Py_VISIT(self->decoder); - Py_VISIT(self->dict); - return 0; + return PyObject_VisitManagedDict(op, visit, arg); } static int @@ -686,7 +683,7 @@ stringio_clear(PyObject *op) Py_CLEAR(self->readnl); Py_CLEAR(self->writenl); Py_CLEAR(self->decoder); - Py_CLEAR(self->dict); + PyObject_ClearManagedDict(op); return 0; } @@ -703,7 +700,7 @@ stringio_dealloc(PyObject *op) } PyUnicodeWriter_Discard(self->writer); (void)stringio_clear(op); - FT_CLEAR_WEAKREFS(op, self->weakreflist); + PyObject_ClearWeakRefs(op); tp->tp_free(self); Py_DECREF(tp); } @@ -930,11 +927,16 @@ _io_StringIO___getstate___impl(stringio *self) if (initvalue == NULL) return NULL; - if (self->dict == NULL) { + if (_PyObject_IsInstanceDictEmpty((PyObject *)self)) { dict = Py_NewRef(Py_None); } else { - dict = PyDict_Copy(self->dict); + dict = PyObject_GenericGetDict((PyObject *)self, NULL); + if (dict == NULL) { + Py_DECREF(initvalue); + return NULL; + } + Py_SETREF(dict, PyDict_Copy(dict)); if (dict == NULL) { Py_DECREF(initvalue); return NULL; @@ -1045,14 +1047,16 @@ _io_StringIO___setstate___impl(stringio *self, PyObject *state) Py_TYPE(dict)->tp_name); return NULL; } - if (self->dict) { - /* Alternatively, we could replace the internal dictionary - completely. However, it seems more practical to just update it. */ - if (PyDict_Update(self->dict, dict) < 0) - return NULL; + /* Alternatively, we could replace the internal dictionary + completely. However, it seems more practical to just update it. */ + PyObject *self_dict = PyObject_GenericGetDict((PyObject *)self, NULL); + if (self_dict == NULL) { + return NULL; } - else { - self->dict = Py_NewRef(dict); + int rc = PyDict_Update(self_dict, dict); + Py_DECREF(self_dict); + if (rc < 0) { + return NULL; } } @@ -1138,12 +1142,6 @@ static PyGetSetDef stringio_getset[] = { {NULL} }; -static struct PyMemberDef stringio_members[] = { - {"__weaklistoffset__", Py_T_PYSSIZET, offsetof(stringio, weakreflist), Py_READONLY}, - {"__dictoffset__", Py_T_PYSSIZET, offsetof(stringio, dict), Py_READONLY}, - {NULL}, -}; - static PyType_Slot stringio_slots[] = { {Py_tp_dealloc, stringio_dealloc}, {Py_tp_doc, (void *)_io_StringIO___init____doc__}, @@ -1151,7 +1149,6 @@ static PyType_Slot stringio_slots[] = { {Py_tp_clear, stringio_clear}, {Py_tp_iternext, stringio_iternext}, {Py_tp_methods, stringio_methods}, - {Py_tp_members, stringio_members}, {Py_tp_getset, stringio_getset}, {Py_tp_init, _io_StringIO___init__}, {Py_tp_new, stringio_new}, diff --git a/Modules/_io/textio.c b/Modules/_io/textio.c index 719f9ef36f49570..bb495ba5efc0327 100644 --- a/Modules/_io/textio.c +++ b/Modules/_io/textio.c @@ -16,7 +16,6 @@ #include "pycore_pyerrors.h" // _PyErr_ChainExceptions1() #include "pycore_pystate.h" // _PyInterpreterState_GET() #include "pycore_unicodeobject.h" // _PyUnicode_AsASCIIString() -#include "pycore_weakref.h" // FT_CLEAR_WEAKREFS() #include "_iomodule.h" @@ -731,8 +730,6 @@ struct textio /* Cache raw object if it's a FileIO object */ PyObject *raw; - PyObject *weakreflist; - PyObject *dict; _PyIO_State *state; }; @@ -1542,7 +1539,7 @@ textiowrapper_clear(PyObject *op) Py_CLEAR(self->errors); Py_CLEAR(self->raw); - Py_CLEAR(self->dict); + PyObject_ClearManagedDict(op); return 0; } @@ -1556,7 +1553,7 @@ textiowrapper_dealloc(PyObject *op) return; self->ok = 0; _PyObject_GC_UNTRACK(self); - FT_CLEAR_WEAKREFS(op, self->weakreflist); + PyObject_ClearWeakRefs(op); (void)textiowrapper_clear(op); tp->tp_free(self); Py_DECREF(tp); @@ -1578,8 +1575,7 @@ textiowrapper_traverse(PyObject *op, visitproc visit, void *arg) Py_VISIT(self->errors); Py_VISIT(self->raw); - Py_VISIT(self->dict); - return 0; + return PyObject_VisitManagedDict(op, visit, arg); } static PyObject * @@ -1946,6 +1942,176 @@ _io_TextIOWrapper_write_impl(textio *self, PyObject *text) return textiowrapper_write_text(self, text); } +static int +textiowrapper_write_list(textio *self, PyObject *strlist) +{ + /* Let normal method lookup handle a write override or a non-inline dict. */ + PyObject *override; + if (!_PyObject_TryGetInstanceAttribute((PyObject *)self, &_Py_ID(write), + &override)) { + return 0; + } + if (override != NULL) { + Py_DECREF(override); + return 0; + } + /* A custom buffer's closed getter can run Python code. Let write() + check it once, after the caller has joined the strings. */ + if (self->raw == NULL && + (self->buffer == NULL || + !Py_IS_TYPE(self->buffer, self->state->PyBytesIO_Type))) { + return 0; + } + if (textiowrapper_check_write(self) == NULL) { + return -1; + } + + /* Stateful codecs must see the whole string. */ + if (!is_asciicompat_encoding(self->encodefunc)) { + return 0; + } + + /* Reserve the known byte lengths up front to avoid repeatedly copying + large prefixes as the writer grows. For uncached UTF-8, the character + count is a lower bound; encoding can still grow the writer as needed. */ + Py_ssize_t capacity = 0; + for (Py_ssize_t i = 0; i < PyList_GET_SIZE(strlist); i++) { + PyObject *text = PyList_GET_ITEM(strlist, i); + assert(PyUnicode_Check(text)); + Py_ssize_t size = PyUnicode_GET_LENGTH(text); + if (self->encodefunc == utf8_encode && !PyUnicode_IS_ASCII(text) && + _PyUnicode_UTF8(text) != NULL) { + (void)PyUnicode_AsUTF8AndSize(text, &size); + } + if (size > PY_SSIZE_T_MAX - capacity) { + PyErr_NoMemory(); + return -2; + } + capacity += size; + } + PyBytesWriter *writer = PyBytesWriter_Create(capacity); + if (writer == NULL) { + return -2; + } + /* Reset the logical size while keeping the reserved allocation. */ + if (PyBytesWriter_Resize(writer, 0) < 0) { + PyBytesWriter_Discard(writer); + return -2; + } + int needflush = 0; + if (self->line_buffering) { + /* print() normally ends with a newline. Check the last part first. */ + for (Py_ssize_t i = PyList_GET_SIZE(strlist); i > 0; i--) { + PyObject *text = PyList_GET_ITEM(strlist, i - 1); + Py_ssize_t size = PyUnicode_GET_LENGTH(text); + if (PyUnicode_FindChar(text, '\n', 0, size, 1) != -1 || + PyUnicode_FindChar(text, '\r', 0, size, 1) != -1) { + needflush = 1; + break; + } + } + } + + const char *writenl = self->writetranslate ? self->writenl : NULL; + for (Py_ssize_t i = 0; i < PyList_GET_SIZE(strlist); i++) { + PyObject *text = PyList_GET_ITEM(strlist, i); + assert(PyUnicode_Check(text)); + Py_ssize_t size = PyUnicode_GET_LENGTH(text); + const char *data; + PyObject *encoded = NULL; + if (PyUnicode_IS_ASCII(text) || + (self->encodefunc == latin1_encode && + PyUnicode_KIND(text) == PyUnicode_1BYTE_KIND)) { + data = PyUnicode_DATA(text); + } + else if (self->encodefunc == utf8_encode) { + if (_PyUnicode_UTF8(text) != NULL) { + data = PyUnicode_AsUTF8AndSize(text, &size); + } + else { + /* Even the "strict" error handler can be replaced and + reconfigure the stream. Handle surrogates in the fallback + so error handlers only see the entire joined string. */ + int kind = PyUnicode_KIND(text); + if (kind != PyUnicode_1BYTE_KIND) { + const void *unicode_data = PyUnicode_DATA(text); + for (Py_ssize_t j = 0; j < size; j++) { + Py_UCS4 ch = PyUnicode_READ(kind, unicode_data, j); + if (Py_UNICODE_IS_SURROGATE(ch)) { + PyBytesWriter_Discard(writer); + return 0; + } + } + } + /* Do not populate the cache just for this write. */ + encoded = _PyUnicode_AsUTF8String(text, "strict"); + if (encoded == NULL) { + PyBytesWriter_Discard(writer); + return -2; + } + data = PyBytes_AS_STRING(encoded); + size = PyBytes_GET_SIZE(encoded); + } + } + else { + /* An ASCII or Latin-1 encoding error needs the joined string. */ + PyBytesWriter_Discard(writer); + return 0; + } + int res = 0; + if (writenl != NULL) { + /* These encodings represent '\n' as a single ASCII byte. */ + const char *end = data + size; + const char *lf; + while ((lf = memchr(data, '\n', end - data)) != NULL) { + if (PyBytesWriter_WriteBytes(writer, data, lf - data) < 0 || + PyBytesWriter_WriteBytes(writer, writenl, -1) < 0) { + res = -1; + break; + } + data = lf + 1; + } + size = end - data; + } + if (res == 0) { + res = PyBytesWriter_WriteBytes(writer, data, size); + } + Py_XDECREF(encoded); + if (res < 0) { + PyBytesWriter_Discard(writer); + return -2; + } + } + PyObject *b = PyBytesWriter_Finish(writer); + if (b == NULL) { + return -2; + } + self->encoding_start_of_stream = 0; + if (textiowrapper_write_bytes(self, b, needflush, self->write_through) < 0) { + return -1; + } + return 1; +} + +int +_PyTextIOWrapper_WriteList(PyObject *file, PyObject *strlist) +{ + assert(PyList_CheckExact(strlist)); + PyObject *module = PyType_GetModule_DuringGC(Py_TYPE(file)); + if (module == NULL || _PyModule_GetDefOrNull(module) != &_PyIO_Module) { + return 0; + } + _PyIO_State *state = get_io_state(module); + if (!Py_IS_TYPE(file, state->PyTextIOWrapper_Type)) { + return 0; + } + int res; + Py_BEGIN_CRITICAL_SECTION(file); + res = textiowrapper_write_list(textio_CAST(file), strlist); + Py_END_CRITICAL_SECTION(); + return res; +} + /* Steal a reference to chars and store it in the decoded_char buffer; */ static void @@ -3565,8 +3731,6 @@ static PyMemberDef textiowrapper_members[] = { {"line_buffering", Py_T_BOOL, offsetof(textio, line_buffering), Py_READONLY}, {"write_through", Py_T_BOOL, offsetof(textio, write_through), Py_READONLY}, {"_finalizing", Py_T_BOOL, offsetof(textio, finalizing), 0}, - {"__weaklistoffset__", Py_T_PYSSIZET, offsetof(textio, weakreflist), Py_READONLY}, - {"__dictoffset__", Py_T_PYSSIZET, offsetof(textio, dict), Py_READONLY}, {NULL} }; diff --git a/Modules/_io/winconsoleio.c b/Modules/_io/winconsoleio.c index 3da4f1df1708a78..a6e884fb9c5fc82 100644 --- a/Modules/_io/winconsoleio.c +++ b/Modules/_io/winconsoleio.c @@ -10,7 +10,6 @@ #include "pycore_fileutils.h" // _Py_BEGIN_SUPPRESS_IPH #include "pycore_object.h" // _PyObject_GC_UNTRACK() #include "pycore_pyerrors.h" // _PyErr_ChainExceptions1() -#include "pycore_weakref.h" // FT_CLEAR_WEAKREFS() #ifdef HAVE_WINDOWS_CONSOLE_IO @@ -216,8 +215,6 @@ typedef struct { unsigned int closefd : 1; char finalizing; unsigned int blksize; - PyObject *weakreflist; - PyObject *dict; char buf[SMALLBUF]; wchar_t wbuf; } winconsoleio; @@ -300,7 +297,6 @@ winconsoleio_new(PyTypeObject *type, PyObject *args, PyObject *kwds) self->writable = 0; self->closefd = 0; self->blksize = 0; - self->weakreflist = NULL; } return (PyObject *) self; @@ -498,15 +494,13 @@ winconsoleio_traverse(PyObject *op, visitproc visit, void *arg) { winconsoleio *self = winconsoleio_CAST(op); Py_VISIT(Py_TYPE(self)); - Py_VISIT(self->dict); - return 0; + return PyObject_VisitManagedDict(op, visit, arg); } static int winconsoleio_clear(PyObject *op) { - winconsoleio *self = winconsoleio_CAST(op); - Py_CLEAR(self->dict); + PyObject_ClearManagedDict(op); return 0; } @@ -519,8 +513,8 @@ winconsoleio_dealloc(PyObject *op) if (_PyIOBase_finalize(op) < 0) return; _PyObject_GC_UNTRACK(self); - FT_CLEAR_WEAKREFS(op, self->weakreflist); - Py_CLEAR(self->dict); + PyObject_ClearWeakRefs(op); + PyObject_ClearManagedDict(op); tp->tp_free(self); Py_DECREF(tp); } @@ -1097,7 +1091,10 @@ _io__WindowsConsoleIO_write_impl(winconsoleio *self, PyTypeObject *cls, len = Py_MIN(len, max_wlen * 3); while (1) { /* Fix for github issues gh-110913 and gh-82052. */ - len = _find_last_utf8_boundary(b->buf, len); + DWORD boundary = _find_last_utf8_boundary(b->buf, len); + if (boundary != 0) { + len = boundary; + } wlen = MultiByteToWideChar(CP_UTF8, 0, b->buf, len, NULL, 0); if (wlen <= max_wlen) { break; @@ -1258,8 +1255,6 @@ static PyGetSetDef winconsoleio_getsetlist[] = { static PyMemberDef winconsoleio_members[] = { {"_blksize", Py_T_UINT, offsetof(winconsoleio, blksize), 0}, {"_finalizing", Py_T_BOOL, offsetof(winconsoleio, finalizing), 0}, - {"__weaklistoffset__", Py_T_PYSSIZET, offsetof(winconsoleio, weakreflist), Py_READONLY}, - {"__dictoffset__", Py_T_PYSSIZET, offsetof(winconsoleio, dict), Py_READONLY}, {NULL} }; diff --git a/Modules/_posixsubprocess.c b/Modules/_posixsubprocess.c index 415e0afbe88125e..8ad01414ad80755 100644 --- a/Modules/_posixsubprocess.c +++ b/Modules/_posixsubprocess.c @@ -373,10 +373,10 @@ _close_range_except(int start_fd, return 0; } -#if defined(HAVE_GETDENTS64) \ +#if defined(_Py_HAVE_GETDENTS64) \ || (defined(__linux__) && defined(HAVE_SYS_SYSCALL_H)) -#ifdef HAVE_GETDENTS64 +#ifdef _Py_HAVE_GETDENTS64 # define py_dirent64 dirent64 #else /* It doesn't matter if d_name has room for NAME_MAX chars; we're using this @@ -392,7 +392,7 @@ struct py_dirent64 { unsigned char d_type; char d_name[256]; /* Filename (null-terminated) */ }; -#endif // !HAVE_GETDENTS64 +#endif // !_Py_HAVE_GETDENTS64 static int _brute_force_closer(int first, int last) @@ -435,7 +435,7 @@ _close_open_fds_safe(int start_fd, int *fds_to_keep, Py_ssize_t fds_to_keep_len) char buffer[sizeof(struct py_dirent64)]; Py_ssize_t bytes; while (1) { -#ifdef HAVE_GETDENTS64 +#ifdef _Py_HAVE_GETDENTS64 /* musl <1.2.4 headers expose `getdents64` as an alias for `getdents`, whose buffer parameter * is `struct dirent *`. We use `void *` to support both that interface and the glibc declaration * without an incompatible-pointer diagnostic. diff --git a/Modules/_remote_debugging/_remote_debugging.h b/Modules/_remote_debugging/_remote_debugging.h index 6080dc4e3815067..532fed28741738b 100644 --- a/Modules/_remote_debugging/_remote_debugging.h +++ b/Modules/_remote_debugging/_remote_debugging.h @@ -121,7 +121,6 @@ typedef struct { #define SIZEOF_GEN_OBJ sizeof(PyGenObject) #define SIZEOF_INTERP_FRAME sizeof(_PyInterpreterFrame) #define SIZEOF_LLIST_NODE sizeof(struct llist_node) -#define SIZEOF_PAGE_CACHE_ENTRY sizeof(page_cache_entry_t) #define SIZEOF_PYOBJECT sizeof(PyObject) #define SIZEOF_SET_OBJ sizeof(PySetObject) #define SIZEOF_TASK_OBJ 4096 @@ -575,7 +574,6 @@ extern int parse_frame_object( RemoteUnwinderObject *unwinder, PyObject** result, uintptr_t address, - uintptr_t* address_of_code_object, uintptr_t* previous_frame ); @@ -584,7 +582,6 @@ extern int parse_frame_from_chunks( PyObject **result, uintptr_t address, uintptr_t *previous_frame, - uintptr_t *stackpointer, StackChunkList *chunks ); diff --git a/Modules/_remote_debugging/asyncio.c b/Modules/_remote_debugging/asyncio.c index 11fa40eb3e5f00c..787f7921d7d74aa 100644 --- a/Modules/_remote_debugging/asyncio.c +++ b/Modules/_remote_debugging/asyncio.c @@ -386,8 +386,7 @@ parse_coro_chain( // Parse the previous frame using the gi_iframe from local copy uintptr_t prev_frame; uintptr_t gi_iframe_addr = coro_address + (uintptr_t)unwinder->debug_offsets.gen_object.gi_iframe; - uintptr_t address_of_code_object = 0; - if (parse_frame_object(unwinder, &name, gi_iframe_addr, &address_of_code_object, &prev_frame) < 0) { + if (parse_frame_object(unwinder, &name, gi_iframe_addr, &prev_frame) < 0) { set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to parse frame object in coro chain"); return -1; } @@ -422,13 +421,12 @@ parse_coro_chain( static PyObject* create_task_result( RemoteUnwinderObject *unwinder, - uintptr_t task_address + uintptr_t task_address, + char is_task ) { PyObject* result = NULL; PyObject *call_stack = NULL; PyObject *tn = NULL; - char task_obj[SIZEOF_TASK_OBJ]; - uintptr_t coro_addr; // Create call_stack first since it's the first tuple element call_stack = PyList_New(0); @@ -444,25 +442,27 @@ create_task_result( goto error; } - // Parse coroutine chain - if (_Py_RemoteDebug_PagedReadRemoteMemory(&unwinder->handle, task_address, - (size_t)unwinder->async_debug_offsets.asyncio_task_object.size, - task_obj) < 0) { - set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to read task object for coro chain"); - goto error; - } - - coro_addr = GET_MEMBER_NO_TAG(uintptr_t, task_obj, unwinder->async_debug_offsets.asyncio_task_object.task_coro); - - if ((void*)coro_addr != NULL) { - if (parse_coro_chain(unwinder, coro_addr, call_stack) < 0) { - set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to parse coroutine chain"); + if (is_task) { + char task_obj[SIZEOF_TASK_OBJ]; + if (_Py_RemoteDebug_PagedReadRemoteMemory(&unwinder->handle, task_address, + (size_t)unwinder->async_debug_offsets.asyncio_task_object.size, + task_obj) < 0) { + set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to read task object for coro chain"); goto error; } - if (PyList_Reverse(call_stack)) { - set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to reverse call stack"); - goto error; + uintptr_t coro_addr = GET_MEMBER_NO_TAG(uintptr_t, task_obj, unwinder->async_debug_offsets.asyncio_task_object.task_coro); + + if ((void*)coro_addr != NULL) { + if (parse_coro_chain(unwinder, coro_addr, call_stack) < 0) { + set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to parse coroutine chain"); + goto error; + } + + if (PyList_Reverse(call_stack)) { + set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to reverse call stack"); + goto error; + } } } @@ -506,33 +506,10 @@ parse_task( goto error; } - if (is_task) { - result = create_task_result(unwinder, task_address); - if (!result) { - set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to create task result"); - goto error; - } - } else { - // Create an empty CoroInfo for non-task objects - RemoteDebuggingState *state = RemoteDebugging_GetStateFromObject((PyObject*)unwinder); - result = PyStructSequence_New(state->CoroInfo_Type); - if (result == NULL) { - set_exception_cause(unwinder, PyExc_MemoryError, "Failed to create empty CoroInfo"); - goto error; - } - PyObject *empty_list = PyList_New(0); - if (empty_list == NULL) { - set_exception_cause(unwinder, PyExc_MemoryError, "Failed to create empty list"); - goto error; - } - PyObject *task_name = PyLong_FromUnsignedLongLong(task_address); - if (task_name == NULL) { - Py_DECREF(empty_list); - set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to create task name"); - goto error; - } - PyStructSequence_SetItem(result, 0, empty_list); // This steals the reference - PyStructSequence_SetItem(result, 1, task_name); // This steals the reference + result = create_task_result(unwinder, task_address, is_task); + if (result == NULL) { + set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to create task result"); + goto error; } if (PyList_Append(render_to, result)) { set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to append task result to render list"); @@ -798,14 +775,12 @@ parse_async_frame_chain( return -1; } PyObject* frame_info = NULL; - uintptr_t address_of_code_object; uintptr_t this_frame = address_of_current_frame; int res = parse_frame_object( unwinder, &frame_info, address_of_current_frame, - &address_of_code_object, &address_of_current_frame ); @@ -894,11 +869,11 @@ append_awaited_by_for_thread( return 0; } -int -append_awaited_by( +static int +append_awaited_info( RemoteUnwinderObject *unwinder, unsigned long tid, - uintptr_t head_addr, + PyObject *task_list, PyObject *result) { PyObject *tid_py = PyLong_FromUnsignedLong(tid); @@ -907,24 +882,16 @@ append_awaited_by( return -1; } - PyObject* awaited_by_for_thread = PyList_New(0); - if (awaited_by_for_thread == NULL) { - Py_DECREF(tid_py); - set_exception_cause(unwinder, PyExc_MemoryError, "Failed to create awaited_by thread list"); - return -1; - } - RemoteDebuggingState *state = RemoteDebugging_GetStateFromObject((PyObject*)unwinder); PyObject *result_item = PyStructSequence_New(state->AwaitedInfo_Type); if (result_item == NULL) { Py_DECREF(tid_py); - Py_DECREF(awaited_by_for_thread); set_exception_cause(unwinder, PyExc_MemoryError, "Failed to create AwaitedInfo"); return -1; } - PyStructSequence_SetItem(result_item, 0, tid_py); // steals ref - PyStructSequence_SetItem(result_item, 1, awaited_by_for_thread); // steals ref + PyStructSequence_SetItem(result_item, 0, tid_py); + PyStructSequence_SetItem(result_item, 1, Py_NewRef(task_list)); if (PyList_Append(result, result_item)) { Py_DECREF(result_item); set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to append awaited_by result item"); @@ -932,8 +899,30 @@ append_awaited_by( } Py_DECREF(result_item); - if (append_awaited_by_for_thread(unwinder, head_addr, awaited_by_for_thread)) - { + return 0; +} + +int +append_awaited_by( + RemoteUnwinderObject *unwinder, + unsigned long tid, + uintptr_t head_addr, + PyObject *result) +{ + PyObject *task_list = PyList_New(0); + if (task_list == NULL) { + set_exception_cause(unwinder, PyExc_MemoryError, "Failed to create awaited_by thread list"); + return -1; + } + + if (append_awaited_info(unwinder, tid, task_list, result) < 0) { + Py_DECREF(task_list); + return -1; + } + + int err = append_awaited_by_for_thread(unwinder, head_addr, task_list); + Py_DECREF(task_list); + if (err < 0) { set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to append awaited_by for thread"); return -1; } @@ -1038,32 +1027,9 @@ process_thread_for_async_stack_trace( return -1; } - // Create AwaitedInfo structure for this thread - PyObject *tid_py = PyLong_FromUnsignedLong(tid); - if (tid_py == NULL) { - Py_DECREF(task_list); - set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to create thread ID"); - return -1; - } - - RemoteDebuggingState *state = RemoteDebugging_GetStateFromObject((PyObject*)unwinder); - PyObject *awaited_info = PyStructSequence_New(state->AwaitedInfo_Type); - if (awaited_info == NULL) { - Py_DECREF(tid_py); - Py_DECREF(task_list); - set_exception_cause(unwinder, PyExc_MemoryError, "Failed to create AwaitedInfo"); - return -1; - } - - PyStructSequence_SetItem(awaited_info, 0, tid_py); // steals ref - PyStructSequence_SetItem(awaited_info, 1, task_list); // steals ref - - if (PyList_Append(result, awaited_info)) { - Py_DECREF(awaited_info); - set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to append AwaitedInfo to result"); - return -1; - } - Py_DECREF(awaited_info); + int err = append_awaited_info(unwinder, tid, task_list, result); + Py_DECREF(task_list); + return err; } return 0; diff --git a/Modules/_remote_debugging/binary_io.h b/Modules/_remote_debugging/binary_io.h index 6a2c5b795823e15..4f740d953f5dcc3 100644 --- a/Modules/_remote_debugging/binary_io.h +++ b/Modules/_remote_debugging/binary_io.h @@ -2,7 +2,7 @@ * Python Remote Debugging Module - Binary I/O Header * * This header provides declarations for high-performance binary file I/O - * for profiling data with optional zstd streaming compression. + * for profiling data with optional zstd compression. ******************************************************************************/ #ifndef Py_BINARY_IO_H @@ -23,7 +23,7 @@ extern "C" { #define BINARY_FORMAT_MAGIC 0x54414348 /* "TACH" (Tachyon) in native byte order */ #define BINARY_FORMAT_MAGIC_SWAPPED 0x48434154 /* Byte-swapped magic for endianness detection */ -#define BINARY_FORMAT_VERSION 1 +#define BINARY_FORMAT_VERSION 2 /* Sentinel values for optional frame fields */ #define OPCODE_NONE 255 /* No opcode captured (u8 sentinel) */ @@ -67,17 +67,32 @@ extern "C" { static_assert(FILE_HEADER_SIZE <= FILE_HEADER_PLACEHOLDER_SIZE, "FILE_HEADER_SIZE exceeds FILE_HEADER_PLACEHOLDER_SIZE"); -/* Sample header field offsets and sizes */ -#define SMP_OFF_THREAD_ID 0 -#define SMP_SIZE_THREAD_ID sizeof(uint64_t) -#define SMP_OFF_INTERPRETER_ID (SMP_OFF_THREAD_ID + SMP_SIZE_THREAD_ID) -#define SMP_SIZE_INTERPRETER_ID sizeof(uint32_t) -#define SMP_OFF_ENCODING (SMP_OFF_INTERPRETER_ID + SMP_SIZE_INTERPRETER_ID) -#define SMP_SIZE_ENCODING sizeof(uint8_t) -#define SAMPLE_HEADER_FIXED_SIZE (SMP_OFF_ENCODING + SMP_SIZE_ENCODING) - -static_assert(SAMPLE_HEADER_FIXED_SIZE == 13, - "SAMPLE_HEADER_FIXED_SIZE must remain 13"); +/* Sample blocks */ +#define BLOCK_MAGIC "TBLK" +#define BLOCK_MAGIC_SIZE (sizeof(BLOCK_MAGIC) - 1) +#define MAX_BLOCK_SIZE (1024 * 1024) +#define MAX_BLOCK_STREAMS 4096 + +/* Block columns */ +enum { + COLUMN_STREAM_TABLE, + COLUMN_SAMPLE_STREAM_IDS, + COLUMN_TIMESTAMP_RESIDUALS, + COLUMN_STATUS_BITMAPS, + COLUMN_STACK_INSTRUCTIONS, + COLUMN_FRAME_IDS, + COLUMN_COUNT +}; + +/* Stream table entry offsets and sizes */ +#define STM_OFF_THREAD_ID 0 +#define STM_SIZE_THREAD_ID sizeof(uint64_t) +#define STM_OFF_INTERPRETER_ID (STM_OFF_THREAD_ID + STM_SIZE_THREAD_ID) +#define STM_SIZE_INTERPRETER_ID sizeof(uint32_t) +#define STREAM_ENTRY_SIZE (STM_OFF_INTERPRETER_ID + STM_SIZE_INTERPRETER_ID) + +static_assert(STREAM_ENTRY_SIZE == 12, + "STREAM_ENTRY_SIZE must remain 12"); /* Footer field offsets and sizes */ #define FTR_OFF_STRINGS 0 @@ -128,14 +143,10 @@ static_assert(PROFILE_STATS_SIZE == 56, #define MIN_STRING_ENTRY_SIZE 1 #define MIN_FRAME_ENTRY_SIZE 7 -/* Buffer sizes: 512KB balances syscall amortization against memory use, - * and aligns well with filesystem block sizes and zstd dictionary windows */ -#define WRITE_BUFFER_SIZE (512 * 1024) -#define COMPRESSED_BUFFER_SIZE (512 * 1024) - /* Compression types */ #define COMPRESSION_NONE 0 #define COMPRESSION_ZSTD 1 +#define COLUMN_CONSTANT 2 /* Column stored as one repeated byte */ /* Profiling configuration. The mode occupies the low three bits and is * stored plus one so zero remains compatible with files written before this @@ -152,17 +163,24 @@ static_assert(PROFILE_STATS_SIZE == 56, /* Stack encoding types for delta compression */ #define STACK_REPEAT 0x00 /* RLE: identical to previous, with count */ -#define STACK_FULL 0x01 /* Full stack (first sample or no match) */ -#define STACK_SUFFIX 0x02 /* Shares N frames from bottom */ -#define STACK_POP_PUSH 0x03 /* Remove M frames, add N frames */ +#define STACK_REPLACE_SUFFIX 0x01 /* Keep outer frames, append new ones */ +#define STACK_REUSE 0x02 /* Stack stored earlier in the block */ /* Maximum stack depth we'll buffer for delta encoding */ #define MAX_STACK_DEPTH 256 +/* Writer's stack cache size per block */ +#define STACK_CACHE_SIZE 4096 +#define STACK_CACHE_BUCKET_SIZE 2 +#define STACK_CACHE_BUCKET_COUNT \ + (STACK_CACHE_SIZE / STACK_CACHE_BUCKET_SIZE) + /* Initial capacities for dynamic arrays - sized to reduce reallocations */ #define INITIAL_STRING_CAPACITY 4096 #define INITIAL_FRAME_CAPACITY 4096 #define INITIAL_THREAD_CAPACITY 256 +#define INITIAL_BLOCK_CAPACITY 256 +#define INITIAL_BLOCK_STREAM_CAPACITY 16 /* ============================================================================ * STATISTICS STRUCTURES @@ -174,7 +192,7 @@ typedef struct { uint64_t repeat_samples; /* Total samples encoded via RLE */ uint64_t full_records; /* Number of full stack records */ uint64_t suffix_records; /* Number of suffix match records */ - uint64_t pop_push_records; /* Number of pop-push records */ + uint64_t reuse_records; /* Number of stack reuse records */ uint64_t total_frames_written;/* Total frame indices written */ uint64_t frames_saved; /* Frames avoided due to delta encoding */ uint64_t bytes_written; /* Total bytes written (before compression) */ @@ -186,7 +204,7 @@ typedef struct { uint64_t repeat_samples; /* Samples decoded from RLE */ uint64_t full_records; /* Full stack records decoded */ uint64_t suffix_records; /* Suffix match records decoded */ - uint64_t pop_push_records; /* Pop-push records decoded */ + uint64_t reuse_records; /* Stack reuse records decoded */ uint64_t total_samples; /* Total samples reconstructed */ uint64_t stack_reconstructions; /* Number of stack array reconstructions */ } BinaryReaderStats; @@ -245,8 +263,6 @@ typedef struct { #else void *cctx; /* Placeholder */ #endif - uint8_t *compressed_buffer; - size_t compressed_buffer_size; } ZstdCompressor; /* Frame entry - combines all frame data for better cache locality. @@ -273,22 +289,25 @@ typedef struct { uint8_t opcode; } FrameKey; -/* Thread entry - tracks per-thread state for delta encoding */ +/* One entry per (thread, interpreter) pair seen in the file */ typedef struct { uint64_t thread_id; - uint64_t prev_timestamp; uint32_t interpreter_id; + /* Stream index in the current block, or UINT32_MAX */ + uint32_t block_stream_index; +} ThreadEntry; + +/* Delta encoding state of a stream, reset with each block */ +typedef struct { + uint64_t prev_timestamp; + size_t thread_index; + uint32_t current_repeat; + /* Previous stack for delta encoding (frame indices, innermost first) */ - uint32_t *prev_stack; size_t prev_stack_depth; - size_t prev_stack_capacity; - - /* RLE pending buffer - samples waiting to be written as a repeat group */ - uint8_t *pending_rle; - size_t pending_rle_bytes; - size_t pending_rle_samples; -} ThreadEntry; + uint32_t prev_stack[MAX_STACK_DEPTH]; +} BlockStream; /* Limit errors occur before emitting an incomplete sample. Other write * failures may leave partial records and must prevent finalization. */ @@ -298,15 +317,65 @@ typedef enum { BINARY_WRITER_BROKEN, } BinaryWriterState; +/* Byte buffer for block data */ +typedef struct { + uint8_t *data; + size_t size; + size_t capacity; +} ByteBuffer; + +/* Column size and encoding */ +typedef struct { + uint32_t decoded_size; + uint32_t stored_size; + uint8_t encoding; +} ColumnDescriptor; + +/* Block header as stored in the file */ +typedef struct { + uint32_t sample_count; + uint64_t base_timestamp; + uint8_t frame_width; + uint8_t common_status_flags; + uint8_t status_bitmap_flags; + ColumnDescriptor columns[COLUMN_COUNT]; +} BlockHeader; + +/* Complete stack that later samples can REUSE */ +typedef struct { + Py_uhash_t hash; + uint32_t offset; + uint32_t depth; +} StackCacheEntry; + +/* REPEAT instruction position and count */ +typedef struct { + uint32_t offset; + uint32_t count; +} Repeat; + +/* Samples buffered until the next flush */ +typedef struct { + ByteBuffer columns[COLUMN_COUNT]; + ByteBuffer repeats; + ByteBuffer transformed; + ByteBuffer compressed; + ByteBuffer output; + StackCacheEntry cache[STACK_CACHE_BUCKET_COUNT][STACK_CACHE_BUCKET_SIZE]; + BlockStream *streams; + size_t stream_capacity; + uint64_t base_timestamp; + uint32_t sample_count; + uint32_t stream_count; + uint32_t max_frame; + size_t size; +} SampleBlock; + /* Main binary writer structure */ typedef struct { FILE *fp; BinaryWriterState state; - - /* Write buffer for batched I/O */ - uint8_t *write_buffer; - size_t buffer_pos; - size_t buffer_size; + SampleBlock *block; /* Compression */ int compression_type; @@ -353,17 +422,37 @@ typedef struct { * BINARY READER STRUCTURES * ============================================================================ */ -/* Per-thread state for stack reconstruction during replay */ +typedef struct { + uint32_t frame_index; + PyObject *frame_info; +} ReaderFrameCacheEntry; + +/* Decoded sample block */ +typedef struct { + BlockHeader header; + uint32_t stream_count; + uint32_t frame_count; + ByteBuffer columns[COLUMN_COUNT]; + const uint32_t *stream_ids; + const uint64_t *timestamps; + const uint8_t *statuses; + const uint32_t *frames; + size_t instruction_offset; + uint32_t frame_offset; +} DecodedBlock; + +/* Per-stream replay state */ typedef struct { uint64_t thread_id; uint32_t interpreter_id; uint64_t prev_timestamp; - - /* Reconstructed stack buffer (frame indices, innermost first) */ - uint32_t *current_stack; + uint32_t current_stack[MAX_STACK_DEPTH]; size_t current_stack_depth; - size_t current_stack_capacity; -} ReaderThreadState; + uint32_t pending_repeats; + int has_stack; + PyObject *batch; + uint8_t batch_status; +} ReaderBlockStream; /* Main binary reader structure */ typedef struct { @@ -378,9 +467,9 @@ typedef struct { /* Decompression state */ int compression_type; - /* Note: ZSTD_DCtx is not stored - created/freed during decompression */ - uint8_t *decompressed_data; - size_t decompressed_size; +#ifdef HAVE_ZSTD + ZSTD_DCtx *dctx; +#endif /* Header metadata */ uint8_t py_major; @@ -409,13 +498,24 @@ typedef struct { /* Parsed frame table: array of FrameEntry structures */ FrameEntry *frames; uint32_t frames_count; + ReaderFrameCacheEntry *frame_info_cache; + size_t frame_info_cache_size; /* Sample data region */ uint8_t *sample_data; size_t sample_data_size; - /* Per-thread state for stack reconstruction (used during replay) */ - ReaderThreadState *thread_states; + ByteBuffer block_data; + ByteBuffer block_stream_ids; + ByteBuffer block_timestamps; + ByteBuffer block_statuses; + ByteBuffer block_frames; + ReaderBlockStream *block_streams; + size_t block_stream_capacity; + int replaying; + + /* Threads seen during replay */ + ThreadEntry *thread_states; size_t thread_state_count; size_t thread_state_capacity; @@ -462,6 +562,14 @@ encode_varint_i32(uint8_t *buf, int32_t value) return encode_varint_u32(buf, zigzag); } +/* Encode signed 64-bit varint (zigzag encoding). Returns bytes written. */ +static inline size_t +encode_varint_i64(uint8_t *buf, int64_t value) +{ + uint64_t zigzag = ((uint64_t)value << 1) ^ (uint64_t)(value >> 63); + return encode_varint_u64(buf, zigzag); +} + /* Decode unsigned 64-bit varint (LEB128). Updates offset only on success. * On error (overflow or incomplete), offset is NOT updated, allowing callers * to detect errors via (offset == prev_offset) check. Sets PyErr on error. */ @@ -525,10 +633,64 @@ decode_varint_i32(const uint8_t *data, size_t *offset, size_t max_size) return (int32_t)((zigzag >> 1) ^ -(int32_t)(zigzag & 1)); } +/* Decode signed 64-bit varint (zigzag encoding). */ +static inline int64_t +decode_varint_i64(const uint8_t *data, size_t *offset, size_t max_size) +{ + size_t saved_offset = *offset; + uint64_t zigzag = decode_varint_u64(data, offset, max_size); + if (*offset == saved_offset) { + return 0; /* decode_varint_u64 already set PyErr */ + } + return (int64_t)((zigzag >> 1) ^ (0 - (zigzag & 1))); +} + /* ============================================================================ * SHARED UTILITY FUNCTIONS * ============================================================================ */ +/* Grow a byte buffer to hold at least size bytes */ +static inline int +buffer_reserve(ByteBuffer *buffer, size_t size) +{ + if (size <= buffer->capacity) { + return 0; + } + size_t capacity = buffer->capacity ? buffer->capacity + : INITIAL_BLOCK_CAPACITY; + while (capacity < size) { + if (capacity > SIZE_MAX / 2) { + PyErr_NoMemory(); + return -1; + } + capacity *= 2; + } + uint8_t *data = PyMem_Realloc(buffer->data, capacity); + if (data == NULL) { + PyErr_NoMemory(); + return -1; + } + buffer->data = data; + buffer->capacity = capacity; + return 0; +} + +static inline void +buffer_clear(ByteBuffer *buffer) +{ + PyMem_Free(buffer->data); + buffer->data = NULL; + buffer->size = 0; + buffer->capacity = 0; +} + +/* Bytes in one status bitmap */ +static inline size_t +status_bitmap_size(uint32_t sample_count) +{ + return ((size_t)sample_count + CHAR_BIT - 1) / CHAR_BIT; +} + /* Generic array growth - returns new pointer or NULL (sets PyErr_NoMemory) * Includes overflow checking for capacity doubling and allocation size. */ static inline void * @@ -716,6 +878,12 @@ void binary_reader_close(BinaryReader *reader); */ PyObject *binary_writer_get_stats(BinaryWriter *writer); +/* + * Decoded bytes of the samples buffered in the current block, not yet + * counted in the writer's bytes_written statistic. + */ +size_t binary_writer_buffered_size(BinaryWriter *writer); + /* * Get reader statistics as a Python dict. * diff --git a/Modules/_remote_debugging/binary_io_reader.c b/Modules/_remote_debugging/binary_io_reader.c index a51a2e475c9e06b..24079f26a137eb3 100644 --- a/Modules/_remote_debugging/binary_io_reader.c +++ b/Modules/_remote_debugging/binary_io_reader.c @@ -27,15 +27,14 @@ * CONSTANTS FOR BINARY FORMAT SIZES * ============================================================================ */ -/* File structure sizes */ -#define MIN_DECOMPRESS_BUFFER_SIZE (64 * 1024) /* Minimum decompression buffer */ - /* Progress callback frequency */ #define PROGRESS_CALLBACK_INTERVAL 1000 /* Cap per-batch RLE samples to bound the timestamp list (gh-151378) */ #define MAX_RLE_BATCH_SAMPLES 8192 +#define FRAME_INFO_CACHE_SIZE 1024 + /* ============================================================================ * BINARY READER IMPLEMENTATION * ============================================================================ */ @@ -230,119 +229,62 @@ reader_parse_profile_stats(BinaryReader *reader, const uint8_t *data, return 0; } -#ifdef HAVE_ZSTD -/* Maximum decompression buffer size to prevent memory exhaustion (1GB) */ -#define MAX_DECOMPRESS_SIZE (1ULL << 30) - -static inline int -reader_decompress_samples(BinaryReader *reader, const uint8_t *data) +static int +reader_decompress(BinaryReader *reader, const uint8_t *compressed_data, + size_t compressed_size, ByteBuffer *output, + size_t decoded_size) { - size_t compressed_size = reader->string_table_offset - FILE_HEADER_PLACEHOLDER_SIZE; - const uint8_t *compressed_data = data + FILE_HEADER_PLACEHOLDER_SIZE; - - /* Validate compressed data region */ - if (reader->string_table_offset < FILE_HEADER_PLACEHOLDER_SIZE) { - PyErr_SetString(PyExc_ValueError, "Invalid string table offset"); - return -1; - } - - ZSTD_DCtx *dctx = ZSTD_createDCtx(); - if (!dctx) { - PyErr_SetString(PyExc_MemoryError, "Failed to create zstd decompression context"); +#ifdef HAVE_ZSTD + /* The data must be exactly one zstd frame */ + size_t frame_size = ZSTD_findFrameCompressedSize(compressed_data, + compressed_size); + if (ZSTD_isError(frame_size) || frame_size != compressed_size) { + PyErr_SetString(PyExc_ValueError, "Corrupted zstd frame header"); return -1; } - /* Try to get exact decompressed size from frame header for optimal allocation */ + /* Check the decompressed size recorded in the frame header before + * allocating for it */ unsigned long long frame_content_size = ZSTD_getFrameContentSize(compressed_data, compressed_size); - size_t alloc_size; - if (frame_content_size == ZSTD_CONTENTSIZE_ERROR) { /* Corrupted frame header - fail early */ - ZSTD_freeDCtx(dctx); PyErr_SetString(PyExc_ValueError, "Corrupted zstd frame header"); return -1; - } else if (frame_content_size != ZSTD_CONTENTSIZE_UNKNOWN && - frame_content_size <= SIZE_MAX && - frame_content_size <= MAX_DECOMPRESS_SIZE) { - alloc_size = (size_t)frame_content_size; - } else { - alloc_size = ZSTD_DStreamOutSize() * 4; - if (alloc_size < MIN_DECOMPRESS_BUFFER_SIZE) { - alloc_size = MIN_DECOMPRESS_BUFFER_SIZE; - } } - - reader->decompressed_data = PyMem_Malloc(alloc_size); - if (!reader->decompressed_data) { - ZSTD_freeDCtx(dctx); - PyErr_NoMemory(); + if (frame_content_size != decoded_size) { + PyErr_SetString(PyExc_ValueError, + "zstd frame size does not match the decoded size"); return -1; } - ZSTD_inBuffer input = { compressed_data, compressed_size, 0 }; - size_t total_output = 0; - size_t last_result = 0; - - while (input.pos < input.size) { - if (total_output >= alloc_size) { - /* Check for overflow before doubling */ - if (alloc_size > SIZE_MAX / 2 || alloc_size * 2 > MAX_DECOMPRESS_SIZE) { - PyMem_Free(reader->decompressed_data); - reader->decompressed_data = NULL; - ZSTD_freeDCtx(dctx); - PyErr_SetString(PyExc_MemoryError, "Decompressed data exceeds maximum size"); - return -1; - } - size_t new_size = alloc_size * 2; - uint8_t *new_buf = PyMem_Realloc(reader->decompressed_data, new_size); - if (!new_buf) { - PyMem_Free(reader->decompressed_data); - reader->decompressed_data = NULL; - ZSTD_freeDCtx(dctx); - PyErr_NoMemory(); - return -1; - } - reader->decompressed_data = new_buf; - alloc_size = new_size; - } - - ZSTD_outBuffer output = { - reader->decompressed_data + total_output, - alloc_size - total_output, - 0 - }; - - last_result = ZSTD_decompressStream(dctx, &output, &input); - if (ZSTD_isError(last_result)) { - PyMem_Free(reader->decompressed_data); - reader->decompressed_data = NULL; - ZSTD_freeDCtx(dctx); - PyErr_Format(PyExc_ValueError, "zstd decompression error: %s", - ZSTD_getErrorName(last_result)); - return -1; - } - - _Py_MSAN_UNPOISON(output.dst, output.pos); - total_output += output.pos; + /* Append the decoded data to the output buffer */ + if (buffer_reserve(output, output->size + decoded_size) < 0) { + return -1; + } + uint8_t *decoded = output->data + output->size; + size_t result = ZSTD_decompressDCtx(reader->dctx, decoded, decoded_size, + compressed_data, compressed_size); + if (ZSTD_isError(result)) { + PyErr_Format(PyExc_ValueError, "zstd decompression error: %s", + ZSTD_getErrorName(result)); + return -1; } - /* Verify decompression is complete (last_result == 0 means frame is complete) */ - if (last_result != 0) { - PyMem_Free(reader->decompressed_data); - reader->decompressed_data = NULL; - ZSTD_freeDCtx(dctx); + /* Verify decompression is complete */ + if (result != decoded_size) { PyErr_SetString(PyExc_ValueError, "Incomplete zstd frame: data may be truncated"); return -1; } - ZSTD_freeDCtx(dctx); - reader->decompressed_size = total_output; - reader->sample_data = reader->decompressed_data; - reader->sample_data_size = reader->decompressed_size; - + _Py_MSAN_UNPOISON(decoded, decoded_size); + output->size += decoded_size; return 0; -} +#else + PyErr_SetString(PyExc_RuntimeError, + "File uses zstd compression but zstd support not compiled in"); + return -1; #endif +} /* Reject a table/run count whose entries cannot fit in the bytes still * available; a malicious file could otherwise drive a huge allocation. @@ -681,10 +623,11 @@ binary_reader_open(PyObject *path) goto error; } - /* Handle compressed data */ if (reader->compression_type == COMPRESSION_ZSTD) { #ifdef HAVE_ZSTD - if (reader_decompress_samples(reader, data) < 0) { + reader->dctx = ZSTD_createDCtx(); + if (!reader->dctx) { + PyErr_SetString(PyExc_MemoryError, "Failed to create zstd decompression context"); goto error; } #else @@ -692,10 +635,10 @@ binary_reader_open(PyObject *path) "File uses zstd compression but zstd support not compiled in"); goto error; #endif - } else { - reader->sample_data = data + FILE_HEADER_PLACEHOLDER_SIZE; - reader->sample_data_size = (size_t)reader->string_table_offset - FILE_HEADER_PLACEHOLDER_SIZE; } + reader->sample_data = data + FILE_HEADER_PLACEHOLDER_SIZE; + reader->sample_data_size = + (size_t)reader->string_table_offset - FILE_HEADER_PLACEHOLDER_SIZE; /* Parse string and frame tables */ if (reader_parse_string_table(reader, data, file_size) < 0) { @@ -713,7 +656,7 @@ binary_reader_open(PyObject *path) } /* Get or create reader thread state for stack reconstruction */ -static ReaderThreadState * +static ThreadEntry * reader_get_or_create_thread_state(BinaryReader *reader, uint64_t thread_id, uint32_t interpreter_id) { @@ -735,192 +678,104 @@ reader_get_or_create_thread_state(BinaryReader *reader, uint64_t thread_id, if (!reader->thread_states) { reader->thread_state_capacity = 16; - reader->thread_states = PyMem_Calloc(reader->thread_state_capacity, sizeof(ReaderThreadState)); + reader->thread_states = PyMem_Calloc(reader->thread_state_capacity, sizeof(ThreadEntry)); if (!reader->thread_states) { PyErr_NoMemory(); return NULL; } - } else if (reader->thread_state_count >= reader->thread_state_capacity) { - ReaderThreadState *new_states = grow_array(reader->thread_states, - &reader->thread_state_capacity, - sizeof(ReaderThreadState)); - if (!new_states) { - return NULL; - } - reader->thread_states = new_states; + } + else if (GROW_ARRAY(reader->thread_states, reader->thread_state_count, + reader->thread_state_capacity, ThreadEntry) < 0) { + return NULL; } - ReaderThreadState *ts = &reader->thread_states[reader->thread_state_count]; - memset(ts, 0, sizeof(ReaderThreadState)); + ThreadEntry *ts = &reader->thread_states[reader->thread_state_count]; + memset(ts, 0, sizeof(ThreadEntry)); ts->thread_id = thread_id; ts->interpreter_id = interpreter_id; - ts->prev_timestamp = reader->start_time_us; - ts->current_stack_capacity = MAX_STACK_DEPTH; - ts->current_stack = PyMem_Malloc(ts->current_stack_capacity * sizeof(uint32_t)); - if (!ts->current_stack) { - PyErr_NoMemory(); - return NULL; - } - // Increment count only after successful allocation to avoid - // leaving a half-initialized entry visible to future lookups reader->thread_state_count++; return ts; } -/* ============================================================================ - * STACK DECODING HELPERS - * ============================================================================ */ - -/* Validate that final_depth fits in the stack buffer. - * Uses uint64_t to prevent overflow on 32-bit platforms. */ -static inline int -validate_stack_depth(ReaderThreadState *ts, uint64_t final_depth) -{ - if (final_depth > ts->current_stack_capacity) { - PyErr_Format(PyExc_ValueError, - "Final stack depth %llu exceeds capacity %zu", - (unsigned long long)final_depth, ts->current_stack_capacity); - return -1; - } - return 0; -} - -/* Decode a full stack from sample data. - * Updates ts->current_stack and ts->current_stack_depth. - * Returns 0 on success, -1 on error (bounds violation). */ -static inline int -decode_stack_full(ReaderThreadState *ts, const uint8_t *data, - size_t *offset, size_t max_size) +static PyObject * +reader_get_frame_info(RemoteDebuggingState *state, BinaryReader *reader, + uint32_t frame_index) { - size_t prev_offset = *offset; - uint32_t depth = decode_varint_u32(data, offset, max_size); - if (*offset == prev_offset) { - return -1; - } - - /* Validate depth against capacity to prevent buffer overflow */ - if (depth > ts->current_stack_capacity) { - PyErr_Format(PyExc_ValueError, - "Stack depth %u exceeds capacity %zu", depth, ts->current_stack_capacity); - return -1; + if (frame_index >= reader->frames_count) { + PyErr_Format(PyExc_ValueError, "Invalid frame index: %u", frame_index); + return NULL; } - - ts->current_stack_depth = depth; - for (uint32_t i = 0; i < depth; i++) { - size_t prev_offset = *offset; - ts->current_stack[i] = decode_varint_u32(data, offset, max_size); - if (*offset == prev_offset) { - return -1; + if (reader->frame_info_cache == NULL) { + size_t cache_size = Py_MIN(reader->frames_count, + (uint32_t)FRAME_INFO_CACHE_SIZE); + reader->frame_info_cache = PyMem_Calloc( + cache_size, sizeof(*reader->frame_info_cache)); + if (reader->frame_info_cache == NULL) { + PyErr_NoMemory(); + return NULL; } + reader->frame_info_cache_size = cache_size; } - return 0; -} - -/* Decode a suffix-encoded stack from sample data. - * The suffix encoding shares frames from the bottom of the previous stack. - * Returns 0 on success, -1 on error (bounds violation). */ -static inline int -decode_stack_suffix(ReaderThreadState *ts, const uint8_t *data, - size_t *offset, size_t max_size) -{ - size_t prev_offset = *offset; - uint32_t shared = decode_varint_u32(data, offset, max_size); - if (*offset == prev_offset) { - return -1; - } - prev_offset = *offset; - uint32_t new_count = decode_varint_u32(data, offset, max_size); - if (*offset == prev_offset) { - return -1; - } - - /* Validate shared doesn't exceed current stack depth */ - if (shared > ts->current_stack_depth) { - PyErr_Format(PyExc_ValueError, - "Shared count %u exceeds current stack depth %zu", - shared, ts->current_stack_depth); - return -1; + ReaderFrameCacheEntry *entry = &reader->frame_info_cache[ + frame_index % reader->frame_info_cache_size]; + if (entry->frame_info != NULL && entry->frame_index == frame_index) { + return Py_NewRef(entry->frame_info); } - /* Use uint64_t to prevent overflow on 32-bit platforms */ - uint64_t final_depth = (uint64_t)shared + new_count; - if (validate_stack_depth(ts, final_depth) < 0) { - return -1; + FrameEntry *frame = &reader->frames[frame_index]; + if (frame->filename_idx >= reader->strings_count || + frame->funcname_idx >= reader->strings_count) { + PyErr_SetString(PyExc_ValueError, "Invalid string index in frame"); + return NULL; } - /* Move shared frames (from bottom of stack) to make room for new frames at the top */ - if (new_count > 0 && shared > 0) { - /* Defensive check: ensure subtraction won't underflow. - * This should already be guaranteed by the check above, but we add - * this assertion as defense-in-depth against stack corruption. */ - if (ts->current_stack_depth < shared) { - PyErr_SetString(PyExc_ValueError, - "Internal error: stack corruption detected in suffix decoding"); - return -1; - } - size_t prev_shared_start = ts->current_stack_depth - shared; - memmove(&ts->current_stack[new_count], - &ts->current_stack[prev_shared_start], - shared * sizeof(uint32_t)); + PyObject *frame_info = PyStructSequence_New(state->FrameInfo_Type); + if (!frame_info) { + return NULL; } - for (uint32_t i = 0; i < new_count; i++) { - size_t prev_offset = *offset; - ts->current_stack[i] = decode_varint_u32(data, offset, max_size); - if (*offset == prev_offset) { - return -1; + /* Build location tuple with full position info */ + PyObject *location; + if (frame->lineno != LOCATION_NOT_AVAILABLE) { + location = Py_BuildValue("(iiii)", + frame->lineno, + frame->end_lineno, + frame->column, + frame->end_column); + if (!location) { + Py_DECREF(frame_info); + return NULL; } } - ts->current_stack_depth = (size_t)final_depth; - return 0; -} - -/* Decode a pop-push encoded stack from sample data. - * Pops frames from the top and pushes new frames. - * Returns 0 on success, -1 on error (bounds violation). */ -static inline int -decode_stack_pop_push(ReaderThreadState *ts, const uint8_t *data, - size_t *offset, size_t max_size) -{ - size_t prev_offset = *offset; - uint32_t pop = decode_varint_u32(data, offset, max_size); - if (*offset == prev_offset) { - return -1; - } - prev_offset = *offset; - uint32_t push = decode_varint_u32(data, offset, max_size); - if (*offset == prev_offset) { - return -1; + else { + location = Py_NewRef(Py_None); } - size_t keep = (ts->current_stack_depth > pop) ? ts->current_stack_depth - pop : 0; - /* Use uint64_t to prevent overflow on 32-bit platforms */ - uint64_t final_depth = (uint64_t)keep + push; - if (validate_stack_depth(ts, final_depth) < 0) { - return -1; + /* Build opcode object */ + PyObject *opcode_obj; + if (frame->opcode != OPCODE_NONE) { + opcode_obj = PyLong_FromLong(frame->opcode); + if (!opcode_obj) { + Py_DECREF(location); + Py_DECREF(frame_info); + return NULL; + } } - - /* Move kept frames (from bottom of stack) to make room for new frames at the top. - * Even when push == 0, we need to move kept frames to index 0 if pop > 0. */ - if (keep > 0) { - memmove(&ts->current_stack[push], - &ts->current_stack[pop], - keep * sizeof(uint32_t)); + else { + opcode_obj = Py_NewRef(Py_None); } - for (uint32_t i = 0; i < push; i++) { - size_t prev_offset = *offset; - ts->current_stack[i] = decode_varint_u32(data, offset, max_size); - /* If offset didn't advance, varint decoding failed */ - if (*offset == prev_offset) { - return -1; - } - } - ts->current_stack_depth = (size_t)final_depth; - return 0; + PyStructSequence_SetItem(frame_info, 0, Py_NewRef(reader->strings[frame->filename_idx])); + PyStructSequence_SetItem(frame_info, 1, location); + PyStructSequence_SetItem(frame_info, 2, Py_NewRef(reader->strings[frame->funcname_idx])); + PyStructSequence_SetItem(frame_info, 3, opcode_obj); + Py_XSETREF(entry->frame_info, Py_NewRef(frame_info)); + entry->frame_index = frame_index; + return frame_info; } -/* Build a Python list of FrameInfo objects from frame indices */ +/* Build a Python list of FrameInfo objects from frame indices. Stacks are + * kept outermost first, but collectors expect innermost first. */ static PyObject * build_frame_list(RemoteDebuggingState *state, BinaryReader *reader, const uint32_t *frame_indices, size_t stack_depth) @@ -931,60 +786,11 @@ build_frame_list(RemoteDebuggingState *state, BinaryReader *reader, } for (size_t k = 0; k < stack_depth; k++) { - uint32_t frame_idx = frame_indices[k]; - if (frame_idx >= reader->frames_count) { - PyErr_Format(PyExc_ValueError, "Invalid frame index: %u", frame_idx); - goto error; - } - - FrameEntry *frame = &reader->frames[frame_idx]; - - if (frame->filename_idx >= reader->strings_count || - frame->funcname_idx >= reader->strings_count) { - PyErr_SetString(PyExc_ValueError, "Invalid string index in frame"); + PyObject *frame_info = reader_get_frame_info( + state, reader, frame_indices[stack_depth - 1 - k]); + if (frame_info == NULL) { goto error; } - - PyObject *frame_info = PyStructSequence_New(state->FrameInfo_Type); - if (!frame_info) { - goto error; - } - - /* Build location tuple with full position info */ - PyObject *location; - if (frame->lineno != LOCATION_NOT_AVAILABLE) { - location = Py_BuildValue("(iiii)", - frame->lineno, - frame->end_lineno, - frame->column, - frame->end_column); - if (!location) { - Py_DECREF(frame_info); - goto error; - } - } - else { - location = Py_NewRef(Py_None); - } - - /* Build opcode object */ - PyObject *opcode_obj; - if (frame->opcode != OPCODE_NONE) { - opcode_obj = PyLong_FromLong(frame->opcode); - if (!opcode_obj) { - Py_DECREF(location); - Py_DECREF(frame_info); - goto error; - } - } - else { - opcode_obj = Py_NewRef(Py_None); - } - - PyStructSequence_SetItem(frame_info, 0, Py_NewRef(reader->strings[frame->filename_idx])); - PyStructSequence_SetItem(frame_info, 1, location); - PyStructSequence_SetItem(frame_info, 2, Py_NewRef(reader->strings[frame->funcname_idx])); - PyStructSequence_SetItem(frame_info, 3, opcode_obj); PyList_SET_ITEM(frame_list, k, frame_info); } @@ -1101,252 +907,754 @@ invoke_progress_callback(PyObject *callback, Py_ssize_t current, uint64_t total) return 0; } -Py_ssize_t -binary_reader_replay(BinaryReader *reader, PyObject *collector, PyObject *progress_callback) +/* ============================================================================ + * SAMPLE BLOCKS + * ============================================================================ */ + +static int +block_error(const char *message) { - PyObject *collect_method; - int has_collect = PyObject_GetOptionalAttrString(collector, "collect", &collect_method); - if (has_collect < 0) { - return -1; + PyErr_Format(PyExc_ValueError, "Invalid sample block: %s", message); + return -1; +} + +static inline int +block_read_varint(const ByteBuffer *column, size_t *offset, uint64_t *value) +{ + size_t prev_offset = *offset; + *value = decode_varint_u64(column->data, offset, column->size); + return *offset == prev_offset ? -1 : 0; +} + +static int +block_read_varint_u32(const ByteBuffer *column, size_t *offset, + uint32_t *value) +{ + size_t prev_offset = *offset; + *value = decode_varint_u32(column->data, offset, column->size); + return *offset == prev_offset ? -1 : 0; +} + +static int +reader_parse_block_header(BinaryReader *reader, const ByteBuffer *section, + size_t *offset, BlockHeader *header) +{ + /* Block magic "TBLK" */ + if (section->size - *offset < BLOCK_MAGIC_SIZE) { + return block_error("truncated header"); } - if (has_collect == 0) { - PyErr_SetString(PyExc_TypeError, "Collector must have a collect() method"); - return -1; + if (memcmp(section->data + *offset, BLOCK_MAGIC, BLOCK_MAGIC_SIZE) != 0) { + return block_error("wrong magic"); } - Py_DECREF(collect_method); + *offset += BLOCK_MAGIC_SIZE; - /* Get module state for struct sequence types */ - PyObject *module = PyImport_ImportModule("_remote_debugging"); - if (!module) { + /* Sample count and base timestamp (varints) */ + if (block_read_varint_u32(section, offset, &header->sample_count) < 0 || + block_read_varint(section, offset, &header->base_timestamp) < 0) { return -1; } - RemoteDebuggingState *state = RemoteDebugging_GetState(module); - Py_DECREF(module); - if (!state) { - PyErr_SetString(PyExc_RuntimeError, "Failed to get module state"); - return -1; + /* Frame ID width and the two status flag masks (1 byte each) */ + if (section->size - *offset < 3) { + return block_error("truncated header"); } - - size_t offset = 0; - Py_ssize_t replayed = 0; - - /* Initial progress callback at 0% */ - if (invoke_progress_callback(progress_callback, 0, reader->sample_count) < 0) { - return -1; + header->frame_width = section->data[(*offset)++]; + header->common_status_flags = section->data[(*offset)++]; + header->status_bitmap_flags = section->data[(*offset)++]; + if (header->sample_count == 0) { + return block_error("no samples"); + } + if (header->frame_width < 1 || header->frame_width > sizeof(uint32_t)) { + return block_error("frame ID width out of range"); + } + if (header->common_status_flags & header->status_bitmap_flags) { + return block_error("common and varying status flags overlap"); } - while (offset < reader->sample_data_size) { - /* Read thread_id (8 bytes) + interpreter_id (4 bytes) + encoding byte */ - if (reader->sample_data_size - offset < SAMPLE_HEADER_FIXED_SIZE) { - PyErr_Format(PyExc_ValueError, - "Truncated sample data: %zu trailing bytes", - reader->sample_data_size - offset); + size_t decoded_total = 0; + for (unsigned column = 0; column < COLUMN_COUNT; column++) { + ColumnDescriptor *descriptor = &header->columns[column]; + /* Decoded size (varint) and encoding (1 byte) */ + if (block_read_varint_u32(section, offset, + &descriptor->decoded_size) < 0) { return -1; } + if (*offset >= section->size) { + return block_error("truncated column descriptor"); + } + descriptor->encoding = section->data[(*offset)++]; + + /* Stored size: implied for raw and constant, a varint for zstd */ + switch (descriptor->encoding) { + case COMPRESSION_NONE: + descriptor->stored_size = descriptor->decoded_size; + break; + case COLUMN_CONSTANT: + descriptor->stored_size = 1; + break; + case COMPRESSION_ZSTD: + if (reader->compression_type != COMPRESSION_ZSTD) { + return block_error("zstd column in an uncompressed file"); + } + if (block_read_varint_u32(section, offset, + &descriptor->stored_size) < 0) { + return -1; + } + break; + default: + return block_error("unknown column encoding"); + } - /* Use memcpy to avoid strict aliasing violations, then byte-swap if needed */ - uint64_t thread_id_raw; - uint32_t interpreter_id_raw; - memcpy(&thread_id_raw, &reader->sample_data[offset], SMP_SIZE_THREAD_ID); - offset += SMP_SIZE_THREAD_ID; + if (descriptor->encoding != COMPRESSION_NONE && + descriptor->decoded_size == 0) { + return block_error("encoded column is empty"); + } + if (descriptor->decoded_size > MAX_BLOCK_SIZE - decoded_total) { + return block_error("decoded columns exceed the block limit"); + } + decoded_total += descriptor->decoded_size; + } - memcpy(&interpreter_id_raw, &reader->sample_data[offset], SMP_SIZE_INTERPRETER_ID); - offset += SMP_SIZE_INTERPRETER_ID; + /* Each sample needs at least one byte of stream index and residual */ + if (header->sample_count > + header->columns[COLUMN_SAMPLE_STREAM_IDS].decoded_size || + header->sample_count > + header->columns[COLUMN_TIMESTAMP_RESIDUALS].decoded_size) { + return block_error("more samples than stored stream indices"); + } + return 0; +} - uint64_t thread_id = SWAP64_IF(reader->needs_swap, thread_id_raw); - uint32_t interpreter_id = SWAP32_IF(reader->needs_swap, interpreter_id_raw); +/* A constant stream index or residual column that holds the same one-byte + * varint for every sample is read from its stored byte, not expanded. */ +static int +constant_varint_column(const DecodedBlock *block, unsigned column, + uint8_t value) +{ + const ColumnDescriptor *descriptor = &block->header.columns[column]; + return (column == COLUMN_SAMPLE_STREAM_IDS || + column == COLUMN_TIMESTAMP_RESIDUALS) && + descriptor->encoding == COLUMN_CONSTANT && + descriptor->decoded_size == block->header.sample_count && + value < 0x80; +} - /* Get or create thread state for reconstruction */ - ReaderThreadState *ts = reader_get_or_create_thread_state(reader, thread_id, interpreter_id); - if (!ts) { - return -1; +static int +decode_block_columns(BinaryReader *reader, const ByteBuffer *section, + size_t *offset, DecodedBlock *block) +{ + const ColumnDescriptor *descriptors = block->header.columns; + /* Find where each column is stored and how much of the shared buffer + * the expanded and decompressed columns need. The buffer is reused + * across blocks and reserved up front so decoded columns never move. */ + uint8_t *stored[COLUMN_COUNT]; + size_t buffered_total = 0; + for (unsigned column = 0; column < COLUMN_COUNT; column++) { + const ColumnDescriptor *descriptor = &descriptors[column]; + if (descriptor->stored_size > section->size - *offset) { + return block_error("column extends past the sample section"); + } + stored[column] = section->data + *offset; + *offset += descriptor->stored_size; + if (descriptor->encoding == COMPRESSION_ZSTD || + (descriptor->encoding == COLUMN_CONSTANT && + !constant_varint_column(block, column, stored[column][0]))) { + buffered_total += descriptor->decoded_size; } + } + ByteBuffer *buffer = &reader->block_data; + if (buffer_reserve(buffer, buffered_total) < 0) { + return -1; + } - /* Read encoding byte */ - uint8_t encoding = reader->sample_data[offset++]; - - switch (encoding) { - case STACK_REPEAT: { - /* RLE repeat: [count: varint] [delta: varint, status: 1]... */ - size_t prev_offset = offset; - uint32_t count = decode_varint_u32(reader->sample_data, &offset, reader->sample_data_size); - /* Detect varint decode failure */ - if (offset == prev_offset) { - PyErr_SetString(PyExc_ValueError, "Malformed varint for RLE count"); - return -1; + buffer->size = 0; + for (unsigned column = 0; column < COLUMN_COUNT; column++) { + const ColumnDescriptor *descriptor = &descriptors[column]; + uint8_t *stored_data = stored[column]; + ByteBuffer *decoded = &block->columns[column]; + decoded->size = descriptor->decoded_size; + switch (descriptor->encoding) { + case COMPRESSION_NONE: + /* Raw columns point into the file data */ + decoded->data = stored_data; + break; + case COLUMN_CONSTANT: + if (constant_varint_column(block, column, stored_data[0])) { + decoded->data = stored_data; + decoded->size = 1; + break; } - - /* Reject a count larger than the remaining bytes can hold; each - * RLE sample needs at least 2 bytes (1-byte min varint + status). */ - if (reader_validate_count("RLE", count, - reader->sample_data_size - offset, 2) < 0) { + decoded->data = buffer->data + buffer->size; + memset(decoded->data, stored_data[0], decoded->size); + buffer->size += decoded->size; + break; + case COMPRESSION_ZSTD: + decoded->data = buffer->data + buffer->size; + if (reader_decompress(reader, stored_data, + descriptor->stored_size, buffer, + decoded->size) < 0) { return -1; } - if ((uint64_t)count > (uint64_t)PY_SSIZE_T_MAX - (uint64_t)replayed) { - PyErr_SetString(PyExc_OverflowError, - "Sample count exceeds Py_ssize_t maximum"); + break; + default: + return block_error("unknown column encoding"); + } + } + return 0; +} + +static int +validate_block_columns(DecodedBlock *block) +{ + size_t stream_table_size = block->columns[COLUMN_STREAM_TABLE].size; + if (stream_table_size % STREAM_ENTRY_SIZE != 0) { + return block_error("stream table has a partial entry"); + } + size_t stream_count = stream_table_size / STREAM_ENTRY_SIZE; + if (stream_count == 0 || stream_count > MAX_BLOCK_STREAMS) { + return block_error("stream count out of range"); + } + block->stream_count = (uint32_t)stream_count; + + size_t frame_ids_size = block->columns[COLUMN_FRAME_IDS].size; + if (frame_ids_size % block->header.frame_width != 0) { + return block_error("frame ID column has a partial value"); + } + block->frame_count = + (uint32_t)(frame_ids_size / block->header.frame_width); + + size_t plane_size = status_bitmap_size(block->header.sample_count); + size_t plane_count = + (size_t)_Py_popcount32(block->header.status_bitmap_flags); + const ByteBuffer *statuses = &block->columns[COLUMN_STATUS_BITMAPS]; + if (statuses->size != plane_size * plane_count) { + return block_error("status bitmaps have the wrong size"); + } + unsigned used_bits = block->header.sample_count % CHAR_BIT; + for (size_t plane = 0; used_bits && plane < plane_count; plane++) { + if (statuses->data[(plane + 1) * plane_size - 1] >> used_bits) { + return block_error("status bitmap padding is not zero"); + } + } + return 0; +} + +static int +reserve_block_streams(BinaryReader *reader, uint32_t stream_count) +{ + if (stream_count <= reader->block_stream_capacity) { + return 0; + } + ReaderBlockStream *streams = PyMem_Realloc( + reader->block_streams, stream_count * sizeof(ReaderBlockStream)); + if (streams == NULL) { + PyErr_NoMemory(); + return -1; + } + reader->block_streams = streams; + reader->block_stream_capacity = stream_count; + return 0; +} + +static int +reader_parse_stream_table(BinaryReader *reader, const DecodedBlock *block) +{ + const uint8_t *stream_table = block->columns[COLUMN_STREAM_TABLE].data; + uint64_t initial_prev_timestamp = + block->header.base_timestamp - reader->sample_interval_us; + memset(reader->block_streams, 0, + block->stream_count * sizeof(ReaderBlockStream)); + for (uint32_t i = 0; i < block->stream_count; i++) { + /* Read thread_id (8 bytes) + interpreter_id (4 bytes) */ + const uint8_t *entry = stream_table + (size_t)i * STREAM_ENTRY_SIZE; + uint64_t thread_id; + uint32_t interpreter_id; + memcpy(&thread_id, entry + STM_OFF_THREAD_ID, STM_SIZE_THREAD_ID); + memcpy(&interpreter_id, entry + STM_OFF_INTERPRETER_ID, + STM_SIZE_INTERPRETER_ID); + ThreadEntry *thread = reader_get_or_create_thread_state( + reader, SWAP64_IF(reader->needs_swap, thread_id), + SWAP32_IF(reader->needs_swap, interpreter_id)); + if (thread == NULL) { + return -1; + } + /* Each thread may appear only once in a block. An index left over + * from an earlier block points at a different thread here. */ + uint32_t previous_index = thread->block_stream_index; + if (previous_index < i && + reader->block_streams[previous_index].thread_id == + thread->thread_id && + reader->block_streams[previous_index].interpreter_id == + thread->interpreter_id) { + return block_error("stream table repeats a thread"); + } + thread->block_stream_index = i; + + /* Each stream's first expected timestamp is the base timestamp */ + ReaderBlockStream *stream = &reader->block_streams[i]; + stream->thread_id = thread->thread_id; + stream->interpreter_id = thread->interpreter_id; + stream->prev_timestamp = initial_prev_timestamp; + } + return 0; +} + +static int +decode_samples(BinaryReader *reader, DecodedBlock *block) +{ + size_t sample_count = block->header.sample_count; + if (buffer_reserve(&reader->block_stream_ids, + sample_count * sizeof(uint32_t)) < 0 || + buffer_reserve(&reader->block_timestamps, + sample_count * sizeof(uint64_t)) < 0) { + return -1; + } + uint32_t *stream_ids = (uint32_t *)reader->block_stream_ids.data; + uint64_t *timestamps = (uint64_t *)reader->block_timestamps.data; + const ByteBuffer *stream_id_column = + &block->columns[COLUMN_SAMPLE_STREAM_IDS]; + const ByteBuffer *residual_column = + &block->columns[COLUMN_TIMESTAMP_RESIDUALS]; + int constant_stream_ids = constant_varint_column( + block, COLUMN_SAMPLE_STREAM_IDS, stream_id_column->data[0]); + int constant_residuals = constant_varint_column( + block, COLUMN_TIMESTAMP_RESIDUALS, residual_column->data[0]); + size_t stream_id_offset = 0; + size_t residual_offset = 0; + int64_t constant_residual = 0; + if (constant_residuals) { + constant_residual = decode_varint_i64(residual_column->data, + &residual_offset, 1); + } + for (uint32_t sample = 0; sample < block->header.sample_count; sample++) { + uint64_t stream_id; + if (constant_stream_ids) { + stream_id = stream_id_column->data[0]; + } + else if (block_read_varint(stream_id_column, &stream_id_offset, + &stream_id) < 0) { + return -1; + } + if (stream_id >= block->stream_count) { + return block_error("sample refers to an unknown stream"); + } + int64_t residual = constant_residual; + if (!constant_residuals) { + size_t prev_offset = residual_offset; + residual = decode_varint_i64(residual_column->data, + &residual_offset, + residual_column->size); + if (residual_offset == prev_offset) { return -1; } + } - reader->stats.repeat_records++; - reader->stats.repeat_samples += count; - - /* Process RLE samples, batching by status */ - PyObject *timestamps_list = NULL; - uint8_t batch_status = 0; - Py_ssize_t batch_idx = 0; - - for (uint32_t i = 0; i < count; i++) { - size_t delta_prev_offset = offset; - uint64_t delta = decode_varint_u64(reader->sample_data, &offset, reader->sample_data_size); - if (offset == delta_prev_offset) { - Py_XDECREF(timestamps_list); - PyErr_SetString(PyExc_ValueError, "Malformed varint in RLE sample data"); - return -1; - } - if (offset >= reader->sample_data_size) { - Py_XDECREF(timestamps_list); - PyErr_SetString(PyExc_ValueError, "Unexpected end of sample data in RLE"); - return -1; - } - uint8_t status = reader->sample_data[offset++]; - ts->prev_timestamp += delta; - - /* Start new batch on first sample or status change */ - if (i == 0 || status != batch_status - || batch_idx >= MAX_RLE_BATCH_SAMPLES) { - if (timestamps_list) { - int rc = emit_sample(state, collector, thread_id, interpreter_id, - batch_status, ts->current_stack, ts->current_stack_depth, - reader, timestamps_list); - Py_DECREF(timestamps_list); - if (rc < 0) { - return -1; - } - } - timestamps_list = PyList_New(0); - if (!timestamps_list) { - return -1; - } - batch_status = status; - batch_idx = 0; - } - - PyObject *ts_obj = PyLong_FromUnsignedLongLong(ts->prev_timestamp); - if (!ts_obj) { - Py_DECREF(timestamps_list); - return -1; - } - int append_rc = PyList_Append(timestamps_list, ts_obj); - Py_DECREF(ts_obj); - if (append_rc < 0) { - Py_DECREF(timestamps_list); - return -1; - } - batch_idx++; - } + /* Timestamp is the expected time plus the stored residual */ + ReaderBlockStream *stream = &reader->block_streams[stream_id]; + stream->prev_timestamp += + reader->sample_interval_us + (uint64_t)residual; - /* Emit final batch */ - if (timestamps_list) { - int rc = emit_sample(state, collector, thread_id, interpreter_id, - batch_status, ts->current_stack, ts->current_stack_depth, - reader, timestamps_list); - Py_DECREF(timestamps_list); - if (rc < 0) { - return -1; - } - } + stream_ids[sample] = (uint32_t)stream_id; + timestamps[sample] = stream->prev_timestamp; + } + if (!constant_stream_ids && stream_id_offset != stream_id_column->size) { + return block_error("stream index column has trailing data"); + } + if (residual_offset != residual_column->size) { + return block_error("timestamp column has trailing data"); + } + block->stream_ids = stream_ids; + block->timestamps = timestamps; + return 0; +} - replayed += count; - reader->stats.total_samples += count; +static int +decode_statuses(BinaryReader *reader, DecodedBlock *block) +{ + if (buffer_reserve(&reader->block_statuses, + block->header.sample_count) < 0) { + return -1; + } + /* Start from the common flags, then add each varying flag's bitmap */ + uint8_t *statuses = reader->block_statuses.data; + memset(statuses, block->header.common_status_flags, + block->header.sample_count); + const uint8_t *plane = block->columns[COLUMN_STATUS_BITMAPS].data; + size_t plane_size = status_bitmap_size(block->header.sample_count); + for (unsigned bit = 0; bit < CHAR_BIT; bit++) { + if (!(block->header.status_bitmap_flags & (1U << bit))) { + continue; + } + for (uint32_t i = 0; i < block->header.sample_count; i++) { + statuses[i] |= ((plane[i / CHAR_BIT] >> (i % CHAR_BIT)) & 1U) + << bit; + } + plane += plane_size; + } + block->statuses = statuses; + return 0; +} - /* Progress callback after batch */ - if ((uint64_t)replayed % PROGRESS_CALLBACK_INTERVAL < (uint64_t)count) { - if (invoke_progress_callback(progress_callback, replayed, reader->sample_count) < 0) { - return -1; - } - } - break; +static int +decode_frame_ids(BinaryReader *reader, DecodedBlock *block) +{ + size_t frame_count = block->frame_count; + size_t frames_size = frame_count * sizeof(uint32_t); + if (buffer_reserve(&reader->block_frames, frames_size) < 0) { + return -1; + } + uint32_t *frames = (uint32_t *)reader->block_frames.data; + block->frames = frames; + if (frame_count == 0) { + return 0; + } + + /* Combine byte b of every frame index from plane b, low byte first */ + memset(frames, 0, frames_size); + const uint8_t *plane = block->columns[COLUMN_FRAME_IDS].data; + for (unsigned byte = 0; byte < block->header.frame_width; byte++) { + for (size_t i = 0; i < frame_count; i++) { + frames[i] |= (uint32_t)plane[i] << (byte * CHAR_BIT); + } + plane += frame_count; + } + for (size_t i = 0; i < frame_count; i++) { + if (frames[i] >= reader->frames_count) { + return block_error("frame index out of range"); } + } + return 0; +} - case STACK_FULL: - case STACK_SUFFIX: - case STACK_POP_PUSH: { - /* All three encodings share: [delta: varint] [status: 1] ... */ - size_t prev_offset = offset; - uint64_t delta = decode_varint_u64(reader->sample_data, &offset, reader->sample_data_size); - /* Detect varint decode failure: offset unchanged means error */ - if (offset == prev_offset) { - PyErr_SetString(PyExc_ValueError, "Malformed varint in sample data"); - return -1; - } - if (offset >= reader->sample_data_size) { - PyErr_SetString(PyExc_ValueError, "Unexpected end of sample data"); - return -1; - } - uint8_t status = reader->sample_data[offset++]; - ts->prev_timestamp += delta; - - if (encoding == STACK_FULL) { - if (decode_stack_full(ts, reader->sample_data, &offset, reader->sample_data_size) < 0) { - return -1; - } - reader->stats.full_records++; - } else if (encoding == STACK_SUFFIX) { - if (decode_stack_suffix(ts, reader->sample_data, &offset, reader->sample_data_size) < 0) { - return -1; - } - reader->stats.suffix_records++; - } else { /* STACK_POP_PUSH */ - if (decode_stack_pop_push(ts, reader->sample_data, &offset, reader->sample_data_size) < 0) { - return -1; - } - reader->stats.pop_push_records++; - } - reader->stats.stack_reconstructions++; +static int +decode_block(BinaryReader *reader, const ByteBuffer *section, + size_t *offset, DecodedBlock *block) +{ + block->instruction_offset = 0; + block->frame_offset = 0; + if (reader_parse_block_header(reader, section, offset, + &block->header) < 0 || + decode_block_columns(reader, section, offset, block) < 0 || + validate_block_columns(block) < 0 || + reserve_block_streams(reader, block->stream_count) < 0 || + reader_parse_stream_table(reader, block) < 0 || + decode_samples(reader, block) < 0 || + decode_statuses(reader, block) < 0 || + decode_frame_ids(reader, block) < 0) { + return -1; + } + return 0; +} - /* Build single-element timestamp list */ - PyObject *ts_obj = PyLong_FromUnsignedLongLong(ts->prev_timestamp); - if (!ts_obj) { - return -1; - } - PyObject *timestamps_list = PyList_New(1); - if (!timestamps_list) { - Py_DECREF(ts_obj); - return -1; - } - PyList_SET_ITEM(timestamps_list, 0, ts_obj); +static int +flush_stream_batch(BinaryReader *reader, RemoteDebuggingState *state, + PyObject *collector, ReaderBlockStream *stream) +{ + if (stream->batch == NULL) { + return 0; + } + int result = emit_sample(state, collector, stream->thread_id, + stream->interpreter_id, stream->batch_status, + stream->current_stack, + stream->current_stack_depth, reader, + stream->batch); + Py_CLEAR(stream->batch); + return result; +} - if (emit_sample(state, collector, thread_id, interpreter_id, status, - ts->current_stack, ts->current_stack_depth, reader, - timestamps_list) < 0) { - Py_DECREF(timestamps_list); - return -1; - } - Py_DECREF(timestamps_list); - if (replayed == PY_SSIZE_T_MAX) { - PyErr_SetString(PyExc_OverflowError, - "Sample count exceeds Py_ssize_t maximum"); - return -1; - } - replayed++; - reader->stats.total_samples++; +static int +decode_stack_repeat(BinaryReader *reader, DecodedBlock *block, + ReaderBlockStream *stream) +{ + const ByteBuffer *instructions = + &block->columns[COLUMN_STACK_INSTRUCTIONS]; + size_t *offset = &block->instruction_offset; + uint64_t count; + if (block_read_varint(instructions, offset, &count) < 0) { + return -1; + } + if (!stream->has_stack) { + return block_error("first stack of a stream cannot repeat"); + } + if (count == 0 || count > block->header.sample_count) { + return block_error("repeat count out of range"); + } + stream->pending_repeats = (uint32_t)(count - 1); + reader->stats.repeat_records++; + reader->stats.repeat_samples++; + return 0; +} + +static int +decode_stack_replace_suffix(BinaryReader *reader, DecodedBlock *block, + ReaderBlockStream *stream) +{ + const ByteBuffer *instructions = + &block->columns[COLUMN_STACK_INSTRUCTIONS]; + size_t *offset = &block->instruction_offset; + uint64_t keep, count; + if (block_read_varint(instructions, offset, &keep) < 0 || + block_read_varint(instructions, offset, &count) < 0) { + return -1; + } + if (keep > stream->current_stack_depth || count > MAX_STACK_DEPTH - keep) { + return block_error("stack prefix out of range"); + } + if (count > block->frame_count - block->frame_offset) { + return block_error("stack uses more frame IDs than stored"); + } + if (count > 0) { + memcpy(&stream->current_stack[keep], + &block->frames[block->frame_offset], count * sizeof(uint32_t)); + } + block->frame_offset += (uint32_t)count; + stream->current_stack_depth = (size_t)(keep + count); + if (keep == 0) { + reader->stats.full_records++; + } + else { + reader->stats.suffix_records++; + } + return 0; +} + +static int +decode_stack_reuse(BinaryReader *reader, DecodedBlock *block, + ReaderBlockStream *stream) +{ + const ByteBuffer *instructions = + &block->columns[COLUMN_STACK_INSTRUCTIONS]; + size_t *offset = &block->instruction_offset; + uint64_t start, count; + if (block_read_varint(instructions, offset, &start) < 0 || + block_read_varint(instructions, offset, &count) < 0) { + return -1; + } + if (count > MAX_STACK_DEPTH) { + return block_error("stack is too deep"); + } + if (start > block->frame_offset || count > block->frame_offset - start) { + return block_error("reused stack is not stored earlier"); + } + if (count > 0) { + memcpy(stream->current_stack, &block->frames[start], + count * sizeof(uint32_t)); + } + stream->current_stack_depth = (size_t)count; + reader->stats.reuse_records++; + return 0; +} + +/* Return 1 if a new stack was decoded, 0 if the stack repeats, -1 on error. */ +static int +decode_stack(BinaryReader *reader, RemoteDebuggingState *state, + PyObject *collector, DecodedBlock *block, + ReaderBlockStream *stream) +{ + /* An earlier REPEAT still covers this sample */ + if (stream->pending_repeats > 0) { + stream->pending_repeats--; + reader->stats.repeat_samples++; + return 0; + } + uint64_t encoding; + if (block_read_varint(&block->columns[COLUMN_STACK_INSTRUCTIONS], + &block->instruction_offset, &encoding) < 0) { + return -1; + } + if (encoding == STACK_REPEAT) { + return decode_stack_repeat(reader, block, stream); + } + + /* The pending batch was collected with the stack being replaced. */ + if (flush_stream_batch(reader, state, collector, stream) < 0) { + return -1; + } + int result; + switch (encoding) { + case STACK_REPLACE_SUFFIX: + result = decode_stack_replace_suffix(reader, block, stream); + break; + case STACK_REUSE: + result = decode_stack_reuse(reader, block, stream); + break; + default: + return block_error("unknown stack instruction"); + } + if (result < 0) { + return -1; + } + stream->has_stack = 1; + reader->stats.stack_reconstructions++; + return 1; +} + +static int +replay_sample(BinaryReader *reader, RemoteDebuggingState *state, + PyObject *collector, DecodedBlock *block, uint32_t sample) +{ + ReaderBlockStream *stream = + &reader->block_streams[block->stream_ids[sample]]; + uint8_t status = block->statuses[sample]; + int new_stack = decode_stack(reader, state, collector, block, stream); + if (new_stack < 0) { + return -1; + } + + /* Start a new batch when the status changes or the batch is full */ + if (stream->batch == NULL || status != stream->batch_status || + PyList_GET_SIZE(stream->batch) >= MAX_RLE_BATCH_SAMPLES) { + if (flush_stream_batch(reader, state, collector, stream) < 0) { + return -1; + } + stream->batch = PyList_New(0); + if (stream->batch == NULL) { + return -1; + } + stream->batch_status = status; + } + PyObject *timestamp = + PyLong_FromUnsignedLongLong(block->timestamps[sample]); + if (timestamp == NULL) { + return -1; + } + int result = PyList_Append(stream->batch, timestamp); + Py_DECREF(timestamp); + if (result < 0) { + return -1; + } + /* Send a new stack's sample immediately so a stream's first sample is + * not held back behind samples from other streams. */ + if (new_stack) { + return flush_stream_batch(reader, state, collector, stream); + } + return 0; +} + +static int +validate_block_consumed(BinaryReader *reader, const DecodedBlock *block) +{ + if (block->instruction_offset != + block->columns[COLUMN_STACK_INSTRUCTIONS].size) { + return block_error("stack instruction column has trailing data"); + } + if (block->frame_offset != block->frame_count) { + return block_error("frame ID column has unused values"); + } + for (uint32_t i = 0; i < block->stream_count; i++) { + if (!reader->block_streams[i].has_stack) { + return block_error("stream table has an unused entry"); + } + if (reader->block_streams[i].pending_repeats != 0) { + return block_error("repeat count extends past the block"); + } + } + return 0; +} + +static int +replay_block(BinaryReader *reader, RemoteDebuggingState *state, + PyObject *collector, PyObject *progress_callback, + DecodedBlock *block, Py_ssize_t *replayed) +{ + if ((uint64_t)block->header.sample_count > + reader->sample_count - (uint64_t)*replayed) { + return block_error("more samples than the file header declares"); + } + int result = 0; + for (uint32_t sample = 0; sample < block->header.sample_count; sample++) { + if (replay_sample(reader, state, collector, block, sample) < 0) { + result = -1; break; } + (*replayed)++; + reader->stats.total_samples++; + if (*replayed % PROGRESS_CALLBACK_INTERVAL == 0 && + invoke_progress_callback(progress_callback, *replayed, + reader->sample_count) < 0) { + result = -1; + break; + } + } + if (result == 0) { + result = validate_block_consumed(reader, block); + } - default: - PyErr_Format(PyExc_ValueError, "Unknown stack encoding: %u", encoding); - return -1; + /* Send the remaining batches, or drop them after an error. */ + for (uint32_t i = 0; i < block->stream_count; i++) { + ReaderBlockStream *stream = &reader->block_streams[i]; + if (result == 0 && + flush_stream_batch(reader, state, collector, stream) < 0) { + result = -1; } + Py_CLEAR(stream->batch); + } - /* Progress callback */ - if (replayed % PROGRESS_CALLBACK_INTERVAL == 0) { - if (invoke_progress_callback(progress_callback, replayed, reader->sample_count) < 0) { - return -1; - } + return result; +} + +static int +replay_blocks(BinaryReader *reader, RemoteDebuggingState *state, + PyObject *collector, PyObject *progress_callback, + Py_ssize_t *replayed) +{ + ByteBuffer section = {reader->sample_data, reader->sample_data_size, 0}; + size_t offset = 0; + while (offset < section.size) { + DecodedBlock block; + if (decode_block(reader, §ion, &offset, &block) < 0 || + replay_block(reader, state, collector, progress_callback, + &block, replayed) < 0) { + return -1; } } + return 0; +} + +static Py_ssize_t +replay_all_samples(BinaryReader *reader, PyObject *collector, + PyObject *progress_callback) +{ + PyObject *collect_method; + int has_collect = PyObject_GetOptionalAttrString(collector, "collect", &collect_method); + if (has_collect < 0) { + return -1; + } + if (has_collect == 0) { + PyErr_SetString(PyExc_TypeError, "Collector must have a collect() method"); + return -1; + } + Py_DECREF(collect_method); + + /* Get module state for struct sequence types */ + PyObject *module = PyImport_ImportModule("_remote_debugging"); + if (!module) { + return -1; + } + RemoteDebuggingState *state = RemoteDebugging_GetState(module); + Py_DECREF(module); + + if (!state) { + PyErr_SetString(PyExc_RuntimeError, "Failed to get module state"); + return -1; + } + if (reader->sample_count > (uint64_t)PY_SSIZE_T_MAX) { + PyErr_SetString(PyExc_OverflowError, + "Sample count exceeds Py_ssize_t maximum"); + return -1; + } + + /* Initial progress callback at 0% */ + if (invoke_progress_callback(progress_callback, 0, reader->sample_count) < 0) { + return -1; + } + + memset(&reader->stats, 0, sizeof(reader->stats)); + Py_ssize_t replayed = 0; + if (replay_blocks(reader, state, collector, progress_callback, + &replayed) < 0) { + return -1; + } if ((uint64_t)replayed != reader->sample_count) { PyErr_Format(PyExc_ValueError, @@ -1354,8 +1662,25 @@ binary_reader_replay(BinaryReader *reader, PyObject *collector, PyObject *progre (unsigned long long)reader->sample_count, replayed); return -1; } + return replayed; +} - /* Final progress callback at 100% */ +Py_ssize_t +binary_reader_replay(BinaryReader *reader, PyObject *collector, PyObject *progress_callback) +{ + if (reader->replaying) { + PyErr_SetString(PyExc_RuntimeError, "Reader is already replaying"); + return -1; + } + reader->replaying = 1; + Py_ssize_t replayed = replay_all_samples(reader, collector, + progress_callback); + reader->replaying = 0; + if (replayed < 0) { + return -1; + } + + /* Final progress callback at 100%; it may close the reader */ if (invoke_progress_callback(progress_callback, replayed, reader->sample_count) < 0) { return -1; } @@ -1453,8 +1778,9 @@ binary_writer_get_stats(BinaryWriter *writer) /* Calculate derived stats */ uint64_t total_records = s->repeat_records + s->full_records + - s->suffix_records + s->pop_push_records; + s->suffix_records + s->reuse_records; uint64_t total_samples = writer->total_samples; + uint64_t bytes_written = s->bytes_written + binary_writer_buffered_size(writer); uint64_t potential_frames = s->total_frames_written + s->frames_saved; double compression_ratio = (potential_frames > 0) ? (double)s->frames_saved / potential_frames * 100.0 : 0.0; @@ -1465,12 +1791,12 @@ binary_writer_get_stats(BinaryWriter *writer) "repeat_samples", s->repeat_samples, "full_records", s->full_records, "suffix_records", s->suffix_records, - "pop_push_records", s->pop_push_records, + "reuse_records", s->reuse_records, "total_records", total_records, "total_samples", total_samples, "total_frames_written", s->total_frames_written, "frames_saved", s->frames_saved, - "bytes_written", s->bytes_written, + "bytes_written", bytes_written, "frame_compression_pct", compression_ratio ); } @@ -1481,7 +1807,7 @@ binary_reader_get_stats(BinaryReader *reader) BinaryReaderStats *s = &reader->stats; uint64_t total_records = s->repeat_records + s->full_records + - s->suffix_records + s->pop_push_records; + s->suffix_records + s->reuse_records; return Py_BuildValue( "{s:K, s:K, s:K, s:K, s:K, s:K, s:K, s:K}", @@ -1489,7 +1815,7 @@ binary_reader_get_stats(BinaryReader *reader) "repeat_samples", s->repeat_samples, "full_records", s->full_records, "suffix_records", s->suffix_records, - "pop_push_records", s->pop_push_records, + "reuse_records", s->reuse_records, "total_records", total_records, "total_samples", s->total_samples, "stack_reconstructions", s->stack_reconstructions @@ -1524,7 +1850,22 @@ binary_reader_close(BinaryReader *reader) } #endif - PyMem_Free(reader->decompressed_data); +#ifdef HAVE_ZSTD + ZSTD_freeDCtx(reader->dctx); +#endif + buffer_clear(&reader->block_data); + buffer_clear(&reader->block_stream_ids); + buffer_clear(&reader->block_timestamps); + buffer_clear(&reader->block_statuses); + buffer_clear(&reader->block_frames); + PyMem_Free(reader->block_streams); + + if (reader->frame_info_cache != NULL) { + for (size_t i = 0; i < reader->frame_info_cache_size; i++) { + Py_XDECREF(reader->frame_info_cache[i].frame_info); + } + PyMem_Free(reader->frame_info_cache); + } if (reader->strings) { for (uint32_t i = 0; i < reader->strings_count; i++) { @@ -1535,12 +1876,7 @@ binary_reader_close(BinaryReader *reader) PyMem_Free(reader->frames); - if (reader->thread_states) { - for (size_t i = 0; i < reader->thread_state_count; i++) { - PyMem_Free(reader->thread_states[i].current_stack); - } - PyMem_Free(reader->thread_states); - } + PyMem_Free(reader->thread_states); PyMem_Free(reader); } diff --git a/Modules/_remote_debugging/binary_io_writer.c b/Modules/_remote_debugging/binary_io_writer.c index 1794017053da503..01331c3722484f7 100644 --- a/Modules/_remote_debugging/binary_io_writer.c +++ b/Modules/_remote_debugging/binary_io_writer.c @@ -2,7 +2,7 @@ * Python Remote Debugging Module - Binary Writer Implementation * * High-performance binary file writer for profiling data with optional zstd - * streaming compression. + * compression. ******************************************************************************/ #ifndef Py_BUILD_CORE_MODULE @@ -11,6 +11,7 @@ #include "binary_io.h" #include "_remote_debugging.h" +#include "pycore_bitutils.h" // _Py_bit_length(), _Py_popcount32() #include "pycore_opcode_utils.h" // MAX_REAL_OPCODE #include #include @@ -27,16 +28,69 @@ * CONSTANTS FOR BINARY FORMAT SIZES * ============================================================================ */ -/* Sample header sizes */ -#define SAMPLE_HEADER_MAX_SIZE 26 /* fixed + max_varint(10) + status(1) + margin */ #define MAX_VARINT_SIZE 10 /* Maximum bytes for a varint64 */ #define MAX_VARINT_SIZE_U32 5 /* Maximum bytes for a varint32 */ -/* Frame buffer: depth varint (max 2 bytes for 256) + 256 frames * 5 bytes/varint + margin */ -#define MAX_FRAME_BUFFER_SIZE ((MAX_STACK_DEPTH * MAX_VARINT_SIZE_U32) + MAX_VARINT_SIZE_U32 + 16) -/* RLE pending buffer (per thread): one entry is a u64 delta varint + status byte */ -#define MAX_RLE_BUF_SIZE (16 * 1024) -#define MAX_RLE_ENTRY_SIZE (MAX_VARINT_SIZE + sizeof(uint8_t)) +#define MAX_BLOCK_HEADER_SIZE \ + (BLOCK_MAGIC_SIZE + MAX_VARINT_SIZE_U32 + MAX_VARINT_SIZE + \ + 3 * sizeof(uint8_t) + \ + COLUMN_COUNT * (2 * MAX_VARINT_SIZE_U32 + sizeof(uint8_t))) + +#define ZSTD_MIN_SIZE 32 +#define STACK_REUSE_MIN_FRAMES 3 + +#define MAX_SAMPLE_SIZE \ + (STREAM_ENTRY_SIZE + 5 * MAX_VARINT_SIZE + sizeof(uint8_t) + \ + MAX_STACK_DEPTH * sizeof(uint32_t) + sizeof(Repeat)) +#define MAX_BITMAP_PADDING (CHAR_BIT - 1) +#define BLOCK_FLUSH_SIZE \ + (MAX_BLOCK_SIZE - MAX_SAMPLE_SIZE - MAX_BITMAP_PADDING) + +static int +buffer_append(ByteBuffer *buffer, const void *data, size_t size) +{ + if (buffer->size > MAX_BLOCK_SIZE || + size > MAX_BLOCK_SIZE - buffer->size) { + PyErr_SetString(PyExc_RuntimeError, + "sample block buffer exceeds the writer safety limit"); + return -1; + } + if (buffer_reserve(buffer, buffer->size + size) < 0) { + return -1; + } + if (size != 0) { + memcpy(buffer->data + buffer->size, data, size); + } + buffer->size += size; + return 0; +} + +static int +buffer_append_varint(ByteBuffer *buffer, uint64_t value) +{ + uint8_t bytes[MAX_VARINT_SIZE]; + size_t size = encode_varint_u64(bytes, value); + return buffer_append(buffer, bytes, size); +} + +static int +column_write(SampleBlock *block, unsigned column, const void *data, + size_t size) +{ + if (buffer_append(&block->columns[column], data, size) < 0) { + return -1; + } + block->size += size; + return 0; +} + +static int +column_write_varint(SampleBlock *block, unsigned column, uint64_t value) +{ + uint8_t bytes[MAX_VARINT_SIZE]; + size_t size = encode_varint_u64(bytes, value); + return column_write(block, column, bytes, size); +} /* Helper macro: convert PyLong to int32, using default_val if conversion fails */ #define PYLONG_TO_INT32_OR_DEFAULT(obj, var, default_val) \ @@ -131,19 +185,6 @@ fwrite_checked_allow_threads(const void *data, size_t size, FILE *fp) return 0; } -/* Forward declaration for writer_write_bytes */ -static inline int writer_write_bytes(BinaryWriter *writer, const void *data, size_t size); - -/* Encode and write a varint u32 - returns 0 on success, -1 on error */ -static inline int -writer_write_varint_u32(BinaryWriter *writer, uint32_t value) -{ - uint8_t buf[MAX_VARINT_SIZE]; - size_t len = encode_varint_u32(buf, value); - return writer_write_bytes(writer, buf, len); -} - - /* ============================================================================ * UTILITY FUNCTIONS * ============================================================================ */ @@ -182,9 +223,9 @@ writer_init_zstd(BinaryWriter *writer) return -1; } - /* Compression level 5: better ratio for repetitive profiling data */ + /* Compression level 3: good ratio without slowing down each block flush */ size_t result = ZSTD_CCtx_setParameter(writer->zstd.cctx, - ZSTD_c_compressionLevel, 5); + ZSTD_c_compressionLevel, 3); if (ZSTD_isError(result)) { PyErr_Format(PyExc_RuntimeError, "Failed to set zstd compression level: %s", ZSTD_getErrorName(result)); @@ -193,16 +234,6 @@ writer_init_zstd(BinaryWriter *writer) return -1; } - /* Use large buffer (512KB) for fewer I/O syscalls */ - writer->zstd.compressed_buffer = PyMem_Malloc(COMPRESSED_BUFFER_SIZE); - if (!writer->zstd.compressed_buffer) { - ZSTD_freeCCtx(writer->zstd.cctx); - writer->zstd.cctx = NULL; - PyErr_NoMemory(); - return -1; - } - writer->zstd.compressed_buffer_size = COMPRESSED_BUFFER_SIZE; - return 0; #else PyErr_SetString(PyExc_RuntimeError, @@ -211,79 +242,6 @@ writer_init_zstd(BinaryWriter *writer) #endif } -static int -writer_flush_buffer(BinaryWriter *writer) -{ - if (writer->buffer_pos == 0) { - return 0; - } - -#ifdef HAVE_ZSTD - if (writer->compression_type == COMPRESSION_ZSTD) { - ZSTD_inBuffer input = { writer->write_buffer, writer->buffer_pos, 0 }; - - while (input.pos < input.size) { - ZSTD_outBuffer output = { - writer->zstd.compressed_buffer, - writer->zstd.compressed_buffer_size, - 0 - }; - - size_t result = ZSTD_compressStream2( - writer->zstd.cctx, &output, &input, ZSTD_e_continue - ); - - if (ZSTD_isError(result)) { - PyErr_Format(PyExc_IOError, "zstd compression error: %s", - ZSTD_getErrorName(result)); - return -1; - } - - _Py_MSAN_UNPOISON(writer->zstd.compressed_buffer, output.pos); - if (output.pos > 0) { - if (fwrite_checked_allow_threads(writer->zstd.compressed_buffer, output.pos, writer->fp) < 0) { - return -1; - } - } - } - } else -#endif - { - if (fwrite_checked_allow_threads(writer->write_buffer, writer->buffer_pos, writer->fp) < 0) { - return -1; - } - } - - writer->buffer_pos = 0; - return 0; -} - -static inline int -writer_write_bytes(BinaryWriter *writer, const void *data, size_t size) -{ - const uint8_t *src = (const uint8_t *)data; - size_t original_size = size; - - while (size > 0) { - size_t space = writer->buffer_size - writer->buffer_pos; - size_t to_copy = (size < space) ? size : space; - - memcpy(writer->write_buffer + writer->buffer_pos, src, to_copy); - writer->buffer_pos += to_copy; - src += to_copy; - size -= to_copy; - - if (writer->buffer_pos == writer->buffer_size) { - if (writer_flush_buffer(writer) < 0) { - return -1; - } - } - } - - writer->stats.bytes_written += original_size; - return 0; -} - /* ============================================================================ * HASH TABLE SUPPORT FUNCTIONS (using _Py_hashtable) * ============================================================================ */ @@ -481,21 +439,16 @@ writer_intern_frame(BinaryWriter *writer, const FrameEntry *entry, uint32_t *ind return 0; } -/* Get or create a thread entry for the given thread_id. - * Returns pointer to ThreadEntry, or NULL on allocation failure. - * If is_new is non-NULL, sets it to 1 if this is a new thread, 0 otherwise. */ +/* Get or create the entry for a (thread, interpreter) pair. */ static ThreadEntry * writer_get_or_create_thread_entry(BinaryWriter *writer, uint64_t thread_id, - uint32_t interpreter_id, int *is_new) + uint32_t interpreter_id) { /* Linear search is OK for small number of threads. * Key is (thread_id, interpreter_id) since same thread_id can exist in different interpreters. */ for (size_t i = 0; i < writer->thread_count; i++) { if (writer->thread_entries[i].thread_id == thread_id && writer->thread_entries[i].interpreter_id == interpreter_id) { - if (is_new) { - *is_new = 0; - } return &writer->thread_entries[i]; } } @@ -506,249 +459,620 @@ writer_get_or_create_thread_entry(BinaryWriter *writer, uint64_t thread_id, "too many threads for binary format"); return NULL; } - if (writer->thread_count >= writer->thread_capacity) { - ThreadEntry *new_entries = grow_array(writer->thread_entries, - &writer->thread_capacity, - sizeof(ThreadEntry)); - if (!new_entries) { - if (PyErr_ExceptionMatches(PyExc_OverflowError)) { - writer->state = BINARY_WRITER_LIMIT_REACHED; - } - return NULL; + if (GROW_ARRAY(writer->thread_entries, writer->thread_count, + writer->thread_capacity, ThreadEntry) < 0) { + if (PyErr_ExceptionMatches(PyExc_OverflowError)) { + writer->state = BINARY_WRITER_LIMIT_REACHED; } - writer->thread_entries = new_entries; + return NULL; } ThreadEntry *entry = &writer->thread_entries[writer->thread_count]; memset(entry, 0, sizeof(ThreadEntry)); entry->thread_id = thread_id; entry->interpreter_id = interpreter_id; - entry->prev_timestamp = writer->start_time_us; - entry->prev_stack_capacity = MAX_STACK_DEPTH; - - entry->prev_stack = PyMem_Calloc(entry->prev_stack_capacity, sizeof(uint32_t)); - if (!entry->prev_stack) { - PyErr_NoMemory(); - return NULL; - } + entry->block_stream_index = UINT32_MAX; writer->thread_count++; - if (is_new) { - *is_new = 1; - } return entry; } -/* Compare two stacks and return the encoding type and parameters. - * Sets: - * - shared_count: number of frames matching from bottom of stack - * - pop_count: frames to remove from prev stack - * - push_count: new frames to add - * - * Returns the best encoding type to use. */ static int -compare_stacks(const uint32_t *prev_stack, size_t prev_depth, - const uint32_t *curr_stack, size_t curr_depth, - size_t *shared_count, size_t *pop_count, size_t *push_count) +encode_column(BinaryWriter *writer, const ByteBuffer *decoded, + ColumnDescriptor *descriptor) { - if (prev_depth == curr_depth) { - int identical = 1; - for (size_t i = 0; i < prev_depth; i++) { - if (prev_stack[i] != curr_stack[i]) { - identical = 0; - break; - } + SampleBlock *block = writer->block; + const uint8_t *data = decoded->data; + size_t size = decoded->size; + uint8_t encoding = COMPRESSION_NONE; + if (size > 1 && memcmp(data, data + 1, size - 1) == 0) { + encoding = COLUMN_CONSTANT; + size = 1; + } +#ifdef HAVE_ZSTD + if (encoding == COMPRESSION_NONE && size >= ZSTD_MIN_SIZE && + writer->compression_type == COMPRESSION_ZSTD) { + size_t compress_bound = ZSTD_compressBound(size); + if (buffer_reserve(&block->compressed, compress_bound) < 0) { + return -1; } - if (identical) { - *shared_count = prev_depth; - *pop_count = 0; - *push_count = 0; - return STACK_REPEAT; + size_t compressed_size = ZSTD_compress2( + writer->zstd.cctx, block->compressed.data, compress_bound, + data, size); + if (ZSTD_isError(compressed_size)) { + PyErr_Format(PyExc_IOError, "zstd compression error: %s", + ZSTD_getErrorName(compressed_size)); + return -1; + } + _Py_MSAN_UNPOISON(block->compressed.data, compressed_size); + uint8_t stored_size[MAX_VARINT_SIZE]; + if (compressed_size + encode_varint_u64(stored_size, compressed_size) + < size) { + data = block->compressed.data; + size = compressed_size; + encoding = COMPRESSION_ZSTD; } } +#endif + descriptor->decoded_size = (uint32_t)decoded->size; + descriptor->stored_size = (uint32_t)size; + descriptor->encoding = encoding; + if (buffer_reserve(&block->output, block->output.size + size) < 0) { + return -1; + } + if (size != 0) { + memcpy(block->output.data + block->output.size, data, size); + } + block->output.size += size; + return 0; +} - /* Find longest common suffix (frames at the bottom/outer part of stack). - * Stacks are stored innermost-first, so suffix is at the end. */ - size_t suffix_len = 0; - size_t min_depth = (prev_depth < curr_depth) ? prev_depth : curr_depth; - - for (size_t i = 0; i < min_depth; i++) { - size_t prev_idx = prev_depth - 1 - i; - size_t curr_idx = curr_depth - 1 - i; - if (prev_stack[prev_idx] == curr_stack[curr_idx]) { - suffix_len++; - } else { - break; +static ByteBuffer * +transform_statuses(SampleBlock *block, uint8_t bitmap_flags) +{ + size_t plane_size = status_bitmap_size(block->sample_count); + unsigned plane_count = (unsigned)_Py_popcount32(bitmap_flags); + if (buffer_reserve(&block->transformed, + plane_size * plane_count) < 0) { + return NULL; + } + const uint8_t *statuses = + block->columns[COLUMN_STATUS_BITMAPS].data; + block->transformed.size = 0; + for (unsigned bit = 0; bit < CHAR_BIT; bit++) { + if (!(bitmap_flags & (1U << bit))) { + continue; + } + uint8_t *plane = block->transformed.data + block->transformed.size; + memset(plane, 0, plane_size); + for (uint32_t sample = 0; sample < block->sample_count; sample++) { + plane[sample / CHAR_BIT] |= ((statuses[sample] >> bit) & 1U) + << (sample % CHAR_BIT); } + block->transformed.size += plane_size; } + return &block->transformed; +} - *shared_count = suffix_len; - *pop_count = prev_depth - suffix_len; - *push_count = curr_depth - suffix_len; - - /* Choose best encoding based on byte cost */ - /* STACK_FULL: 1 (type) + 1-2 (depth) + sum(frame varints) */ - /* STACK_SUFFIX: 1 (type) + 1-2 (shared) + 1-2 (new_count) + sum(new frame varints) */ - /* STACK_POP_PUSH: 1 (type) + 1-2 (pop) + 1-2 (push) + sum(new frame varints) */ - - /* If no common suffix, use full stack */ - if (suffix_len == 0) { - return STACK_FULL; +static ByteBuffer * +transform_frame_ids(SampleBlock *block, unsigned width) +{ + ByteBuffer *frames = &block->columns[COLUMN_FRAME_IDS]; + size_t frame_count = frames->size / sizeof(uint32_t); + if (buffer_reserve(&block->transformed, + frame_count * width) < 0) { + return NULL; } - - /* If only adding frames (suffix == prev_depth), use SUFFIX */ - if (*pop_count == 0 && *push_count > 0) { - return STACK_SUFFIX; + const uint32_t *frame_ids = (const uint32_t *)frames->data; + for (unsigned byte = 0; frame_count && byte < width; byte++) { + uint8_t *plane = + block->transformed.data + (size_t)byte * frame_count; + for (uint32_t frame = 0; frame < frame_count; frame++) { + plane[frame] = + (uint8_t)(frame_ids[frame] >> (byte * CHAR_BIT)); + } } + block->transformed.size = frame_count * width; + return &block->transformed; +} - /* If popping and/or pushing, use POP_PUSH if it saves bytes */ - /* Heuristic: POP_PUSH is better when we're modifying top frames */ - if (*pop_count > 0 || *push_count > 0) { - /* Use full stack if sharing less than half the frames */ - if (suffix_len < curr_depth / 2) { - return STACK_FULL; +static ByteBuffer * +transform_stack_instructions(SampleBlock *block) +{ + ByteBuffer *instructions = + &block->columns[COLUMN_STACK_INSTRUCTIONS]; + size_t repeat_count = block->repeats.size / sizeof(Repeat); + if (repeat_count == 0) { + return instructions; + } + const Repeat *repeats = (const Repeat *)block->repeats.data; + size_t copied_offset = 0; + block->transformed.size = 0; + for (size_t i = 0; i < repeat_count; i++) { + if (buffer_append(&block->transformed, + instructions->data + copied_offset, + repeats[i].offset - copied_offset) < 0 || + buffer_append_varint( + &block->transformed, STACK_REPEAT) < 0 || + buffer_append_varint( + &block->transformed, repeats[i].count) < 0) { + return NULL; } - return STACK_POP_PUSH; + copied_offset = repeats[i].offset; + } + if (buffer_append(&block->transformed, + instructions->data + copied_offset, + instructions->size - copied_offset) < 0) { + return NULL; } + return &block->transformed; +} - return STACK_FULL; +static void +block_reset(BinaryWriter *writer) +{ + SampleBlock *block = writer->block; + /* Only threads with a stream in this block need resetting */ + for (uint32_t i = 0; i < block->stream_count; i++) { + size_t thread = block->streams[i].thread_index; + writer->thread_entries[thread].block_stream_index = UINT32_MAX; + } + for (unsigned column = 0; column < COLUMN_COUNT; column++) { + block->columns[column].size = 0; + } + block->sample_count = 0; + block->stream_count = 0; + block->repeats.size = 0; + block->max_frame = 0; + block->size = 0; + memset(block->cache, 0, sizeof(block->cache)); } -/* Write common sample header: thread_id(8) + interpreter_id(4) + encoding(1). - * Returns 0 on success, -1 on failure. */ -static inline int -write_sample_header(BinaryWriter *writer, ThreadEntry *entry, uint8_t encoding) +static unsigned +block_frame_width(const SampleBlock *block) +{ + if (block->max_frame == 0) { + return 1; + } + return (unsigned)(_Py_bit_length(block->max_frame) + CHAR_BIT - 1) + / CHAR_BIT; +} + +static void +block_status_flags(const SampleBlock *block, uint8_t *common_flags, + uint8_t *bitmap_flags) +{ + uint8_t common_status = UINT8_MAX; + uint8_t any_status = 0; + const uint8_t *statuses = block->columns[COLUMN_STATUS_BITMAPS].data; + for (uint32_t sample = 0; sample < block->sample_count; sample++) { + common_status &= statuses[sample]; + any_status |= statuses[sample]; + } + *common_flags = common_status; + *bitmap_flags = any_status ^ common_status; +} + +static size_t +write_block_header(const BlockHeader *header, uint8_t *bytes) { - uint8_t header[SAMPLE_HEADER_FIXED_SIZE]; - memcpy(header + SMP_OFF_THREAD_ID, &entry->thread_id, SMP_SIZE_THREAD_ID); - memcpy(header + SMP_OFF_INTERPRETER_ID, &entry->interpreter_id, SMP_SIZE_INTERPRETER_ID); - header[SMP_OFF_ENCODING] = encoding; - return writer_write_bytes(writer, header, SAMPLE_HEADER_FIXED_SIZE); + /* Magic "TBLK", sample count and base timestamp (varints), then frame + * ID width and the two status flag masks (1 byte each) */ + size_t size = BLOCK_MAGIC_SIZE; + memcpy(bytes, BLOCK_MAGIC, BLOCK_MAGIC_SIZE); + size += encode_varint_u32(bytes + size, header->sample_count); + size += encode_varint_u64(bytes + size, header->base_timestamp); + bytes[size++] = header->frame_width; + bytes[size++] = header->common_status_flags; + bytes[size++] = header->status_bitmap_flags; + + /* Each column: decoded size (varint), encoding (1 byte), and the stored + * size (varint) for zstd only */ + for (unsigned column = 0; column < COLUMN_COUNT; column++) { + const ColumnDescriptor *descriptor = &header->columns[column]; + size += encode_varint_u32(bytes + size, descriptor->decoded_size); + bytes[size++] = descriptor->encoding; + if (descriptor->encoding == COMPRESSION_ZSTD) { + size += encode_varint_u32(bytes + size, descriptor->stored_size); + } + } + return size; } -/* Flush pending RLE samples for a thread. - * Writes the RLE record to the output buffer. - * Returns 0 on success, -1 on failure. */ static int -flush_pending_rle(BinaryWriter *writer, ThreadEntry *entry) +block_flush(BinaryWriter *writer) { - if (entry->pending_rle_samples == 0) { + SampleBlock *block = writer->block; + if (block->sample_count == 0) { return 0; } + BlockHeader header = {0}; + header.sample_count = block->sample_count; + header.base_timestamp = block->base_timestamp; + header.frame_width = (uint8_t)block_frame_width(block); + block_status_flags(block, &header.common_status_flags, + &header.status_bitmap_flags); - /* Write RLE record: - * [thread_id: 8] [interpreter_id: 4] [STACK_REPEAT: 1] [count: varint] - * [timestamp_delta_1: varint] [status_1: 1] ... [timestamp_delta_N: varint] [status_N: 1] - */ - - if (write_sample_header(writer, entry, STACK_REPEAT) < 0) { + /* Convert each column to its file form, then encode it */ + /* Leave room for the header, which is written once its size is known */ + if (buffer_reserve(&block->output, MAX_BLOCK_HEADER_SIZE) < 0) { return -1; } - - if (writer_write_varint_u32(writer, (uint32_t)entry->pending_rle_samples) < 0) { + block->output.size = MAX_BLOCK_HEADER_SIZE; + size_t decoded_payload_size = 0; + for (unsigned column = 0; column < COLUMN_COUNT; column++) { + ByteBuffer *decoded = &block->columns[column]; + switch (column) { + case COLUMN_STATUS_BITMAPS: + decoded = transform_statuses(block, header.status_bitmap_flags); + break; + case COLUMN_STACK_INSTRUCTIONS: + decoded = transform_stack_instructions(block); + break; + case COLUMN_FRAME_IDS: + decoded = transform_frame_ids(block, header.frame_width); + break; + } + if (decoded == NULL) { + return -1; + } + if (decoded->size > + MAX_BLOCK_SIZE - decoded_payload_size) { + PyErr_SetString(PyExc_RuntimeError, + "sample block exceeds the decoded size limit"); + return -1; + } + decoded_payload_size += decoded->size; + if (encode_column(writer, decoded, &header.columns[column]) < 0) { + return -1; + } + } + /* Put the header right before the columns and write the block */ + uint8_t header_bytes[MAX_BLOCK_HEADER_SIZE]; + size_t header_size = write_block_header(&header, header_bytes); + size_t unused = MAX_BLOCK_HEADER_SIZE - header_size; + uint8_t *header_start = block->output.data + unused; + memcpy(header_start, header_bytes, header_size); + if (fwrite_checked_allow_threads(header_start, + block->output.size - unused, + writer->fp) < 0) { return -1; } + writer->stats.bytes_written += decoded_payload_size; + block_reset(writer); + return 0; +} - if (writer_write_bytes(writer, entry->pending_rle, entry->pending_rle_bytes) < 0) { - return -1; +/* FNV-1a over the frame indices, the same in every process */ +static Py_uhash_t +stack_hash(const uint32_t *stack, size_t depth) +{ + uint64_t hash = 0xcbf29ce484222325; + for (size_t i = 0; i < depth; i++) { + hash = (hash ^ stack[i]) * 0x100000001b3; + } + /* Mix the high bits into the low bits that select the cache bucket */ + hash ^= hash >> 33; + hash *= 0xff51afd7ed558ccd; + hash ^= hash >> 33; + return (Py_uhash_t)hash; +} + +static size_t +common_outer_frames(const BlockStream *stream, const uint32_t *stack, + size_t depth) +{ + size_t shared = 0; + while (shared < depth && shared < stream->prev_stack_depth && + stack[depth - 1 - shared] == + stream->prev_stack[stream->prev_stack_depth - 1 - shared]) { + shared++; } + return shared; +} - writer->stats.repeat_records++; - writer->stats.repeat_samples += entry->pending_rle_samples; - /* Each RLE sample saves writing the entire stack */ - writer->stats.frames_saved += entry->pending_rle_samples * entry->prev_stack_depth; +static int +cached_stack_matches(const StackCacheEntry *cached, + const ByteBuffer *frames, Py_uhash_t hash, + const uint32_t *stack, size_t depth) +{ + if (cached->hash != hash || cached->depth != depth) { + return 0; + } + const uint32_t *frame_ids = (const uint32_t *)frames->data; + for (size_t frame = 0; frame < depth; frame++) { + if (stack[frame] != + frame_ids[cached->offset + depth - 1 - frame]) { + return 0; + } + } + return 1; +} + +static StackCacheEntry * +stack_cache_lookup(StackCacheEntry *bucket, const ByteBuffer *frames, + Py_uhash_t hash, const uint32_t *stack, size_t depth) +{ + for (size_t i = 0; i < STACK_CACHE_BUCKET_SIZE; i++) { + if (cached_stack_matches(&bucket[i], frames, hash, stack, depth)) { + return &bucket[i]; + } + } + return NULL; +} - entry->pending_rle_bytes = 0; - entry->pending_rle_samples = 0; +static void +stack_cache_insert(StackCacheEntry *bucket, StackCacheEntry entry) +{ + memmove(&bucket[1], &bucket[0], + (STACK_CACHE_BUCKET_SIZE - 1) * sizeof(*bucket)); + bucket[0] = entry; +} +static int +write_stack_repeat(BinaryWriter *writer, BlockStream *stream, size_t depth) +{ + SampleBlock *block = writer->block; + if (stream->current_repeat == UINT32_MAX) { + Repeat repeat = { + (uint32_t)block->columns[COLUMN_STACK_INSTRUCTIONS].size, 1}; + if (buffer_append(&block->repeats, + &repeat, sizeof(repeat)) < 0) { + return -1; + } + block->size += sizeof(repeat); + stream->current_repeat = + (uint32_t)(block->repeats.size / sizeof(repeat) - 1); + writer->stats.repeat_records++; + } + else { + Repeat *repeats = (Repeat *)block->repeats.data; + repeats[stream->current_repeat].count++; + } + writer->stats.repeat_samples++; + writer->stats.frames_saved += depth; return 0; } -/* Write a single sample with the specified encoding. - * Returns 0 on success, -1 on failure. */ static int -write_sample_with_encoding(BinaryWriter *writer, ThreadEntry *entry, - uint64_t timestamp_delta, uint8_t status, - int encoding_type, - const uint32_t *frame_indices, size_t stack_depth, - size_t shared_count, size_t pop_count, size_t push_count) +write_stack_reuse(BinaryWriter *writer, + const StackCacheEntry *cached, size_t depth) { - /* Header: thread_id(8) + interpreter_id(4) + encoding(1) + delta(varint) + status(1) */ - uint8_t header_buf[SAMPLE_HEADER_MAX_SIZE]; - memcpy(header_buf + SMP_OFF_THREAD_ID, &entry->thread_id, SMP_SIZE_THREAD_ID); - memcpy(header_buf + SMP_OFF_INTERPRETER_ID, &entry->interpreter_id, SMP_SIZE_INTERPRETER_ID); - header_buf[SMP_OFF_ENCODING] = (uint8_t)encoding_type; - size_t varint_len = encode_varint_u64( - header_buf + SAMPLE_HEADER_FIXED_SIZE, - timestamp_delta); - header_buf[SAMPLE_HEADER_FIXED_SIZE + varint_len] = status; - - if (writer_write_bytes(writer, header_buf, - SAMPLE_HEADER_FIXED_SIZE + varint_len + 1) < 0) { + SampleBlock *block = writer->block; + if (column_write_varint(block, COLUMN_STACK_INSTRUCTIONS, + STACK_REUSE) < 0 || + column_write_varint(block, COLUMN_STACK_INSTRUCTIONS, + cached->offset) < 0 || + column_write_varint(block, COLUMN_STACK_INSTRUCTIONS, depth) < 0) { return -1; } + writer->stats.reuse_records++; + writer->stats.frames_saved += depth; + return 0; +} - uint8_t frame_buf[MAX_FRAME_BUFFER_SIZE]; - size_t frame_buf_pos = 0; - size_t frames_written = 0; +static int +write_stack_replace_suffix(BinaryWriter *writer, StackCacheEntry *bucket, + Py_uhash_t hash, const uint32_t *stack, + size_t depth, size_t shared) +{ + SampleBlock *block = writer->block; + ByteBuffer *frames = &block->columns[COLUMN_FRAME_IDS]; + size_t new_frames = depth - shared; + if (column_write_varint(block, COLUMN_STACK_INSTRUCTIONS, + STACK_REPLACE_SUFFIX) < 0 || + column_write_varint(block, COLUMN_STACK_INSTRUCTIONS, shared) < 0 || + column_write_varint(block, COLUMN_STACK_INSTRUCTIONS, + new_frames) < 0) { + return -1; + } - switch (encoding_type) { - case STACK_FULL: - /* [depth: varint] [frame_idx: varint]... */ - frame_buf_pos += encode_varint_u32(frame_buf, (uint32_t)stack_depth); - for (size_t i = 0; i < stack_depth; i++) { - frame_buf_pos += encode_varint_u32(frame_buf + frame_buf_pos, frame_indices[i]); + /* Append the new inner frames, outermost first */ + size_t stack_offset = frames->size / sizeof(uint32_t); + if (new_frames != 0) { + if (buffer_reserve( + frames, frames->size + new_frames * sizeof(uint32_t)) < 0) { + return -1; } - frames_written = stack_depth; - writer->stats.full_records++; - break; - - case STACK_SUFFIX: - /* [shared_count: varint] [new_count: varint] [new_frame_idx: varint]... */ - frame_buf_pos += encode_varint_u32(frame_buf, (uint32_t)shared_count); - frame_buf_pos += encode_varint_u32(frame_buf + frame_buf_pos, (uint32_t)push_count); - /* New frames are at the top (beginning) of current stack */ - for (size_t i = 0; i < push_count; i++) { - frame_buf_pos += encode_varint_u32(frame_buf + frame_buf_pos, frame_indices[i]); + uint32_t *new_frame_ids = (uint32_t *)(frames->data + frames->size); + for (size_t i = 0; i < new_frames; i++) { + uint32_t frame = stack[new_frames - 1 - i]; + new_frame_ids[i] = frame; + if (frame > block->max_frame) { + block->max_frame = frame; + } } - frames_written = push_count; - writer->stats.suffix_records++; - /* Saved writing shared_count frames */ - writer->stats.frames_saved += shared_count; - break; - - case STACK_POP_PUSH: - /* [pop_count: varint] [push_count: varint] [new_frame_idx: varint]... */ - frame_buf_pos += encode_varint_u32(frame_buf, (uint32_t)pop_count); - frame_buf_pos += encode_varint_u32(frame_buf + frame_buf_pos, (uint32_t)push_count); - /* New frames are at the top (beginning) of current stack */ - for (size_t i = 0; i < push_count; i++) { - frame_buf_pos += encode_varint_u32(frame_buf + frame_buf_pos, frame_indices[i]); + frames->size += new_frames * sizeof(uint32_t); + block->size += new_frames * sizeof(uint32_t); + } + /* Complete stacks are cached so later samples can REUSE them */ + if (shared == 0) { + if (bucket != NULL) { + stack_cache_insert(bucket, (StackCacheEntry){ + hash, (uint32_t)stack_offset, (uint32_t)depth}); } - frames_written = push_count; - writer->stats.pop_push_records++; - /* Saved writing shared_count frames (stack_depth - push_count if we had written full) */ - writer->stats.frames_saved += shared_count; - break; - - default: - PyErr_SetString(PyExc_RuntimeError, "Invalid stack encoding type"); + writer->stats.full_records++; + } + else { + writer->stats.suffix_records++; + } + writer->stats.total_frames_written += new_frames; + writer->stats.frames_saved += shared; + return 0; +} + +static int +write_stack(BinaryWriter *writer, BlockStream *stream, int new_stream, + const uint32_t *stack, size_t depth) +{ + SampleBlock *block = writer->block; + size_t shared = common_outer_frames(stream, stack, depth); + /* Same stack as this stream's previous sample: extend its REPEAT */ + if (!new_stream && shared == depth && + shared == stream->prev_stack_depth) { + return write_stack_repeat(writer, stream, depth); + } + + /* Enough new frames to be worth looking up an earlier complete stack */ + Py_uhash_t hash = 0; + StackCacheEntry *bucket = NULL; + StackCacheEntry *cached = NULL; + if (depth - shared >= STACK_REUSE_MIN_FRAMES) { + hash = stack_hash(stack, depth); + bucket = block->cache[hash % STACK_CACHE_BUCKET_COUNT]; + cached = stack_cache_lookup(bucket, &block->columns[COLUMN_FRAME_IDS], + hash, stack, depth); + } + + int result; + if (cached != NULL) { + /* Stack already stored in this block: point to its frames */ + result = write_stack_reuse(writer, cached, depth); + } + else { + /* Keep the shared outer frames and write only the new inner ones */ + result = write_stack_replace_suffix(writer, bucket, hash, stack, + depth, shared); + } + if (result < 0) { return -1; } + stream->current_repeat = UINT32_MAX; + memcpy(stream->prev_stack, stack, depth * sizeof(uint32_t)); + stream->prev_stack_depth = depth; + return 0; +} + +size_t +binary_writer_buffered_size(BinaryWriter *writer) +{ + SampleBlock *block = writer->block; + if (block == NULL || block->sample_count == 0) { + return 0; + } + size_t size = block->columns[COLUMN_STREAM_TABLE].size + + block->columns[COLUMN_SAMPLE_STREAM_IDS].size + + block->columns[COLUMN_TIMESTAMP_RESIDUALS].size + + block->columns[COLUMN_STACK_INSTRUCTIONS].size; + const Repeat *repeats = (const Repeat *)block->repeats.data; + size_t repeat_count = block->repeats.size / sizeof(Repeat); + uint8_t bytes[MAX_VARINT_SIZE]; + for (size_t i = 0; i < repeat_count; i++) { + size += encode_varint_u64(bytes, STACK_REPEAT) + + encode_varint_u64(bytes, repeats[i].count); + } + uint8_t common_flags, bitmap_flags; + block_status_flags(block, &common_flags, &bitmap_flags); + size += (size_t)_Py_popcount32(bitmap_flags) + * status_bitmap_size(block->sample_count); + size += block->columns[COLUMN_FRAME_IDS].size / sizeof(uint32_t) + * block_frame_width(block); + return size; +} - if (writer_write_bytes(writer, frame_buf, frame_buf_pos) < 0) { +static int +block_add_stream(BinaryWriter *writer, ThreadEntry *entry) +{ + SampleBlock *block = writer->block; + if (GROW_ARRAY(block->streams, block->stream_count, + block->stream_capacity, BlockStream) < 0) { return -1; } + /* Stream table entry: thread_id (8 bytes) + interpreter_id (4 bytes) */ + uint8_t stream_entry[STREAM_ENTRY_SIZE]; + memcpy(stream_entry + STM_OFF_THREAD_ID, + &entry->thread_id, sizeof(entry->thread_id)); + memcpy(stream_entry + STM_OFF_INTERPRETER_ID, + &entry->interpreter_id, sizeof(entry->interpreter_id)); + if (column_write(block, COLUMN_STREAM_TABLE, + stream_entry, sizeof(stream_entry)) < 0) { + return -1; + } + BlockStream *stream = &block->streams[block->stream_count]; + stream->thread_index = (size_t)(entry - writer->thread_entries); + stream->prev_timestamp = block->base_timestamp - writer->sample_interval_us; + stream->current_repeat = UINT32_MAX; + stream->prev_stack_depth = 0; + entry->block_stream_index = block->stream_count++; + return 0; +} - writer->stats.total_frames_written += frames_written; +static int +block_add_sample(BinaryWriter *writer, ThreadEntry *entry, + uint64_t timestamp, uint8_t status, + const uint32_t *stack, size_t depth) +{ + static_assert(BLOCK_FLUSH_SIZE + MAX_SAMPLE_SIZE + MAX_BITMAP_PADDING == + MAX_BLOCK_SIZE, + "block flush size needs headroom for one complete sample"); + + SampleBlock *block = writer->block; + /* Start a new block if this sample might not fit */ + if (block->size >= BLOCK_FLUSH_SIZE || + (entry->block_stream_index == UINT32_MAX && + block->stream_count == MAX_BLOCK_STREAMS)) { + if (block_flush(writer) < 0) { + return -1; + } + } + if (block->sample_count == 0) { + block->base_timestamp = timestamp; + } + int new_stream = entry->block_stream_index == UINT32_MAX; + if (new_stream && block_add_stream(writer, entry) < 0) { + return -1; + } + BlockStream *stream = &block->streams[entry->block_stream_index]; + /* Store the difference from the expected timestamp */ + uint64_t residual = + timestamp - stream->prev_timestamp - writer->sample_interval_us; + uint8_t residual_bytes[MAX_VARINT_SIZE]; + size_t residual_size = + encode_varint_i64(residual_bytes, (int64_t)residual); + if (column_write_varint(block, COLUMN_SAMPLE_STREAM_IDS, + entry->block_stream_index) < 0 || + column_write(block, COLUMN_TIMESTAMP_RESIDUALS, + residual_bytes, residual_size) < 0 || + column_write(block, COLUMN_STATUS_BITMAPS, + &status, sizeof(status)) < 0) { + return -1; + } + if (write_stack(writer, stream, new_stream, stack, depth) < 0) { + return -1; + } + stream->prev_timestamp = timestamp; + block->sample_count++; + writer->total_samples++; return 0; } +static void +block_destroy(SampleBlock *block) +{ + if (block == NULL) { + return; + } + for (unsigned i = 0; i < COLUMN_COUNT; i++) { + buffer_clear(&block->columns[i]); + } + buffer_clear(&block->repeats); + buffer_clear(&block->transformed); + buffer_clear(&block->compressed); + buffer_clear(&block->output); + PyMem_Free(block->streams); + PyMem_Free(block); +} + BinaryWriter * binary_writer_create(PyObject *path, uint64_t sample_interval_us, int compression_type, uint64_t start_time_us, int profiling_mode, int capture_features) { + if (compression_type != COMPRESSION_NONE && + compression_type != COMPRESSION_ZSTD) { + PyErr_SetString(PyExc_ValueError, "invalid compression type"); + return NULL; + } BinaryWriter *writer = PyMem_Calloc(1, sizeof(BinaryWriter)); if (!writer) { PyErr_NoMemory(); @@ -761,12 +1085,18 @@ binary_writer_create(PyObject *path, uint64_t sample_interval_us, int compressio writer->profiling_mode = profiling_mode; writer->capture_features = capture_features; - writer->write_buffer = PyMem_Malloc(WRITE_BUFFER_SIZE); - if (!writer->write_buffer) { + writer->block = PyMem_Calloc(1, sizeof(SampleBlock)); + if (writer->block == NULL) { + PyErr_NoMemory(); + goto error; + } + writer->block->streams = PyMem_Malloc( + INITIAL_BLOCK_STREAM_CAPACITY * sizeof(BlockStream)); + if (writer->block->streams == NULL) { PyErr_NoMemory(); goto error; } - writer->buffer_size = WRITE_BUFFER_SIZE; + writer->block->stream_capacity = INITIAL_BLOCK_STREAM_CAPACITY; writer->string_hash = _Py_hashtable_new_full( string_hash_func, @@ -964,74 +1294,29 @@ process_thread_sample(BinaryWriter *writer, PyObject *thread_info, if (status_long == -1 && PyErr_Occurred()) { return -1; } + if (status_long < 0 || status_long > UINT8_MAX) { + writer->state = BINARY_WRITER_LIMIT_REACHED; + PyErr_Format(PyExc_OverflowError, + "thread status %ld exceeds the binary format range", + status_long); + return -1; + } uint8_t status = (uint8_t)status_long; ThreadEntry *entry = writer_get_or_create_thread_entry( - writer, thread_id, interpreter_id, NULL); + writer, thread_id, interpreter_id); if (!entry) { return -1; } - /* Calculate timestamp delta */ - uint64_t delta = timestamp_us - entry->prev_timestamp; - /* Process frames and build current stack */ uint32_t curr_stack[MAX_STACK_DEPTH]; size_t curr_depth; if (build_frame_stack(writer, frame_list, curr_stack, &curr_depth) < 0) { return -1; } - - /* Compare with previous stack to determine encoding */ - size_t shared_count, pop_count, push_count; - int encoding = compare_stacks( - entry->prev_stack, entry->prev_stack_depth, - curr_stack, curr_depth, - &shared_count, &pop_count, &push_count); - - if (encoding == STACK_REPEAT) { - /* Buffer this sample for RLE. - * - * STACK_REPEAT also covers the "first sample for a fresh thread, - * empty stack" case: a new ThreadEntry has prev_stack_depth == 0 - * and a zero-initialized prev_stack, so compare_stacks() returns - * STACK_REPEAT against an empty curr_stack (depth 0). Buffering - * it here is correct; the RLE flush path emits it as a normal - * STACK_REPEAT record. */ - if (entry->pending_rle == NULL) { - entry->pending_rle = PyMem_Malloc(MAX_RLE_BUF_SIZE); - if (!entry->pending_rle) { - PyErr_NoMemory(); - return -1; - } - } - if (entry->pending_rle_bytes + MAX_RLE_ENTRY_SIZE > MAX_RLE_BUF_SIZE - && flush_pending_rle(writer, entry) < 0) { - return -1; - } - entry->pending_rle_bytes += encode_varint_u64( - entry->pending_rle + entry->pending_rle_bytes, delta); - entry->pending_rle[entry->pending_rle_bytes++] = status; - entry->pending_rle_samples++; - } else { - /* Stack changed - flush any pending RLE first */ - if (flush_pending_rle(writer, entry) < 0) { - return -1; - } - - if (write_sample_with_encoding(writer, entry, delta, status, encoding, - curr_stack, curr_depth, - shared_count, pop_count, push_count) < 0) { - return -1; - } - - memcpy(entry->prev_stack, curr_stack, curr_depth * sizeof(uint32_t)); - entry->prev_stack_depth = curr_depth; - } - - entry->prev_timestamp = timestamp_us; - writer->total_samples++; - return 0; + return block_add_sample( + writer, entry, timestamp_us, status, curr_stack, curr_depth); } int @@ -1078,47 +1363,10 @@ binary_writer_write_sample(BinaryWriter *writer, PyObject *stack_frames, uint64_ int binary_writer_finalize(BinaryWriter *writer) { - for (size_t i = 0; i < writer->thread_count; i++) { - if (flush_pending_rle(writer, &writer->thread_entries[i]) < 0) { - return -1; - } - } - - if (writer_flush_buffer(writer) < 0) { + if (block_flush(writer) < 0) { return -1; } -#ifdef HAVE_ZSTD - /* Finalize compression stream */ - if (writer->compression_type == COMPRESSION_ZSTD && writer->zstd.cctx) { - ZSTD_inBuffer input = { NULL, 0, 0 }; - size_t remaining; - - do { - ZSTD_outBuffer output = { - writer->zstd.compressed_buffer, - writer->zstd.compressed_buffer_size, - 0 - }; - - remaining = ZSTD_compressStream2(writer->zstd.cctx, &output, &input, ZSTD_e_end); - - if (ZSTD_isError(remaining)) { - PyErr_Format(PyExc_IOError, "zstd finalization error: %s", - ZSTD_getErrorName(remaining)); - return -1; - } - - _Py_MSAN_UNPOISON(writer->zstd.compressed_buffer, output.pos); - if (output.pos > 0) { - if (fwrite_checked_allow_threads(writer->zstd.compressed_buffer, output.pos, writer->fp) < 0) { - return -1; - } - } - } while (remaining > 0); - } -#endif - /* Use 64-bit file position for >2GB files */ file_offset_t string_table_offset = FTELL64(writer->fp); if (string_table_offset < 0) { @@ -1314,13 +1562,12 @@ binary_writer_destroy(BinaryWriter *writer) Py_fclose(writer->fp); } - PyMem_Free(writer->write_buffer); + block_destroy(writer->block); #ifdef HAVE_ZSTD if (writer->zstd.cctx) { ZSTD_freeCCtx(writer->zstd.cctx); } - PyMem_Free(writer->zstd.compressed_buffer); #endif if (writer->string_hash) { @@ -1339,13 +1586,7 @@ binary_writer_destroy(BinaryWriter *writer) } PyMem_Free(writer->frame_entries); - if (writer->thread_entries) { - for (size_t i = 0; i < writer->thread_count; i++) { - PyMem_Free(writer->thread_entries[i].prev_stack); - PyMem_Free(writer->thread_entries[i].pending_rle); - } - PyMem_Free(writer->thread_entries); - } + PyMem_Free(writer->thread_entries); PyMem_Free(writer); } diff --git a/Modules/_remote_debugging/clinic/module.c.h b/Modules/_remote_debugging/clinic/module.c.h index 9895286cdb80196..28e5a3ca32ac803 100644 --- a/Modules/_remote_debugging/clinic/module.c.h +++ b/Modules/_remote_debugging/clinic/module.c.h @@ -1168,7 +1168,7 @@ PyDoc_STRVAR(_remote_debugging_BinaryWriter_get_stats__doc__, "Get encoding statistics for the writer.\n" "\n" "Returns a dict with encoding statistics including\n" -"repeat/full/suffix/pop-push record counts, frames written/saved, and\n" +"repeat/full/suffix/reuse record counts, frames written/saved, and\n" "compression ratio."); #define _REMOTE_DEBUGGING_BINARYWRITER_GET_STATS_METHODDEF \ @@ -1740,4 +1740,4 @@ _remote_debugging_get_gc_stats(PyObject *module, PyObject *const *args, Py_ssize exit: return return_value; } -/*[clinic end generated code: output=2667550106e045bd input=a9049054013a1b77]*/ +/*[clinic end generated code: output=ef0d1eaac8262c2c input=a9049054013a1b77]*/ diff --git a/Modules/_remote_debugging/frame_cache.c b/Modules/_remote_debugging/frame_cache.c index 1b2d9a60f408b91..63d1331015a548d 100644 --- a/Modules/_remote_debugging/frame_cache.c +++ b/Modules/_remote_debugging/frame_cache.c @@ -318,9 +318,8 @@ frame_cache_store( // Validate we have a complete stack before caching. // Only cache if last_frame_visited matches base_frame_addr (the sentinel // at the bottom of the stack). Note: we use last_frame_visited rather than - // addrs[num_addrs-1] because the base frame is visited but not added to the - // addrs array (it returns frame==NULL from is_frame_valid due to - // owner==FRAME_OWNED_BY_INTERPRETER). + // addrs[num_addrs-1] because the base frame's address is not added to the + // addrs array. if (base_frame_addr != 0 && last_frame_visited != base_frame_addr) { // Incomplete stack - don't cache (graceful degradation) return 0; diff --git a/Modules/_remote_debugging/frames.c b/Modules/_remote_debugging/frames.c index 19a9b5369148a98..3f8d628a6314e6e 100644 --- a/Modules/_remote_debugging/frames.c +++ b/Modules/_remote_debugging/frames.c @@ -36,7 +36,7 @@ process_single_stack_chunk( return -1; } - if (_Py_RemoteDebug_PagedReadRemoteMemory(&unwinder->handle, chunk_addr, current_size, this_chunk) < 0) { + if (_Py_RemoteDebug_ReadRemoteMemory(&unwinder->handle, chunk_addr, current_size, this_chunk) < 0) { PyMem_RawFree(this_chunk); set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to read stack chunk"); return -1; @@ -65,7 +65,7 @@ process_single_stack_chunk( } this_chunk = tmp; - if (_Py_RemoteDebug_PagedReadRemoteMemory(&unwinder->handle, chunk_addr, actual_size, this_chunk) < 0) { + if (_Py_RemoteDebug_ReadRemoteMemory(&unwinder->handle, chunk_addr, actual_size, this_chunk) < 0) { PyMem_RawFree(this_chunk); set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to reread stack chunk with correct size"); return -1; @@ -195,11 +195,8 @@ parse_frame_buffer( RemoteUnwinderObject *unwinder, PyObject** result, const char *frame, - uintptr_t* address_of_code_object, uintptr_t* previous_frame ) { - *address_of_code_object = 0; - *previous_frame = GET_MEMBER(uintptr_t, frame, unwinder->debug_offsets.interpreter_frame.previous); uintptr_t code_object = GET_MEMBER_NO_TAG(uintptr_t, frame, unwinder->debug_offsets.interpreter_frame.executable); int frame_valid = is_frame_valid(unwinder, (uintptr_t)frame, code_object); @@ -217,8 +214,6 @@ parse_frame_buffer( } #endif - *address_of_code_object = code_object; - CodeObjectContext code_ctx = { .code_addr = code_object, .instruction_pointer = instruction_pointer, @@ -232,7 +227,6 @@ parse_frame_object( RemoteUnwinderObject *unwinder, PyObject** result, uintptr_t address, - uintptr_t* address_of_code_object, uintptr_t* previous_frame ) { char frame[SIZEOF_INTERP_FRAME]; @@ -249,7 +243,7 @@ parse_frame_object( STATS_INC(unwinder, memory_reads); STATS_ADD(unwinder, memory_bytes_read, SIZEOF_INTERP_FRAME); - return parse_frame_buffer(unwinder, result, frame, address_of_code_object, previous_frame); + return parse_frame_buffer(unwinder, result, frame, previous_frame); } int @@ -258,41 +252,16 @@ parse_frame_from_chunks( PyObject **result, uintptr_t address, uintptr_t *previous_frame, - uintptr_t *stackpointer, StackChunkList *chunks ) { - void *frame_ptr = find_frame_in_chunks(chunks, address); - if (!frame_ptr) { + const char *frame = find_frame_in_chunks(chunks, address); + if (!frame) { PyErr_Format(PyExc_RuntimeError, "Frame at address 0x%lx not found in stack chunks", address); set_exception_cause(unwinder, PyExc_RuntimeError, "Frame not found in stack chunks"); return -1; } - char *frame = (char *)frame_ptr; - *previous_frame = GET_MEMBER(uintptr_t, frame, unwinder->debug_offsets.interpreter_frame.previous); - *stackpointer = GET_MEMBER(uintptr_t, frame, unwinder->debug_offsets.interpreter_frame.stackpointer); - uintptr_t code_object = GET_MEMBER_NO_TAG(uintptr_t, frame_ptr, unwinder->debug_offsets.interpreter_frame.executable); - int frame_valid = is_frame_valid(unwinder, (uintptr_t)frame, code_object); - if (frame_valid != 1) { - return frame_valid; - } - - uintptr_t instruction_pointer = GET_MEMBER(uintptr_t, frame, unwinder->debug_offsets.interpreter_frame.instr_ptr); - - // Get tlbc_index for free threading builds - int32_t tlbc_index = 0; -#ifdef Py_GIL_DISABLED - if (unwinder->debug_offsets.interpreter_frame.tlbc_index != 0) { - tlbc_index = GET_MEMBER(int32_t, frame, unwinder->debug_offsets.interpreter_frame.tlbc_index); - } -#endif - - CodeObjectContext code_ctx = { - .code_addr = code_object, - .instruction_pointer = instruction_pointer, - .tlbc_index = tlbc_index, - }; - return parse_code_object(unwinder, result, &code_ctx); + return parse_frame_buffer(unwinder, result, frame, previous_frame); } /* ============================================================================ @@ -316,7 +285,6 @@ process_frame_chain( int parse_result = 0; PyObject *frame = NULL; uintptr_t next_frame_addr = 0; - uintptr_t stackpointer = 0; last_frame_addr = frame_addr; if (++frame_count > MAX_FRAME_CHAIN_DEPTH) { @@ -326,30 +294,33 @@ process_frame_chain( } assert(frame_count <= MAX_FRAME_CHAIN_DEPTH); + // The base frame is a sentinel with previous == NULL: reaching it is + // enough, no need to read it. Still read it when it is the GC frame + // so the marker gets emitted. + if (frame_addr == ctx->base_frame_addr + && !(unwinder->gc && frame_addr == ctx->gc_frame)) { + break; + } + if (ctx->chunks && ctx->chunks->count > 0) { parse_result = parse_frame_from_chunks( - unwinder, &frame, frame_addr, &next_frame_addr, &stackpointer, ctx->chunks); + unwinder, &frame, frame_addr, &next_frame_addr, ctx->chunks); if (parse_result == 0) { goto parsed_frame; } PyErr_Clear(); } - { - uintptr_t address_of_code_object = 0; - if (ctx->prefetch.frame && ctx->prefetch.frame_addr == frame_addr) { - parse_result = parse_frame_buffer( - unwinder, &frame, ctx->prefetch.frame, - &address_of_code_object, &next_frame_addr); - } - else { - parse_result = parse_frame_object( - unwinder, &frame, frame_addr, - &address_of_code_object, &next_frame_addr); - } - if (parse_result < 0) { - set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to parse frame object in chain"); - return -1; - } + if (ctx->prefetch.frame && ctx->prefetch.frame_addr == frame_addr) { + parse_result = parse_frame_buffer( + unwinder, &frame, ctx->prefetch.frame, &next_frame_addr); + } + else { + parse_result = parse_frame_object( + unwinder, &frame, frame_addr, &next_frame_addr); + } + if (parse_result < 0) { + set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to parse frame object in chain"); + return -1; } parsed_frame: @@ -543,17 +514,16 @@ try_full_cache_hit( } PyObject *current_frame = NULL; - uintptr_t code_object_addr = 0; uintptr_t previous_frame = 0; int parse_result; if (ctx->prefetch.frame && ctx->prefetch.frame_addr == ctx->frame_addr) { parse_result = parse_frame_buffer(unwinder, ¤t_frame, ctx->prefetch.frame, - &code_object_addr, &previous_frame); + &previous_frame); } else { parse_result = parse_frame_object(unwinder, ¤t_frame, ctx->frame_addr, - &code_object_addr, &previous_frame); + &previous_frame); } if (parse_result < 0) { return -1; diff --git a/Modules/_remote_debugging/module.c b/Modules/_remote_debugging/module.c index 303af4a4505fea7..c77ff9ae0abd5b5 100644 --- a/Modules/_remote_debugging/module.c +++ b/Modules/_remote_debugging/module.c @@ -1842,8 +1842,8 @@ _remote_debugging_BinaryWriter_set_stats_impl(BinaryWriterObject *self, /* Finalize the writer, cache total_samples, and destroy it. * * The cache assignment must happen AFTER binary_writer_finalize(): finalize - * flushes pending RLE records and updates the file header before the writer is - * destroyed. Caching before finalize would preserve a count for a file that + * flushes the open sample block and updates the file header before the writer + * is destroyed. Caching before finalize would preserve a count for a file that * might still fail to finish. */ static int binary_writer_finalize_and_cache(BinaryWriterObject *self) @@ -1957,13 +1957,13 @@ _remote_debugging.BinaryWriter.get_stats Get encoding statistics for the writer. Returns a dict with encoding statistics including -repeat/full/suffix/pop-push record counts, frames written/saved, and +repeat/full/suffix/reuse record counts, frames written/saved, and compression ratio. [clinic start generated code]*/ static PyObject * _remote_debugging_BinaryWriter_get_stats_impl(BinaryWriterObject *self) -/*[clinic end generated code: output=06522cd52544df89 input=a8bb8c8682ccd34b]*/ +/*[clinic end generated code: output=06522cd52544df89 input=09c317b6289ae311]*/ { if (!self->writer) { PyErr_SetString(PyExc_ValueError, "Writer is closed"); @@ -2044,6 +2044,22 @@ static PyType_Spec BinaryWriter_spec = { #define BinaryReader_CAST(op) ((BinaryReaderObject *)(op)) +static int +binary_reader_close_checked(BinaryReaderObject *self) +{ + if (self->reader && self->reader->replaying) { + PyErr_SetString(PyExc_RuntimeError, + "Cannot close or reinitialize the reader " + "while it is replaying"); + return -1; + } + if (self->reader) { + binary_reader_close(self->reader); + self->reader = NULL; + } + return 0; +} + /*[clinic input] class _remote_debugging.BinaryReader "BinaryReaderObject *" "&PyBinaryReader_Type" [clinic start generated code]*/ @@ -2066,10 +2082,9 @@ _remote_debugging_BinaryReader___init___impl(BinaryReaderObject *self, PyObject *filename) /*[clinic end generated code: output=f04b33ee5c5e6dbf input=9d7cbe8b4f1a97c9]*/ { - if (self->reader) { - binary_reader_close(self->reader); + if (binary_reader_close_checked(self) < 0) { + return -1; } - self->reader = binary_reader_open(filename); if (!self->reader) { return -1; @@ -2163,9 +2178,8 @@ static PyObject * _remote_debugging_BinaryReader_close_impl(BinaryReaderObject *self) /*[clinic end generated code: output=ad0238cf5240b4f8 input=b919a66c737712d5]*/ { - if (self->reader) { - binary_reader_close(self->reader); - self->reader = NULL; + if (binary_reader_close_checked(self) < 0) { + return NULL; } Py_RETURN_NONE; } @@ -2200,9 +2214,8 @@ _remote_debugging_BinaryReader___exit___impl(BinaryReaderObject *self, PyObject *exc_tb) /*[clinic end generated code: output=2acdd36cfdc14e4a input=87284243d7935835]*/ { - if (self->reader) { - binary_reader_close(self->reader); - self->reader = NULL; + if (binary_reader_close_checked(self) < 0) { + return NULL; } Py_RETURN_FALSE; } diff --git a/Modules/_remote_debugging/object_reading.c b/Modules/_remote_debugging/object_reading.c index 6f138f3de3503b6..ed1f07c9143e0fc 100644 --- a/Modules/_remote_debugging/object_reading.c +++ b/Modules/_remote_debugging/object_reading.c @@ -185,9 +185,6 @@ read_py_bytes( uintptr_t address, Py_ssize_t max_len ) { - PyObject *result = NULL; - char *buf = NULL; - // Read the entire PyBytesObject at once char bytes_obj[SIZEOF_BYTES_OBJ]; int res = _Py_RemoteDebug_PagedReadRemoteMemory( @@ -198,7 +195,7 @@ read_py_bytes( ); if (res < 0) { set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to read PyBytesObject"); - goto err; + return NULL; } Py_ssize_t len = GET_MEMBER(Py_ssize_t, bytes_obj, unwinder->debug_offsets.bytes_object.ob_size); @@ -209,36 +206,26 @@ read_py_bytes( return NULL; } - buf = (char *)PyMem_RawMalloc(len+1); - if (buf == NULL) { - PyErr_NoMemory(); + PyBytesWriter *writer = PyBytesWriter_Create(len); + if (writer == NULL) { set_exception_cause(unwinder, PyExc_MemoryError, "Failed to allocate buffer for bytes reading"); return NULL; } size_t offset = (size_t)unwinder->debug_offsets.bytes_object.ob_sval; - res = _Py_RemoteDebug_PagedReadRemoteMemory(&unwinder->handle, address + offset, len, buf); + res = _Py_RemoteDebug_PagedReadRemoteMemory(&unwinder->handle, address + offset, + len, PyBytesWriter_GetData(writer)); if (res < 0) { + PyBytesWriter_Discard(writer); set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to read bytes data from remote memory"); - goto err; + return NULL; } - buf[len] = '\0'; - result = PyBytes_FromStringAndSize(buf, len); + PyObject *result = PyBytesWriter_Finish(writer); if (result == NULL) { set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to create PyBytes from remote bytes data"); - goto err; } - - PyMem_RawFree(buf); - assert(result != NULL); return result; - -err: - if (buf != NULL) { - PyMem_RawFree(buf); - } - return NULL; } long diff --git a/Modules/_testbuffer.c b/Modules/_testbuffer.c index 8b6b617aafa427d..8b37dc7bb34bed9 100644 --- a/Modules/_testbuffer.c +++ b/Modules/_testbuffer.c @@ -2171,7 +2171,6 @@ ndarray_tobytes(PyObject *self, PyObject *dummy) Py_buffer *src = &ndbuf->base; Py_buffer dest; PyObject *ret = NULL; - char *mem; if (ND_C_CONTIGUOUS(ndbuf->flags)) return PyBytes_FromStringAndSize(src->buf, src->len); @@ -2180,14 +2179,13 @@ ndarray_tobytes(PyObject *self, PyObject *dummy) assert(src->strides != NULL); assert(src->ndim > 0); - mem = PyMem_Malloc(src->len); - if (mem == NULL) { - PyErr_NoMemory(); + PyBytesWriter *writer = PyBytesWriter_Create(src->len); + if (writer == NULL) { return NULL; } dest = *src; - dest.buf = mem; + dest.buf = PyBytesWriter_GetData(writer); dest.suboffsets = NULL; dest.strides = strides_from_shape(ndbuf, 0); if (dest.strides == NULL) @@ -2195,11 +2193,12 @@ ndarray_tobytes(PyObject *self, PyObject *dummy) if (copy_buffer(&dest, src) < 0) goto out; - ret = PyBytes_FromStringAndSize(mem, src->len); + ret = PyBytesWriter_Finish(writer); + writer = NULL; out: PyMem_XFree(dest.strides); - PyMem_Free(mem); + PyBytesWriter_Discard(writer); return ret; } @@ -2445,7 +2444,7 @@ py_buffer_to_contiguous(PyObject *self, PyObject *args) int flags; char ord; Py_buffer view; - char *buf = NULL; + PyBytesWriter *writer = NULL; if (!PyArg_ParseTuple(args, "OOi", &obj, &order, &flags)) { return NULL; @@ -2460,21 +2459,21 @@ py_buffer_to_contiguous(PyObject *self, PyObject *args) goto out; } - buf = PyMem_Malloc(view.len); - if (buf == NULL) { - PyErr_NoMemory(); + writer = PyBytesWriter_Create(view.len); + if (writer == NULL) { goto out; } - if (PyBuffer_ToContiguous(buf, &view, view.len, ord) < 0) { + if (PyBuffer_ToContiguous(PyBytesWriter_GetData(writer), &view, view.len, ord) < 0) { goto out; } - ret = PyBytes_FromStringAndSize(buf, view.len); + ret = PyBytesWriter_Finish(writer); + writer = NULL; out: PyBuffer_Release(&view); - PyMem_XFree(buf); + PyBytesWriter_Discard(writer); return ret; } diff --git a/Modules/_testcapi/mem.c b/Modules/_testcapi/mem.c index 257568172de8b19..822ae661944a24b 100644 --- a/Modules/_testcapi/mem.c +++ b/Modules/_testcapi/mem.c @@ -13,6 +13,8 @@ # include // proc_pidinfo(), PROC_PIDTASKINFO # include // struct proc_taskinfo # endif +#else +# define TARGET_OS_OSX 0 #endif #ifdef __FreeBSD__ diff --git a/Modules/_testinternalcapi/test_cases.c.h b/Modules/_testinternalcapi/test_cases.c.h index 8e7ba1f69553dda..8bb3f857437792a 100644 --- a/Modules/_testinternalcapi/test_cases.c.h +++ b/Modules/_testinternalcapi/test_cases.c.h @@ -6799,6 +6799,8 @@ INSTRUCTION_STATS(GET_ANEXT); _PyStackRef aiter; _PyStackRef awaitable; + _PyStackRef null; + _PyStackRef none; aiter = stack_pointer[-1]; _PyFrame_SetStackPointer(frame, stack_pointer); _PyFrame_StackPointerValidate(frame); @@ -6808,8 +6810,12 @@ JUMP_TO_LABEL(error); } awaitable = PyStackRef_FromPyObjectSteal(awaitable_o); + null = PyStackRef_NULL; + none = PyStackRef_None; stack_pointer[0] = awaitable; - stack_pointer += 1; + stack_pointer[1] = null; + stack_pointer[2] = none; + stack_pointer += 3; ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__); DISPATCH(); } @@ -6824,6 +6830,8 @@ INSTRUCTION_STATS(GET_AWAITABLE); _PyStackRef iterable; _PyStackRef iter; + _PyStackRef null; + _PyStackRef none; iterable = stack_pointer[-1]; _PyFrame_SetStackPointer(frame, stack_pointer); _PyFrame_StackPointerValidate(frame); @@ -6839,8 +6847,12 @@ JUMP_TO_LABEL(error); } iter = PyStackRef_FromPyObjectSteal(iter_o); + null = PyStackRef_NULL; + none = PyStackRef_None; stack_pointer[0] = iter; - stack_pointer += 1; + stack_pointer[1] = null; + stack_pointer[2] = none; + stack_pointer += 3; ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__); DISPATCH(); } @@ -10120,7 +10132,7 @@ assert(_PyOpcode_Deopt[opcode] == (LOAD_GLOBAL)); JUMP_TO_PREDICTED(LOAD_GLOBAL); } - #if Py_GIL_DISABLED + #ifdef Py_GIL_DISABLED int increfed = _Py_TryIncrefCompareStackRef(&entries[index].me_value, res_o, &res); if (!increfed) { UPDATE_MISS_STATS(LOAD_GLOBAL); @@ -10187,7 +10199,7 @@ assert(_PyOpcode_Deopt[opcode] == (LOAD_GLOBAL)); JUMP_TO_PREDICTED(LOAD_GLOBAL); } - #if Py_GIL_DISABLED + #ifdef Py_GIL_DISABLED int increfed = _Py_TryIncrefCompareStackRef(&entries[index].me_value, res_o, &res); if (!increfed) { UPDATE_MISS_STATS(LOAD_GLOBAL); diff --git a/Modules/binascii.c b/Modules/binascii.c index 57684f8336e64ed..406234cac1ed238 100644 --- a/Modules/binascii.c +++ b/Modules/binascii.c @@ -2390,10 +2390,8 @@ binascii_a2b_qp_impl(PyObject *module, Py_buffer *data, int header, Py_ssize_t in, out; char ch; const unsigned char *ascii_data; - unsigned char *odata; unsigned char *stripped = NULL; Py_ssize_t datalen = 0; - PyObject *rv; ascii_data = data->buf; datalen = data->len; @@ -2443,14 +2441,12 @@ binascii_a2b_qp_impl(PyObject *module, Py_buffer *data, int header, } } - /* We allocate the output same size as input, this is overkill. - */ - odata = (unsigned char *) PyMem_Calloc(1, datalen); - if (odata == NULL) { + PyBytesWriter *writer = PyBytesWriter_Create(datalen); + if (writer == NULL) { PyMem_Free(stripped); - PyErr_NoMemory(); return NULL; } + unsigned char *odata = PyBytesWriter_GetData(writer); in = out = 0; while (in < datalen) { @@ -2492,10 +2488,8 @@ binascii_a2b_qp_impl(PyObject *module, Py_buffer *data, int header, out++; } } - rv = PyBytes_FromStringAndSize((char *)odata, out); - PyMem_Free(odata); PyMem_Free(stripped); - return rv; + return PyBytesWriter_FinishWithSize(writer, out); } static int @@ -2535,9 +2529,7 @@ binascii_b2a_qp_impl(PyObject *module, Py_buffer *data, int quotetabs, { Py_ssize_t in, out; const unsigned char *databuf; - unsigned char *odata; Py_ssize_t datalen = 0, odatalen = 0; - PyObject *rv; unsigned int linelen = 0; unsigned char ch; int crlf = 0; @@ -2622,13 +2614,11 @@ binascii_b2a_qp_impl(PyObject *module, Py_buffer *data, int quotetabs, odatalen += delta; } - /* We allocate the output same size as input, this is overkill. - */ - odata = (unsigned char *) PyMem_Calloc(1, odatalen); - if (odata == NULL) { - PyErr_NoMemory(); + PyBytesWriter *writer = PyBytesWriter_Create(odatalen); + if (writer == NULL) { return NULL; } + unsigned char *odata = PyBytesWriter_GetData(writer); in = out = linelen = 0; while (in < datalen) { @@ -2698,9 +2688,7 @@ binascii_b2a_qp_impl(PyObject *module, Py_buffer *data, int quotetabs, } } } - rv = PyBytes_FromStringAndSize((char *)odata, out); - PyMem_Free(odata); - return rv; + return PyBytesWriter_FinishWithSize(writer, out); } /* List of functions defined in the module */ diff --git a/Modules/clinic/_cursesmodule.c.h b/Modules/clinic/_cursesmodule.c.h index 553843ab96a7af7..39512bc6cad3d68 100644 --- a/Modules/clinic/_cursesmodule.c.h +++ b/Modules/clinic/_cursesmodule.c.h @@ -520,7 +520,7 @@ _curses_window_attrset(PyObject *self, PyObject *arg) return return_value; } -#if defined(HAVE_CURSES_WATTR_GET) +#if defined(_Py_HAVE_CURSES_WATTR_GET) PyDoc_STRVAR(_curses_window_attr_get__doc__, "attr_get($self, /)\n" @@ -540,9 +540,9 @@ _curses_window_attr_get(PyObject *self, PyObject *Py_UNUSED(ignored)) return _curses_window_attr_get_impl((PyCursesWindowObject *)self); } -#endif /* defined(HAVE_CURSES_WATTR_GET) */ +#endif /* defined(_Py_HAVE_CURSES_WATTR_GET) */ -#if defined(HAVE_CURSES_WATTR_SET) +#if defined(_Py_HAVE_CURSES_WATTR_SET) PyDoc_STRVAR(_curses_window_attr_set__doc__, "attr_set($self, attr, pair=0, /)\n" @@ -583,9 +583,9 @@ _curses_window_attr_set(PyObject *self, PyObject *const *args, Py_ssize_t nargs) return return_value; } -#endif /* defined(HAVE_CURSES_WATTR_SET) */ +#endif /* defined(_Py_HAVE_CURSES_WATTR_SET) */ -#if defined(HAVE_CURSES_WATTR_ON) +#if defined(_Py_HAVE_CURSES_WATTR_ON) PyDoc_STRVAR(_curses_window_attr_on__doc__, "attr_on($self, attr, /)\n" @@ -614,9 +614,9 @@ _curses_window_attr_on(PyObject *self, PyObject *arg) return return_value; } -#endif /* defined(HAVE_CURSES_WATTR_ON) */ +#endif /* defined(_Py_HAVE_CURSES_WATTR_ON) */ -#if defined(HAVE_CURSES_WATTR_OFF) +#if defined(_Py_HAVE_CURSES_WATTR_OFF) PyDoc_STRVAR(_curses_window_attr_off__doc__, "attr_off($self, attr, /)\n" @@ -645,9 +645,9 @@ _curses_window_attr_off(PyObject *self, PyObject *arg) return return_value; } -#endif /* defined(HAVE_CURSES_WATTR_OFF) */ +#endif /* defined(_Py_HAVE_CURSES_WATTR_OFF) */ -#if defined(HAVE_CURSES_WCOLOR_SET) +#if defined(_Py_HAVE_CURSES_WCOLOR_SET) PyDoc_STRVAR(_curses_window_color_set__doc__, "color_set($self, pair, /)\n" @@ -676,7 +676,7 @@ _curses_window_color_set(PyObject *self, PyObject *arg) return return_value; } -#endif /* defined(HAVE_CURSES_WCOLOR_SET) */ +#endif /* defined(_Py_HAVE_CURSES_WCOLOR_SET) */ PyDoc_STRVAR(_curses_window_getattrs__doc__, "getattrs($self, /)\n" @@ -2867,7 +2867,7 @@ _curses_filter(PyObject *module, PyObject *Py_UNUSED(ignored)) #endif /* defined(HAVE_CURSES_FILTER) */ -#if defined(HAVE_CURSES_NOFILTER) +#if defined(_Py_HAVE_CURSES_NOFILTER) PyDoc_STRVAR(_curses_nofilter__doc__, "nofilter($module, /)\n" @@ -2891,7 +2891,7 @@ _curses_nofilter(PyObject *module, PyObject *Py_UNUSED(ignored)) return _curses_nofilter_impl(module); } -#endif /* defined(HAVE_CURSES_NOFILTER) */ +#endif /* defined(_Py_HAVE_CURSES_NOFILTER) */ PyDoc_STRVAR(_curses_baudrate__doc__, "baudrate($module, /)\n" @@ -3566,7 +3566,7 @@ PyDoc_STRVAR(_curses_getwin__doc__, #define _CURSES_GETWIN_METHODDEF \ {"getwin", (PyCFunction)_curses_getwin, METH_O, _curses_getwin__doc__}, -#if defined(HAVE_CURSES_SCR_DUMP) +#if defined(_Py_HAVE_CURSES_SCR_DUMP) PyDoc_STRVAR(_curses_scr_dump__doc__, "scr_dump($module, filename, /)\n" @@ -3583,9 +3583,9 @@ PyDoc_STRVAR(_curses_scr_dump__doc__, #define _CURSES_SCR_DUMP_METHODDEF \ {"scr_dump", (PyCFunction)_curses_scr_dump, METH_O, _curses_scr_dump__doc__}, -#endif /* defined(HAVE_CURSES_SCR_DUMP) */ +#endif /* defined(_Py_HAVE_CURSES_SCR_DUMP) */ -#if defined(HAVE_CURSES_SCR_DUMP) +#if defined(_Py_HAVE_CURSES_SCR_DUMP) PyDoc_STRVAR(_curses_scr_restore__doc__, "scr_restore($module, filename, /)\n" @@ -3602,9 +3602,9 @@ PyDoc_STRVAR(_curses_scr_restore__doc__, #define _CURSES_SCR_RESTORE_METHODDEF \ {"scr_restore", (PyCFunction)_curses_scr_restore, METH_O, _curses_scr_restore__doc__}, -#endif /* defined(HAVE_CURSES_SCR_DUMP) */ +#endif /* defined(_Py_HAVE_CURSES_SCR_DUMP) */ -#if defined(HAVE_CURSES_SCR_DUMP) +#if defined(_Py_HAVE_CURSES_SCR_DUMP) PyDoc_STRVAR(_curses_scr_init__doc__, "scr_init($module, filename, /)\n" @@ -3621,9 +3621,9 @@ PyDoc_STRVAR(_curses_scr_init__doc__, #define _CURSES_SCR_INIT_METHODDEF \ {"scr_init", (PyCFunction)_curses_scr_init, METH_O, _curses_scr_init__doc__}, -#endif /* defined(HAVE_CURSES_SCR_DUMP) */ +#endif /* defined(_Py_HAVE_CURSES_SCR_DUMP) */ -#if defined(HAVE_CURSES_SCR_DUMP) && defined(HAVE_CURSES_SCR_SET) +#if defined(_Py_HAVE_CURSES_SCR_DUMP) && defined(_Py_HAVE_CURSES_SCR_SET) PyDoc_STRVAR(_curses_scr_set__doc__, "scr_set($module, filename, /)\n" @@ -3639,7 +3639,7 @@ PyDoc_STRVAR(_curses_scr_set__doc__, #define _CURSES_SCR_SET_METHODDEF \ {"scr_set", (PyCFunction)_curses_scr_set, METH_O, _curses_scr_set__doc__}, -#endif /* defined(HAVE_CURSES_SCR_DUMP) && defined(HAVE_CURSES_SCR_SET) */ +#endif /* defined(_Py_HAVE_CURSES_SCR_DUMP) && defined(_Py_HAVE_CURSES_SCR_SET) */ PyDoc_STRVAR(_curses_halfdelay__doc__, "halfdelay($module, tenths, /)\n" @@ -3778,7 +3778,7 @@ _curses_has_key(PyObject *module, PyObject *arg) #endif /* defined(HAVE_CURSES_HAS_KEY) */ -#if defined(HAVE_CURSES_DEFINE_KEY) +#if defined(_Py_HAVE_CURSES_DEFINE_KEY) PyDoc_STRVAR(_curses_define_key__doc__, "define_key($module, definition, keycode, /)\n" @@ -3834,9 +3834,9 @@ _curses_define_key(PyObject *module, PyObject *const *args, Py_ssize_t nargs) return return_value; } -#endif /* defined(HAVE_CURSES_DEFINE_KEY) */ +#endif /* defined(_Py_HAVE_CURSES_DEFINE_KEY) */ -#if defined(HAVE_CURSES_KEY_DEFINED) +#if defined(_Py_HAVE_CURSES_KEY_DEFINED) PyDoc_STRVAR(_curses_key_defined__doc__, "key_defined($module, definition, /)\n" @@ -3876,9 +3876,9 @@ _curses_key_defined(PyObject *module, PyObject *arg) return return_value; } -#endif /* defined(HAVE_CURSES_KEY_DEFINED) */ +#endif /* defined(_Py_HAVE_CURSES_KEY_DEFINED) */ -#if defined(HAVE_CURSES_KEYOK) +#if defined(_Py_HAVE_CURSES_KEYOK) PyDoc_STRVAR(_curses_keyok__doc__, "keyok($module, keycode, enable, /)\n" @@ -3921,7 +3921,7 @@ _curses_keyok(PyObject *module, PyObject *const *args, Py_ssize_t nargs) return return_value; } -#endif /* defined(HAVE_CURSES_KEYOK) */ +#endif /* defined(_Py_HAVE_CURSES_KEYOK) */ PyDoc_STRVAR(_curses_init_color__doc__, "init_color($module, color_number, r, g, b, /)\n" @@ -4364,7 +4364,7 @@ PyDoc_STRVAR(_curses_set_term__doc__, #define _CURSES_SET_TERM_METHODDEF \ {"set_term", (PyCFunction)_curses_set_term, METH_O, _curses_set_term__doc__}, -#if defined(HAVE_CURSES_NEW_PRESCR) +#if defined(_Py_HAVE_CURSES_NEW_PRESCR) PyDoc_STRVAR(_curses_new_prescr__doc__, "new_prescr($module, /)\n" @@ -4387,7 +4387,7 @@ _curses_new_prescr(PyObject *module, PyObject *Py_UNUSED(ignored)) return _curses_new_prescr_impl(module); } -#endif /* defined(HAVE_CURSES_NEW_PRESCR) */ +#endif /* defined(_Py_HAVE_CURSES_NEW_PRESCR) */ #if defined(HAVE_CURSES_ESCDELAY) @@ -4731,7 +4731,7 @@ _curses_meta(PyObject *module, PyObject *arg) return return_value; } -#if (defined(HAVE_CURSES_GETMOUSE) || defined(PDCURSES)) && (defined(HAVE_CURSES_HAS_MOUSE) || defined(PDCURSES)) +#if (defined(HAVE_CURSES_GETMOUSE) || defined(PDCURSES)) && (defined(_Py_HAVE_CURSES_HAS_MOUSE) || defined(PDCURSES)) PyDoc_STRVAR(_curses_has_mouse__doc__, "has_mouse($module, /)\n" @@ -4751,7 +4751,7 @@ _curses_has_mouse(PyObject *module, PyObject *Py_UNUSED(ignored)) return _curses_has_mouse_impl(module); } -#endif /* (defined(HAVE_CURSES_GETMOUSE) || defined(PDCURSES)) && (defined(HAVE_CURSES_HAS_MOUSE) || defined(PDCURSES)) */ +#endif /* (defined(HAVE_CURSES_GETMOUSE) || defined(PDCURSES)) && (defined(_Py_HAVE_CURSES_HAS_MOUSE) || defined(PDCURSES)) */ #if (defined(HAVE_CURSES_GETMOUSE) || defined(PDCURSES)) @@ -5645,7 +5645,7 @@ _curses_termattrs(PyObject *module, PyObject *Py_UNUSED(ignored)) return _curses_termattrs_impl(module); } -#if defined(HAVE_CURSES_TERM_ATTRS) +#if defined(_Py_HAVE_CURSES_TERM_ATTRS) PyDoc_STRVAR(_curses_term_attrs__doc__, "term_attrs($module, /)\n" @@ -5668,7 +5668,7 @@ _curses_term_attrs(PyObject *module, PyObject *Py_UNUSED(ignored)) return _curses_term_attrs_impl(module); } -#endif /* defined(HAVE_CURSES_TERM_ATTRS) */ +#endif /* defined(_Py_HAVE_CURSES_TERM_ATTRS) */ PyDoc_STRVAR(_curses_termname__doc__, "termname($module, /)\n" @@ -6244,7 +6244,7 @@ _curses_slk_attr(PyObject *module, PyObject *Py_UNUSED(ignored)) #endif /* (defined(NCURSES_EXT_FUNCS) || defined(PDCURSES)) */ -#if defined(HAVE_CURSES_SLK_ATTR_ON) +#if defined(_Py_HAVE_CURSES_SLK_ATTR_ON) PyDoc_STRVAR(_curses_slk_attr_on__doc__, "slk_attr_on($module, attr, /)\n" @@ -6273,9 +6273,9 @@ _curses_slk_attr_on(PyObject *module, PyObject *arg) return return_value; } -#endif /* defined(HAVE_CURSES_SLK_ATTR_ON) */ +#endif /* defined(_Py_HAVE_CURSES_SLK_ATTR_ON) */ -#if defined(HAVE_CURSES_SLK_ATTR_OFF) +#if defined(_Py_HAVE_CURSES_SLK_ATTR_OFF) PyDoc_STRVAR(_curses_slk_attr_off__doc__, "slk_attr_off($module, attr, /)\n" @@ -6304,9 +6304,9 @@ _curses_slk_attr_off(PyObject *module, PyObject *arg) return return_value; } -#endif /* defined(HAVE_CURSES_SLK_ATTR_OFF) */ +#endif /* defined(_Py_HAVE_CURSES_SLK_ATTR_OFF) */ -#if defined(HAVE_CURSES_SLK_ATTR_SET) +#if defined(_Py_HAVE_CURSES_SLK_ATTR_SET) PyDoc_STRVAR(_curses_slk_attr_set__doc__, "slk_attr_set($module, attr, pair=0, /)\n" @@ -6346,9 +6346,9 @@ _curses_slk_attr_set(PyObject *module, PyObject *const *args, Py_ssize_t nargs) return return_value; } -#endif /* defined(HAVE_CURSES_SLK_ATTR_SET) */ +#endif /* defined(_Py_HAVE_CURSES_SLK_ATTR_SET) */ -#if defined(HAVE_CURSES_SLK_COLOR) +#if defined(_Py_HAVE_CURSES_SLK_COLOR) PyDoc_STRVAR(_curses_slk_color__doc__, "slk_color($module, pair, /)\n" @@ -6377,7 +6377,7 @@ _curses_slk_color(PyObject *module, PyObject *arg) return return_value; } -#endif /* defined(HAVE_CURSES_SLK_COLOR) */ +#endif /* defined(_Py_HAVE_CURSES_SLK_COLOR) */ #if defined(HAVE_CURSES_USE_ENV) @@ -6715,4 +6715,4 @@ _curses_has_extended_color_support(PyObject *module, PyObject *Py_UNUSED(ignored #ifndef _CURSES_ASSUME_DEFAULT_COLORS_METHODDEF #define _CURSES_ASSUME_DEFAULT_COLORS_METHODDEF #endif /* !defined(_CURSES_ASSUME_DEFAULT_COLORS_METHODDEF) */ -/*[clinic end generated code: output=422461e62cb86010 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=417ad397dfaa96ce input=a9049054013a1b77]*/ diff --git a/Modules/clinic/posixmodule.c.h b/Modules/clinic/posixmodule.c.h index e992a4d1b8f7971..c2f183f70378367 100644 --- a/Modules/clinic/posixmodule.c.h +++ b/Modules/clinic/posixmodule.c.h @@ -6322,7 +6322,7 @@ os_wait(PyObject *module, PyObject *Py_UNUSED(ignored)) #endif /* defined(HAVE_WAIT) */ -#if (defined(HAVE_PIDFD_OPEN) || (defined(__linux__) && defined(__NR_pidfd_open) && !(defined(__ANDROID__) && __ANDROID_API__ < 31))) +#if (defined(_Py_HAVE_PIDFD_OPEN) || (defined(__linux__) && defined(__NR_pidfd_open) && !(defined(__ANDROID__) && __ANDROID_API__ < 31))) PyDoc_STRVAR(os_pidfd_open__doc__, "pidfd_open($module, /, pid, flags=0)\n" @@ -6397,9 +6397,9 @@ os_pidfd_open(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObjec return return_value; } -#endif /* (defined(HAVE_PIDFD_OPEN) || (defined(__linux__) && defined(__NR_pidfd_open) && !(defined(__ANDROID__) && __ANDROID_API__ < 31))) */ +#endif /* (defined(_Py_HAVE_PIDFD_OPEN) || (defined(__linux__) && defined(__NR_pidfd_open) && !(defined(__ANDROID__) && __ANDROID_API__ < 31))) */ -#if (defined(HAVE_PIDFD_GETFD) || (defined(__linux__) && defined(__NR_pidfd_getfd) && !(defined(__ANDROID__) && __ANDROID_API__ < 31))) +#if (defined(_Py_HAVE_PIDFD_GETFD) || (defined(__linux__) && defined(__NR_pidfd_getfd) && !(defined(__ANDROID__) && __ANDROID_API__ < 31))) PyDoc_STRVAR(os_pidfd_getfd__doc__, "pidfd_getfd($module, /, pidfd, targetfd, *, flags=0)\n" @@ -6484,7 +6484,7 @@ os_pidfd_getfd(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObje return return_value; } -#endif /* (defined(HAVE_PIDFD_GETFD) || (defined(__linux__) && defined(__NR_pidfd_getfd) && !(defined(__ANDROID__) && __ANDROID_API__ < 31))) */ +#endif /* (defined(_Py_HAVE_PIDFD_GETFD) || (defined(__linux__) && defined(__NR_pidfd_getfd) && !(defined(__ANDROID__) && __ANDROID_API__ < 31))) */ #if defined(HAVE_SETNS) @@ -13763,4 +13763,4 @@ os__emscripten_log(PyObject *module, PyObject *const *args, Py_ssize_t nargs, Py #ifndef OS__EMSCRIPTEN_LOG_METHODDEF #define OS__EMSCRIPTEN_LOG_METHODDEF #endif /* !defined(OS__EMSCRIPTEN_LOG_METHODDEF) */ -/*[clinic end generated code: output=c92854d21c8f6290 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=077c57e5f93bd790 input=a9049054013a1b77]*/ diff --git a/Modules/clinic/signalmodule.c.h b/Modules/clinic/signalmodule.c.h index 6125f253ac0380d..26139ff0fc0cd39 100644 --- a/Modules/clinic/signalmodule.c.h +++ b/Modules/clinic/signalmodule.c.h @@ -689,7 +689,7 @@ signal_pthread_kill(PyObject *module, PyObject *const *args, Py_ssize_t nargs) #endif /* defined(HAVE_PTHREAD_KILL) */ -#if (defined(HAVE_PIDFD_SEND_SIGNAL) || (defined(__linux__) && defined(__NR_pidfd_send_signal) && !(defined(__ANDROID__) && __ANDROID_API__ < 31))) +#if (defined(_Py_HAVE_PIDFD_SEND_SIGNAL) || (defined(__linux__) && defined(__NR_pidfd_send_signal) && !(defined(__ANDROID__) && __ANDROID_API__ < 31))) PyDoc_STRVAR(signal_pidfd_send_signal__doc__, "pidfd_send_signal($module, pidfd, signalnum, siginfo=None, flags=0, /)\n" @@ -754,7 +754,7 @@ signal_pidfd_send_signal(PyObject *module, PyObject *const *args, Py_ssize_t nar return return_value; } -#endif /* (defined(HAVE_PIDFD_SEND_SIGNAL) || (defined(__linux__) && defined(__NR_pidfd_send_signal) && !(defined(__ANDROID__) && __ANDROID_API__ < 31))) */ +#endif /* (defined(_Py_HAVE_PIDFD_SEND_SIGNAL) || (defined(__linux__) && defined(__NR_pidfd_send_signal) && !(defined(__ANDROID__) && __ANDROID_API__ < 31))) */ #ifndef SIGNAL_ALARM_METHODDEF #define SIGNAL_ALARM_METHODDEF @@ -807,4 +807,4 @@ signal_pidfd_send_signal(PyObject *module, PyObject *const *args, Py_ssize_t nar #ifndef SIGNAL_PIDFD_SEND_SIGNAL_METHODDEF #define SIGNAL_PIDFD_SEND_SIGNAL_METHODDEF #endif /* !defined(SIGNAL_PIDFD_SEND_SIGNAL_METHODDEF) */ -/*[clinic end generated code: output=2a04ec31f49b1c93 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=48d30418971f312a input=a9049054013a1b77]*/ diff --git a/Modules/clinic/socketmodule.c.h b/Modules/clinic/socketmodule.c.h index 09041d92a305793..5a1a128b304f028 100644 --- a/Modules/clinic/socketmodule.c.h +++ b/Modules/clinic/socketmodule.c.h @@ -2152,7 +2152,7 @@ _socket_if_nameindex(PyObject *module, PyObject *Py_UNUSED(ignored)) #endif /* (defined(HAVE_IF_NAMEINDEX) || defined(MS_WINDOWS_DESKTOP)) */ -#if (defined(HAVE_IF_NAMETOINDEX) || defined(MS_WINDOWS_DESKTOP)) +#if (defined(_Py_HAVE_IF_NAMETOINDEX) || defined(MS_WINDOWS_DESKTOP)) PyDoc_STRVAR(_socket_if_nametoindex__doc__, "if_nametoindex($module, oname, /)\n" @@ -2184,9 +2184,9 @@ _socket_if_nametoindex(PyObject *module, PyObject *arg) return return_value; } -#endif /* (defined(HAVE_IF_NAMETOINDEX) || defined(MS_WINDOWS_DESKTOP)) */ +#endif /* (defined(_Py_HAVE_IF_NAMETOINDEX) || defined(MS_WINDOWS_DESKTOP)) */ -#if (defined(HAVE_IF_INDEXTONAME) || defined(MS_WINDOWS_DESKTOP)) +#if (defined(_Py_HAVE_IF_INDEXTONAME) || defined(MS_WINDOWS_DESKTOP)) PyDoc_STRVAR(_socket_if_indextoname__doc__, "if_indextoname($module, if_index, /)\n" @@ -2215,7 +2215,7 @@ _socket_if_indextoname(PyObject *module, PyObject *arg) return return_value; } -#endif /* (defined(HAVE_IF_INDEXTONAME) || defined(MS_WINDOWS_DESKTOP)) */ +#endif /* (defined(_Py_HAVE_IF_INDEXTONAME) || defined(MS_WINDOWS_DESKTOP)) */ #if defined(CMSG_LEN) @@ -2449,4 +2449,4 @@ _socket_CMSG_SPACE(PyObject *module, PyObject *arg) #ifndef _SOCKET_CMSG_SPACE_METHODDEF #define _SOCKET_CMSG_SPACE_METHODDEF #endif /* !defined(_SOCKET_CMSG_SPACE_METHODDEF) */ -/*[clinic end generated code: output=bca43f33969d67ad input=a9049054013a1b77]*/ +/*[clinic end generated code: output=e6e677d3de0c63e4 input=a9049054013a1b77]*/ diff --git a/Modules/mmapmodule.c b/Modules/mmapmodule.c index ce5c59cb0212707..06099a3b8b2e0be 100644 --- a/Modules/mmapmodule.c +++ b/Modules/mmapmodule.c @@ -1567,22 +1567,16 @@ mmap_subscript_impl(mmap_object *self, PyObject *item) else if (step == 1) return _safe_PyBytes_FromStringAndSize(self->data + start, slicelen); else { - char *result_buf = (char *)PyMem_Malloc(slicelen); - PyObject *result; - - if (result_buf == NULL) - return PyErr_NoMemory(); - - if (safe_copy_to_slice(result_buf, self->data, start, step, - slicelen) < 0) - { - result = NULL; + PyBytesWriter *writer = PyBytesWriter_Create(slicelen); + if (writer == NULL) { + return NULL; } - else { - result = PyBytes_FromStringAndSize(result_buf, slicelen); + if (safe_copy_to_slice(PyBytesWriter_GetData(writer), self->data, + start, step, slicelen) < 0) { + PyBytesWriter_Discard(writer); + return NULL; } - PyMem_Free(result_buf); - return result; + return PyBytesWriter_Finish(writer); } } else { diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c index c6f89fc2454e88a..619154309b40e31 100644 --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -10870,7 +10870,7 @@ os_wait_impl(PyObject *module) // This system call always crashes on older Android versions. -#if defined(HAVE_PIDFD_OPEN) \ +#if defined(_Py_HAVE_PIDFD_OPEN) \ || (defined(__linux__) && defined(__NR_pidfd_open) \ && !(defined(__ANDROID__) && __ANDROID_API__ < 31)) /*[clinic input] @@ -10888,7 +10888,7 @@ static PyObject * os_pidfd_open_impl(PyObject *module, pid_t pid, unsigned int flags) /*[clinic end generated code: output=5c7252698947dc41 input=03058b32c389f874]*/ { -#ifdef HAVE_PIDFD_OPEN +#ifdef _Py_HAVE_PIDFD_OPEN int fd = pidfd_open(pid, flags); #else int fd = syscall(__NR_pidfd_open, pid, flags); @@ -10901,7 +10901,7 @@ os_pidfd_open_impl(PyObject *module, pid_t pid, unsigned int flags) #endif -#if defined(HAVE_PIDFD_GETFD) \ +#if defined(_Py_HAVE_PIDFD_GETFD) \ || (defined(__linux__) && defined(__NR_pidfd_getfd) \ && !(defined(__ANDROID__) && __ANDROID_API__ < 31)) /*[clinic input] @@ -10922,7 +10922,7 @@ os_pidfd_getfd_impl(PyObject *module, int pidfd, int targetfd, unsigned int flags) /*[clinic end generated code: output=e1a1415a13c7137f input=ef6417fb10deb1cc]*/ { -#ifdef HAVE_PIDFD_GETFD +#ifdef _Py_HAVE_PIDFD_GETFD int fd = pidfd_getfd(pidfd, targetfd, flags); #else int fd = syscall(__NR_pidfd_getfd, pidfd, targetfd, flags); diff --git a/Modules/readline.c b/Modules/readline.c index 643b1fcb040290d..fab0d9aeaa80aeb 100644 --- a/Modules/readline.c +++ b/Modules/readline.c @@ -1543,7 +1543,7 @@ setup_readline(readlinestate *mod_state) /* The name must be defined before initialization */ rl_readline_name = "python"; -#ifdef HAVE_RL_CHANGE_ENVIRONMENT +#ifdef _Py_HAVE_RL_CHANGE_ENVIRONMENT /* Prevent readline from setting the LINES and COLUMNS environment * variables: ncurses prefers them over an ioctl() query, so a stale value * left after a resize breaks SIGWINCH / KEY_RESIZE handling (gh-46927). */ diff --git a/Modules/signalmodule.c b/Modules/signalmodule.c index addf6b456b1f95c..85ed113a17e5e81 100644 --- a/Modules/signalmodule.c +++ b/Modules/signalmodule.c @@ -1304,7 +1304,7 @@ signal_pthread_kill_impl(PyObject *module, unsigned long thread_id, // This system call always crashes on older Android versions. -#if defined(HAVE_PIDFD_SEND_SIGNAL) \ +#if defined(_Py_HAVE_PIDFD_SEND_SIGNAL) \ || (defined(__linux__) && defined(__NR_pidfd_send_signal) \ && !(defined(__ANDROID__) && __ANDROID_API__ < 31)) /*[clinic input] @@ -1329,7 +1329,7 @@ signal_pidfd_send_signal_impl(PyObject *module, int pidfd, int signalnum, PyErr_SetString(PyExc_TypeError, "siginfo must be None"); return NULL; } -#ifdef HAVE_PIDFD_SEND_SIGNAL +#ifdef _Py_HAVE_PIDFD_SEND_SIGNAL int res = pidfd_send_signal(pidfd, signalnum, NULL, flags); #else int res = syscall(__NR_pidfd_send_signal, pidfd, signalnum, NULL, flags); diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c index 4df03284cbdd24f..7a76882e1633e10 100644 --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -606,14 +606,14 @@ _PyLong_##NAME##_Converter(PyObject *obj, void *ptr) \ return 1; \ } -#if defined(HAVE_IF_INDEXTONAME) || defined(MS_WINDOWS_DESKTOP) +#if defined(_Py_HAVE_IF_INDEXTONAME) || defined(MS_WINDOWS_DESKTOP) # ifdef MS_WINDOWS UNSIGNED_INT_CONVERTER(NetIfindex, NET_IFINDEX) # else # define _PyLong_NetIfindex_Converter _PyLong_UnsignedInt_Converter # define NET_IFINDEX unsigned int # endif -#endif // defined(HAVE_IF_INDEXTONAME) || defined(MS_WINDOWS_DESKTOP) +#endif // defined(_Py_HAVE_IF_INDEXTONAME) || defined(MS_WINDOWS_DESKTOP) /*[python input] class NET_IFINDEX_converter(CConverter): @@ -7403,7 +7403,7 @@ _socket_if_nameindex_impl(PyObject *module) #endif // defined(HAVE_IF_NAMEINDEX) || defined(MS_WINDOWS_DESKTOP) -#if defined(HAVE_IF_NAMETOINDEX) || defined(MS_WINDOWS_DESKTOP) +#if defined(_Py_HAVE_IF_NAMETOINDEX) || defined(MS_WINDOWS_DESKTOP) /*[clinic input] _socket.if_nametoindex @@ -7433,9 +7433,9 @@ _socket_if_nametoindex_impl(PyObject *module, PyObject *oname) return PyLong_FromUnsignedLong(index); } -#endif // defined(HAVE_IF_NAMETOINDEX) || defined(MS_WINDOWS_DESKTOP) +#endif // defined(_Py_HAVE_IF_NAMETOINDEX) || defined(MS_WINDOWS_DESKTOP) -#if defined(HAVE_IF_INDEXTONAME) || defined(MS_WINDOWS_DESKTOP) +#if defined(_Py_HAVE_IF_INDEXTONAME) || defined(MS_WINDOWS_DESKTOP) /*[clinic input] @permit_long_summary @@ -7460,7 +7460,7 @@ _socket_if_indextoname_impl(PyObject *module, NET_IFINDEX index) return PyUnicode_DecodeFSDefault(name); } -#endif // defined(HAVE_IF_INDEXTONAME) || defined(MS_WINDOWS_DESKTOP) +#endif // defined(_Py_HAVE_IF_INDEXTONAME) || defined(MS_WINDOWS_DESKTOP) #ifdef CMSG_LEN diff --git a/Objects/abstract.c b/Objects/abstract.c index 6bb1ac572b0c6e8..ff56cdb9efb39b8 100644 --- a/Objects/abstract.c +++ b/Objects/abstract.c @@ -2600,6 +2600,28 @@ check_class(PyObject *cls, const char *error) return -1; } +/* Return 1 if inst.__class__ would resolve to object's own getset descriptor, + i.e. evaluate to Py_TYPE(inst). Types with a custom tp_getattro or an + overridden __class__ return 0 and take the generic lookup. */ +static inline int +class_attr_is_type(PyObject *inst) +{ + PyTypeObject *tp = Py_TYPE(inst); + if (tp->tp_getattro != PyObject_GenericGetAttr) { + return 0; + } + _PyStackRef cref; + _PyType_LookupStackRefAndVersion(tp, &_Py_ID(__class__), &cref); + if (PyStackRef_IsNull(cref)) { + return 0; + } + PyObject *descr = PyStackRef_AsPyObjectBorrow(cref); + int res = (Py_IS_TYPE(descr, &PyGetSetDescr_Type) + && PyDescr_TYPE(descr) == &PyBaseObject_Type); + PyStackRef_CLOSE(cref); + return res; +} + static int object_isinstance(PyObject *inst, PyObject *cls) { @@ -2608,6 +2630,9 @@ object_isinstance(PyObject *inst, PyObject *cls) if (PyType_Check(cls)) { retval = PyObject_TypeCheck(inst, (PyTypeObject *)cls); if (retval == 0) { + if (class_attr_is_type(inst)) { + return 0; + } retval = PyObject_GetOptionalAttr(inst, &_Py_ID(__class__), &icls); if (icls != NULL) { if (icls != (PyObject *)(Py_TYPE(inst)) && PyType_Check(icls)) { diff --git a/Objects/bytes_methods.c b/Objects/bytes_methods.c index 414afeb7bb003c7..4021a0e82236942 100644 --- a/Objects/bytes_methods.c +++ b/Objects/bytes_methods.c @@ -377,6 +377,7 @@ _Py_bytes_maketrans(Py_buffer *frm, Py_buffer *to) } #define FASTSEARCH fastsearch +#define STRINGLIB_IS_UNICODE 0 #define STRINGLIB(F) stringlib_##F #define STRINGLIB_CHAR char #define STRINGLIB_SIZEOF_CHAR 1 diff --git a/Objects/dictobject.c b/Objects/dictobject.c index 5d3edbc4ed9578a..4a1d1b069a21ed8 100644 --- a/Objects/dictobject.c +++ b/Objects/dictobject.c @@ -8663,16 +8663,22 @@ frozendict_new(PyTypeObject *type, PyObject *args, PyObject *kwds) return d; } -static void +static int transfer_keys_and_values_lock_held(PyObject *res, PyObject *dict) { PyDictObject *new = (PyDictObject *)res; PyDictObject *old = (PyDictObject *)dict; assert(can_modify_dict(old)); + if (old->ma_values && old->ma_values->embedded) { + PyErr_SetString(PyExc_ValueError, + "can't convert dict with embedded values"); + return -1; + } + // Fast path: do nothing on an empty dict: if (old->ma_keys == Py_EMPTY_KEYS) { - return; + return 0; } Py_ssize_t used = old->ma_used; @@ -8690,6 +8696,7 @@ transfer_keys_and_values_lock_held(PyObject *res, PyObject *dict) new->ma_keys = keys; new->ma_values = values; ASSERT_CONSISTENT(new); + return 0; } PyObject * @@ -8705,9 +8712,14 @@ PyDict_AsFrozenDictAndClear(PyObject *dict) return NULL; } + int transfer; Py_BEGIN_CRITICAL_SECTION(dict); - transfer_keys_and_values_lock_held(res, dict); + transfer = transfer_keys_and_values_lock_held(res, dict); Py_END_CRITICAL_SECTION(); + if (transfer < 0) { + Py_DECREF(res); + return NULL; + } _PyObject_GC_TRACK(res); assert(_PyFrozenDictObject_CAST(res)->ma_hash == -1); diff --git a/Objects/iterobject.c b/Objects/iterobject.c index fcef1cc9c301434..683683ceaa6600c 100644 --- a/Objects/iterobject.c +++ b/Objects/iterobject.c @@ -238,11 +238,33 @@ PyCallIter_New(PyObject *callable, PyObject *sentinel) return _PyCallIter_NewEx(callable, sentinel, NULL); } +// Sharing a callable iterator between threads, or calling __setstate__() +// on an iterator that is in use, is not supported: the values produced are +// unspecified, but the interpreter must not crash. + +static inline PyObject * +calliter_swap(PyObject **field, PyObject *value) +{ +#ifdef Py_GIL_DISABLED + return _Py_atomic_exchange_ptr(field, value); +#else + PyObject *old = *field; + *field = value; + return old; +#endif +} + +static inline PyObject * +calliter_load(PyObject **field) +{ + return Py_XNewRef(FT_ATOMIC_LOAD_PTR_ACQUIRE(*field)); +} + static void calliter_exhaust(calliterobject *it) { - Py_CLEAR(it->it_callable); - Py_CLEAR(it->it_sentinel); + _PyObject_XDecRefDelayed(calliter_swap(&it->it_callable, NULL)); + _PyObject_XDecRefDelayed(calliter_swap(&it->it_sentinel, NULL)); } static void @@ -270,19 +292,26 @@ static PyObject * calliter_iternext(PyObject *op) { calliterobject *it = (calliterobject*)op; - PyObject *result; + PyObject *callable, *sentinel, *stop_exc, *result; - if (it->it_callable == NULL) { + callable = calliter_load(&it->it_callable); + if (callable == NULL) { return NULL; } + result = _PyObject_CallNoArgs(callable); + Py_DECREF(callable); - result = _PyObject_CallNoArgs(it->it_callable); /* The call can exhaust the iterator re-entrantly. */ - if (result != NULL && it->it_callable != NULL) { - if (it->it_sentinel == NULL) { + if (result != NULL && + FT_ATOMIC_LOAD_PTR_RELAXED(it->it_callable) != NULL) + { + sentinel = calliter_load(&it->it_sentinel); + if (sentinel == NULL) { return result; /* Common case, fast path */ } - int ok = PyObject_RichCompareBool(it->it_sentinel, result, Py_EQ); + /* The comparison can run code that replaces the sentinel. */ + int ok = PyObject_RichCompareBool(sentinel, result, Py_EQ); + Py_DECREF(sentinel); if (ok == 0) { return result; /* Common case, fast path */ } @@ -291,14 +320,19 @@ calliter_iternext(PyObject *op) calliter_exhaust(it); } } - else if (PyErr_ExceptionMatches(it->it_stop_exc)) { - PyErr_Clear(); - calliter_exhaust(it); - } - else if (PyErr_ExceptionMatches(PyExc_StopIteration)) { - /* It would be mistaken for the end of the iteration (see PEP 479). */ - _PyErr_FormatFromCause(PyExc_RuntimeError, - "callable raised StopIteration"); + else if (result == NULL) { + stop_exc = calliter_load(&it->it_stop_exc); + if (PyErr_ExceptionMatches(stop_exc)) { + PyErr_Clear(); + calliter_exhaust(it); + } + else if (PyErr_ExceptionMatches(PyExc_StopIteration)) { + /* It would be mistaken for the end of the iteration (see + PEP 479). */ + _PyErr_FormatFromCause(PyExc_RuntimeError, + "callable raised StopIteration"); + } + Py_DECREF(stop_exc); } Py_XDECREF(result); return NULL; @@ -314,22 +348,29 @@ calliter_reduce(PyObject *op, PyObject *Py_UNUSED(ignored)) * call must be before access of iterator pointers. * see issue #101765 */ - if (it->it_callable == NULL) { - return Py_BuildValue("N(())", iter); + PyObject *callable = calliter_load(&it->it_callable); + PyObject *sentinel = calliter_load(&it->it_sentinel); + PyObject *stop_exc = calliter_load(&it->it_stop_exc); + PyObject *res; + if (callable == NULL) { + res = Py_BuildValue("N(())", iter); } /* Only the sentinel can be passed as an argument of iter(), so other attributes are restored from the state (see calliter_setstate()). */ - if (it->it_sentinel == NULL) { - return Py_BuildValue("N(OO)(()O)", iter, it->it_callable, Py_None, - it->it_stop_exc); + else if (sentinel == NULL) { + res = Py_BuildValue("N(OO)(()O)", iter, callable, Py_None, stop_exc); } - else if (it->it_stop_exc == PyExc_StopIteration) { - return Py_BuildValue("N(OO)", iter, it->it_callable, it->it_sentinel); + else if (stop_exc == PyExc_StopIteration) { + res = Py_BuildValue("N(OO)", iter, callable, sentinel); } else { - return Py_BuildValue("N(OO)((O)O)", iter, it->it_callable, Py_None, - it->it_sentinel, it->it_stop_exc); + res = Py_BuildValue("N(OO)((O)O)", iter, callable, Py_None, + sentinel, stop_exc); } + Py_XDECREF(callable); + Py_XDECREF(sentinel); + Py_DECREF(stop_exc); + return res; } static PyObject * @@ -349,11 +390,12 @@ calliter_setstate(PyObject *op, PyObject *state) if (_PyEval_CheckExceptTypeValid(_PyThreadState_GET(), stop_exc) < 0) { return NULL; } - if (it->it_callable != NULL) { - Py_XSETREF(it->it_sentinel, - PyTuple_GET_SIZE(sentinel) ? - Py_NewRef(PyTuple_GET_ITEM(sentinel, 0)) : NULL); - Py_SETREF(it->it_stop_exc, Py_NewRef(stop_exc)); + if (FT_ATOMIC_LOAD_PTR_RELAXED(it->it_callable) != NULL) { + PyObject *value = PyTuple_GET_SIZE(sentinel) ? + Py_NewRef(PyTuple_GET_ITEM(sentinel, 0)) : NULL; + _PyObject_XDecRefDelayed(calliter_swap(&it->it_sentinel, value)); + _PyObject_XDecRefDelayed(calliter_swap(&it->it_stop_exc, + Py_NewRef(stop_exc))); } Py_RETURN_NONE; diff --git a/Objects/longobject.c b/Objects/longobject.c index 09bfe4ad4641281..4a3117c43de5e12 100644 --- a/Objects/longobject.c +++ b/Objects/longobject.c @@ -3755,7 +3755,13 @@ x_add(PyLongObject *a, PyLongObject *b) size_a = size_b; size_b = size_temp; } } - z = long_alloc(size_a+1); + assert(size_a >= 1); + /* A carry out of the top digit is only possible if the top digits sum + to at least PyLong_MASK; only then allocate a digit for it. */ + digit top_sum = a->long_value.ob_digit[size_a - 1] + + (size_b == size_a ? b->long_value.ob_digit[size_b - 1] : 0); + int extra_digit = top_sum >= PyLong_MASK; + z = long_alloc(size_a + extra_digit); if (z == NULL) return NULL; for (i = 0; i < size_b; ++i) { @@ -3768,8 +3774,13 @@ x_add(PyLongObject *a, PyLongObject *b) z->long_value.ob_digit[i] = carry & PyLong_MASK; carry >>= PyLong_SHIFT; } - z->long_value.ob_digit[i] = carry; - return long_normalize(z); + if (extra_digit) { + z->long_value.ob_digit[i] = carry; + return long_normalize(z); + } + assert(carry == 0); + assert(z->long_value.ob_digit[i - 1] != 0); + return z; } /* Subtract the absolute values of two integers. */ diff --git a/Objects/mimalloc/arena-abandon.c b/Objects/mimalloc/arena-abandon.c index 460c80fc22782ff..fdc2a4a902a1daa 100644 --- a/Objects/mimalloc/arena-abandon.c +++ b/Objects/mimalloc/arena-abandon.c @@ -240,17 +240,20 @@ static mi_segment_t* mi_arena_segment_clear_abandoned_next_field(mi_arena_field_ mi_arena_t* arena = mi_arena_from_index(arena_idx); if (arena != NULL) { bool has_lock = false; + // CPython: gh-161751: for a full visit, take the lock before reading the bitmap. + // A reclaim from another sub-process can clear our bit for a short time. + if (previous->visit_all && mi_option_is_enabled(mi_option_visit_abandoned)) { + mi_lock_acquire(&arena->abandoned_visit_lock); + has_lock = true; + } // visit the abandoned fields (starting at previous_idx) for (; field_idx < arena->field_count; field_idx++, bit_idx = 0) { size_t field = mi_atomic_load_relaxed(&arena->blocks_abandoned[field_idx]); if mi_unlikely(field != 0) { // skip zero fields quickly - // we only take the arena lock if there are actually abandoned segments present + // a reclaim tries to take the lock only when the field has abandoned segments if (!has_lock && mi_option_is_enabled(mi_option_visit_abandoned)) { - has_lock = (previous->visit_all ? (mi_lock_acquire(&arena->abandoned_visit_lock),true) : mi_lock_try_acquire(&arena->abandoned_visit_lock)); + has_lock = mi_lock_try_acquire(&arena->abandoned_visit_lock); if (!has_lock) { - if (previous->visit_all) { - _mi_error_message(EFAULT, "internal error: failed to visit all abandoned segments due to failure to acquire the visitor lock"); - } // skip to next arena break; } diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index 0fe765c36848f25..5b9e8985d1bea8e 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -14737,7 +14737,7 @@ intern_common(PyInterpreterState *interp, PyObject *s /* stolen */, /* Statically allocated strings must be already interned. */ assert(!_PyUnicode_STATE(s).statically_allocated); -#if Py_GIL_DISABLED +#ifdef Py_GIL_DISABLED /* In the free-threaded build, all interned strings are immortal */ immortalize = 1; #endif diff --git a/PC/pyconfig.h b/PC/pyconfig.h index f969525c1db5c7c..9de3786a0c769b4 100644 --- a/PC/pyconfig.h +++ b/PC/pyconfig.h @@ -98,17 +98,15 @@ WIN32 is still required for the locale module. #endif /* Define to 1 when compiling for experimental free-threaded builds */ -#ifdef Py_GIL_DISABLED +#if defined(Py_GIL_DISABLED) && Py_GIL_DISABLED == 0 /* We undefine if it was set to zero because all later checks are #ifdef. * Note that non-Windows builds do not do this, and so every effort should * be made to avoid defining the variable at all when not desired. However, * sysconfig.get_config_var always returns a 1 or a 0, and so it seems likely * that a build backend will define it with the value. */ -#if Py_GIL_DISABLED == 0 #undef Py_GIL_DISABLED #endif -#endif /* Compiler specific defines */ diff --git a/PCbuild/pyproject.props b/PCbuild/pyproject.props index ca8330c004cccea..67296c22806c3e4 100644 --- a/PCbuild/pyproject.props +++ b/PCbuild/pyproject.props @@ -52,7 +52,7 @@ <_DebugPreprocessorDefinition>NDEBUG; <_DebugPreprocessorDefinition Condition="$(Configuration) == 'Debug'">_DEBUG; <_PyStatsPreprocessorDefinition>PyStats; - <_PyStatsPreprocessorDefinition Condition="$(PySTATS) != ''">Py_STATS; + <_PyStatsPreprocessorDefinition Condition="$(PySTATS) != ''">Py_STATS=1; <_PlatformPreprocessorDefinition>_WIN32; <_PlatformPreprocessorDefinition Condition="$(Platform) == 'x64'">_WIN64; <_PlatformPreprocessorDefinition Condition="$(Platform) == 'x64' and $(PlatformToolset) != 'ClangCL'">_M_X64;$(_PlatformPreprocessorDefinition) diff --git a/Platforms/Apple/testbed/Python.xcframework/build/utils.sh b/Platforms/Apple/testbed/Python.xcframework/build/utils.sh index bbe3620a1397311..53801215377fd86 100755 --- a/Platforms/Apple/testbed/Python.xcframework/build/utils.sh +++ b/Platforms/Apple/testbed/Python.xcframework/build/utils.sh @@ -14,7 +14,7 @@ # * There is an `app_packages` folder that contains installed Python packages. # ----- # set -e -# source $PROJECT_DIR/Python.xcframework/build/build_utils.sh +# source "$PROJECT_DIR/Python.xcframework/build/utils.sh" # install_python Python.xcframework app app_packages # ----- @@ -101,9 +101,9 @@ install_dylib () { MODULE_PATH=$(dirname "$FULL_EXT") MODULE_NAME=$(echo $EXT | cut -d "." -f 1) # The location of the extension file, relative to the bundle - RELATIVE_EXT=${FULL_EXT#$CODESIGNING_FOLDER_PATH/} + RELATIVE_EXT=${FULL_EXT#"$CODESIGNING_FOLDER_PATH"/} # The path to the extension file, relative to the install base - PYTHON_EXT=${RELATIVE_EXT/$INSTALL_BASE/} + PYTHON_EXT=${RELATIVE_EXT/"$INSTALL_BASE"/} # The full dotted name of the extension module, constructed from the file path. FULL_MODULE_NAME=$(echo $PYTHON_EXT | cut -d "." -f 1 | tr "/" "."); # A bundle identifier; not actually used, but required by Xcode framework packaging @@ -152,8 +152,8 @@ install_dylib () { process_dylibs () { PYTHON_XCFRAMEWORK_PATH=$1 LIB_PATH=$2 - find "$CODESIGNING_FOLDER_PATH/$LIB_PATH" -name "*.so" | while read FULL_EXT; do - install_dylib $PYTHON_XCFRAMEWORK_PATH "$LIB_PATH/" "$FULL_EXT" + find "$CODESIGNING_FOLDER_PATH/$LIB_PATH" -name "*.so" | while read -r FULL_EXT; do + install_dylib "$PYTHON_XCFRAMEWORK_PATH" "$LIB_PATH/" "$FULL_EXT" done } @@ -168,13 +168,13 @@ install_python() { PYTHON_XCFRAMEWORK_PATH=$1 shift - install_stdlib $PYTHON_XCFRAMEWORK_PATH + install_stdlib "$PYTHON_XCFRAMEWORK_PATH" PYTHON_VER=$(ls -1 "$CODESIGNING_FOLDER_PATH/python/lib" | grep -E "^python3\.\d+[a-z]*$") echo "Install Python $PYTHON_VER standard library extension modules..." - process_dylibs $PYTHON_XCFRAMEWORK_PATH python/lib/$PYTHON_VER/lib-dynload + process_dylibs "$PYTHON_XCFRAMEWORK_PATH" python/lib/$PYTHON_VER/lib-dynload - for package_path in $@; do + for package_path in "$@"; do echo "Installing $package_path extension modules ..." - process_dylibs $PYTHON_XCFRAMEWORK_PATH $package_path + process_dylibs "$PYTHON_XCFRAMEWORK_PATH" "$package_path" done } diff --git a/Platforms/Apple/testbed/iOSTestbed.xcodeproj/project.pbxproj b/Platforms/Apple/testbed/iOSTestbed.xcodeproj/project.pbxproj index 6ac40a325045bd6..b2131822424044e 100644 --- a/Platforms/Apple/testbed/iOSTestbed.xcodeproj/project.pbxproj +++ b/Platforms/Apple/testbed/iOSTestbed.xcodeproj/project.pbxproj @@ -250,7 +250,7 @@ ); runOnlyForDeploymentPostprocessing = 0; shellPath = /bin/sh; - shellScript = "set -e\nsource $PROJECT_DIR/Python{{abiflags}}.xcframework/build/utils.sh\ninstall_python Python{{abiflags}}.xcframework app app_packages\n"; + shellScript = "set -e\nsource \"$PROJECT_DIR/Python{{abiflags}}.xcframework/build/utils.sh\"\ninstall_python Python{{abiflags}}.xcframework app app_packages\n"; showEnvVarsInLog = 0; }; /* End PBXShellScriptBuildPhase section */ diff --git a/Platforms/Apple/testbed/tvOSTestbed.xcodeproj/project.pbxproj b/Platforms/Apple/testbed/tvOSTestbed.xcodeproj/project.pbxproj index e14489ad0770a0f..941f6a266bc42a0 100644 --- a/Platforms/Apple/testbed/tvOSTestbed.xcodeproj/project.pbxproj +++ b/Platforms/Apple/testbed/tvOSTestbed.xcodeproj/project.pbxproj @@ -236,7 +236,7 @@ ); runOnlyForDeploymentPostprocessing = 0; shellPath = /bin/sh; - shellScript = "set -e\n\nsource $PROJECT_DIR/Python{{abiflags}}.xcframework/build/utils.sh\n\ninstall_python Python{{abiflags}}.xcframework app app_packages\n"; + shellScript = "set -e\n\nsource \"$PROJECT_DIR/Python{{abiflags}}.xcframework/build/utils.sh\"\n\ninstall_python Python{{abiflags}}.xcframework app app_packages\n"; showEnvVarsInLog = 0; }; /* End PBXShellScriptBuildPhase section */ diff --git a/Platforms/Apple/testbed/visionOSTestbed.xcodeproj/project.pbxproj b/Platforms/Apple/testbed/visionOSTestbed.xcodeproj/project.pbxproj index ee19e22d533cc83..53ddec1b6f29e40 100644 --- a/Platforms/Apple/testbed/visionOSTestbed.xcodeproj/project.pbxproj +++ b/Platforms/Apple/testbed/visionOSTestbed.xcodeproj/project.pbxproj @@ -250,7 +250,7 @@ ); runOnlyForDeploymentPostprocessing = 0; shellPath = /bin/sh; - shellScript = "set -e\n\nsource $PROJECT_DIR/Python{{abiflags}}.xcframework/build/utils.sh\n\ninstall_python Python{{abiflags}}.xcframework app app_packages\n"; + shellScript = "set -e\n\nsource \"$PROJECT_DIR/Python{{abiflags}}.xcframework/build/utils.sh\"\n\ninstall_python Python{{abiflags}}.xcframework app app_packages\n"; showEnvVarsInLog = 0; }; /* End PBXShellScriptBuildPhase section */ diff --git a/Platforms/Apple/testbed/watchOSTestbed.xcodeproj/project.pbxproj b/Platforms/Apple/testbed/watchOSTestbed.xcodeproj/project.pbxproj index d5220c60e65c441..02b98a685cea6c0 100644 --- a/Platforms/Apple/testbed/watchOSTestbed.xcodeproj/project.pbxproj +++ b/Platforms/Apple/testbed/watchOSTestbed.xcodeproj/project.pbxproj @@ -236,7 +236,7 @@ ); runOnlyForDeploymentPostprocessing = 0; shellPath = /bin/sh; - shellScript = "set -e\n\nsource $PROJECT_DIR/Python{{abiflags}}.xcframework/build/utils.sh\n\ninstall_python Python{{abiflags}}.xcframework app app_packages\n"; + shellScript = "set -e\n\nsource \"$PROJECT_DIR/Python{{abiflags}}.xcframework/build/utils.sh\"\n\ninstall_python Python{{abiflags}}.xcframework app app_packages\n"; showEnvVarsInLog = 0; }; /* End PBXShellScriptBuildPhase section */ diff --git a/Platforms/emscripten/config.site-wasm32-emscripten b/Platforms/emscripten/config.site-wasm32-emscripten index 3c9a1c2c2568b0e..62640cc772d7e9b 100644 --- a/Platforms/emscripten/config.site-wasm32-emscripten +++ b/Platforms/emscripten/config.site-wasm32-emscripten @@ -19,7 +19,6 @@ ac_cv_func_posix_spawn=no ac_cv_func_posix_spawnp=no ac_cv_func_eventfd=no ac_cv_func_memfd_create=no -ac_cv_func_prlimit=no # The rest is based on pyodide # https://github.com/pyodide/pyodide/blob/main/cpython/pyconfig.undefs.h @@ -45,8 +44,6 @@ ac_cv_func_seteuid=no ac_cv_func_setegid=no ac_cv_func_getresuid=no ac_cv_func_getresgid=no -ac_cv_func_setresuid=no -ac_cv_func_setresgid=no # Syscalls not implemented in emscripten # [Errno 52] Function not implemented @@ -89,7 +86,7 @@ ac_cv_func_setresuid=no ac_cv_func_setresgid=no # Emscripten does not support hard links, always fails with errno 34 -# "Too many links". See emscripten_syscall_stubs.c +# "Too many links". ac_cv_func_link=no ac_cv_func_linkat=no diff --git a/Platforms/emscripten/config.toml b/Platforms/emscripten/config.toml index 39835844bbb365f..68ac87371fb9030 100644 --- a/Platforms/emscripten/config.toml +++ b/Platforms/emscripten/config.toml @@ -17,8 +17,8 @@ pythoninfo-args = [ [dependencies.libffi] url = "https://github.com/libffi/libffi/releases/download/v{version}/libffi-{version}.tar.gz" -version = "3.4.6" -shasum = "b0dea9df23c863a7a50e825440f3ebffabd65df1497108e5d437747843895a4e" +version = "3.8.1" +shasum = "81e879c094cc5ec3186c48510c6385ac6b3e866192e8bc71a35245703757ee40" [dependencies.mpdec] url = "https://www.bytereef.org/software/mpdecimal/releases/mpdecimal-{version}.tar.gz" diff --git a/Platforms/emscripten/make_libffi.sh b/Platforms/emscripten/make_libffi.sh index 3c75c4d51271025..c05e7c92904a811 100755 --- a/Platforms/emscripten/make_libffi.sh +++ b/Platforms/emscripten/make_libffi.sh @@ -1,7 +1,7 @@ #!/bin/bash set +e -export CFLAGS="-O2 -fPIC -DWASM_BIGINT" +export CFLAGS="-O2 -fPIC" export CXXFLAGS="$CFLAGS" # Build paths diff --git a/Python/bltinmodule.c b/Python/bltinmodule.c index 72f2e356c34f7c8..2e597ade6d0dc83 100644 --- a/Python/bltinmodule.c +++ b/Python/bltinmodule.c @@ -18,6 +18,7 @@ #include "pycore_pystate.h" // _PyThreadState_GET() #include "pycore_pythonrun.h" // _Py_SourceAsString() #include "pycore_tuple.h" // _PyTuple_Recycle() +#include "pycore_unicodeobject.h" // _Py_LATIN1_CHR() #include "clinic/bltinmodule.c.h" @@ -2240,8 +2241,6 @@ builtin_print_impl(PyObject *module, PyObject * const *objects, PyObject *file, int flush) /*[clinic end generated code: output=38d8def56c837bcc input=ff35cb3d59ee8115]*/ { - int i; - if (file == Py_None) { file = PySys_GetAttr(&_Py_ID(stdout)); if (file == NULL) { @@ -2279,29 +2278,106 @@ builtin_print_impl(PyObject *module, PyObject * const *objects, return NULL; } - for (i = 0; i < objects_length; i++) { - if (i > 0 && PyFile_WriteObject(sep, file, Py_PRINT_RAW) < 0) { - Py_DECREF(file); - return NULL; - } - if (PyFile_WriteObject(objects[i], file, Py_PRINT_RAW) < 0) { - Py_DECREF(file); - return NULL; - } + if (objects_length > PY_SSIZE_T_MAX / 2) { + Py_DECREF(file); + return PyErr_NoMemory(); } - if (PyFile_WriteObject(end, file, Py_PRINT_RAW) < 0) { + /* Reserve space for the arguments, separators, and terminator. */ + PyObject *parts = PyList_New(Py_MAX(1, objects_length * 2)); + if (parts == NULL) { Py_DECREF(file); return NULL; } + /* Keep this private list out of GC inspection during callbacks. */ + PyObject_GC_UnTrack(parts); + /* Keep only accumulated parts visible when a conversion fails. */ + Py_SET_SIZE(parts, 0); + PyObject *exception, *saved_exception, *line; + for (Py_ssize_t i = 0; i < objects_length; i++) { + if (i > 0 && PyList_Append(parts, sep) < 0) { + goto write; + } + PyObject *arg = PyObject_Str(objects[i]); + if (arg == NULL) { + goto write; + } + int err = PyList_Append(parts, arg); + Py_DECREF(arg); + if (err < 0) { + goto write; + } + } + if (PyList_Append(parts, end) < 0) { + goto write; + } + +write: + /* Write the accumulated parts even if converting an argument failed. + Handle that exception while writing, as in a Python finally block. */ + exception = PyErr_GetRaisedException(); + saved_exception = PyErr_GetHandledException(); + if (exception != NULL) { + if (PyList_GET_SIZE(parts) == 0) { + /* An empty write may discard a text stream's buffered input. */ + PyErr_SetRaisedException(exception); + exception = NULL; + goto done; + } + PyErr_SetHandledException(exception); + } - if (flush) { - if (_PyFile_Flush(file) < 0) { - Py_DECREF(file); - return NULL; + int written = _PyTextIOWrapper_WriteList(file, parts); + if (written < 0) { + if (written == -2 && PyErr_ExceptionMatches(PyExc_MemoryError)) { + goto write_parts; } + goto done; + } + if (written) { + goto flush; + } + + line = PyUnicode_Join(&_Py_STR(empty), parts); + if (line != NULL) { + PyObject *result = PyObject_CallMethodOneArg(file, &_Py_ID(write), line); + Py_DECREF(line); + if (result == NULL) { + goto done; + } + Py_DECREF(result); + goto flush; + } + +write_parts: + /* If preparing the output fails (usually from lack of memory), try + writing the parts separately before propagating the exception. */ + Py_XSETREF(exception, PyErr_GetRaisedException()); + PyErr_SetHandledException(exception); + for (Py_ssize_t i = 0; i < PyList_GET_SIZE(parts); i++) { + PyObject *result = PyObject_CallMethodOneArg( + file, &_Py_ID(write), PyList_GET_ITEM(parts, i)); + if (result == NULL) { + goto done; + } + Py_DECREF(result); } - Py_DECREF(file); +flush: + if (flush && _PyFile_Flush(file) < 0) { + goto done; + } + PyErr_SetRaisedException(exception); + exception = NULL; + +done: + PyErr_SetHandledException(saved_exception); + Py_XDECREF(saved_exception); + Py_XDECREF(exception); + Py_DECREF(parts); + Py_DECREF(file); + if (PyErr_Occurred()) { + return NULL; + } Py_RETURN_NONE; } diff --git a/Python/bytecodes.c b/Python/bytecodes.c index 42ad83daa1ea461..c6f1f61f3e08fe1 100644 --- a/Python/bytecodes.c +++ b/Python/bytecodes.c @@ -1641,19 +1641,23 @@ dummy_func( iter = PyStackRef_FromPyObjectSteal(iter_o); } - inst(GET_ANEXT, (aiter -- aiter, awaitable)) { + inst(GET_ANEXT, (aiter -- aiter, awaitable, null, none)) { PyObject *awaitable_o = _PyEval_GetANext(PyStackRef_AsPyObjectBorrow(aiter)); if (awaitable_o == NULL) { ERROR_NO_POP(); } awaitable = PyStackRef_FromPyObjectSteal(awaitable_o); + null = PyStackRef_NULL; + none = PyStackRef_None; } - inst(GET_AWAITABLE, (iterable -- iter)) { + inst(GET_AWAITABLE, (iterable -- iter, null, none)) { PyObject *iter_o = _PyEval_GetAwaitable(PyStackRef_AsPyObjectBorrow(iterable), oparg); PyStackRef_CLOSE(iterable); ERROR_IF(iter_o == NULL); iter = PyStackRef_FromPyObjectSteal(iter_o); + null = PyStackRef_NULL; + none = PyStackRef_None; } family(SEND, INLINE_CACHE_ENTRIES_SEND) = { @@ -2278,7 +2282,7 @@ dummy_func( assert(index < DK_SIZE(keys)); PyObject *res_o = FT_ATOMIC_LOAD_PTR_CONSUME(entries[index].me_value); DEOPT_IF(res_o == NULL); - #if Py_GIL_DISABLED + #ifdef Py_GIL_DISABLED int increfed = _Py_TryIncrefCompareStackRef(&entries[index].me_value, res_o, &res); DEOPT_IF(!increfed); #else @@ -2297,7 +2301,7 @@ dummy_func( PyDictUnicodeEntry *entries = DK_UNICODE_ENTRIES(keys); PyObject *res_o = FT_ATOMIC_LOAD_PTR_CONSUME(entries[index].me_value); DEOPT_IF(res_o == NULL); - #if Py_GIL_DISABLED + #ifdef Py_GIL_DISABLED int increfed = _Py_TryIncrefCompareStackRef(&entries[index].me_value, res_o, &res); DEOPT_IF(!increfed); #else diff --git a/Python/codegen.c b/Python/codegen.c index a975705576a167c..ae60caf195f23ca 100644 --- a/Python/codegen.c +++ b/Python/codegen.c @@ -618,8 +618,6 @@ codegen_unwind_fblock(compiler *c, location *ploc, RETURN_IF_ERROR(codegen_call_exit_with_nones(c, *ploc)); if (info->fb_type == COMPILE_FBLOCK_ASYNC_WITH) { ADDOP_I(c, *ploc, GET_AWAITABLE, 2); - ADDOP(c, *ploc, PUSH_NULL); - ADDOP_LOAD_CONST(c, *ploc, Py_None); ADD_YIELD_FROM(c, *ploc, 1); } ADDOP(c, *ploc, POP_TOP); @@ -2225,8 +2223,6 @@ codegen_async_for(compiler *c, stmt_ty s) /* SETUP_FINALLY to guard the __anext__ call */ ADDOP_JUMP(c, loc, SETUP_FINALLY, except); ADDOP(c, loc, GET_ANEXT); - ADDOP(c, loc, PUSH_NULL); - ADDOP_LOAD_CONST(c, loc, Py_None); USE_LABEL(c, send); ADD_YIELD_FROM(c, loc, 1); ADDOP(c, loc, POP_BLOCK); /* for SETUP_FINALLY */ @@ -4843,8 +4839,6 @@ codegen_async_comprehension_generator(compiler *c, location loc, ADDOP_JUMP(c, loc, SETUP_FINALLY, except); ADDOP(c, loc, GET_ANEXT); - ADDOP(c, loc, PUSH_NULL); - ADDOP_LOAD_CONST(c, loc, Py_None); USE_LABEL(c, send); ADD_YIELD_FROM(c, loc, 1); ADDOP(c, loc, POP_BLOCK); @@ -5167,8 +5161,6 @@ codegen_comprehension(compiler *c, expr_ty e, int type, if (is_async_comprehension && type != COMP_GENEXP) { ADDOP_I(c, loc, GET_AWAITABLE, 0); - ADDOP(c, loc, PUSH_NULL); - ADDOP_LOAD_CONST(c, loc, Py_None); ADD_YIELD_FROM(c, loc, 1); } @@ -5312,8 +5304,6 @@ codegen_async_with_inner(compiler *c, stmt_ty s, int pos) ADDOP_I(c, loc, LOAD_SPECIAL, SPECIAL___AENTER__); ADDOP_I(c, loc, CALL, 0); ADDOP_I(c, loc, GET_AWAITABLE, 1); - ADDOP(c, loc, PUSH_NULL); - ADDOP_LOAD_CONST(c, loc, Py_None); ADD_YIELD_FROM(c, loc, 1); ADDOP_JUMP(c, loc, SETUP_WITH, final); @@ -5349,8 +5339,6 @@ codegen_async_with_inner(compiler *c, stmt_ty s, int pos) */ RETURN_IF_ERROR(codegen_call_exit_with_nones(c, loc)); ADDOP_I(c, loc, GET_AWAITABLE, 2); - ADDOP(c, loc, PUSH_NULL); - ADDOP_LOAD_CONST(c, loc, Py_None); ADD_YIELD_FROM(c, loc, 1); ADDOP(c, loc, POP_TOP); @@ -5364,8 +5352,6 @@ codegen_async_with_inner(compiler *c, stmt_ty s, int pos) ADDOP(c, loc, PUSH_EXC_INFO); ADDOP(c, loc, WITH_EXCEPT_START); ADDOP_I(c, loc, GET_AWAITABLE, 2); - ADDOP(c, loc, PUSH_NULL); - ADDOP_LOAD_CONST(c, loc, Py_None); ADD_YIELD_FROM(c, loc, 1); RETURN_IF_ERROR(codegen_with_except_finish(c, cleanup)); @@ -5552,8 +5538,6 @@ codegen_visit_expr_impl(compiler *c, expr_ty e, bool result_is_unused) case Await_kind: VISIT(c, expr, e->v.Await.value); ADDOP_I(c, loc, GET_AWAITABLE, 0); - ADDOP(c, loc, PUSH_NULL); - ADDOP_LOAD_CONST(c, loc, Py_None); ADD_YIELD_FROM(c, loc, 1); break; case Compare_kind: diff --git a/Python/crossinterp.c b/Python/crossinterp.c index d189d7bcefe313e..a91b45ba2cbcf45 100644 --- a/Python/crossinterp.c +++ b/Python/crossinterp.c @@ -2018,6 +2018,14 @@ _PyXI_ApplyError(_PyXI_error *error, const char *failure) PyThreadState *tstate = PyThreadState_Get(); if (failure != NULL) { + if (error->override != NULL + && error->override->code == _PyXI_ERR_NO_MEMORY) + { + PyErr_NoMemory(); + } + else { + PyErr_SetString(PyExc_InterpreterError, failure); + } xi_error_clear(error); return NULL; } diff --git a/Python/executor_cases.c.h b/Python/executor_cases.c.h index bc85ccd9ddb4fc1..5df8120e7424b93 100644 --- a/Python/executor_cases.c.h +++ b/Python/executor_cases.c.h @@ -8877,11 +8877,13 @@ break; } - case _GET_ANEXT_r12: { + case _GET_ANEXT_r13: { CHECK_CURRENT_CACHED_VALUES(1); ASSERT_WITHIN_STACK_BOUNDS_IGNORING_CACHE(__FILE__, __LINE__); _PyStackRef aiter; _PyStackRef awaitable; + _PyStackRef null; + _PyStackRef none; _PyStackRef _stack_item_0 = _tos_cache0; aiter = _stack_item_0; stack_pointer[0] = aiter; @@ -8896,21 +8898,23 @@ JUMP_TO_ERROR(); } awaitable = PyStackRef_FromPyObjectSteal(awaitable_o); - _tos_cache1 = awaitable; - _tos_cache0 = aiter; - _tos_cache2 = PyStackRef_ZERO_BITS; - SET_CURRENT_CACHED_VALUES(2); - stack_pointer += -1; - ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__); + null = PyStackRef_NULL; + none = PyStackRef_None; + _tos_cache2 = none; + _tos_cache1 = null; + _tos_cache0 = awaitable; + SET_CURRENT_CACHED_VALUES(3); ASSERT_WITHIN_STACK_BOUNDS_IGNORING_CACHE(__FILE__, __LINE__); break; } - case _GET_AWAITABLE_r11: { + case _GET_AWAITABLE_r13: { CHECK_CURRENT_CACHED_VALUES(1); ASSERT_WITHIN_STACK_BOUNDS_IGNORING_CACHE(__FILE__, __LINE__); _PyStackRef iterable; _PyStackRef iter; + _PyStackRef null; + _PyStackRef none; _PyStackRef _stack_item_0 = _tos_cache0; oparg = CURRENT_OPARG(); iterable = _stack_item_0; @@ -8932,10 +8936,12 @@ JUMP_TO_ERROR(); } iter = PyStackRef_FromPyObjectSteal(iter_o); + null = PyStackRef_NULL; + none = PyStackRef_None; + _tos_cache2 = none; + _tos_cache1 = null; _tos_cache0 = iter; - _tos_cache1 = PyStackRef_ZERO_BITS; - _tos_cache2 = PyStackRef_ZERO_BITS; - SET_CURRENT_CACHED_VALUES(1); + SET_CURRENT_CACHED_VALUES(3); ASSERT_WITHIN_STACK_BOUNDS_IGNORING_CACHE(__FILE__, __LINE__); break; } @@ -10440,7 +10446,7 @@ SET_CURRENT_CACHED_VALUES(0); JUMP_TO_JUMP_TARGET(); } - #if Py_GIL_DISABLED + #ifdef Py_GIL_DISABLED int increfed = _Py_TryIncrefCompareStackRef(&entries[index].me_value, res_o, &res); if (!increfed) { UOP_STAT_INC(uopcode, miss); @@ -10485,7 +10491,7 @@ SET_CURRENT_CACHED_VALUES(0); JUMP_TO_JUMP_TARGET(); } - #if Py_GIL_DISABLED + #ifdef Py_GIL_DISABLED int increfed = _Py_TryIncrefCompareStackRef(&entries[index].me_value, res_o, &res); if (!increfed) { UOP_STAT_INC(uopcode, miss); diff --git a/Python/generated_cases.c.h b/Python/generated_cases.c.h index ee9b64107bb9f24..3af7863e80d2c6a 100644 --- a/Python/generated_cases.c.h +++ b/Python/generated_cases.c.h @@ -6799,6 +6799,8 @@ INSTRUCTION_STATS(GET_ANEXT); _PyStackRef aiter; _PyStackRef awaitable; + _PyStackRef null; + _PyStackRef none; aiter = stack_pointer[-1]; _PyFrame_SetStackPointer(frame, stack_pointer); _PyFrame_StackPointerValidate(frame); @@ -6808,8 +6810,12 @@ JUMP_TO_LABEL(error); } awaitable = PyStackRef_FromPyObjectSteal(awaitable_o); + null = PyStackRef_NULL; + none = PyStackRef_None; stack_pointer[0] = awaitable; - stack_pointer += 1; + stack_pointer[1] = null; + stack_pointer[2] = none; + stack_pointer += 3; ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__); DISPATCH(); } @@ -6824,6 +6830,8 @@ INSTRUCTION_STATS(GET_AWAITABLE); _PyStackRef iterable; _PyStackRef iter; + _PyStackRef null; + _PyStackRef none; iterable = stack_pointer[-1]; _PyFrame_SetStackPointer(frame, stack_pointer); _PyFrame_StackPointerValidate(frame); @@ -6839,8 +6847,12 @@ JUMP_TO_LABEL(error); } iter = PyStackRef_FromPyObjectSteal(iter_o); + null = PyStackRef_NULL; + none = PyStackRef_None; stack_pointer[0] = iter; - stack_pointer += 1; + stack_pointer[1] = null; + stack_pointer[2] = none; + stack_pointer += 3; ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__); DISPATCH(); } @@ -10118,7 +10130,7 @@ assert(_PyOpcode_Deopt[opcode] == (LOAD_GLOBAL)); JUMP_TO_PREDICTED(LOAD_GLOBAL); } - #if Py_GIL_DISABLED + #ifdef Py_GIL_DISABLED int increfed = _Py_TryIncrefCompareStackRef(&entries[index].me_value, res_o, &res); if (!increfed) { UPDATE_MISS_STATS(LOAD_GLOBAL); @@ -10185,7 +10197,7 @@ assert(_PyOpcode_Deopt[opcode] == (LOAD_GLOBAL)); JUMP_TO_PREDICTED(LOAD_GLOBAL); } - #if Py_GIL_DISABLED + #ifdef Py_GIL_DISABLED int increfed = _Py_TryIncrefCompareStackRef(&entries[index].me_value, res_o, &res); if (!increfed) { UPDATE_MISS_STATS(LOAD_GLOBAL); diff --git a/Python/lock.c b/Python/lock.c index a2d47eea64a006b..59dfe772a0365f1 100644 --- a/Python/lock.c +++ b/Python/lock.c @@ -25,7 +25,7 @@ static const PyTime_t TIME_TO_BE_FAIR_NS = 1000*1000; // Spin for a bit before parking the thread. This is only enabled for // `--disable-gil` builds because it is unlikely to be helpful if the GIL is // enabled. -#if Py_GIL_DISABLED +#ifdef Py_GIL_DISABLED static const int MAX_SPIN_COUNT = 40; #else static const int MAX_SPIN_COUNT = 0; diff --git a/Python/modsupport.c b/Python/modsupport.c index bab21d1b2be5b51..f944b1c05ba9889 100644 --- a/Python/modsupport.c +++ b/Python/modsupport.c @@ -767,7 +767,7 @@ int PyABIInfo_Check(PyABIInfo *info, const char *module_name) /* Free-threading/GIL */ uint16_t gilflags = info->flags & (PyABIInfo_GIL | PyABIInfo_FREETHREADED); -#if Py_GIL_DISABLED +#ifdef Py_GIL_DISABLED if (gilflags == PyABIInfo_GIL) { return _abiinfo_raise(module_name, "incompatible with free-threaded CPython"); diff --git a/Python/optimizer_bytecodes.c b/Python/optimizer_bytecodes.c index 6284885c428832b..6648b72e66e281d 100644 --- a/Python/optimizer_bytecodes.c +++ b/Python/optimizer_bytecodes.c @@ -2763,13 +2763,21 @@ dummy_func(void) { } } - op(_GET_ANEXT, (aiter -- aiter, awaitable)) { + op(_GET_ANEXT, (aiter -- aiter, awaitable, null, none)) { if (sym_matches_type(aiter, &PyAsyncGen_Type)) { awaitable = sym_new_type(ctx, &_PyAsyncGenASend_Type); } else { awaitable = sym_new_not_null(ctx); } + null = PyJitRef_Borrow(sym_new_null(ctx)); + none = PyJitRef_Borrow(sym_new_const(ctx, Py_None)); + } + + op(_GET_AWAITABLE, (iterable -- iter, null, none)) { + iter = sym_new_not_null(ctx); + null = PyJitRef_Borrow(sym_new_null(ctx)); + none = PyJitRef_Borrow(sym_new_const(ctx, Py_None)); } // END BYTECODES // diff --git a/Python/optimizer_cases.c.h b/Python/optimizer_cases.c.h index 9cd20c495db4418..c86121fcfc0e79a 100644 --- a/Python/optimizer_cases.c.h +++ b/Python/optimizer_cases.c.h @@ -1872,6 +1872,8 @@ case _GET_ANEXT: { JitOptRef aiter; JitOptRef awaitable; + JitOptRef null; + JitOptRef none; aiter = stack_pointer[-1]; if (sym_matches_type(aiter, &PyAsyncGen_Type)) { awaitable = sym_new_type(ctx, &_PyAsyncGenASend_Type); @@ -1879,17 +1881,30 @@ else { awaitable = sym_new_not_null(ctx); } - CHECK_STACK_BOUNDS(1); + null = PyJitRef_Borrow(sym_new_null(ctx)); + none = PyJitRef_Borrow(sym_new_const(ctx, Py_None)); + CHECK_STACK_BOUNDS(3); stack_pointer[0] = awaitable; - stack_pointer += 1; + stack_pointer[1] = null; + stack_pointer[2] = none; + stack_pointer += 3; ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__); break; } case _GET_AWAITABLE: { JitOptRef iter; + JitOptRef null; + JitOptRef none; iter = sym_new_not_null(ctx); + null = PyJitRef_Borrow(sym_new_null(ctx)); + none = PyJitRef_Borrow(sym_new_const(ctx, Py_None)); + CHECK_STACK_BOUNDS(2); stack_pointer[-1] = iter; + stack_pointer[0] = null; + stack_pointer[1] = none; + stack_pointer += 2; + ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__); break; } diff --git a/Python/perf_jit_trampoline.c b/Python/perf_jit_trampoline.c index 07aa8e769bf2d7d..da6a7e69633db52 100644 --- a/Python/perf_jit_trampoline.c +++ b/Python/perf_jit_trampoline.c @@ -672,7 +672,7 @@ static void perf_map_jit_write_entry_with_name( uint64_t thread_id = 0; pthread_threadid_np(NULL, &thread_id); ev.thread_id = (uint32_t)thread_id; -#elif defined(HAVE_GETTID) +#elif defined(_Py_HAVE_GETTID) ev.thread_id = gettid(); #else ev.thread_id = syscall(SYS_gettid); // Get thread ID via system call diff --git a/Python/pystate.c b/Python/pystate.c index ce7197c4e054283..4c95669f9fa1b09 100644 --- a/Python/pystate.c +++ b/Python/pystate.c @@ -2294,7 +2294,7 @@ tstate_deactivate(PyThreadState *tstate) assert(tstate_is_bound(tstate)); assert(tstate->_status.active); -#if Py_STATS +#ifdef Py_STATS _PyStats_Detach((_PyThreadStateImpl *)tstate); #endif diff --git a/Python/remote_debug.h b/Python/remote_debug.h index a68e4c6102e5a7d..0c55fd3a0354e3a 100644 --- a/Python/remote_debug.h +++ b/Python/remote_debug.h @@ -155,8 +155,6 @@ get_page_size(void) { typedef struct page_cache_entry { uintptr_t page_addr; // page-aligned base address char *data; - int valid; - struct page_cache_entry *next; } page_cache_entry_t; #define MAX_PAGES 1024 @@ -211,7 +209,6 @@ _Py_RemoteDebug_FreePageCache(proc_handle_t *handle) PyMem_RawFree(handle->pages[i].data); } handle->pages[i].data = NULL; - handle->pages[i].valid = 0; } handle->page_cache_count = 0; } @@ -219,9 +216,6 @@ _Py_RemoteDebug_FreePageCache(proc_handle_t *handle) UNUSED static void _Py_RemoteDebug_ClearCache(proc_handle_t *handle) { - for (int i = 0; i < handle->page_cache_count; i++) { - handle->pages[i].valid = 0; - } handle->page_cache_count = 0; } @@ -244,7 +238,6 @@ _Py_RemoteDebug_InitProcHandle(proc_handle_t *handle, pid_t pid) { handle->page_cache_count = 0; for (int i = 0; i < MAX_PAGES; i++) { handle->pages[i].data = NULL; - handle->pages[i].valid = 0; } handle->pid = pid; @@ -1539,7 +1532,7 @@ _Py_RemoteDebug_PagedReadRemoteMemory(proc_handle_t *handle, // between profiler samples, so entries are packed at the front. for (int i = 0; i < handle->page_cache_count; i++) { page_cache_entry_t *entry = &handle->pages[i]; - if (entry->valid && entry->page_addr == page_base) { + if (entry->page_addr == page_base) { memcpy(out, entry->data + offset_in_page, size); return 0; } @@ -1566,7 +1559,6 @@ _Py_RemoteDebug_PagedReadRemoteMemory(proc_handle_t *handle, } entry->page_addr = page_base; - entry->valid = 1; handle->page_cache_count++; memcpy(out, entry->data + offset_in_page, size); return 0; diff --git a/Python/specialize.c b/Python/specialize.c index 067bae59f6073b7..b5cf4ebc31142ab 100644 --- a/Python/specialize.c +++ b/Python/specialize.c @@ -30,7 +30,7 @@ * InternalDocs/interpreter.md `Specialization` section. */ -#if Py_STATS +#ifdef Py_STATS #define SPECIALIZATION_FAIL(opcode, kind) \ do { \ PyStats *s = _PyStats_GET(); \ @@ -42,7 +42,7 @@ do { \ } while (0) #else # define SPECIALIZATION_FAIL(opcode, kind) ((void)0) -#endif // Py_STATS +#endif // !Py_STATS static void fixup_getiter(_Py_CODEUNIT *instruction, int flags) diff --git a/Python/thread_pthread.h b/Python/thread_pthread.h index e660bca84068604..c7f245acddb290c 100644 --- a/Python/thread_pthread.h +++ b/Python/thread_pthread.h @@ -18,7 +18,7 @@ #include #include /* pause(), also getthrid() on OpenBSD */ -#ifdef HAVE_GETTID +#ifdef _Py_HAVE_GETTID # include // gettid() #elif defined(__linux__) # include /* syscall(SYS_gettid) */ @@ -384,7 +384,7 @@ PyThread_get_thread_native_id(void) #ifdef __APPLE__ uint64_t native_id; (void) pthread_threadid_np(NULL, &native_id); -#elif defined(HAVE_GETTID) +#elif defined(_Py_HAVE_GETTID) pid_t native_id; native_id = gettid(); #elif defined(__linux__) diff --git a/Tools/build/check_capi_macros.py b/Tools/build/check_capi_macros.py old mode 100644 new mode 100755 index 62ae1fd0bb1c941..86c433a11683487 --- a/Tools/build/check_capi_macros.py +++ b/Tools/build/check_capi_macros.py @@ -1,3 +1,4 @@ +#! /usr/bin/env python3 """ Check that all macros defined by the Python C API have a name starting with "Py". Ignore names listed by check_capi_macros_ignored.txt: macros with an diff --git a/Tools/build/check_capi_macros_ignored.txt b/Tools/build/check_capi_macros_ignored.txt index 1800a2c4ec8946c..154f1dbfca039c9 100644 --- a/Tools/build/check_capi_macros_ignored.txt +++ b/Tools/build/check_capi_macros_ignored.txt @@ -116,45 +116,23 @@ HAVE_CONNECT HAVE_COPY_FILE_RANGE HAVE_CTERMID HAVE_CTERMID_R -HAVE_CURSES_DEFINE_KEY HAVE_CURSES_ESCDELAY HAVE_CURSES_FILTER HAVE_CURSES_GETMOUSE HAVE_CURSES_H HAVE_CURSES_HAS_KEY -HAVE_CURSES_HAS_MOUSE HAVE_CURSES_IMMEDOK -HAVE_CURSES_IS_KEYPAD -HAVE_CURSES_IS_LEAVEOK HAVE_CURSES_IS_PAD HAVE_CURSES_IS_TERM_RESIZED -HAVE_CURSES_KEYOK -HAVE_CURSES_KEY_DEFINED -HAVE_CURSES_NEW_PRESCR -HAVE_CURSES_NOFILTER HAVE_CURSES_RESIZETERM HAVE_CURSES_RESIZE_TERM -HAVE_CURSES_SCR_DUMP -HAVE_CURSES_SCR_SET HAVE_CURSES_SET_ESCDELAY HAVE_CURSES_SET_TABSIZE -HAVE_CURSES_SLK_ATTR_OFF -HAVE_CURSES_SLK_ATTR_ON -HAVE_CURSES_SLK_ATTR_SET -HAVE_CURSES_SLK_COLOR HAVE_CURSES_SYNCOK HAVE_CURSES_TABSIZE -HAVE_CURSES_TERM_ATTRS HAVE_CURSES_TYPEAHEAD HAVE_CURSES_USE_ENV -HAVE_CURSES_USE_SCREEN -HAVE_CURSES_USE_WINDOW -HAVE_CURSES_WATTR_GET -HAVE_CURSES_WATTR_OFF -HAVE_CURSES_WATTR_ON -HAVE_CURSES_WATTR_SET HAVE_CURSES_WCHGAT -HAVE_CURSES_WCOLOR_SET HAVE_DB_H HAVE_DECLSPEC_DLL HAVE_DECL_PR_SET_VMA_ANON_NAME @@ -235,7 +213,6 @@ HAVE_GDBM_H HAVE_GDBM_NDBM_H HAVE_GETADDRINFO HAVE_GETC_UNLOCKED -HAVE_GETDENTS64 HAVE_GETEGID HAVE_GETENTROPY HAVE_GETEUID @@ -279,16 +256,13 @@ HAVE_GETSERVBYNAME HAVE_GETSERVBYPORT HAVE_GETSID HAVE_GETSOCKNAME -HAVE_GETTID HAVE_GETUID HAVE_GRANTPT HAVE_GRP_H HAVE_HSTRERROR HAVE_HTOLE64 HAVE_ICONV_H -HAVE_IF_INDEXTONAME HAVE_IF_NAMEINDEX -HAVE_IF_NAMETOINDEX HAVE_INET_ATON HAVE_INET_NTOA HAVE_INET_PTON @@ -389,9 +363,6 @@ HAVE_PANELW HAVE_PANEL_H HAVE_PATHCONF HAVE_PAUSE -HAVE_PIDFD_GETFD -HAVE_PIDFD_OPEN -HAVE_PIDFD_SEND_SIGNAL HAVE_PIPE HAVE_PIPE2 HAVE_PLOCK @@ -442,7 +413,6 @@ HAVE_RECVFROM HAVE_RENAMEAT HAVE_RL_APPEND_HISTORY HAVE_RL_CATCH_SIGNAL -HAVE_RL_CHANGE_ENVIRONMENT HAVE_RL_COMPDISP_FUNC_T HAVE_RL_COMPLETION_APPEND_CHARACTER HAVE_RL_COMPLETION_DISPLAY_MATCHES_HOOK diff --git a/Tools/build/generate-build-details.py b/Tools/build/generate-build-details.py index 4fb49bf139b225f..a6f537fc0d020da 100644 --- a/Tools/build/generate-build-details.py +++ b/Tools/build/generate-build-details.py @@ -10,6 +10,7 @@ import importlib.machinery import json import os +import re import sys import sysconfig @@ -18,17 +19,98 @@ from typing import Any +PATCHLEVEL_H = os.path.join( + os.path.dirname(__file__), '..', '..', 'Include', 'patchlevel.h', +) +RELEASE_LEVELS = {0xA: 'alpha', 0xB: 'beta', 0xC: 'candidate', 0xF: 'final'} + + def version_info_to_dict(obj: sys._version_info) -> dict[str, Any]: field_names = ('major', 'minor', 'micro', 'releaselevel', 'serial') return {field: getattr(obj, field) for field in field_names} +def get_version_info() -> dict[str, Any]: + """Return the target's version as a version_info dict. + + sys.version_info describes the running interpreter, which under + --with-build-python can differ from the target in micro version and + release level. Read Include/patchlevel.h instead when it is available. + """ + try: + with open(PATCHLEVEL_H, encoding='utf-8') as f: + patchlevel = f.read() + except FileNotFoundError: + return version_info_to_dict(sys.version_info) + defines = dict(re.findall(r'^#define\s+(PY_\w+)\s+(\S+)', patchlevel, re.M)) + # PY_RELEASE_LEVEL is defined as another macro, e.g. PY_RELEASE_LEVEL_ALPHA. + level = int(defines[defines['PY_RELEASE_LEVEL']], 16) + return { + 'major': int(defines['PY_MAJOR_VERSION']), + 'minor': int(defines['PY_MINOR_VERSION']), + 'micro': int(defines['PY_MICRO_VERSION']), + 'releaselevel': RELEASE_LEVELS[level], + 'serial': int(defines['PY_RELEASE_SERIAL']), + } + + +def hexversion(version_info: dict[str, Any]) -> int: + level = {name: value for value, name in RELEASE_LEVELS.items()} + return int( + version_info['major'] << 24 + | version_info['minor'] << 16 + | version_info['micro'] << 8 + | level[version_info['releaselevel']] << 4 + | version_info['serial'] + ) + + def get_dict_key(container: dict[str, Any], key: str) -> dict[str, Any]: for part in key.split('.'): container = container[part] return container +def get_extension_suffixes() -> list[str]: + """Return the extension module suffixes of the target interpreter. + + importlib.machinery.EXTENSION_SUFFIXES describes the running interpreter, + which is not the target under --with-build-python. On POSIX, rebuild the + list from the target configuration instead, following + _PyImport_DynLoadFiletab in Python/dynload_*.c. The two are kept in sync + by test_build_details.BuildDetailsTargetTests.test_native_extension_suffixes. + """ + if os.name != 'posix': + return importlib.machinery.EXTENSION_SUFFIXES + + dynloadfile = sysconfig.get_config_var('DYNLOADFILE') + if dynloadfile == 'dynload_stub.o': + return [] + if dynloadfile == 'dynload_hpux.o': + return [sysconfig.get_config_var('EXT_SUFFIX'), '.sl'] + if dynloadfile != 'dynload_shlib.o': + raise ValueError(f'Unsupported DYNLOADFILE: {dynloadfile!r}') + + soabi = sysconfig.get_config_var('SOABI') + if sysconfig.get_config_var('MACHDEP') == 'cygwin': + return [f'.{soabi}.dll', '.dll'] + + suffixes = [f'.{soabi}.so'] + alt_soabi = sysconfig.get_config_var('ALT_SOABI') + if alt_soabi: + suffixes.append(f'.{alt_soabi}.so') + platform = sysconfig.get_config_var('SOABI_PLATFORM') + if not sysconfig.get_config_var('Py_GIL_DISABLED'): + if platform: + suffixes.append(f'.abi3-{platform}.so') + suffixes.append('.abi3.so') + if platform: + suffixes.append(f'.abi3t-{platform}.so') + suffixes.append('.abi3t.so') + suffixes.append('.so') + return suffixes + + def generate_data(schema_version: str) -> collections.defaultdict[str, Any]: """Generate the build-details.json data (PEP 739). @@ -54,22 +136,30 @@ def generate_data(schema_version: str) -> collections.defaultdict[str, Any]: ) data['platform'] = sysconfig.get_platform() + version_info = get_version_info() data['language']['version'] = sysconfig.get_python_version() - data['language']['version_info'] = version_info_to_dict(sys.version_info) + data['language']['version_info'] = version_info data['implementation'] = vars(sys.implementation).copy() - data['implementation']['version'] = version_info_to_dict(sys.implementation.version) + data['implementation']['version'] = version_info + data['implementation']['hexversion'] = hexversion(version_info) # Fix cross-compilation if '_multiarch' in data['implementation']: data['implementation']['_multiarch'] = sysconfig.get_config_var('MULTIARCH') + if 'supports_isolated_interpreters' in data['implementation']: + # Mirrors Python/sysmodule.c. + data['implementation']['supports_isolated_interpreters'] = ( + sysconfig.get_config_var('MACHDEP') not in ('wasi', 'emscripten') + ) - data['abi']['flags'] = list(sys.abiflags) + data['abi']['flags'] = list(sysconfig.get_config_var('ABIFLAGS')) data['suffixes']['source'] = importlib.machinery.SOURCE_SUFFIXES data['suffixes']['bytecode'] = importlib.machinery.BYTECODE_SUFFIXES #data['suffixes']['optimized_bytecode'] = importlib.machinery.OPTIMIZED_BYTECODE_SUFFIXES #data['suffixes']['debug_bytecode'] = importlib.machinery.DEBUG_BYTECODE_SUFFIXES - data['suffixes']['extensions'] = importlib.machinery.EXTENSION_SUFFIXES + extension_suffixes = get_extension_suffixes() + data['suffixes']['extensions'] = extension_suffixes LIBDIR = sysconfig.get_config_var('LIBDIR') LIBPL = sysconfig.get_config_var('LIBPL') @@ -118,17 +208,16 @@ def generate_data(schema_version: str) -> collections.defaultdict[str, Any]: # regardless. # # We only want to set abi.extension_suffix and stable_abi_suffix if - # extension modules are supported. - if has_dynamic_library: + # extension modules are supported, regardless of how libpython is built. + if extension_suffixes: data['abi']['extension_suffix'] = sysconfig.get_config_var('EXT_SUFFIX') - # EXTENSION_SUFFIXES has been constant for a long time, and currently we - # don't have a better information source to find the stable ABI suffix. - for suffix in importlib.machinery.EXTENSION_SUFFIXES: + for suffix in extension_suffixes: if suffix.startswith('.abi'): data['abi']['stable_abi_suffix'] = suffix break + if has_dynamic_library: data['libpython']['dynamic'] = os.path.join(dynamic_library_dir, LDLIBRARY) # FIXME: Not sure if windows has a different dll for the stable ABI, and # even if it does, currently we don't have a way to get its name. diff --git a/Tools/c-analyzer/c_parser/parser/_regexes.py b/Tools/c-analyzer/c_parser/parser/_regexes.py index c1a8ab3ad2f15da..65fe2f929a942e3 100644 --- a/Tools/c-analyzer/c_parser/parser/_regexes.py +++ b/Tools/c-analyzer/c_parser/parser/_regexes.py @@ -340,7 +340,10 @@ def _ind(text, level=1, edges='both'): # This is actually a "constant expression". \d+ | - [^'",}}]+ + [^'",(}}]+ + | + # Parens may hold commas (e.g. from offsetof()). + [^'";}}]+ (?= ; ) ) )? \s* diff --git a/configure b/configure index e033b0217aefad5..0e93826d3494c6d 100755 --- a/configure +++ b/configure @@ -21009,12 +21009,6 @@ if test "x$ac_cv_func_gai_strerror" = xyes then : printf "%s\n" "#define HAVE_GAI_STRERROR 1" >>confdefs.h -fi -ac_fn_c_check_func "$LINENO" "getdents64" "ac_cv_func_getdents64" -if test "x$ac_cv_func_getdents64" = xyes -then : - printf "%s\n" "#define HAVE_GETDENTS64 1" >>confdefs.h - fi ac_fn_c_check_func "$LINENO" "getegid" "ac_cv_func_getegid" if test "x$ac_cv_func_getegid" = xyes @@ -21177,12 +21171,6 @@ if test "x$ac_cv_func_getsid" = xyes then : printf "%s\n" "#define HAVE_GETSID 1" >>confdefs.h -fi -ac_fn_c_check_func "$LINENO" "gettid" "ac_cv_func_gettid" -if test "x$ac_cv_func_gettid" = xyes -then : - printf "%s\n" "#define HAVE_GETTID 1" >>confdefs.h - fi ac_fn_c_check_func "$LINENO" "getuid" "ac_cv_func_getuid" if test "x$ac_cv_func_getuid" = xyes @@ -21195,24 +21183,12 @@ if test "x$ac_cv_func_grantpt" = xyes then : printf "%s\n" "#define HAVE_GRANTPT 1" >>confdefs.h -fi -ac_fn_c_check_func "$LINENO" "if_indextoname" "ac_cv_func_if_indextoname" -if test "x$ac_cv_func_if_indextoname" = xyes -then : - printf "%s\n" "#define HAVE_IF_INDEXTONAME 1" >>confdefs.h - fi ac_fn_c_check_func "$LINENO" "if_nameindex" "ac_cv_func_if_nameindex" if test "x$ac_cv_func_if_nameindex" = xyes then : printf "%s\n" "#define HAVE_IF_NAMEINDEX 1" >>confdefs.h -fi -ac_fn_c_check_func "$LINENO" "if_nametoindex" "ac_cv_func_if_nametoindex" -if test "x$ac_cv_func_if_nametoindex" = xyes -then : - printf "%s\n" "#define HAVE_IF_NAMETOINDEX 1" >>confdefs.h - fi ac_fn_c_check_func "$LINENO" "initgroups" "ac_cv_func_initgroups" if test "x$ac_cv_func_initgroups" = xyes @@ -21357,24 +21333,6 @@ if test "x$ac_cv_func_pause" = xyes then : printf "%s\n" "#define HAVE_PAUSE 1" >>confdefs.h -fi -ac_fn_c_check_func "$LINENO" "pidfd_open" "ac_cv_func_pidfd_open" -if test "x$ac_cv_func_pidfd_open" = xyes -then : - printf "%s\n" "#define HAVE_PIDFD_OPEN 1" >>confdefs.h - -fi -ac_fn_c_check_func "$LINENO" "pidfd_getfd" "ac_cv_func_pidfd_getfd" -if test "x$ac_cv_func_pidfd_getfd" = xyes -then : - printf "%s\n" "#define HAVE_PIDFD_GETFD 1" >>confdefs.h - -fi -ac_fn_c_check_func "$LINENO" "pidfd_send_signal" "ac_cv_func_pidfd_send_signal" -if test "x$ac_cv_func_pidfd_send_signal" = xyes -then : - printf "%s\n" "#define HAVE_PIDFD_SEND_SIGNAL 1" >>confdefs.h - fi ac_fn_c_check_func "$LINENO" "pipe" "ac_cv_func_pipe" if test "x$ac_cv_func_pipe" = xyes @@ -22101,6 +22059,299 @@ fi + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for getdents64" >&5 +printf %s "checking for getdents64... " >&6; } +if test ${ac_cv_func_getdents64+y} +then : + printf %s "(cached) " >&6 +else case e in #( + e) cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include +int +main (void) +{ +(void)getdents64 + ; + return 0; +} +_ACEOF +if ac_fn_c_try_compile "$LINENO" +then : + ac_cv_func_getdents64=yes +else case e in #( + e) ac_cv_func_getdents64=no ;; +esac +fi +rm -f core conftest.err conftest.$ac_objext conftest.beam conftest.$ac_ext + ;; +esac +fi +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_func_getdents64" >&5 +printf "%s\n" "$ac_cv_func_getdents64" >&6; } + if test "x$ac_cv_func_getdents64" = xyes +then : + +printf "%s\n" "#define _Py_HAVE_GETDENTS64 1" >>confdefs.h + +fi + + + + + + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for gettid" >&5 +printf %s "checking for gettid... " >&6; } +if test ${ac_cv_func_gettid+y} +then : + printf %s "(cached) " >&6 +else case e in #( + e) cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include +int +main (void) +{ +(void)gettid + ; + return 0; +} +_ACEOF +if ac_fn_c_try_compile "$LINENO" +then : + ac_cv_func_gettid=yes +else case e in #( + e) ac_cv_func_gettid=no ;; +esac +fi +rm -f core conftest.err conftest.$ac_objext conftest.beam conftest.$ac_ext + ;; +esac +fi +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_func_gettid" >&5 +printf "%s\n" "$ac_cv_func_gettid" >&6; } + if test "x$ac_cv_func_gettid" = xyes +then : + +printf "%s\n" "#define _Py_HAVE_GETTID 1" >>confdefs.h + +fi + + + + + + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for if_indextoname" >&5 +printf %s "checking for if_indextoname... " >&6; } +if test ${ac_cv_func_if_indextoname+y} +then : + printf %s "(cached) " >&6 +else case e in #( + e) cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include +int +main (void) +{ +(void)if_indextoname + ; + return 0; +} +_ACEOF +if ac_fn_c_try_compile "$LINENO" +then : + ac_cv_func_if_indextoname=yes +else case e in #( + e) ac_cv_func_if_indextoname=no ;; +esac +fi +rm -f core conftest.err conftest.$ac_objext conftest.beam conftest.$ac_ext + ;; +esac +fi +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_func_if_indextoname" >&5 +printf "%s\n" "$ac_cv_func_if_indextoname" >&6; } + if test "x$ac_cv_func_if_indextoname" = xyes +then : + +printf "%s\n" "#define _Py_HAVE_IF_INDEXTONAME 1" >>confdefs.h + +fi + + + + + + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for if_nametoindex" >&5 +printf %s "checking for if_nametoindex... " >&6; } +if test ${ac_cv_func_if_nametoindex+y} +then : + printf %s "(cached) " >&6 +else case e in #( + e) cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include +int +main (void) +{ +(void)if_nametoindex + ; + return 0; +} +_ACEOF +if ac_fn_c_try_compile "$LINENO" +then : + ac_cv_func_if_nametoindex=yes +else case e in #( + e) ac_cv_func_if_nametoindex=no ;; +esac +fi +rm -f core conftest.err conftest.$ac_objext conftest.beam conftest.$ac_ext + ;; +esac +fi +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_func_if_nametoindex" >&5 +printf "%s\n" "$ac_cv_func_if_nametoindex" >&6; } + if test "x$ac_cv_func_if_nametoindex" = xyes +then : + +printf "%s\n" "#define _Py_HAVE_IF_NAMETOINDEX 1" >>confdefs.h + +fi + + + + + + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for pidfd_open" >&5 +printf %s "checking for pidfd_open... " >&6; } +if test ${ac_cv_func_pidfd_open+y} +then : + printf %s "(cached) " >&6 +else case e in #( + e) cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include +int +main (void) +{ +(void)pidfd_open + ; + return 0; +} +_ACEOF +if ac_fn_c_try_compile "$LINENO" +then : + ac_cv_func_pidfd_open=yes +else case e in #( + e) ac_cv_func_pidfd_open=no ;; +esac +fi +rm -f core conftest.err conftest.$ac_objext conftest.beam conftest.$ac_ext + ;; +esac +fi +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_func_pidfd_open" >&5 +printf "%s\n" "$ac_cv_func_pidfd_open" >&6; } + if test "x$ac_cv_func_pidfd_open" = xyes +then : + +printf "%s\n" "#define _Py_HAVE_PIDFD_OPEN 1" >>confdefs.h + +fi + + + + + + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for pidfd_getfd" >&5 +printf %s "checking for pidfd_getfd... " >&6; } +if test ${ac_cv_func_pidfd_getfd+y} +then : + printf %s "(cached) " >&6 +else case e in #( + e) cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include +int +main (void) +{ +(void)pidfd_getfd + ; + return 0; +} +_ACEOF +if ac_fn_c_try_compile "$LINENO" +then : + ac_cv_func_pidfd_getfd=yes +else case e in #( + e) ac_cv_func_pidfd_getfd=no ;; +esac +fi +rm -f core conftest.err conftest.$ac_objext conftest.beam conftest.$ac_ext + ;; +esac +fi +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_func_pidfd_getfd" >&5 +printf "%s\n" "$ac_cv_func_pidfd_getfd" >&6; } + if test "x$ac_cv_func_pidfd_getfd" = xyes +then : + +printf "%s\n" "#define _Py_HAVE_PIDFD_GETFD 1" >>confdefs.h + +fi + + + + + + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for pidfd_send_signal" >&5 +printf %s "checking for pidfd_send_signal... " >&6; } +if test ${ac_cv_func_pidfd_send_signal+y} +then : + printf %s "(cached) " >&6 +else case e in #( + e) cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include +int +main (void) +{ +(void)pidfd_send_signal + ; + return 0; +} +_ACEOF +if ac_fn_c_try_compile "$LINENO" +then : + ac_cv_func_pidfd_send_signal=yes +else case e in #( + e) ac_cv_func_pidfd_send_signal=no ;; +esac +fi +rm -f core conftest.err conftest.$ac_objext conftest.beam conftest.$ac_ext + ;; +esac +fi +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_func_pidfd_send_signal" >&5 +printf "%s\n" "$ac_cv_func_pidfd_send_signal" >&6; } + if test "x$ac_cv_func_pidfd_send_signal" = xyes +then : + +printf "%s\n" "#define _Py_HAVE_PIDFD_SEND_SIGNAL 1" >>confdefs.h + +fi + + + + + # os.statx uses Linux's statx function. AIX also has a function named statx, # but it's unrelated. Check only on Linux (including Android). case $ac_sys_system in #( @@ -29853,7 +30104,7 @@ printf "%s\n" "$ac_cv_readline_rl_change_environment" >&6; } then : -printf "%s\n" "#define HAVE_RL_CHANGE_ENVIRONMENT 1" >>confdefs.h +printf "%s\n" "#define _Py_HAVE_RL_CHANGE_ENVIRONMENT 1" >>confdefs.h fi @@ -32026,6 +32277,9 @@ fi + + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function is_pad" >&5 printf %s "checking for curses function is_pad... " >&6; } if test ${ac_cv_lib_curses_is_pad+y} @@ -32087,6 +32341,7 @@ fi + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function is_term_resized" >&5 printf %s "checking for curses function is_term_resized... " >&6; } if test ${ac_cv_lib_curses_is_term_resized+y} @@ -32148,6 +32403,7 @@ fi + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function resize_term" >&5 printf %s "checking for curses function resize_term... " >&6; } if test ${ac_cv_lib_curses_resize_term+y} @@ -32209,6 +32465,7 @@ fi + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function resizeterm" >&5 printf %s "checking for curses function resizeterm... " >&6; } if test ${ac_cv_lib_curses_resizeterm+y} @@ -32270,6 +32527,7 @@ fi + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function immedok" >&5 printf %s "checking for curses function immedok... " >&6; } if test ${ac_cv_lib_curses_immedok+y} @@ -32331,6 +32589,7 @@ fi + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function syncok" >&5 printf %s "checking for curses function syncok... " >&6; } if test ${ac_cv_lib_curses_syncok+y} @@ -32392,6 +32651,7 @@ fi + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function wchgat" >&5 printf %s "checking for curses function wchgat... " >&6; } if test ${ac_cv_lib_curses_wchgat+y} @@ -32453,6 +32713,7 @@ fi + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function filter" >&5 printf %s "checking for curses function filter... " >&6; } if test ${ac_cv_lib_curses_filter+y} @@ -32514,9 +32775,10 @@ fi - { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function nofilter" >&5 -printf %s "checking for curses function nofilter... " >&6; } -if test ${ac_cv_lib_curses_nofilter+y} + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function has_key" >&5 +printf %s "checking for curses function has_key... " >&6; } +if test ${ac_cv_lib_curses_has_key+y} then : printf %s "(cached) " >&6 else case e in #( @@ -32542,8 +32804,8 @@ int main (void) { - #ifndef nofilter - void * volatile x = (void *)nofilter; (void)x; + #ifndef has_key + void * volatile x = (void *)has_key; (void)x; #endif ; @@ -32552,9 +32814,9 @@ main (void) _ACEOF if ac_fn_c_try_link "$LINENO" then : - ac_cv_lib_curses_nofilter=yes + ac_cv_lib_curses_has_key=yes else case e in #( - e) ac_cv_lib_curses_nofilter=no ;; + e) ac_cv_lib_curses_has_key=no ;; esac fi rm -f core conftest.err conftest.$ac_objext conftest.beam \ @@ -32562,12 +32824,12 @@ rm -f core conftest.err conftest.$ac_objext conftest.beam \ ;; esac fi -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_nofilter" >&5 -printf "%s\n" "$ac_cv_lib_curses_nofilter" >&6; } - if test "x$ac_cv_lib_curses_nofilter" = xyes +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_has_key" >&5 +printf "%s\n" "$ac_cv_lib_curses_has_key" >&6; } + if test "x$ac_cv_lib_curses_has_key" = xyes then : -printf "%s\n" "#define HAVE_CURSES_NOFILTER 1" >>confdefs.h +printf "%s\n" "#define HAVE_CURSES_HAS_KEY 1" >>confdefs.h fi @@ -32575,9 +32837,10 @@ fi - { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function has_key" >&5 -printf %s "checking for curses function has_key... " >&6; } -if test ${ac_cv_lib_curses_has_key+y} + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function typeahead" >&5 +printf %s "checking for curses function typeahead... " >&6; } +if test ${ac_cv_lib_curses_typeahead+y} then : printf %s "(cached) " >&6 else case e in #( @@ -32603,8 +32866,8 @@ int main (void) { - #ifndef has_key - void * volatile x = (void *)has_key; (void)x; + #ifndef typeahead + void * volatile x = (void *)typeahead; (void)x; #endif ; @@ -32613,9 +32876,9 @@ main (void) _ACEOF if ac_fn_c_try_link "$LINENO" then : - ac_cv_lib_curses_has_key=yes + ac_cv_lib_curses_typeahead=yes else case e in #( - e) ac_cv_lib_curses_has_key=no ;; + e) ac_cv_lib_curses_typeahead=no ;; esac fi rm -f core conftest.err conftest.$ac_objext conftest.beam \ @@ -32623,12 +32886,12 @@ rm -f core conftest.err conftest.$ac_objext conftest.beam \ ;; esac fi -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_has_key" >&5 -printf "%s\n" "$ac_cv_lib_curses_has_key" >&6; } - if test "x$ac_cv_lib_curses_has_key" = xyes +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_typeahead" >&5 +printf "%s\n" "$ac_cv_lib_curses_typeahead" >&6; } + if test "x$ac_cv_lib_curses_typeahead" = xyes then : -printf "%s\n" "#define HAVE_CURSES_HAS_KEY 1" >>confdefs.h +printf "%s\n" "#define HAVE_CURSES_TYPEAHEAD 1" >>confdefs.h fi @@ -32636,9 +32899,10 @@ fi - { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function has_mouse" >&5 -printf %s "checking for curses function has_mouse... " >&6; } -if test ${ac_cv_lib_curses_has_mouse+y} + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function use_env" >&5 +printf %s "checking for curses function use_env... " >&6; } +if test ${ac_cv_lib_curses_use_env+y} then : printf %s "(cached) " >&6 else case e in #( @@ -32664,8 +32928,8 @@ int main (void) { - #ifndef has_mouse - void * volatile x = (void *)has_mouse; (void)x; + #ifndef use_env + void * volatile x = (void *)use_env; (void)x; #endif ; @@ -32674,9 +32938,9 @@ main (void) _ACEOF if ac_fn_c_try_link "$LINENO" then : - ac_cv_lib_curses_has_mouse=yes + ac_cv_lib_curses_use_env=yes else case e in #( - e) ac_cv_lib_curses_has_mouse=no ;; + e) ac_cv_lib_curses_use_env=no ;; esac fi rm -f core conftest.err conftest.$ac_objext conftest.beam \ @@ -32684,12 +32948,12 @@ rm -f core conftest.err conftest.$ac_objext conftest.beam \ ;; esac fi -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_has_mouse" >&5 -printf "%s\n" "$ac_cv_lib_curses_has_mouse" >&6; } - if test "x$ac_cv_lib_curses_has_mouse" = xyes +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_use_env" >&5 +printf "%s\n" "$ac_cv_lib_curses_use_env" >&6; } + if test "x$ac_cv_lib_curses_use_env" = xyes then : -printf "%s\n" "#define HAVE_CURSES_HAS_MOUSE 1" >>confdefs.h +printf "%s\n" "#define HAVE_CURSES_USE_ENV 1" >>confdefs.h fi @@ -32697,9 +32961,10 @@ fi - { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function is_keypad" >&5 -printf %s "checking for curses function is_keypad... " >&6; } -if test ${ac_cv_lib_curses_is_keypad+y} + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function set_escdelay" >&5 +printf %s "checking for curses function set_escdelay... " >&6; } +if test ${ac_cv_lib_curses_set_escdelay+y} then : printf %s "(cached) " >&6 else case e in #( @@ -32725,8 +32990,8 @@ int main (void) { - #ifndef is_keypad - void * volatile x = (void *)is_keypad; (void)x; + #ifndef set_escdelay + void * volatile x = (void *)set_escdelay; (void)x; #endif ; @@ -32735,9 +33000,9 @@ main (void) _ACEOF if ac_fn_c_try_link "$LINENO" then : - ac_cv_lib_curses_is_keypad=yes + ac_cv_lib_curses_set_escdelay=yes else case e in #( - e) ac_cv_lib_curses_is_keypad=no ;; + e) ac_cv_lib_curses_set_escdelay=no ;; esac fi rm -f core conftest.err conftest.$ac_objext conftest.beam \ @@ -32745,12 +33010,12 @@ rm -f core conftest.err conftest.$ac_objext conftest.beam \ ;; esac fi -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_is_keypad" >&5 -printf "%s\n" "$ac_cv_lib_curses_is_keypad" >&6; } - if test "x$ac_cv_lib_curses_is_keypad" = xyes +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_set_escdelay" >&5 +printf "%s\n" "$ac_cv_lib_curses_set_escdelay" >&6; } + if test "x$ac_cv_lib_curses_set_escdelay" = xyes then : -printf "%s\n" "#define HAVE_CURSES_IS_KEYPAD 1" >>confdefs.h +printf "%s\n" "#define HAVE_CURSES_SET_ESCDELAY 1" >>confdefs.h fi @@ -32758,9 +33023,10 @@ fi - { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function is_leaveok" >&5 -printf %s "checking for curses function is_leaveok... " >&6; } -if test ${ac_cv_lib_curses_is_leaveok+y} + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function set_tabsize" >&5 +printf %s "checking for curses function set_tabsize... " >&6; } +if test ${ac_cv_lib_curses_set_tabsize+y} then : printf %s "(cached) " >&6 else case e in #( @@ -32786,8 +33052,8 @@ int main (void) { - #ifndef is_leaveok - void * volatile x = (void *)is_leaveok; (void)x; + #ifndef set_tabsize + void * volatile x = (void *)set_tabsize; (void)x; #endif ; @@ -32796,9 +33062,9 @@ main (void) _ACEOF if ac_fn_c_try_link "$LINENO" then : - ac_cv_lib_curses_is_leaveok=yes + ac_cv_lib_curses_set_tabsize=yes else case e in #( - e) ac_cv_lib_curses_is_leaveok=no ;; + e) ac_cv_lib_curses_set_tabsize=no ;; esac fi rm -f core conftest.err conftest.$ac_objext conftest.beam \ @@ -32806,12 +33072,12 @@ rm -f core conftest.err conftest.$ac_objext conftest.beam \ ;; esac fi -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_is_leaveok" >&5 -printf "%s\n" "$ac_cv_lib_curses_is_leaveok" >&6; } - if test "x$ac_cv_lib_curses_is_leaveok" = xyes +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_set_tabsize" >&5 +printf "%s\n" "$ac_cv_lib_curses_set_tabsize" >&6; } + if test "x$ac_cv_lib_curses_set_tabsize" = xyes then : -printf "%s\n" "#define HAVE_CURSES_IS_LEAVEOK 1" >>confdefs.h +printf "%s\n" "#define HAVE_CURSES_SET_TABSIZE 1" >>confdefs.h fi @@ -32819,9 +33085,9 @@ fi - { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function typeahead" >&5 -printf %s "checking for curses function typeahead... " >&6; } -if test ${ac_cv_lib_curses_typeahead+y} + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses variable ESCDELAY" >&5 +printf %s "checking for curses variable ESCDELAY... " >&6; } +if test ${ac_cv_lib_curses_ESCDELAY+y} then : printf %s "(cached) " >&6 else case e in #( @@ -32847,32 +33113,29 @@ int main (void) { - #ifndef typeahead - void * volatile x = (void *)typeahead; (void)x; - #endif + int x = ESCDELAY; (void)x; ; return 0; } _ACEOF -if ac_fn_c_try_link "$LINENO" +if ac_fn_c_try_compile "$LINENO" then : - ac_cv_lib_curses_typeahead=yes + ac_cv_lib_curses_ESCDELAY=yes else case e in #( - e) ac_cv_lib_curses_typeahead=no ;; + e) ac_cv_lib_curses_ESCDELAY=no ;; esac fi -rm -f core conftest.err conftest.$ac_objext conftest.beam \ - conftest$ac_exeext conftest.$ac_ext +rm -f core conftest.err conftest.$ac_objext conftest.beam conftest.$ac_ext ;; esac fi -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_typeahead" >&5 -printf "%s\n" "$ac_cv_lib_curses_typeahead" >&6; } - if test "x$ac_cv_lib_curses_typeahead" = xyes +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_ESCDELAY" >&5 +printf "%s\n" "$ac_cv_lib_curses_ESCDELAY" >&6; } + if test "x$ac_cv_lib_curses_ESCDELAY" = xyes then : -printf "%s\n" "#define HAVE_CURSES_TYPEAHEAD 1" >>confdefs.h +printf "%s\n" "#define HAVE_CURSES_ESCDELAY 1" >>confdefs.h fi @@ -32880,9 +33143,9 @@ fi - { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function use_env" >&5 -printf %s "checking for curses function use_env... " >&6; } -if test ${ac_cv_lib_curses_use_env+y} + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses variable TABSIZE" >&5 +printf %s "checking for curses variable TABSIZE... " >&6; } +if test ${ac_cv_lib_curses_TABSIZE+y} then : printf %s "(cached) " >&6 else case e in #( @@ -32908,32 +33171,29 @@ int main (void) { - #ifndef use_env - void * volatile x = (void *)use_env; (void)x; - #endif + int x = TABSIZE; (void)x; ; return 0; } _ACEOF -if ac_fn_c_try_link "$LINENO" +if ac_fn_c_try_compile "$LINENO" then : - ac_cv_lib_curses_use_env=yes + ac_cv_lib_curses_TABSIZE=yes else case e in #( - e) ac_cv_lib_curses_use_env=no ;; + e) ac_cv_lib_curses_TABSIZE=no ;; esac fi -rm -f core conftest.err conftest.$ac_objext conftest.beam \ - conftest$ac_exeext conftest.$ac_ext +rm -f core conftest.err conftest.$ac_objext conftest.beam conftest.$ac_ext ;; esac fi -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_use_env" >&5 -printf "%s\n" "$ac_cv_lib_curses_use_env" >&6; } - if test "x$ac_cv_lib_curses_use_env" = xyes +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_TABSIZE" >&5 +printf "%s\n" "$ac_cv_lib_curses_TABSIZE" >&6; } + if test "x$ac_cv_lib_curses_TABSIZE" = xyes then : -printf "%s\n" "#define HAVE_CURSES_USE_ENV 1" >>confdefs.h +printf "%s\n" "#define HAVE_CURSES_TABSIZE 1" >>confdefs.h fi @@ -32941,9 +33201,11 @@ fi - { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function new_prescr" >&5 -printf %s "checking for curses function new_prescr... " >&6; } -if test ${ac_cv_lib_curses_new_prescr+y} + + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function define_key" >&5 +printf %s "checking for curses function define_key... " >&6; } +if test ${ac_cv_lib_curses_define_key+y} then : printf %s "(cached) " >&6 else case e in #( @@ -32969,8 +33231,8 @@ int main (void) { - #ifndef new_prescr - void * volatile x = (void *)new_prescr; (void)x; + #ifndef define_key + void * volatile x = (void *)define_key; (void)x; #endif ; @@ -32979,9 +33241,9 @@ main (void) _ACEOF if ac_fn_c_try_link "$LINENO" then : - ac_cv_lib_curses_new_prescr=yes + ac_cv_lib_curses_define_key=yes else case e in #( - e) ac_cv_lib_curses_new_prescr=no ;; + e) ac_cv_lib_curses_define_key=no ;; esac fi rm -f core conftest.err conftest.$ac_objext conftest.beam \ @@ -32989,12 +33251,12 @@ rm -f core conftest.err conftest.$ac_objext conftest.beam \ ;; esac fi -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_new_prescr" >&5 -printf "%s\n" "$ac_cv_lib_curses_new_prescr" >&6; } - if test "x$ac_cv_lib_curses_new_prescr" = xyes +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_define_key" >&5 +printf "%s\n" "$ac_cv_lib_curses_define_key" >&6; } + if test "x$ac_cv_lib_curses_define_key" = xyes then : -printf "%s\n" "#define HAVE_CURSES_NEW_PRESCR 1" >>confdefs.h +printf "%s\n" "#define _Py_HAVE_CURSES_DEFINE_KEY 1" >>confdefs.h fi @@ -33002,9 +33264,11 @@ fi - { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function use_screen" >&5 -printf %s "checking for curses function use_screen... " >&6; } -if test ${ac_cv_lib_curses_use_screen+y} + + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function has_mouse" >&5 +printf %s "checking for curses function has_mouse... " >&6; } +if test ${ac_cv_lib_curses_has_mouse+y} then : printf %s "(cached) " >&6 else case e in #( @@ -33030,8 +33294,8 @@ int main (void) { - #ifndef use_screen - void * volatile x = (void *)use_screen; (void)x; + #ifndef has_mouse + void * volatile x = (void *)has_mouse; (void)x; #endif ; @@ -33040,9 +33304,9 @@ main (void) _ACEOF if ac_fn_c_try_link "$LINENO" then : - ac_cv_lib_curses_use_screen=yes + ac_cv_lib_curses_has_mouse=yes else case e in #( - e) ac_cv_lib_curses_use_screen=no ;; + e) ac_cv_lib_curses_has_mouse=no ;; esac fi rm -f core conftest.err conftest.$ac_objext conftest.beam \ @@ -33050,12 +33314,12 @@ rm -f core conftest.err conftest.$ac_objext conftest.beam \ ;; esac fi -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_use_screen" >&5 -printf "%s\n" "$ac_cv_lib_curses_use_screen" >&6; } - if test "x$ac_cv_lib_curses_use_screen" = xyes +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_has_mouse" >&5 +printf "%s\n" "$ac_cv_lib_curses_has_mouse" >&6; } + if test "x$ac_cv_lib_curses_has_mouse" = xyes then : -printf "%s\n" "#define HAVE_CURSES_USE_SCREEN 1" >>confdefs.h +printf "%s\n" "#define _Py_HAVE_CURSES_HAS_MOUSE 1" >>confdefs.h fi @@ -33063,9 +33327,11 @@ fi - { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function use_window" >&5 -printf %s "checking for curses function use_window... " >&6; } -if test ${ac_cv_lib_curses_use_window+y} + + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function is_keypad" >&5 +printf %s "checking for curses function is_keypad... " >&6; } +if test ${ac_cv_lib_curses_is_keypad+y} then : printf %s "(cached) " >&6 else case e in #( @@ -33091,8 +33357,8 @@ int main (void) { - #ifndef use_window - void * volatile x = (void *)use_window; (void)x; + #ifndef is_keypad + void * volatile x = (void *)is_keypad; (void)x; #endif ; @@ -33101,9 +33367,9 @@ main (void) _ACEOF if ac_fn_c_try_link "$LINENO" then : - ac_cv_lib_curses_use_window=yes + ac_cv_lib_curses_is_keypad=yes else case e in #( - e) ac_cv_lib_curses_use_window=no ;; + e) ac_cv_lib_curses_is_keypad=no ;; esac fi rm -f core conftest.err conftest.$ac_objext conftest.beam \ @@ -33111,12 +33377,12 @@ rm -f core conftest.err conftest.$ac_objext conftest.beam \ ;; esac fi -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_use_window" >&5 -printf "%s\n" "$ac_cv_lib_curses_use_window" >&6; } - if test "x$ac_cv_lib_curses_use_window" = xyes +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_is_keypad" >&5 +printf "%s\n" "$ac_cv_lib_curses_is_keypad" >&6; } + if test "x$ac_cv_lib_curses_is_keypad" = xyes then : -printf "%s\n" "#define HAVE_CURSES_USE_WINDOW 1" >>confdefs.h +printf "%s\n" "#define _Py_HAVE_CURSES_IS_KEYPAD 1" >>confdefs.h fi @@ -33124,9 +33390,11 @@ fi - { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function key_defined" >&5 -printf %s "checking for curses function key_defined... " >&6; } -if test ${ac_cv_lib_curses_key_defined+y} + + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function is_leaveok" >&5 +printf %s "checking for curses function is_leaveok... " >&6; } +if test ${ac_cv_lib_curses_is_leaveok+y} then : printf %s "(cached) " >&6 else case e in #( @@ -33152,8 +33420,8 @@ int main (void) { - #ifndef key_defined - void * volatile x = (void *)key_defined; (void)x; + #ifndef is_leaveok + void * volatile x = (void *)is_leaveok; (void)x; #endif ; @@ -33162,9 +33430,9 @@ main (void) _ACEOF if ac_fn_c_try_link "$LINENO" then : - ac_cv_lib_curses_key_defined=yes + ac_cv_lib_curses_is_leaveok=yes else case e in #( - e) ac_cv_lib_curses_key_defined=no ;; + e) ac_cv_lib_curses_is_leaveok=no ;; esac fi rm -f core conftest.err conftest.$ac_objext conftest.beam \ @@ -33172,12 +33440,12 @@ rm -f core conftest.err conftest.$ac_objext conftest.beam \ ;; esac fi -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_key_defined" >&5 -printf "%s\n" "$ac_cv_lib_curses_key_defined" >&6; } - if test "x$ac_cv_lib_curses_key_defined" = xyes +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_is_leaveok" >&5 +printf "%s\n" "$ac_cv_lib_curses_is_leaveok" >&6; } + if test "x$ac_cv_lib_curses_is_leaveok" = xyes then : -printf "%s\n" "#define HAVE_CURSES_KEY_DEFINED 1" >>confdefs.h +printf "%s\n" "#define _Py_HAVE_CURSES_IS_LEAVEOK 1" >>confdefs.h fi @@ -33185,9 +33453,11 @@ fi - { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function term_attrs" >&5 -printf %s "checking for curses function term_attrs... " >&6; } -if test ${ac_cv_lib_curses_term_attrs+y} + + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function key_defined" >&5 +printf %s "checking for curses function key_defined... " >&6; } +if test ${ac_cv_lib_curses_key_defined+y} then : printf %s "(cached) " >&6 else case e in #( @@ -33213,8 +33483,8 @@ int main (void) { - #ifndef term_attrs - void * volatile x = (void *)term_attrs; (void)x; + #ifndef key_defined + void * volatile x = (void *)key_defined; (void)x; #endif ; @@ -33223,9 +33493,9 @@ main (void) _ACEOF if ac_fn_c_try_link "$LINENO" then : - ac_cv_lib_curses_term_attrs=yes + ac_cv_lib_curses_key_defined=yes else case e in #( - e) ac_cv_lib_curses_term_attrs=no ;; + e) ac_cv_lib_curses_key_defined=no ;; esac fi rm -f core conftest.err conftest.$ac_objext conftest.beam \ @@ -33233,12 +33503,12 @@ rm -f core conftest.err conftest.$ac_objext conftest.beam \ ;; esac fi -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_term_attrs" >&5 -printf "%s\n" "$ac_cv_lib_curses_term_attrs" >&6; } - if test "x$ac_cv_lib_curses_term_attrs" = xyes +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_key_defined" >&5 +printf "%s\n" "$ac_cv_lib_curses_key_defined" >&6; } + if test "x$ac_cv_lib_curses_key_defined" = xyes then : -printf "%s\n" "#define HAVE_CURSES_TERM_ATTRS 1" >>confdefs.h +printf "%s\n" "#define _Py_HAVE_CURSES_KEY_DEFINED 1" >>confdefs.h fi @@ -33246,9 +33516,11 @@ fi - { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function define_key" >&5 -printf %s "checking for curses function define_key... " >&6; } -if test ${ac_cv_lib_curses_define_key+y} + + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function keyok" >&5 +printf %s "checking for curses function keyok... " >&6; } +if test ${ac_cv_lib_curses_keyok+y} then : printf %s "(cached) " >&6 else case e in #( @@ -33274,8 +33546,8 @@ int main (void) { - #ifndef define_key - void * volatile x = (void *)define_key; (void)x; + #ifndef keyok + void * volatile x = (void *)keyok; (void)x; #endif ; @@ -33284,9 +33556,9 @@ main (void) _ACEOF if ac_fn_c_try_link "$LINENO" then : - ac_cv_lib_curses_define_key=yes + ac_cv_lib_curses_keyok=yes else case e in #( - e) ac_cv_lib_curses_define_key=no ;; + e) ac_cv_lib_curses_keyok=no ;; esac fi rm -f core conftest.err conftest.$ac_objext conftest.beam \ @@ -33294,12 +33566,12 @@ rm -f core conftest.err conftest.$ac_objext conftest.beam \ ;; esac fi -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_define_key" >&5 -printf "%s\n" "$ac_cv_lib_curses_define_key" >&6; } - if test "x$ac_cv_lib_curses_define_key" = xyes +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_keyok" >&5 +printf "%s\n" "$ac_cv_lib_curses_keyok" >&6; } + if test "x$ac_cv_lib_curses_keyok" = xyes then : -printf "%s\n" "#define HAVE_CURSES_DEFINE_KEY 1" >>confdefs.h +printf "%s\n" "#define _Py_HAVE_CURSES_KEYOK 1" >>confdefs.h fi @@ -33307,9 +33579,11 @@ fi - { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function keyok" >&5 -printf %s "checking for curses function keyok... " >&6; } -if test ${ac_cv_lib_curses_keyok+y} + + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function new_prescr" >&5 +printf %s "checking for curses function new_prescr... " >&6; } +if test ${ac_cv_lib_curses_new_prescr+y} then : printf %s "(cached) " >&6 else case e in #( @@ -33335,8 +33609,8 @@ int main (void) { - #ifndef keyok - void * volatile x = (void *)keyok; (void)x; + #ifndef new_prescr + void * volatile x = (void *)new_prescr; (void)x; #endif ; @@ -33345,9 +33619,9 @@ main (void) _ACEOF if ac_fn_c_try_link "$LINENO" then : - ac_cv_lib_curses_keyok=yes + ac_cv_lib_curses_new_prescr=yes else case e in #( - e) ac_cv_lib_curses_keyok=no ;; + e) ac_cv_lib_curses_new_prescr=no ;; esac fi rm -f core conftest.err conftest.$ac_objext conftest.beam \ @@ -33355,12 +33629,12 @@ rm -f core conftest.err conftest.$ac_objext conftest.beam \ ;; esac fi -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_keyok" >&5 -printf "%s\n" "$ac_cv_lib_curses_keyok" >&6; } - if test "x$ac_cv_lib_curses_keyok" = xyes +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_new_prescr" >&5 +printf "%s\n" "$ac_cv_lib_curses_new_prescr" >&6; } + if test "x$ac_cv_lib_curses_new_prescr" = xyes then : -printf "%s\n" "#define HAVE_CURSES_KEYOK 1" >>confdefs.h +printf "%s\n" "#define _Py_HAVE_CURSES_NEW_PRESCR 1" >>confdefs.h fi @@ -33368,9 +33642,11 @@ fi - { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function set_escdelay" >&5 -printf %s "checking for curses function set_escdelay... " >&6; } -if test ${ac_cv_lib_curses_set_escdelay+y} + + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function nofilter" >&5 +printf %s "checking for curses function nofilter... " >&6; } +if test ${ac_cv_lib_curses_nofilter+y} then : printf %s "(cached) " >&6 else case e in #( @@ -33396,8 +33672,8 @@ int main (void) { - #ifndef set_escdelay - void * volatile x = (void *)set_escdelay; (void)x; + #ifndef nofilter + void * volatile x = (void *)nofilter; (void)x; #endif ; @@ -33406,9 +33682,9 @@ main (void) _ACEOF if ac_fn_c_try_link "$LINENO" then : - ac_cv_lib_curses_set_escdelay=yes + ac_cv_lib_curses_nofilter=yes else case e in #( - e) ac_cv_lib_curses_set_escdelay=no ;; + e) ac_cv_lib_curses_nofilter=no ;; esac fi rm -f core conftest.err conftest.$ac_objext conftest.beam \ @@ -33416,12 +33692,12 @@ rm -f core conftest.err conftest.$ac_objext conftest.beam \ ;; esac fi -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_set_escdelay" >&5 -printf "%s\n" "$ac_cv_lib_curses_set_escdelay" >&6; } - if test "x$ac_cv_lib_curses_set_escdelay" = xyes +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_nofilter" >&5 +printf "%s\n" "$ac_cv_lib_curses_nofilter" >&6; } + if test "x$ac_cv_lib_curses_nofilter" = xyes then : -printf "%s\n" "#define HAVE_CURSES_SET_ESCDELAY 1" >>confdefs.h +printf "%s\n" "#define _Py_HAVE_CURSES_NOFILTER 1" >>confdefs.h fi @@ -33429,9 +33705,11 @@ fi - { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function set_tabsize" >&5 -printf %s "checking for curses function set_tabsize... " >&6; } -if test ${ac_cv_lib_curses_set_tabsize+y} + + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function slk_attr_off" >&5 +printf %s "checking for curses function slk_attr_off... " >&6; } +if test ${ac_cv_lib_curses_slk_attr_off+y} then : printf %s "(cached) " >&6 else case e in #( @@ -33457,8 +33735,8 @@ int main (void) { - #ifndef set_tabsize - void * volatile x = (void *)set_tabsize; (void)x; + #ifndef slk_attr_off + void * volatile x = (void *)slk_attr_off; (void)x; #endif ; @@ -33467,9 +33745,9 @@ main (void) _ACEOF if ac_fn_c_try_link "$LINENO" then : - ac_cv_lib_curses_set_tabsize=yes + ac_cv_lib_curses_slk_attr_off=yes else case e in #( - e) ac_cv_lib_curses_set_tabsize=no ;; + e) ac_cv_lib_curses_slk_attr_off=no ;; esac fi rm -f core conftest.err conftest.$ac_objext conftest.beam \ @@ -33477,12 +33755,12 @@ rm -f core conftest.err conftest.$ac_objext conftest.beam \ ;; esac fi -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_set_tabsize" >&5 -printf "%s\n" "$ac_cv_lib_curses_set_tabsize" >&6; } - if test "x$ac_cv_lib_curses_set_tabsize" = xyes +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_slk_attr_off" >&5 +printf "%s\n" "$ac_cv_lib_curses_slk_attr_off" >&6; } + if test "x$ac_cv_lib_curses_slk_attr_off" = xyes then : -printf "%s\n" "#define HAVE_CURSES_SET_TABSIZE 1" >>confdefs.h +printf "%s\n" "#define _Py_HAVE_CURSES_SLK_ATTR_OFF 1" >>confdefs.h fi @@ -33490,9 +33768,11 @@ fi - { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function wattr_get" >&5 -printf %s "checking for curses function wattr_get... " >&6; } -if test ${ac_cv_lib_curses_wattr_get+y} + + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function slk_attr_on" >&5 +printf %s "checking for curses function slk_attr_on... " >&6; } +if test ${ac_cv_lib_curses_slk_attr_on+y} then : printf %s "(cached) " >&6 else case e in #( @@ -33518,8 +33798,8 @@ int main (void) { - #ifndef wattr_get - void * volatile x = (void *)wattr_get; (void)x; + #ifndef slk_attr_on + void * volatile x = (void *)slk_attr_on; (void)x; #endif ; @@ -33528,9 +33808,9 @@ main (void) _ACEOF if ac_fn_c_try_link "$LINENO" then : - ac_cv_lib_curses_wattr_get=yes + ac_cv_lib_curses_slk_attr_on=yes else case e in #( - e) ac_cv_lib_curses_wattr_get=no ;; + e) ac_cv_lib_curses_slk_attr_on=no ;; esac fi rm -f core conftest.err conftest.$ac_objext conftest.beam \ @@ -33538,12 +33818,12 @@ rm -f core conftest.err conftest.$ac_objext conftest.beam \ ;; esac fi -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_wattr_get" >&5 -printf "%s\n" "$ac_cv_lib_curses_wattr_get" >&6; } - if test "x$ac_cv_lib_curses_wattr_get" = xyes +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_slk_attr_on" >&5 +printf "%s\n" "$ac_cv_lib_curses_slk_attr_on" >&6; } + if test "x$ac_cv_lib_curses_slk_attr_on" = xyes then : -printf "%s\n" "#define HAVE_CURSES_WATTR_GET 1" >>confdefs.h +printf "%s\n" "#define _Py_HAVE_CURSES_SLK_ATTR_ON 1" >>confdefs.h fi @@ -33551,9 +33831,11 @@ fi - { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function wattr_set" >&5 -printf %s "checking for curses function wattr_set... " >&6; } -if test ${ac_cv_lib_curses_wattr_set+y} + + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function slk_attr_set" >&5 +printf %s "checking for curses function slk_attr_set... " >&6; } +if test ${ac_cv_lib_curses_slk_attr_set+y} then : printf %s "(cached) " >&6 else case e in #( @@ -33579,8 +33861,8 @@ int main (void) { - #ifndef wattr_set - void * volatile x = (void *)wattr_set; (void)x; + #ifndef slk_attr_set + void * volatile x = (void *)slk_attr_set; (void)x; #endif ; @@ -33589,9 +33871,9 @@ main (void) _ACEOF if ac_fn_c_try_link "$LINENO" then : - ac_cv_lib_curses_wattr_set=yes + ac_cv_lib_curses_slk_attr_set=yes else case e in #( - e) ac_cv_lib_curses_wattr_set=no ;; + e) ac_cv_lib_curses_slk_attr_set=no ;; esac fi rm -f core conftest.err conftest.$ac_objext conftest.beam \ @@ -33599,12 +33881,12 @@ rm -f core conftest.err conftest.$ac_objext conftest.beam \ ;; esac fi -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_wattr_set" >&5 -printf "%s\n" "$ac_cv_lib_curses_wattr_set" >&6; } - if test "x$ac_cv_lib_curses_wattr_set" = xyes +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_slk_attr_set" >&5 +printf "%s\n" "$ac_cv_lib_curses_slk_attr_set" >&6; } + if test "x$ac_cv_lib_curses_slk_attr_set" = xyes then : -printf "%s\n" "#define HAVE_CURSES_WATTR_SET 1" >>confdefs.h +printf "%s\n" "#define _Py_HAVE_CURSES_SLK_ATTR_SET 1" >>confdefs.h fi @@ -33612,9 +33894,11 @@ fi - { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function wattr_on" >&5 -printf %s "checking for curses function wattr_on... " >&6; } -if test ${ac_cv_lib_curses_wattr_on+y} + + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function slk_color" >&5 +printf %s "checking for curses function slk_color... " >&6; } +if test ${ac_cv_lib_curses_slk_color+y} then : printf %s "(cached) " >&6 else case e in #( @@ -33640,8 +33924,8 @@ int main (void) { - #ifndef wattr_on - void * volatile x = (void *)wattr_on; (void)x; + #ifndef slk_color + void * volatile x = (void *)slk_color; (void)x; #endif ; @@ -33650,9 +33934,9 @@ main (void) _ACEOF if ac_fn_c_try_link "$LINENO" then : - ac_cv_lib_curses_wattr_on=yes + ac_cv_lib_curses_slk_color=yes else case e in #( - e) ac_cv_lib_curses_wattr_on=no ;; + e) ac_cv_lib_curses_slk_color=no ;; esac fi rm -f core conftest.err conftest.$ac_objext conftest.beam \ @@ -33660,12 +33944,12 @@ rm -f core conftest.err conftest.$ac_objext conftest.beam \ ;; esac fi -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_wattr_on" >&5 -printf "%s\n" "$ac_cv_lib_curses_wattr_on" >&6; } - if test "x$ac_cv_lib_curses_wattr_on" = xyes +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_slk_color" >&5 +printf "%s\n" "$ac_cv_lib_curses_slk_color" >&6; } + if test "x$ac_cv_lib_curses_slk_color" = xyes then : -printf "%s\n" "#define HAVE_CURSES_WATTR_ON 1" >>confdefs.h +printf "%s\n" "#define _Py_HAVE_CURSES_SLK_COLOR 1" >>confdefs.h fi @@ -33673,9 +33957,11 @@ fi - { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function wattr_off" >&5 -printf %s "checking for curses function wattr_off... " >&6; } -if test ${ac_cv_lib_curses_wattr_off+y} + + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function term_attrs" >&5 +printf %s "checking for curses function term_attrs... " >&6; } +if test ${ac_cv_lib_curses_term_attrs+y} then : printf %s "(cached) " >&6 else case e in #( @@ -33701,8 +33987,8 @@ int main (void) { - #ifndef wattr_off - void * volatile x = (void *)wattr_off; (void)x; + #ifndef term_attrs + void * volatile x = (void *)term_attrs; (void)x; #endif ; @@ -33711,9 +33997,9 @@ main (void) _ACEOF if ac_fn_c_try_link "$LINENO" then : - ac_cv_lib_curses_wattr_off=yes + ac_cv_lib_curses_term_attrs=yes else case e in #( - e) ac_cv_lib_curses_wattr_off=no ;; + e) ac_cv_lib_curses_term_attrs=no ;; esac fi rm -f core conftest.err conftest.$ac_objext conftest.beam \ @@ -33721,12 +34007,12 @@ rm -f core conftest.err conftest.$ac_objext conftest.beam \ ;; esac fi -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_wattr_off" >&5 -printf "%s\n" "$ac_cv_lib_curses_wattr_off" >&6; } - if test "x$ac_cv_lib_curses_wattr_off" = xyes +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_term_attrs" >&5 +printf "%s\n" "$ac_cv_lib_curses_term_attrs" >&6; } + if test "x$ac_cv_lib_curses_term_attrs" = xyes then : -printf "%s\n" "#define HAVE_CURSES_WATTR_OFF 1" >>confdefs.h +printf "%s\n" "#define _Py_HAVE_CURSES_TERM_ATTRS 1" >>confdefs.h fi @@ -33734,9 +34020,11 @@ fi - { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function wcolor_set" >&5 -printf %s "checking for curses function wcolor_set... " >&6; } -if test ${ac_cv_lib_curses_wcolor_set+y} + + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function use_screen" >&5 +printf %s "checking for curses function use_screen... " >&6; } +if test ${ac_cv_lib_curses_use_screen+y} then : printf %s "(cached) " >&6 else case e in #( @@ -33762,8 +34050,8 @@ int main (void) { - #ifndef wcolor_set - void * volatile x = (void *)wcolor_set; (void)x; + #ifndef use_screen + void * volatile x = (void *)use_screen; (void)x; #endif ; @@ -33772,9 +34060,9 @@ main (void) _ACEOF if ac_fn_c_try_link "$LINENO" then : - ac_cv_lib_curses_wcolor_set=yes + ac_cv_lib_curses_use_screen=yes else case e in #( - e) ac_cv_lib_curses_wcolor_set=no ;; + e) ac_cv_lib_curses_use_screen=no ;; esac fi rm -f core conftest.err conftest.$ac_objext conftest.beam \ @@ -33782,12 +34070,12 @@ rm -f core conftest.err conftest.$ac_objext conftest.beam \ ;; esac fi -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_wcolor_set" >&5 -printf "%s\n" "$ac_cv_lib_curses_wcolor_set" >&6; } - if test "x$ac_cv_lib_curses_wcolor_set" = xyes +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_use_screen" >&5 +printf "%s\n" "$ac_cv_lib_curses_use_screen" >&6; } + if test "x$ac_cv_lib_curses_use_screen" = xyes then : -printf "%s\n" "#define HAVE_CURSES_WCOLOR_SET 1" >>confdefs.h +printf "%s\n" "#define _Py_HAVE_CURSES_USE_SCREEN 1" >>confdefs.h fi @@ -33795,9 +34083,11 @@ fi - { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function slk_attr_on" >&5 -printf %s "checking for curses function slk_attr_on... " >&6; } -if test ${ac_cv_lib_curses_slk_attr_on+y} + + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function use_window" >&5 +printf %s "checking for curses function use_window... " >&6; } +if test ${ac_cv_lib_curses_use_window+y} then : printf %s "(cached) " >&6 else case e in #( @@ -33823,8 +34113,8 @@ int main (void) { - #ifndef slk_attr_on - void * volatile x = (void *)slk_attr_on; (void)x; + #ifndef use_window + void * volatile x = (void *)use_window; (void)x; #endif ; @@ -33833,9 +34123,9 @@ main (void) _ACEOF if ac_fn_c_try_link "$LINENO" then : - ac_cv_lib_curses_slk_attr_on=yes + ac_cv_lib_curses_use_window=yes else case e in #( - e) ac_cv_lib_curses_slk_attr_on=no ;; + e) ac_cv_lib_curses_use_window=no ;; esac fi rm -f core conftest.err conftest.$ac_objext conftest.beam \ @@ -33843,12 +34133,12 @@ rm -f core conftest.err conftest.$ac_objext conftest.beam \ ;; esac fi -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_slk_attr_on" >&5 -printf "%s\n" "$ac_cv_lib_curses_slk_attr_on" >&6; } - if test "x$ac_cv_lib_curses_slk_attr_on" = xyes +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_use_window" >&5 +printf "%s\n" "$ac_cv_lib_curses_use_window" >&6; } + if test "x$ac_cv_lib_curses_use_window" = xyes then : -printf "%s\n" "#define HAVE_CURSES_SLK_ATTR_ON 1" >>confdefs.h +printf "%s\n" "#define _Py_HAVE_CURSES_USE_WINDOW 1" >>confdefs.h fi @@ -33856,9 +34146,11 @@ fi - { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function slk_attr_off" >&5 -printf %s "checking for curses function slk_attr_off... " >&6; } -if test ${ac_cv_lib_curses_slk_attr_off+y} + + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function wattr_get" >&5 +printf %s "checking for curses function wattr_get... " >&6; } +if test ${ac_cv_lib_curses_wattr_get+y} then : printf %s "(cached) " >&6 else case e in #( @@ -33884,8 +34176,8 @@ int main (void) { - #ifndef slk_attr_off - void * volatile x = (void *)slk_attr_off; (void)x; + #ifndef wattr_get + void * volatile x = (void *)wattr_get; (void)x; #endif ; @@ -33894,9 +34186,9 @@ main (void) _ACEOF if ac_fn_c_try_link "$LINENO" then : - ac_cv_lib_curses_slk_attr_off=yes + ac_cv_lib_curses_wattr_get=yes else case e in #( - e) ac_cv_lib_curses_slk_attr_off=no ;; + e) ac_cv_lib_curses_wattr_get=no ;; esac fi rm -f core conftest.err conftest.$ac_objext conftest.beam \ @@ -33904,12 +34196,12 @@ rm -f core conftest.err conftest.$ac_objext conftest.beam \ ;; esac fi -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_slk_attr_off" >&5 -printf "%s\n" "$ac_cv_lib_curses_slk_attr_off" >&6; } - if test "x$ac_cv_lib_curses_slk_attr_off" = xyes +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_wattr_get" >&5 +printf "%s\n" "$ac_cv_lib_curses_wattr_get" >&6; } + if test "x$ac_cv_lib_curses_wattr_get" = xyes then : -printf "%s\n" "#define HAVE_CURSES_SLK_ATTR_OFF 1" >>confdefs.h +printf "%s\n" "#define _Py_HAVE_CURSES_WATTR_GET 1" >>confdefs.h fi @@ -33917,9 +34209,11 @@ fi - { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function slk_attr_set" >&5 -printf %s "checking for curses function slk_attr_set... " >&6; } -if test ${ac_cv_lib_curses_slk_attr_set+y} + + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function wattr_off" >&5 +printf %s "checking for curses function wattr_off... " >&6; } +if test ${ac_cv_lib_curses_wattr_off+y} then : printf %s "(cached) " >&6 else case e in #( @@ -33945,8 +34239,8 @@ int main (void) { - #ifndef slk_attr_set - void * volatile x = (void *)slk_attr_set; (void)x; + #ifndef wattr_off + void * volatile x = (void *)wattr_off; (void)x; #endif ; @@ -33955,9 +34249,9 @@ main (void) _ACEOF if ac_fn_c_try_link "$LINENO" then : - ac_cv_lib_curses_slk_attr_set=yes + ac_cv_lib_curses_wattr_off=yes else case e in #( - e) ac_cv_lib_curses_slk_attr_set=no ;; + e) ac_cv_lib_curses_wattr_off=no ;; esac fi rm -f core conftest.err conftest.$ac_objext conftest.beam \ @@ -33965,12 +34259,12 @@ rm -f core conftest.err conftest.$ac_objext conftest.beam \ ;; esac fi -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_slk_attr_set" >&5 -printf "%s\n" "$ac_cv_lib_curses_slk_attr_set" >&6; } - if test "x$ac_cv_lib_curses_slk_attr_set" = xyes +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_wattr_off" >&5 +printf "%s\n" "$ac_cv_lib_curses_wattr_off" >&6; } + if test "x$ac_cv_lib_curses_wattr_off" = xyes then : -printf "%s\n" "#define HAVE_CURSES_SLK_ATTR_SET 1" >>confdefs.h +printf "%s\n" "#define _Py_HAVE_CURSES_WATTR_OFF 1" >>confdefs.h fi @@ -33978,9 +34272,11 @@ fi - { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function slk_color" >&5 -printf %s "checking for curses function slk_color... " >&6; } -if test ${ac_cv_lib_curses_slk_color+y} + + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function wattr_on" >&5 +printf %s "checking for curses function wattr_on... " >&6; } +if test ${ac_cv_lib_curses_wattr_on+y} then : printf %s "(cached) " >&6 else case e in #( @@ -34006,8 +34302,8 @@ int main (void) { - #ifndef slk_color - void * volatile x = (void *)slk_color; (void)x; + #ifndef wattr_on + void * volatile x = (void *)wattr_on; (void)x; #endif ; @@ -34016,9 +34312,9 @@ main (void) _ACEOF if ac_fn_c_try_link "$LINENO" then : - ac_cv_lib_curses_slk_color=yes + ac_cv_lib_curses_wattr_on=yes else case e in #( - e) ac_cv_lib_curses_slk_color=no ;; + e) ac_cv_lib_curses_wattr_on=no ;; esac fi rm -f core conftest.err conftest.$ac_objext conftest.beam \ @@ -34026,12 +34322,12 @@ rm -f core conftest.err conftest.$ac_objext conftest.beam \ ;; esac fi -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_slk_color" >&5 -printf "%s\n" "$ac_cv_lib_curses_slk_color" >&6; } - if test "x$ac_cv_lib_curses_slk_color" = xyes +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_wattr_on" >&5 +printf "%s\n" "$ac_cv_lib_curses_wattr_on" >&6; } + if test "x$ac_cv_lib_curses_wattr_on" = xyes then : -printf "%s\n" "#define HAVE_CURSES_SLK_COLOR 1" >>confdefs.h +printf "%s\n" "#define _Py_HAVE_CURSES_WATTR_ON 1" >>confdefs.h fi @@ -34039,9 +34335,11 @@ fi - { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses variable ESCDELAY" >&5 -printf %s "checking for curses variable ESCDELAY... " >&6; } -if test ${ac_cv_lib_curses_ESCDELAY+y} + + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function wattr_set" >&5 +printf %s "checking for curses function wattr_set... " >&6; } +if test ${ac_cv_lib_curses_wattr_set+y} then : printf %s "(cached) " >&6 else case e in #( @@ -34067,29 +34365,32 @@ int main (void) { - int x = ESCDELAY; (void)x; + #ifndef wattr_set + void * volatile x = (void *)wattr_set; (void)x; + #endif ; return 0; } _ACEOF -if ac_fn_c_try_compile "$LINENO" +if ac_fn_c_try_link "$LINENO" then : - ac_cv_lib_curses_ESCDELAY=yes + ac_cv_lib_curses_wattr_set=yes else case e in #( - e) ac_cv_lib_curses_ESCDELAY=no ;; + e) ac_cv_lib_curses_wattr_set=no ;; esac fi -rm -f core conftest.err conftest.$ac_objext conftest.beam conftest.$ac_ext +rm -f core conftest.err conftest.$ac_objext conftest.beam \ + conftest$ac_exeext conftest.$ac_ext ;; esac fi -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_ESCDELAY" >&5 -printf "%s\n" "$ac_cv_lib_curses_ESCDELAY" >&6; } - if test "x$ac_cv_lib_curses_ESCDELAY" = xyes +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_wattr_set" >&5 +printf "%s\n" "$ac_cv_lib_curses_wattr_set" >&6; } + if test "x$ac_cv_lib_curses_wattr_set" = xyes then : -printf "%s\n" "#define HAVE_CURSES_ESCDELAY 1" >>confdefs.h +printf "%s\n" "#define _Py_HAVE_CURSES_WATTR_SET 1" >>confdefs.h fi @@ -34097,9 +34398,11 @@ fi - { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses variable TABSIZE" >&5 -printf %s "checking for curses variable TABSIZE... " >&6; } -if test ${ac_cv_lib_curses_TABSIZE+y} + + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function wcolor_set" >&5 +printf %s "checking for curses function wcolor_set... " >&6; } +if test ${ac_cv_lib_curses_wcolor_set+y} then : printf %s "(cached) " >&6 else case e in #( @@ -34125,35 +34428,39 @@ int main (void) { - int x = TABSIZE; (void)x; + #ifndef wcolor_set + void * volatile x = (void *)wcolor_set; (void)x; + #endif ; return 0; } _ACEOF -if ac_fn_c_try_compile "$LINENO" +if ac_fn_c_try_link "$LINENO" then : - ac_cv_lib_curses_TABSIZE=yes + ac_cv_lib_curses_wcolor_set=yes else case e in #( - e) ac_cv_lib_curses_TABSIZE=no ;; + e) ac_cv_lib_curses_wcolor_set=no ;; esac fi -rm -f core conftest.err conftest.$ac_objext conftest.beam conftest.$ac_ext +rm -f core conftest.err conftest.$ac_objext conftest.beam \ + conftest$ac_exeext conftest.$ac_ext ;; esac fi -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_TABSIZE" >&5 -printf "%s\n" "$ac_cv_lib_curses_TABSIZE" >&6; } - if test "x$ac_cv_lib_curses_TABSIZE" = xyes +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_wcolor_set" >&5 +printf "%s\n" "$ac_cv_lib_curses_wcolor_set" >&6; } + if test "x$ac_cv_lib_curses_wcolor_set" = xyes then : -printf "%s\n" "#define HAVE_CURSES_TABSIZE 1" >>confdefs.h +printf "%s\n" "#define _Py_HAVE_CURSES_WCOLOR_SET 1" >>confdefs.h fi + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for ncurses-style curses function getmouse" >&5 printf %s "checking for ncurses-style curses function getmouse... " >&6; } if test ${ac_cv_lib_curses_getmouse+y} @@ -34207,6 +34514,7 @@ fi + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function scr_dump" >&5 printf %s "checking for curses function scr_dump... " >&6; } if test ${ac_cv_lib_curses_scr_dump+y} @@ -34260,7 +34568,7 @@ printf "%s\n" "$ac_cv_lib_curses_scr_dump" >&6; } if test "x$ac_cv_lib_curses_scr_dump" = xyes then : -printf "%s\n" "#define HAVE_CURSES_SCR_DUMP 1" >>confdefs.h +printf "%s\n" "#define _Py_HAVE_CURSES_SCR_DUMP 1" >>confdefs.h fi @@ -34268,6 +34576,8 @@ fi + + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for curses function scr_set" >&5 printf %s "checking for curses function scr_set... " >&6; } if test ${ac_cv_lib_curses_scr_set+y} @@ -34321,12 +34631,14 @@ printf "%s\n" "$ac_cv_lib_curses_scr_set" >&6; } if test "x$ac_cv_lib_curses_scr_set" = xyes then : -printf "%s\n" "#define HAVE_CURSES_SCR_SET 1" >>confdefs.h +printf "%s\n" "#define _Py_HAVE_CURSES_SCR_SET 1" >>confdefs.h fi + + CPPFLAGS=$ac_save_cppflags fi diff --git a/configure.ac b/configure.ac index d654f680643041c..234e2fbc920822f 100644 --- a/configure.ac +++ b/configure.ac @@ -5757,15 +5757,15 @@ AC_CHECK_FUNCS([ \ copy_file_range ctermid dladdr dup explicit_bzero explicit_memset \ faccessat fchmod fchmodat fchown fchownat fdopendir fdwalk fexecve \ fpathconf fstatat ftruncate futimens futimes futimesat \ - gai_strerror getdents64 getegid geteuid getgid getgrent getgrgid getgrgid_r \ + gai_strerror getegid geteuid getgid getgrent getgrgid getgrgid_r \ getgrnam_r getgrouplist gethostname getitimer getloadavg getlogin getlogin_r \ getpeername getpgid getpid getppid getpriority _getpty \ getpwent getpwnam_r getpwuid getpwuid_r getresgid getresuid getrusage getsid \ - gettid getuid grantpt if_indextoname if_nameindex \ - if_nametoindex initgroups kill killpg lchown linkat \ + getuid grantpt if_nameindex \ + initgroups kill killpg lchown linkat \ lockf lstat lutimes madvise mbrtowc memrchr mkdirat mkfifo mkfifoat \ mknod mknodat mktime mmap mremap nice openat opendir pathconf pause \ - pidfd_open pidfd_getfd pidfd_send_signal pipe \ + pipe \ plock poll ppoll posix_fadvise posix_fallocate posix_openpt \ pread preadv preadv2 process_vm_readv \ pthread_cond_timedwait_relative_np pthread_condattr_setclock pthread_init \ @@ -5788,7 +5788,13 @@ PY_CHECK_FUNC_PRIVATE([freadlink], [@%:@include ]) PY_CHECK_FUNC_PRIVATE([strerror_r], [@%:@include ]) # FreeBSD pthread_attr_get_np() PY_CHECK_FUNC_PRIVATE([pthread_attr_get_np], [@%:@include ]) - +PY_CHECK_FUNC_PRIVATE([getdents64], [@%:@include ]) +PY_CHECK_FUNC_PRIVATE([gettid], [@%:@include ]) +PY_CHECK_FUNC_PRIVATE([if_indextoname], [@%:@include ]) +PY_CHECK_FUNC_PRIVATE([if_nametoindex], [@%:@include ]) +PY_CHECK_FUNC_PRIVATE([pidfd_open], [@%:@include ]) +PY_CHECK_FUNC_PRIVATE([pidfd_getfd], [@%:@include ]) +PY_CHECK_FUNC_PRIVATE([pidfd_send_signal], [@%:@include ]) # os.statx uses Linux's statx function. AIX also has a function named statx, # but it's unrelated. Check only on Linux (including Android). @@ -7255,7 +7261,7 @@ AS_VAR_IF([with_readline], [no], [ ) ]) AS_VAR_IF([ac_cv_readline_rl_change_environment], [yes], [ - AC_DEFINE([HAVE_RL_CHANGE_ENVIRONMENT], [1], [Define if you have readline 6.3]) + AC_DEFINE([_Py_HAVE_RL_CHANGE_ENVIRONMENT], [1], [Define if you have readline 6.3]) ]) # check for readline 4.2 @@ -7660,6 +7666,7 @@ dnl lacks, for example term_attrs() in the narrow ncurses library. AC_DEFUN([PY_CHECK_CURSES_FUNC], [ AS_VAR_PUSHDEF([py_var], [ac_cv_lib_curses_$1]) AS_VAR_PUSHDEF([py_define], [HAVE_CURSES_]m4_toupper($1)) + AS_VAR_PUSHDEF([py_define], m4_ifblank([$3], [[HAVE_CURSES_]m4_toupper($1)], [$3])) AC_CACHE_CHECK( [for curses function $1], [py_var], @@ -7680,6 +7687,13 @@ AC_DEFUN([PY_CHECK_CURSES_FUNC], AS_VAR_POPDEF([py_define]) ]) +dnl PY_CHECK_CURSES_FUNC_PRIVATE(FUNCTION, [INCLUDES], [AC_DEFINE-VAR]) +dnl Similar to PY_CHECK_CURSES_FUNC but define macro _Py_HAVE_CURSES_xxx, +dnl instead of HAVE_CURSES_xxx +AC_DEFUN([PY_CHECK_CURSES_FUNC_PRIVATE], +[ PY_CHECK_CURSES_FUNC([$1], [$2], m4_ifblank([$3], [[_Py_HAVE_CURSES_]m4_toupper($1)], [$3])) +]) + dnl PY_CHECK_CURSES_VAR(VARIABLE) dnl Like PY_CHECK_CURSES_FUNC, but for an integer variable (or macro), such as dnl ESCDELAY, which a function probe cannot detect. @@ -7712,36 +7726,37 @@ PY_CHECK_CURSES_FUNC([immedok]) PY_CHECK_CURSES_FUNC([syncok]) PY_CHECK_CURSES_FUNC([wchgat]) PY_CHECK_CURSES_FUNC([filter]) -PY_CHECK_CURSES_FUNC([nofilter]) PY_CHECK_CURSES_FUNC([has_key]) -PY_CHECK_CURSES_FUNC([has_mouse]) -PY_CHECK_CURSES_FUNC([is_keypad]) -PY_CHECK_CURSES_FUNC([is_leaveok]) PY_CHECK_CURSES_FUNC([typeahead]) PY_CHECK_CURSES_FUNC([use_env]) -PY_CHECK_CURSES_FUNC([new_prescr]) -PY_CHECK_CURSES_FUNC([use_screen]) -PY_CHECK_CURSES_FUNC([use_window]) -PY_CHECK_CURSES_FUNC([key_defined]) -PY_CHECK_CURSES_FUNC([term_attrs]) -PY_CHECK_CURSES_FUNC([define_key]) -PY_CHECK_CURSES_FUNC([keyok]) PY_CHECK_CURSES_FUNC([set_escdelay]) PY_CHECK_CURSES_FUNC([set_tabsize]) dnl The X/Open attr_t and soft-label attribute functions are absent on old dnl SVr4 curses (e.g. illumos). -PY_CHECK_CURSES_FUNC([wattr_get]) -PY_CHECK_CURSES_FUNC([wattr_set]) -PY_CHECK_CURSES_FUNC([wattr_on]) -PY_CHECK_CURSES_FUNC([wattr_off]) -PY_CHECK_CURSES_FUNC([wcolor_set]) -PY_CHECK_CURSES_FUNC([slk_attr_on]) -PY_CHECK_CURSES_FUNC([slk_attr_off]) -PY_CHECK_CURSES_FUNC([slk_attr_set]) -PY_CHECK_CURSES_FUNC([slk_color]) PY_CHECK_CURSES_VAR([ESCDELAY]) PY_CHECK_CURSES_VAR([TABSIZE]) +PY_CHECK_CURSES_FUNC_PRIVATE([define_key]) +PY_CHECK_CURSES_FUNC_PRIVATE([has_mouse]) +PY_CHECK_CURSES_FUNC_PRIVATE([is_keypad]) +PY_CHECK_CURSES_FUNC_PRIVATE([is_leaveok]) +PY_CHECK_CURSES_FUNC_PRIVATE([key_defined]) +PY_CHECK_CURSES_FUNC_PRIVATE([keyok]) +PY_CHECK_CURSES_FUNC_PRIVATE([new_prescr]) +PY_CHECK_CURSES_FUNC_PRIVATE([nofilter]) +PY_CHECK_CURSES_FUNC_PRIVATE([slk_attr_off]) +PY_CHECK_CURSES_FUNC_PRIVATE([slk_attr_on]) +PY_CHECK_CURSES_FUNC_PRIVATE([slk_attr_set]) +PY_CHECK_CURSES_FUNC_PRIVATE([slk_color]) +PY_CHECK_CURSES_FUNC_PRIVATE([term_attrs]) +PY_CHECK_CURSES_FUNC_PRIVATE([use_screen]) +PY_CHECK_CURSES_FUNC_PRIVATE([use_window]) +PY_CHECK_CURSES_FUNC_PRIVATE([wattr_get]) +PY_CHECK_CURSES_FUNC_PRIVATE([wattr_off]) +PY_CHECK_CURSES_FUNC_PRIVATE([wattr_on]) +PY_CHECK_CURSES_FUNC_PRIVATE([wattr_set]) +PY_CHECK_CURSES_FUNC_PRIVATE([wcolor_set]) + dnl Probe for the X/Open getmouse(MEVENT *) signature specifically: PDCurses dnl declares an incompatible getmouse(void) unless built for the ncurses mouse API. AC_CACHE_CHECK([for ncurses-style curses function getmouse], @@ -7756,8 +7771,9 @@ AS_VAR_IF([ac_cv_lib_curses_getmouse], [yes], dnl scr_dump gates scr_dump/scr_restore/scr_init; scr_set is separate, since dnl old SVr4 curses (e.g. illumos) has the former but not scr_set. -PY_CHECK_CURSES_FUNC([scr_dump]) -PY_CHECK_CURSES_FUNC([scr_set]) +PY_CHECK_CURSES_FUNC_PRIVATE([scr_dump]) +PY_CHECK_CURSES_FUNC_PRIVATE([scr_set]) + CPPFLAGS=$ac_save_cppflags ])dnl have_curses != no ])dnl save env diff --git a/pyconfig.h.in b/pyconfig.h.in index dba7d193bcb3d19..1d4f445a8b1fdaa 100644 --- a/pyconfig.h.in +++ b/pyconfig.h.in @@ -185,9 +185,6 @@ /* Define if you have the 'ctermid_r' function. */ #undef HAVE_CTERMID_R -/* Define if you have the 'define_key' function. */ -#undef HAVE_CURSES_DEFINE_KEY - /* Define if you have the 'ESCDELAY' variable. */ #undef HAVE_CURSES_ESCDELAY @@ -203,105 +200,42 @@ /* Define if you have the 'has_key' function. */ #undef HAVE_CURSES_HAS_KEY -/* Define if you have the 'has_mouse' function. */ -#undef HAVE_CURSES_HAS_MOUSE - /* Define if you have the 'immedok' function. */ #undef HAVE_CURSES_IMMEDOK -/* Define if you have the 'is_keypad' function. */ -#undef HAVE_CURSES_IS_KEYPAD - -/* Define if you have the 'is_leaveok' function. */ -#undef HAVE_CURSES_IS_LEAVEOK - /* Define if you have the 'is_pad' function. */ #undef HAVE_CURSES_IS_PAD /* Define if you have the 'is_term_resized' function. */ #undef HAVE_CURSES_IS_TERM_RESIZED -/* Define if you have the 'keyok' function. */ -#undef HAVE_CURSES_KEYOK - -/* Define if you have the 'key_defined' function. */ -#undef HAVE_CURSES_KEY_DEFINED - -/* Define if you have the 'new_prescr' function. */ -#undef HAVE_CURSES_NEW_PRESCR - -/* Define if you have the 'nofilter' function. */ -#undef HAVE_CURSES_NOFILTER - /* Define if you have the 'resizeterm' function. */ #undef HAVE_CURSES_RESIZETERM /* Define if you have the 'resize_term' function. */ #undef HAVE_CURSES_RESIZE_TERM -/* Define if you have the 'scr_dump' function. */ -#undef HAVE_CURSES_SCR_DUMP - -/* Define if you have the 'scr_set' function. */ -#undef HAVE_CURSES_SCR_SET - /* Define if you have the 'set_escdelay' function. */ #undef HAVE_CURSES_SET_ESCDELAY /* Define if you have the 'set_tabsize' function. */ #undef HAVE_CURSES_SET_TABSIZE -/* Define if you have the 'slk_attr_off' function. */ -#undef HAVE_CURSES_SLK_ATTR_OFF - -/* Define if you have the 'slk_attr_on' function. */ -#undef HAVE_CURSES_SLK_ATTR_ON - -/* Define if you have the 'slk_attr_set' function. */ -#undef HAVE_CURSES_SLK_ATTR_SET - -/* Define if you have the 'slk_color' function. */ -#undef HAVE_CURSES_SLK_COLOR - /* Define if you have the 'syncok' function. */ #undef HAVE_CURSES_SYNCOK /* Define if you have the 'TABSIZE' variable. */ #undef HAVE_CURSES_TABSIZE -/* Define if you have the 'term_attrs' function. */ -#undef HAVE_CURSES_TERM_ATTRS - /* Define if you have the 'typeahead' function. */ #undef HAVE_CURSES_TYPEAHEAD /* Define if you have the 'use_env' function. */ #undef HAVE_CURSES_USE_ENV -/* Define if you have the 'use_screen' function. */ -#undef HAVE_CURSES_USE_SCREEN - -/* Define if you have the 'use_window' function. */ -#undef HAVE_CURSES_USE_WINDOW - -/* Define if you have the 'wattr_get' function. */ -#undef HAVE_CURSES_WATTR_GET - -/* Define if you have the 'wattr_off' function. */ -#undef HAVE_CURSES_WATTR_OFF - -/* Define if you have the 'wattr_on' function. */ -#undef HAVE_CURSES_WATTR_ON - -/* Define if you have the 'wattr_set' function. */ -#undef HAVE_CURSES_WATTR_SET - /* Define if you have the 'wchgat' function. */ #undef HAVE_CURSES_WCHGAT -/* Define if you have the 'wcolor_set' function. */ -#undef HAVE_CURSES_WCOLOR_SET - /* Define to 1 if you have the header file. */ #undef HAVE_DB_H @@ -552,9 +486,6 @@ /* Define this if you have flockfile(), getc_unlocked(), and funlockfile() */ #undef HAVE_GETC_UNLOCKED -/* Define to 1 if you have the 'getdents64' function. */ -#undef HAVE_GETDENTS64 - /* Define to 1 if you have the 'getegid' function. */ #undef HAVE_GETEGID @@ -684,9 +615,6 @@ /* Define if you have the 'getsockname' function. */ #undef HAVE_GETSOCKNAME -/* Define to 1 if you have the 'gettid' function. */ -#undef HAVE_GETTID - /* Define to 1 if you have the 'getuid' function. */ #undef HAVE_GETUID @@ -705,15 +633,9 @@ /* Define to 1 if you have the header file. */ #undef HAVE_ICONV_H -/* Define to 1 if you have the 'if_indextoname' function. */ -#undef HAVE_IF_INDEXTONAME - /* Define to 1 if you have the 'if_nameindex' function. */ #undef HAVE_IF_NAMEINDEX -/* Define to 1 if you have the 'if_nametoindex' function. */ -#undef HAVE_IF_NAMETOINDEX - /* Define if you have the 'inet_aton' function. */ #undef HAVE_INET_ATON @@ -1009,15 +931,6 @@ /* Define to 1 if you have the 'pause' function. */ #undef HAVE_PAUSE -/* Define to 1 if you have the 'pidfd_getfd' function. */ -#undef HAVE_PIDFD_GETFD - -/* Define to 1 if you have the 'pidfd_open' function. */ -#undef HAVE_PIDFD_OPEN - -/* Define to 1 if you have the 'pidfd_send_signal' function. */ -#undef HAVE_PIDFD_SEND_SIGNAL - /* Define to 1 if you have the 'pipe' function. */ #undef HAVE_PIPE @@ -1164,9 +1077,6 @@ /* Define if you can turn off readline's signal handling. */ #undef HAVE_RL_CATCH_SIGNAL -/* Define if you have readline 6.3 */ -#undef HAVE_RL_CHANGE_ENVIRONMENT - /* Define to 1 if the system has the type 'rl_compdisp_func_t'. */ #undef HAVE_RL_COMPDISP_FUNC_T @@ -2141,6 +2051,72 @@ /* Define if you have the 'cospi' function. */ #undef _Py_HAVE_COSPI +/* Define if you have the 'define_key' function. */ +#undef _Py_HAVE_CURSES_DEFINE_KEY + +/* Define if you have the 'has_mouse' function. */ +#undef _Py_HAVE_CURSES_HAS_MOUSE + +/* Define if you have the 'is_keypad' function. */ +#undef _Py_HAVE_CURSES_IS_KEYPAD + +/* Define if you have the 'is_leaveok' function. */ +#undef _Py_HAVE_CURSES_IS_LEAVEOK + +/* Define if you have the 'keyok' function. */ +#undef _Py_HAVE_CURSES_KEYOK + +/* Define if you have the 'key_defined' function. */ +#undef _Py_HAVE_CURSES_KEY_DEFINED + +/* Define if you have the 'new_prescr' function. */ +#undef _Py_HAVE_CURSES_NEW_PRESCR + +/* Define if you have the 'nofilter' function. */ +#undef _Py_HAVE_CURSES_NOFILTER + +/* Define if you have the 'scr_dump' function. */ +#undef _Py_HAVE_CURSES_SCR_DUMP + +/* Define if you have the 'scr_set' function. */ +#undef _Py_HAVE_CURSES_SCR_SET + +/* Define if you have the 'slk_attr_off' function. */ +#undef _Py_HAVE_CURSES_SLK_ATTR_OFF + +/* Define if you have the 'slk_attr_on' function. */ +#undef _Py_HAVE_CURSES_SLK_ATTR_ON + +/* Define if you have the 'slk_attr_set' function. */ +#undef _Py_HAVE_CURSES_SLK_ATTR_SET + +/* Define if you have the 'slk_color' function. */ +#undef _Py_HAVE_CURSES_SLK_COLOR + +/* Define if you have the 'term_attrs' function. */ +#undef _Py_HAVE_CURSES_TERM_ATTRS + +/* Define if you have the 'use_screen' function. */ +#undef _Py_HAVE_CURSES_USE_SCREEN + +/* Define if you have the 'use_window' function. */ +#undef _Py_HAVE_CURSES_USE_WINDOW + +/* Define if you have the 'wattr_get' function. */ +#undef _Py_HAVE_CURSES_WATTR_GET + +/* Define if you have the 'wattr_off' function. */ +#undef _Py_HAVE_CURSES_WATTR_OFF + +/* Define if you have the 'wattr_on' function. */ +#undef _Py_HAVE_CURSES_WATTR_ON + +/* Define if you have the 'wattr_set' function. */ +#undef _Py_HAVE_CURSES_WATTR_SET + +/* Define if you have the 'wcolor_set' function. */ +#undef _Py_HAVE_CURSES_WCOLOR_SET + /* Define if compiler supports __builtin_shufflevector with 128-bit vectors AND the target architecture has native SIMD (not just API availability) */ #undef _Py_HAVE_EFFICIENT_BUILTIN_SHUFFLEVECTOR @@ -2151,19 +2127,43 @@ /* Define if you have the 'freadlink' function. */ #undef _Py_HAVE_FREADLINK +/* Define if you have the 'getdents64' function. */ +#undef _Py_HAVE_GETDENTS64 + +/* Define if you have the 'gettid' function. */ +#undef _Py_HAVE_GETTID + /* Define if you have a working iconv() function. */ #undef _Py_HAVE_ICONV +/* Define if you have the 'if_indextoname' function. */ +#undef _Py_HAVE_IF_INDEXTONAME + +/* Define if you have the 'if_nametoindex' function. */ +#undef _Py_HAVE_IF_NAMETOINDEX + /* Define to 1 if libgcc __register_frame and __deregister_frame are linkable. */ #undef _Py_HAVE_LIBGCC_EH_FRAME_REGISTRATION +/* Define if you have the 'pidfd_getfd' function. */ +#undef _Py_HAVE_PIDFD_GETFD + +/* Define if you have the 'pidfd_open' function. */ +#undef _Py_HAVE_PIDFD_OPEN + +/* Define if you have the 'pidfd_send_signal' function. */ +#undef _Py_HAVE_PIDFD_SEND_SIGNAL + /* Define if you have the 'PR_SET_VMA_ANON_NAME' constant. */ #undef _Py_HAVE_PR_SET_VMA_ANON_NAME /* Define if you have the 'pthread_attr_get_np' function. */ #undef _Py_HAVE_PTHREAD_ATTR_GET_NP +/* Define if you have readline 6.3 */ +#undef _Py_HAVE_RL_CHANGE_ENVIRONMENT + /* Define if you have the 'sinpi' function. */ #undef _Py_HAVE_SINPI