You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Install a static installation description file as part of the Python installation #107956
Ship a static file that describes various aspects of the installation as part of the Python installation.
UPDATE: Proposed as PEP 739, which has been accepted.
Pitch
Shipping such a file would make it much easier to introspect a Python installation without having to run the interpreter. There are many use-cases for this, but some key would be for eg. Python launchers, and cross-compilation tooling.
But I agree with the overall goal of providing the actual info needed for building and installing extensions. We want the new fields to be the actual commands, not just the ones that were used to build CPython. This probably ties into #108064 to document these commands.
I'm not entirely sure I understand the distinction you are making between "the actual commands" and "the ones that were used to build CPython".
The commands used for building CPython are actual real commands, they contain for example the name of a specific compiler. That's actually problematic because build systems such as meson, cmake, autotools, (and to a small extent that I wish were a lot larger, setuptools) etc. cannot accept "actual commands" from sysconfig since they need to be able to use different commands, and sometimes compile different languages that aren't used in the CPython build system at all (such as mingw Fortran on Windows, which is a GCC component that, yes, is getting mixed with MSVC-built pythons).
Any proposed API for CPython that wants to replace current usage of sysconfig.get_config_vars as the preferred information source used by build systems, cannot hardcode the name of a compiler to use. If it does hardcode the name of a compiler to use then we (meson, in this case) will simply ignore that new API and keep using sysconfig.get_config_vars with a hacked up heuristic -- our current one to find the correct library / import library is, depending on Unix/windows and also CPython / PyPy, a mixture of templating the hardcoded word python /pypy3-c, the version number, DEBUG_EXT, ABIFLAGS, py_version_nodot, etc.
Basically, what we would like to see in order to make our jobs as buildsystem developers easier, is a way to do for Windows what pkg-config --cflags --libs python3 does on Linux and macOS. (And it should of course work cross platform, so we can use it on Unix platforms as a fallback when pkg-config isn't installed.)
We don't really want to know the commands to use -- we have our own commands that we're probably already using to build (embedded copies of?) regular C/C++/Fortran libraries into static libraries that will then get linked with a python binding file and the python import library. We need the flexibility to choose our own toolchain, because to us, libpython is just "yet another dependency" irrespective of the fact that the libpython dependency happens to parallel the runtime we are building plugins for.
If people want to document and know an example command capable of being copy-pasted and run to build an extension without resorting to a build system like meson, cmake, or autotools... then IMHO this is best as documentation, not as part of a stdlib API. A stdlib API should only contain the python-specific parts that would be used by the documentation. It's up to the documentation to then explain how to find/choose/activate/run a compiler.
And yeah, by "commands" I really just meant the compile-time options required, in some format that a library can figure out which actual options to use with its own compiler.
Thanks for clarifying this. I appear to have missed that response while writing my own, lol...
I think a large part of my confusion was that I typically think of, say, cl.exe as a command, and things like header search directories or import libraries as flags. Different terminologies I guess.
We're more in agreement than you think 😉 I'm just stating things very simply compared to the depth that you've thought about them, and I think you're assuming that I've also thought these through 110% and am making definitive design statements. I'm not - I'm just standing somewhere that I can see the problem and generally waving in their direction (I should add, I'm confident in that direction, I'm just not spelling out as full an implementation plan as you are).
Gotcha. I wasn't assuming that, but wasn't sure the tone you were going for (yay nonverbal communication). For me it's easier to spell everything out, to make sure we are on the same page and there isn't any misunderstanding.
I definitely agree that not all the sysconfig data is useful.1 Even so, for practicality, having a deprecated-on-release subkey of the new data that is literally "what sysconfig would show, for better or worse" makes it much easier for devs to migrate to the new data. Once they're there, they can start using the new data when it's available (this also means a distributor can generate a "sysconfig-only" version of the data for older releases if they want). Without this, devs who might need it won't switch, because they don't have a nice fallback for the older versions of Python they support. Consider, they might read the file and still need to launch Python to introspect itself - we ought to be able to avoid that on first release.
If the only thing people are missing is access to the sysconfig data, I am personally okay with requiring them to launch an interpreter during migration. From the cross-builds side, the data can simply be changed by setting _PYTHON_SYSCONFIGDATA_NAME, the key thing for this use-case would be that people stop introspecting other data.
Agreed that replacing sysconfig isn't the goal, but it is a good test for whether we've provided enough information. Right now, the gaps in this area are filled by people using sysconfig and often making assumptions about how Python is normally installed - if we can't replace all of those with this file (plus some runtime calculation, of course), I think we've missed the mark. People shouldn't need to use sysconfig if we get this right, whether they can launch the interpreter or not.
I agree. Though, we shouldn't forget that this file isn't targeting all the use-cases covered by sysconfig (eg. the install layout), for now at least.
And yeah, by "commands" I really just meant the compile-time options required, in some format that a library can figure out which actual options to use with its own compiler. That doc page ought to end up with specific compiler commands as examples, but we wouldn't put those in here. However, there should be enough information in this file that a program/script can figure out how to match the original compiler settings enough to get a compatible extension module. Again, if we can't provide that, I think we've missed our goal, so this is more of a validation test than a specific feature.
That makes more sense, we're on the same page then.
Any proposed API for CPython that wants to replace current usage of sysconfig.get_config_vars as the preferred information source used by build systems, cannot hardcode the name of a compiler to use. If it does hardcode the name of a compiler to use then we (meson, in this case) will simply ignore that new API and keep using sysconfig.get_config_vars with a hacked up heuristic -- our current one to find the correct library / import library is, depending on Unix/windows and also CPython / PyPy, a mixture of templating the hardcoded word python /pypy3-c, the version number, DEBUG_EXT, ABIFLAGS, py_version_nodot, etc.
My goal, as I also stated above, is to provide all the compiler-related information in a compiler-agnostic way. Downstream users, like Meson, should be able to take that information and give it to whichever compiler backend it wants to use and have that be able to generate the actual compiler-specific commands. This should be a 1st party supported use-case.
From the cross-builds side, the data can simply be changed by setting _PYTHON_SYSCONFIGDATA_NAME, the key thing for this use-case would be that people stop introspecting other data.
This doesn't work on Windows - no file is generated. But we still may need to get fields for a non-executable runtime (such as an ARM64 build on an x64 machine).
we shouldn't forget that this file isn't targeting all the use-cases covered by sysconfig (eg. the install layout)
I assume by "install layout" you mean how to install wheels? It ought to cover the locations where CPython itself installed things to, such as its own headers and libs, right? (Those are "compiler-related information," I guess.)
I assume by "install layout" you mean how to install wheels? It ought to cover the locations where CPython itself installed things to, such as its own headers and libs, right? (Those are "compiler-related information," I guess.)
Well, the install scheme really includes both what wheels typically use as well as a bit more -- it's actually legal to install headers in a wheel, which maps to the same location as the CPython headers in "include" / "platinclude". I don't think that wheels can do anything with the stdlib location though... which is on that
The install layout from sysconfig doesn't currently say anything about the directory where libpython itself is though -- maybe it should? 🤔 People keep on wanting to package up C/++ libraries via tools like auditwheel repair / delvewheel and it's a big pain to do cross platform what with adding the DLL directory, or embedding private paths, having a single conventional directory that python itself guarantees is in the DLL search path for all modules could be handy...
it's actually legal to install headers in a wheel, which maps to the same location as the CPython headers in "include" / "platinclude".
Legal, but unspecified, and not portable. The current include and platinclude in sysconfig are where to find CPython's headers, there's no guarantee you can install anything there.
The plan right now is not to specify those. So we'd include the actual directory where CPython stuff is installed to, and anyone who interprets that as "I can also install stuff here" is going off-label.
@FFY00 clarified that the location would be an implementation detail, though they'd like to make an effort to keep it consistent when possible (discussion)
Outcome: As discussed above, this should probably go to an installation-specific directory. stdlib was the only proposed location that meets this criterion, and no-one raised an issue against using stdlib specifically, so AFAICT this seems to be the best choice.
Note: In the future, we can also place the file in another place, if for example we want to improve discoverability (discussion, discussion)
Outcome: I don't think there is a clear choice here, but TOML and JSON seem to be the stronger options. It seems TOML is the preferred option when looking at the format alone, but it raises concerns regarding the implementation and technical viability. JSON, on the other hand, does not provide any technical challenges, but it's less desirable as a format.
File name
(the extension isn't very relevant here, as that depends on the file format)
Outcome: There hasn't been much discussion about the file name. install-details.toml seems to be the leading option, although the lack of discussion does make it a very strong candidate.
Sorry if missed anything, misunderstood anything, or if there was any bias. Please let me know what you think, and if the proposed outcomes seem correct.
Also, this is my first time writing a discussion summary of this kind, so I am sorry if you feel misrepresented in any way. Please let me know if that is the case, so that I can try to prevent it in the future and try to improve.
It's a good summary. The only bias I feel is there is (apparently/weakly) dismissing my proposals because nobody else is talking about them 😆
Some responses that I started posting on the PR, but make more sense here:
though all major languages I am aware of have a TOML parser library
As far as I know, neither Bash nor PowerShell have anything built in, which means you can't write a native script to handle it. (PowerShell definitely has JSON, and I'm not sure about Bash, but I bet that a command line tool like jq is far more common than... stoml? I don't know what the standard tool would be here.)
.NET certainly has nothing native, though adding additional dependencies is only sometimes an issue. But when it is, writing a basic parser is far more likely than jumping through whatever hoops are necessary to get one.
LTS versions of Python likely to be on existing Linux distros don't have it, which means system scripts or tools on those will need an additional dependency to handle the file. Again, not impossible, but potentially complicated enough that people will reach for str.partition('=') instead and hope.
If we support different data types [in the file], the user implementation [of a parser] wouldn't be trivial
What different types do we need? If certain fields are defined as being int or float, it's easy enough for any language to parse those. But I expect most fields are going to be strings, and the escaping rules for anything other than key=value are going to make editing by hand just as complicated as parsing. The biggest advantage of key=value is that the final string is literally as it's read, and the biggest restriction is that we can't embed newlines trivially.1
Footnotes
My vote here would be for an empty key to indicate that the value should be appended after a newline to the previous key. Trailing backslashes are very likely to occur in paths on Windows, so those aren't a good option here IMHO. ↩
I'd like to raise an additional defense of using json over toml:
@brettcannon thinks that if users are meant to ever read or write the file, TOML can make sense, but if not, JSON would be better (discussion)
It sounds to me like there's no real concern that users are meant to write the file, just read it:
Yes, I expect people to read, and in some situations, write this file, but not too frequently. Some use-cases:
Manually introspecting an installation without running the interpreter, which is the most relevant when working with cross-builds, but certainly not limited to it
This plays a bit with my outer goal of trying to make it possible to build most extensions, etc. without having to run the interpreter, where in certain scenarios, I think users might need to write the file themselves (eg. when it's not available — I expand into this below)
I do hear the rationale for reading the file, mainly for debugging -- in general, data formats that a human can read somehow, are beneficial absent compelling need otherwise. But I'm not sure what the rationale for writing one is.
That there isn't one already existing? Why is this a reason to write one? I would think it's a reason to raise a request for your python binary distributor to add one. I doubt software will be dropping support for running against a python that doesn't have this file, which means old pythons are covered... and new pythons should have the file, right?
...
If it's only interesting to read the file, not to write it, then I still think json is a good choice. The main problem with json is that it's annoying to write correctly (in particular, the requirement to separate elements in a list with commas, but raise a syntax error if the last element has a non-ambiguous trailing comma).
Reading it is mostly easy, you just pretty-print it when originally generating the json file. Optionally, if you want comments, you have to "cheat" and hack those in by creating json entries called "__comment": "this is a comment" which isn't very thrilling, but on the other hand I usually feel like I don't really need comments in a json file except when I am adding notes for the next person to edit the file (assuming that said json file is intended to be edited by hand, instead of generated).
As far as I know, neither Bash nor PowerShell have anything built in, which means you can't write a native script to handle it. (PowerShell definitely has JSON, and I'm not sure about Bash, but I bet that a command line tool like jq is far more common than... stoml? I don't know what the standard tool would be here.)
jq is very common, yes. Every time I've ever wanted to parse a toml file from a shell though, I ended up finding one of like 8 different programs all called "toml2json", then passing that to jq.
@AA-Turner seems to agree with @FFY00, but points out that using JSON would make the implementation easier
I'd be +1 TOML if we had a writer in the stdlib. I'm maybe +0 though currently, as the TOML implementation in the PR seems a little fragile currently. JSON does have the benefits Steve mentioned for PS/bash, etc, though.
Activity
eli-schwartz commentedon Aug 25, 2023
I'm not entirely sure I understand the distinction you are making between "the actual commands" and "the ones that were used to build CPython".
The commands used for building CPython are actual real commands, they contain for example the name of a specific compiler. That's actually problematic because build systems such as meson, cmake, autotools, (and to a small extent that I wish were a lot larger, setuptools) etc. cannot accept "actual commands" from sysconfig since they need to be able to use different commands, and sometimes compile different languages that aren't used in the CPython build system at all (such as mingw Fortran on Windows, which is a GCC component that, yes, is getting mixed with MSVC-built pythons).
Any proposed API for CPython that wants to replace current usage of sysconfig.get_config_vars as the preferred information source used by build systems, cannot hardcode the name of a compiler to use. If it does hardcode the name of a compiler to use then we (meson, in this case) will simply ignore that new API and keep using sysconfig.get_config_vars with a hacked up heuristic -- our current one to find the correct library / import library is, depending on Unix/windows and also CPython / PyPy, a mixture of templating the hardcoded word
python/pypy3-c, the version number, DEBUG_EXT, ABIFLAGS, py_version_nodot, etc.Basically, what we would like to see in order to make our jobs as buildsystem developers easier, is a way to do for Windows what
pkg-config --cflags --libs python3does on Linux and macOS. (And it should of course work cross platform, so we can use it on Unix platforms as a fallback when pkg-config isn't installed.)We don't really want to know the commands to use -- we have our own commands that we're probably already using to build (embedded copies of?) regular C/C++/Fortran libraries into static libraries that will then get linked with a python binding file and the python import library. We need the flexibility to choose our own toolchain, because to us, libpython is just "yet another dependency" irrespective of the fact that the libpython dependency happens to parallel the runtime we are building plugins for.
If people want to document and know an example command capable of being copy-pasted and run to build an extension without resorting to a build system like meson, cmake, or autotools... then IMHO this is best as documentation, not as part of a stdlib API. A stdlib API should only contain the python-specific parts that would be used by the documentation. It's up to the documentation to then explain how to find/choose/activate/run a compiler.
eli-schwartz commentedon Aug 25, 2023
Thanks for clarifying this. I appear to have missed that response while writing my own, lol...
I think a large part of my confusion was that I typically think of, say, cl.exe as a command, and things like header search directories or import libraries as flags. Different terminologies I guess.
FFY00 commentedon Sep 1, 2023
Gotcha. I wasn't assuming that, but wasn't sure the tone you were going for (yay nonverbal communication). For me it's easier to spell everything out, to make sure we are on the same page and there isn't any misunderstanding.
If the only thing people are missing is access to the
sysconfigdata, I am personally okay with requiring them to launch an interpreter during migration. From the cross-builds side, the data can simply be changed by setting_PYTHON_SYSCONFIGDATA_NAME, the key thing for this use-case would be that people stop introspecting other data.I agree. Though, we shouldn't forget that this file isn't targeting all the use-cases covered by sysconfig (eg. the install layout), for now at least.
That makes more sense, we're on the same page then.
My goal, as I also stated above, is to provide all the compiler-related information in a compiler-agnostic way. Downstream users, like Meson, should be able to take that information and give it to whichever compiler backend it wants to use and have that be able to generate the actual compiler-specific commands. This should be a 1st party supported use-case.
zooba commentedon Sep 1, 2023
This doesn't work on Windows - no file is generated. But we still may need to get fields for a non-executable runtime (such as an ARM64 build on an x64 machine).
I assume by "install layout" you mean how to install wheels? It ought to cover the locations where CPython itself installed things to, such as its own headers and libs, right? (Those are "compiler-related information," I guess.)
Other than that, agreed with all the rest.
eli-schwartz commentedon Sep 1, 2023
Well, the install scheme really includes both what wheels typically use as well as a bit more -- it's actually legal to install headers in a wheel, which maps to the same location as the CPython headers in "include" / "platinclude". I don't think that wheels can do anything with the stdlib location though... which is on that
The install layout from sysconfig doesn't currently say anything about the directory where libpython itself is though -- maybe it should? 🤔 People keep on wanting to package up C/++ libraries via tools like auditwheel repair / delvewheel and it's a big pain to do cross platform what with adding the DLL directory, or embedding private paths, having a single conventional directory that python itself guarantees is in the DLL search path for all modules could be handy...
zooba commentedon Sep 5, 2023
Legal, but unspecified, and not portable. The current
includeandplatincludeinsysconfigare where to find CPython's headers, there's no guarantee you can install anything there.The plan right now is not to specify those. So we'd include the actual directory where CPython stuff is installed to, and anyone who interprets that as "I can also install stuff here" is going off-label.
FFY00 commentedon Sep 5, 2023
Okay, to move things forward, let's set some of the base implementation details, so that GH-108483 is unblocked.
I am gonna try to summarize the discussions so far, and what I think to be the most reasonable outcome for each topic.
stdlib(discussion)sys.prefix(discussion)stdlibwas the only proposed location that meets this criterion, and no-one raised an issue against usingstdlibspecifically, so AFAICT this seems to be the best choice.key=valueformat to TOML (discussion)key=valueformat (discussion)(the extension isn't very relevant here, as that depends on the file format)
sysconfig.json(discussion)sys.prefixas the file locationsysconfig, so doesn't think "sysconfig" is a good option (discussion)install-details.toml(discussion)install-details.tomlseems to be the leading option, although the lack of discussion does make it a very strong candidate.Sorry if missed anything, misunderstood anything, or if there was any bias. Please let me know what you think, and if the proposed outcomes seem correct.
Also, this is my first time writing a discussion summary of this kind, so I am sorry if you feel misrepresented in any way. Please let me know if that is the case, so that I can try to prevent it in the future and try to improve.
zooba commentedon Sep 5, 2023
It's a good summary. The only bias I feel is there is (apparently/weakly) dismissing my proposals because nobody else is talking about them 😆
Some responses that I started posting on the PR, but make more sense here:
As far as I know, neither Bash nor PowerShell have anything built in, which means you can't write a native script to handle it. (PowerShell definitely has JSON, and I'm not sure about Bash, but I bet that a command line tool like
jqis far more common than...stoml? I don't know what the standard tool would be here.).NET certainly has nothing native, though adding additional dependencies is only sometimes an issue. But when it is, writing a basic parser is far more likely than jumping through whatever hoops are necessary to get one.
LTS versions of Python likely to be on existing Linux distros don't have it, which means system scripts or tools on those will need an additional dependency to handle the file. Again, not impossible, but potentially complicated enough that people will reach for
str.partition('=')instead and hope.What different types do we need? If certain fields are defined as being int or float, it's easy enough for any language to parse those. But I expect most fields are going to be strings, and the escaping rules for anything other than
key=valueare going to make editing by hand just as complicated as parsing. The biggest advantage ofkey=valueis that the final string is literally as it's read, and the biggest restriction is that we can't embed newlines trivially.1Footnotes
My vote here would be for an empty key to indicate that the value should be appended after a newline to the previous key. Trailing backslashes are very likely to occur in paths on Windows, so those aren't a good option here IMHO. ↩
eli-schwartz commentedon Sep 5, 2023
I'd like to raise an additional defense of using json over toml:
It sounds to me like there's no real concern that users are meant to write the file, just read it:
I do hear the rationale for reading the file, mainly for debugging -- in general, data formats that a human can read somehow, are beneficial absent compelling need otherwise. But I'm not sure what the rationale for writing one is.
That there isn't one already existing? Why is this a reason to write one? I would think it's a reason to raise a request for your python binary distributor to add one. I doubt software will be dropping support for running against a python that doesn't have this file, which means old pythons are covered... and new pythons should have the file, right?
...
If it's only interesting to read the file, not to write it, then I still think json is a good choice. The main problem with json is that it's annoying to write correctly (in particular, the requirement to separate elements in a list with commas, but raise a syntax error if the last element has a non-ambiguous trailing comma).
Reading it is mostly easy, you just pretty-print it when originally generating the json file. Optionally, if you want comments, you have to "cheat" and hack those in by creating json entries called
"__comment": "this is a comment"which isn't very thrilling, but on the other hand I usually feel like I don't really need comments in a json file except when I am adding notes for the next person to edit the file (assuming that said json file is intended to be edited by hand, instead of generated).jq is very common, yes. Every time I've ever wanted to parse a toml file from a shell though, I ended up finding one of like 8 different programs all called "toml2json", then passing that to jq.
AA-Turner commentedon Sep 5, 2023
I'd be +1 TOML if we had a writer in the stdlib. I'm maybe +0 though currently, as the TOML implementation in the PR seems a little fragile currently. JSON does have the benefits Steve mentioned for PS/bash, etc, though.
A
encukou commentedon Oct 11, 2023
FWIW, feature flags that affect the stable ABI are currently:
pythonGH-107956: install a static description file
pythonGH-107956: install build-details.json (PEP 739)
GH-107956: install build-details.json (PEP 739) (#130069)
FFY00 commentedon Mar 28, 2025
We can close this now 😊
zooba commentedon Mar 31, 2025
Has Windows support been added somewhere? What's the plan there?
pythonGH-107956: add sysconfig._build_details
pythonGH-107956: add sysconfig._build_details
pythonGH-107956: add sysconfig._build_details
pythonGH-107956: add sysconfig._build_details
pythonGH-107956: add sysconfig._build_details
Fix CI: migrate BuildDetailsLibpythonTests to sysconfig._build_detail…
Fix CI: migrate BuildDetailsLibpythonTests to sysconfig._build_detail…
pythonGH-107956: add sysconfig._build_details