You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I just learned of this issue. Rather than adding has_netloc, etc. attributes, why not use None to distinguish missing values as is preferred above, but add a new boolean keyword argument to urlparse(), etc. to get the new behavior (e.g. "allow_none" to parallel "allow_fragments")?
It seems like this would be more elegant, IMO, because it would lead to the API we really want. For example, the ParseResult(), etc. signatures and repr() values would be simpler. Changing the default value of the new keyword arguments would also provide a clean and simple deprecation pathway in the future, if desired.
I like this option. I suppose choosing which option to take is a compromise between compatiblity and simplicity. In the short term, the “allows_none” option requires user code to be updated. In the long term it may break compatibility. But the “has_netloc” etc option is more complex.
One thing against the 'has_netloc' etc. solution is that
while it guarantees round-trips (urlunsplit(urlsplit('...')) etc.),
it is conditional on 'urlunsplit' getting 'SplitResult' object.
When 'urlunsplit' gets a normal tuple, it is helpless.
I opened issue #99962 discussing alternative solutions based on a new representation (instead of the empty string) for undefined components.
But after learning about this issue, I think that @cjerdonek’s solution using a None representation for undefined components with a new flag parameter (allow_none) in the parsing functions (urldefrag(), urlparse(), and urlsplit()) is the best solution, as it is the simplest representation for undefined components, it is backward compatible with the flag parameter defaulting to the legacy behaviour, and it does not create yet another parsing function.
@cjerdonek’s solution is not unprecedented in the standard library. For instance @erlend-aasland has recently chosen a similar approach in PR #93823 for implementing the new PEP-249 manual commit mode in the sqlite3 module, by introducing a new autocommit parameter in the connect() function which changes the behaviour of the Connection.__init__(), Connection.execute(), Connection.executemany(), Connection.executescript(), Connection.commit(), Connection.rollback(), and Connection.close() methods.
Add a boolean option allow_none in urlparse(), urlsplit() and urldefrag() -- if it is true, then non defined components are represented as None instead of an empty string.
Add a boolean option keep_empty in urlunparse(), urlunsplit() and the geturl() method in all six result classes -- if it is true, keep empty non-None components in the resulting string.
Now you can call a parsing function with allow_none=True to get unambiguous result and then call an unparsing function with keep_empty=True to preserve the original empty components or with keep_empty=False (default) to strip empty components. Calling with keep_empty=True after allow_none=False is an error -- it produces result with added ";?#" if corresponding components were absent in the original.
But preserving the original representation is more common than stripping empty components, so in most cases we want to use keep_empty=True after allow_none=True. It makes sense to make this default in the geturl() method, since we know whether a named tuple is the result of the call with allow_none=True.
So we need to encode the value of allow_none in the nametuple results. There are two ways:
Add an attribute. It should not be a part of the tuple for backward compatibility. (A keyword-only parameter can be added in the constructor, although this complicates the things. We need to override _replace() and __replace__() to preserve that attribute in the copies.
Add different classes for allow_none=True. Currently There are 6 result classes (DefragResult,
SplitResult, ParseResult and the corresponding bytes counterparts), not including bytes classes and mixins, this amount can be simply doubled.
I hesitate before choosing a way.
Alternative to boolean options is addition of a new set of parsing and unparsing functions. newurlparse(string) is shorter than urlparse(string, allow_none=True). Old functions could be deprecated and removed in future.
Just to provide a real-world example where this matters, I have a test init script that creates a test database alongside a regular database and resets its state, based on an environment variable.
The code looks like:
db=urlparse(os.environ["DATABASE_URL"]
systemdb=urlunparse(db._replace(path="/postgres"))
subprocess.run(["psql", systemdb, "-c", "CREATE DATABASE IF NOT EXISTS test"])
testdb=urlunparse(db._replace(path="/test")
One of our devs uses postgres://localhost/appdb as their connection string, and everything works fine for them. Another uses postgres:///appdb as their connection string, and the script fails because systemdb is set to postgres:/postgres and testdb is set to postgres:/test instead of postgres:///postgres and postgres:///test.
Note that python does the right thing for file URLs, but this seems to be coded as an exception rather than falling out of the standard rules of URL parsing.
Activity
rbtcollins commentedon Aug 2, 2015
See also bpo-6631
cjerdonek commentedon Jul 31, 2018
I just learned of this issue. Rather than adding has_netloc, etc. attributes, why not use None to distinguish missing values as is preferred above, but add a new boolean keyword argument to urlparse(), etc. to get the new behavior (e.g. "allow_none" to parallel "allow_fragments")?
It seems like this would be more elegant, IMO, because it would lead to the API we really want. For example, the ParseResult(), etc. signatures and repr() values would be simpler. Changing the default value of the new keyword arguments would also provide a clean and simple deprecation pathway in the future, if desired.
vadmium commentedon Aug 4, 2018
I like this option. I suppose choosing which option to take is a compromise between compatiblity and simplicity. In the short term, the “allows_none” option requires user code to be updated. In the long term it may break compatibility. But the “has_netloc” etc option is more complex.
openandclose commentedon Jun 10, 2020
I found another related issue (bpo-37969).
I also filed one myself (bpo-40938).
---
One thing against the 'has_netloc' etc. solution is that
while it guarantees round-trips (urlunsplit(urlsplit('...')) etc.),
it is conditional on 'urlunsplit' getting 'SplitResult' object.
When 'urlunsplit' gets a normal tuple, it is helpless.
merwok commentedon Feb 12, 2022
See also bpo-46337
geryogam commentedon Dec 3, 2022
I opened issue #99962 discussing alternative solutions based on a new representation (instead of the empty string) for undefined components.
But after learning about this issue, I think that @cjerdonek’s solution using a
Nonerepresentation for undefined components with a new flag parameter (allow_none) in the parsing functions (urldefrag(),urlparse(), andurlsplit()) is the best solution, as it is the simplest representation for undefined components, it is backward compatible with the flag parameter defaulting to the legacy behaviour, and it does not create yet another parsing function.@cjerdonek’s solution is not unprecedented in the standard library. For instance @erlend-aasland has recently chosen a similar approach in PR #93823 for implementing the new PEP-249 manual commit mode in the
sqlite3module, by introducing a new autocommit parameter in theconnect()function which changes the behaviour of theConnection.__init__(),Connection.execute(),Connection.executemany(),Connection.executescript(),Connection.commit(),Connection.rollback(), andConnection.close()methods.serhiy-storchaka commentedon Aug 25, 2024
I tried to implement the @cjerdonek's idea.
urlparse(),urlsplit()andurldefrag()-- if it is true, then non defined components are represented as None instead of an empty string.urlunparse(),urlunsplit()and thegeturl()method in all six result classes -- if it is true, keep empty non-None components in the resulting string.Now you can call a parsing function with
allow_none=Trueto get unambiguous result and then call an unparsing function withkeep_empty=Trueto preserve the original empty components or withkeep_empty=False(default) to strip empty components. Calling withkeep_empty=Trueafterallow_none=Falseis an error -- it produces result with added ";?#" if corresponding components were absent in the original.But preserving the original representation is more common than stripping empty components, so in most cases we want to use
keep_empty=Trueafterallow_none=True. It makes sense to make this default in thegeturl()method, since we know whether a named tuple is the result of the call withallow_none=True.So we need to encode the value of
allow_nonein the nametuple results. There are two ways:_replace()and__replace__()to preserve that attribute in the copies.allow_none=True. Currently There are 6 result classes (DefragResult,SplitResult, ParseResult and the corresponding bytes counterparts), not including bytes classes and mixins, this amount can be simply doubled.
I hesitate before choosing a way.
Alternative to boolean options is addition of a new set of parsing and unparsing functions.
newurlparse(string)is shorter thanurlparse(string, allow_none=True). Old functions could be deprecated and removed in future.pythongh-67041: Allow to distinguish between empty and not defined UR…
pythongh-67041: Allow to distinguish between empty and not defined UR…
jcdyer commentedon Nov 14, 2025
Just to provide a real-world example where this matters, I have a test init script that creates a test database alongside a regular database and resets its state, based on an environment variable.
The code looks like:
One of our devs uses
postgres://localhost/appdbas their connection string, and everything works fine for them. Another usespostgres:///appdbas their connection string, and the script fails becausesystemdbis set topostgres:/postgresandtestdbis set topostgres:/testinstead ofpostgres:///postgresandpostgres:///test.Note that python does the right thing for file URLs, but this seems to be coded as an exception rather than falling out of the standard rules of URL parsing.
gh-67041: Allow to distinguish between empty and not defined URI comp…
pythongh-67041: Allow to distinguish between empty and not defined UR…