Skip to content

urllib.parse wrongly strips empty #fragment, ?query, //netloc #67041

Closed

Description

@soilandreyes
BPO 22852
Nosy @orsenthil, @rbtcollins, @merwok, @cjerdonek, @berkerpeksag, @vadmium, @demianbrecht, @openandclose
Files
  • has_netloc.patch
  • has_netloc.v2.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2014-11-12.10:23:30.997>
    labels = ['type-feature', 'library', '3.11']
    title = 'urllib.parse wrongly strips empty #fragment, ?query, //netloc'
    updated_at = <Date 2022-02-12.11:48:41.130>
    user = 'https://bugs.python.org/soilandreyes'

    bugs.python.org fields:

    activity = <Date 2022-02-12.11:48:41.130>
    actor = 'eric.araujo'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['Library (Lib)']
    creation = <Date 2014-11-12.10:23:30.997>
    creator = 'soilandreyes'
    dependencies = []
    files = ['38465', '38633']
    hgrepos = ['279']
    issue_num = 22852
    keywords = ['patch', 'needs review']
    message_count = 17.0
    messages = ['231070', '231099', '235582', '237999', '238220', '238246', '238251', '238255', '238258', '238264', '238897', '244516', '247905', '322778', '323123', '371181', '413124']
    nosy_count = 10.0
    nosy_names = ['orsenthil', 'rbcollins', 'eric.araujo', 'chris.jerdonek', 'berker.peksag', 'martin.panter', 'piotr.dobrogost', 'demian.brecht', 'soilandreyes', 'op368']
    pr_nums = []
    priority = 'normal'
    resolution = None
    stage = 'patch review'
    status = 'open'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue22852'
    versions = ['Python 3.11']

    Linked PRs

    Activity

    1. rbtcollins commented on Aug 2, 2015

      @rbtcollins
      Member

      See also bpo-6631

    2. cjerdonek commented on Jul 31, 2018

      @cjerdonek
      Member

      I just learned of this issue. Rather than adding has_netloc, etc. attributes, why not use None to distinguish missing values as is preferred above, but add a new boolean keyword argument to urlparse(), etc. to get the new behavior (e.g. "allow_none" to parallel "allow_fragments")?

      It seems like this would be more elegant, IMO, because it would lead to the API we really want. For example, the ParseResult(), etc. signatures and repr() values would be simpler. Changing the default value of the new keyword arguments would also provide a clean and simple deprecation pathway in the future, if desired.

    3. vadmium commented on Aug 4, 2018

      @vadmium
      Member

      I like this option. I suppose choosing which option to take is a compromise between compatiblity and simplicity. In the short term, the “allows_none” option requires user code to be updated. In the long term it may break compatibility. But the “has_netloc” etc option is more complex.

    4. openandclose commented on Jun 10, 2020

      openandclosemannequin
      Mannequin

      I found another related issue (bpo-37969).

      I also filed one myself (bpo-40938).

      ---

      One thing against the 'has_netloc' etc. solution is that
      while it guarantees round-trips (urlunsplit(urlsplit('...')) etc.),
      it is conditional on 'urlunsplit' getting 'SplitResult' object.
      When 'urlunsplit' gets a normal tuple, it is helpless.

    5. merwok commented on Feb 12, 2022

      @merwok
      Member

      See also bpo-46337

    6. transferred this issue fromon Apr 10, 2022
    7. geryogam commented on Dec 3, 2022

      @geryogam
      Contributor

      I opened issue #99962 discussing alternative solutions based on a new representation (instead of the empty string) for undefined components.

      But after learning about this issue, I think that @cjerdonek’s solution using a None representation for undefined components with a new flag parameter (allow_none) in the parsing functions (urldefrag(), urlparse(), and urlsplit()) is the best solution, as it is the simplest representation for undefined components, it is backward compatible with the flag parameter defaulting to the legacy behaviour, and it does not create yet another parsing function.

      @cjerdonek’s solution is not unprecedented in the standard library. For instance @erlend-aasland has recently chosen a similar approach in PR #93823 for implementing the new PEP-249 manual commit mode in the sqlite3 module, by introducing a new autocommit parameter in the connect() function which changes the behaviour of the Connection.__init__(), Connection.execute(), Connection.executemany(), Connection.executescript(), Connection.commit(), Connection.rollback(), and Connection.close() methods.

    8. serhiy-storchaka commented on Aug 25, 2024

      @serhiy-storchaka
      Member

      I tried to implement the @cjerdonek's idea.

      • Add a boolean option allow_none in urlparse(), urlsplit() and urldefrag() -- if it is true, then non defined components are represented as None instead of an empty string.
      • Add a boolean option keep_empty in urlunparse(), urlunsplit() and the geturl() method in all six result classes -- if it is true, keep empty non-None components in the resulting string.

      Now you can call a parsing function with allow_none=True to get unambiguous result and then call an unparsing function with keep_empty=True to preserve the original empty components or with keep_empty=False (default) to strip empty components. Calling with keep_empty=True after allow_none=False is an error -- it produces result with added ";?#" if corresponding components were absent in the original.

      But preserving the original representation is more common than stripping empty components, so in most cases we want to use keep_empty=True after allow_none=True. It makes sense to make this default in the geturl() method, since we know whether a named tuple is the result of the call with allow_none=True.

      So we need to encode the value of allow_none in the nametuple results. There are two ways:

      • Add an attribute. It should not be a part of the tuple for backward compatibility. (A keyword-only parameter can be added in the constructor, although this complicates the things. We need to override _replace() and __replace__() to preserve that attribute in the copies.
      • Add different classes for allow_none=True. Currently There are 6 result classes (DefragResult,
        SplitResult, ParseResult and the corresponding bytes counterparts), not including bytes classes and mixins, this amount can be simply doubled.

      I hesitate before choosing a way.

      Alternative to boolean options is addition of a new set of parsing and unparsing functions. newurlparse(string) is shorter than urlparse(string, allow_none=True). Old functions could be deprecated and removed in future.

    9. added a commit that references this issue on Aug 25, 2024
    10. added a commit that references this issue on Aug 31, 2024
    11. jcdyer commented on Nov 14, 2025

      @jcdyer

      Just to provide a real-world example where this matters, I have a test init script that creates a test database alongside a regular database and resets its state, based on an environment variable.

      The code looks like:

      db = urlparse(os.environ["DATABASE_URL"]
      systemdb = urlunparse(db._replace(path="/postgres"))
      subprocess.run(["psql", systemdb, "-c", "CREATE DATABASE IF NOT EXISTS test"])
      testdb = urlunparse(db._replace(path="/test")

      One of our devs uses postgres://localhost/appdb as their connection string, and everything works fine for them. Another uses postgres:///appdb as their connection string, and the script fails because systemdb is set to postgres:/postgres and testdb is set to postgres:/test instead of postgres:///postgres and postgres:///test.

      Note that python does the right thing for file URLs, but this seems to be coded as an exception rather than falling out of the standard rules of URL parsing.

      >>> urlunparse(urlparse("file:///abc"))
      'file:///abc'
      >>> urlunparse(urlparse("postgres:///abc"))
      'postgres:/abc'
    12. added a commit that references this issue on Jan 22, 2026
    13. added a commit that references this issue on Feb 15, 2026
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.15bugs and security fixesstdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions