Description
Running make linkcheck in Doc folder outputs thousands of broken and redirected status. It would be nice to clean this up so we can link-check Python in CI/CD for each commit, but right now it is too polluted.
Some of these occurrences could/should be fixed in the docs itself, others can benefit from Sphinx's linkcheck configs e.g. linkcheck_ignore and linkcheck_allow_redirect.
See linkcheck-log.txt for the full log as of commit f2b7ecb.
Fun stats obtained from the above file:
- Of its 8327 lines, where 7824 are related to BPO -> GH issues. Of 7824, 5744 lines are from whatsnew/changelog and only 20 are not from whatsnew/
- 241 lines are redirection of CPython CVS URL, fixing /tree/ to /blob/ in GitHub URL
- 28 lines are broken links, where one broken link is 'https://' from an example in whatsnet/changelog
The way I see, this steps divide in:
- Clean BPO to GH redirection messages
- Fix broken links
- Clean CPython CVS URL redirections
- Clean GH issues to GH PR redirections
- Fix the rest of the occurrences.
linkcheck_allow_redirect and linkcheck_ignore can be very handy in this case. linkcheck_allow_redirect makes 'ok' status a redirect that is being spotted by linkcheck, and we have linkcheck_ignore as the last resource.
Questions I have before implementing the solution:
- Documentation hosted by Read The Docs may have language enabled so example.com is redirected to example.com/en/latest. To handle occurrences, I could add them to linkcheck_allow_redirect or we can use sphinx-ext-intersphinx to map a keyword to the documentation URL (e.g 'rtd' for read-the-docs docs). The last option allows to map to proper language of the target URL linked, similar on how Weblate did)
- Is there any restrictions to fix broken/redirect links in old whatsnew/.rst?
- Is there any restrictions to fix broken/redirect links in old whatsnew/changelog.rst (i.e. Misc/NEWS)?
- Should I create a single Pull Request for all the fixes?
Linked PRs
- GH-103484: Docs: add linkcheck allowed redirects entries for most cases #103569
- [3.11] GH-103484: Docs: add linkcheck allowed redirects entries for most cases (GH-103569) #103602
- GH-103484: Fix broken links reported by linkcheck #103608
- [3.11] GH-103484: Fix broken links reported by linkcheck (GH-103608) #103683
- GH-103484: Fix redirected permanently URLs #104001
- [3.11] GH-103484: Fix redirected permanently URLs (GH-104001) #104088
- GH-103484: Tell linkcheck to ignore debian manpage redirects #123019
- [3.12] GH-103484: Tell linkcheck to ignore debian manpage redirects (GH-123019) #124136
- [3.13] GH-103484: Tell linkcheck to ignore debian manpage redirects (GH-123019) #124137
- GH-103484: Fix permanently redirects reported by linkcheck #124144
- [3.13] GH-103484: Fix permanently redirects reported by linkcheck (GH-124144) #124151
- [3.12] GH-103484: Fix permanently redirects reported by linkcheck (GH-124144) #124152
- GH-103484: Fix broken links reported by linkcheck #124169
- [3.13] GH-103484: Fix broken links reported by linkcheck (GH-124169) #124179
- [3.12] GH-103484: Fix broken links reported by linkcheck (GH-124169) #124180
- [3.12]: gh-103484: Relink _xxsubinterpretersmodule.c in whatsnew/3.12.rst #124183
- gh-103484: Fix links reported as "broken" by linkcheck #140012
- gh-103484: Remove the outdated generator tutorials footnote in unittest.mock examples #162166
- [3.15] gh-103484: Remove an outdated footnote in the
unittest.mockexamples (GH-162166) #162200 - [3.14] gh-103484: Remove an outdated footnote in the
unittest.mockexamples (GH-162166) #162201
Metadata
Metadata
Assignees
Labels
Projects
- StatusShow more project fieldsTodo
Activity
merwok commentedon Apr 16, 2023
A workflow in the main branch applies to other branches?! What a strange system.
rffontenelle commentedon Apr 16, 2023
Yep. It is a centralized way to control what routines will run in which events (push, pull requests, etc.) and in which branches.
hugovk commentedon Apr 17, 2023
Are you sure?
I made a test repo with
mainand3.11branches:Each branch has a workflow that does
echo "hi from <branch-name>":A push to each branch echoes the matching version number. So it runs its own workflow, not the one from
main:Also, making branches from each, and a PR back, also echoes the matching version number, not the one from
main:rffontenelle commentedon Apr 17, 2023
I was sure until now. I mean, what's the point of having branch filters with they don't actually filter on which branch a workflow can or cannot be run?
hugovk commentedon Apr 17, 2023
I expect it's to have as few changes as possible between branches to make it easier for the bot to backport (and for humans when the bot can't do it).
rffontenelle commentedon Apr 17, 2023
Well, I stand corrected. That means a linkcheck job will need to be added to both main and 3.11. No problem.
GH-103484: Docs: add linkcheck allowed redirects entries for most cas…
pythonGH-103484: Docs: add linkcheck allowed redirects entries for mo…
hugovk commentedon Apr 18, 2023
We probably don't want to run linkcheck on the CI for a couple of reasons:
GH-103484: Docs: add linkcheck allowed redirects entries for most cas…
rffontenelle commentedon Apr 18, 2023
Fixes to broken links now submitted.
Merge branch 'main' into superopt
GH-103484: Fix broken links reported by linkcheck (#103608)
pythonGH-103484: Fix broken links reported by linkcheck (python#103608)
rffontenelle commentedon Apr 22, 2023
To reduce this time, a workaround is to include in linkcheck_ignore the links to https://bugs.python.org/ and https://github.com/python/cpython/issues/. A local test (editing conf.py) reduced the time to just 5 minutes!
I would prefer this is happen only via command-line option e.g.
make -C Doc linkcheck SPHINXOPTS='--keep-going -D linkcheck_ignore=<expr>', but I was unable to find the proper<expr>. Any suggestion?Indeed, this is something out of control of cpython. We could set higher value in linkcheck_retries and linkcheck_timeout, though.
merwok commentedon Apr 22, 2023
What about defining a custom environment variable (instead of using SPHINXOPTS) that you can read in the config code?
rffontenelle commentedon Apr 23, 2023
OMG, awesome idea.
It is simply the case of checking if
$CIenvironment variable is set, because GitHub Actions sets it. Then, insert the desired URLs:It does reduce run time to 5 minutes. See my test.
[3.11] GH-103484: Fix broken links reported by linkcheck (GH-103608) (#…
rffontenelle commentedon Apr 23, 2023
@hugovk have you considered my idea on adding linkcheck to GitHub Actions with 1) linkcheck_ignore BPO and GH issues to reduce time to 5 minutes; 2) set higher timeout and retries values ?
Also, I recommend disabling
-Wflag to error on broken links, but setting up a problem matcher to annotate both broken (as error) and redirects (as warning).I can propose a PR for better analysis.
rffontenelle commentedon Apr 30, 2023
New PR, now aiming occurrences of "redirected permanently". After this one, there is only "redirected with Found" and linkcheck is OK.
GH-103484: Fix redirected permanently URLs (#104001)
pythonGH-103484: Fix redirected permanently URLs (python#104001)
[3.11] GH-103484: Fix redirected permanently URLs (GH-104001) (#104088)
Merge branch 'main' into inlinecomp2
Merge branch 'main' into cached_property_docs
rffontenelle commentedon Aug 14, 2024
After some time, I see new wild links popping up from linkcheck output.
My main problems now are these, which I hope to get :
pypi role via extlinks, PyPI's URL normalization and "redirected permanently":
conf.py uses extlinks extension to create the
:pypi:`packagename`role, but PyPI is normalizing the URL to lowercase letters and dash instead of underscore. So we're having "redirected permanently" status on these links. For instance,:pypi:`Urwid`results in https://pypi.org/project/Urwid/, but PyPI redirects to https://pypi.org/project/urwid/.Is it possible for extlinks to generate an URL with the proper replacements (lowercase letters and dash instead of underscore) so I can avoid adding an ignore exception to linkcheck?
Links to other Sphinx-based documentation causing "redirect with found"
Python docs links to many other Sphinx-based documentation, which have its URL appended with language and version (e.g. https://babel.pocoo.org/ to https://babel.pocoo.org/en/latest/). Adding exceptions or Interpshinx might add a lot of lines to conf.py, because it would be a least one line per documentation URL
For the record, these are the currently non-ok output of linkcheck:
status "broken"
howto/curses.rst:541: [broken] https://linux.die.net/man/3/ncurses: 403 Client Error: Forbidden for url: https://linux.die.net/man/3/ncurses library/functools.rst:220: [broken] https://en.wikipedia.org/wiki/Cache_replacement_policies#Least_recently_used_(LRU): Anchor 'Least_recently_used_%28LRU%29' not found library/select.rst:279: [broken] https://linux.die.net/man/4/epoll: 403 Client Error: Forbidden for url: https://linux.die.net/man/4/epoll library/ssl.rst:2757: [broken] https://www.openssl.org/docs/man1.1.1/man1/ciphers.html#CIPHER-LIST-FORMAT: Anchor 'CIPHER-LIST-FORMAT' not found library/statistics.rst:995: [broken] https://www.statisticshowto.com/probability-and-statistics/z-score/: 403 Client Error: Forbidden for url: https://www.statisticshowto.com/probability-and-statistics/z-score/ whatsnew/2.4.rst:759: [broken] https://developer-old.gnome.org/glib/2.26/: HTTPSConnectionPool(host='developer-old.gnome.org', port=443): Max retries exceeded with url: /glib/2.26/ (Caused by NameResolutionError(": Failed to resolve 'developer-old.gnome.org' ([Errno -2] Name or service not known)")) whatsnew/2.6.rst:191: [broken] https://svn.python.org/view/tracker/importer/: unauthorized whatsnew/3.12.rst:361: [broken] https://github.com/python/cpython/tree/main/Modules/_xxsubinterpretersmodule.c: 404 Client Error: Not Found for url: https://github.com/python/cpython/tree/main/Modules/_xxsubinterpretersmodule.c whatsnew/3.2.rst:1650: [broken] https://www.openssl.org/docs/man1.0.2/man1/ciphers.html#CIPHER-LIST-FORMAT: Anchor 'CIPHER-LIST-FORMAT' not found whatsnew/3.4.rst:1962: [broken] https://ltp.sourceforge.net/coverage/lcov.php: 404 Client Error: Not Found for url: https://ltp.sourceforge.net/coverage/lcov.php whatsnew/3.8.rst:2228: [broken] https://ark.intel.com/content/www/us/en/ark/products/76088/intel-core-i7-4960hq-processor-6m-cache-up-to-3-80-ghz.html: 403 Client Error: Forbidden for url: https://ark.intel.com/content/www/us/en/ark/products/76088/intel-core-i7-4960hq-processor-6m-cache-up-to-3-80-ghz.htmlstatus "redirected permanently"
status "redirected temporarily"
status "redirected with Found"
status "timeout"
GH-103484: Tell linkcheck to ignore debian manpage redirects (#123019)
hugovk commentedon Aug 16, 2024
linkcheck_allowed_redirects?cpython/Doc/conf.py
Line 534 in e2f2dc7
linkcheck_allowed_redirects? It allows regular expressions.pythonGH-103484: Tell linkcheck to ignore debian manpage redirects (p…
pythonGH-103484: Tell linkcheck to ignore debian manpage redirects (p…
pythonGH-103484: Tell linkcheck to ignore debian manpage redirects (p…
[3.12] GH-103484: Tell linkcheck to ignore debian manpage redirects (G…
GH-103484: Fix permanently redirects reported by linkcheck (GH-124144)
pythonGH-103484: Fix permanently redirects reported by linkcheck (pyt…
pythonGH-103484: Fix permanently redirects reported by linkcheck (pyt…
[3.12] GH-103484: Fix permanently redirects reported by linkcheck (GH…
GH-103484: Fix broken links reported by linkcheck (#124169)
pythonGH-103484: Fix broken links reported by linkcheck (pythonGH-124169
pythonGH-103484: Fix broken links reported by linkcheck (pythonGH-124169
vstinner commentedon Sep 17, 2024
Fixed by ab80c6b. Backports will follow.
[3.12] GH-103484: Fix broken links reported by linkcheck (GH-124169) (#…
rffontenelle commentedon Sep 17, 2024
@vstinner I was hoping to include at least another PR for "redirect with found" reported by linkcheck, (see above comment #103484 (comment)), and then we can close this. Can you please keep this open for now?
pythonGH-103484: Fix permanently redirects reported by linkcheck (pyt…
pythonGH-103484: Fix broken links reported by linkcheck (python#124169)
[3.13] GH-103484: Tell linkcheck to ignore debian manpage redirects (G…
[3.13] GH-103484: Fix broken links reported by linkcheck (GH-124169) (#…
[3.13] GH-103484: Fix permanently redirects reported by linkcheck (GH…
[3.12]: gh-103484: Relink _xxsubinterpretersmodule.c in whatsnew/3.12…
StanFromIreland commentedon Oct 12, 2025
I ran
make linkcheckon main (which took a while!), the full log: anseo!Broken cases, ignoring 403's and false positives they are all fixed by #140012:
warnings
pythongh-103484: Fix dead links to David Beazley's generator tutorials
gh-103484: Remove an outdated footnote in the `unittest.mock` examples (