Repository navigation
Description
This is a summary of several issues concerning import-time profiling when using line_profiler.autoprofile. I'm working on fixes (see the linked PRs), but this issue is mainly for documenting purposes.
Only the last target profiled in multi-target imports FIXED
Click to expand
(See #433.)
In line_profiler/autoprofile/profmod_extractor.py::ProfmodExtractor, ProfmodExtractor.run() is only set up for single-target imports, so in an import statement like
from foo import bar, bazbar would not be profiled unless a line_profiler/autoprofile/ast_profile_transformer.py::AstProfileTransformer is also used (kernprof -l --prof-imports --prof-mod=<path/to/the/profiled/code>).
Remedies/Mitigations
- FIX: handling multi-target (from-)import statements #434: deprecates
ProfmodExtractor.run(), replacing it with.extract_all()which assumes multi-target imports.
Star-imports (from ... import *) result in SyntaxError
Click to expand
This was first reported by @Erotemic and Fable as a part of TTsangSC#5 (see dev/planning/fable-review-fullrepo-2026-07-05.md). Essentially, since ProfmodExtractor and AstProfileTransformer both do these simple substitutions:
import <foo.bar>
->import <foo.bar>; prof.add_imported_function_or_module(<foo.bar>)import <foo.bar> as <baz>
->import <foo.bar>; prof.add_imported_function_or_module(<baz>)from <foo.bar> import <baz>
->from <foo.bar> import <baz>; prof.add_imported_function_or_module(<baz>)from <foo.bar> import <baz> as <foobar>
->from <foo.bar> import <baz> as <foobar>; prof.add_imported_function_or_module(<foobar>)
A star-import of the form from <foo.bar> import * would result in the syntactically wrong from <foo.bar> import *; prof.add_imported_function_or_module(*). As such, if
- The module from which the star-import pulls names is among the
--prof-modtargets (kernprof -l --prof-mod=<foo.bar>), or - Full-module profiling with import rewriting is requested (
kernprof -l --prof-imports --prof-mod=<path/to/the/profiled/code>)
line_profiler/autoprofile/autoprofile.py::run() will error out with SyntaxError.
Remedies/Mitigations
- FIX: handling multi-target (from-)import statements #434: includes the stop-gap measure of catching star-imports, filtering them out, and warning the user against them.
- ENH: handle nested imports and star-imports TTsangSC/line_profiler#6:
- Introduces a new helper function-slash-pseudo-method
line_profiler/autoprofile/line_profiler_utils.py::add_star_import()which actually handles (dynamically) recovering and profiling the imported names. - Adds a new
kernprofflag--prof-star-importsand the config key-value pairs[tool.line_profiler.kernprof]::prof-star-importsand[tool.line_profiler.autoprofile]::prof_star_importsto control whether to interpolateadd_star_import()nodes into the profiled-code AST as withadd_imported_function_or_module().
- Introduces a new helper function-slash-pseudo-method
Nested imports and top-level imports not handled on equal footing
Click to expand
Consider the following code (say, ./script.py):
from foo import bar
try:
from spam import ham
except ImportError:
from other_spam import hamSuppose that we do kernprof -l --prof-mod=foo,spam ./script.py.
- Since
./script.pyisn't in--prof-mod, we don't do a full module rewrite, and just insertprof.add_imported_function_or_module()calls withProfmodExtractor. - However,
ProfmodExtractoronly handles top-level imports, i.e. those directly in the module body without being nested inside compound statements, like the try-except bloc we have. - As such, only
foo.barwill be passed to the profiler, andspam.hamis missed.
Now suppose that we do instead kernprof -l --prof-mod=foo,spam --prof-mod=./script.py --prof-imports ./script.py.
- Since
./script.pyis in--prof-modand--prof-importsis set, we do a full module AST rewrite withAstProfileTransformer. - While
AstProfileTransformerwalks the entire AST and thus is able to catch all the import statements, we lose granularity because all three offoo.bar,spam.ham, andother_spam.hamwill be profiled (if their resp.importstatements are executed), and there is no option to just select the former two and exclude the last.
Remedies/Mitigations
-
ENH: handle nested imports and star-imports TTsangSC/line_profiler#6:
- Adds
kernprofflag--prof-nested-importsand config options[tool.line_profiler.kernprof]::prof-nested-importsand[tool.line_profiler.autoprofile.import_discovery]::*to control the profiling of imports (e.g. exclude imports inside function bodies and loops by default). - Extends
ProfmodExtractor.extract_all()(introduced in FIX: handling multi-target (from-)import statements #434) to also recover import targets nested inside eligible compound statements (as dictated by the above config options), so that--prof-modoptions are respected. - Updates
AstProfileTransformer(used for--prof-imports) so that it also respects the aforementioned config and doesn’t just rewrite arbitrary imports.
Note though that this changes the semantics of
--prof-imports, in that it no longer unconditionally causes imports to be profiled, being now also subjected to scoping restrictions by--prof-nested-imports. - Adds
On-import profiling deduplication results in inconsistent profiling
Click to expand
Consider the following code (say, ./script.py):
def foo() -> None:
from textwrap import indent
print(indent('foo\n''bar\n''baz', '* '))
def spam() -> None:
from textwrap import indent
print(indent('spam\n''ham\n''eggs', '- '))
if __name__ == '__main__':
spam()Suppose that we do kernprof -l --prof-mod=./script.py --prof-imports ./script.py. This rewrites the entire module with AstProfileTransformer and supposedly profiles all (1) locally-defined functions and (2) imports nested arbitrarily deep in the AST. Now AstProfileTransformer keeps track of the prof.add_imported_function_or_module() nodes it (or any prior preprocessing, via the profiled_imports: list[str] parameter) interpolated, and avoids profiling the same import target twice (because add_imported_function_or_module() can descend deep into an object and incur substantial overhead). This is well and all, but in the above example, it results in no profiling data being collected for textwrap.indent(), because prof.add_imported_function_or_module(indent) is only inserted in foo() and not bar().
Remedies/Mitigations
- ENH: handle nested imports and star-imports TTsangSC/line_profiler#6:
- Updates
AstProfileTransformerso that:- It accepts
profiled_imports: list[str]as before, but since this leaves us with insufficient info for deduplication, aDeprecationWarningis issued. - It also takes
profiled_imports: Mapping[...]which is the same format as returned byProfmodExtractor.extract_all()(introduced in FIX: handling multi-target (from-)import statements #434), and uses that to do context-aware on-import profiling deduplication.
- It accepts
- Updates
AstTreeProfiler.profile()and how it usesAstProfileTransformerto make use of the fixed feature.
- Updates
Activity