Repository navigation
Fix bug with recursive calls not being counted correctly - #444
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #444 +/- ##
==========================================
+ Coverage 84.95% 85.12% +0.17%
==========================================
Files 21 21
Lines 2412 2413 +1
Branches 376 376
==========================================
+ Hits 2049 2054 +5
+ Misses 263 259 -4
Partials 100 100 see 1 file with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
|
Hi, welcome back! Just wanted to say that this looks exciting. But unfortunately I won't be able to do a proper review until a week later (~20th); currently in the middle of a messy relocation.1 Please see if this is fine with your timeline. EDIT: speaking of 3.15, do you happen to know if the current Cython code works as-is for it? Or should we be changing something? Footnotes
|
|
Hey TTsang, if you can't do a proper review for a week then that's fine; this can wait. However, it may be useful if Erotemic can review this and if it's okay then merge it, and then the larger next PR can be reviewed by both of you. This PR is relatively short and sweet. For 3.15, the current Cython code seems to work as-is. I was testing 3.15.0 in my local test matrix earlier and it profiled fine after doing the requisite magic with tracing and such. |
|
Update on the speedup numbers I shared in the initial PR description:
Each of those numbers is a geometric mean reduction of overhead. 3.12 was really the turning point where the speedups began to be massive (I didn't measure 3.12 because Cython is broken on it). The sys.monitoring code that applies to 3.12 and above seems to have been designed without much thought put into reducing overhead, so there was tons of low-hanging fruit there that I picked. I'm excited to send in the speedup PRs once this is merged! Once the speedups are done, I plan to work on making sure line_profiler behaves under the free-threaded build of Python. This will be a major challenge (so it will take a while), but I'll figure it out! |
Happen to know the guy who did that part and can tell you he didn't ;)
In principle it (at least the Cython core) seems that it should be alright, since each thread should only interact with its own data via That said, I remember that our Cython core is currently incompatible with multiple interpreters (e.g. Footnotes
|
|
Hey @Erotemic, do you happen to have time to take a look at this? Totally fine if not, I was just hoping to get to the overhead reductions and speedups sometime soon so that other people can benefit, and this is a bit of a blocker. Also, looking at your activity graph, I think it could be useful to give TTsang commit and merge permissions so that you don't have to spend as much time on this project. TTsang has been consistently active here for a long time and seems dedicated to the success of line_profiler, and it could help reduce any burden on you. |
|
@Theelx This looks good to merge. I don't see anything obviously incorrect, and the motivation makes sense to me. We need to know the time at the start and end of a particular code line, and currently we use the hash of its code (modulo details) to do that. But if a line ends up making a recursive call to itself, the inner scope can record its own measurement of that line before the outer scope has finished, and because both lines have the same code, they have the same key, and thus the inner scope clobbers the outer scope. So instead we are using the address of the Python frame (which contains the code object), so that keys on both the code and the context it was executed in. Then later we can correctly aggregate across multiple frames that measure the same code. I had GPT 6 in chat mode review the PR and I've had it distill the discussion into a list of suggestions to improve the PR quality. Your choice if you take any of these on. I think points 1, 2, and 4 look like they are worth it. 6 is a nitpick that doesn't bother me, but its probably right. Point 3 I think indulges the LLM tendency to over-test things, but its not a terrible idea. Point 5 doesn't seem worth it if it costs additional complexity, and the proposed solution might cause memory churn, so it might not be the right way to address the issue. Details🤖 1. Avoid redundant hash-map lookups (performance)Currently, if deref(last_map).count(frame_key):
old = deref(last_map)[frame_key]That performs two lookups. The exit path similarly checks Consider finding the entry once and reusing the iterator: cdef LastTimeMap.iterator it
it = deref(last_map).find(frame_key)
has_last = it != deref(last_map).end()
if has_last:
old = deref(it).second
has_last = old.block_hash == block_hash
# On exit, reuse the same iterator:
if not is_line_event and it != deref(last_map).end():
deref(last_map).erase(it)The line-event update could similarly use the iterator for an existing entry, avoiding another lookup through 2. Clarify what
|
Erotemic
left a comment
There was a problem hiding this comment.
Consider if you want to implement any of the GPT suggested improvements, here. But this can be merged as is. So make the changes or let me know if you want to defer.
|
Hey @Erotemic, thanks for the suggestions! Suggestions 1 and 4 are already handled by my speedup branches that will come shortly after this is merged I'll implement 2, 3, and 6 now. |
Hey everyone! Sorry for the long absence from working on line profiler. When Python 3.15.0 released a few days ago, I remembered that they have a new profiling module, and that reminded me that line-profiler was a thing that I'd been meaning to speed up for a while. I have some speedups in store that will massively reduce the overhead from line-profiler. On 3.12, the code goes from running ~14.7x slower when being profiled to ~2.6x slower (geomean) across like 5 different scenarios, and there are similar speedups from 3.10 to 3.15. The public API in this optimized version is also fully compatible with the current API. I'm excited to share it, however it'll be a massive PR (currently +2023 lines and -850), so I want to split it up into a few smaller ones that are more targeted.
This PR contains the only bug fix that the speedups rely on. @notEvil found this bug and mentioned it in #230, and I chose to fix it here. Once this is merged, I can start sending in some PRs for the massive speedups!