gh-69605: Add module autocomplete to PyREPL by tomasr8 · Pull Request #129329 · python/cpython

tomasr8 · 2025-01-27T00:55:24Z

DPO discussion thread: https://discuss.python.org/t/looking-for-feedback-on-adding-import-autocomplete-to-pyrepl/82281

Adds a module autocomplete functionality to the PyREPL. Some examples first:

import<tab>importfoo<tab>importfoo.<tab>importfooasbar, baz<tab>from<tab>fromfoo<tab>fromfooimport<tab>fromfooimportbar<tab>fromfooimport (barasbaz, qux<tab>

(Check the tests to see a complete list of what is and is not supported)

The implementation is based on the original patch by @vadmium and adapted to PyREPL.

It can autcomplete both import and from ... import statements as long as the import fragment is valid. It uses a tokenizer/parser-based approach so that it can correctly recognize more complex contructs such as from foo import (bar as baz, qux<tab>.

Module search is done by pkgutil.iter_modules.

I wasn't sure where exactly to put this - for now it hooks into ReadlineAlikeReader.get_completions, let me know if there's a better place for it!

Note that, if there are any import completions available, normal completions are turned off (so that e.g. import m<tab> will not suggest import map().

Feedback welcome :)

Issue: Readline completion of module names in import statements #69605

picnixz · 2025-01-27T06:37:51Z

I have a branch which adds a filtering logic to the completer (see https://discuss.python.org/t/repl-introduce-generic-filters-to-filter-auto-completion-matches/77553) and I wondered whether we could expand the Completer interface in order to have a more generic hook category. Both your work and mine can be orthogonal but we might take this opportunity to make the completer class more generic (I haven't looked at the PR in details as I'm travelling now)

tomasr8 · 2025-01-27T09:34:30Z

Right, I remember seeing your dpo post before! I think it's a good idea :) With this PR I decided to target PyREPL since it is less stable than rlcompleter and thus it's easier to expand/make changes. Though we could definitely think about unifying the interface with that of rlcompleter. For now the API is a bit ad hoc so I'd welcome some discussion in that direction 🙂

tomasr8 · 2025-01-27T09:48:44Z

 return result

+ def get_module_completions(self) -> list[str]:
+ completer = ModuleCompleter(namespace={'__package__': '_pyrepl'}) # TODO: namespace?


One thing I wasn't sure about is how to handle relative imports in regards to __package__. In PyREPL, it is set to _pyrepl so I replicate that here but maybe we want it to be configurable?

hugovk · 2025-01-30T12:23:22Z

On macOS, type import re<tab>:

>>> import re
[ complete but not unique ]

Press <tab> again:

>>> import re
re reprlib readline resource requests

Then change your mind and press ctrl+C, and you get:

>>> import re
KeyboardInterrupt readline resource requests
>>>
re reprlib readline resource requests

I'd expect something more like:

>>> import re
KeyboardInterrupt
>>>

Press ctrl+C a couple more times:

>>> import re
KeyboardInterrupt readline resource requests
>>>
KeyboardInterrupt readline resource requests
>>>
KeyboardInterrupt readline resource requests
>>>
re reprlib readline resource requests

I'd expect something more like:

>>> import re
KeyboardInterrupt
>>>
KeyboardInterrupt
>>>
KeyboardInterrupt
>>>

tomasr8 · 2025-01-30T13:11:12Z

Answering here what I wrote on Discord - the ctrl+C behaviour is present for the existing name/attribute autocomplete as well, this PR is not changing that. I think it's something we could add in a separate PR though!

gaogaotiantian · 2025-02-25T01:23:15Z

I only did a quick look at the code. Are you trying to import the module during completion?

tomasr8 · 2025-02-25T08:30:49Z

I only did a quick look at the code. Are you trying to import the module during completion?

Yes I try to import it to ensure it's a valid package and so that I can easily list the submodules (e.g. import foo.<tab>). IPython does something similar. Is it a concern?

gaogaotiantian · 2025-02-25T16:45:59Z

Yes I try to import it to ensure it's a valid package and so that I can easily list the submodules (e.g. import foo.<tab>). IPython does something similar. Is it a concern?

I think so. Implicitly import a module during auto-completion seems a bit unintentional to me. For example, importing torch could take 2~3 seconds, and to user it would be a freeze during completion. Some libraries monkeypatch stdlib or even Python during import, and that's some hidden side effect. Personally I don't like this implicit behavior. Also, will the import only triggered if . is in the text? Or for all possible completion? Say I do import t<tab> does it import all the modules starting with a t?

tomasr8 · 2025-02-25T21:23:25Z

Say I do import t does it import all the modules starting with a t?

This PR uses pkgutil.iter_modules so just typing import t<tab> does not actually import anything. The results of pkgutil.iter_modules are also cached so the top-level packages are only searched once.

I do import in case you type import torch.x<tab>, but I think I might be able to avoid that.

What I think I cannot avoid is an import in case you type from foo import x<tab>. In order for me to know which objects are available for import, I need to actually import the module. I don't think there is a way around that? What do you think?

gaogaotiantian · 2025-02-25T22:06:08Z

Personally, I would prefer if the auto-completer never imports any module without my knowledge, even if that means it can't do its job in some cases. It's okay if it completes when the module already imported, but importing a new module during auto-completion seems wrong to me. I think we should at least get opinions from a few other core devs about this, because in my mind this is a relatively large new behavior.

tomasr8 · 2025-02-25T22:20:12Z

I think we should at least get opinions from a few other core devs about this, because in my mind this is a relatively large new behavior.

I admit I do not fully understand the implications of this so I think getting more opinions is a good idea :) Do you want me to start a discussion on Discord?

gaogaotiantian · 2025-02-25T22:26:54Z

Discord or/and dpo maybe? We should at least get some opinions from repl owners.

tomasr8 · 2025-02-25T22:46:18Z

ok! It's getting a bit late here, so I'll do it tomorrow 🙂

tomasr8 · 2025-03-10T15:28:37Z

I updated the PR following the discussion on DPO: https://discuss.python.org/t/looking-for-feedback-on-adding-import-autocomplete-to-pyrepl/82281

Modules are no longer imported. Modules are discovered using a combination of pkgutil.iter_modules and submodule_search_locations.
Removed support for module attributes. This relied on actually importing the modules. We can add this once we have decided in which way we should discover module attributes.

pablogsal · 2025-04-19T01:16:46Z

 return result

+ def get_module_completions(self) -> list[str]:
+ completer = ModuleCompleter(namespace={'__package__': '_pyrepl'}) # TODO: namespace?


Either remove the TODO or we should handle this differently

I removed the TODO and left a comment. Inside pyrepl, __package__ is set to _pyrepl so the module completer should use the same value when resolving relative packages.

pablogsal · 2025-04-19T01:20:19Z

+ def get_module_completions(self) -> list[str]:
+ completer = ModuleCompleter(namespace={'__package__': '_pyrepl'}) # TODO: namespace?
+ line = self.get_line()
+ return completer.get_completions(line)


Can this raise? What do we want to do if any of the pkgutil raises for any reason?

It shouldn't unless iter_modules or find_spec raise. Since those can call code from user-supplied finders, they can raise an arbitrary exception. I could add a simple except Exception block and simply return no completions if that happens. To the user it would look like no completions are available rather than outright crashing the repl.

Added in fd81999

pablogsal · 2025-04-19T01:21:57Z

+ """Global module cache"""
+ if not self._global_cache or self._curr_sys_path != sys.path:
+ self._curr_sys_path = sys.path[:]
+ self._global_cache = list(pkgutil.iter_modules())


Nothing to do here but I am a bit concerned about how this can perform for a lot of packages

See this #129329 (comment) with timings

pablogsal · 2025-04-19T01:23:39Z

+ if self.code.rstrip().endswith('import') and self.code.endswith(' '):
+ return Result(from_name=self.parse_empty_from_import(), name='')
+ if self.code.rstrip().endswith('from') and self.code.endswith(' '):
+ return Result(from_name='')


This calls code.rstrip() a lot, maybe is worth setting that as an attribute or just always use self.code = code.rstrip(). What do you think?
At the very least maybe set it as a local so you don't strip twice

I added it as a local since it's only called in 3 places

pablogsal · 2025-04-19T01:25:56Z

 return result

+ def get_module_completions(self) -> list[str]:
+ completer = ModuleCompleter(namespace={'__package__': '_pyrepl'}) # TODO: namespace?


Is this creating one of this guys per request? Should we cache the ModuleCompleter?

it is, and yes we should cache it. Updated in 5c11124 (I also moved the completer to a separate file to avoid having to reorder everything when I added it to the ReadlineConfig)

Also changed in f4e290a so that each Reader gets a new ModuleCompleter instance (with its own module cache)

pablogsal · 2025-04-19T01:26:55Z

@tomasr8 can you try to play a bit with some edge cases (tons of packages, etc) to see how the performance is for extreme circumstances? I want to be aware of the ways this can "break"

tomasr8 · 2025-04-19T06:48:31Z

Thanks for taking the time to review this, I really appreciate it!

@tomasr8 can you try to play a bit with some edge cases (tons of packages, etc) to see how the performance is for extreme circumstances? I want to be aware of the ways this can "break"

When it comes to performance, the biggest bottleneck is the call to pkgutil.iter_modules() to find all top-level packages.
(the result is cached so we pay the cost only the first time you hit TAB).

Here are some timings for lots of packages. I installed the top 1000 pypi packages (at least those that can be installed on 3.14 so 836 packages installed):

~ pip freeze | wc -l
836

The timings are on a laptop with Intel ultra 9 185H CPU and a new-ish SSD (relevant because pkgutil.iter_modules interacts with the file system).

For those ~800 packages in a single location, pkgutil.iter_modules takes about 0.03s:

>>>importtime, pkgutil>>>start=time.time(); list(pkgutil.iter_modules()); time.time() -start0.03072071075439453

Typing import <tab> takes about 0.03s as well so most of the cost is really inside pkgutil.iter_modules:

>>>import<tab>0.03162884712219238

Subsequent completion requests are much faster since the results are cached:

>>>import<tab>0.002469778060913086

Another thing I tried is multiple search locations (i.e. multiple sys.path entries):

5 different locations and ~4000 packages total:

`pkgutil.iter_modules`: 0.3535459041595459sInitial`import <tab>`: 0.3592982292175293sSubsequent`import <tab>`: 0.0021576881408691406s

10 different locations and ~8000 packages total:

`pkgutil.iter_modules`: 0.6836235523223877sInitial`import <tab>`: 0.6652779579162598sSubsequent`import <tab>`: 0.0002522468566894531s

The initial search is about 0.35s and 0.68s respectively, while the subsequent ones are again very cheap.

hugovk · 2025-04-23T10:58:29Z

I still get this failure locally:

======================================================================
FAIL: test_import_completions (test.test_pyrepl.test_pyrepl.TestPyReplModuleCompleter.test_import_completions) (code='import path\t\n')
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/hugo/github/python/cpython/main/Lib/test/test_pyrepl/test_pyrepl.py", line 938, in test_import_completionsself.assertEqual(output, expected)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^AssertionError: 'import path' != 'import pathlib'
- import path
+ import pathlib
? +++

It's because my env isn't a clean slate because I've pip installed some packages for testing on 3.14, and a dependency is interfering:

Python 3.14.0a7+ (heads/completer:f4e290a03f1, Apr 23 2025, 13:32:14) [Clang 16.0.0 (clang-1600.0.26.6)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import path<tab>
pathlib pathvalidate

❯ ./python.exe -m pip freeze | grep pathpathvalidate==3.2.0

I don't think this needs to block initial merge before the freeze, but would be good to fix.

(PS there's a merge conflict on some imports.)

tomasr8 · 2025-04-23T12:24:12Z

Weird, I tested this locally with some scripts on sys.path and it passed for me. Sorry about that, I'll take a look later today!

tomasr8 · 2025-04-23T20:41:58Z

Ok so iter_modules also gets FileFinders from sys.path so I had to change the approach but it should work now. Tested locally with pathvalidate installed and the tests are passing.

hugovk · 2025-04-24T08:49:06Z

Passing now, thanks!

pablogsal · 2025-04-25T01:24:37Z

LGTM, fantastic job @tomasr8

tomasr8 added 2 commits January 27, 2025 00:19

Add module autocomplete to PyREPL
b3bcd67

Add news entry
bcd3527

tomasr8 requested review from ambv, lysnikolaou and pablogsal as code owners January 27, 2025 00:55

bedevere-appBot added the awaiting review label Jan 27, 2025

bedevere-appBot mentioned this pull request Jan 27, 2025
Readline completion of module names in import statements #69605
Closed

Merge branch 'main' into completer
df3e8ec

tomasr8 commented Jan 27, 2025
View reviewed changes

Merge branch 'main' into completer
48ee6ad

tomasr8 added the topic-repl Related to the interactive shell label Feb 22, 2025

hugovk reviewed Feb 25, 2025
View reviewed changes

tomasr8 added 4 commits March 8, 2025 14:52

Remove attribute completion, never import modules
0917d24

Add type annotations
589cf63

fix some mypy issues
62d0b55

Pass explicit None to find_spec
46ca249

neutrinoceros mentioned this pull request Mar 9, 2025
ENH: ensure that dir(astropy) lists subpackages astropy/astropy#17598
Merged
1 task

tomasr8 added 3 commits April 17, 2025 16:38

Merge branch 'main' into completer
75e4b55

Do not suggest modules which are not legal identifiers
3c13f86

Make the tests more robust
8eb656f

pablogsal reviewed Apr 19, 2025
View reviewed changes

tomasr8 added 3 commits April 19, 2025 08:29

Remove todo comment
7a2fde0

Move to a separate file and cache ModuleCompleter
5c11124

Avoid calling rstrip more than once
10da15b

tomasr8 added 3 commits April 19, 2025 09:01

Catch exceptions
fd81999

Fix tests
8fba3d3

Every Reader has its own ModuleCompleter instance
f4e290a

tomasr8 requested a review from pablogsal April 20, 2025 18:48

tests: Only look for modules in the stdlib
602121d

pablogsal approved these changes Apr 25, 2025
View reviewed changes

bedevere-appBot added awaiting merge and removed awaiting review labels Apr 25, 2025

pablogsal merged commit c3a7118 into python:mainApr 25, 2025

bedevere-appBot removed the awaiting merge label Apr 25, 2025

tomasr8 deleted the completer branch April 25, 2025 07:31

pllim mentioned this pull request Apr 25, 2025
Revisit dir(astropy) implementation when Python 3.14 is minversion astropy/astropy#18055
Open

devdanzin mentioned this pull request May 6, 2025
gh-69605: Add PyREPL import autocomplete feature to 'What's New' #133358
Merged

picnixz mentioned this pull request May 18, 2025
Suggest the possible module in ModuleNotFoundError #134175
Closed

Uh oh!

Uh oh!

Conversation

tomasr8 commented Jan 27, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

picnixz commented Jan 27, 2025

Uh oh!

tomasr8 commented Jan 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hugovk commented Jan 30, 2025

Uh oh!

tomasr8 commented Jan 30, 2025

Uh oh!

gaogaotiantian commented Feb 25, 2025

Uh oh!

tomasr8 commented Feb 25, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gaogaotiantian commented Feb 25, 2025

Uh oh!

tomasr8 commented Feb 25, 2025

Uh oh!

gaogaotiantian commented Feb 25, 2025

Uh oh!

tomasr8 commented Feb 25, 2025

Uh oh!

gaogaotiantian commented Feb 25, 2025

Uh oh!

tomasr8 commented Feb 25, 2025

Uh oh!

tomasr8 commented Mar 10, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tomasr8Apr 19, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pablogsal commented Apr 19, 2025

Uh oh!

tomasr8 commented Apr 19, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hugovk commented Apr 23, 2025

Uh oh!

tomasr8 commented Apr 23, 2025

Uh oh!

tomasr8 commented Apr 23, 2025

tomasr8 commented Jan 27, 2025•
edited
Loading

tomasr8Apr 19, 2025•
edited
Loading

tomasr8 commented Apr 19, 2025•
edited
Loading