Support for PEP 484 style # type comments

I'd like to open a discussion about whether flake8 and pyflakes should include support for # type: comments as standardized by PEP 484. The way PEP 484 is currently written, names used in such type comments must be imported using regular import statements. When such an imported name is used exclusively in a # type comment it may elicit a spurious warning about an unused import ("F401 UnusedImport"). While it's easy to shut this up by putting # noqa on the import line, that gets ugly quick. (That's the solution we have adopted so far at Dropbox, but we really need something better.)

While you might think that it would be enough to suppress F401 about from typing import ... it is also common to have to import some other class that is not instantiated directly in the code but still needs to be named in a # type comment. This is especially prevalent in Python 2 code where signature annotations are used; this is a recent addition to PEP 484: https://www.python.org/dev/peps/pep-0484/#suggested-syntax-for-python-2-7-and-straddling-code.

I've developed a hack to deal with this, but it is pretty darn ugly. The problem is that the UnusedImport warning (and similar warnings like UndefinedName and UnusedVariable) are implemented by pyflakes, which takes an AST. Comments are not represented in the AST. So my hack is to append extra lines to the end of the file containing expressions that are copied (almost) literally from the # type comments -- but this is really hideous. A more principled implementation might be to switch to the parser implemented in lib2to3, which does represent comments; but that could be a big effort, plus it's slower.

Finally -- I couldn't find a tracker for pyflakes, otherwise I would have reported this there. I actually developed the hack twice -- once for pyflakes (as a diff to pyflakes/api.py) and then again for flake8 (as a monkeypatch for pep8.Checker.check_ast).

Thoughts?

So I'd definitely like to see this myself. My other concern was how this will work with pydocstyle (formerly pep257) but it seems to work just fine.

I'm not certain this is something that pyflakes, pep8, or flake8 can fix right now. I'm working on flake8 3.0 which will be (ideally be far more powerful) and I think we'll be able to handle this, if not in flake8 core than in a plugin to handle silencing UnusedImport errors for comments like this.

---

For no other reason than you said you couldn't find it, pyflakes links:

• https://github.com/pyflakes/pyflakes
• https://bugs.launchpad.net/pyflakes

Milestone changed to 3.0.0

Added bug:confirmed label

Added component:pyflakes feature:accepted labels

Reassigned to @sigmavirus24

@gvanrossum question for you: Does mypy now support Python 2.7? The README says it supports 3.2+. If so, how're you using the # type: comments? I think I'd like to start using this for flake8 3.0 development.

[Strange, gitlab bounced my email response]

@gvanrossum question for you: Does mypy now support Python 2.7?

Yes, using the notation I added to PEP 484 recently (linked from the first comment in this issue).

The README says it supports 3.2+.

Oh, sorry. We've been so busy implementing all sorts of stuff that updating the docs and doing a release have taken a backseat.

If so, how're you using the # type: comments? I think I'd like to start using this for flake8 3.0 development.

We use them instead of signature annotations. Mypy treats them as equivalent.

Let us know if you run into any issues -- we're eager to get more (early) users!

So, a couple notes for myself (and others watching this) and a question/thought:

Notes:

• mypy still needs to be installed on Python 3 in order to run
• the PyPI package is mypy-lang not mypy

Question/Thought:

I tried running mypy on flake8 3.0's source tree which is partially hinted with the # type: comments. I had avoided using from typing import ... in each file because with a few exceptions most of the types are straight-forward. If I want to use some of the generics from typing though, that module now becomes a runtime dependency for flake8, right?

In other words, by importing from the typing module in Flake8 unconditionally for mypy, then typing becomes something that is listed as a requirement for Flake8 (on Python 2, and 3.x where x < 5). I'm not sure Flake8 wants to do that, but I also wonder if other projects wanting to use the # type: comments will feel the same. In short, I'm wondering if mypy could instead look for tagged comments (similar to the # type: comments) for imports. In that case, typing, et. all, would need to be installed whenever running mypy but otherwise it wouldn't be necessary. (That would also have the benefit of side-stepping this issue.)

Yes, the Python version used to run mypy is orthogonal to the Python version that it checks -- it's best to think of mypy as a binary. The mypy-lang name is unfortunate but can't be helped until we hear back from the owner of the "mypy" package on PyPI (last uploaded in 2011, seemingly no longer responding to that email address).

And yes, if you need to declare an argument as e.g. returning a list of integers, you have to us from typing import List and then use List[int] in the annotation.

We've had extensive discussions in the Dropbox mypy team about whether to support importing things in comments so as to avoid the runtime dependency on the typing module. (Ironically the reason to look at this was the desire to reduce noise from flake8 about those imports.)

In the end we agreed not to do this because it's not just simple imports (though that's the first use case one typically encounters) -- in order to always avoid importing the typing module, you would also need to support type variable definitions, type aliasing, cast() calls, and (perhaps most importantly) defining new generic types.

However, if you're not defining new generic types or functions (so you also don't need to declare type variables), all your annotations are in comments, and you don't need type aliases or cast(), then there's a trick: mypy still analyzes code inside if False: so you can get away with writing

if False:
    from typing import Any, List, Dict, ...

Hope this helps!

Removed bug:confirmed component:pyflakes feature:accepted labels

Milestone changed to %3.x.0

I'm looking into fixing this. The obvious choice here would be to adopt typed-ast in Flake8. The problem is, it should rather be adopted in pyflakes.

Currently pyflakes directly relies on the built-in AST module and has a ton of if/else blocks with checks based on the current running Python version to decide what nodes are available, what attributes they contain, etc. This is different from typed-ast which decouples the version of the interpreter from the version of the analyzed AST. Arguably, only supporting ast27 and ast35 would simplify the source of pyflakes. However, it would then lose the ability to find syntax errors against the currently running interpreter (say, availability of f-strings on Python 3.5). We can work around that by pre-parsing with the built-in parser first for syntax errors but that slows things down.

The bigger problem is that the change of the AST provider is viral. All AST plugins with visitors or isinstance/type() is checks would fail. So this poses a pretty significant backwards incompatibility. And there's no way to merge the type comments with the built-in AST module pre-3.6 without subclassing, which again would break compatibility for some plugins.

Another issue is the current unavailability of typed-ast for Windows.

Ian, I think this is a worthwhile improvement for Flake8 4.0, which specifically enables running checks on 2.x source code from a Python 3 interpreter, and vice versa. It would also simplify pyflakes (given how complicated it is to integrate with pyflakes or customize its behavior, I start daydreaming about a rewrite).

Is someone working on this?

@toogy what is there to work on? typed-ast still has far too many open TODO items that imply instability in the project. Until that actually stabilizes, this will not land. The standard library does not preserve comments, and without that happening PyFlakes cannot be adapted to understand the imports from typing.

mentioned in issue #342 (closed)

mentioned in issue #352