Automatic fixing by plugins
So there had been a request for this previously (by @joe-gordon0 if my memory is serving me well) but I just cannot find it. I've reached out to GitLab about it (https://twitter.com/sigmavirus24/status/811238094271840261).
I believe the original issue was #84 but that is, for some reason, lost.
tl;dr There is immense interest in having Flake8 provide fixes for the violations it reports in the style of go-fmt, either by doing the fixing in-place or allowing plugins to print out how to apply the fix.
The Problem/Reasoning
Most people use Flake8, presently, to prevent style issues from being introduced and to force collaborators to follow a specific style. People, however, want to begin using Flake8 to just correct that style without human intervention. When Flake8 3.0 was written (from scratch) I implemented the ability for plugins to register and be notified of specific violations happening. The idea was for this architecture to then be utilized for plugins to fix a specific violation for users.
Since then, I've realized that we also need to have the ability to prioritize which violations notify plugins first. On top of that, we need some good interface for plugins to modify a file. For this, I've been looking at RedBaron. The problems with this, right now, are:
- RedBaron is slow (for now)
- Passing around a mutable FST representation might not be the best way of doing this and @ambv has somewhat recommended against it
On top of that, we'll be reimplementing (to a degree) in a far less opinionated manner what Google has already built and it might just be better to collaborate with them to make their system pluggable.
Why make things pluggable?
A lot of PEP-0008 is written ambiguously and allows different styles as "valid". That means that pycodestyle has a lot of things it accepts as valid and just as many it accepts as invalid. Further, that means each person can use a style they prefer and still have it be valid. All of this combined means that if one plugin fixes one style violation one way, people are going to complain it doesn't fix it the way they like. As such, they should be able to swap these out for each other and have it simply work the way they want. There can be 5 ways to not cause Exxx and so there are likely an equal number of people who use each of those 5 ways. If Flake8 said "This is the one way to do it" we'd get so much hatemail that I'd probably quit. I'd rather just let people do what they want than come down with one opinion.
Activity
I recommended against mutating the FST as response to raised errors because it causes surprising effects. One fixer can trigger another warning. If you executed the fixers recursively in this case, we cannot guarantee it will ever finish (fixer cycles, etc.).
yapf took a more holistic approach: unwrap lines, permutate wrapping while applying penalty points for things the user doesn't like, choose the formatting with the least penalty points. This is the algorithm used by clang-format. This gives much better results but still causes surprises every now and then. Specifically, applying correct weights in config becomes an art form.
gofmt is more simplistic, AFAICT it has limited configuration, similar to JSON pretty printers, etc.
Don't get me wrong, auto-fixers are an awesome productivity boost, I'd very much like to see them. It's just that it's a hard problem and in practice the result is semi-automatic at best with FST, or semi-elegant with applied weights or pretty printers.
yapf uses lib2to3 underneath, which has a big warning in the docs: https://docs.python.org/2/library/2to3.html#module-lib2to3. I am not sure if someone has forked lib2to3 or if the warning is outdated.
having played around with autopep8 I have seen some of the issues @ambv describes. And I think semi-automatic is what we should aim for.
@sigmavirus24 how many pep8 rules have low risk of causing other issues and have few ways of fixing? Perhaps aiming to solve those cases first may make things a little easier?
Or we could just embrace yapf and be done with it?
lib2to3 is very unlikely to change anymore. Also, the usage of it by yapf is limited mostly to unwinding lines.
There is no inherently "safe" subset of rules, whatever changes line lengths or affects other formatting might trigger warnings that weren't there before. Not to mention people running custom plugins that may have additional checks. So, the way to introduce "safe" rules would be to suggest auto-fixes to the user only after we applied them temporarily and confirmed no additional warnings are generated. The complication here is that we'd have to track line and column changes to differentiate between existing and additional warnings. Better yet, those additional warnings might be triggered outside of the lines that are fixed, for example when cyclomatic complexity is increased or a name used before isn't anymore.
If it sounds like I tried this in the past, it's because I did ;-) In the end it seems yapf is simply much more predictable. Funnily, it also sometimes generates code that disagrees with pycodestyle if the configuration isn't in complete sync :D
Or we could just embrace yapf and be done with it?
It sounds like yapf has improved. What I think is missing, however, is the ability to have one configuration for the autofixer and flake8. Most people already have a project style-guide in the way of select and ignore rules for Flake8.
Perhaps the best thing for us to do would be to have a way to generate a yapf config from Flake8's?
@sigmavirus24 just using yapf sounds like a good idea, less re-inventing the wheel needed.
