Deadlock with very large number of files
Installing flake8
$ virtualenv venv -ppython2.7
$ venv/bin/pip install flake8
$ ./venv/bin/flake8 --bug-report
{
"dependencies": [
{
"dependency": "setuptools",
"version": "3.4.4"
}
],
"platform": {
"python_implementation": "CPython",
"python_version": "2.7.6",
"system": "Linux"
},
"plugins": [
{
"plugin": "mccabe",
"version": "0.5.2"
},
{
"plugin": "pycodestyle",
"version": "2.0.0"
},
{
"plugin": "pyflakes",
"version": "1.2.3"
}
],
"version": "3.2.0"
}
Issue Description
With lots of files, flake8 deadlocks. Deadlock is confirmed by seeing all processes at either wait
or futex_wait
in ps
.
In the 2.x line this does not seem to be a problem.
Reproduction
(This is affecting a real repository with real files, I've simulated this with 100,001 blank files below).
mkdir -p bar
touch bar/a{0..100000}.py
flake8 bar -j8
In a separate tab open:
watch -n.1 "ps -wwwelfy | grep $USER |grep flake8 | grep -v grep | grep -v watch"
In my case, after ~3 minutes the watch panel reaches this state:
Every .1s: ps -wwwelfy | grep asottile |grep flake8 | grep -v grep | grep -v watch Fri Nov 18 11:31:49 2016
S asottile 1484265 1029227 27 80 0 591748 171242 wait 11:21 pts/29 00:02:44 /tmp/fpp/venv/bin/python2.7 /tmp/fpp/venv/bin/flake8 bar -j8
S asottile 1487064 1484265 3 80 0 583904 190214 futex_ 11:22 pts/29 00:00:18 /tmp/fpp/venv/bin/python2.7 /tmp/fpp/venv/bin/flake8 bar -j8
S asottile 1487072 1484265 3 80 0 583920 190244 futex_ 11:22 pts/29 00:00:19 /tmp/fpp/venv/bin/python2.7 /tmp/fpp/venv/bin/flake8 bar -j8
S asottile 1487077 1484265 3 80 0 583916 190221 futex_ 11:22 pts/29 00:00:18 /tmp/fpp/venv/bin/python2.7 /tmp/fpp/venv/bin/flake8 bar -j8
S asottile 1487085 1484265 3 80 0 583920 190221 futex_ 11:22 pts/29 00:00:18 /tmp/fpp/venv/bin/python2.7 /tmp/fpp/venv/bin/flake8 bar -j8
S asottile 1487093 1484265 3 80 0 583924 190222 futex_ 11:22 pts/29 00:00:18 /tmp/fpp/venv/bin/python2.7 /tmp/fpp/venv/bin/flake8 bar -j8
S asottile 1487097 1484265 3 80 0 583924 190222 futex_ 11:22 pts/29 00:00:18 /tmp/fpp/venv/bin/python2.7 /tmp/fpp/venv/bin/flake8 bar -j8
S asottile 1487110 1484265 3 80 0 583956 190221 futex_ 11:22 pts/29 00:00:18 /tmp/fpp/venv/bin/python2.7 /tmp/fpp/venv/bin/flake8 bar -j8
S asottile 1487111 1484265 3 80 0 583920 190222 futex_ 11:22 pts/29 00:00:18 /tmp/fpp/venv/bin/python2.7 /tmp/fpp/venv/bin/flake8 bar -j8
Regression
With an older flake8
, this finishes much quicker and does not deadlock:
$ flake8 --version
2.6.2 (pycodestyle: 2.0.0, pyflakes: 1.2.3, mccabe: 0.5.2) CPython 2.7.6 on Linux
$ time flake8 bar -j8
real 0m15.752s
user 0m14.612s
sys 0m1.104s