opened
merge request
!1
"removing .idea pycharm stuff and creating gitignore file"
at
dlab-indecol / moose
created project
dlab-indecol / moose
commented on
issue #3
"web_trawler should be able to trawl all web pages in a domain, especially now that there's a y/n prompt"
at
dlab-indecol / web_trawler
but we can still have the option non-interactive, just trawl all first/second order links...
commented on
issue #3
"web_trawler should be able to trawl all web pages in a domain, especially now that there's a y/n prompt"
at
dlab-indecol / web_trawler
Non-interactive mode with regard to the file links should still be default, I agree. But if you'd like the trawler to jump to linked web pages in...
Web_trawler already uses both multiprocessing and multithreading. I think all that's needed is to catch certain kinds of errors, time.sleep for a w...