web_trawler should be able to trawl all web pages in a domain, especially now that there's a y/n prompt

If implemented, interactive mode for adding linked web pages for trawling should be the default (and maybe there should not even be an uninteractive mode, since some domains are huge). And there should be a stop or break command in addition to y/n.

I think this full-domain-walking-by-links makes more sense than going just a single step from the original page. As in, I imagine the former will be useful more often than the latter.

changed the description

added enhancement suggestion labels

In general in favour of this, but:

1. There should always be a non-interactive mode; this was the first purpose of the programme and should be kept (as e.g. for wiod - we want a command line to download all xls files linking from a given webadress)

2. I see the possibility for something like "web_trawler -i" which follows each link recursively. You than should have the following options at each node: a,A: download all (showing which files first) f,F: filter (which would open a prompt for your regex expression or filetypes - showing which files first) n,N: do nothing, next A,F,N would afterwards automatically go to the next node, a,f,n would instead go through each subnode of the current one at any point you can do s: show the current url path u: abort the current path, go up one level x: abort immediately

1. Non-interactive mode with regard to the file links should still be default, I agree. But if you'd like the trawler to jump to linked web pages in the same domain, to trawl them as well, then that jumping should be interactive by default. Or perhaps your suggestion in point no. 2 is better.

2. Filtering after each node visited seems like it would cause a lot of repetition. How about this: First gather up all the download links from all nodes and then give the option to filter? In other words: on each node you have the option to trawl it or not, i.e. y/n. And once all y'ed nodes have been trawled, you get a list of what's about to be downloaded, and have the option to filter it further (further than the whitelist/blacklist presumably already given).

Not completely sure what you mean by going up a level, since web_trawler doesn't really understand a web site's hierarchical structure. It just follows links, and checks if they belong to the same top level domain as the original node.

1. but we can still have the option non-interactive, just trawl all first/second order links
2. sounds good
3. in order to implement the full interactive option with completely open link following, I think you needs some mechanism to go back/up the "link stack"