daughter directories

Great work guys. Is there any way to make the daughter functionality better? e.g. from http://www.wiod.org/database/wiots16 there are daughters http://www.wiod.org/database/niots16 http://www.wiod.org/database/int_suts16

etc, but the search also grabs everything from the header bar e.g. http://www.wiod.org/database/wiots13

can you exclude the header bar from the search?

The wiod.org header bar can certainly be excluded. But it'll be hard to find a more or less universal solution. We're dependent on html standards or conventions to figure out what is in a header or menubar. And unfortunately, there's just a bunch of web dev framework specific standards. Would it be an acceptable solution to get a y/n prompt per link to include files linked to from them or not? This would of course only work with command line use.

I'll discuss with Konstantin. Maybe he has a better idea.

Yes, a good idea would be a y/n for each page/directory – rather than at the file level.

assigned to @gormr

added discussion enhancement suggestion labels

created branch 2-daughter-directories

The y/n prompt has now been implemented and uploaded to PyPI. Install with pip install web_trawler --upgrade. Note that the -i-flag has changed meaning to "interactive" (see the updated readme).

Run web_trawler http://www.wiod.org/database/wiots16 -ai to get asked about whether to trawl each linked web page.

closed