[AG-TECH] Mailing List Search Engine

[Richard Lawley]

You can make wget ignore the robots.txt file by placing robots=off in
your .wgetrc file (from the wget manual:
http://qcd.nersc.gov/utilities/wget.html#SEC29)

Richard

> And I have been failed to get new emails from an ANL web site
>
(http://www-unix.mcs.anl.gov/web-mail-archive/lists/ag-tech/maillist.htm
l)
> using by 'wget' command, because of restriction by 'robot.txt'
(http://www-unix.mcs.anl.gov/robot.txt).
> 
> I'd like ANL admins to permit me getting the emails by 'wget' or tell
me another site which does not restrict 'wget'.
> 
> -- 
> Hiroyuki Komatsu