This document describes the state of utility sitegrab.

At the moment it is a command line utility only.

You can get quite a bit of information on this utility by reading
submit.html.

Usage is:

  sitegrab <site-url> <url-prefix> <max-depth>

eg:

  sitegrab http://www.dcs.ex.ac.uk/~aba/ http://www.dcs.ex.ac.uk/ 3
  sitegrab http://www.dcs.ex.ac.uk/~aba/rsa/ http://www.dcs.ex.ac.uk/~aba/ 10

Can also hack file: urls, and (wow) eternity urls via an eternity server
(not sure why you'd want to do this mind).

sitegrab is sensible about recursion.  It remembers where it's been and
won't go there again.  It tries to normalize urls, but if you get creative
you could probably confuse it into visiting the same page twice.

It just prints out a list of urls it could find linked to from the site/page
you point it at.

url-prefix is a string which all urls to be followed must have as a prefix.
For example if I had a link to the www.dcs home page from my web pages, the
first example above would start wandering around that page.  This is
probably not what you want.  The second example restricts things to being
under my home page.

max-depth is the maximum level of recursion to allow.  0 is no recursion,
and just includes inline images, and the page itself.  1 includes pages
linked directly from the site main page, and their inline images, as well as
the main page itself, and inline images linked on the main page.



