The spider should crawl more than the specified site, to do this it needs to store outbound links. It probably needs a separate database for storing all links for future crawl.
Log in to post a comment.
