iframe contents ignored

[Garrett-R]

Iframes are sometimes used to have parts of sites controlled by a CMS.

It would nice to have the option of inspecting the iframe's content and for any links that are to the site being indexed, having those included.

It would have to take into account the tag, since if the base tag matches the site being indexed, then all relative URLs should be crawled.