Distributed crawling has shown that it can overcome important
limitations of the centralized crawling paradigm. However, the
distributed nature of current distributed crawlers is not fully
utilized. The optimal benefits of this approach are usually limited to
the sites hosting the crawler. We describe IPMicra, a distributed
location aware web crawler that utilizes an IP address hierarchy and
allows crawling of links in a near optimal location aware manner. The
crawler outperforms earlier distributed crawling approaches without a
significant overhead.