Niocchi

Niocchi is a java asynchronous crawl library implemented with NIO. It is designed to crawl several thousands of hosts in parallel on a single low end server.

It is currently being used in production by Enormo to crawl thousands of websites daily, and by Vitalprix.

Index

Introduction
Requirements
License
Package organization
Architecture
Usage
Caveats
To Do
Download
About the Authors

Introduction

Most of the java crawling libraries use standard synchronous java IO. That means crawling N documents in parallel requires at least N running threads. Even if each thread is not taking a lot of resources while fetching the content, that approach becomes costly when crawling at a large scale. On the contrary, doing asynchronous I/O by using the NIO package introduced in java 1.4 allows the crawling of many documents in parallel using one single thread.

Requirements

Niocchi requires java 1.5 or above.

License

This software is licensed under the Apache license version 2.0.

Package organization

org.niocchi.core holds the library itself.
org.niocchi.gc holds an implementation example of a very simple crawler that reads the URL to crawl from a file and saves the crawled documents.
org.niocchi.monitor holds a utility thread that can be used by the crawler to provide real time information through a telnet connexion.
org.niocchi.rc holds an implementation example of a RedirectionController.
org.niocchi.resources holds a few implementation examples of the Resource and ResourceCreator classes.
org.niocchi.urlpools holds a few implementation examples of the URLPool class.

Architecture

A Query encapsulates a URL and implements methods to check its status after being crawled.
A Resource holds the crawled content and implements methods to save it.
Each Query is associated to a Resource. To crawl one URL, one Resource needs to be taken from the pool of resources. Once the URL is crawled and its content processed, the Resource is returned to the pool. The number of available Resources is fixed and controls how many URL can be crawled in parallel at any time. This number is set through the ResourcePool constructor.
When a Query is crawled, its associated Resource will be processed by one of the workers.
The URLPool acts as a source of URLs to crawl into which the crawled taps. It's an interface that must be implemented to provide URLs to the crawler.
The crawler has been designed as "active", meaning it consumes URLs from the URLPool, as opposed to being "passive" and waiting to be given URL. When the crawler starts, it will get URLs to crawl from the URLPool until all resources are consumed or hasNextQuery() return false or getNextQuery() returns null. Each time a Query is crawled and processed and its Resource returned to the ResourcePool, the crawler requests other URLs to crawl from the URLPool until all resources are consumed or hasNextQuery() return false or getNextQuery() returns null. If all URLs have been crawled and no more are immediately available to crawl, the crawler will recheck every second for available URLs to crawl.
When a Query has been crawled, it is put into a FIFO of queries to be processed. One of the Workers will take it and process the content of its associated Resource. The work is done in the processResource() method. The Query is then returned to the URLPool which can examine the crawl status and the result of the processing. Lastly, the Query associated Resource is returned to the Resource pool.
In order to not block during host name resolution, the Crawler uses two additional threads. ResolverQueue resolves the URL coming from the URLPool and RedirectionResolverQueue resolves the URLs gotten from redirections.

This architecture is represented in the following diagram:

Usage

In order to use Niocchi, the following interface and abstract classes must be implemented.

Resource and ResourceCreator

Subclass these classes and implement Resource.isValid() and ResourceCreator.createResource() or use one of the two provided implementations for HTML and pictures types of resources.

Worker

Subclass Worker and implement processResource(Query). This is where you do whatever needs to be done with the crawled content. Check the DiskSaveWorker class for a example of implementation.
You will usually instanciate 1 worker per CPU core.

URLPool

Implement the URLPool interface.

getNextQuery() returns a Query to crawl or null if there isn't any available Query yet.
hasNextQuery() returns false only when there are no more queries to crawl and the crawler must terminate after the last queries still being crawled are processed.
setProcessed(Query) is called by the crawler to inform the URLPool when a Query has been crawled and its resource processed. This is typically used by the URLPool to check the crawl status and log the error in case of a failure or to get more URL to crawl in case of success. A typical example where getNextQuery() returns null but hasNextQuery() returns true is when the URLPool is waiting for some processed resources from which more URL to crawl have been extracted to come back. Check the urlpools package for examples of implementation.

Caveats

Niocchi crawls the content in memory. Each Resource has a buffer which default size is 100KB. That means the crawler will eat up 100 x number of allocated resources in the pool KB. If a document is too big to fit in this buffer, the Query getStatus() method will return the INCOMPLETE (-2) error code. The default size can be changed with the static method Resource.setCapacity().
Each socket consumes one file descriptor. If you intend to crawl a large number of documents in parallel and reach the default limit of the system, you can raise it with ulimit.
There is no control regarding how many URL can be crawled for a same host in parallel implemented in this version. Crawl regulation and politeness has to be implemented in the URLPool.

To Do

Lots of documentation to write.
Improve the ResolverQueue to resolve host names in parallel by using a pool of threads.
Add an option to crawl resources on disk instead of in memory.

Download

Niocchi 1.0
Apache Commons Logging

About the Authors

Niocchi has been written by François-Louis Mommens and has received contributions by Iván de Prado Alonso and Marc Gracia