Integrating Apache Nutch With Apache Solr on Ubuntu Server

In our previous tutorials, we written the steps to install Apache Nutch on Ubuntu Server and also how to install Apache Solr on Ubuntu Server. Integrating Apache Nutch With Apache Solr Will Offer a Web UI, Options to Visually Search and Use Extended Functions of Apache Nutch. Our guide on installing Apache Solr uses older version of Solr (at present). We are using Apache Nutch 1.x – in previous articles we discussed that Apache Nutch 1.x and Apache Nutch 2.x has difference and practically they are kind of “different software”.

 

Integrating Apache Nutch With Apache Solr

 

You must read our previous guides to understand what we are talking about. The seed URLs file we created named seed.txt under urls/ directory, if has following content :

Then we can configure Regular Expression Filters by opening the file conf/regex-urlfilter.txt with our needed regex, like :

That is for limiting crawl, adding logic of crawl etc. Download DMOZ RDF dump from somewhere (DMOZ officially closed), it has zillions of URLs. If that RDF dump’s name is content.rdf.u8.gz, then do these steps (we already shown these kind of steps in earlier guides) :

We ended the Nutch guide after the above step explaining segments. Here are advanced steps :

We already have Apache Solr installed, take that installation is at $HOME/apache-solr, which we can refer as ${APACHE_SOLR_HOME}.

create resources for a new Nutch Solr core :

Copy the Nutch’s schema.xml into the conf directory :

There should not be managed-schema :

Now, start the Solr server :

Create the nutch core :

Add the core name to the Solr server URL in Nutch’s configaration file :

Now, we can pass through reverse proxy for server to reach the front-end which for localhost is at URL :

We can run commands like :

We can run a crawl with Nutch and index the data in Solr. Using the same seed list :

After the crawl, the data will be indexed in Solr. We can view it via Solr webUI. Go to http://localhost:8983/solr, load the nutchCore, on left hand side of panel, click on Query to search documents in the indexed data. *:*. That return things indexed by Solr.