Rich Documents Processing: On the Search or Application Side

when indexing so called “rich documents” we should sometimes think about, where we want those documents to be processes – should we send them to apache solr (or other search engine, like elasticsearch) and forget about them or whether we should use apache tika before sending the document and send the extracted content along with other information for indexation.

options

as i wrote a few lines above we have two options – the first one is sending the binaries to search engine and use extractingrequesthandler (information about integrating solr with apache tika can be found here ) in solr case, so it will make all the work for us. the second option is to use the same functionality (almost the same) to parse binary documents and get their contents before sending them to solr. of course there is a third option, not possible in most cases – get the documents you want to index in a format understandable by solr

processing on the search server side

the simplest approach is to process your “rich documents” on the search server side. lets assume its apache solr. we configure the extractingrequesthandler in the way we want it to work and we forget about everything else. but its not the right approach every time. you can imagine a situation when your indexing server is almost 100% utilized. if you would add another source of generating load you would probably suffer from performance problems. in such cases you will probably want to do it the other way.

processing outside of the search server

if the amount of rich documents is huge or your indexing server is almost completely utilized than it may be a good idea to process your binary files before sending them to your indexing server. using apache tika for example we are able to build (quite easily) a good and reliable solution to process rich documents in your application. of course such approach require a bit of knowledge about java (or any other language you will use for content extraction). such approach can save us from a situation where our indexing server is overloaded and because of the amount of data we can’t do anything with it.

a few words at the end

once every few weeks we will be publishing posts that don’t cover one of the apache solr functionalities, but instead discuss some overall search problem or describe architecture of system with search as their part. we hope that such posts will allow us and you to look a bit wider on search topics than only from apache solr point of view.