Search Components Online

WordNet: An Open Source Thesaurus

The fine folks at Princeton have an online database of synonyms that can also be used with your own code.  This is pretty amazing, given how carefully guarded most of the traditional thesaurus providers have been.

See the main site, the very generous license (free for commercial use and modification under reasonable conditions), doc, stats and download.

Of course for a search application you'll need to actually do something with it.  There is a WordNet analyzer listed in the Lucene sandbox, though the link is broken.  But we have it on good authority that it does work...

Here's an article on stemming and lemmatization that includes info on thesaurus based searching, among other things.

CATMaker: Summarization Tool for Medical Text

CATMaker is an interesting tool for medical text.  It helps generate a summary of a case and a "bottom line" assessment.
http://www.cebm.net/index.aspx?o=1216

It's produced by the Center for Evidence Based Based Medicine (CEBM).  It might be open source.

Summary tools can be very helpful when used with search engines.  They can help give a better summary in a results list, or when used at index time, give the search engine a smaller, more relevant set of text to search over.

Java tool for NLP

In Natural Language Processing, search indexing, and document processing, you often need to identify the language in which the document is written before you can do any significant processing. Cedric Champeau has posted a handy Java language tool on his blog that lets you identify the language using an n-gram tokenizer. From his page:

JLangDetect is based on n-gram tokenization. Basically, texts are tokenized with different token sizes. For example, given the text "cat", n-gram tokenization for 1 to 3 token sizes will produce the following tokens :

• c
• a
• t
• ca
• at
• cat

The idea is to tokenize a large set of documents in a given language and record token statistics. When you need to identify a language, then you'll tokenize it the same way, and you'll be able to score the input string against several token stats.

The tool is licensed under the Apache 2.0 license. Links to download the tools are provided on his page. For your convenience:

• Binary : jlangdetect-0.1.jar
• Source: jlangdetect-0.1-sources.jar
• Javadoc: jlangdetect-0.1-javadoc.jar
• Europarl pre-compiled corpus: ngrams-europarl.zip

Indexing SharePoint connectors for Documentum and FileNet

Microsoft has announced the availability of two free indexing connectors for its Search Server and Search Server Express products. The same search kernel is embedded in SharePoint 2007, so these indexing connectors should let you search EMC's Documentum and IBM;s FileNet content from within SharePoint 2007. Note these indexing connectors are not yet available for FAST ESP - at least not from Microsoft.

Download the Documentum connector here, or the FileNet connector here.

The Microsoft Enterprise Search Blog has links to some cool installation videos for Documentum and FileNet as well (requires SilverLight).

OpenPipeline.org seeks to Standardize Document Processing

Our friends over at Dieselpoint have launched an open source initiative at OpenPipeline.org to try and make the process of filtering and parsing documents a more orderly process.  More and more vendors are using the pipeline model, we even did an article about the FAST pipeline some time back.  Go Chris!

http://openpipeline.org

JQuery Tips on Text Searchng

Johann Burkard has published a number of JQuery tips in two postings on text manipulation and on text searching as just two of the interesting posts on JavaScript, Java and other technology topics. His first post shows simply how to remove a word using the replace method (based on our favorite, regex), while his second post has a great little function for searching for text that uses a recursive search for nodes that contain a pattern. It includes a cool demo of its capabilities as a link right under the code listing.

As the say, write less, Do More with JQuery.