Search Components Online

Java HTML Parsers

Before you index your content into a search engine, you've gotta parse it.

Since many HTML pages either don't conform to the strict XHTML standards, or have blatant typos, you typically can't use JDOM or other XML tools to edit them.  And there are some HTML pages that even JTidy can fix.

Here are some other resources:

Open source list of HTML Parsers
http://java-source.net/open-source/html-parsers

Java's own HTML parser (surprisingly, part of the Swing UI kit)

• javax.swing.text.html.HTMLEditorKit
• http://www.java2s.com/Code/Java/Development-Class/HTMLDocumentElementIteratorExample.htm
• http://www.java2s.com/Tutorial/Java/0120__Development/UsejavaxswingtexthtmlHTMLEditorKittoparseHTML.htm
• http://www.java2s.com/Tutorial/Java/0120__Development/ParseHTML.htm
• http://www.rgagnon.com/javadetails/java-0424.html
• http://www.rgagnon.com/javadetails/java-0639.html
• http://forums.java.net/jive/message.jspa?messageID=58806
• http://java.sun.com/products/jfc/tsc/articles/bookmarks/

And one related link with a ton of examples speaking HTTP
http://jan.newmarch.name/distjava/http/lecture2.html

Java tool for NLP

In Natural Language Processing, search indexing, and document processing, you often need to identify the language in which the document is written before you can do any significant processing. Cedric Champeau has posted a handy Java language tool on his blog that lets you identify the language using an n-gram tokenizer. From his page:

JLangDetect is based on n-gram tokenization. Basically, texts are tokenized with different token sizes. For example, given the text "cat", n-gram tokenization for 1 to 3 token sizes will produce the following tokens :

• c
• a
• t
• ca
• at
• cat

The idea is to tokenize a large set of documents in a given language and record token statistics. When you need to identify a language, then you'll tokenize it the same way, and you'll be able to score the input string against several token stats.

The tool is licensed under the Apache 2.0 license. Links to download the tools are provided on his page. For your convenience:

• Binary : jlangdetect-0.1.jar
• Source: jlangdetect-0.1-sources.jar
• Javadoc: jlangdetect-0.1-javadoc.jar
• Europarl pre-compiled corpus: ngrams-europarl.zip

PDF Text Extraction Tool

FreewareFiles.com has a nice PDF text extraction tool.  From the description:

Ease Pdf to Text Extractor is a free software designed to extract text from Adobe PDF files. It does NOT need Adobe Acrobat software.It processes at very high speed and you can convert multiple PDF files to text files at one time. The program is freeware, which means that you can use it either personally or commercially for free.

Warning: a few dynamic ads on this site.
http://www.freewarefiles.com/Ease-PDF-To-Text-Extractor_program_43618.html

JQuery Tips on Text Searchng

Johann Burkard has published a number of JQuery tips in two postings on text manipulation and on text searching as just two of the interesting posts on JavaScript, Java and other technology topics. His first post shows simply how to remove a word using the replace method (based on our favorite, regex), while his second post has a great little function for searching for text that uses a recursive search for nodes that contain a pattern. It includes a cool demo of its capabilities as a link right under the code listing.

As the say, write less, Do More with JQuery.

Joomla CMS meets free IBM/Yahoo Search Engine

The free IBM/Yahoo search engine has been popular since it shipped last year.  And of course the open source Joomla! CMS continues to go like gang busters.  But what if they every got together?  Dream no more!

The folks over at TnR have got you covered with a new connector.  From their site:

TnR Global Enterprise Search Component for Joomla driven by IBM Yahoo! OmniFind Edition is a standard Joomla component that can be installed into the Joomla open source content management system. The component (and a matching search module) provide the Joomla site with direct access to an OmniFind search system. The Joomla administrator can define the look of the search box and the search results using a standard Smarty template. The OmniFind enterprise search can be placed on any server the Joomla system can reach, allowing it to be behind a corporate firewall.

http://www.tnrsearch.com/index.php/Latest/Esearch-Joomla-Component.html

Free MySQL Table Optimizer from Erik Hallander

This guy's website is pretty interesting too.

http://www.erikhallander.com/blog/2008/free-stuff-automatic-mysql-table-optimizer.html

db Software Laboratory's free Database Browser

This looks like a really nice tool for database administrators.

Although this download is free, their site has a lot of other cool stuff (not free) for ETL ("Extract, Transform, Load") and database conversion tools.

http://www.dbsoftlab.com/index.php/Freeware/Database-Browser.html

PureText: Desktop Tray Icon / Utility Removes Text Formatting for Copy and Paste

This utility sits in your Windows Desktop tray and gives you a new hotkey to paste text between applications without all the annoying formatting.

http://www.stevemiller.net/puretext/

PDF to MS Word, HTML or Text

These folks are offering quite a few free and low cost file conversion tools for PDF, which would be handy to have if you were writing a web spider that needed to handle PDF.

http://pdf-to-html-word.com/download.htm then see the download page.

Indian company offering Free Desktop Search

Indian software company Crave Worldwide is offering their "ExtraSearch FREE" desktop engine.

From their Website:

ExtraSearch Free is a unique search tool which allows to search for files on your PC and filter the search results using different filters and criterias.
ExtraSearch allows you to search files and filter search results using file name and extension (with wildcards), file date, attributes, file size and contents of the file too.  You can easily export the search results to an HTML file, CSV file (easy to import in excel) and text file.
ExtraSearch Free provides a powerful set of tools to double search files on your PC.

http://www.cwwsoft.com/extrasearch-free.html