Search Components Online

- Carrot^2: Source for Free Federated Search Connectors

It ships with something like 12 predefined federation connectors... need to find the term they use... that ship with it.

Needs link...

ref. to Screen Scraping....

Java HTML Parsers

Before you index your content into a search engine, you've gotta parse it.

Since many HTML pages either don't conform to the strict XHTML standards, or have blatant typos, you typically can't use JDOM or other XML tools to edit them.  And there are some HTML pages that even JTidy can fix.

Here are some other resources:

Open source list of HTML Parsers
http://java-source.net/open-source/html-parsers

Java's own HTML parser (surprisingly, part of the Swing UI kit)

• javax.swing.text.html.HTMLEditorKit
• http://www.java2s.com/Code/Java/Development-Class/HTMLDocumentElementIteratorExample.htm
• http://www.java2s.com/Tutorial/Java/0120__Development/UsejavaxswingtexthtmlHTMLEditorKittoparseHTML.htm
• http://www.java2s.com/Tutorial/Java/0120__Development/ParseHTML.htm
• http://www.rgagnon.com/javadetails/java-0424.html
• http://www.rgagnon.com/javadetails/java-0639.html
• http://forums.java.net/jive/message.jspa?messageID=58806
• http://java.sun.com/products/jfc/tsc/articles/bookmarks/

And one related link with a ton of examples speaking HTTP
http://jan.newmarch.name/distjava/http/lecture2.html

PDF to MS Word, HTML or Text

These folks are offering quite a few free and low cost file conversion tools for PDF, which would be handy to have if you were writing a web spider that needed to handle PDF.

http://pdf-to-html-word.com/download.htm then see the download page.