We are using Oracle Text for full text search. Now that we are using other databases such as DB2, MySQL, etc, we need to consider using the native database API, or standardize on the 3rd party full-text search software.
We had a look at Sphinx Search. This provides direct APIs to MySQL, and ODBC to other databases. Their most well-known user is Craiglist. Looks very scalable, especially if you can shard the indexes.
Features we like:
– Fast-up to 4 times faster than MySQL full-text search.
– Ability to configure the score, giving more importance to different fields (e.g. weighted fields).
– Largest web-site using Sphinx handles a database of 4 TB of documents.