We had a look at Sphinx Search. This provides direct APIs to MySQL, and ODBC to other databases. Their most well-known user is Craiglist. Looks very scalable, especially if you can shard the indexes.