Comments on: Django and full-text search

By: Francois

Francois — Sat, 12 Dec 2009 16:26:28 +0000

After what’s probably the longest moderation time in recorded history (err, so sorry about that…), I’ll try an answer, which is the least I can do: after all this search, I was still using the LIKE statement (well, the “__icontains” filter in Django). IIRC the idea was that at some point I could create a MySQL index, if the need ever arose for faster search (but it didn’t).

Here’s the source as used in Clusterify, now that it’s on Github:

http://github.com/clusterify/clusterify/blob/master/utils.py

And example of use (search for “get_query”):

http://github.com/clusterify/c.....s/views.py

By: Joseph Turian

Joseph Turian — Mon, 08 Jun 2009 00:25:12 +0000

Francois, I have also heard about haystack. It looks very django-y, with Whoosh and Solr backends. However, I have been having some difficulty getting it working just yet.

What solution are you currently using?

By: piramida

piramida — Thu, 19 Feb 2009 19:58:15 +0000

Thanks for the links, have not heard about Whoosh – would follow it’s development. If that happens in a django module there would be much more ways to manipulate the process in case it starts slowing the system down Run a separate indexer thread watching the db or having a queue is one idea. Direct ORM access to data creates many easy solutions.

By: francois

francois — Thu, 19 Feb 2009 15:35:07 +0000

Quickly looking at the source, though, I think the part of your comment concerning signals is not addressed: I might be wrong, but it seems indexing will happen directly while the HTTP request is processed (ie. synchronously, in the same thread).

By: francois

francois — Thu, 19 Feb 2009 15:18:50 +0000

Actually there are pure Python indexing engines, but all I’ve seen have a “slower than C counterparts” mention somewhere. See for example this thread:

http://stackoverflow.com/questions/438315/is-there-a-pure-python-lucene

which seems to point to Whoosh (http://whoosh.ca/) as the best alternative right now.

More quick searching brought this: it seems a week ago someone started a project on Github to make a Django-Whoosh module:

http://github.com/ericflo/djan.....ree/master

Maybe the solution _is_ on the way.

By: piramida

piramida — Thu, 19 Feb 2009 08:56:15 +0000

Maybe it’s time to publish a project description on your new site for a django module which would do good indexed relevance-sorted FT searching done in python (maybe embedding C if needed but I think python would work)? Have an indexer task running listening for changes and updating index for all models marked for indexing.

The problem is real, we’re using Lucene but having to keep Tomcat up just for that and having on save and on delete signals syncing the index through POST requests is a completely bad django-fu, so I’d be glad to get rid of it.