Dovecot Full-Text Search

Intro

I am using my own e-mail server for a few years now, but I almost never use search in my mailbox, so I did not really noticed how slow it was. All because the heavy lifting is left to the client to do.

Enter the FTS aka full-text search, a wonderful solution that forces server to do the dirty part, so your phone has more juice for browsing reddit.

https://en.wikipedia.org/wiki/Full-text_search

Picking the solution

My server is based on Dovecot, so first we need to find out what does it support and how can we make it work.

https://doc.dovecot.org/configuration_manual/fts/

Options:

• Apache Solr, Java and complicated, pass.
• Lucene, it’s using CLucene library, which is very old and has some bugs, pass.
• Dovecot Pro FTS, non-free, pass.
• Squat, Dovecot’s own search index. Obsolete in v2.1+, pass.
• fts-xapian, has shorter Github readme.md, so I picked this one versus Elastic based two. C++ is a plus, even two of them.

https://en.wikipedia.org/wiki/Xapian

Installation

The official installation notes are pretty self explanatory and can be found here

$ apt-get build-dep dovecot-core
$ apt-get install dovecot-dev libxapian30

Build it yourself

$ git clone https://github.com/grosjo/fts-xapian
$ cd fts-xapian

$ autoreconf -vi
$ ./configure --with-dovecot=/usr/lib/dovecot
$ make
$ sudo make install

Debian 11

The good people of Debian have packaged it all up, so this is all you need:

apt-get install dovecot-fts-xapian

Configuration

Add this to the plugin section of your /etc/dovecot/dovecot.conf file:

    ###
    ### fts-xapian - full text search (source in /opt/fts-xapian)
    ### https://github.com/grosjo/fts-xapian
    ###

	plugin = fts fts_xapian

	fts = xapian
	fts_xapian = partial=2 full=20 attachments=0 verbose=0

	fts_autoindex = yes
	fts_enforced = yes

	fts_autoindex_exclude = \Trash

Restart Dovecot:

systemctl restart dovecot

Re-index all the mailboxes:

$ doveadm index -A -q \*

Testing

You should see the speed increase in seaches immediatly.

If you check the logs, you will see that the indexing is going on as you search.

/var/log/mail/dovecot.log

Aug 18 14:29:30 sablun dovecot: indexer-worker(xo@sablun.org)<2395994><dsnRCSaazv5ejFqk:U6AOAKrJe18SmyQAv0xPeQ>: Indexed 0 messages in Archive
Aug 18 14:29:30 sablun dovecot: indexer-worker(xo@sablun.org)<2395994><sZzUCSaay/5ejFqk:GBxIAarJe18SmyQAv0xPeQ>: Indexed 0 messages in INBOX