25 June 2013 by Published in: Code 140 comments

You may be concerned that the NSA is reading your e-mail. Is there really anything you can do about it though? After all, you don’t really want to move off of GMail / Google Apps. And no place you would host is any better.

Except, you know, hosting it yourself. The way that e-mail was originally designed to work. We’ve all just forgotten because, you know, webapps-n-stuff. It’s a lot of work, mkay, and I’m a lazy software developer.

Today we kill your excuses. Because I’m going to show you exactly how to do it, it’s going to take about two hours to set up, and it’s a “set it and forget it” kind of setup. Not only that, but it is actually going to be better than GMail, from a purely features perspective. It might surprise you to learn that people continue to develop email server software in a post-Google-apps world, and that the state of self-hosted is much better than you remember.

Now fair warning: it took me about two days to figure the stuff out you’re going to see in this blogpost, starting from knowing basically nothing about modern e-mail servers. But now that I’ve figured it out, if you don’t ask too many questions you can implement it from these notes in just two hours. So take this not just as a guide for setting up an e-mail server, but as two days of free consulting, that just happens to produce a complete recipe for a modern, fully-featured, fast email server at the end. You’re really going to turn down free consulting? Come on, buckle down and do this.

So bookmark this blog post, block off a Saturday next month, and get it done. Seriously. If you are still using GMail (or Yahoo, or arbitrary US-based email company) in August, your right to complain about the NSA spying on you is revoked. If you’re complaining about government spying on the Internet, or in a gathering of programmers, and you won’t take basic steps to do anything about it, then you’re a hypocrite, full-stop. I will personally come to your terminal and demand the return of your complain license. Pick a weekend, get it done. Or just admit that you’re okay with it. Either way, just be consistent.

Edit Some people are complaining that the NSA is pulling all the e-mail over the wire anyway, so encrypting your own server is stupid. However, these people are not aware of just how much e-mail goes over TLS. For example, if you follow this guide, every message to or from a GMail / Google Apps account goes over TLS. I measured it, and that’s a third of my ham volume. And Google isn’t a special snowflake; any reasonably-configured mail will be encrypted over the wire. It’s 2013; people have been getting TLS into wide adoption for awhile now.

In the next two hours, we’re going to fix this.  You’re going to build a modern, 2013, e-mail stack. From scratch. Or something.

  • You’re going to host your own mail.
  • It’s going to be encrypted on the server, locked-on-boot, SSH on reboots to unlock
  • While we’re at it, let’s fix some things that annoy me about GMail:
    • Better SPAM detection.  Yes, you can beat the Big G.
    • Push support that actually works.  Little known fact: Google discontinued push for iOS earlier this year.  Also little known fact: push has never worked in OSX Mail.app.  We’re gonna fix it.  Lightning fast push on all the devices.
    • Full-text search. Believe it or not, mail search on my iPhone is pretty bad under GMail. Three out of four Google Suggest results for “iPhone mail search” are “doesn’t work”, “not working”, and “slow”. Our version is going to be working, working, and fast. Take that, Google.

I’m going to assume:

  • You’re running Debian Wheezy
  • You know your way around Linux more or less. Experience running Apache or Lighttpd or Nginx, etc.
  • You have root access
  • Nobody really cares if you mess anything up.
  • You’re not stupid and you know how to google things

Hang on just a minute though. Doesn’t hosting your own mail require beefy hardware to deal with the spams? Not really. I get a lot of mail, and this is my graph:

Screen Shot 2013-06-13 at 4.24.05 PM

Doesn’t it go down a lot though? Getting up in the middle of the night to do server reboots and that?

Again, not really. RFC 2821 requires that mail servers try for “at least 4-5 days” to deliver your mail. Let’s just say, that’s a lot more forgiving than running Apache.

A crash course on e-mail

The people who have designed e-mail software are, on the whole, people with a lot more problems than you.  Unfortunately this means there are a lot of separate little projects that work together.

  • Postfix – this is called an “MTA” in the email lingo.  Essentially it runs SMTP, and delivers incoming mail to Dovecot.
  • Dovecot – this is the “LDA” in the email lingo.  Essentially it runs IMAP.
  • EncFS – this is what we use to encrypt our email store
  • opendkim – implements DKIM authentication
  • spamd – antispam
  • z-push – better push support
  • mySQL – really we’re going to use this to hold information about users and domains, but nothing really sensitive.

Let’s get started!

Security Foreward

I assume you kinda-sorta know how to configure a secure server on the Internet and have done so for your mail server. If you don’t know what I’m talking about, time to read my first 5 minutes on a server.

DNS

We should probably start configuring your DNS, since that stuff takes awhile. If you have an existing domain, you might want to lower the TTL on your MX records to the smallest possible setting. That way you can switch over when you’re ready.

For this article, I’m going to set up my mailserver running on a subdomain – drew@awesomebox.sealedabstract.com. So I configure an MX record that points mail for the subdomain to my server.

EncFS

$ apt-get install encfs

mkdir /encrypted-mail /decrypted-mail
chgrp mail /decrypted-mail/
chmod -R g+rw /decrypted-mail/
gpasswd -a mail fuse
chgrp fuse /dev/fuse; chmod g+rw /dev/fuse


root@li212-205:~# encfs /encrypted-mail /decrypted-mail -o --public
Creating new encrypted volume.


Please choose from one of the following options:
 enter "x" for expert configuration mode,
 enter "p" for pre-configured paranoia mode,
 anything else, or an empty line will select standard mode.
?> p

Paranoia configuration selected.

Configuration finished.  The filesystem to be created has
the following properties:
Filesystem cipher: "ssl/aes", version 3:0:2
Filename encoding: "nameio/block", version 3:0:1
Key Size: 256 bits
Block Size: 1024 bytes, including 8 byte MAC header
Each file contains 8 byte header with unique IV data.
Filenames encoded using IV chaining mode.
File data IV is chained to filename IV.
File holes passed through to ciphertext.

-------------------------- WARNING --------------------------
The external initialization-vector chaining option has been
enabled.  This option disables the use of hard links on the
filesystem. Without hard links, some programs may not work.
The programs 'mutt' and 'procmail' are known to fail.  For
more information, please see the encfs mailing list.
If you would like to choose another configuration setting,
please press CTRL-C now to abort and start over.

Now you will need to enter a password for your filesystem.
You will need to remember this password, as there is absolutely
no recovery mechanism.  However, the password can be changed
later using encfsctl.

New Encfs Password: 
Verify Encfs Password:

It’s that simple.  /decrypted-mail is now a regular directory.  /encrypted-mail is that same data, just encrypted.

It’s important to use the --public option with EncFS. This is because by default, EncFS goes to great lengths to be a lot narrower than the standard UNIX permissions model. Meanwhile, we have a lot of UNIX users who expect standard UNIX behaviors.

Any time we want to mount the /decrypted-mail dir, we run the same encfs command again, and this time it mounts the folder, instead of creating it.

OpSec

It’s important as a practical matter that you keep other people off your root. It goes without saying, but there are a lot of attack vectors (like physical access, or access through a VPS admin panel) that could potentially allow your host, acting under the direction of a government agent, to run commands as root, and take your mail.

For security reasons, I’m not going to disclose the measures that I take to avoid others gaining root on my system. A good start might be changing your root password, or keeping your mail server under your pillow at night.

Postfix

apt-get install postfix postfix-mysql dovecot-core dovecot-imapd dovecot-mysql mysql-server dovecot-lmtpd

Postfix prompts you with a bunch of information–the one that you want is “Internet Site”. It’s also the default.

Postfix prompts you for a “mail name” – I chose awesomebox.sealedabstract.com.

About 60 seconds later, you’ve got a mail server. Now to configure it. First we need to derive our password.

doveadm pw -s SHA512-CRYPT

This will give you a string like {SHA512-CRYPT}$6$gJ8hXjMn/lePALEt$JMX1jd... The part after “{SHA512-CRYPT}” is the hash for your password. It always starts with “$6$”.

mysqladmin -p create mailserver
mysql -p mailserver
mysql> GRANT SELECT ON mailserver.* TO 'mailuser'@'127.0.0.1' IDENTIFIED BY 'mailuserpass';
mysql> FLUSH PRIVILEGES;

mysql> CREATE TABLE `virtual_domains` (
  `id` int(11) NOT NULL auto_increment,
  `name` varchar(50) NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

mysql> CREATE TABLE `virtual_users` (
  `id` int(11) NOT NULL auto_increment,
  `domain_id` int(11) NOT NULL,
  `password` varchar(106) NOT NULL,
  `email` varchar(100) NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `email` (`email`),
  FOREIGN KEY (domain_id) REFERENCES virtual_domains(id) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

mysql> CREATE TABLE `virtual_aliases` (
  `id` int(11) NOT NULL auto_increment,
  `domain_id` int(11) NOT NULL,
  `source` varchar(100) NOT NULL,
  `destination` varchar(100) NOT NULL,
  PRIMARY KEY (`id`),
  FOREIGN KEY (domain_id) REFERENCES virtual_domains(id) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

mysql> INSERT INTO `mailserver`.`virtual_domains`
  (`id` ,`name`)
VALUES
  ('1', 'awesomebox.sealedabstract.com');

You could also configure additional domains here if desired.

mysql> INSERT INTO `mailserver`.`virtual_users`
  (`id`, `domain_id`, `password` , `email`)
VALUES
  ('1', '1', '$6$YOURPASSWORDHASH', 'drew@awesomebox.sealedabstract.com');

Again, you could also configure multiple users here.

mysql> exit

Okay, now let’s configure postfix. Let’s back up the original file:

cp /etc/postfix/main.cf /etc/postfix/main.cf.orig
nano /etc/postfix/main.cf

Comment out the “TLS parameters”, and use these instead:

smtpd_tls_cert_file=/etc/ssl/certs/dovecot.pem
smtpd_tls_key_file=/etc/ssl/private/dovecot.pem
smtpd_use_tls=yes
smtpd_tls_auth_only = yes
smtp_tls_security_level = may
smtp_tls_loglevel = 2
smtpd_tls_received_header = yes

Below the TLS parameters, paste these:

smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes

smtpd_recipient_restrictions =
        permit_sasl_authenticated,
        permit_mynetworks,
        reject_unauth_destination

There’s a line that sets “mydestination” to a wide variety of domains. Make sure it’s only set to localhost.

mydestination = localhost

At the very bottom of the file, paste this:

virtual_transport = lmtp:unix:private/dovecot-lmtp
virtual_mailbox_domains = mysql:/etc/postfix/mysql-virtual-mailbox-domains.cf
virtual_mailbox_maps = mysql:/etc/postfix/mysql-virtual-mailbox-maps.cf
virtual_alias_maps = mysql:/etc/postfix/mysql-virtual-alias-maps.cf
local_recipient_maps = $virtual_mailbox_maps

This essentially points postfix at Dovecot and the MySQL tables.

Okay, so now we’re going to create the three files we specified earlier.

nano /etc/postfix/mysql-virtual-mailbox-domains.cf

.

user = mailuser
password = mailuserpass
hosts = 127.0.0.1
dbname = mailserver
query = SELECT 1 FROM virtual_domains WHERE name='%s'

Similarly, for /etc/postfix/mysql-virtual-mailbox-maps.cf:

user = mailuser
password = mailuserpass
hosts = 127.0.0.1
dbname = mailserver
query = SELECT 1 FROM virtual_users WHERE email='%s'

Finally, for /etc/postfix/mysql-virtual-alias-maps.cf:

user = mailuser
password = mailuserpass
hosts = 127.0.0.1
dbname = mailserver
query = SELECT destination FROM virtual_aliases WHERE source='%s'

So let’s restart postfix:

service restart postfix

And verify that it works:

$ postmap -q awesomebox.sealedabstract.com mysql:/etc/postfix/mysql-virtual-mailbox-domains.cf
1
$ postmap -q drew@awesomebox.sealedabstract.com mysql:/etc/postfix/mysql-virtual-mailbox-maps.cf
1

Looks good.

Dovecot

So first, let’s backup all the config files:

cp /etc/dovecot/dovecot.conf /etc/dovecot/dovecot.conf.orig
cp /etc/dovecot/conf.d/10-mail.conf /etc/dovecot/conf.d/10-mail.conf.orig
cp /etc/dovecot/conf.d/10-auth.conf /etc/dovecot/conf.d/10-auth.conf.orig
cp /etc/dovecot/dovecot-sql.conf.ext /etc/dovecot/dovecot-sql.conf.ext.orig
cp /etc/dovecot/conf.d/10-master.conf /etc/dovecot/conf.d/10-master.conf.orig
cp /etc/dovecot/conf.d/10-ssl.conf /etc/dovecot/conf.d/10-ssl.conf.orig

And edit the main one:

nano /etc/dovecot/dovecot.conf

Down at the bottom, we’re going to enable imap:

protocols = imap

So far, so good. Now edit another config file:

nano /etc/dovecot/conf.d/10-mail.conf

We patch the variables listed below:

mail_location = maildir:/decrypted-mail/%d/%n
mail_privileged_group = mail
first_valid_uid = 0

Now we edit the auth file:

nano /etc/dovecot/conf.d/10-auth.conf

Here are the values we patch:

disable_plaintext_auth = yes
auth_mechanisms = plain login
#INSERT a hashtag in front of the following import.  This separates your mail server's login from UNIX logins.
#!include auth-system.conf.ext
#REMOVE the hashtag in front of the following import.  This points it at mysql for authentication.
!include auth-sql.conf.ext

Now let’s configure that SQL in more detail:

nano /etc/dovecot/conf.d/auth-sql.conf.ext

.

passdb {
  driver = sql
  args = /etc/dovecot/dovecot-sql.conf.ext
}
userdb {
  driver = static
  args = uid=mail gid=mail home=/decrypted-mail/%d/%n
}

Edit yet another config file

nano /etc/dovecot/dovecot-sql.conf.ext

.

driver = mysql
connect = host=127.0.0.1 dbname=mailserver user=mailuser password=mailuserpass
default_pass_scheme = SHA512-CRYPT
password_query = SELECT email as user, password FROM virtual_users WHERE email='%u';

Now I know what you’re thinking. SHA512? Why not bcrypt?

In theory dovecot supports the argument BLF-CRYPT here (e.g. bcrypt) but in practice only if your libc provides bcrypt. SUSE is the only Linux that has bcrpt in its libc. This topic comes up enough that Ulrich Drepper has an entire webpage on this topic.

In practice you can do any of the following: A) use SHA512-CRYPT, B) Switch to SUSE, C) Switch to Real Unix, D) whine at Ulrich Drepper, E) whine at NIST. I’ve chosen choice A. If it makes you feel any better, it’s been vetted by HP, IBM, RH, and Sun.

chown -R mail:dovecot /etc/dovecot
chmod -R o-rwx /etc/dovecot

.

nano /etc/dovecot/conf.d/10-master.conf

We’re setting our ports to zero. This essentially forces people to use “secure” sockets. You know, cause we’re paranoid.

service imap-login {
  inet_listener imap {
    port = 0  
  }
…

service pop3-login {
  inet_listener pop3 {
    port = 0  
  }
...

For the service lmtp, we use as follows:

service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    mode = 0666
    group = postfix
    user = postfix
  }

  # Create inet listener only if you can't use the above UNIX socket
  #inet_listener lmtp {
    # Avoid making LMTP visible for the entire internet
    #address =
    #port =
  #}
  user=mail
}

We’ll replace the whole “service auth” and “service auth-worker” section with this one:

service auth {
  # auth_socket_path points to this userdb socket by default. It's typically
  # used by dovecot-lda, doveadm, possibly imap process, etc. Its default
  # permissions make it readable only by root, but you may need to relax these
  # permissions. Users that have access to this socket are able to get a list
  # of all usernames and get results of everyone's userdb lookups.
  unix_listener /var/spool/postfix/private/auth {
    mode = 0666
    user = postfix
    group = postfix
  }

  unix_listener auth-userdb {
    mode = 0600
    user = mail
    #group =
  }

  # Postfix smtp-auth
  #unix_listener /var/spool/postfix/private/auth {
  #  mode = 0666
  #}

  # Auth process is run as this user.
  user = dovecot
}
service auth-worker {
  # Auth worker process is run as root by default, so that it can access
  # /etc/shadow. If this isn't necessary, the user should be changed to
  # $default_internal_user.
  user = mail
}

Generate those SSL certs:

openssl req -new -x509 -days 1000 -nodes -out "/etc/ssl/certs/dovecot.pem" -keyout "/etc/ssl/private/dovecot.pem"

And set them in the config file:

nano /etc/dovecot/conf.d/10-ssl.conf
ssl_cert = </etc/ssl/certs/dovecot.pem
ssl_key = </etc/ssl/private/dovecot.pem
ssl = required

Let’s kick the server:

service dovecot restart

At this point, it should basically work. You should be able to send and receive mail. Go ahead and try it! You should be running IMAP on 993 (standard SSL port) and you should be able to log in with the e-mail and password you set.

Note that you can get debug information with

tail /var/log/mail.log

You can also test “over-the-wire TLS”. When you send a message, mail.log should show this:

Jul 15 19:20:52 li212-205 postfix/smtp[17453]: Untrusted TLS connection established to ASPMX.L.GOOGLE.com[2607:f8b0:4003:c02::1b]:25: TLSv1.2 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)

When you receive a message, this header will appear:

Received: from mail-ob0-f169.google.com (mail-ob0-f169.google.com [209.85.214.169]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mailserver.com (Postfix) with ESMTPS id 7F90E1CC71D for <drew@sealedabstract.com>; Mon, 15 Jul 2013 19:28:07 -0400 (EDT)

However, we should set up some more things.

Getting through SPAM filters

If setting up an e-mail server was that easy, spammers would do it. Alas.

You should go read Jeff Atwood’s ‘so you’d like to send some e-mail’ for the details, but we’re going to set up SPF, OpenDKIM, and (maybe) PTR.

OpenDKIM

apt-get install opendkim opendkim-tools
mkdir -pv /etc/opendkim/
chown -Rv opendkim:opendkim /etc/opendkim
chmod go-rwx /etc/opendkim/*
cd /etc/opendkim/
opendkim-genkey -r -h rsa-sha256 -d awesomebox.sealedabstract.com -s mail
mv -v mail.private mail
cat mail.txt

This will output our DKIM key to the terminal. Then, we install it on our DNS server. My ZONE file looks like this. Unfortunately it doesn’t wrap very well in this blog post.

mail._domainkey.awesomebox      300 TXT "v=DKIM1; h=rsa-sha256; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDbLRiNXX9zxAtdw45Vsd35d/1VQZUFG8ejfQu6aql74obJhTESaqANBbOkNaD0xb+2kuN/w+2725Gv2tTPAcHfBZogyazkwtNrlNQV2h3q4ub/UTkn0AHeq0P/RMtmhV+hKRzk0hcYWPlzWMoR5ZGWwMYdhbocTeBX4Mc2pWEYewIDAQAB"

Returning to our Wheezy box, let’s configure postfix to use that key:

nano /etc/opendkim/KeyTable
awesomebox.sealedabstract.com awesomebox.sealedabstract.com:mail:/etc/opendkim/mail

.

nano  /etc/opendkim/SigningTable
*@awesomebox.sealedabstract.com awesomebox.sealedabstract.com

.

nano /etc/opendkim/TrustedHosts
127.0.0.1

.

nano /etc/opendkim.conf
##
## opendkim.conf -- configuration file for OpenDKIM filter
##
Canonicalization        relaxed/relaxed
ExternalIgnoreList      refile:/etc/opendkim/TrustedHosts
InternalHosts           refile:/etc/opendkim/TrustedHosts
KeyTable                refile:/etc/opendkim/KeyTable
LogWhy                  Yes
MinimumKeyBits          1024
Mode                    sv
PidFile                 /var/run/opendkim/opendkim.pid
SigningTable            refile:/etc/opendkim/SigningTable
Socket                  inet:8891@localhost
Syslog                  Yes
SyslogSuccess           Yes
TemporaryDirectory      /var/tmp
UMask                   022
UserID                  opendkim:opendkim

Now we return to /etc/postfix/main.cf and add (to the bottom)

smtpd_milters           = inet:127.0.0.1:8891
non_smtpd_milters       = $smtpd_milters
milter_default_action   = accept

Allright, let’s check it out!

service opendkim restart
service postfix restart

SPF

SPF is simpler. All you have to do is edit your zone:

awesomebox      300 TXT "v=spf1 mx -all"

All I am saying here is “the mail server for awesomebox can send mail for awesomebox” You’d think it would be obvious, but recall at the start, the observation that this system was designed by people who have far more problems than you. At scale, you probably have more than one mail server.

You can also debug your DKIM (and SPF) settings with e.g. this tool.

Reverse PTR

This is not as hard as Jeff leads you to believe. If you’re hosting with Linode (and you should) it’s right on the config panel.

Linodes->[name]->Remote Access->Reverse DNS->Type in your hostname. Done.

Spam

Let’s move on to everybody’s favorite: fighting spam. The inbound kind.

People argue about what is the best antispam tool. Argument over: dspam. I’ve found it quite a bit better than e.g. SpamAssassin and the other traditional tools. It takes some time to train, so it doesn’t work “right out of the box”. But after a few days, my spam detection has been great.

apt-get install dspam dovecot-antispam postfix-pcre dovecot-sieve

patch these in /etc/dspam/dspam.conf:

Home /decrypted-mail/dspam
TrustedDeliveryAgent "/usr/sbin/sendmail"
UntrustedDeliveryAgent "/usr/lib/dovecot/deliver -d %u"
Tokenizer osb

IgnoreHeader X-Spam-Status
IgnoreHeader X-Spam-Scanned
IgnoreHeader X-Virus-Scanner-Result
IgnoreHeader X-Virus-Scanned
IgnoreHeader X-DKIM
IgnoreHeader DKIM-Signature
IgnoreHeader DomainKey-Signature
IgnoreHeader X-Google-Dkim-Signature

ParseToHeaders on
ChangeModeOnParse off
ChangeUserOnParse full
ServerPID               /var/run/dspam/dspam.pid
ServerDomainSocketPath  "/var/run/dspam/dspam.sock"
ClientHost      /var/run/dspam/dspam.sock

Of course, we have to create the directory in question

mkdir /decrypted-mail/dspam
chown dspam:dspam /decrypted-mail/dspam

As best as I can tell, the preferences in /etc/dpsam/dspam.conf are completely ignored. If you want to edit them, the right place is /etc/dspam/default.prefs. Totally logical, right? Anyway, patch these values:

spamAction=deliver         # { quarantine | tag | deliver } -> default:quarantine
signatureLocation=headers  # { message | headers } -> default:message
showFactors=on

Okay, now we’re editing /etc/postfix/master.cf. These go at the end.

dspam     unix  -       n       n       -       10      pipe
  flags=Ru user=dspam argv=/usr/bin/dspam --deliver=innocent,spam --user $recipient -i -f $sender -- $recipient
dovecot   unix  -       n       n       -       -       pipe
  flags=DRhu user=mail:mail argv=/usr/lib/dovecot/deliver -f ${sender} -d ${recipient}

And now:

nano /etc/postfix/dspam_filter_access
/./   FILTER dspam:unix:/run/dspam/dspam.sock

Now we’ll patch the end of /etc/postfix/main.cf

# new settings for dspam
dspam_destination_recipient_limit = 1 #only scan one mail at a time
smtpd_client_restrictions =
   permit_sasl_authenticated #localhost doesn't get scanned
   check_client_access pcre:/etc/postfix/dspam_filter_access #run dspam on everything else

Integrating dspam with imap:

nano /etc/dovecot/conf.d/20-imap.conf
mail_plugins = $mail_plugins antispam

Integrating dspam with lmtp:

protocol lmtp {
# Space separated list of plugins to load (default is global mail_plugins).
mail_plugins = $mail_plugins sieve
}

Telling sieve to move spam into a Spam folder. Edit /decrypted-mail/awesomebox.sealedabstract.com/drew/.dovecot.sieve (e.g. for your username and domain)

require ["regex", "fileinto", "imap4flags"];

# Catch mail tagged as Spam, except Spam retrained and delivered to the mailbox
if allof (header :regex "X-DSPAM-Result" "^(Spam|Virus|Bl[ao]cklisted)$",
          not header :contains "X-DSPAM-Reclassified" "Innocent") {

  # Mark as read
  setflag "\\Seen";

  # Move into the Junk folder
  fileinto "Spam";

  # Stop processing here
  stop;
}

And then we’ll configure /etc/dovecot/conf.d/90-plugin.conf. Note that, there’s just one plugin {} dict, and this stuff goes inside it.

plugin {
...
   # Antispam (DSPAM)
   antispam_backend = dspam
   antispam_allow_append_to_spam = YES
   antispam_spam = Spam;Junk
   antispam_trash = trash;Trash
   antispam_signature = X-DSPAM-Signature
   antispam_signature_missing = error
   antispam_dspam_binary = /usr/bin/dspam
   antispam_dspam_args = --user;%u;--deliver=;--source=error
   antispam_dspam_spam = --class=spam
   antispam_dspam_notspam = --class=innocent
   antispam_dspam_result_header = X-DSPAM-Result
}

Give postfix and dovecot a kick:

service postfix restart
service dovecot restart

And we’re good to go. Incoming messages should have headers like

X-Dspam-Result: Innocent
X-Dspam-Processed: Wed Jun 12 21:46:08 2013
X-Dspam-Confidence: 0.9899
X-Dspam-Probability: 0.0000
X-Dspam-Signature: 51b9246071121935811689
X-Dspam-Factors: 27, Received*12+Jun, 0.01000, Received*12+Jun, 0.01000, Received*Postfix+with, 0.01000, Received*with+#+id, 0.01000, Received*awesomebox.sealedabstract.com+#+12, 0.01000, Content-Type*text+plain, 0.01000, Received*Postfix+#+ESMTP, 0.01000, Received*li212-205.members.linode.com+Postfix, 0.01000, Received*drew+#+#+#+Jun, 0.01000, Received*Wed+#+Jun, 0.01000, Received*Wed+#+Jun, 0.01000, Received*li212-205.members.linode.com+#+with, 0.01000, Received*Wed+#+#+2013, 0.01000, Received*Wed+#+#+2013, 0.01000, Received*Postfix+#+#+id, 0.01000, Received*li212-205.members.linode.com+#+#+#+id, 0.01000, Received*ESMTP+id, 0.01000, Date*12+Jun, 0.01000, Received*for+#+#+#+12, 0.01000, Date*Jun+2013, 0.01000, Received*by+#+Postfix, 0.01000, Received*by+#+with, 0.01000, Received*awesomebox.sealedabstract.com+#+#+Jun, 0.01000, Received*by+#+#+#+ESMTP, 0.01000, Date*Wed+#+#+2013, 0.01000, Received*drew+#+#+12, 0.01000, Received*for+#+awesomebox.sealedabstract.com, 0.01000

To train the message as spam, move it to a folder called “Spam” or “Junk” on any device. Otherwise it’ll be trained as ham.

Push

So long story short, push is complicated. On IMAP, people generally use “IMAP IDLE”, which is basically a fancy way of saying they leave the IMAP connection open all the time. Alas, it turns out to be bad for mobile devices in a lot of dimensions, although you can configure it to be better. Anyway, the iPhone doesn’t support it, ostensibly on battery life grounds.

In theory Apple Mail supports IDLE on OSX, but in practice there is some debate about this. In my experience there is some incompatibility between GMail’s IDLE and Apple Mail’s IDLE, but Dovecot and Mail.app seem to get along just fine. I’m calling it now and blaming PRISM.

Believe it or not, Apple actually uses Dovecot as part of OSX Server–which obviously seems to manage push mail with the whole Apple universe of platforms. As a result, Apple has an implementation of push-on-Dovecot that you can in fact replicate. However, mostly it shuffles data off to its closed-source tool. On the other hand, the closed-source tool mostly does a thing that’s fairly well-known involving the APN system, so it wouldn’t be that hard to clone.

However, there’s an even simpler solution. There’s a tool called “z-push” (Debian rebrands it “d-push” on trademark grounds) that emulates Exchange ActiveSync, much like the old Google Sync did. In fact, I’m fairly certain you could use it to get the Google Sync behavior on GMail again if you wanted.

apt-get install d-push

Now Exchange uses HTTPS. Which means you need an HTTP server. I happen to use lighttpd already, so that’s what I’m going to use. I’m told it’s “designed for” Apache, so it’s probably easy to configure with that server too.

Patch these in /etc/d-push/config.php:

define('IMAP_PORT', 993);
define('IMAP_OPTIONS', '/ssl/novalidate-cert'); //only use this when IMAP_SERVER is 'localhost'!
define('STATE_DIR', '/decrypted-mail/dpush-state/');
define('IMAP_SENTFOLDER', 'Sent Messages');

My lighttpd config rewrites all incoming URLs to wordpress by default. It’s how you get really great URLs like the ones on the top of this post. So I have to “opt out” the Exchange stuff from this system. If you don’t use mod_rewrite, you won’t have to do this. Editing /etc/lighttpd/lighttpd.conf:

url.rewrite-once = (
    "^/(Microsoft-Server-ActiveSync*)(\?.*)?$" => "$1$2",
…
)

Note that, if you’re not careful with the rewrite rules, lighttpd will slice the params off the URL before they get passed to d-push. Then you get errors like this:

[INFO] Version='2.0-1.1' method='GET' from='XXXX' cmd='' getUser='drew@awesomebox.sealedabstract.com' devId='' devType=''
[FATAL] FatalException: Requested the d-push URL without the required GET parameters - code: 0

Several people seem to be having this problem on the Internet.

If you don’t have it already, you need mod_alias listed under server.modules in your /etc/lighttpd/lighttpd.conf. I think it is there by default, but I’m not entirely sure.

Now you need to point lighttpd to dpush. Basically anywhere in your /etc/lighttpd/lighttpd.conf, do this:

alias.url += ("/Microsoft-Server-ActiveSync" => "/usr/share/d-push/index.php")

If you don’t have an SSL cert, you can generate one from bash:

openssl req -x509 -newkey rsa:2048 -keyout lighttpd-key.pem -out lighttpd-cert.pem -days 1001 -nodes
cat lighttpd-key.pem lighttpd-cert.pem > lighttpd-all.pem

Back in nano, let’s turn that SSL on, if it’s not on already:

$SERVER["socket"] == ":443" {
     ssl.engine                  = "enable"
     ssl.pemfile                 = "/etc/ssl/lighttpd-all.pem"
 }

And give lighttpd a kick:

service lighttpd restart

There you go. At this point you should be able to connect via the “Exchange” settings on an iPhone.

A quick note that I needed a patch /usr/share/d-push/backend/imap/imap.php to get my “Deleted Messages” folder working:

else if($lid == "trash" || $lid == "deleted messages") {
            $folder->parentid = "0";
            $folder->displayname = "Trash";
            $folder->type = SYNC_FOLDER_TYPE_WASTEBASKET;
            $this->wasteID = $id;
        }

A quick note on debugging — the z-push documentation suggests that to get logs you need to create some file called “debug.txt” in an unspecified location and then chmod it 777. The Debian Gods, however, have seen fit to give us a /var/log/d-push/d-push.log file to check instead. You can also check /var/log/lighttpd/error.log and access.log at the same path.

Full-text search

Unfortunately, we need Java.

apt-get install solr-tomcat dovecot-solr

Due to some bug, you have to pull down solr-schema.xml from the orig.tar.gz package, linked here.

cp /path/to/solr-schema.xml /etc/solr/conf/schema.xml
service tomcat6 restart

Back to patching /etc/dovecot/conf.d/20-imap.conf. The “antispam” is there from when we installed dspam.

mail_plugins = $mail_plugins antispam fts fts_solr

Now we hit /etc/dovecot/conf.d/90-plugin.conf:

plugin {
  ...
  fts = solr
  fts_solr = break-imap-search url=http://localhost:8080/solr/
}

“break-imap-search” is a command that essentially says we should actually do full-text-search, which is against the IMAP specification.

VERY IMPORTANT. By default, tomcat6 is globally accessible, which means just anybody with a web browser can query your mail! We need to turn this off. Inside /etc/tomcat6/server.xml, there is a line called

<Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" URIEncoding="UTF-8" redirectPort="8443" />

We need to add address="127.0.0.1" onto that:

<Connector address="127.0.0.1" port="8080" protocol="HTTP/1.1" 
           connectionTimeout="20000" 
           URIEncoding="UTF-8"
           redirectPort="8443" />

For reasons that are not immediately clear to me, Solr comes with some queries out of the box that are configured for the schema that it ships with. You will see things in your log like this:

SEVERE: org.apache.solr.common.SolrException: undefined field text
at org.apache.solr.schema.IndexSchema.getDynamicFieldType(IndexSchema.java:1330)
at org.apache.solr.schema.IndexSchema$SolrQueryAnalyzer.getAnalyzer(IndexSchema.java:408)
at org.apache.solr.schema.IndexSchema$SolrIndexAnalyzer.reusableTokenStream(IndexSchema.java:383)
at org.apache.lucene.queryParser.QueryParser.getFieldQuery(QueryParser.java:574)
at org.apache.solr.search.SolrQueryParser.getFieldQuery(SolrQueryParser.java:206)
at org.apache.lucene.queryParser.QueryParser.Term(QueryParser.java:1429)
at org.apache.lucene.queryParser.QueryParser.Clause(QueryParser.java:1317)
at org.apache.lucene.queryParser.QueryParser.Query(QueryParser.java:1245)
at org.apache.lucene.queryParser.QueryParser.TopLevelQuery(QueryParser.java:1234)
at org.apache.lucene.queryParser.QueryParser.parse(QueryParser.java:206)
at org.apache.solr.search.LuceneQParser.parse(LuceneQParserPlugin.java:79)
at org.apache.solr.search.QParser.getQuery(QParser.java:143)
at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:105)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:165)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1376)
at org.apache.solr.core.QuerySenderListener.newSearcher(QuerySenderListener.java:59)
at org.apache.solr.core.SolrCore$3.call(SolrCore.java:1182)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)

To fix this, you need to place <!-- and --> around text in /etc/solr/conf/solrconfig.xml. It should look like this:

<lst name="defaults">
       <str name="echoParams">explicit</str>
       <int name="rows">10</int>
       <!--<str name="df">text</str>-->
     </lst>

We should also patch the data directory to be on the encrypted partition:

<dataDir>/decrypted-mail/solr</dataDir>

And give it the proper permissions:

mkdir /decrypted-mail/solr
chown -R tomcat6:tomcat6 /decrypted-mail/solr

Give tomcat and dovecot a kick:

service dovecot restart
service tomcat6 restart

I’ve found that it sometimes helps to remove dovecot’s indexes:

rm -f /decrypted-mail/awesomebox.sealedabstract.com/drew/dovecot.index*
rm -f /decrypted-mail/awesomebox.sealedabstract.com/drew/.MyMailboxName1/dovecot.index*
rm -f /decrypted-mail/awesomebox.sealedabstract.com/drew/.MyMailboxName2/dovecot.index*

IMAP search should work at this point. Unfortunately, if you’re using push (and thus z-push, and thus ActiveSync) it’s a little more complicated.

As best as I can tell, the official z-push doesn’t support this. However, there is a vibrant z-push fork community. fmbiete’s fork seems to be way ahead of upstream. Looks like somebody could use a new maintainer.

Anyway, I simply copied fmbiete’s fork to /usr/share/d-push, making sure to back up and restore the config.php file. z-push/d-push seems to want a little more of a kick when you move between versions; this is the recipe that worked for me:

rm /decrypted-mail/dpush-state/*

Followed by deleting and re-adding the account on my iPhone. You get a few “can’t connect to the server” errors, but that’s just because you’ve thoroughly confused the sync on the iPhone side by nuking everything.

Debugging magic

There’s a neat trick to debugging IMAP search I discovered while I was working on this.

openssl s_client -connect localhost:993
1 login drew@sealedabstract.com [password]
2 Select "Inbox"
3 Search text "test"

That will let you test what search and search performance look like on the IMAP side. You can also see what solr is up to:

tail /var/log/tomcat6/catalina.2013-06-13.log
INFO: [] webapp=/solr path=/select params={fl=uid,score&sort=uid+asc&q=body:"Test"&fq=%2Bbox:123a6f3ac671b951ca310000e529c561+%2Buser:"drew@sealedabstract.com"&rows=179} hits=2 status=0 QTime=3 
Jun 13, 2013 4:23:16 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/select params={fl=uid,score&sort=uid+asc&q=body:"Test"&fq=%2Bbox:e6825420817bb951a9380000e529c561+%2Buser:"drew@sealedabstract.com"&rows=3} hits=0 status=0 QTime=1 
Jun 13, 2013 4:23:16 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/select params={fl=uid,score&sort=uid+asc&q=body:"Test"&fq=%2Bbox:7301d918ab87b95174400000e529c561+%2Buser:"drew@sealedabstract.com"&rows=5205} hits=364 status=0 QTime=4 

You can query solr directly:

curl 'http://localhost:8080/solr/select?fl=uid,score&rows=5205&sort=uid+asc&q=body:%22Why%20bec%22&fq=%2Bbox:7301d918ab87b95174400000e529c561+%2Buser:%22drew%40sealedabstract.com%22'

This means I’m searching for “test”.

Also, when in doubt: restart tomcat. In my testing it doesn’t react even to things like filesystem changes (???) without a kick.

Conclusion

We did it! We’ve got a fast, modern, e-mail stack that (in my view, anyway) is much better than GMail, not only in the security aspect, but even in areas like speed, spam, and compatibility.

There is, however, more we could do:

  • webmail – I don’t really need one, but I’ve heard good things about roundcube. Apple used to ship it in 10.7 Server, but stopped suddenly. Take that for whatever it’s worth.
  • requiring TLS encryption – E-mail isn’t secure, but many servers route it over SSL anyway if both endpoints support it. Requiring TLS makes you non-RFC-compliant, so you might lose mail, but it also keeps the NSA from scooping up your mail in some datacenter.
  • Greylisting. This is a cool idea involving providing a “temporary failure” each time a new sender sends a message. Legitimate senders will keep retrying, and after a few minutes you add them to a “good” list. Spammers will give up, and their mail messages will never even hit disk. It’s actually really easy configure, just apt-get install postgrey and then add check_policy_service inet:127.0.0.1:10023 under smtpd_client_restrictions in /etc/postfix/main.cf.

So, go forth and run your own mail. Now, your right to complain about government spying has been rightly restored. Enjoy your HN flamewars.


Want me to build your app / consult for your company / speak at your event? Good news! I'm an iOS developer for hire.

Like this post? Contribute to the coffee fund so I can write more like it.

Comments

  1. Fri 23rd Aug 2013 at 5:49 am

    @Derek: I would suggest you add “t=y” to the DNS TXT record to activate “Testing Mode”. See https://www.ietf.org/rfc/rfc4871.txt for details.

  2. Fri 23rd Aug 2013 at 5:54 am

    First thanks a lot for your good write up, helped a lot!

    I found one incompliance with the current DKIM RFC4871 from the IETF (https://www.ietf.org/rfc/rfc4871.txt): You used “rsa-sha256” as the value of “h” in your DNS-Record but should use “sha256” instead. So the command to generate your keys should read:

    opendkim-genkey -r -h sha256 -d awesomebox.sealedabstract.com -s mail

    and your DNS-Entry should replace “h=rsa-sha256” with “h=sha256”. Then all verifiers I could find pass, including the strict one from verifier.port25.com.

  3. Libby
    Sun 25th Aug 2013 at 9:03 am

    Thank you, Daps!! I had the same problem and that fixed it.

  4. ted
    Sun 25th Aug 2013 at 11:06 am

    @temp

    I followed the fix here
    https://forum.linode.com/viewtopic.php?t=10329&p=59751
    which resolved the ‘modules.dep.bin’ issue. Running the full encfs /encrypted-mail /decrypted-mail -o --public command returned a different error, but running it without the -o then worked.

  5. Mon 26th Aug 2013 at 8:49 am

    Nice article. I read it a couple of days ago and then lost it and tried to find it again ever since, because the topic came up in several discussions and I wanted to be able to point people somewhere. Googleing for “mail server encfs dovecot postfix howto” did the trick.

    Some thoughts:

    It might be best to ensure that Postfix only uses secure memory. One option is to disable swap completely, alternatively you can encrypt your swap space (I think using LUKS) or audit postfix to use mlock(2) to ensure no email data is ever written to the disk in plain. I’m not sure postfix and dovecot were designed to only use secure memory because (1) the data is written to the mail spool anyway, which is usually unencrypted, (2) Total surveillance was long thought to be unrealistic. As the recent revelations have shown, this is not true.
    Same thing goes for the mail queue: You should ensure that the mail queue, where mail that can’t be delivered immediately is stored, is encrypted so the NSA can’t harvest your email from old, unlinked files in the mail queue. Using encfs on the mail queue and starting postfix manually after a server reboot should to the trick.
    Also make sure that files in /tmp can’t be recovered after a system reboot. Make /tmp a ramdisk by mount -t tmpfs none /tmp. Note that RAM-disks are unsuitable for storing the mail queue because you normally want your undelivered outgoing email to be delivered after an unexpected reboot.

    One thing I personally haven’t done yet is using DKIM. A problem I see with DKIM is that it isn’t MIME aware and thus it’s currently impossible to sign only parts of a multipart message. It also is rather unsuitable for posting to mailing lists. Perhaps a subsequent version of the standard will eliminate these issues.

    Feel free to contact me via email.

    Best,

    Moritz

  6. Lomax
    Mon 26th Aug 2013 at 7:56 pm

    Thanks for the fantastic writeup!

    I came across and resolved the following issues:

    postfix didn’t seem to like the comments starting mid-line in /etc/postfix/main.cf – they needed to be removed.

    /etc/dovecot/dovecot.conf needs “lmtp” adding to its “protocols = …” section to turn this on.

    brandonchecketts’s email test tool didn’t like the format of the DKIM key produced by OpenDKIM. The error message was something like “does not support hash algorithm ‘sha256′”, producing the T_DKIM_INVALID SpamAssassin tag. This was resolved by just removing the options at the start of the key, so it just looked like: k=rsa; p=… (This is what gmail’s DKIM key looks like).

    And finally the contents of /etc/opendkim/KeyTable is supposed to be one line – as indeed it is in this post if you view source, the wrapping had me confused. This one produces errors like “resource unavailable: d2i_PrivateKey_bio() failed”.

    Also, since I did this setup on a clean Debian install, I had a few issues installing d-push and lighttpd. The package depends on apache, but recommends lighttpd (?), so I got this going by removing apache2.2-bin after installing d-push, then installing lighttpd, then doing “dpkg_reconfigure lighttpd” and selecting lighttpd when prompted to do the configuration. This takes care of most of the config. The only other thing I had to do was symlink conf-available/10-fastcgi.conf and 15-fastcgi-php.conf into conf-enabled in order to make PHP work (otherwise you get 403 Forbidden when browsing to https://…/Microsoft-Server-ActiveSync

    The only other things I did were to put the secret keys on the encfs, and stop all the services auto-starting, so they can get started manually after mounting it. I used the following script:

    !/bin/bash

    sudo encfs /encrypted-mail /decrypted-mail –public
    for s in mysql postfix dovecot opendkim dspam lighttpd tomcat6
    do
    sudo service $s start
    done

    So far it’s working great – thanks again.
    The only other thing I think would be valuable is any advice you can offer about OpSec. I appreciate you don’t want to give away specific techniques you’re using, but in general, can you shed any more light on what kinds of things to guard against?

    I believe as long as it’s on a decent VM (not OpenVZ – but KVM or Xen VPS, or a real dedicated box), the admin can’t trivially browse around your stuff. I guess there is always the possibility that they could remount your rootfs, change the root password and get in that way without having to shut down the box, but that all sounds a bit ninja!

  7. Lomax
    Mon 26th Aug 2013 at 8:06 pm

    @Derek S

    Same situation as me. You can just add the DKIM key in a TXT record called mail._domainkey (no dot on the end) or mail._domainkey.yourdomain.com. (dot on the end, meaning it’s fully qualified).

    Then you can test it using the testing utility linked above: http://www.brandonchecketts.com/emailtest.php

  8. Tue 27th Aug 2013 at 2:51 am

    can i first suggest that the SPF record you recommend be edited as you should never ever use mx in an spf record as its considered a bonehead move (not quite as bonehead as using ptr:domain but close)

    it should simply be v=spf1 ip4: -all

    as you know your own ip and it costs 0 further dns lookups for receivers to find

    mx is more expensive to receivers as it costs a minimum of 2 dns lookups (of 10 total allowed before an spf fail) so if you have multiple servers used by multiple users the benefit of ip4 becomes obvious

    that said as fas as ptr/dkim/spf i would also suggest looking at http://www.alandoherty.net/info/mailservers/
    for a list of all the simple things that can be done in dns and server-config to ensure that others know your servers are legit mail servers not infected forgers

  9. Tue 27th Aug 2013 at 2:52 am

    sorry in my previous comment it removed some text
    i had typed
    v=spf1 ip4:{your-ip-here} -all

  10. Rorto
    Fri 30th Aug 2013 at 7:40 am

    Just let me now when Java will not be necessary anymore to achieve this.

  11. Justin
    Sat 31st Aug 2013 at 5:08 pm

    For folks having issues with encfs … –public, make sure you’re not copying/pasting from above. After following Daps0l’s tips, I was still getting an error; but it was because –public is a two-hyphen parameter, and copying/pasting was giving me an mdash or something.

  12. Pm
    Sat 31st Aug 2013 at 8:32 pm

    I’m having a similar situation to Temp. I’ve installed the fuse-util package but when I attempt ‘which fuse’ it still tells me that it’s missing.

  13. Thu 05th Sep 2013 at 12:41 pm

    I figured out the Debian Linode issue with EncFS.

    Debian 7 Linodes are using EncFS version 1.7.4 (not 1.7.4-2.4+b1, which I’m assuming is a later version than 1.7.4). Also, I don’t know if this fix is even relevant to that version number.

    At any rate, the real EncFS command is:
    encfs –public /encrypted_mail /decrypted_mail

    By putting -o –public at the end of the command, as in the guide, it’s passing the options to fusermount (the mount program for Fuse), when in reality, it needs to be passed to EncFS itself, hence putting it toward the front/top of the command.

    Hope this helps, and thanks for the guide!

  14. Donald S
    Thu 05th Sep 2013 at 7:06 pm

    Well that didn’t work. Trying again:

    @Derek S
    DomainKeys are found in the _domainkey subdomain, and use a selector to identify which key to use. For the root domain your DNS record should be selector._domainkey 300 TXT "v=DKIM1;...", giving a FQDN of selector._domainkey.example.com.
    Drew uses ‘mail’ as the selector but you can use pretty much anything you want, e.g. if you want to use multiple domain keys for different things.

  15. X
    Fri 06th Sep 2013 at 2:01 pm

    This is too complex, no-one will bother to do this, and it’s not because they’re hypocrites or lazy – this amount of effort simply doesn’t scale well, and 2 hours is a very conservative estimate except for people who are already experienced with running servers.

    One way to improve the situation would be if you turned all the things in this blog post, into a script that automates the entire process, with (a) robust error handling in case a user’s system doesn’t fit your exact assumptions and (b) robust abort-resume behaviour if the process gets interrupted. If something really really isn’t automatable (e.g. setting up reverse DNS), you can build it as a Q&A prompt into the script, and (a) and (b) properties should allow them to cancel-then-resume the script once they’ve done that step.

    Up to the challenge?

  16. X
    Fri 06th Sep 2013 at 2:03 pm

    whoops, I meant optimistic estimate rather than conservative estimate.

  17. Fri 06th Sep 2013 at 9:17 pm

    “I’m going to assume:”

    Bad idea. Thanks for the tutorial, but…

    I’m not running Debian, never had any experience running Apache or any other server, and haven’t got a server to use, but I still want secure email that I host myself and the right to complain about the treasonous war the NSSA is levying on the public. Maybe I’ll eventually get something like a RaspberryPI and find a way to do it with that but not for now. Most people are similar, and also don’t know their way around linux (I’m currently running a version of Puppy linux on my desktop, based on Slackware) and that is why they use GMail or some other third party.

    If we can’t figure out a way that the average joe can do this easily, then somebody needs to make that(or a step towards it) their goal whether it requires making cheap plug servers available or whatever.

    I have an N800(Nokia internet tablet circa 2006) that runs Maemo(a version of linux Nokia developed) and it is the only computer I have that’s (almost)always on. We need something that can enable people to host such an encrypted service on something like that or a RaspberryPI or even their “smart” phone. Not saying it’s your problem, just that this is what is needed.

  18. Lefty Goldblatt
    Sat 07th Sep 2013 at 11:03 am

    Could this all be pre-packaged into a virtualbox image? Then folks could just launch the image and worry about editing whatever needs changed?
    Less chance of problems maybe.

  19. siavash
    Sat 07th Sep 2013 at 8:25 pm

    Hi
    I’m just on the first lines of your post but I’m impressed enough to submit a comment right now.
    I’ve been using Internet for more than 12 years and I’m sure that this is one of the most incredible posts I’ve ever seen.
    As I’m a GNU user for a while and even before this NSA thing, I was planning to do everything on my own server.
    Now I’m using Owncloud for pretty much everything except Email and this post is exactly what I was waiting for.
    I’ll comment again if there was a problem during the setup and after its done.
    Thanks for sharing your knowledge and wish you the best.

  20. Jesse Kline
    Sat 07th Sep 2013 at 8:58 pm

    I agree that those of us who are concened about privacy should take steps to protect our data. I’ve been looking into seting up my own mail server using a Raspberry Pi, but I’m getting stuck on the DNS. I currently have a domain hosted with Dreamhost. The email service is outsourced to Google. I would like to circumvent Gmail by setting up secure.mydomain.com and having a dynamic DNS service update the DNS record to point to my home address.

    I’m not sure if I can point a subdomain to dyndns or another dynamic IP service (I’d prefer not to have to pay for it). I’ve found scripts to update the Dreamhost DNS records, but it only allows my to create A records — I have to fully host a subdomain in order to change the MX records.

    Can anyone point me to a good howto guide for creating a subdomain for email and using dynamic DNS to keep the record up to date?

  21. Steiner
    Mon 09th Sep 2013 at 3:48 pm

    Now that it is known that NSA can crack SSL, how about adding PGP into the mix?

  22. Me
    Tue 10th Sep 2013 at 7:17 pm

    Just a quck note about

    encfs /encrypted-mail /decrypted-mail -o –public

    I believe this command is incorrect as is.

    It should either be:
    encfs /encrypted-mail /decrypted-mail –public

    OR
    encfs /encrypted-mail /decrypted-mail -o allow_other blah_blah_other_fuse_args_here_that_that_equtes_to_public_above

    the docs for 1.7.4 either use –public
    OR
    use -o with the list of fuse_args which will give you equivalent meaning to public. If you run the command as is in the tutorial above –public will be treated as a fuse_arg, which it isnt, and fuse will throw an exception.

  23. Me
    Tue 10th Sep 2013 at 11:06 pm

    Ran into two more errors in the logs.

    The first were of this type
    … status=deferred (connect to mailserver[private/dovecot-lmtp]: No such file or directory) …

    This was fixed by editing
    nano /etc/dovecot/dovecot.conf
    and setting:
    protocols = imap lmtp

    This presented another set of errors of the form:
    … status=deferred (lost connection with mailserver[private/dovecot-lmtp] while receiving the initial server greeting) …

    This error lead to the solution…
    …lmtp(11805): Fatal: Error reading configuration: Invalid settings: postmaster_address setting not given…

    That variable needs to be set in
    nano /etc/dovecot/conf.d/15-lda.conf
    in my case its the first variable you can set in the file.
    I chose to create an alias in the mysql called postmaster@mydomain.com that goes to myrealemail@mydomain.com, then stick postmaster@mydomain.com into that variable. OR you can just stick yourrealemail@yourdomain.com in there.

    Give dovecot and postfix a kick.
    /etc/init.d/dovecot restart
    /etc/init.d/postfix restart

    Now I can recieve mails from externals.

    I still cannot send mail as I think my ISP has 25 outgoing blocked. What a bummer.

    What I still cannot understand though is the purpose of encfs. The system is going to be on all the time so the email with be always unencrypted in /decrypted-mail.

    There is also no where to show how to mount /decrypted-mail if you need to reboot. I suppose it can be done in fstab.

    Now onto see if I can complete this thing…its been over two hours now.

  24. Me
    Wed 11th Sep 2013 at 2:51 pm

    Just another little gotcha….

    When sending email you may get a 4.7.1 service unavailable message.

    Checking the logs further shows that opendkim is getting a permission denied error.

    /etc/opendkim is chowned to opendkim user but not it’s contents, so just do a
    chown -R opendkim:opendkim /etc/opendkim

    then you know what to do… kick opendkim and kick postfix
    Check logs to make sure opendkim permission errors are gone.

  25. Fri 13th Sep 2013 at 6:27 am

    I’m sure I’m being uber paranoid here but isn’t Linode hosted in the US?
    So they like pretty much like every service provider in the US could be asked to provide your email, and it wouldn’t be hard for them.

  26. Gerry
    Sat 14th Sep 2013 at 9:23 am

    What is the minimum hardware requirements for such stack?

    CPU / MEM / DISK ?

  27. fmbiete
    Sun 15th Sep 2013 at 6:15 am

    Great post!!

    Nice to see that someone got to work z-push-contrib + dovecot + solr

  28. Daps0l
    Wed 18th Sep 2013 at 8:40 am

    @Eva Lacy

    Well the e-mail database is encrypted with encfs, so if the host/harddisk is given to authorities they can not decrypt it and thus can not access your e-mails.

    However, in general it’s true that if someone ‘evil’ has physical access to your host, you’re usually screwed. Because one can obtain the encryption keys from memory as long as the server is running.

    But, obviously that does not mean you shouldn’t make it ‘as difficult as you can’ to protect your privacy! :)

  29. gpunkt
    Tue 03rd Dec 2013 at 8:57 am

    the initial directories, decrypted and encrypted, are they in /home/root/filename or do you create a user like vmailuser and go with this? I’ll just begin trying this and see where it leads if I create a new user to deal with the mail because I see no other user than root in your codes, I don’t feel like using root for this idea.

  30. Pablo
    Mon 09th Dec 2013 at 4:00 am

    Hi,
    great tutorial! Everything is working fine here but the SMTP.

    The log shows me:
    Dec 9 09:47:18 ip-172-XXX-XXX-XXX dovecot: auth-worker: mysql(127.0.0.1): Connected to database mailserver
    Dec 9 09:47:18 ip-172-XXX-XXX-XXX dovecot: imap-login: Login: user=me@XXX.XXX, method=PLAIN, rip=189.XXX.XXX.XXX, lip=172.XXX.XXX.XXX, mpid=7171, TLS
    Dec 9 09:47:19 ip-172-31-40-52 dovecot: imap(me@XXX.XXX): Connection closed bytes=13/344

    But I have no answer to my SMTP request.

    I’m using the same server used in IMAP, so:

    IMAP (this is working fine!):
    Username: me@XXX.XXX
    Incomeserver: mail.XXX.XXX
    Port: 993 (already open in my firewall)
    Connection type: SSL

    SMTP (this is NOT working):
    Sending server: mail.XXX.XXX
    Port: 587 (already open in my firewall)
    Connection: Start TLS
    Username: me@XXX.XXXX
    Password: XXX
    Connection type: PLAIN

    Actually I’ve tried all combinations in SMTP settings:
    None + Plain
    None + Login
    None + CRAM-MD5
    None + DIGEST-MD5

    Start TLS + Plain
    Start TLS + Login
    Start TLS + CRAM-MD5
    Start TLS + DIGEST-MD5

    SSL + Plain
    SSL + Login
    SSL + CRAM-MD5
    SSL + DIGEST-MD5

    None of these settings allowed me to connect to my SMTP server.

    I’m using 2244 as SSH port. Could be something related?

    The mail client I’m using is AirMail (Mac).

    Could you please send some words about it?

    Thanks in advanced,

  31. Wed 11th Dec 2013 at 9:31 pm

    Rather than the “chown -R opendkim…” above, I’d recommend this, as it doesn’t give opendkim write permissions it doesn’t need:

    cd /etc/opendkim
    chown root:opendkim mail
    chmod g+r mail

    Also, “chmod go-rwx /etc/opendkim/” doesn’t work, as you’ve noticed if you’ve tried it. Leave off the trailing “/“.

    Great how-to! It’s really helping me patch up the ol’ mail server.

  32. atak
    Sat 04th Jan 2014 at 12:11 pm

    When I run the encfs command I get fuse –public is not an valid option.

  33. Charles
    Thu 09th Jan 2014 at 10:55 am

    Nice tutorial! I’ve been running my own mail server since college, most recently with Exim + Dovecot + LUKS. A few observations:

    1) Some ISPs won’t let you run a server at home, or want to charge you a lot of money to do so (Insight performs regular port scans and will cut off your service if they find anything)

    2) If your residential Internet service goes down temporarily, RFC compliant remote servers will keep trying to deliver mail for 4-5 days as the article says, but many of them will issue a notification to the sender after the FIRST failed delivery attempt that delivery is being delayed, which can look unprofessional if you’re using this for work (a professor at business school gave me a lecture about this when he got one of these notifications trying to e-mail me something)

    3) The encryption method they use (EncFS) leaks metadata about the files it’s storing; full disk encryption is always more secure

Comments are closed.

Powered by WordPress