amazon ec2 - After reading lots of articles on sendmail relay tls port 587 gmail, still having problems

Environment is Amazon EC2, running Amazon Linux (Centos-like) with sendmail 8.14.4 and Cyrus sasl 2.1.23. The machine has an elastic IP address that is the target of a domain name, and reverse DNS is setup for it. MX record points at an external server, so the machine does not deal with any incoming internet email, only outgoing. For all outgoing off-node email, I want to use a TLS authenticated connection to smtp.googlemail.com.

There are lots of tutorials and articles about this kind of setup (several on this site and its sister sites), as it seems to be a fairly popular way of dealing with email in the cloud. I've been reading every one I can find, trying different things, and studying /var/log/maillog. But for the life of me, I'm stumped. It's not that I can't send email with sendmail, or that I can't send it through an authenticated TLS connection to smtp.googlelemail.com: it's just that I can't get sendmail to send it through such a connection! I admit I'm a sendmail newbie, and I know it has a reputation for difficult configuration, so I've really put a lot of time into it. But I've run out of clues and ideas at this point.

I have one php application on the machine that uses Zend Framework (1.11), and it uses Zend_Mail_Transport_Smtp. In the constructor for the transport I specify smtp.googlemail.com, port 587, tls, name@gmail.com, and password. The email from that application gets sent lickity-split and arrives with nice clean headers (I also added a TXT record including google's spf txt records to the domain's zone file).

I have another application that uses php mail directly, and php mail relies on sendmail. That application also is able to send mail that arrives okay under the default sendmail configuration supplied with the Amazon Linux AMI (which does not relay through googleemail). But the headers in the arriving messages are not so clean, at least one thing being that there is a "neutral" complaint from the spf check. So I'm not stuck without email, it's just that I'd like to be able to use sendmail for its greater reliably with queueing outgoing mail in case of multi-user bursts of emails coming from the website (as all those new users of the Zend Framework application sign up for accounts :-), and I don't want users looking at the headers and thinking things may not be quite right.

So the goal here is just to modify the existing working sendmail configuration to relay outgoing mail through a port 587 tls connection to googlemail, just the way that the Zend Framework application does with no problem. And then I can modify the Zend Framework application to use sendmail to get the overload protection of queueing.

How hard can it be I thought...

What I'm finding is that with the changes in place in the sendmail configuration, outgoing messages invariably get stuck with this message in /var/log/maillog:

"timeout waiting for input from googlemail-smtp.l.google.com. during client greeting"

The messages then go into the mailq and stay there, failing each time they get re-tried.

Here are diffs between the sendmail.cf that works by sending directly, and the one modified to relay through google:

# diff sendmail.mc-orig sendmail.mc-new
26c27,30

< dnl define(`SMART_HOST', `smtp.your.provider')dnl
---
> define(`SMART_HOST', `[smtp.googlemail.com]')dnl
> define(`RELAY_MAILER_ARGS', `TCP $h 587')dnl
> define(`ESMTP_MAILER_ARGS', `TCP $h 587')dnl
> FEATURE(authinfo, `Hash -o /etc/mail/authinfo.db')dnl
52,53c56,59
< dnl TRUST_AUTH_MECH(`EXTERNAL DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
< dnl define(`confAUTH_MECHANISMS', `EXTERNAL GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
---

> dnl # TRUST_AUTH_MECH(`EXTERNAL DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
> dnl # define(`confAUTH_MECHANISMS', `EXTERNAL GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
> TRUST_AUTH_MECH(`PLAIN LOGIN')dnl
> define(`confAUTH_MECHANISMS', `PLAIN LOGIN')dnl

Here is the (sanitized) mode 600 root-owned authinfo file:
AuthInfo:[smtp.googlemail.com] "U:user@gmail.com" P:"xyzzy" "M:PLAIN"

I also started saslauthd, and edited /usr/lib/sasl2/Sendmail.conf to read:
pwcheck_method:saslauthd
mech_list: login plain

I did not set up any certificates, because as far as I could tell they are not necessary if sendmail is only using TLS on outgoing connections; and indeed the port 587 connection works in the Zend Framework application without certificates installed (or saslauthd running for that matter).

The level 29 maillog for a message sent with the relaying sendmail config looks like this:

[31528]: q1F5dYcB031528: from=userapp.com+admin@gmail.com, size=7085, class=0, nrcpts=1, msgid=<3fb0c7a36a63b2b855336d0865b345a7@bugs.userapp.com>, relay=nobody@localhost
[31529]: NOQUEUE: connect from localhost [127.0.0.1]
[31529]: AUTH: available mech=PLAIN LOGIN, allowed mech=PLAIN LOGIN
[31529]: q1F5dYYY031529: Milter: no active filter
[31529]: q1F5dYYY031529: --- 220 name.compute-1.internal ESMTP Sendmail 8.14.4/8.14.4; Wed, 15 Feb 2012 05:39:34 GMT
[31529]: q1F5dYYY031529: <-- EHLO name.compute-1.internal

[31529]: q1F5dYYY031529: --- 250-name.compute-1.internal Hello localhost [127.0.0.1], pleased to meet you
[31529]: q1F5dYYY031529: --- 250-ENHANCEDSTATUSCODES
[31529]: q1F5dYYY031529: --- 250-PIPELINING
[31529]: q1F5dYYY031529: --- 250-8BITMIME
[31529]: q1F5dYYY031529: --- 250-SIZE
[31529]: q1F5dYYY031529: --- 250-DSN
[31529]: q1F5dYYY031529: --- 250-ETRN
[31529]: q1F5dYYY031529: --- 250-AUTH PLAIN LOGIN
[31529]: q1F5dYYY031529: --- 250-DELIVERBY
[31529]: q1F5dYYY031529: --- 250 HELP

[31529]: q1F5dYYY031529: <-- MAIL From: SIZE=7085 AUTH=userapp.com+2Badmin@gmail.com
[31529]: ruleset=trust_auth, arg1=userapp.com+2Badmin@gmail.com, relay=localhost [127.0.0.1], reject=550 5.7.1 ... not authenticated
[31529]: q1F5dYYY031529: --- 250 2.1.0 ... Sender ok
[31529]: q1F5dYYY031529: <-- RCPT To:
[31529]: q1F5dYYY031529: --- 250 2.1.5 ... Recipient ok
[31529]: q1F5dYYY031529: <-- DATA
[31529]: q1F5dYYY031529: --- 354 Enter mail, end with "." on a line by itself
[31529]: q1F5dYYY031529: from=, size=7190, class=0, nrcpts=1, msgid=<3fb0c7a36a63b2b855336d0865b345a7@bugs.userapp.com>, proto=ESMTP, daemon=MTA, relay=localhost [127.0.0.1]
[31529]: q1F5dYYY031529: --- 250 2.0.0 q1F5dYYY031529 Message accepted for delivery
[31528]: q1F5dYcB031528: to=user@example.com, ctladdr=userapp.com+admin@gmail.com (99/99), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=37085, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (q1F5dYYY031529 Message accepted for delivery)

[31529]: q1F5dYYZ031529: <-- QUIT
[31529]: q1F5dYYZ031529: --- 221 2.0.0 name.compute-1.internal closing connection
[31531]: q1F5dYYY031529: SMTP outgoing connect on name.compute-1.interna
[31531]: q1F5dYYY031529: timeout waiting for input from googlemail-smtp.l.google.com. during client greeting
[31531]: q1F5dYYY031529: to=, delay=00:05:00, xdelay=00:05:00, mailer=relay, pri=127190, relay=googlemail-smtp.l.google.com. [74.125.91.16], dsn=4.0.0, stat=Deferred: Connection timed out with googlemail-smtp.l.google.com.  

I do see the "ruleset=trust_auth, ... not authenticated" message, but besides not knowing how to fix it, I also see that it's immediately followed by an ok message, and the log shows it proceeding onward to try to connect to the relay, so I think that has nothing to do with the timeout... If I'm wrong and someone could tell me how to fix it, that would be great!

The maillog for a message sent with the unmodified config that works without relaying (note that "user@example.com" is actually an address with an mx record for a Network Solutions server, which is why the final line has a relay= for a netsol.net host):

[31425]: q1F5VtLr031425: from=userapp+admin@gmail.com, size=6743, class=0, nrcpts=1, msgid=<8aa8ddfdc691cb86896329126a4eb6ef@bugs.userapp.com>, relay=nobody@localhost
[31426]: NOQUEUE: connect from localhost [127.0.0.1]
[31426]: AUTH: available mech=PLAIN LOGIN, allowed mech=EXTERNAL GSSAPI KERBEROS_V4 DIGEST-MD5 CRAM-MD5
[31426]: q1F5VuPd031426: Milter: no active filter
[31426]: q1F5VuPd031426: --- 220 name.compute-1.internal ESMTP Sendmail 8.14.4/8.14.4; Wed, 15 Feb 2012 05:31:56 GMT
[31426]: q1F5VuPd031426: <-- EHLO name.compute-1.internal
[31426]: q1F5VuPd031426: --- 250-name.compute-1.internal Hello localhost [127.0.0.1], pleased to meet you
[31426]: q1F5VuPd031426: --- 250-ENHANCEDSTATUSCODES
[31426]: q1F5VuPd031426: --- 250-PIPELINING

[31426]: q1F5VuPd031426: --- 250-8BITMIME
[31426]: q1F5VuPd031426: --- 250-SIZE
[31426]: q1F5VuPd031426: --- 250-DSN
[31426]: q1F5VuPd031426: --- 250-ETRN
[31426]: q1F5VuPd031426: --- 250-DELIVERBY
[31426]: q1F5VuPd031426: --- 250 HELP
[31426]: q1F5VuPd031426: <-- MAIL From: SIZE=6743
[31426]: q1F5VuPd031426: --- 250 2.1.0 ... Sender ok
[31426]: q1F5VuPd031426: <-- RCPT To:
[31426]: q1F5VuPd031426: --- 250 2.1.5 ... Recipient ok

[31426]: q1F5VuPd031426: <-- DATA
[31426]: q1F5VuPd031426: --- 354 Enter mail, end with "." on a line by itself
[31426]: q1F5VuPd031426: from=, size=6848, class=0, nrcpts=1, msgid=<8aa8ddfdc691cb86896329126a4eb6ef@bugs.userapp.com>, proto=ESMTP, daemon=MTA, relay=localhost [127.0.0.1]
[31426]: q1F5VuPd031426: --- 250 2.0.0 q1F5VuPd031426 Message accepted for delivery
[31425]: q1F5VtLr031425: to=user@example.com, ctladdr=userapp+admin@gmail.com (99/99), delay=00:00:01, xdelay=00:00:00, mailer=relay, pri=36743, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (q1F5VuPd031426 Message accepted for delivery)
[31428]: q1F5VuPd031426: SMTP outgoing connect on name.compute-1.interna
[31426]: q1F5VuPe031426: <-- QUIT
[31426]: q1F5VuPe031426: --- 221 2.0.0 name.compute-1.internal closing connection
[31428]: q1F5VuPd031426: to=, delay=00:00:00, xdelay=00:00:00, mailer=esmtp, pri=126848, relay=inbound.domain.netsolmail.net. [205.178.149.7], dsn=2.0.0, stat=Sent (OK FB/29-06630-5434B3F4)
[31428]: q1F5VuPd031426: done; delay=00:00:00, ntries=1  

Here is what I get using telnet:

# telnet smtp.googlemail.com 587
Trying 74.125.93.16...
Connected to smtp.googlemail.com.
Escape character is '^]'.
220 mx.google.com ESMTP j17sm7987765qaj.9


502 5.5.1 Unrecognized command. j17sm7987765qaj.9
STARTTLS
503 5.5.1 EHLO/HELO first. j17sm7987765qaj.9
EHLO localhost
250-mx.google.com at your service, [nnn.nnn.nnn.nnn]
250-SIZE 35882577
250-8BITMIME
250-STARTTLS
250 ENHANCEDSTATUSCODES
STARTTLS

220 2.0.0 Ready to start TLS

Any help greatly appreciated!

Answer

You seem to have a small typo in your password declaration, so try the following:

AuthInfo:googlemail.com "U:user@gmail.com" "P:xyzzy" "M:PLAIN"
AuthInfo:google.com "U:user@gmail.com" "P:xyzzy" "M:PLAIN"

Do not forget to run makemap and rebuild authinfo.db:

makemap hash authinfo < authinfo