Email Delivery, inbound and outbound
Just like with email aliases, forwards and mailboxes, the way that email works can often be described by analogy with regular postal mail (marked with italics).
- Email is delivered from the sender to the recipient's mailbox using SMTP (Simple Mail Transfer Protocol). This is the same for incoming or outgoing mail.
- The program that is responsible for accepting the delivery of email or sending it on is called an MTA (Mail Transfer Agent). In the NetManager, we use sendmail for this.
- Your email client (e.g. Thunderbird, webmail, Outlook) is separate from this. It will use POP3 or IMAP to access the email in your mailbox, but what we are discussing here is how an email server sends and receives email.
- Sending email and receiving email are different even though SMTP is used for both. This is an important distinction. While the method of sending and receiving postal mail is the same (putting an envelope into a slot), they are obviously very different in practice. To receive mail, it gets put through your letterbox by a postman, but to send mail, you don't just push out letters backwards through your letterbox. You need to leave the house and actively make it happen.
- To receive email, the server (MTA) listens out for incoming email. This is the same as you sitting by the letterbox waiting for a delivery. If mail does not get delivered to your letterbox because of being badly addressed, lost or held up, there's not much you can do about it.
- To send email, the server may either:
- Deliver it directly to the recipient. This is like you putting a letter through someone's letter box
- Send it via an email relay which will then handle the delivery (perhaps via another email relay down the line). This is like you putting a letter in a Royal Mail postbox and trusting it to be delivered
- Some Internet connections, especially those for schools, are very restrictive and do not allow you to send or receive mail directly. All email is forced to go via the ISP's servers. This is akin to you being in a prison where all your mail in and out has to pass through the guards. People trying to deliver to you directly will be stopped at the gate. If the guards choose to throw your mail away (or just accidentally lose it), you probably won't know and neither will people sending you mail.
- Because it is free to send email (whereas it usually costs a stamp and an envelope to post a letter), it is very easy to send junk email (a.k.a spam). Therefore many email servers are choosy about what email they accept and will:
- look at the email itself to decide whether to accept it. This is like the postman asking you to sign for a letter that you can see is junk from its envelope. If you refuse to sign, it will be returned to the sender
- look at the server attempting to deliver the mail to decide whether to trust it. This is similar to you refusing to accept any deliveries except from a Royal Mail postman or a courier company you know.
- Related to the above, if people decide not to accept your email or mail, you cannot force them to do so and they may not tell you why they do not want to accept it. For postal mail, you could try to improve your reputation by sending it in a nice envelope and sending it via a trusted courier. This is similar to you throwing away junk without opening it if it is clearly junk, but choosing to open plain envelopes (just in case it's something exciting!).
- Because of the problem of junk email or spam, most email servers will not accept email unless either:
- the recipient is someone it knows about, i.e. it will be delivering it to a local mailbox This is the same as you only accepting mail that is addressed to you.
- the sender is someone it trusts because it is part of the same network, e.g. you are sending email out from your computer to your ISP's server for it to deliver on your behalf. This is the same as you using a franking machine for your mail, you need to have the right frank on the envelope for Royal Mail to accept it
- The domain part of an email address (the bit after the @) is used to decide where to deliver email to. The server trying to send email to a domain looks up a list of email servers that will accept email in a central directory (technically, it looks for MX records in DNS). It will try to deliver the email to each of those servers in descending priority order. If the server that takes the email is not the highest priority it will try to deliver it onwards for you. This is similar to you specifying "please deliver to neighbour" when ordering something. It may be delivered directly, but if not, your neighbour will bring it round
Email is not being delivered to you
- If your MX records in DNS are wrong (and these may not be under your control), then email will be delivered to the wrong place.
- You may have all email delivered to the ISP's email relay who should then forward it on. They may forget that they should be doing this and try to deliver it elsewhere instead. This is especially hard to solve if your ISP stops you receiving email directly.
Case study 1:
An unnamed LEA kept changing the MX records and/or internal servers so that email for a school no longer got delivered to the school's mail server.
Solution: take control of the school's domain name and set email to be delivered directly to the school server with our servers as MX backups.
When you send email out, it just disappears
- This will only happen if sending email out via another server (e.g. your ISP's mail relay). If you send out directly, then you would get bounces on errors or get information in the logs.
- The server you are sending out via may be silently throwing away all email or it may be trying and failing to deliver, but not letting you know
Case study 2:
The same unnamed LEA altered their published email relay, which had worked for years, so that all email was accepted but discarded.
Solution: check that port 25 is not blocked outbound and send email out directly. If port 25 is blocked, subscribe to an email-forwarding service, locate other ports that may be open such as 2525 and 587 and use these to send out via the email-forwarding service
When sending email out directly, most email is delivered, but you cannot email anyone at your ISP
If your NetManager is on the ISP's private range of addresses, it may not be possible to speak to their external-facing email servers because of firewall configuration problems at their end incorrectly assuming that you would only speak to their email servers from outside
Case study 3: The very same unnamed LEA have email servers for their gov.uk domain that cannot be contacted from within the LEA network. When their email relay was being used, it could deliver the emails because of exceptions they'd made to allow this, but emails sent directly time out.
Solution: use mailertables within NetManager to deliver the gov.uk domain to the internal email server (perhaps the email relay will still work for that) or subscribe to an external email-forwarding service and use that.
When sending email out either directly or via an email relay, most email is delivered, but some emails are either delayed or bounced with messages relating to IP having poor reputation
Multiple customers will be using the email relay or the ISP's firewall. To servers in the outside world, it looks like all the email from all those customers is coming from a single IP address. If just one computer at one of those customers has a virus that causes it to send spam, this can make the single IP address get blacklisted and some servers will refuse to accept mail from it.
Case study 4: The Internet service through an unnamed education-focused ISP blocks all direct email outbound on port 25, therefore you need to use an email relay. Unfortunately, the ISP's email relay keeps getting blacklisted as a spam-sender due to it being used by many schools (some of which have virus-infected computers).
Solution: subscribe to an email-forwarding service, locate other ports that may be open such as 2525 and 587 and use these to send out via the email-forwarding service
Case study 5: A NetManager had two IP addresses set that could be used to talk to the Internet (IP1 and IP2). This was for historic reasons as it had replaced multiple other servers. The school's dedicated remote access was configured to point to IP2, but IP1 was the primary address on the NetManager as this is where email was being delivered to. When IP2 talked to the Internet, all traffic appeared to come from the dedicated remote access address, but when IP1 talked to the Internet, it just used the ISP's standard firewall IP address which was blacklisted for having sent spam.
mail_source to the remote access IP address (IP2) so that all mail is sent from that address
When sending email out directly, most email is delivered, but some emails are either delayed or bounced with messages relating to PTR or reverse DNS
Just as you can convert a network name such as mail.google.com to an IP address, you can also attempt to do the reverse by looking up what are known as PTR records. Setting up PTR records is generally out of your control as they are managed by the people responsible for allocating your Internet IP addresses. Because this requires input from an ISP and so is not widely open to abuse, some mail servers attempt to do such a reverse lookup and refuse to accept email that comes from a server that does not have a PTR record. ISPs often overlook setting up PTR records.
Solution: if your mail server has a name such as
mail.domain.sch.uk and its IP address is
18.104.22.168, then ask your ISP to set up a PTR record for
22.214.171.124 to point to
When sending email out directly, most email is delivered, but email to multiple Google accounts is delayed (perhaps with "Warning: could not send message for past 4 hours" messages)
The NetManager will try to economise on the number of connections it makes to destination email servers. If is notices that the destination email server is the same for multiple recipients of an email, it will only send the email once. This is all as it should be. Unfortunately, Google email servers only accept email for one domain at a time, i.e. you can send a single email to email@example.com and firstname.lastname@example.org with no problems, but if you try to send it to email@example.com, then the email to users @domain2.com will be deferred with the message
451-4.3.0 Multiple destination domains per transaction is unsupported.
Solution: this is not a problem, the NetManager will try again and the email will be delivered to all users even if it takes multiple attempts.