Free email validation API for web forms

Email validation is a hard problem that every single developer, building anything on the web, has to deal with.  We actually have been working on email validation ourselves for years (more on that below). We looked at the validation services available and weren't satisfied with any of them, either for performance, accuracy or completeness reasons.  So today, we're releasing a project we affectionately called Guardpost, as our newest API, and are also pulling back the curtain to show exactly how we built it.

We're launching this as a free service that anyone collecting email through web forms can (and should!) use.   You'll need a Mailgun account to use the service, but you don't have to send your emails through Mailgun. If you want to get started right away, check out the API documentation, or a little sample jquery app, as an example of how to use Guardpost in a signup form.

We suggest using Guardpost as part of your email collection form to reduce typos and false address submission in conjunction with a link emailed to the address to confirm the email is valid (double opt-in). Of course, you can use Mailgun to send the double opt-in email, as well. This is not intended to be a bulk mailing list cleaning service and we reserve the right to disable your account if we see it being used as such.

To call the Guardpost API, just use the publishable API Key in the My Account tab of your Mailgun account (the one with the "pub-key" prefix).

Now, on to to the technical details:

Why is email validation so hard?

Address validation is hard for multiple reasons:

  1. Email address syntax is fairly complex, complex enough that it is difficult to express with a regular expression, while being readable and maintainable for developers.
  2. There is no single definition for what is valid syntax, for an email address, and what is not. The definitions that do exist frequently conflict.
  3. The Internet runs on the Robustness principle, and because of that mail servers will accept addresses that do not conform standards, but are otherwise understandable.

Why did we create an Email Validation API?

There are 3 main reasons we feel like we needed to build our own service.

  1. Our goal is not to make a perfect address validator that can validate every single address that has ever been created. Our goal is to build a realistic address validator for the types of addresses we see everyday.
  2. We've sent billions of emails and collected a lot of ESP data. We know that gmail.com is a valid MX host while gmali.com is not.
  3. Furthermore, the validator is ESP specific, so we can go way beyond valid syntax checks, bring in specific requirement for Gmail vs. Yahoo vs. Hotmail.

 

What does the Validation service do?

Our validator service actually consists of five micro-services:

1. A recursive descent parser for syntax validation

Email address syntax is fairly complex, enough to make a pure regular expression based approach cumbersome and unmaintainable (check out RFC 2822 and RFC 5322 about proper email format then this discussion on Stackoverflow if you need some convincing). That why we wrote a parser that analyzes addresses, and determines if they are valid or not, based on a formal grammar.

What is a formal grammar? Formal grammars (and specifically in our case a context free grammar) are a set of rules that define the structure of a string. For example, it allows us to transform something we intuitively understand, like an address list, into something formal that a computer can parse.

So what would the context free grammar for an address list look like? Something like this:

address-list -> address ( delimiter address )*

What we have defined here is an address list, and we are saying it is composed of a single address, followed by zero or more delimiter and single address pairs. For example the following would be a valid address list:

john@example.com, smith@example.com

While the following would not be:

john@example.com smith@example.com

What's really nice about recursive descent parsers is that we can take the grammar rules and turn them into code in a fairly straightforward manner. Here is pseudo-code for the above address list example:

   def get-address-list():
      address_list = []

      # get address
      address = get_address()
      if address is None:
         return None
      address_list.append(addr)

      while True:
         # get delimiter
         delimiter = get_delimiter()
            if delimiter is None:
               break

         # get address
         address = get_address()
         if address is None:
             break
         address_list.append(address)

      return address_list

Just like that, one by one, we slowly built grammar for every part of an email address. We spent hours pouring over RFCs, looking at bounces, looking at what mail servers actually accept (which is different sometimes from what RFC says), reading how other people were solving this problem to eventually define our own context free grammar for email addresses:

   address-list      ->    address { delimiter address }  
   mailbox           ->    name-addr-rfc | name-addr-lax | addr-spec | url

   name-addr-rfc     ->    [ display-name-rfc ] angle-addr-rfc
   display-name-rfc  ->    [ whitespace ] word { whitespace word }
   angle-addr-rfc    ->    [ whitespace ] < addr-spec > [ whitespace ]

   name-addr-lax     ->    [ display-name-lax ] angle-addr-lax
   display-name-lax  ->    [ whitespace ] word { whitespace word } whitespace
   angle-addr-lax    ->    addr-spec [ whitespace ]

   addr-spec         ->    [ whitespace ] local-part @ domain [ whitespace ]
   local-part        ->    dot-atom | quoted-string
   domain            ->    dot-atom

   word              ->    word-ascii
   word-ascii        ->    atom | quoted-string
   whitespace        ->    whitespace-ascii

We built our parser around the above grammar for what we think is a realistic email address syntax.  Again, this is not just based on RFC, but what we see ESPs accepting from sending billions of emails.

2. Domain Name Service (DNS) lookups

Just because an email address is syntactically valid, doesn't mean that anyone will receive mail at that address. To receive mail, you have to have a server that will listen for incoming messages, this server is called a Mail Exchanger (MX) and is usually defined in your DNS records. That's why, in addition to syntax checks, we look up the domains DNS records to see if a MX server has been defined to accept mail.

3. Mail Exchanger existence checks

Again, due to the robustness principle, just because a host does not define MX records does not mean they can't accept mail. Mail servers will often fall-back to A records to try and deliver mail. That's why we go one step further than just a DNS query, we ping the Mail Exchanger to make sure that it actually exists.

4. Custom Email Service Provider (ESP) grammar

Being liberal in what you accept is just one part of the robustness principle, the second is be conservative in what you send. Because of that, most ESPs actually have a fairly stringent rules for the local-part (before the @ symbol) you can actually create. For example, Yahoo Mail addresses can only contain letters, numbers, underscores, and one period. So while an address like, "John Smith"@yahoo.com is completely syntactically valid, it does not actually exist at Yahoo Mail and will bounce. That's why if we know the Mail Exchanger the mail is going to, and we know the big ones like Yahoo, Google, Microsoft (including Hotmail), AOL, and Apple we validate against their more stringent rules.

5. Suggestion Service

Email addresses are frequently mistyped. Instead of @yahoo.com, you might type @yaho.com, that's why, as part of our validation service, if we detect a misspelled word, we offer suggestions so you don't miss mail due to a typo.  Here's what that looks like in the jquery demo app [source] we mentioned above.

What we don't provide.

We've talked a lot about what we provide, but there are a few things we should make clear we don't provide:

  1. We don't check if a mailbox actually exists on a server. This used to be a feature of SMTP, but due to abuse by spammers, this was disabled by most major ESPs.
  2. We are not a mailing list cleanup service. This service is meant to be used to increase sign-up accuracy, not to cleanup mailing lists purchased from questionable sources. We will be monitoring usage and reserve the right to shut down anyone that is using it inappropriately.

So that's it.  We hope you enjoy the service and it makes your life easier.  If you have any questions or comments, let us know.

Happy emailing!

The Mailgunners

comments powered by Disqus

Mailgun Get posts by email

Like what you're reading? Get these posts delivered to your inbox.

No spam, ever.