We’ve looked after our own mailing list for years. We had a nice database of users hosted on the website that was pretty flexible, easy for us to use and very easy to which to sign up.

But sending out mail is another matter entirely. We currently have around 700 addresses on our list, and it has become increasingly difficult to find a server that is prepared to send out that many messages. We had to resort to splitting mailouts, VPNing to York and dodging angry system administrators in order to get our mail out. And culling the dead emails from the list wasn’t much fun either.

So we’ve finally given up and got a York Lists account. In many ways it’s great: emailing the list is now as easy as, um, sending an email, and it deletes invalid addresses automatically.

A nice personality

But it’s a bit of an ugly thing. We used to have this signup form. The lists service gave us this one instead. It doesn’t match the rest of the site and provides far too much useless information on screen.

It would be nice to use the old signup form with our new lists account. And luckily we can.

Submitting to the lists service

First, create a form that submits to the lists service. I made this by copying the fields from the lists subscription form. Change the form action to point at your list. I’ve hidden a few of the fields as I don’t consider them important.

<form class="SubscribeForm" method="post" action="https://lists.york.ac.uk/mailman/subscribe/breakz-soc">
<h3>Sign me up!</h3>

<label for="name">Name</label>
<input class="SubscribeText" id="name" type="text" name="fullname" size="35" maxlength="255" />

<label for="email">Email</label>
<input class="SubscribeText" id="email" type="text" name="email" size="35" maxlength="255" />

<input type="hidden" name="pw" value="" />
<input type="hidden" name="pw-conf" value="" />
<input type="hidden" name="digest" value="0" />

<input class="SubscribeSubmit" type="submit" name="email-button" value="Subscribe" title="Click to submit details and join mailing list." />
</form>

The form should now work nicely, but unfortunately you’ll end up on the lists service site after submitting. It would look a lot more professional if we could stay on site throughout.

Keeping it on the site

And we can do this with a bit of AJAX. When someone submits the form, we send the data on to the lists service, get back its response and return a success (or failure) message. The AJAX itself is pretty simple, but unfortunately we’re not allowed to send data cross domain for security reasons.

Luckily we can get round this by using an AJAX proxy, which takes a local request and redirects it to an external domain. The one I used is the PHP AJAX proxy by Iacovos Constantinou.

We need to add a couple more elements to the form so we can give feedback to the user. I stuck these just after the submit button.

<img id="ajaxBusy" src="http://www.breakzdjs.com/images/progress.gif" alt="Busy..." />
<div id="ajaxResult"></div>

Apart from that, the form is identical. (So it still works for those without JavaScript.)

Driving the form with JavaScript

I used jQuery because I think it’s great. Along with other JavaScript libraries, it has a nice, high level interface to the AJAX functionality.

The code below intercepts the form submit function and posts the form data on to the AJAX proxy. Once we get a return value from the AJAX proxy (the ‘thanks for subscribing’ HTML page from the lists server), we process it to see what the list server said. The server success or error message is between the </h1> tag and the <hr> (sic.) tag, so we use a regexp to cull the rest.

The list server returns a verbose (and largely uninteresting) message if the subscription succeeds, starting with “Your subscription request has been received, and will soon be acted upon.” We search for that and, if it’s there, show a rather terser and to-the-point message for the user. If not, tell the user there’s a problem and print the server response. The code also throws up a link to the lists server subscribe page, just in case something funny is going on at this end.

jQuery(document).ready(function()
{
	var ajaxProxy = "http://www.breakzdjs.com/ajax-proxy.php";
	var listUrl = "https://lists.york.ac.uk/mailman/subscribe/breakz-soc";
	var signupUrl = "https://lists.york.ac.uk/mailman/listinfo/breakz-soc";

	jQuery("form.SubscribeForm").submit(function()
	{
		jQuery("#ajaxBusy").show();

		jQuery.post(ajaxProxy,
		{
			csurl: escape(listUrl),
			fullname: jQuery("#name").val(),
			email: jQuery("#email").val(),
			pw: "",
			digest: "0"
		},
		function(data)
		{
			jQuery("#ajaxBusy").hide();

			// Remove HTML guff.  Interesting bit is between </h1> and <hr>.
			data = data.replace(/(^(.|\n)*<\/h1>|<hr>(.|\n)*$)/gi, "");

			// Failed
			if (data.search("Your subscription request has been received, and will soon be acted upon.") == -1)
			{
				jQuery("#ajaxResult").html("<h3>Ooh dear.  Something went wrong.</h3><p>" + data + "</p><p>Keeps going wrong?  <a href=\"" + signupUrl + "\">Try signing up on the mailing list server&nbsp&raquo;</a></p>").fadeIn("fast");
			}
			else
			{
				jQuery("#ajaxResult").html("<h3>Thanks!</h3><p>You should get an email from the list service we use to confirm your subscription.  Just click the link in that and you&rsquo;re done.</p>").fadeIn("fast");
			}
		});

		return false;
	});
});

Ready to rumble

And that’s it. See it in action

• On our new subscribe page
• On the Pole Exercise site
• Download the code

Don’t forget to edit all of the files so they point to your lists URL.

Things have been largely positive, but there are a few gotchas we’ve come up against so far. I’m going to provide a quick setup guide and then outline these issues so you hopefully don’t have to follow the same (occasionally torturous) path we did.

Should I switch to Gmail?

Advantages:

• Much more space (4GB vs. 50MB)
• Emails not eaten by MessageLabs – MessageLabs is the spam solution used on the York email accounts. Although effective, it does sometimes delay (by hours or days) a significant number of legitimate messages, or occasionally doesn’t deliver them at all.
• The web access interface is much nicer than the York one

Disadvantages:

• Spam protection not quite as good – We get about 200 spam messages per day through to Breakz. Almost all are filtered out, but Gmail seems slightly worse than MessageLabs was.
• Unclear about the TOS / SLA / privacy (though this is largely true for the York solution too)
• Getting mail out of Gmail if you or YUSU decide to move to another system at some point (this can be overcome with POP / IMAP – see below)

So it seems like a good thing to do.

Setting it all up

Log in to Gmail. Change your name in the settings bit (top right).

Go to the York email settings page (use your socs account to log in) and set the account to forward to your new YUSU address (“please divert my mail to”).

Anything else?

If you’re just planning on sending a few messages and only using the Gmail web interface then you’re done. Pat yourself on the back and go and have a coffee or something. Go on, shoo!

Otherwise, please read on…

Using your own domain

For a number of years, we’ve used info@breakzdjs.com as our public-facing address. Until recently, this just forwarded to our socs account. We’ve now changed this so it goes to the Gmail one instead.

As well as looking more professional, this solution means we can move the actual location of our email around (as we have just done) without changing our contact details. If you’d like to do something similar, your host’s control panel should have email forwarding options.

If you do this and use the Gmail web interface, you should set up a reply-to address in the Gmail settings so mail doesn’t appear to return to your YUSU account. You should also do this in your mail client if you’re using the Gmail SMTP server (see next) as Gmail will helpfully change the from address to your @yusu.org one.

Accessing Gmail through an email client

If you’d rather read your mail with Outlook or Thunderbird (and I have no idea why you wouldn’t want to), Gmail supports both POP and IMAP access. You can enable both via the web interface’s settings bit (Forwarding and POP/IMAP tab).

Google’s implementation of POP is non-standard, as it tries to sync your web and local inboxes. This works fine with just one person reading the mail, but breaks if (as is likely) more than one person is accessing the account via POP. The first person gets messages delivered as expected, but subsequent people will get nothing. Clearly, this is shit. (It also breaks if you read the mail on more than one computer.)

A much better choice is Gmail’s recently released IMAP option (as used by York for the past 10+ years). At the moment, to get IMAP to show up, you have to set the interface language to ‘English (US)’.

Update: The way Gmail does IMAP is non-standard too. In particular, deleting a message from the inbox does not, actually, delete it. There’s a list of how IMAP and Gmail actions sync here.

Google guides:

• How to set up POP access
• How to set up IMAP access

Sending to large mailing lists

Gmail places quite tight restrictions on the number of emails you can send per day. It’s only possible to email 500 people per day from the web interface, and just 100 via Google’s SMTP server. If, like us, you have over 500 people on your mailing list, this is clearly useless.

There are two principal ways round this:

• Use a list management service
• Use a different email server

Using a list management service

The university provides a mailing list service that quite a few societies already use. There’s info on the site about how to apply for one.

You could also consider the YUSU mailing / membership list service (part of the new website – log in as your society to access), though when I tried it briefly it appeared to be rather unfriendly and somewhat broken.

We don’t use either of these options as we want to be able to record more information about our members (e.g. whether they are interested in DJ lessons), occasionally need to email specific groups (newly registered, DJs, etc.) and want to allow people to join the list via our site.

Using a different email server

If you’re on campus, you can get round this by using the York SMTP server. Get rid of the Gmail outgoing server settings (don’t forget to reset the authentication to default) and enter smtp.york.ac.uk instead.

If you’re off campus, you may be able to use your ISP’s server (check with your ISP for settings). However, quite a few ISPs are even stricter than Gmail on sending email. If you’re with one of them, you can use the York SMTP server once you’ve connected to the university web cache or VPN. (You may also be able to use your department’s email server and VPN instead.) Unfortunately this means you’ll have to connect to the VPN each time you want to send an email to your mailing list.

Think that’s it! Hope that helps – otherwise, post a comment or get in touch.