Fun with Joe Jobs
by Ed Sawicki
Accelerated Learning Center
Tailored Computers
October 4, 2007
This article is being written as events unfold. It's about 1:30 AM on October 4, 2007 and I'm writing a chapter of my latest book. A few minutes earlier, I glanced over at my mail server's status screen and noticed activity. My mail server is configured to display activity using a format much simpler than the standard log files. Right now it looks like this but the screen is scrolling at about one line per second:
01:32 NOSUCHUSER: mta-in2.mtapool.hostyours.com (206.83.74.51) FROM:<> TO:biera-lehtinen@panug.org 01:32 NOSUCHUSER: deep-oral-sluts.com (66.197.17.122) FROM:<> TO:man717@panug.org 01:32 NOSUCHUSER: mta-in2.mtapool.hostyours.com (206.83.74.51) FROM:<> TO:mistry@panug.org 01:32 NOSUCHUSER: cumeils.prima.com.ar (200.42.0.138) FROM:<> TO:vrouenraets@panug.org 01:32 NOSUCHUSER: cumeils.prima.com.ar (200.42.0.138) FROM:<> TO:ciaran@panug.org 01:32 NOSUCHUSER: cumeils.prima.com.ar (200.42.0.138) FROM:<> TO:ermanno912@panug.org 01:32 NOSUCHUSER: mail.sohonet.hu (212.52.166.7) FROM:<> TO:kesslerwsdx@panug.org 01:32 NOSUCHUSER: mail.sohonet.hu (212.52.166.7) FROM:<> TO:sunnie@panug.org 01:32 NOSUCHUSER: mail.sohonet.hu (212.52.166.7) FROM:<> TO:maert_waide@panug.org 01:32 NOSUCHUSER: mta-in2.mtapool.hostyours.com (206.83.74.51) FROM:<> TO:serafin.siippainen@panug.org 01:32 NOSUCHUSER: mta-in2.mtapool.hostyours.com (206.83.74.51) FROM:<> TO:hvamstg@panug.org 01:32 NOSUCHUSER: mail.sohonet.hu (212.52.166.7) FROM:<> TO:ayse217@panug.org 01:32 NOSUCHUSER: mta-in2.mtapool.hostyours.com (206.83.74.51) FROM:<> TO:keaten.kasprick@panug.org 01:32 NOSUCHUSER: mta-in2.mtapool.hostyours.com (206.83.74.51) FROM:<> TO:maisano@panug.org 01:32 NOSUCHUSER: mta-in2.mtapool.hostyours.com (206.83.74.51) FROM:<> TO:laing@panug.org 01:32 NOSUCHUSER: mta-in2.mtapool.hostyours.com (206.83.74.51) FROM:<> TO:cecilia.hammel@panug.org 01:32 NOSUCHUSER: mta-in2.mtapool.hostyours.com (206.83.74.51) FROM:<> TO:hae496@panug.org 01:32 NOSUCHUSER: mail.sohonet.hu (212.52.166.7) FROM:<> TO:blue@panug.org 01:32 NOSUCHUSER: mta-in2.mtapool.hostyours.com (206.83.74.51) FROM:<> TO:tobey@panug.org 01:32 NOSUCHUSER: mta-in2.mtapool.hostyours.com (206.83.74.51) FROM:<> TO:lantermandptcc@panug.org
This is telling me that the mail system is receiving mail addressed to non-existant users in the panug.org domain. The mail is coming from legitimate mail servers making it appear that clients of these ISPs are sending the spam (hacked Windows boxes). However, notice that the FROM: field has no address. If these mail servers were sending mail from their clients, the FROM: field would contain an address of some sort.
It's time to look at some of these mail messages. Right now my mail server is rejecting all of them because they're not addressed to users on the system. I'll direct them to my inbox. I make one addition to my Postfix virtual server file:
@panug.org edand reload Postfix:
postfix reload
Within seconds, my inbox starts filling with messages. I quickly remove the line I added to the virtual server file and reload Postfix to make it stop. I read the first message. It says:
This is the mail system at host mta-in2.mtapool.hostyours.com. I'm sorry to have to inform you that your message could not be delivered to one or more recipients. It's attached below.followed by the message it could not deliver. The message is about enlarging genitalia and includes this URL if I want to buy their product:
http://www.marionshop.com/
The message contains this header line:
Received: from [84.171.230.169] (p54ABC605.dip.t-dialin.net [84.171.198.5]) by mta-in2.mtapool.hostyours.com
The host at 84.171.230.169 is sending out spam and using the panug.org domain to do it. This is called a Joe Job in email circles. The spammer sends out mail to lots of hosts on the Internet using random panug.org FROM: addresses and my server gets swamped by the rejection messages. The stream of rejection messages is called backscatter. The danger is that my mail server will be blacklisted by those who aren't clever enough to realize that my server is a victim. I need to do something about this quick.
I spend a minute deciding what to do. I can't prevent the spammer from sending spam with the panug.org address. It's unlikely I can get the ISP to do anything quickly. What do I do now to stop the backscatter?
Aha, I can change the panug.org DNS to point to another mail server. Then that server will get blacklisted instead of mine. I can point it to:
- The host sending the spam; 84.171.230.169
- The mail server for the genitalia enlarging company; marionshop.com
- The ISP's mail server
I do a whois on 84.171.230.169 to see who the ISP is and get their abuse address. I send them a message telling them of the problem and ask them where I should point my MX record, which will determine where the backscatter goes to. I bet they've never had that question before and won't know how to respond. For now, I decide to point my MX at the spammer: 84.171.230.169.
I edit the panug.org DNS file and see that I wisely set the TTL to only 60 seconds. Most books on DNS recommend setting the TTL to a much higher value; not my book. I want DNS changes to take effect quickly and this is one situation where rapid changes are nice. I do a DNS lookup of 84.171.230.169 and see that the host name is p54ABC605.dip.t-dialin.net. I change the MX line to:
MX 1 p54ABC605.dip.t-dialin.net.and reload the zone. Almost exactly one minute later, the mail server status screen stops scrolling. The backscatter is now going back to the spammer.
The spammer is probably a hacked Windows box - there's nobody there to notice or to care. Nobody is going to have any incentive to stop sending spam using panug.org addreses. Now that I have time to think, it's probably better to send the backscatter to either the ISP or to marionshop.com. I choose marionshop.com.
A whois tells me marionshop.com is in Omsk, Russia. A dig tells me their mail server is marionshop.com. So I alter my DNS so the panug.org MX record points to marionshop.com and I reload the DNS. In one minute they'll start getting the backscatter. I wonder what they're going to do.
About a half-hour later I still haven't heard from the spammer's ISP. They need a nudge. I add another MX record (with the same priority value) to the panug.org DNS file with the ISP's mail server name. I notice that panug.org has a SPF record. I remove it and reload the DNS. Sixty seconds later, the backscatter is going to both the ISP and the genitalia enlarging company. Now I wait.
For the next 11 hours I would occassionally place a third MX record in the DNS, listing my mail server with the same priority as the other two. I would see that the spammer was still spamming and I'd remove my server. Then, after 11 hours, it stopped. I have no idea whether the spammer just ran out of addresses to send spam to, if the genitalia enlarging company made it stop, or whether the ISP blocked the spammer. The ISP never responded to me, but that's typical.
The following sections are being written days after the attack.
Postmortem
Since the joe job attack, I've asked associates who work in the IT industry how they would handle such an attack. All of them said they would not be able to solve the problem themselves - they would call their ISP for help. One said his ISP would tell him the solution is to block all incoming email for the duration of the attack. Not wishing to turn off all incoming email, he would "ride out the storm", hope that some legitimate email would squeak through, and deal with the block lists later.
When most of these people heard how I solved the problem, they had lots of comments and questions - mostly about DNS:
How can you have your mail deliberately sent to someone else?
But it would take hours for changes to the DNS MX records to take effect! How did you do it in 60 seconds?
Isn't a TTL of only 60 seconds poor practice and violates some RFC?
Our ISP handles our DNS and our requests for changes may take hours or days.
We can't have our email sent to other servers. Are you nuts?
This tells me that most IT shops are not equipped to deal with Internet-borne attacks, such as the type described in this article. They've never thought about it. It's certainly not on their list of disaster planning (business continuance) concerns.
Since the attack, I've developed a few techniques for dealing with attacks like this in ways that are far more acceptable than the brute force solution I used that night, turning off all incoming email, or riding out the storm. The solutions involve knowing how to think about the problem and being a little clever. Oh yes, and being capable of getting around the limitations of popular software, such as DNS software.
I've already started adding features to my company's eScrubber email service to deal with these attacks.
Revealing Information
As a result of the Joe Job attack, I now have a few thousand email messages from the mail servers that bounced the non-deliverable spam back to me. I've been going through them just for research purposes. Some are revealing - too revealing. Here's an example and my favorite. I've obfuscated the actual email addresses:
A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:
pipe to |/home/amail/log-bounces
generated by amail@example.com
(ultimately generated from bc@example.com)
local delivery failed
The following text was generated during the delivery attempt:
------ pipe to |/home/amail/log-bounces
generated by amail@example.com
(ultimately generated from bc@example.com) ------
DBD::mysql::st execute failed: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '' at line 1 at
/home/amail/log-bounces line 69, line 23.
DBD::mysql::st fetchrow_hashref failed: fetch() without execute() at /home/amail/log-bounces line 71, line 23.
Can't use an undefined value as a HASH reference at /home/amail/log-bounces line 71, line 23.
This appears to be a mail server using some sort of Perl program and the MySQL database that's been written or configured improperly. Because of the way this system has been engineered, outsiders now know of the problem, which may be an exploitable security hole.
Here's another example that shows another possible security hole:
designr2@cp48.example.net
(generated from finance@example.ru)
mailbox /home/designr2/mail/inbox has the wrong mode 644 (660 expected):
retry timeout exceeded
Well-written mail systems would not reveal these errors to the world.