What The Heck Happened?
Our site and both forums were down for about 16 hours yesterday (30 Aug) and today. We assumed it was a server problem, filed the normal request for info, and finally went home and got a good night's sleep. This morning, we found out what happened.
We have a computer (named "wildcat") used by the marketing director. It is used (among other things) to send STAR FLEET ALERT, the PDF press releases about new products, to retailers, wholesalers, gamers, and media who ASK to get them. After Vanessa left, I would send the Star Fleet Alerts myself. I would do them, Email them to "marketing@", then go down the hall and re-mail them to the "batches" of Email addresses Vanessa had collected. Whenever I went down the hall to that computer to send the Alerts, I would deal with any Email had arrived for "marketing@" or "support@" which had not been read in 2-3 days (since nobody worked at that computer). We decided two weeks ago that this was bad customer service and had all of the mail from those two addresses redirected to me, and I deal with it each day as it arrives.
This had an unintended consequence, as the first time I sent the PDF by Email to that computer, it redirected back to my computer. So I had Matt set up an Email address (Wildcat@) that does not forward to me so I could Email things to that computer. (I could just pop them over there via the LAN, but I was overcome by ennui.) A day or two later, on 24 Aug, I had a Star Fleet Alert to send out. So I Emailed it to Wildcat@ and walked down the hall and set up the five different Emails (Star Fleet Alerts, depending on what they are, go to various combinations of people; some only go to wholesalers, some only go to the media, etc. This one went to everybody) and transferred the Star Fleet Alert PDF to them. I didn't realize that it was sending them "from Wildcat@" instead of "from Marketing@" but when I noticed, I considered this to be almost irrelevant. When I saw what was happening, I changed that computer so that even if it wasn't receiving marketing@ Emails (which are still going to me), it was sending all of its email FROM marketing@.
Somebody got one of the 24 Aug "from Wildcat@" Emails, and not realizing it was something he had ASKED for, and not bothering to Email Wildcat@ and say "why are you sending this to me?", filed a complaint with his ISP (AOL), which forwarded the complaint to our host, who send us a notice of: "What are you doing sending spam? That is against the rules!" But we never got the notice because they sent it to an Email address we rarely use which has been "spoofed" by some Russian spammers who send a million spams a day with that as the phony return address. Of those million emails, anything that doesn't go through (over 1,000 Emails per DAY) bounces back as "this Email did not go through" to ... me as "the guy who sent it" even though, of course, I never sent it. So I had every spam filter known to man added to that address and one of them trashed the notice from our host, and the follow up "why didn't you answer our Email" notice. When they didn't get an answer, they locked down our account so they didn't get sued (or shut down) for sending Spam. We Emailed them asking what happened and only got the answer only because I (suspecting something was up) visually scanned over 1,000 "killed Russian spams" to find it.
The problem was solved within minutes (our host is VERY good!), and steps are being taken to make sure it never happens again. The host will now send such warnings to every Email address in the company, a special pre-filter will pick out everything they send and put it into a priority folder, and a special note was be sent to every AOL address on the mailing batch (and the culprit fessed up and apologized for the panic, saying he gets so much spam he automatically forwards everything he doesn't recognize to the complaint desk).
I want to thank all of our friends who called or Emailed to warn us that something was happening (all of whom knew they were probably wasting their time, but all of whom knew we'd rather hear 99 times than zero times). I am sure there is a lesson to be learned here (and I think we have already learned and implemented it), but for now, I just need to get back to work.
We have a computer (named "wildcat") used by the marketing director. It is used (among other things) to send STAR FLEET ALERT, the PDF press releases about new products, to retailers, wholesalers, gamers, and media who ASK to get them. After Vanessa left, I would send the Star Fleet Alerts myself. I would do them, Email them to "marketing@", then go down the hall and re-mail them to the "batches" of Email addresses Vanessa had collected. Whenever I went down the hall to that computer to send the Alerts, I would deal with any Email had arrived for "marketing@" or "support@" which had not been read in 2-3 days (since nobody worked at that computer). We decided two weeks ago that this was bad customer service and had all of the mail from those two addresses redirected to me, and I deal with it each day as it arrives.
This had an unintended consequence, as the first time I sent the PDF by Email to that computer, it redirected back to my computer. So I had Matt set up an Email address (Wildcat@) that does not forward to me so I could Email things to that computer. (I could just pop them over there via the LAN, but I was overcome by ennui.) A day or two later, on 24 Aug, I had a Star Fleet Alert to send out. So I Emailed it to Wildcat@ and walked down the hall and set up the five different Emails (Star Fleet Alerts, depending on what they are, go to various combinations of people; some only go to wholesalers, some only go to the media, etc. This one went to everybody) and transferred the Star Fleet Alert PDF to them. I didn't realize that it was sending them "from Wildcat@" instead of "from Marketing@" but when I noticed, I considered this to be almost irrelevant. When I saw what was happening, I changed that computer so that even if it wasn't receiving marketing@ Emails (which are still going to me), it was sending all of its email FROM marketing@.
Somebody got one of the 24 Aug "from Wildcat@" Emails, and not realizing it was something he had ASKED for, and not bothering to Email Wildcat@ and say "why are you sending this to me?", filed a complaint with his ISP (AOL), which forwarded the complaint to our host, who send us a notice of: "What are you doing sending spam? That is against the rules!" But we never got the notice because they sent it to an Email address we rarely use which has been "spoofed" by some Russian spammers who send a million spams a day with that as the phony return address. Of those million emails, anything that doesn't go through (over 1,000 Emails per DAY) bounces back as "this Email did not go through" to ... me as "the guy who sent it" even though, of course, I never sent it. So I had every spam filter known to man added to that address and one of them trashed the notice from our host, and the follow up "why didn't you answer our Email" notice. When they didn't get an answer, they locked down our account so they didn't get sued (or shut down) for sending Spam. We Emailed them asking what happened and only got the answer only because I (suspecting something was up) visually scanned over 1,000 "killed Russian spams" to find it.
The problem was solved within minutes (our host is VERY good!), and steps are being taken to make sure it never happens again. The host will now send such warnings to every Email address in the company, a special pre-filter will pick out everything they send and put it into a priority folder, and a special note was be sent to every AOL address on the mailing batch (and the culprit fessed up and apologized for the panic, saying he gets so much spam he automatically forwards everything he doesn't recognize to the complaint desk).
I want to thank all of our friends who called or Emailed to warn us that something was happening (all of whom knew they were probably wasting their time, but all of whom knew we'd rather hear 99 times than zero times). I am sure there is a lesson to be learned here (and I think we have already learned and implemented it), but for now, I just need to get back to work.
posted by Federation Commander at 1:30 PM
<< Home