Dispatches From The Geeks

News and Announcements from the MCS Systems Group

Sending and receiving attachments may not work

It seems that the Exchange server almost all of us are using is blocking attachments that have executable files. What this means is that if you receive a file (including a ZIP file) that contains a windows executable, or even an executable shell script, it’ll get blocked. It’s supposed to notify you of this, but that appears to not be working reliably at the moment.

I’m working with CIS on this (from both a policy and execution standpoint), but in the meantime if you don’t get an email you were expecting, it may be because it contains a “forbidden” attachment. In these cases, until we figure this out, you should work with the person sending the attachment to get the file via some other method, such as sharing via a file service (box.com, Dropbox, Google, etc.) or send to an alternate account.

I’m really sorry about this — if I could turn it off right now I would, trust me. We’ll figure it out, though, and I’ll keep working with everyone involved to make sure the solution works for all sides. Please continue to let me know any issues you do come across. Thanks.

Written by Craig Stacey

February 1, 2014 at 12:20 pm

Posted in Uncategorized

CIS IT Maintenance Weekend January 17-19

Please see the following announcement from CIS. The only thing that seems to affect CELS is the telephone system work and some Argonne web applications being patched. Argonne Business Systems will be inaccessible on that Saturday.

IT Maintenance Weekend Coming January 17-19

Major maintenance activities are planned for the weekend of January 17-19. Work will commence at 5:00 pm on Fri, 1/17 and continue through 7:00 pm on Sun. 1/19. Expect that any laboratory network and core IT services may be effected throughout the weekend. Please note;

  • All telephone systems will be down for resiliency testing between 6:00 and 7:00 pm Friday. This includes emergency phones and 911 services. A PA announcement will be made when this happens. In the event of emergency, employees should contact emergency services from their cell phones (630-252-1911).

Major Impacts

Friday evening

  • All telephone systems will be down for resiliency testing. A PA announcement will be made. ~15 min. outage expected
  • All voicemail services down. ~2 hr. outage expected
  • Web applications down for patching


  • Building 201, network outage much of the day
    • do not expect to have network access or get to your desktop computers in Bldg. 201 on Saturday
  • Wireless networks, rolling outages throughout the day
  • Many business systems & SCADA down for the morning
  • Login servers down in the afternoon, majority of business application inaccessible
  • Web applications server upgrade, ~2 hr. outage expected


  • Most everything back to normal by early morning
  • Full verification throughout the day


Written by Craig Stacey

January 10, 2014 at 11:33 am

Posted in Uncategorized

Reduced coverage today

Due to the treacherous conditions, a number of the Systems team are working remotely today. We’re online and accessible, but any activities requiring in-person actions might be delayed. As always, report any issues to systems@mcs.anl.gov.

Thanks, and stay safe (and warm)!

β€” Craig

Written by Craig Stacey

January 7, 2014 at 10:09 am

Posted in Uncategorized

Brief outage of WordPress sites

At 5:00pm today there will be a brief outage of all the WordPress sites on press3.mcs.anl.gov including those below. This is so applied security updates can take effect. The outage is expected to only last a few minutes and an all clear will be sent when the server and sites are back. If this outage poses a problem please email systems@mcs.anl.gov so we can work with you on it.


Written by Craig Stacey

December 18, 2013 at 1:38 pm

Posted in Uncategorized

Next CIS Maintenance Weekend, Jan 17-19 2014

The next CIS maintenance weekend will be Friday January 17 through Sunday January 19, 2014. The scope of the work to be done on this weekend is still being determined. Please let us know if there are particular concerns you have with this window.



Written by Craig Stacey

December 12, 2013 at 9:31 am

Posted in Uncategorized

Mail migrations moving along

Hey, gang!

At my request, CIS took a break from the migrations while I was away on vacation over Thanksgiving. Now that I’m back, we’re going to continue them up until the break, then pick up again in the new year. The plan remains as it was — you’ll get a notification a couple of days before the migration will take place, one when it starts, and one when it finishes. Remember, if the migration slot you’ve been assigned doesn’t work for you, simply let them know you’d like to be rescheduled. I saw the first batch of announcements for Thursday evening’s migrations go out today. Thanks!

Written by Craig Stacey

December 10, 2013 at 3:12 pm

Posted in Uncategorized

Unscheduled WordPress Service Interruption


In coordination with the Cyber security office, our wordpress infrastructure is back online as of 14:40 this afternoon and we have restored all the wordpress sites listed in the initial message.

Very fortunately we were able to restore them to a known clean state from backups that were made at 01:00:00 11/21/20213 (one AM this morning). This means that very few, (possibly no), changes were lost. However if you have made any changes to any of these sites this morning then you will need to make those changes again. We regret the inconvenience, but it was necessary to be confident that no residual evil remained.

The short story on the incident is that a very new vulnerability in a wordpress plugin made it possible for a baddy to insert some content on the server at 10:49 Am this morning. We became aware of it by 11 AM and soon after had made the content made unavailable to the internet. The server was patched, restored to a known clean state and put back into production by 14:40.

If you would like more details please feel free to contact us at systems@mcs.anl.gov, we’d be happy to take the opportunity to brag about how well we managed the issue.

Written by Craig Stacey

November 21, 2013 at 3:21 pm

Posted in Uncategorized

Unscheduled WordPress Service Interruption


We regret to announce that we needed to disable access to some of the web sites that we manage. At this time we are investigating suspicious activity on many of our WordPress web sites.

For the time being the following sites are unavailable for web access, any attempt to access any of these sites will timeout:

argo-osr.org extremecomputingtraining.anl.gov press3.mcs.anl.gov symposium30.cels.anl.gov
cerisc.mcs.anl.gov inside.cels.anl.gov sc11.anl.gov tcs.anl.gov
estrfi.cels.anl.gov papka.alcf.anl.gov sc12.anl.gov http://www.mpich.org

We will restore access to the websites as soon as we can, we’ll send out updates as necessary, and will have more details of the exact nature of the issue as soon as we can.

We regret this interruption in service, and thank you for your patience.

More details to come

Please direct any inquiries to systems@mcs.anl.gov

Max Trefonides for MCS Systems

Written by Craig Stacey

November 21, 2013 at 1:27 pm

Posted in Uncategorized

Phishing mail

A phishing mail is making the rounds purporting to be regarding uchicago webmail. This is a malicious mail and should be disregarded.

If you did follow the link and put any information in, please let me know. Thanks.

Written by Craig Stacey

November 17, 2013 at 1:18 pm

Posted in Uncategorized

Adobe password leak

Adobe announced that the user data of nearly 150 million people was exposed to hackers. I’ve included a link to an article on the break-in from The Guardian. The database of lost information is huge, however, there’s a tool to see if your data has been compromised that’s linked in the Guardian article.

I’m waiting to hear if ANL Cyber is running through the database to find affected users, but regardless, there’s two things you should do.

1) Adobe has been sending out notices to people with Adobe accounts, encouraging them to change their passwords. This is good advice, regardless of whether your data is exposed.

2) Hopefully, you don’t reuse passwords across different sites. But if you do, then you can assume that whoever knows your adobe password now can know your password on any site where you used the same one. As such, aside from making your Adobe password unique, you should also change your password anywhere else that you used that password (and make each of them unique as well).

The Guardian article can be a bit technical, but it’s worth a look, especially for the last few paragraphs where it links to tools you can use for password management. I use 1Password myself, but I know others who use the other ones linked and are happy.



Written by Craig Stacey

November 13, 2013 at 1:04 pm

Posted in Uncategorized