Dispatches From The Geeks

News and Announcements from the MCS Systems Group

Monday’s power work

Monday’s power work will not generally affect servers run by us. Systems Administrators of affected systems will notify their users directly, but the affected systems are Mira, Beagle, TRACC, and some portions of Magellan.

The power outage in December took out much of the data center, but left most of the office side of the building up. This is the opposite of that situation – most of the data center (with the exception of two Power Distribution Units) will stay up, but the office side of things will generally lose power.

Written by Craig Stacey

January 20, 2016 at 1:40 pm

Posted in Uncategorized

B240/TCS to open at 10:30 am, this Monday, January 25th

For those of your in B240 and are supported by us in CELS Systems, a couple of notes:

  1. Your desktop will almost certainly lose power and reboot. Before you leave on Friday, you should cleanly shut down your machine.
  2. Upon your return on Monday, even if you did cleanly shut it down, it’s possible it will have come back on its own. However, it may have come back before things were ready for it to do so. As such, please reboot your computer if anything seems out of the ordinary before opening a trouble ticket with us – I promise it’s the first thing we’re going to do once we get a ticket and it might get you up and running quicker.
  3. After you’ve rebooted your computer, if things are still messed up, please let us know at help@cels.anl.gov (or systems@mcs.anl.gov) and we’ll look into it. Please be patient, we’re short-staffed and will be dealing with a building that just rebooted. :)

Written by Craig Stacey

January 20, 2016 at 11:31 am

Posted in Uncategorized

Update: CIS Maintenance Weekend this weekend.

Here’s the official list of affected services:

http://today.anl.gov/2016/01/it-maintenance-weekend-jan-15-17/

Note that Single Sign On (SSO) which allows external apps to authenticate against Argonne accounts (like Box, Workday, etc.) is being temporarily migrated offsite so that it will remain up during the network outage. And CIS is confident this outage won’t last more than 15 minutes.

Thanks!

Written by Craig Stacey

January 15, 2016 at 2:53 pm

Posted in Uncategorized

CIS Maintenance Weekend this weekend.

This is a reminder of the outage happening this weekend. The takeaway is still the same, anything behind the lab’s firewall will be unavailable for a brief time tomorrow morning, with some periods of up and down time throughout the morning. Things should all be normal by noon.

The official story from CIS is that e-mail and other services are not affected, but that’s not going to be the case. As such, the official announcement from them isn’t accurate and I don’t want to send it on since it implies a false sense of what the situation really is.

So be prepared for brief outages of lab-hosted services tomorrow between 8 and noon.

Thanks. My original notice follows below.

===

CIS Maintenance weekend is the weekend after this coming one. I normally forward these announcements on to you, but I noticed some possible errors in the list of affected services, and I want to get some clarification from them on it. I’ll send a complete list when I have it, however based on the fact that the Argonne network will be undergoing a major upgrade on the morning of Saturday, January 16, we can expect access to services from offsite that are behind the lab’s firewall to be inaccessible.

While this does not include services Systems runs here in building 240, nor does it include ALCF’s machines, it does include services hosted in building 221. This includes lab e-mail, lab web servers (includinghttp://www.mcs.anl.gov), and services that Systems runs in that building. Because that building has the most stable power situation (generator-backed UPS), we’ve housed all our critical servers there including Systems-provided websites and services such as WordPress, Mediawiki, Confluence, Jira, gitlab, etc.

The official outage window is from 8AM through noon, though the work is expected to take no more than an hour if things go as expected; the extra time is for the unexpected.

I’ll send an official list of affected services once it’s nailed down, but I wanted to get this word out quickly in case there are issues we need to be aware of regarding sustaining the outage at that time. If it poses a particular problem for you, let me know.

Written by Craig Stacey

January 15, 2016 at 8:30 am

Posted in Uncategorized

CIS Maintenance Weekend January 15-17

CIS Maintenance weekend is the weekend after this coming one. I normally forward these announcements on to you, but I noticed some possible errors in the list of affected services, and I want to get some clarification from them on it. I’ll send a complete list when I have it, however based on the fact that the Argonne network will be undergoing a major upgrade on the morning of Saturday, January 16, we can expect access to services from offsite that are behind the lab’s firewall to be inaccessible.

While this does not include services Systems runs here in building 240, nor does it include ALCF’s machines, it does include services hosted in building 221. This includes lab e-mail, lab web servers (including http://www.mcs.anl.gov), and services that Systems runs in that building. Because that building has the most stable power situation (generator-backed UPS), we’ve housed all our critical servers there including Systems-provided websites and services such as WordPress, Mediawiki, Confluence, Jira, gitlab, etc.

The official outage window is from 8AM through noon, though the work is expected to take no more than an hour if things go as expected; the extra time is for the unexpected.

I’ll send an official list of affected services once it’s nailed down, but I wanted to get this word out quickly in case there are issues we need to be aware of regarding sustaining the outage at that time. If it poses a particular problem for you, let me know.

Thanks.

Written by Craig Stacey

January 7, 2016 at 4:51 pm

Posted in Uncategorized

Phishing mail

Some of you may be getting very obvious phishing notices saying your anl.gov mail account is over limit. ANL Cyber has been notified, and you can safely ignore and delete the message. Thanks, and happy new year!

Written by Craig Stacey

January 4, 2016 at 9:53 am

Posted in Uncategorized

MCS Website fixed

In working with the ANL web team, we were finally able to locate the issue and correct it yesterday afternoon by around 5:20. A change made globally to http://www.anl.gov (which is where the main MCS website is hosted) to bring mobile support to the website and make some font changes had unexpected behavior on how our website was rendered. We’re still not sure exactly why the issue presented the way it did, but it was some interaction between the main website and our proxy running at http://www.mcs.anl.gov that caused an issue in rendering. The problem was not present on the testing website, which is why they didn’t believe there was an issue pushing out the update.

We’ll be meeting with the team in January to make sure further rollouts don’t hit us this way, as well as go over the plan for updating our proxy configuration.

Written by Craig Stacey

December 16, 2015 at 8:36 am

Posted in Uncategorized

MCS Website not displaying properly

Changes within the ANL webserver content seems to have broken the MCS website (hosted on their server). We’ve alerted CIS to the issue and are awaiting word on what happened and how to bring things back quickly.

Written by Craig Stacey

December 15, 2015 at 10:52 am

Posted in Uncategorized

FYI: Users may get an SSH host key warning when logging into login.mcs.anl.gov. This is expected as the key has been recently updated.

Written by Craig Stacey

December 9, 2015 at 1:15 pm

Posted in Uncategorized

File server outage

This morning’s power work unexpectedly took out one of our file servers (for reasons we have yet to understand). We’re in the process of bringing it back. This affects most home directories for Unix.

Written by Craig Stacey

December 7, 2015 at 9:58 am

Posted in Uncategorized

Follow

Get every new post delivered to your Inbox.

Join 55 other followers