Please report any oddities you find, but we believe everything is back with no evidence of any data loss. I’ll post a full post mortem tomorrow.
We’re almost back, but have hit a snag with the database. Hopefully, the next update I send will be a restore announcement. Stand by.
As building 240 tenants know, there is major power work scheduled for June 1st that will take out power to the entire building. This, unfortunately, includes the data center in building 240, home to a number of computers. The administrators of specific systems (LCRC, Beagle, Magellan, etc.) will be notifying their users of what this means for them, but this announcement is a broader announcement of the general MCS and CELS computing infratructure and what will be affected.
The short answer is that it’s easier to say what won’t be affected. Mail services we provide (forwarding for mcs.anl.gov, alcf.anl.gov, cels.anl.gov, ci.uchicago.edu, etc.) and their mailing lists will be unaffected. Most web sites we host (WordPress, Confluence, etc.) will remain up. We’ll notify site owners of any exceptions to this. CIS-provided services (e-mail, web, business systems) are generally unaffected. Externally hosted services (Box, Dayforce, TAMS) are unaffected.
Now for the info you really need — what will be down. All MCS/CELS file and compute servers will be down. This includes SSH logins (login.mcs.anl.gov), unix and Mac home file servers, linux compute servers, all desktops, and all networking in building 240.
The outage window for the power work is slated to be 7AM to 7PM on Monday, June 1. As such, we will begin taking systems down prior to that so they shut down cleanly. You can expect MCS/CELS compute systems to be down beginning by 6AM on that day. Once the power comes back on, most of our systems should come back fairly quickly, and we hope to be back to normal well within that outage window, though we are beholden to the power work being completed before we can start bringing things back.
Because your network files will not be available, we encourage you to make sure you have files and data you need locally for that day. Getting accustomed to working in Box can help with that. We’ll be contacting administrative users in the lead-up to this outage to assist in getting you comfortable working with Box and getting your files moved there. You can find more information at http://inside.anl.gov/services/box.
We’ll send more announcements on this as we get closer to the date. You can also keep up to date via twitter (@mcssys) or WordPress (https://mcssys.wordpress.com).
Due to power work being performed in the data center the following computer servers were unexpectedly taken down and will remain down for the duration of the work which is expected to last approximately 3.5 hours:
There was also a brief outage of login1 and login2.mcs.anl.gov due to the work. We apologize for any inconvenience or disruption this may cause.
We have fixed the problem with attachments and images, as well as the firewall issues. Access to https://collab.cels.anl.gov should work as expected. I’ve updated the banner on collab.mcs.anl.gov to point to the new site, and we will later revert the old site to the pointer to the new URL.
If you find any issues with your Confluence spaces, please let us know at firstname.lastname@example.org.
Thanks for your patience.
We discovered attachments and images did not migrate with the site. As such, we’re bringing collab.mcs.anl.gov back for now, and turning off collab.cels.anl.gov. For the time being, please consider collab.mcs.anl.gov read only. If we don’t have a fix in place within the next couple of hours, we’ll fully release collab.mcs.anl.gov back to read/write and take another stab at a full migration at a later date.
An announcement will be made to this list when this is finalized. For minor updates, if you’re not already you can keep an eye on @mcssys on Twitter (https://twitter.com/mcssys) for further developments.
Some issues remain on the upgrade from collab.mcs.anl.gov (Confluence v4) to collab.cels.anl.gov (Confluence v5). Primarily, the firewall exceptions to allow access from offsite are not yet in place. I’ve put in a request to have that done as soon as possible, and with luck should be in place before too long.
Secondly, any custom images you’ve uploaded into your space may not have transferred properly. We still have the old database and server, so please take a look at your spaces in Confluence and let us know if there’s data, images, or info you need recovered.
We’ll send another announcement once the firewall issue is fixed, but in the interim is should be accessible from any internal network, including the VPN.
As part of our plan to merge the contents of Confluence running on xcollab.mcs.anl.gov and collab.mcs.anl.gov we will be moving https://collab.mcs.anl.gov/ to https://collab.cels.anl.gov/ and upgrading the Confluence software from 4.3 to 5.7.1
This hostname migration will occur late Saturday May 2nd (2015-05-02). During that time, it will not be possible to update or add pages to your Spaces on https://collab.mcs.anl.gov/.
Once the migration is complete, the site will be replaced with a link to https://collab.cels.anl.gov/ and will eventually be brought off-line entirely.
Once https://collab.cels.anl.gov/ is up and the migration is completed, users browsing the old address will be shown a notice directing them to the new address.
During the following week we will work individually with owners of Spaces hosted on https://xcollab.mcs.anl.gov/ to migrate them to the new server as well.
This message applies to the people who are connecting to Dayforce via rdp.mcs.anl.gov rather than some other method. If what you’re doing is working for you, you can safely disregard this message.
Recently, a Google Chrome update disabled native support of NPAPI plugins. This affects a number of plugins, including Silverlight, which is required by Dayforce. There is a workaround, which I’ll go into shortly, but the issue remains this is a temporary workaround and we can expect Chrome to remove support (rather than disable it) eventually.
So the real fix is to switch to Firefox. As such, I’ve added a new shortcut to everyone’s desktop on RDP labelled "Dayforce (via Firefox)". If you launch that shortcut, it will eventually get you to Dayforce (it may ask you some first-time Firefox user questions such as importing bookmarks and choosing a default browser). You’ll eventually need to allow the Silverlight plugin, which it will remind you to do. Make sure you "Allow and Remember" to prevent it from asking every time.
If you wish to continue using Chrome, you’ll need to take an extra step after launching the old Chrome shortcut. You’ll see a page like this:
Ignore step 1. Silverlight is installed, it just doesn’t know it. You need to follow the rest of the steps, however, to get Dayforce to start working again. Once following those steps (including relaunching) Dayforce will work in Chrome as it used to.
Again, this only applies to people using rdp.mcs.anl.gov to get to Dayforce. If you’re happily using http://dash.anl.gov (the officially supported method), or using your own browser and silverlight extenstion, you don’t need to change what you’re doing.