Dispatches From The Geeks

News and Announcements from the MCS Systems Group

Data Center work complete

The previously announced Data Center outage (https://mcssys.wordpress.com/2019/08/13/building-240-data-center-outage-5pm-sep-6-sep-9/) is complete. 

At this point we’ve got the MCS/CELS Linux environment powered back up and operational. The BIO environment was back by 1:30 thanks to early availability of power in that part of the data center.

Systems administrators of other more targeted systems will announce when they are back up on the appropriate mailing list.

The work accomplished this weekend included some necessary work with real beneficial results including bringing the building’s water chillers back online (so we now have 2 sources of chilled water), and the replacement of our failing UPS system with temporary UPSes (which you can see parked in trailers outside the east side of the data center). This will allow us to have reliable battery-backed UPS power to ride the gap from now until our new UPSes come online when the data center expansion comes online.

Thanks for your patience!

Written by Craig Stacey

September 9, 2019 at 5:59 pm

Posted in Uncategorized

COMPLETED: GitLab maintenance window

Maintenance of our GitLab services has been completed. Please let us know if you encounter any issues by emailing help@cels.anl.gov

– CELS Systems

Written by Craig Stacey

August 27, 2019 at 12:20 pm

Posted in Uncategorized

STARTING: GitLab maintenance window

Beginning work to update our GitLab services at
https://gitlab.cels.anl.gov and https://xgitlab.cels.anl.gov

Services will be disrupted during this maintenance window.

Service is expected to be restored by 13:00 CT and may be available sooner than that.

We will send a followup email upon completion of the work.

– CELS Systems

Written by Craig Stacey

August 27, 2019 at 12:00 pm

Posted in Uncategorized

GitLab maintenance window 2019-08-27 from 12:00 to 13:00 CT

A maintenance window of our GitLab services has been planned for 2019-08-27 at 12:00 CT in order to update the software.

GitLab services provided by gitlab.cels.anl.gov and xgitlab.cels.anl.gov will be unavailable during maintenance.

Service is expected to be restored by 13:00 CT, and may be available sooner than that.

If this window poses undue inconvenience please let us know, we can reschedule if needed.

– CELS Systems

Written by Craig Stacey

August 26, 2019 at 9:22 am

Posted in Uncategorized

Building 240 Data Center Outage, 5PM Sep 6 – Sep 9.

The building 240 Data Center cooling outage is set for Saturday, September 7 through Monday, September 9, inclusive. The expectation is that the work will be complete on Sunday, September 8, but we have to allow a day as a contingency. The building’s water-cooling system was taken down just prior to the construction of the building extension, and since then we’ve been running on the lab’s chilled water supply. This was a temporary arrangement, and we now need to revert to due to capacity issues related to other systems coming online. This work requires that the cooling for the data center will be offline during this time while the building management brings the relocated cooling system back online.

This affects the entire MCS Linux environment, including desktop workstations and home directories. It also affects the BIO file services (Y drive and X drive for linux computers). We will be powering these systems down beginning at 5PM on Friday, September 6 in order to be fully shut down for the early morning shut-off on Saturday. We advise you to move what data you need into Box before the work takes place so you can access it during the outage if required.

Services we provide which do not use your MCS home directory will not be generally affected. Some web services may see a brief service outage over the weekend as we take the opportunity to migrate to newer hardware, though we will keep these as low-impact as possible.
It is the hope of CELS Systems that if the work is completed in the expected window, we can start bringing computers and servers back online Monday morning in short order. However, it may not happen until later in the day depending on the state of the cooling system work. For planning purposes, the safe bet is to presume the workstation environment may not be available until Tuesday, September 10, though we will do everything in our power to make sure that’s not the case. We will send an announcement if we will not make the Monday morning window.

Expect other announcements from the operators of affected systems such as LCRC (Bebop/Blues), ALCF, Magellan, and others. We will send a reminder to this list prior to the outage. Thank you.

Written by Craig Stacey

August 13, 2019 at 9:46 am

Posted in Uncategorized

COMPLETED: GitLab maintenance window

Maintenance of our GitLab services has been completed. Please let us know if you encounter any issues by emailing help@cels.anl.gov

– CELS Systems

Written by Craig Stacey

July 30, 2019 at 12:32 pm

Posted in Uncategorized

STARTING: GitLab maintenance window

Beginning work to update our GitLab services at
https://gitlab.cels.anl.gov and https://xgitlab.cels.anl.gov

Services will be disrupted during this maintenance window.

Service is expected to be restored by 13:00 CT and may be available sooner than that.

We will send a followup email upon completion of the work.

– CELS Systems

Written by Craig Stacey

July 30, 2019 at 11:59 am

Posted in Uncategorized

Announcements: LinkedIn Learning, data center outage update, and gitolite (git.mcs.anl.gov) retirement

Hi, folks. I’m going to keep this short and sweet, with the critical bits up front, and explanations to follow for those interested.

1) CELS Employees can now request a LinkedIn Learning account. See https://virtualhelpdesk.cels.anl.gov/docs/cloud-and-external-services-provided-by-cels/linkedin-learning/ for details.

2) The previously announced data center outage now looks to be set for September 7 through September 9. More details to follow.

3) We need to migrate existing projects off “git.mcs.anl.gov” as soon as we can, as the server will lose its external conduits a week from today.

If any of these topics are intriguing to you, read on.

LinkedIn Learning:

Back in May we announced this service would become available to CELS users, and it’s ready to roll. You will need to request a new account to use this, and if you’re on an existing account it will not be renewed. We will be contacting existing users with instructions on how to export your courses and any relevant certificates. The service is not yet integrated with ANL logins, however there’s a strong possibility this will change shortly after the new fiscal year. Stay tuned for more info on that.

For now, see https://virtualhelpdesk.cels.anl.gov/docs/cloud-and-external-services-provided-by-cels/linkedin-learning/ to get started.

Data Center outage:

The current planning date for the cooling outage is Saturday, September 7 through Monday, September 9. Expectation is that the work will be complete on Sunday, September 8, but we have to allow a day as a contingency. The cooling for the data center will be offline during this time while the building management brings the relocated cooling system back online. The building’s water cooling system was taken down just prior to the construction on the building, and since then we’ve been running on the lab’s chilled water supply. This was a temporary arrangement, and we now need to get off it due to capacity issues.

Only computers in the data center in this building will be affected by this. We will post a more exhaustive list as we get closer to the weekend. This will affect unix home file systems in the MCS environment, including workstations.

Gitolite retirement:

We need to retire the git.mcs.anl.gov server and service. It is hosted on an end of life OS and for security reasons we need to remove external access to these hosts by July 22. After that, they will still be online, but only accessible from onsite. We’d like to help you get set up on xgitlab.cels.anl.gov, so if you’d like help in doing it, drop us a line at help@cels.anl.gov and let us know.

Thanks!

Written by Craig Stacey

July 15, 2019 at 4:28 pm

Posted in Uncategorized

COMPLETE: JIRA maintenance window

Maintenance of our JIRA service has been completed. Please let us know if you encounter any issues by emailing help@cels.anl.gov

– CELS Systems

Written by Craig Stacey

July 11, 2019 at 5:41 pm

Posted in Uncategorized

STARTING: JIRA maintenance window

Beginning work on our JIRA service at https://jira.cels.anl.gov

Service will be disrupted during this maintenance window.

Service is expected to be restored by 19:00 CDT and may be available sooner than that.

We will send a followup email upon completion of the work.

– CELS Systems

Written by Craig Stacey

July 11, 2019 at 5:01 pm

Posted in Uncategorized