Last night’s data transfer went quite well. The last bits are being put in place now. Most users will have their mail accessible early Sunday afternoon, and all users by end of day.[Edit: Let's say most users by mid-afternoon.]
It’s been a long day, and I’m quite punch drunk, so this notice will be brief and scant on details. More info coming tomorrow.The data transfer is in its final throes. At some point on Sunday afternoon, your mail will be accessible again. Mail is being delivered into your mailbox, you just can’t read it yet.For the 100 users who already had Zimbra accounts prior to this, your mail will take a little longer, since we have to sync the two accounts together into one. At various times tomorrow your existing Zimbra account may be inaccessible while this synchronization is happening.By the time tomorrow is through, this process will be complete and we’ll be back in full operation.We apologize for the long delay on this. We didn’t expect it to take this long, due to the various unexpected bottlenecks. But the good news is that it is proceeding smoothly and generally as planned.Enjoy your vacation from e-mail — it’ll all be there for you to read when we’re back.
For those who are checking in on the blog:Most things are back. Mail and Zimbra accounts are still offline. We’ll send an announcement (and post here) when they’re ready. ETA is still up in the air, as the data transfer is (obviously) taking longer than we’d expected.Stay tuned.
I’m going to keep this brief and to the point, since we’re all going a bit nuts getting things in place for tomorrow.MCS is going offline tomorrow from 6:00 AM until 6:30 PM due to a building-wide power outage. Any and all malfunctions and outages are expected and known. We’ll be doing some work on some systems that may have UPS power. But if you get access to anything, this should not be construed as an all clear and it may suddenly power down without warning. LCRC, TeraGrid and ALCF are handling announcements for their gear (though ALCF’s only outage is the BG/L rack, I believe).In short: No computing tomorrow. No e-mail, no web (except a placeholder), nothing. The whole building is losing power. No lights. No air conditioning. Nothing. Stay away. Computer no workee. In fact, before you leave tonight, make sure you’ve saved everything and logged out.If you send us mail saying something isn’t working, and it’s between 6 AM and 6:30 PM on Saturday, you will be mocked. Mercilessly. Sorry, but we need to entertain ourselves somehow during this. :)Please read the last 5 or 6 posts on http://www.mcs.anl.gov/systems/blog if you’ve no idea as to the scope of what’s going on. But, when we’re done, we’ll have successfully completely hosed up your mail and printing. The good news is that we’ll also send directions on how you can unhose it.Specifically, after things are back up, the all clear will be sent, and a whole fresh set of documents will be published showing how to connect to your new mail account, how to add your new printers, and how to take advantage of all the blinky lights and gadgetry the deployment of the new services provided.So, enjoy yet another computing-free Saturday! In the immortal words of former Vice-President Al Gore… Peace out, y’all!
Another semi-regular update on the mail server upgrade and migration. This is happening on Saturday, April 26th. Please see http://www.mcs.anl.gov/systems/blog?cat=1 for past updates and details. Throughout the week I’ll be sending notes on changes you can expect to see or things you need to do to be ready. Friday, I posted about making sure you have an Argonne Domain account — you’ll definitely need that to be able to read any new mail.Today, I’m covering a couple of thoughts that occurred over the weekend, brought on by some questions some of you have asked. So let’s get to those:I don’t use my e-mail account, do I need it?Nope, you don’t. You will need an e-mail account only if you want to use any of the features of the Zimbra service. The features you’re most likely to want are: having a mailbox, having a calendar on the server, or accessing the shared calendars. If you don’t want any of them, you don’t need an account on the server. We do need a way to reach you via e-mail, however, so please be sure your personal accounts page at accounts.mcs.anl.gov is up to date.Okay, I don’t need the e-mail account. Do I still need a domain account?As covered in Friday’s update, if you’re onsite more frequently than a few days a year, you’ve almost assuredly got one. But if you don’t actually have one, do you need one right now? No. Will you need one eventually? Yes. Right now we’re using our own back-end for identification and authentication for a number of resources. This leads to the mishmash of usernames and passwords you need to keep straight. Eventually, we’ll be down to a single identity, and that’s going to be the Argonne Domain account. Our new account management system is going to be using that as its back-end, so at some point in the coming year we’ll be migrating people over to that. I see this e-mail switch as being the first big push in getting most of our non-Argonne users set up, since it’s now become a lot simpler to get this account set up.Is any mail not being copied to the new server?Yes. First up, if you use POP for e-mail, then it’s likely your mail isn’t even on our server, since POP by default downloads the mail to the client machine and deletes it from the server. Some POP clients can be configured to leave the mail on the server for a period of time, or indefinitely. If you are using POP, which we don’t recommend, and you leave copies on the server, you will likely be downloading a duplicate copy of that mail when the switch is complete.But, aside from any mail that only lives on your machines and not on our server, there’s other mail we’re not copying over. Specifically, we’re not copying over your Trash and Viral mailboxes. The former because those mailboxes can get rather large and presumably the user has decided it is trash, and the latter because it’s mail we’ve designated as highly likely to be malicious. Quarantine is being copied — that’s where .zip files end up for a number of users. The SPAM folder is also being copied, since we don’t want to make the judgement call that you’ve decided it’s actually SPAM.Both of the folders we’re not copying will continue to be available on the old mail server as long as it remains in operation — probably another month.Now, as promised in the last update, an overview of forwarding mail, setting server-side filters, and setting Auto-reply or Out of Office message.I’m kind of cheating by choosing those as topics, because they’re all dead simple. In all cases, they’re done through the web interface, which has thorough documentation. And because the web interface is available from anywhere (no VPN or SSH keys required), you’ll be able to set any of these from any web browser.In fact, all of the preferences for how your account operates are handled from this web interface. There’s also a new feature not available on our current system, New Mail Notification. You can choose to have a message get sent to a provided e-mail address notifying you of new mail. This is different from simply forwarding the mail, since the content of the mail is kep local to the server. If your MCS mailbox is low traffic and you prefer not to regularly check it, this might be a handy option for you.We’re making good progress on the migration, and still expect to hit our target on Saturday. If all goes as planned, when you check your mail after Saturday evening, you’ll see all your old mail on the old server followed by instructions on how to read your mail on the new server.
As announced on Wednesday, we’re going to be completing the migration to the new mail server on Saturday, April 26th during the building-wide power outage. It’s my intention over the next week to send out the occasional update on important issues you need to be aware of in an effort to make this transition go as smoothly as possible. There’s a lot of good information in here for everyone, so if you could read the whole message, there’ll be fewer surprises ahead.We’ve been going over the pieces involved with this transition for some time, and have covered a lot of the issues. But, I won’t lie to you, there are things we’re discovering or just now considering as we get further along. I say this a lot… it’s not the problems you can think of that get you, it’s the ones you don’t think of.The big thing we need to tackle before the 26th is mapping your MCS identity to your Argonne identity. We’re in the process right now of populating our user database (affectionately known as userbase) with the Argonne domain usernames of all the employees in the divisions we support (CELS, MCS, and ALCF). But we’ve got 500 e-mail accounts, so that’s not going to cover all of them. Some of you, for example, don’t even have an Argonne domain account yet. That’s what this mail is going to address.If you’re an employee in MCS, ALCF, or CELS, we’ve got you covered. If you’re not, we’re going to need a little help from you to get things lined up for a smooth transition on the 26th.Consider these criteria:
- Do you have an Argonne domain account?
- Do you know what the username for that account is?
- Do you know the password?
If you think the answer to all of the above questions are “Yes”, there’s a quick way to test it. Argonne has a credentials site that lets you find out the status of your account. It’s a common phishing tactic to get you to click a link in e-mail that results in a login page, so I’m not going to make a clickable hyperlink here. You should open your browser and visit the credentials.anl.gov site. If you’re not onsite, you’ll need to put https:// in front of that URL.The left side of that page has a Username/Password test. Try it out. If it works, great!Do you have an Argonne domain account?If you work onsite at the Laboratory, you almost assuredly do, since it’s required for the various training being on-site requires.But, if you don’t, and you want or have an MCS e-mail account, you need to get one, and you can do that via the Cyber Gatepass system.Begin by visiting webapps.anl.gov/collaborator (putting the https:// in there if you’re offsite), and follow the instructions on that page to begin the process. I apologize for the duplication of effort in terms of collecting this information — we’re in the process of making this new database our central store for identity such that this shouldn’t be an issue anymore.You’ll get a username that starts with “ac.” followed by something related to your name. The “ac” stands for Argonne Collaborator, in case you were wondering. After your account is approved, you’ll be given instructions on how to change your temporary password into something meaningful.Important note for Sponsors: You’re going to get an e-mail from the Cyber GatePass system requesting you visit the webapps.inside.anl.gov/cgp site to approve requests. It’s a quick process, but I wanted to give you a heads up on this since you’ve not likely seen these messages before.If you don’t know your domain account username or password, contact the Help Desk (email@example.com) for help. By this point, you have an Argonne domain account, and you know the username and password. You’re going to want to remember that password, it’s going to be your e-mail password after the 26th.Now, so we can get your account set up, we need you to do one of the following:Login to your MCS account page at accounts.mcs.anl.gov/account.php — this page uses your MCS Workstation username and password, which is also your current e-mail username and password. Add your Argonne domain username to the page (under Account Information). Make sure everything else there is up to date as well.-or-Send us an e-mail at firstname.lastname@example.org with your MCS username, and what your Argonne domain username is, and we’ll get your record up to date. Remember, if you’re an employee in CELS, ALCF, or MCS we have this information and will be taking care of it for you.That covers the big issue for today. The documentation on using the new system is coming along well, and will be unveiled a day or two before the switchover takes place. As we write this documentation, we catch other issues on which it would be nice to give some advance notice. My next update is going to cover some of the changes in how you do things like forward your mail, use server-side filtering, and set an auto-reply or Out of Office message.
Hi, folks. There’s a lot to cover here, so let’s just get down to it.Topics:
- Building 221 Power Outage, April 26 2008
- Mail server update
- Printing update
- Web update
Power Outage: April 26 2008Yes, you read that correctly. We’re going to have a full building power outage on April 26, 2008. We’ll get through this together, but first, let’s visit a happy place so we’re in the right frame of mind.You’re sitting in a reclining chair on a beach. Waves are lapping in. Your umbrella is just nicely shading the sun, while a cool sea breeze gently swirls around. Off in the distance, you faintly hear a seagull. Your refreshing beverage of choice is in your hand.Ahh, okay. Now, we proceed.See, the outage on the 5th took care of the emergency and fixed the bad panels. Originally, they wanted to take the whole building down and do the required maintenance on the whole electrial system for the building. However, available labor and uncertainty over what might come back up shoved that aside, and only the essential work of getting our 4 power panels back up was done.Now the maintenance on the other panels downstairs needs to happen. And to do that, the whole building has to be without power. So everything’s coming down again. On Saturday, April 26th, 2008.You know, I need to go back to the happy place for a second.Mmmm… Guinness. (Hey, if Bailey’s has ads with people drinking Bailey’s on a beach, I can drink Guinness on a beach!)Okay, back again. So, here are the details:
- Date: Saturday, April 26th, 2008
- Time: 06:00 – 18:30
- What will be down: Almost everything.
It’s easier if I say what’s going to stay up. Network, ALCF CryptoCARDs, database servers, and name service. Normally, the web server would stay up, but we’re taking the opportunity to do some upgrades to the machine. So it will be down for a portion of the time. We’ll set up a dummy web server that shows we’re down for maintenance and will answer on all the addresses that the current server uses. Right now, that list is:
As soon as the web server is upgraded, it’ll come back online. I don’t expect it will be down for the full outage window, but things don’t always go as expected so… hang on, going to the happy place again.Hey, look, a hermit crab! Neat!Okay, as I was saying, don’t be terribly surprised if the web server isn’t back online until the outage window is over. You’ll be able to send mail, but you won’t be able to receive mail. More on that below.This time, and I’ve confirmed this with FMS, the entire building will be without power. This means no overhead lighting, no computers, no air conditioning. In other words, stay away. Also, since everything’s going down, it would be a good idea to log out and power down your desktop computers when you leave on Friday (the 25th, not this coming Friday).We’ll send out a reminder about this as it approaches.Mail Server UpgradeWe are truly in the home stretch here. So much so, in fact, that we’re targeting the April 26th outage as the cutover to the new server. But before we go into details, some more good news.Those of you who attended or watched my talk in January got a preview of the new mail service. I also stated in that talk it would cost the division $8.00 per year per user for this. A bargain, in my not so humble opinion. Well, I’ve received word that the pricing has changed. It’s now going to cost us, rounded to the nearest penny, zero dollars and zero cents.Yup, the lab’s decided that the Zimbra service is going to be part of their core services. Hooray! I don’t need to go to the happy place yet!We’ve got another change from my talk in January. Specifically, the password scheme.Originally, our plan was to have the e-mail service password be independent of any other passwords. After discussions and looking over past correspondence, it’s become clear that what most of you, our beloved users, want is fewer passwords. To that end, we’re going to have the Zimbra server use the Argonne domain password — the same one you use for logging in to inside.anl.gov and other Argonne services. Ultimately, everything that we use a password for in the general MCS/ALCF/CELS core IT infrastructure will use that scheme. There are some significant advantages to be had with this move, including the speed with which we can roll out some of the new projects we’ve been working on, like the new account management system, trouble ticket system, and, of course, the new mail server.I don’t want to bore everyone with the details here, so if you have questions or concerns I’m more than happy to answer them. I will say we’re also working on alternate paths to allowing us to go to a single password, some of which involve using your MCS credentials (whatever they may turn out to be) to access lab services, and offering One Time Password (OTP) generators to people who’d like that level of security.Okay, back to the actual details of the mail server move. I’m in the process of writing the documentation, but here’s what you can expect:
- April 24th or 25th: Announcement of the new documentation, for those who want to get a jump on the game.
- April 26th: Mail is down during power outage — this also applies to users who are currently reading their mail on Zimbra right now, like me. When power comes back up, you’ll find your MCS mailbox is empty, except for a single message. That message will contain the details on what you need to do to talk to the new mail server.
At that point, everything will be moved over to the new Zimbra server. All existing mail and calendars you have will be on a single account. Your e-mail address will not change (unless you want it to). If you’re in MCS, your primary address will be email@example.com, ALCF will have firstname.lastname@example.org, and CELS will have email@example.com — but you can also have a name in either or both of the domains you’re not in primarily if you’d like.Consider this a teaser, the full documentation will have all the details.We’re very excited about this. We hope everyone sees why we’re so gung-ho on this move — a number of us are already living on the new server and are quite pleased.Printing updateWe’re also going to make our new print server production on the 26th. This new print server will be used for all architectures, including Linux, Mac OS X, and Windows. At that time, we’ll also be taking the old printers offline and retiring them, because the Canon copiers will be up and online as our printing/faxing/scanning solution.Here’s what’s going to change:
- lw3, lw4, ljcolor, lw9, lw16 are all pulled from service and retired.
- The copier/scanners in D255 (B & W) and A224 (B & W and color) become the primary printers upstairs in 221.
- The copier/scanner in D168 (color) moves to where lw16 is currently located (alcove at C130) and becomes primary printer for downstairs in 221.
- The copier/scanner in 360-l158 comes online as a printer/scanner.
- All other existing printers stay as they are. As those printers die, whether to replace them or shift the workload to one of the copiers is a decision made by consensus among the user, supervisor and Systems.
The docs on using the new print server and the copiers as printer, scanner, and fax are coming soon. Using these for printing is especially advantageous, since our lease on them includes toner — our only extra cost is the paper.Web updateIn the last Dispatch, I alluded to a seminar highlighting the use of the CELS Website Content Management tool. Well, it’s happening.
- Location: 221-A216
- Time: Wednesday, April 23, 2008, 2:00 PM
If you’ve not yet played with the tool, or even if you have, I strongly encourage you to attend. The seminar will be driven by Beth Cerny-Patiño, who built the forms and the websites that use them. We’ll also be highlighting what a research group site or project site can look like when driven by the data entered into these forms. It’s possible to have a group site that looks great, is up to date with the latest information, and without requiring the person entering that information to know any HTML or use any web authoring software.Part of what’s driving this is that we need to get the old web servers turned off. So we need to get sites up on the new server, and the more sites that are driven dynamically from this data, the easier it is for everyone involved. Right now, all the content on the main MCS site is driven from this data. Likewise, the CI website, CELS website, and parts of the ALCF website are all also driven by this same data. And, you’ll notice when you visit these sites, data applicable to multiple sites will show up on those sites.So come, check it out, ask questions!Well, that’s a lot to digest, so let’s just end this Dispatch here. I’m still vaguely on target of having these updates come out monthly, so expect another one in May.And now it’s off to my happy place. To quote the great Joel Hodgson, sometimes I go off into my own little world. But it’s okay… they know me there.
Today I bring you tales of…Intrigue!Romance!Espionage!Barbecue!Asparagus farming!Pants mending!Well, no, not really. I just thought after yesterday’s play writing adventure I’d deliver today’s update in the form of a 50′s movie trailer.You’ll be on the edge of your seat as electricians deftly work with power distribution panels and enormous breakers! You’ll marvel as a rag-tag team of sysadmins disassemble, reassemble, recable, and relocate racks of computers! You’ll stare in dull silence as those same sysadmins munch donuts, sip coffee and type on their laptops!No one will be admitted during the last 15 minutes! Because the electronic door strikes won’t have any power!Don’t give away the secret ending to your friends! Because I’m going to do it right now!Power Panels 1, 2, 3, and 4 are back in operation.And there was much rejoicing.The downside is that due to a shortage of available labor, and the general uncertainty as to what actually is the overarching problem actually is, the entire maintenance was not performed. However, when the rest will be performed is also an open question, since they want to have a better handle on what went wrong before they mess with the other power.So, for the interim, at least, we’ve got us a machine room that’s back to full capacity, and a bunch of sysadmins who are happy to be out of crisis mode. At least, for this particular crisis.Jazz is in the process of coming back, expect a notice from John on that on the once-again-functional LCRC mailing list. Cosmea will come back online tomorrow, full-disclosure and breadboard users should expect a notice from Rick or Narayan.This marks the last of the daily updates I’ll be sending out through e-mail. If you’d like to keep tabs on what the aforementioned rag-tag team is up to, I’ll be posting somewhat regularly at http://www.mcs.anl.gov/systems/blog.Once again, I want to thank everyone for their patience and understanding. We’ve all been getting the good vibes everyone’s been sending, and we appreciate it.Adieu!
Emergency maintenance needs to be conducted on the power distribution panels on the service floor of Building 221 tomorrow. The outage window is 6:00 AM until 6:30 PM.During this time, building 221 will have no power, including all offices and desktop computers, as well as the machine rooms.All machines in the MCS machine rooms will be down, with some exceptions. The machines that will be down (aside from any systems already down) include all clusters, BG/L, TeraGrid, and full-disclosure (the SiCortex).Network connectivity will stay up, as will machines in our Tier One service rack, including the file servers and web server. Mail service will be down, though all mail will still be delivered, just delayed.We apologize for the abrupt notice on this — it’s related to the power situation we’re facing in the machine room, but has a broader scope than that. If this work is not done, any panel could go down and stay down until this work is done. Specifically, the mechanisms that house the large main breakers on the distribution panels are not working, which means no tripped breaker can be re-seated.Thanks for your understanding.
Ladies and Gentlemen,I present a one act play. I’d tell you the title, but it’s not suitable for a family-friendly mailing list.[Scene: My office]Look! There’s a light at the end of the tunnel. The wiring is done, all the panels are ready to come up!Oh, what’s that you say? That light at the end of the tunnel is a train? Oh. Of course.Those big 300 pound 600 amp breaker boxes aren’t going back into the panels? Huh. And things had been going so smoothly up to this point.You mean we have to do that electrical maintenance that was postponed a few weeks back to fix this? And it involves taking power down for the whole building? Of course.Oh, and all the numbers on my lottery ticket hit, but a week after the drawing I paid for? Wow, of course!And… scene!Okay, so I’m not a playwright. But I do deliver a stirring message, and that message is that everyone should go out and enjoy one of the few fine weather weekends we’ve got so far this year, because there’s no computing going on here. An officially worded notice will follow this mailing, just to make sure everyone gets that this is really happening.So tomorrow, when the sun is shining and you’re catching spring fever, think of us who will be in the dark and quiet machine room, taking the downtime as an opportunity to run cables, move machines, move racks, and undo all the power hacks we had in place. In the dark. Going uphill both ways.Sorry for the short notice on this. The alternative is to be without power for 2 more weeks, and that’s after a week of downtime already.With any luck, the next installment of this saga will simply be a short message of success. May the fates smile on us more than they have been so far.