One of the benefits of having the clustered filers (FAS3050c) is that I can, in most cases, perform a system upgrade without having to disrupt services running on either system. The process is a little complex but well worth the payoff as in our environment I literally have thousands of students connecting to the storage at a time. Below is a slightly modified version of my notes from the upgrade (use at your own risk). I followed the directions from NetApp's upgrade guide. Although, I will note that their directions were not exact. I had differing outputs from commands at times which made me a little nervous. All in all the upgrade went pretty smooth and the systems have been running solid since.
Download from now.netapp.com under Download Software – DataOnTap – FAS 3050c
- new Shelf Firmware from now.netapp.com (all shelf firmware updates)
- new Disk Firmware from now.netapp.com (all disk firmware updates)
- newest release of Filer Firmware CFE 3.1
- newest GA Release of DataOnTap 7.2.3
- docs for DataOnTap 7.2.3
- Mounted \\filerA\c$
- Mounted \\filerB\c$
- Made of backup of c$\etc\ folder on both systems (minus log files)
- Copy to c$\backup\etc_8-24-2007 - From shelf zip file to the etc\shelf_fw on the both filerA and filerB
- From shelf zip file to the etc\shelf_fw on the both filerA and filerA
- From disk zip file to the etc\disk_fw on the both filerA and filerB
- From disk zip file to the etc\disk_fw on the both filerA and filerA
- Login to the appliance console.
- Check current shelf firmware version ( > sysconfig -v )
- Enter Advanced privileges ( > priv set advanced )
- Start the update ( > storage download shelf )
- This will upgrade the shelf firmware on all the disk shelves in the system. (If you wish to only update the disk shelves attached to a specific adapter, enter storage download shelf adapter_number instead). - Accept the update, Press y for yes and hit enter.
- To verify the new shelf firmware, ( > sysconfig -v )
- Exit Advanced privlieges ( > priv set admin )
Disk firmware is automatically updated on reboot if there are updated files in the disk_fw folder. To keep the system from updating too many disks at once set or verify the following option.
- ( > options raid.background_disk_fw_update.enable)
- if it is set to off, I recommend you change it to on
- Downloaded the newest General Deployment Release, in this case it was Data ONTAP 7.2.3.
- Verified our system met all requirements for running the downloaded release, updates were required for Disk firmware and shelf firmware (which was done above)
- Checked known problems and limitations of the new release to see if any would affect our environment. No potential problems found.
- Compared bug fixes from current version of OnTap 7.0.5 to new version of 7.2.3. There were many bug fixes that could potentially effect our environment which makes the upgrade needed.
- Downloaded newest documentation for 7.2.3
With C$ mapped on both filers I ran the downloaded OS install (self extracting zip files) to the respective \etc directories. This is the first step and copies all the needed files over to the filers. Once completed, we preforme the procedure below from the NOW upgrade guide for Windows Clients.
- start the install on both systems ( > download )
- Checked the cluster status ( > cf status ) to make sure cluster failover was enabled
- Had filerB takeover services for filerA ( > cf takeover )
- This causes filerA to reboot - During reboot of filerA hit ( ctrl-c ) to enter into maintenance mode
- From maintenance mode type ( > halt ) to do a full reboot
- Hit ( del ) during memory test to get to the CFE prompt
- start the firmware update of the filer from the CFE> prompt using ( CFE> update_flash )
- Now reboot, type ( bye ) at console after update was finished to reboot filerA
- filerA is now in a …waiting for giveback state
- Now to give services back to filerA we have to force it using ( > cf giveback –f ) from filerB
- This is required since we are now on different version of DataOnTap between systems in the cluster. - Giveback successful, checked firmeware and os version on filerA using ( > sysconfig –v )
- After checking services on both systems it's time to upgrade filerB
- Have filerA take over the services of filerB ( > cf takeover –n )
- Type ( > halt ) from filerB to reboot it
- During reboot of filerB hit ( ctrl-c ) to enter into maintenance mode
- From maintenance mode type ( > halt ) to do a full reboot
- Hit ( del ) during memory test to get to the CFE prompt
- start the firmware update of the filer from the CFE> prompt using ( CFE> update_flash )
- Typed ( bye ) at console after update was finished to reboot filerB
- filerB is now in a …waiting for giveback state
- Now to give services back to filerB we have to force it using ( > cf giveback –f ) from filerA
- This is required since we are now on different version of DataOnTap between systems in the cluster. - Giveback successful, checked firmeware and os version on filerB using ( > sysconfig –v )
- Both systems should now show the updated firmware and OnTap version 7.2.3
- You should also notice that any out of date disk firmware is automatically updated. In my case I went from NA07 to NA08 on many of the disks.
My final steps were to test system connections
- We use the following NetApp services: CIFS, FTP, HTTP, FCP via VMWARE. All worked fine. I Also checked our student websites and our web based FTP software that connects to the filer.
- Checked Domain connection using cifs testdc ( filerA> cifs testdc )
- appeared fine
Nice post. A few suggestions, from me, someone who has upgraded a few hundred filers in my day.
ReplyDeleteI'd make a note that while FC shelf firmware upgrades will be non-disruptive, a SATA shelf will suspend IO to that shelf for ~30 seconds. Thus downtime for the services running from aggregates hosted on that shelf/loop may be required.
Just copy that 723_setup_e.exe file right to /etc/software, and use the "software install" command to expand it on the Filer. It's much quicker than using CIFS to copy the files as they are expanded.
And of course, always make sure you take a Snapshot of each volume, and make sure you send an Autosupport message before and after.
"options autosupport.doit pre-upgrade"
That'll make sure that NetApp Global Services has something to work with in case something does go wrong.
I'm glad to hear you have no trouble doing these upgrades yourself. If you ever tire of working at a U, let me know. We'll see if we can't find you something to do @ NetApp. ;)
Very good job on this.
ReplyDelete