NetApp 3040a Clustered Link Aggregation - vif

I've got (2) NetApp 3040a clustered systems both running LACP Aggregated vifs (nics) for my NFS VMWare Connections. One cluster is running on Cisco Catlyist 3750's and the other is running on a Cisco Catlyist 4507. Both switches are setup redundantly. The fail over / load balance is excellent. Here's how I set it up:

My switches are set to IP Load Balance (global switch setting)

Commands I used to setup the nics on the NetApp. This puts onboard nic c and d and add on card port c and d in an aggregated LACP vif called SANAprivate. I use this for private NFS traffic for my VMWare ESX Hosts. The next command sets the IP Address info and adds the partner vif for cluster failovers / non-disruptive SAN upgrades.
> vif create lacp SANAprivate -b ip e0c e0d e4c e4d

> ifconfig SANAprivate 192.168.217.11 up netmask 255.255.255.0 broadcast 192.168.217.255 -wins mediatype auto trusted partner SANBprivate

> vif status SANAprivate
default: transmit 'IP Load balancing', VIF Type 'multi_mode', fail 'log'
private: 4 links, transmit 'IP Load balancing', VIF Type 'lacp' fail 'default'
VIF Status Up Addr_set
up:
e4d: state up, since 30Jan2009 07:47:56 (7+08:17:02)
mediatype: auto-1000t-fd-up
flags: enabled
active aggr, aggr port: e0d
input packets 8106183, input bytes 9157734620
input lacp packets 22869, output lacp packets 21163
output packets 502026, output bytes 229370476
up indications 2, broken indications 0
drops (if) 0, drops (link) 0
indication: up at 30Jan2009 07:47:56
consecutive 0, transitions 2
e4c: state up, since 30Jan2009 07:47:54 (7+08:17:04)
mediatype: auto-1000t-fd-up
flags: enabled
active aggr, aggr port: e0d
input packets 912352, input bytes 82064164
input lacp packets 22874, output lacp packets 21162
output packets 4173173, output bytes 1334844804
up indications 2, broken indications 0
drops (if) 0, drops (link) 0
indication: up at 30Jan2009 07:47:54
consecutive 0, transitions 2
e0c: state up, since 30Jan2009 07:47:53 (7+08:17:05)
mediatype: auto-1000t-fd-up
flags: enabled
active aggr, aggr port: e0d
input packets 2356250, input bytes 569112124
input lacp packets 22857, output lacp packets 21160
output packets 873913, output bytes 121767134
up indications 2, broken indications 0
drops (if) 0, drops (link) 0
indication: up at 30Jan2009 07:47:53
consecutive 0, transitions 2
e0d: state up, since 30Jan2009 07:47:53 (7+08:17:05)
mediatype: auto-1000t-fd-up
flags: enabled
active aggr, aggr port: e0d
input packets 3886952, input bytes 2231755682
input lacp packets 22877, output lacp packets 21160
output packets 1772975, output bytes 1653703494
up indications 2, broken indications 0
drops (if) 0, drops (link) 0
indication: up at 30Jan2009 07:47:53
consecutive 0, transitions 2

Cisco Switch Config
We tested this by pulling Cables from each of the 4 nics up to 3 at a time, so each nic would be by itself and with other nics while pulling data from the link aggregation. We setup multiple connections so we were pulling more than 1 nics worth of bandwidth. I have had very good results with this configuration and have not seen any issues with teaming the onboard nics and the addon nics.

interface Port-channel10
description NetApp Filer Public Links
switchport
switchport access vlan 463
switchport mode access
!
interface GigabitEthernet1/1
description stfSan-e0a
switchport access vlan 463
switchport mode access
channel-group 10 mode active
!
interface GigabitEthernet1/2
description stfSan-e4a
switchport access vlan 463
switchport mode access
channel-group 10 mode active
!
interface GigabitEthernet2/1
description stfSan-e0b
switchport access vlan 463
switchport mode access
channel-group 10 mode active
!
interface GigabitEthernet2/2
description stfSan-e4b
switchport access vlan 463
switchport mode access
channel-group 10 mode active
!

VMWare over NFS on a NetApp - ASIS (deduplication) WOW

I have a NetApp 3040c cluster that I'm using for NFS, iSCSI and FC connectivity to my VMWare ESX Servers. NFS has proven to be fast and reliable. I'm running the following system on NFS without any problems:

• Active Directory Domain Controller (Server 2008) - (16,000 users)
  
• Exchange 2007 (CAS) Client Access Server on Server 2008
• Exchange 2007 HUB on Server 2008
• ILM / MIIS on Server 2003
• IIS on Server 2003 with over 800 websites
• IIS on Server 2008
• Full Exchange 2007 Test enviorment (3 servers Mailbox, HUB, CAS and 1 DC)
• Blackberry Access Server
• Wireless Raidus Server

These Virtual Machines take up 462 GB of disk space... but the really cool part is that because I'm running these all on my NetApps over NFS with DeDuplication turned on I'm seeing a 77% storage savings... so they are only taking up 105 GB of disk space on the storage system.

sanb> df -s -g /vol/esxNFS
Filesystem used saved %saved
/vol/esxNFS/ 105GB 357GB 77%

I'll I can say is wow.

NetApp 3040c SAN Cabling Diagram - Typical Fiber Channel

The following is a typical cabling diagram of a NetApp 3040c with 4 fiber channel disk shelves in a single loop configuration. This particular system is also dual connected into 2 fiber channel switches as part of a highly available SAN design. The 3040c Filers in this diagram include (2) additional 4 port fiber channel cards to allow both multiple Fiber Channel networks to connect to the SAN and allow maximum availability of the disks.

NetApp 3040c Direct Attach VMWare ESX

This isn't documented... but you can actually direct attach you VMWare ESX Servers to the NetApp 3050c / 3040c Filer Systems. Here's my setup. (2) Dell 2950 systems with 2 fiber channel ports each. I have the Dell 2950's directly attached into my NetApp FAS 3040c systems with crossover for failover of the NetApp Cluster. See image below. The green and blue lines represent fiber cables from the ESX Servers to the NetApp Filers. The red and dark blue lines are the fiber cables from the NetApp heads to the DS14Mk4 / DS14Mk2 disk shelves.

What you do is setup the Fiber Channel to essentially contain a virtual switch. Both adapters 0c and 0d show 3 adapters, 1 online, 1 standby and 1 partner. This will allow the failover in VMWare when you use the cluster failover for maintenance and such.

Slot: 0c
Description: Fibre Channel Target Adapter 0c
Adapter Name: 0c_0
Adapter Type: Local
Status: ONLINE

Adapter Name: 0c_1
Adapter Type: Standby
Status: OFFLINE

Adapter Name: 0c_2
Adapter Type: Partner
Status: ONLINE

Slot: 0d
Description: Fibre Channel Target Adapter 0d
Adapter Name: 0d_0
Adapter Type: Local
Status: ONLINE

Adapter Name: 0d_1
Adapter Type: Standby
Status: OFFLINE

Adapter Name: 0d_2
Adapter Type: Partner
Status: ONLINE

Although this setup will only work for 2 ESX Servers, it can be very useful when you have a limited budget (no need for fiber switches) but need the bandwidth from Fiber Channel. If you are on a serious budget you might want to consider VMWare over NFS with the NetApp.

Blackboard on VMWare ESX with Network Appliance

I've been working with our E-Learning Team on designing a new Platform for Blackboard (Blackboard is a learning / course management tool). Our design has concluded with running Blackboard Enterprise on (4) Dell R900 Quad Proc / Quad Core systems with 32gb RAM via VMWare ESX 3.5 connected to a Network Appliance 3040c SAN. We will be running (28) 300 GB fiber channel drives over 4GB fiber for the back end drives. This should provide plenty of disk IO. The High IO apps like our SQL Database VMs will run over Fiber Channel and the smaller systems like QuestionMark will run via VMWare over NFS.

This design, while very new, will provide a very reliable, highly available learning management system. All systems, both Hardware and Apps are configured in some kind of cluster to minimize all single points of failure.

Performance really should be no issue, as the horsepower we can give the Virtual Machines is higher than we were able to dedicate on the old standalone physical systems.

Filer Panic - NetApp 3050c - DataOnTap 7.2.3

I was surprised to find an email this morning from my Filers and NetApp support telling me one of my filers had a little panic over the night.

RPANIC:Saving 674M to /etc/crash/core.101185944.2008-03-05.06_16_18.nz ("Protection Fault accessing address 0x0000008c from EIP 0x1426f54 in process FTPPool03 on release NetApp Release 7.2.3") via sparecore

According to the logs and the auto support message it appears that an FTP Error occurred from a poorly written FTP Client that caused the filer to PANIC. This is disturbing. It's saying I can write crappy FTP Software and cause the NetApp Filers to panic. I run my filers in a cluster so during the panic the filer that didn't panic took over the one that did and life went on normally. Now I need to wait until after hours to perform the giveback because already this morning there are over 100 connections via CIFS to the filer and bringing it out of cluster takeover mode requires the cifs service be shutdown and restarted on the partner system.

The Command is easy though: > cf giveback

Panic Message:Protection Fault accessing address 0x0000008c from EIP 0x1426f54 in process FTPPool03 on release NetApp Release 7.2.3

Bug: 264711

Title: disconnecting FTP session during the "LIST" command may panic filer

Description: During a session from an external FTP client to the FTP service running on the storage appliance itself (a service of the Data ONTAP kernel), if the FTP control connection unexpectedly disconnects at the same time the appliance is processing an Passive FTP "LIST" command (or equivalent operation), the appliance may suffer an interruption of service.


Workaround: Correctly written FTP clients on a healthy network are less likely to provoke an abrupt disconnection.

NetApp 3050c upgrade of DataOnTap 7.0.5 to DataOnTap 7.2.3

Performing a non-disruptive upgrade of our Network Appliance FAS 3050c (clustered filer configuration)

One of the benefits of having the clustered filers (FAS3050c) is that I can, in most cases, perform a system upgrade without having to disrupt services running on either system. The process is a little complex but well worth the payoff as in our environment I literally have thousands of students connecting to the storage at a time. Below is a slightly modified version of my notes from the upgrade (use at your own risk). I followed the directions from NetApp's upgrade guide. Although, I will note that their directions were not exact. I had differing outputs from commands at times which made me a little nervous. All in all the upgrade went pretty smooth and the systems have been running solid since.

Download from now.netapp.com under Download Software – DataOnTap – FAS 3050c

• new Shelf Firmware from now.netapp.com (all shelf firmware updates)
• new Disk Firmware from now.netapp.com (all disk firmware updates)
• newest release of Filer Firmware CFE 3.1
• newest GA Release of DataOnTap 7.2.3
• docs for DataOnTap 7.2.3

Copied and Made backups of files

• Mounted \\filerA\c$
• Mounted \\filerB\c$
• Made of backup of c$\etc\ folder on both systems (minus log files)
  - Copy to c$\backup\etc_8-24-2007
• From shelf zip file to the etc\shelf_fw on the both filerA and filerB
• From shelf zip file to the etc\shelf_fw on the both filerA and filerA
• From disk zip file to the etc\disk_fw on the both filerA and filerB
• From disk zip file to the etc\disk_fw on the both filerA and filerA

Shelf Firmware

• Login to the appliance console.
• Check current shelf firmware version ( > sysconfig -v )
  
• Enter Advanced privileges ( > priv set advanced )
  
• Start the update ( > storage download shelf )
  - This will upgrade the shelf firmware on all the disk shelves in the system. (If you wish to only update the disk shelves attached to a specific adapter, enter storage download shelf adapter_number instead).
• Accept the update, Press y for yes and hit enter.
• To verify the new shelf firmware, ( > sysconfig -v )
  
• Exit Advanced privlieges ( > priv set admin )

Disk Firmware

Disk firmware is automatically updated on reboot if there are updated files in the disk_fw folder. To keep the system from updating too many disks at once set or verify the following option.

• ( > options raid.background_disk_fw_update.enable)
  - if it is set to off, I recommend you change it to on
  

DataOnTap Update

1. Downloaded the newest General Deployment Release, in this case it was Data ONTAP 7.2.3.
2. Verified our system met all requirements for running the downloaded release, updates were required for Disk firmware and shelf firmware (which was done above)
3. Checked known problems and limitations of the new release to see if any would affect our environment. No potential problems found.
4. Compared bug fixes from current version of OnTap 7.0.5 to new version of 7.2.3. There were many bug fixes that could potentially effect our environment which makes the upgrade needed.
5. Downloaded newest documentation for 7.2.3

Update Procedure

With C$ mapped on both filers I ran the downloaded OS install (self extracting zip files) to the respective \etc directories. This is the first step and copies all the needed files over to the filers. Once completed, we preforme the procedure below from the NOW upgrade guide for Windows Clients.

1. start the install on both systems ( > download )
   
2. Checked the cluster status ( > cf status ) to make sure cluster failover was enabled
3. Had filerB takeover services for filerA ( > cf takeover )
   - This causes filerA to reboot
   
4. During reboot of filerA hit ( ctrl-c ) to enter into maintenance mode
5. From maintenance mode type ( > halt ) to do a full reboot
6. Hit ( del ) during memory test to get to the CFE prompt
7. start the firmware update of the filer from the CFE> prompt using ( CFE> update_flash )
   
8. Now reboot, type ( bye ) at console after update was finished to reboot filerA
9. filerA is now in a …waiting for giveback state
10. Now to give services back to filerA we have to force it using ( > cf giveback –f ) from filerB
    - This is required since we are now on different version of DataOnTap between systems in the cluster.
    
11. Giveback successful, checked firmeware and os version on filerA using ( > sysconfig –v )
12. After checking services on both systems it's time to upgrade filerB
    
13. Have filerA take over the services of filerB ( > cf takeover –n )
    
14. Type ( > halt ) from filerB to reboot it
    
15. During reboot of filerB hit ( ctrl-c ) to enter into maintenance mode
16. From maintenance mode type ( > halt ) to do a full reboot
17. Hit ( del ) during memory test to get to the CFE prompt
18. start the firmware update of the filer from the CFE> prompt using ( CFE> update_flash )
    
19. Typed ( bye ) at console after update was finished to reboot filerB
20. filerB is now in a …waiting for giveback state
21. Now to give services back to filerB we have to force it using ( > cf giveback –f ) from filerA
    - This is required since we are now on different version of DataOnTap between systems in the cluster.
22. Giveback successful, checked firmeware and os version on filerB using ( > sysconfig –v )
1. Both systems should now show the updated firmware and OnTap version 7.2.3


23. You should also notice that any out of date disk firmware is automatically updated. In my case I went from NA07 to NA08 on many of the disks.

My final steps were to test system connections

1. We use the following NetApp services: CIFS, FTP, HTTP, FCP via VMWARE. All worked fine. I Also checked our student websites and our web based FTP software that connects to the filer.
2. Checked Domain connection using cifs testdc ( filerA> cifs testdc )
   - appeared fine

Filer Panic - NetApp FAS 3050c cluster mode

Filer Panic - NetApp FAS 3050c cluster mode

I recently encountered a panic of one of my filers. I run (2) FAS 3050c NetApp filers in a clustered configuration. Here's what happened. One of the guys in the server room (to remain nameless) was messing around behind the server rack and somehow broke one of my fiber cables connecting filerB to it's disk shelves (the shelves I run as my san that VMWare ESX is connected to). No biggie, filerA quickly noticed the outage and picked up services for filerB. I got notice from autosupport about the outage and was quickly in the server room to check it out. After going through filerB back and forth and checking connections I attempted to run a ( cf giveback ) from filerA to give filerB it's services back. filerB didn't like it and quickly threw the services back to filerA. This got me worried.

So i called NetApp support, who by the way already had a case opened up because of the failover that had happened. After a couple hours on with support we decided it was either a faulty ESH2 module, a faulty cable, or something else (like a bad disk not reporting itself as bad in the loop). So NetApp sent out some parts, 3 new drives, a new cable and 2 ESH2 modules. (Now I'm glad I have the hardware warrenty).

I get the parts the next day and get everything replaced. Again, everything looks to be normal but when checking the disk / shelf status with a ( > fcadmin device_map ) from within maintenance mode we noticed the filer was not recognizing the shelf's in it's loop but services were still running fine from filerA. (see below) At this point we decided we need to wait until I can take the entire system (both head units in the cluster) down so we can do an invasive test to see what piece of hardware is having the problem. Also, running an ( > aggr status -r ) showed me the filer thought all the disks in the shelf had failed.

> *> fcadmin device_map
Loop Map for channel 0a:
Translated Map: Port Count 24
7 29 28 27 25 26 23 22 21 20 16 19 18 17 24 39 38 37 36 32 35 34 33 40
Shelf mapping:

Shelf Unknown: 16 17 18 19 20 21 22 23 24 25 26 27 28 29 32 33 34 35 36 37 38 39 40

Loop Map for channel 0b:
Translated Map: Port Count 17
7 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
Shelf mapping:
Shelf Unknown: 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

Chuck at NetApp support recommends I first try just powering off the systems, then powering off the shelves and letting them sit for a couple minutes. (I first had to shut down all my VM's running the from the SAN and shutdown all services running on both systems... with thousands of users this was a pain). He then has me power on the shelves, let them fully come up and then power on filerB by itself into maintenance mode. We again run the ( > fcadmin device_map ) (see below) and now the filer is seeing it's shelves. Apparently there is a bug with the shelf firmware version I am on (just one version back) that causes certain panics to stay in memory. Hence, our problem.

> *> fcadmin device_map
Loop Map for channel 0a:
Loop Map for channel 0a:
7 29 28 27 25 26 23 22 21 20 16 19 18 17 24 39 38 37 36 32 35 34 33 40
Shelf mapping:
Shelf 1: 29 28 27 26 25 24 23 22 21 20 19 18 17 16

Shelf 2: XXX XXX XXX XXX XXX 40 39 38 37 36 35 34 33 32

Loop Map for channel 0b:
Translated Map: Port Count 17
7 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
Shelf mapping:
Shelf 1: 29 28 27 26 25 24 23 22 21 20 19 18 17 16

Target SES devices on this loop:
Shelf 1: 14 15

I now shut filerB back down, bring up filerA (which is still running service for both systems) and then bring filerB up. filerB comes up in a ...waiting for giveback state. I issue a ( > cf giveback ) from filerA and filerB takes back it's services. Were back up and running.

NetApp recommended I upgrade the firmware as soon as possible and noted that the power cycle of the shelf is a temporary fix to the memory issue.