NetApp 3040c SAN Cabling Diagram - Typical Fiber Channel

The following is a typical cabling diagram of a NetApp 3040c with 4 fiber channel disk shelves in a single loop configuration. This particular system is also dual connected into 2 fiber channel switches as part of a highly available SAN design. The 3040c Filers in this diagram include (2) additional 4 port fiber channel cards to allow both multiple Fiber Channel networks to connect to the SAN and allow maximum availability of the disks.

NetApp 3040c Direct Attach VMWare ESX

This isn't documented... but you can actually direct attach you VMWare ESX Servers to the NetApp 3050c / 3040c Filer Systems. Here's my setup. (2) Dell 2950 systems with 2 fiber channel ports each. I have the Dell 2950's directly attached into my NetApp FAS 3040c systems with crossover for failover of the NetApp Cluster. See image below. The green and blue lines represent fiber cables from the ESX Servers to the NetApp Filers. The red and dark blue lines are the fiber cables from the NetApp heads to the DS14Mk4 / DS14Mk2 disk shelves.

What you do is setup the Fiber Channel to essentially contain a virtual switch. Both adapters 0c and 0d show 3 adapters, 1 online, 1 standby and 1 partner. This will allow the failover in VMWare when you use the cluster failover for maintenance and such.

Slot: 0c
Description: Fibre Channel Target Adapter 0c
Adapter Name: 0c_0
Adapter Type: Local
Status: ONLINE

Adapter Name: 0c_1
Adapter Type: Standby
Status: OFFLINE

Adapter Name: 0c_2
Adapter Type: Partner
Status: ONLINE

Slot: 0d
Description: Fibre Channel Target Adapter 0d
Adapter Name: 0d_0
Adapter Type: Local
Status: ONLINE

Adapter Name: 0d_1
Adapter Type: Standby
Status: OFFLINE

Adapter Name: 0d_2
Adapter Type: Partner
Status: ONLINE

Although this setup will only work for 2 ESX Servers, it can be very useful when you have a limited budget (no need for fiber switches) but need the bandwidth from Fiber Channel. If you are on a serious budget you might want to consider VMWare over NFS with the NetApp.

VMWare over NFS vs FiberChannel (FC)

VMWare over NFS has been quite the buzz over the past few months, especially when it comes to NetApp filers like the 3070c. I personally run VMWare over FC connected into a NetApp 3050c on hi-speed fiber channel drives. What I like about it is it's fast and it works well. It's really not that hard to setup and a denial of service attack on my IP network won't hurt my VM's access to the SAN.

With Fiber channel you have to create LUNs for the VMWare box to access. There are 2 schools of though here... either create 1 big LUN and run all your VM's from it - or - create a new LUN for each VM.

With NFS, things change. Apparently you can take advantage of de-duplication, no single disk I/O, and you can use VMDK thin provisioning. What I'd like to see is a good white paper on NFS vs Fiber Channel including setup instructions and why you would choose one over the other.