NetApp 3040a Clustered Link Aggregation - vif

I've got (2) NetApp 3040a clustered systems both running LACP Aggregated vifs (nics) for my NFS VMWare Connections. One cluster is running on Cisco Catlyist 3750's and the other is running on a Cisco Catlyist 4507. Both switches are setup redundantly. The fail over / load balance is excellent. Here's how I set it up:

My switches are set to IP Load Balance (global switch setting)

Commands I used to setup the nics on the NetApp. This puts onboard nic c and d and add on card port c and d in an aggregated LACP vif called SANAprivate. I use this for private NFS traffic for my VMWare ESX Hosts. The next command sets the IP Address info and adds the partner vif for cluster failovers / non-disruptive SAN upgrades.
> vif create lacp SANAprivate -b ip e0c e0d e4c e4d

> ifconfig SANAprivate 192.168.217.11 up netmask 255.255.255.0 broadcast 192.168.217.255 -wins mediatype auto trusted partner SANBprivate

> vif status SANAprivate
default: transmit 'IP Load balancing', VIF Type 'multi_mode', fail 'log'
private: 4 links, transmit 'IP Load balancing', VIF Type 'lacp' fail 'default'
VIF Status Up Addr_set
up:
e4d: state up, since 30Jan2009 07:47:56 (7+08:17:02)
mediatype: auto-1000t-fd-up
flags: enabled
active aggr, aggr port: e0d
input packets 8106183, input bytes 9157734620
input lacp packets 22869, output lacp packets 21163
output packets 502026, output bytes 229370476
up indications 2, broken indications 0
drops (if) 0, drops (link) 0
indication: up at 30Jan2009 07:47:56
consecutive 0, transitions 2
e4c: state up, since 30Jan2009 07:47:54 (7+08:17:04)
mediatype: auto-1000t-fd-up
flags: enabled
active aggr, aggr port: e0d
input packets 912352, input bytes 82064164
input lacp packets 22874, output lacp packets 21162
output packets 4173173, output bytes 1334844804
up indications 2, broken indications 0
drops (if) 0, drops (link) 0
indication: up at 30Jan2009 07:47:54
consecutive 0, transitions 2
e0c: state up, since 30Jan2009 07:47:53 (7+08:17:05)
mediatype: auto-1000t-fd-up
flags: enabled
active aggr, aggr port: e0d
input packets 2356250, input bytes 569112124
input lacp packets 22857, output lacp packets 21160
output packets 873913, output bytes 121767134
up indications 2, broken indications 0
drops (if) 0, drops (link) 0
indication: up at 30Jan2009 07:47:53
consecutive 0, transitions 2
e0d: state up, since 30Jan2009 07:47:53 (7+08:17:05)
mediatype: auto-1000t-fd-up
flags: enabled
active aggr, aggr port: e0d
input packets 3886952, input bytes 2231755682
input lacp packets 22877, output lacp packets 21160
output packets 1772975, output bytes 1653703494
up indications 2, broken indications 0
drops (if) 0, drops (link) 0
indication: up at 30Jan2009 07:47:53
consecutive 0, transitions 2

Cisco Switch Config
We tested this by pulling Cables from each of the 4 nics up to 3 at a time, so each nic would be by itself and with other nics while pulling data from the link aggregation. We setup multiple connections so we were pulling more than 1 nics worth of bandwidth. I have had very good results with this configuration and have not seen any issues with teaming the onboard nics and the addon nics.

interface Port-channel10
description NetApp Filer Public Links
switchport
switchport access vlan 463
switchport mode access
!
interface GigabitEthernet1/1
description stfSan-e0a
switchport access vlan 463
switchport mode access
channel-group 10 mode active
!
interface GigabitEthernet1/2
description stfSan-e4a
switchport access vlan 463
switchport mode access
channel-group 10 mode active
!
interface GigabitEthernet2/1
description stfSan-e0b
switchport access vlan 463
switchport mode access
channel-group 10 mode active
!
interface GigabitEthernet2/2
description stfSan-e4b
switchport access vlan 463
switchport mode access
channel-group 10 mode active
!

Exchange 2007 - Blackberry Enterprise Server (BES) Setup

Here are a couple things I had to do to get the Blackberry Enterprise Server (BES) running with Exchange 2007. This stuff wasn't clear in the install guide. Especially number 1 below.

1. Give the BESAdmin account permission on my exchange databases. I had to do it on all of our databases. Here's the command for Database07

add-adpermission -user BESAdmin –identity “Database07” -accessrights GenericRead, GenericWrite -extendedrights Send-As, Receive-As, ms-Exch-Store-Admin

2. Give the BESAdmin account extended rights

Add-ADPermission -Identity "BESAdmin" -User "BESAdmin" -AccessRights GenericRead,GenericWrite,ExtendedRight -extendedrights Send-As,Receive-As,Receive-As,ms-Exch-Store-Admin

3. Add the BESAdmin to the Exchange View Only Administrators Group in Active Directory.

Powershell: Exchange 2007 - BES - Blackberry Enterprise Server

Here a quick little snippet from a script I run when setting up users for our BES (Blackberry Enterprise Server) environment with Exchange 2007. Essentially, the script is just giving the BESAdmin account Send-As permission on the AD Account. You could do this on your entire User OU in the domain, but for security purposes we've decided to only set the permission for the Blackberry users.

# Open the File of User Names and Put it in the Pipeline
$import = Import-Csv "NewBlackBerryAccounts.txt"
$domain = "mydomain.com"

# Loop Through the CSV File, creating accounts
$import | Foreach {
   # Set Vars
   $StrUserName = $_.Username

   $user = get-qaduser $StrUserName@$domain
   if($user) {
      $dn = $user.DN
      Add-ADPermission -Identity $dn -User 'mydomain\BESAdmin' -ExtendedRights 'Send-as'
   } else {
      write-host Username $_.Username not found
   }
}

Powershell Progress Bar with Time Countdown

I needed to add a 15 minute pause in a script that we were using to create exchange mailboxes and this little Powershell progress bar with a countdown timer worked really nice.

###===========================
### Pause Program for 15 min
### - Matt Brown, 2008
###===========================
$x = 15*60
$length = $x / 100
while($x -gt 0) {
  $min = [int](([string]($x/60)).split('.')[0])
  $text = " " + $min + " minutes " + ($x % 60) + " seconds left"
  Write-Progress "Pausing Script" -status $text -perc ($x/$length)
  start-sleep -s 1
  $x--
}

Powershell: Monitor IIS Application Pool or Site

We have an exchange IIS Application Pool stopping every so often because of some Entourage client problems. It would cause the Application Pool to stop and therefore break owa access, which was a problem.

So while we are working with Microsoft on a permanent solution I quickly put together a powershell script to run every 30 seconds and check the state of the application pool. If the MSExchangeOWAAppPool is stopped, then the script starts it. It uses the IIS Provider Tools snapin for powershell.

### -----------------------------------------------
### Written by Matt Brown - 12:46 PM 11/3/2008
### Powershell script to check MSExchangeOWAAppPool
### Requires IIS Administration Provider Tools
### -----------------------------------------------

### Make sure Snapin is loaded
$add = 1
get-PSSnapin * | foreach {
 if($_.Name -match 'IIsProviderSnapIn') {
  $add = 0
 }
}
if($add) {
 add-PSSnapin IIsProviderSnapIn
}

#######################################
######## Check AppPool State ##########
#######################################
Write-Host "`n"
Write-Host "#####################################"
Write-Host "Running check on MSExchangeOWAAppPool"
Write-Host "#####################################"
while($true) {

 $state = Get-WebItemState IIS:\AppPools\MSExchangeOWAAppPool
 if($state -eq "Stopped") {
  Start-WebItem IIS:\AppPools\MSExchangeOWAAppPool
  $now = Get-Date –f "yyyy-MM-dd HH:mm:ss"
  $MsgBody = "CAS01 AppPool needed a restart " + $now
  Write-Host $MsgBody
 }
 Start-Sleep -s 30
}

Powershell: New Active Directory Objects Report

Here's a quick Powershell script to send you a report of the most recent additions to Active Directory.

### --------------------------------------------
### Written by Matt Brown - 12:13 PM 10/22/2008
###
### AD Report on new objects created in the
### last 24 hours
### Requires Quest Powershell extenstions for AD
### --------------------------------------------

#######################################
####### Load Required Snapin's ########
#######################################
## Add Quest AD Snapin Tool
$addAD = 1
get-PSSnapin * | foreach {
  if($_.Name -match 'quest.activeroles.admanagement') {
    $addAD = 0
  }
}
if($addAD) { add-PSSnapin quest.activeroles.admanagement }

#######################################
########### Setup Log File ############
#######################################
$Today=get-date
$filename="NewADObjects_"+($Today.year).ToString()+"_"
$filename+=($Today.month).ToString()+"_"+($Today.day).ToString()+".txt"

#######################################
### Get AD Formated Date 24 hrs ago ###
#######################################
$currentDate = [System.DateTime]::Now
$currentDateUtc = $currentDate.ToUniversalTime()
$creationDate = $currentDateUtc.AddHours(- 24)
$YYYY = $creationDate.Year.ToString()
$MM = $creationDate.Month.ToString();
if ($MM.Length -eq 1) {$MM="0" + $MM};
$DD = $creationDate.Day.ToString();
if ($DD.Length -eq 1) {$DD="0" + $DD};
$creationDateStr = $YYYY + $MM + $DD + '000000.0Z'

$MsgBody = "###################`n"
$MsgBody += " New AD Objects`n"
$MsgBody += "###################`n`n"

write-host $creationDateStr
$newobjects = Get-QADObject -ldapfilter "(whenCreated>=$creationDateStr)" 
                -SizeLimit 30000 | sort type

if($newobjects) {
  $newobjects | out-file $filename
  $type = ""
  $newobjects | foreach {
    if($_.Type -eq $type) {
      $MsgBody += " " + $_.Name + "`n"
    } else {
      $type = $_.Type
      $MsgBody += "`nNew " + $_.Type + "(s)`n"
      $MsgBody += "===================================`n"
      $MsgBody += " " + $_.Name + "`n"
    }
  }
}

#######################################
############ Email Report #############
#######################################

function SendEmail($body) {
  $message = New-Object System.Net.Mail.MailMessage
  $message.From = "myemail@domain.com"
  $message.To.Add("myeamail@domain.com")
  $message.Subject = "Active Directory - new object report"
  $message.Body = $body

  $smtp = New-Object System.net.Mail.SmtpClient
  $smtp.Host = "smtp.mailserver.com"
  $smtp.UseDefaultCredentials = $true
  $smtp.Send($message)
}

SendEmail($MsgBody)

Update GAL Display Name - powershell

We recently decided to change on how our Global Address list is displayed from using the format to the , format.

Powershell made quick work of this task and took about 10 minutes with 2500 users. Here's the script.

###=====================================
### Update Exchange Global Address List Display
### - Matt Brown, 2008
###=====================================
$Users = Get-User -ResultSize unlimited |
 where {
   ($_.RecipientTypeDetails -eq "MailUser")
   -or ($_.RecipientTypeDetails -eq "UserMailbox")
 }

ForEach ($Person in $Users) {
   $NewName = $User.LastName + ", "
   $NewName += $User.FirstName + " "
   $NewName += $user.Initials
  
   # get rid of trailing spaces caused by blank initials
   $NewName = $NewName.Trim()
   Set-User $User -Name $NewName -DisplayName $NewName
   $NewName = $Null
}

Don't forget to update the OAB after this is done so your outlook clients will update.

Active Directory - Removing SID History

I use a couple of great tools from joeware.net to remove a SID from a users SID History. I had a problem where the wrong user was mapped over during a migration when we were colasping multiple domains into one.

The 2 tools I used from joeware were adfind and admod, both free.

adfind -h IT-DC01 -default -f sAMAccountName=jackuser sidhistory

dn:CN=Jack User,OU=Employees,OU=People,DC=mydomain,DC=edu
>sIDHistory: S-1-5-23-4189335451-1674751469-1023141700-3124
>sIDHistory: S-1-5-23-4217985222-169311000002009-1212232504-146495

This listed the current SID's in the history of the users account. After deciding which one I wanted to removie I used admod to remove it.

admod -b "CN=Jack User,OU=Employees,OU=People,DC=mydomain,DC=edu"
sidhistory:-:S-1-5-23-4217985222-1000002009-1212232504-146495

Sid Removed and now where ready to take that SID and add it to the correct user account.

admod -b "CN=Jackie User,OU=Employees,OU=People,DC=mydomain,DC=edu"
sidhistory:+:S-1-5-23-4217985222-1000002009-1212232504-146495

Note: I found out after this post that this option does not work with SIDHistory. You will need to use the VB Script or ADMT to migrate the sid from the source domain.

VMWare over NFS on a NetApp - ASIS (deduplication) WOW

I have a NetApp 3040c cluster that I'm using for NFS, iSCSI and FC connectivity to my VMWare ESX Servers. NFS has proven to be fast and reliable. I'm running the following system on NFS without any problems:

• Active Directory Domain Controller (Server 2008) - (16,000 users)
  
• Exchange 2007 (CAS) Client Access Server on Server 2008
• Exchange 2007 HUB on Server 2008
• ILM / MIIS on Server 2003
• IIS on Server 2003 with over 800 websites
• IIS on Server 2008
• Full Exchange 2007 Test enviorment (3 servers Mailbox, HUB, CAS and 1 DC)
• Blackberry Access Server
• Wireless Raidus Server

These Virtual Machines take up 462 GB of disk space... but the really cool part is that because I'm running these all on my NetApps over NFS with DeDuplication turned on I'm seeing a 77% storage savings... so they are only taking up 105 GB of disk space on the storage system.

sanb> df -s -g /vol/esxNFS
Filesystem used saved %saved
/vol/esxNFS/ 105GB 357GB 77%

I'll I can say is wow.

ADMT Migrating Computers from 2003 to 2008

In order to migrate computers from a 2000 or 2003 Active Directory Domain to a 2008 Active Directory Domain you need to set a group policy in your 2008 domain that allows cryptography algorithms compatible with Windows NT 4.0. Otherwise, the migration will bring the computer account over but will not change the domain the computer is in.

You can find this Group Policy setting here:
In the Group Policy Management Editor console, expand Computer Configuration, expand Policies, expand Administrative Templates, expand System, click Net Logon, and then double-click Allow cryptography algorithms compatible with Windows NT 4.0. Based on (http://support.microsoft.com/kb/942564)

Another thing I found was that that Target Domain Account you are using to transfer the computers over need to have local admin privileges on the systems you are migrating from the source domain and the ADMT System needs to have access to the computer. (IE: Firewall opened for at least that machine).

MySQL - find bad email addresses in database

I needed a quick way to find bad email addresses in my mysql database that was not being verified when the users entered them into the form. That will be the next fix.

A quick select using:

SELECT name,email FROM `D_User` WHERE `email` NOT LIKE '%@%.%'

... and I had all the bad email addresses.

VMWare ESX VM Network Aggregation / EtherChannel / LACP

I recently went through a series of tests with our VMWare ESX 3.5 environment to test link aggregation and failover. We tested a variety of Link aggregation methods including LACP, PgaP, and standard etherchannel while experimenting with different settings on the vSwitch.

Here's what the lab consisted of: a Dell R900 with 8 physical nics (3 used for the test), a Cisco 4507 Switch with 2 Gig high speed blades, 2 laptops and 3 Virtual Machines. This would also work on 2 cisco 3750's connected with a stackwise cable. We setup a virtual switch on the ESX Server that was just for Virtual Machine networks with 3 pnics. Our ports are set for trunk mode so we can have multiple vlan's on our VM's. We setup a ping from each of the Virtual Machines to one of the laptops with a command like: "ping 10.0.0.100 -t -w 500" and setup the same from the 2 laptops back to the virtual machines. This way we could see how many packets we lost on each setting change when unplugging a cable from the switch or taking a blade offline.

After quite a bit of testing we found Standard Etherchannel to work the best. With a standard etherchannel setup we would loose between 1 and 3 packets (at the faster retry time) if a network cable or switch blade was brought offline. In my opinion, this was an acceptable behavior, although I would like to see an LACP aggregation running.

Here's what our final configuration looked like:

VMWare ESX vSwitch Configuration: (under vSwitch Properties->Nic Teaming)
Load Balancing: Route based on IP Hash
Network Failover Detection: Link Status Only
(this could be beacon probing depending on your network)
Notify Switches: Yes
Failback: Yes




Switch Config (Cisco 4507 or 3750)
# Set Switch load balance to IP
port-channel load-balance src-dst-ip

# Add port 1/2 to group
interface GigabitEthernet1/2
switchport mode trunk
channel-group 1 mode on

# Add port 2/1 to group
interface GigabitEthernet2/1
switchport mode trunk
channel-group 1 mode on

# Add port 2/2 to group
interface GigabitEthernet2/2
switchport mode trunk
channel-group 1 mode on

# Setup Port Channel Group
interface Port-channel1
switchport switchport mode trunk
spanning-tree portfast trunk