I was surprised to find out after installing VMWare ESX 3.5 on one of our 4 new Dell R900s that the numbering of my nics was dependent on which one we chose to setup up the Service Console on and that the rest of the nics would get an interrupted numbering depending on where nic0 was located. After a little poking around I figured out you can reset this numbering by editing the following file: > /etc/vmware/esx.conf from the command line. I used vi. I also recommend backing this file up (> cp esx.conf esx.conf.bak) before you try editing it because it is essential to ESX.
if you open this file and scroll about half way down you will see the numbering of you nics. Here's what one of mine looked like. You can see that nic0 ended up somewhere in the middle instead of the first port on my on board card like I wanted.
All you have to do it go through and change the names of the ports to the nic number you want. Save the file and reboot the ESX Server. This is faster than re-installing ESX, especially if you have done any configs on the system.
You can verify your nics by typing: esxcfg-nics –l