Drobo Elite Equals Epic FAIL

SMB Storage Device Not Ready For VMware

Drobo recently released the latest version of the Drobo Elite. They should have kept it in the oven a little longer. We decided to use this SMB storage device because it is VMware Certified, low-cost and has built in 2-disk redundancy.

This Drobo has 8 bays and can handle any size, speed and type of disk. We filled it with 2TB 10K SATA drives from Western Digital.

The original design consisted of two Dell R510 servers running Vsphere 4.1 with fail-over mode enabled. All VMs will reside on the Drobo.

The first sign of trouble appeared during a tech support all to Drobo. I needed some help configuring the iSCSI connections so I could add the datastores to the VMware hosts. The tech I spoke to didn’t know how yet as the model was brand new. Uh-oh.

I figured it out on my own. The Drobo Dashboard software needs to be installed on a physical machine on the same network as the iSCSI connections. From there, you can create partitions and set the RAID level. I chose the 2-disk redundancy level. I would regret that decision later. I created several 2TB partitions

Once the partitions were made, I was able to add the datastores to both Vsphere hosts and all looked good. I copied the ISO files up and installed Small Business Server 2011 without a hitch.

The virtual SBS Server had a dedicated GB NIC on its own vSwitch. The vNIC was the Intel 1000 emulator. During troubleshooting I did try the VMXNET3 vNIC and it made no real difference in throughput. The physical NIC was connected to a Juniper EX2200 switch with the MTU size set to 1500 on the switch and vNIC.

So, we get onsite and install the infrastructure. All is fine until we actually start copying files from physical machines on the network to the SBS server. The throughput was very poor. Just using RDP to get into the server was agonizingly slow. I got that rock in my stomach feeling that we may be in trouble.

Symptoms included:

– 16kbps to 1 MB/ sec throughput on the vNIC.
– Latency in the 800 ms range for Reads and Writes to the datastore.
– Timeouts were likely but unconfirmed. We had some file copy resets and errors.

Troubleshooting:

1. Tried Jumbo Frames on the vNIC and Juniper switch. It helped a little at first, but the latency in reads/writes continued
2. Added the VMXNET3 adapter and ran the Fix My Network wizard in SBS to establish the new vNIC.
3. Swapped the ethernet cables with new CAT6
4. Moved the iSCSI connections off the switch. Configured crossover cables and connected directly to the vHost

All of those made no difference. We figured at that point that it was probably the overhead of the 2-disk redundancy of the Drobo. It’s a software based RAID and most likely resource intensive.

Now we were neck deep in trouble. The client had been in the office the day after the installation. Mail Services, Internet Access, BES and File Access were all very slow and had disconnects.

Finally, we said hell with it. We had to move the virtual SBS off of the Drobo. The problem was the vHosts weren’t designed with enough disk space to house the SBS as is. I used the VMware Converter to run a V2V conversion. In the process I shrunk the disks to fit in the vHost.

A 120 GB disk took 8 hours to get from the Drobo to the vHost over the cross connected iSCSI connections.

Once on the vHost, the network throughput increased to 20MBps. Latency dropped to 16 ms from 800 ms. No network drops and all services responded quickly.

We plan on dismissing the Drobo- Office Space Style. I’ll be the one with the bat.

The Drobo is next.

Advertisements


Categories: IT Pros

Tags: , , , ,

10 replies

  1. I have work with a couple of customers that have the same basic setup as you were describing.
    Can’t say I’m 100% sold on the solution. I can’t say I’ve had the same experience as you have.
    I was curious, based on how I’m reading the description I’m confused on how you had the vSwitchs setup.
    One for regular LAN traffic attached to the VM guest?
    One dedicated to vSphere datastore for the iSCSI traffic? This vSwitch is not accessible to any of the guests, only the host.
    Is this how it was configured?

    • Hi Kurt.
      We have 6 physical NICs in each vSphere Host. Here’s how I split them up:
      vSwitch0 – Management Network (vmnic 0,2)
      vSwitch1 – VM Traffic (vmnic1). SBS2011 is the only server on this switch
      vSwitch2 – iSCSI-vMotion (vmnic5). This vSwitch was on the ISCSI VLAN on the Juniper Switch. Different subnet.
      vSwitch3 – VM Traffic 2 (vmnic3). vCenter is the only server on this switch

      The Drobo 800i has 2 iSCSI NICs. So, I had one connected to each host. I’m in conversations with Drobo support. If I get anything interesting, I’ll post it.

  2. Have you tried changing the switch?

  3. I have the same setup using a b800i with 3 HP DL360 G7 servers. They are connected to a set of redundant Gig-E switches, as is the Drobo. We’re currently running vSphere 4.1 Update 1 and plan to go to 5.0 shortly. We’re also getting exactly the same performance issues. It’s absolutely hideous.

    So far, Drobo support for this problem has been very much less than stellar (to put things nicely). I’m about to ask for a refund and look for an alternate iSCSI platform that can actually handle VMware.

    I’m going to be at VMworld in a couple of weeks and plan to stop by the Drobo booth to express my displeasure – politely and professionally. I’m really disappointed with this unit so far…!

  4. Currently, I am in Drobo hell right now too. I bought my Elite back in January, so I don’t know if I can return mine. But I’ve been through every optimization, setting, hard drive configuration (dual disk fail protect vs. 1 disk), direct connected iSCSI vs. hardware switch connected. And nothing makes this unit even remotely usable with VMWare.

    I’ve called for help and they say they can’t help with vmware configuration or testing. You have to have an NTFS LUN which a Windows PC directly connects to.

    I really wanted to like the Drobo – I really did, but it’s been a nightmare to set up, test and ultimately discover it won’t work in a small business environment.

  5. Can anybody provide Hardware specifications for your NIC’s? Ie make and model. I am assuming most of you are using Dell servers, and these come with Broadcom NIC’s. There are several different configuration changes you need to make with these NIC’s in order for the Drobo and Dell server to play nice with each other

    • In my case, we’re using HP Proliant DL360 G7 servers. Dell isn’t in the picture at all. Since making the change, we are still occassionally getting the same Disk Read / Write Latency errors. If you have additional tuning parameters for these to clean this up, I’m certainly willing to listen.

      I have a set of Dell 1950 servers that I am setting up in my basement for a home lab. Drobo, as a result of my experiences here, isn’t even on my list as a storage solution. I’m looking at NetGear and QNap platforms instead.

  6. You truly help it become search simple with your presentation however obtain this condition to generally be truly a very important factor i consider I would hardly ever have an understanding of. It seems like very sophisticated and intensely huge for me. I’m just impatient within your up coming distribute, Let me try and obtain the cling than it!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Brad Hedlund

stuff and nonsense

MyVirtuaLife.Net

Every cloud has a silver lining.

Live Virtually

A Storage and Virtualization Blog

Virtualization Team

VMware ESX/ESXi - ESX server - Virtualization - vCloud Director, tutorials, how-to, video

www.hypervizor.com/

Just another WordPress.com site

VirtualKenneth's Blog - hqVirtual | hire quality

Virtualization Blog focused on VMware Environments

Virtu-Al.Net

Virtually everything is POSHable

Gabes Virtual World

Your P.I. on virtualization

Yellow Bricks

by Duncan Epping

Wahl Network

Technical Solutions for Technical People

Joking's Blog

Phoenix-based IT guy with too much to say...

%d bloggers like this: