Avaya IP Office and the Rogue DHCP Server

Leave it to a phone system to cry havoc on a Small Business Server 2011 network. In this case it was an older version (5) of Avaya IP office. To make matters worse, the VOIP capability isn’t being used. It’s a straight digital phone system.

We receive a morning call from a client that some Windows 7 systems can’t get online. Not all, some. You have to confirm that kind of thing. A couple phone calls to the offline users revealed that they couldn’t see any network shares or get to the internet. This was the first time this had happened at this site. The SBS environment was about a month old and performing quite well. I never like the first time problems…something changed, now to figure out what.

Luckily, it didn’t take long. After remotely connecting to the server (and exhaling the sigh of relief that I could), the SBS Server Manager showed me the Big Red X. The DHCP Server had stopped. No problem. Click Start, the service state changed to Started (Ah, OK. That was easy), service changed to Stopped (Mothe******.). This is a SBS self-protection feature. It will stop its own DHCP Service if it sees another one on the network.

So I hop in the truck and head out there. Once on-site, I set up the offline users with static IP addresses to get them online and turn the heat down. While doing this, I notice that none of them have IP addresses. All of them had the standard (169.x.x.x) IP address that windows machines give themselves if they can’t find a DHCP server. This worried me a little because I had expected them to be getting an address from somewhere. If there is a rogue DHCP server on the network, it sucked because it won’t even give out addresses. What a tease, and not the good kind. I’m in a race against time now as the other users will drop off the network as their DHCP leases expire.

I catch a break by finding out the Avaya phone vendor had made a visit on-site the week before. A-ha. I call Matt, our phone guy. He says by default the Avaya IP Office software DHCP service is on, and if a change had been made recently that could be our culprit. I login to the Avaya IP Office and notice that DHCP is turned on, but for a different subnet. I ensure DHCP is disabled for both subnets, alert the client that the phones will going down and reboot the phone system.

After the Avaya system reboots, the SBS is still angry. The DHCP service still won’t start. Rebooting the SBS doesn’t change anything. Checking the premises for wireless routers, WAPs or anything else that could be a DHCP server came up empty. It had to be the Avaya, and we had to do something soon, before the leases expired. It wasn’t just the leases though; SBS is a control freak. It has to know and be everything on the network. If the DHCP service didn’t start, other errors would appear soon.

Matt looked into the Avaya protocols, while I looked to try and trick SBS into thinking it was the only DHCP server. Turns out, there’s no trick; just a registry entry to tell SBS to ignore anything it detects on the DHCP Ports (UDP 67,68).

To disable rogue DHCP detection on the SBS

To disable rogue DHCP detection on the SBS by creating following registry entry:

1. Open Registry Editor.

2. Navigate to

3. Create a new DWORD with the following registry value:
Value name:   DisableRogueDetection
Data type:      REG_DWORD
Radix:            Binary
Value data:    (Hexadecimal) 1, which will be saved as 0x00000001

4. Restart the server.

That worked. DHCP started, and we were in good shape. Matt and I wanted answers, though, as we can’t have Avaya systems hijacking our SBS environments. Avaya has the DHCP and TFTP services bundled into one on/off switch. The DHCP service runs on the standard UDP ports 67,68, while TFTP runs on UDP 69. The problem here is that with DHCP disabled, the BOOTP protocol is still active on the DHCP reserved ports. Disabling BOOTP will clear the UDP ports of traffic that trigger the DHCP Shutdown on the SBS.

If the Avaya BOOTP is disabled, as it is now in our environment, all phones needing configuration information will need the IP address of the DHCP and TFTP servers entered manually.

Whew. Just another day at the office.


Categories: IT Pros

Tags: , , , , , ,

3 replies

  1. Just an FYI this happens because the IPO Admin Manager application runs its own BOOTP/TFTP Server by default, so even with DHCP turned off on the IPO, the Manager application will trigger the rogue detection. You can turn it off in Manager though, which turns off the baked-in TFTP Server as well. This is not only for older versions, this is happening on current builds as well (7.x.xx)

  2. Also, it doesn’t seem as though the application log shows that dhcp stops because of the rogue detection. Looks like it just crashes. You will see the following in the application event log. Also, a reboot doesn’t seem to be required.

    Faulting application name: svchost.exe_DHCPServer, version: 6.1.7600.16385, time stamp: 0x4a5bc3c1
    Faulting module name: dhcpssvc.dll, version: 6.1.7600.16385, time stamp: 0x4a5bdece
    Exception code: 0xc0000005
    Fault offset: 0x000000000004da70
    Faulting process id: 0xb24
    Faulting application start time: 0x01cca551af265012
    Faulting application path: C:\Windows\system32\svchost.exe
    Faulting module path: c:\windows\system32\dhcpssvc.dll
    Report Id: f2fbe982-1144-11e1-88ed-78e3b51a8fb0

  3. Also if you happen to be using Avaya’s Voicemail PRO server software, be aware that the Microsoft Windows firewall must be TURNED OFF for it to function correctly. Best way to do this is to create an appropriately named Group Policy that has a single setting to turn off the firewall and then link it to the appropriate server(s).

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

Brad Hedlund

stuff and nonsense


Every cloud has a silver lining.

Live Virtually

A Storage and Virtualization Blog

Virtualization Team

VMware ESX/ESXi - ESX server - Virtualization - vCloud Director, tutorials, how-to, video


Just another WordPress.com site

VirtualKenneth's Blog - hqVirtual | hire quality

Virtualization Blog focused on VMware Environments


Virtually everything is POSHable

Gabes Virtual World

Your P.I. on virtualization

Yellow Bricks

by Duncan Epping

Wahl Network

Technical Solutions for Technical People

Joking's Blog

Phoenix-based IT guy with too much to say...

%d bloggers like this: