Right after upgrading to the latest HP Procurve™ switch gear at two facilities, a client lost connectivity to its Avaya™ telephone switches at each facility. These switches were operating for the previous 18 months on the pre-existing switch gear without issue. An easy fix seemed to be just around the corner, but it was not to be.
Identifying the problem
This turned out to be an elusive problem after the initial course of troubleshooting turned up no definitive answers. When looking at the switch, it showed no packets or data was hitting the switch port, even though the Avaya™ gear had a link. When looking from the perspective of the Avaya™ gear, the network staff was able to ping the IP address on the Avaya™ port, but couldn’t get to the switch nor the default router. It was time for a packet analyzer.
At first, the analyzer was not picking up any packets coming from the Avaya™ product. Not a single broadcast, ARP, or other packet was emitted from the stingy system. This was quite abnormal for anything with a working OS – total silence is not indicative of any system, much less our previous experiences with Avaya™ gear. We were using the built in mirroring feature of the Procurve™ gear (similar to SPAN on Cisco™), to redirect traffic from the port being monitored. This has been used dozens of times, at this client, with positive results. However, one must remember that mirror or SPAN ports forward packets meeting specific requirements. Packets with layer two errors are not forwarded by these switches mirroring function. However, we already verified the port was showing absolutely no errors, and no counter was incrementing for incoming traffic. This was a bit perplexing, since we could place the Avaya™ gear onto another switch, and just like a light switch, the data started flowing to and from the Avaya™ gear.
With this knowledge, I placed my trusty Ethernet Tap between the Procurve™ and the Avaya™ gear and noticed something interesting. The Avaya™ gear was carrying on normal conversations (or at least it thought) with any system that communicated with it. When sending pings from the default router, the Avaya™ responded promptly to each and every ICMP packet. When an ARP request came for the Avaya™ gear, it immediately responded with a reply. This was the evidence we were looking for. The piece of the puzzle that we did not have from the previous troubleshooting.
Problem resolved – (Band-aid applied until vendor patch is provided)
Using an intermediate fix of connecting the Avaya™ gear to an alternate switch at the facilities wiring closet, we were able to get the switch back to operational status. We opened a ticket with HP Procurve™ tech support with the dump of the technical data from switch, and the packet capture. The tech support person could not find a resolution, so he escalated it to a Tier 2 engineer, who again escalated to a Tier 3 engineer. The Tier 3 engineer opened a trouble ticket with a high level Avaya™ engineer, and the two discussed the issue.
It turns out that the Avaya™ equipment does not use the standard IEEE 802.3 convention of a 7 byte preamble followed by a single byte start of frame delimiter prior to the rest of the Ethernet framing data. It also turns out that the latest HP Procurve™ switch forces the incoming packet to be in compliance with the 802.3 specification, or it dumps the packet(s) without even incrementing the ports receive packet/byte counter (don’t worry, some Cisco™ switch gear acts the same way). So, what is the permanent solution you ask? Avaya™ is currently investigating changing their code to use the 802.3 specification, and HP™ is taking the hardline of making sure their new gear enforces the IEEE standard. When the permanent fix does come, it will be in the form of a patch to the Avaya™ firmware – make sure to look for it.
