My trip isn’t going as planned

A client had developed an in house application that utilized data sources from multiple servers. It was an application that uses GIS data to plan trips, and was experiencing uncharacteristically slow application response times. The application had been running for several years, in various incarnations, and had been recently ‘tweaked’ by a software engineering contractor who had been hired after the lead developer left the organization five months earlier.

The application had been performing for the previous several months with an average response time 4.5 seconds. I had helped the client setup a testing environment using synthetic transactions to measure the response time, and we had plenty of supporting data for the prior time periods. The testing system was reporting times in excess of 30 seconds to perform the same function.

Identifying the problem

After discussing the code changes with the new software engineer, I began collecting data with the analyzer to determine the source of the latency. When looking at the response time between the client and the front end HTTP server, it appeared that everything in this space was performing as expected. The application, session, transport, etc. were all looking fine. Next, I moved the analyzer to the data path between the HTTP server and the Oracle™ database server. From here I was able to see several samples of a large delay between the two servers. A request would go out from the HTTP server, and a long period of time elapsed prior to the database server response. Continuing the troubleshooting required an examination of the rest of the systems architecture, so I needed to capture data between the Oracle™ server and the servers holding the GIS data. Now that I had data from all the data paths in this four tier client server application, I was able to begin the process of moving to identifying the problem.

As with any analysis, it requires one to take an iterative approach of piecing small pieces of the larger puzzle together. Once you have a few pieces, you interconnect them to determine if the entire puzzle is in front of you, or pieces are missing (requiring additional data collection and analysis). In this case, I pieced together the chain of data flowing from the client to the front-end HTTP server, next came the data from the HTTP server to the Oracle™ server, followed by the data from the Oracle™ server to the GIS server. In going through the tasks involved to deliver a valid response to the user, multiple sub-tasks are going on behind the scenes (reflected by the modules within the application) and this data can be seen using the analyzer. In this case, it was clear that the vast majority of the latency was between the HTTP server and the Oracle™ server, since the HTTP and GIS aspects were responding well.

In looking closely at the data traveling between the two servers, the HTTP server was making two separate SQL calls to the Oracle™ server. In looking at these two calls, the majority of the latency was coming from the second SQL call. The second SQL query took, on average, eight times longer to complete.

Problem resolved

I took my findings to the new software engineer, and he said there was a mistake in the analysis. He was adamant that there was only a single SQL query initiated from the HTTP server to the Oracle™ database. When I walked through multiple sessions captured during the analysis, I convinced him that indeed, there were two SQL calls going to database, instead of just one. I also convinced him that it was the second call that was causing the problem. Based upon this information (since we had the actual SQL syntax to search for in the code), he reviewed the code and located the module making the call. It soon became evident that the program was not supposed to be running the code in the module, and he had a logic problem to resolve.

Later that day, the software engineer made the appropriate changes to the code, and wanted a verification that the second SQL call was not being issued. Even though the response times returned to normal (averaging 4.5 seconds) after the change, we needed to verify that the second call was not being made. I reconnected the analyzer to the systems in question, ran several tests, and concluded that the code was running properly (not placing the second SQL call).

The TIES that bind

I was asked by a client to identify a significant response time problem with an application they were ready to launch into full production. This application, called TIES, was developed in house that consisted of several man years of development time, and was primary to their core business function. During the development cycle, testing was performed to determine if the application had acceptable response time, which it did. Based upon this information, they went into phase one of the production rollout. As with many organizations, the initial testing proved to be inadequate, and the application (and its users) was now experiencing significant latency now that a portion of their user community was asked to use in a production environment.

The organization was under considerable pressure to launch the application into full production, due to their phasing out of its predecessor. They had lost the staff to properly maintain the legacy system, and were compelled to cutover to the new system post haste. Obviously, they were now at a standstill until this problem could be resolved.

Identifying the problem

In gathering the necessary information required to begin systematically troubleshoot this issue, I had a lengthy conversation with the lead developer. We discussed system architecture, interfaces to the database, data flow, expected response time, recent code changes, previous testing results, and much more. Armed with this information, I now was ready to utilize a packet analyzer with the appropriate filters in place. I moved to several locations in the path of the data stream due to this being a multi-tier client server application. After obtaining multiple examples of application usage (users completing specific tasks within the application) at each data path, I was ready to begin the analysis.

Problem resolved

Once I had gone over the packet captures in significant detail, I was able to locate an area within the application of significant and consistent latency right after a SQL call went to the database. In each case, it was taking 10-11 seconds for the database server to respond with the result set. Since the client was seeing response times of around 13 seconds, I new that this was the cause of the problem. I had the actual SQL code used in these database queries and went to the lead developer with this information.

Upon analysis, it was identified that the SQL call going out to the database was correct, and expected. What was not expected was the time taken to run the query, which had previously taken significantly less time during the testing phase. Upon additional analysis using Oracle tools, it was determined the indexing on the tables was inadequate, and inefficient. However, this indexing problem only began showing up after the system was under a more significant load presented by the user community.

Once the table indexing was properly adjusted, the application response time increased significantly. I once again placed the analyzer inline with the client, and was able to verify that with this change, the application went from 13 seconds to sub 3 seconds for completion in all cases. The client began adding more users into the mix, and additional packet analysis showed the application could sustain sub 3 seconds response time even with a significant increase in the number of users accessing the system.

With this information, the IT management team decided to proceed with the launch of the application into full production.

Telephones that ramble without a standard preamble

Right after upgrading to the latest HP Procurve™ switch gear at two facilities, a client lost connectivity to its Avaya™ telephone switches at each facility. These switches were operating for the previous 18 months on the pre-existing switch gear without issue. An easy fix seemed to be just around the corner, but it was not to be.

Identifying the problem

This turned out to be an elusive problem after the initial course of troubleshooting turned up no definitive answers. When looking at the switch, it showed no packets or data was hitting the switch port, even though the Avaya™ gear had a link. When looking from the perspective of the Avaya™ gear, the network staff was able to ping the IP address on the Avaya™ port, but couldn’t get to the switch nor the default router. It was time for a packet analyzer.

At first, the analyzer was not picking up any packets coming from the Avaya™ product. Not a single broadcast, ARP, or other packet was emitted from the stingy system. This was quite abnormal for anything with a working OS – total silence is not indicative of any system, much less our previous experiences with Avaya™ gear. We were using the built in mirroring feature of the Procurve™ gear (similar to SPAN on Cisco™), to redirect traffic from the port being monitored. This has been used dozens of times, at this client, with positive results. However, one must remember that mirror or SPAN ports forward packets meeting specific requirements. Packets with layer two errors are not forwarded by these switches mirroring function. However, we already verified the port was showing absolutely no errors, and no counter was incrementing for incoming traffic. This was a bit perplexing, since we could place the Avaya™ gear onto another switch, and just like a light switch, the data started flowing to and from the Avaya™ gear.

With this knowledge, I placed my trusty Ethernet Tap between the Procurve™ and the Avaya™ gear and noticed something interesting. The Avaya™ gear was carrying on normal conversations (or at least it thought) with any system that communicated with it. When sending pings from the default router, the Avaya™ responded promptly to each and every ICMP packet. When an ARP request came for the Avaya™ gear, it immediately responded with a reply. This was the evidence we were looking for. The piece of the puzzle that we did not have from the previous troubleshooting.

Problem resolved – (Band-aid applied until vendor patch is provided)

Using an intermediate fix of connecting the Avaya™ gear to an alternate switch at the facilities wiring closet, we were able to get the switch back to operational status. We opened a ticket with HP Procurve™ tech support with the dump of the technical data from switch, and the packet capture. The tech support person could not find a resolution, so he escalated it to a Tier 2 engineer, who again escalated to a Tier 3 engineer. The Tier 3 engineer opened a trouble ticket with a high level Avaya™ engineer, and the two discussed the issue.

It turns out that the Avaya™ equipment does not use the standard IEEE 802.3 convention of a 7 byte preamble followed by a single byte start of frame delimiter prior to the rest of the Ethernet framing data. It also turns out that the latest HP Procurve™ switch forces the incoming packet to be in compliance with the 802.3 specification, or it dumps the packet(s) without even incrementing the ports receive packet/byte counter (don’t worry, some Cisco™ switch gear acts the same way). So, what is the permanent solution you ask? Avaya™ is currently investigating changing their code to use the 802.3 specification, and HP™ is taking the hardline of making sure their new gear enforces the IEEE standard. When the permanent fix does come, it will be in the form of a patch to the Avaya™ firmware – make sure to look for it.

Is this packet looking big, or it just me?

A client, who recently changed some of their IP subnets, was experiencing a lot of delay when transferring data from one of their critical systems. The data being transferred, a vehicle location system, is updated in a real time manner using GPS. This was having a significant impact on their ability to deliver accurate information to the internal and external user community.

In this case, the client communicated to me the data transfers, which normally take a few seconds, were taking a few minutes. So, by the time the client was getting the information, the data was already outdated.

The server sending the data was an older HP server connected to an FDDI ring. They paid more than a million dollars for the entire system, including the server, workstations, and software. It had not been upgraded in a long time and had been in place for years. Prior to this cutover, the client could not update the IP addressing due to the lack of support from the vendor. This system had to continue using the same datacomm gear and IP addressing that was assigned several years earlier.

Identifying the problem

In cutting over to the ‘new’ IP subnet for the rest of the network users, a router was added between the real time system and the other business servers. After connecting the analyzer, it soon became obvious that a good deal of ICMP packets originating from the new router were headed towards the real time server. These messages were telling the server that packet fragmentation was required. This was because the real time server was sending out packets at the MTU (Maximum Transfer Unit) size of its local connection – FDDI. The MTU for FDDI is more than 4000 bytes, where the Ethernet router in the middle is about 1500 bytes. The problem appeared to be initially created by the two servers negotiating what MTU size they could agree to use. They were both on LAN segments that allowed much larger packet sizes (FDDI) than the Ethernet router now connecting them together.

There are a few different mechanisms systems can use to ensure the MTU size does not exceed the limit of the WANs/LANs they are traversing. This is typically referred to as path MTU discovery. The real time server was not performing the PMTU discovery, and worse yet, it was ignoring the ICMP messages of the router, which was getting overloaded and dropping many of the packets. This had not occurred before because both servers were connected via FDDI rings and the MTU for FDDI could be used without problem. Now that there was a ‘smaller’ LAN connecting them, and the real time server was not conducting PMTU discovery nor heeding the fragmentation messages of the router, the packet loss was large and retransmissions were causing very lengthy session times.

Problem resolved

With the discovery of the MTU sizing problems, I conducted a bit of research to find out how to hard set the MTU size on the older HPUX OS to match the Ethernet MTU size for this connection, and not the larger MTU size of FDDI. After obtaining this information, I presented the solution to the UNIX staff the following morning. They immediately implemented the change on the real time server, and we retested the data transfer session. The speed was once again between two and three seconds, and a check of the analyzer showed no additional ICMP fragmentation messages were being sent.

Three new ticket vending machines – 48,000 dollars, The ability to process credit cards – priceless

A client called me to verify whether their new and existing ticket sales kiosks were compliant with the tougher PCI credit card standards. As of Jan 1, 2007, the credit card companies (Payment Card Industry – PCI) mandated a new security compliance initiative call PCI DSS v.1.1 to allow businesses and other organizations to process credit card information. Any business or organization that processes credit card data must comply with this new standard, or face the consequence – do a web search on TJ Maxx’s™ inability to properly secure their cardholder data.

The core of the PCI DSS (Data Security Standard) is a group of principles and accompanying requirements, around which the specific elements of the DSS are organized. Requirement number four specifically deals with ensuring credit cardholder data is transferred in an encrypted manner across public or open networks.

Identifying the problem

In this case, the client was not aware of a problem, but wanted to make sure their organization was in compliance with the new PCI DSS standard, as they did not want to face potential fines and public scrutiny. In addition to ensuring the organization was meeting the other eleven DSS requirements, I am focusing this write-up only on requirement number four – discussed above.

I went to the remote location where the client has several of their ticket vending machines. These are basically kiosks that sell tickets to consumers in an automated manner. The client had a recent infusion of new kiosks and software, for processing credit cards. They have these kiosks at multiple facilities, so we analyzed the facility with the highest concentration of machines. I setup the analyzer and waited for a customer to use their credit card on the machine. In looking at the data transmission, it was clear that the client was connecting to an outside credit card processing agency. It was also clear that an SSL session was being initiated by the kiosk, and transmitting the credit cardholder information using encryption. In looking further in the packets, I was able to certify that the client connection was using SSL v.3 and was using a valid Verisign™ certificate. Additionally, I was able to certify that the kiosks were doing this while using 128 bit encryption.

Problem resolved

Even though there was not a problem to resolve in a literal sense, the client received the packet captures as documentation with the relevant information highlighted certifying that they were indeed in compliance with requirement number four of PCI DSS v.1.1. We could now focus on the eleven other requirements.

Am I insecure, or what?

This is a case where the IT manager had an idea that the development team was not properly securing the HR data transfers between their legacy system and the ERP system. There was a perceived lack of understanding of the data privacy implications during the data updates between the systems.

The IT manager asked that I perform a packet analysis session with the intention of intercepting data to/from the ERP backend database servers. The purpose of this packet analysis session is to determine if the data/ information captured to/from the ERP servers places the organization at risk from eaves dropping and other security concerns, specifically in dealing with personal employee information.

Identifying the problem

Making sure to have a documented sign-off from the manager requesting that I purposefully look for sensitive data being transferred to/from the system, I began with connecting the analyzer in front of the ERP servers.

After understanding the traffic paths of the systems in question, I looked at a few snapshots of data from the analyzer. Not knowing what exactly to expect, I setup some very detailed capture filters (based upon some initial data captures) and let the analyzer run for five days. When I returned, I had captured multiple files of data, and began looking through the data payloads within the captured packets.

It was clear that the data transfers between the ERP system and the legacy system were going over FTP in clear text. Not only was I able to capture the login credentials required to initiate the transfer, but the data being transferred between the systems as well.

In many cases, people know that sensitive data can be accessed on their internal systems, but it is a far more imposing to show the decision makers their personal information. So in addition to showing the root level passwords captured while the server was logging onto the remote system, I printed out a single copy of the payroll details for both the IT manager and the CEO of the company. Payroll information related to gross salary, deductions, taxes, and net payroll numbers for this pay period and year to date were on this document. This included the SSN, birthdate, position, race, hire date, employee ID was included as well. It didn’t take long for the IT manager to understand the implications of this while looking at all the numbers printed on this sheet, which perfectly matched his latest paystub.

In addition to being able to see this information, other security implications were immediately obvious. These included – 1) the logon credentials of the systems and people logging into the system to initiate the file transfer, this allowed the potential for data file and OS file level manipulation, and 2) the file structure was visible when the server to server transfer was occurring. You could see the location and name of the payroll file for the entire organization.

Problem resolved

Recommendations were immediately made to shutdown the FTP services on this and other servers within the computer room. Secure Shell (SSH) was initially implemented and further investigation was conducted to determine and appropriate long term solution.

Our time clocks keep getting lost

One of my clients uses time clocks (for its payroll system) at several of their larger facilities. A large portion of their staff use these time clocks when entering and leaving work. This data is tied into their payroll system, which uses CyberShift™ as their workforce management software. On several occasions, they ‘lost’ one or more of the time clocks, where the server could no longer see the time clock. They were working with the CyberShift™ technical support staff, which was not able to come up with a ‘fix’. They called me with the expectation of determining why this was happening.

Identifying the problem

Since this was not an everyday occurrence, I asked the client to call me in when it happened again to begin the analysis. Upon arrival, I learned that they had lost connections to all time clocks at a specific site. I went to their main site where the computer room housed the system server, to verify if the server could connect to the time clocks from there. I viewed several connection attempts, but I did not view any data coming from the time clocks in question. After this, we went to the site where the time clocks resided and connected the packet analyzer. This time, we could see the time clocks receive data from the server, and send data back to the server. However, each data stream was abrupt, and new TCP sessions were initiated for each request coming in from the server.

So, both the server and the time clocks were sending data to each other, but only the data from the server was actually getting to the time clocks. The data being sent by the time clocks was not getting to the server.

Since, 1) everything appeared normal at this time, and 2) we knew the data was not getting back to the server, we focused on determining why the data could be seen on the local LAN, but not getting through to the LAN where the server resided. They were using a T1 to connect this remote site to their main site. They had recently cutover their switch gear to a different manufacturer, and also installed a new router due to the cutover. Thinking they may have a routing issue, they stated that no other applications or users were exhibiting any problems/symptoms.

We looked for the obvious things first – packet loss on the local LAN, packet loss on the T1, routing configuration, etc. No problems with any of this. So, this required a more detailed look through the packet decodes. Starting with layer one of the OSI model and working up towards layer seven (this can be done in reverse as well), we began going over the packets from the time clocks to the server in fine detail. We had already looked at the upper layers during the initial analysis and noted that layer three (network layer) through seven (application layer) appeared normal (no IP address problems, no DNS problems, no transport problems, applications were communicating on valid port numbers, etc. etc), so we focused the analysis at the lower layers (< layer three). Having already looked into layer one and layer two issues (using the snmp counters) on the switch and router, we next looked at layer two on the packet decodes. This was where the problem was located. In going through the packet decodes, it was determined the MAC address for the router was different for the time clocks than for all other traffic on the network. In doing more investigation, it was found that the time clocks were using the ‘old’ routers MAC address, even though a new router was being used.

The packet analyzer was reconnected to the network and let it run for quite awhile. During this time, the time clock never sent an ARP packet out for the default gateway. It continued to use the MAC address of the previous router, even though it had been removed earlier during the week. We were able to document the time clocks ARP cache never timed out for communicating with the router, even though it had changed three days prior.

Problem resolved

To get the client back online that day – we did a temporary fix by depowering and repowering the time clock (which was under lock and key). This cleared the ARP cache for the timeclock, requiring the time clock to send out an ARP packet to locate the default gateway, which responded properly.

After communicating this information to CyberShift™, they communicated they purchased the time clocks from a separate manufacturer. When we spoke with the technical support department at the time clock manufacturer about the ARP cache never timing out, they communicated that they used a third party NIC card. They needed to contact this company prior to accepting the results. After a few con-calls with CyberShift™, the time clock manufacturer, and the NIC card manufacturer, it was decided that a firmware upgrade would correct the problem. They sent the new firmware code, and we loaded it onto some test time clocks. We retested to see if the time clocks could establish communication with the new router when the MAC address changed on the default gateway (by either ARP’ing for the gateway after an ARP cache timeout, or listening to the gratuitous ARP sent from the ‘new’ router when it came online). It did neither. This information was communicated to the time clock manufacturer, and they once again went back to researching the problem.

The time clock manufacturer got back in contact with the client about 30 days after the initial troubleshooting. They agreed that it was a defect in the time clock system, and asked that the client replace all time clocks at all facilities with their new time clock (which had been tested, and did not exhibit the same problem). This task was performed, and the problem has not returned. Kudos to CyberShift™, and the time clock manufacturer for taking ownership and getting a permanent solution to the problem.

The new phone system is transferring more than phone calls

Recently, a client asked me to take a look at the cause of multiple TFTP (Trivial File Transfer Protocol) requests coming from unknown computers directed at their datacommunication switch gear. Not knowing if this was traffic initiated by a virus, they needed an in depth look. Their switch equipment manufacturer notified them, during a tech support call, that the TFTP requests could cause the switches OS to halt if the requests continued.

Identifying the problem

After connecting the analyzer onto the appropriate place on their network, I was able to locate the source/destination path for the TFTP traffic. It was identified that the TFTP traffic was coming from a group of approximately twelve workstations within a specific department. It was also identified that the TFTP data requests were not destined solely for the datacomm switches, but to all stations within the subnet. The workstations were sending TFTP requests to a broadcast MAC address, and requesting a specific file from any/all stations on their local network. Not good.

Further investigation found that these workstations were in a call center, and used an application specific for the phone systems call recording. Also, I looked at the specific TFTP request and found that it had a unique string. In researching this TFTP string, it was traced to the recently installed Avaya™ phone system.

Looking further at the packet decodes for this TFTP issue resulted in several additional findings of potential security issues within the various software modules supported on the Avaya™ system. Based upon this research, several trouble tickets were started with Avaya’s™ tech support department.

Problem resolved

Unfortunately, many times when small, medium, or large software and hardware manufacturers are alerted to deficiencies in their code (software or firmware), these companies take a hands off attitude for making corrections. In this case, Avaya™ stated that the TFTP’s broadcasts were essential to software modules the client was running, and this code could not be changed. Avaya™ communicated the TFTP broadcasts were used to activate the voicemail light on the phones at the customer site. This came as a surprise, since the phones at this organization were not connected to the data communications network (they were on a dedicated cable plant) and have no mechanism to ‘see’ these broadcasted TFTP packets. Instead of placing a checkbox within the Avaya™ software to turn this feature off, and stop the broadcasts (since the phones were not even connected to the same network), they decided it was ‘an integral function of the software’ and could not be changed. Too bad for Avaya™ customers.

In this case, since we were dealing with a non-responsive vendor, we had to take matters into our own hands and resolve the problem in a different manner. One of the least costly ways to correct this issue was to install personal firewall software on each of the one dozen workstations. The firewall on each workstation needed to be configured to disallow TFTP to a broadcast address. TFTP packets were allowed to be directed at specific systems (using unicast packets) since this transmission was required for their call recording and hunt group lists. With the firewall configured in this manner the phone system will continue to operate properly, and the data communication switches no longer are subjected to this barrage of TFTP file requests.

Post mortem

Based upon the analysis conducted on the Avaya™ phone system, I would highly recommend that organizations using this system (specifically the IP Office system) be aware of several known (and potentially unknown) security threats to this system. As of this writing, much of the communication between workstations and servers using the IP Office system uses TFTP for transferring data to/from the server. This communication is a base function of the system architecture, and can be seen each minute of the day (in the organization I looked at anyway). The majority of these connections DO NOT USE credentials to obtain this information. I could easily obtain phone system information connecting to the server and issuing ‘get’ commands. You can obtain various system information like hunt groups, user lists, system users, etc. and you can get this without using an analyzer or having a password.

Additionally, there are known problems with password hashing, where the hash of the password can be used, among other things, to reboot the IP Office server, when using specific commands like –

“tftp -i <Address of IP Office Server> get “/nasystem/reboot/<insert captured password hash>”

There are other security issues with the Avaya™ system related to local passwords being saved in clear text on the computer (find this using regedit), and snmp credentialing accepting any string that has the same number of characters as the proper credentials.

Some of these issues are being corrected by Avaya™, some are not, so make sure you have the latest IP Office software.

In summary, educate yourself to the specific vulnerabilities of your phone system.

The latest network diet – getting two gigabytes thinner

At a client where I am onsite on a regular basis, I decided to take advantage of the slower than normal activity workload during the holidays, and take a look at some of the applications/systems traversing the infrastructure. I wanted to locate systems that might be exhibiting symptoms or problems, even though users were not placing help desk calls for these systems.

Identifying the problem

In looking at the main through-points on the network, I was able to uncover several misbehaving applications/systems. Some of these applications/systems were merely misconfigured, while others contain poorly written modules.

I made up a list of the top five worst offenders, and worked with the appropriate IT group in each case to make changes required to alleviate the problems.

Brief Synopsis of the issues

1) One of the systems on the network had 30 wireless access points (of the more than 72 on this system) attempting to connect to a management server at a rate of over 11,000 connections an hour (about 3 connection attempts a second). This might not sound particularly egregious at first blush, but the caveat to this one is the server refused these connection attempts, stating that it did not have a ‘listener’ for the server to connect to.

2) The disk backup system in use at the client also exhibited a problem where the backup server refused connections from approximately 52 clients resulting in 5,500 unanswered connection attempts each hour of the day. Just as in the case above, the server stated it did not have an appropriate ‘listener’ running on the server.

3) One of the servers on the network, used for discovery and inventory purposes, needed route table configuration changes, after receiving more than 1000 route redirects during the course of the analysis.

4) The server used to manage the printing infrastructure required updating due to having more than 1300 queries to non-existing services/printers on the network. It was found to be making more than 15,000 calls each day (using snmp and netbios).

5) A specialized vehicle tracking system was found to be contacting multiple systems with data requests to servers that could not respond. This was occurring hundreds of times a day, and was taking up resources from the other servers.

Problem resolved

In each of the cases, further investigation was conducted on the configuration and/or architecture of the system and its applications. In each case, configuration changes were made where possible. Some of the systems required communication with the vendor’s software development team requesting a code modification for specific module of the code. Among other items, the subsequent changes resulted in 1) reducing the overhead for the affected servers/computers required for processing these packets, and 2) removing two gigabytes of LAN/WAN traffic each month for the non-needed data produced by these five systems.

The case of the missing link

In this case, the client was adding video security cameras to various locations throughout their organization. Since the video feeds were to run over the main networks, they decided to add a separate video VLAN to segment the traffic from the other corporate traffic. While doing this, they visited each site and changed the configuration on their main Cisco™ routers (actually Layer 3 switches). All was well, except the last site. At this last site, the two (redundant) gigabit uplinks linking the facilities main switches to the router continued to shutdown. Instead of a 30 minute cutover (as it had been at the other facilities) the staff was there for hours attempting to determine the cause of the link failure. After many hours of troubleshooting, they determined they would not be able to bring up the links as planned. To return the network to operational status, they disconnected the redundant link and shutdown spanning tree on the main VLAN (the cause of the link shutdown).

During the course of the next week, the client spent a good deal of time on phone with Cisco™ TAC support, attempting to isolate the problem. They were advised to upgrade the IOS code (since Cisco™ said the problem may lie within the OS), and were asked to send in the logs, along with the configuration. A week after the first attempt, they were able to obtain another maintenance window to install the new OS on the router, and attempted several configuration changes (ones suggested by Cisco™, and others found by the client staff conducting research). In each case, they were unsuccessful in bringing up the redundant link (or even the primary link with spanning tree running), even though the configuration of the network and the router was identical to the other sites. They spent several more hours on the phone with Cisco™ TAC support, with the same outcome, and once again had to remove the redundant link and shutdown spanning tree on the main VLAN (to bring the site back into service).

Identifying the problem

During the course of the following week the client planned to try additional configuration changes and once again obtained a maintenance window. This is when I was asked to analyze the situation, with the hope of identifying the source of the problem. In traveling to this facility, I placed the packet analyzer on the main link of the trunk (which was now acting only as an access link due to the malfunctioning redundant link) between the router and the main layer 2 switch gear at the facility. Even though the redundant link was no longer connected, I was hoping the reason behind the problem was continuing to occur on the network, and I could identify its source.

After capturing about a half million packets, I brought the analyzer back to the office, and the work of looking for the needle in the haystack began. After just a few moments, I was able to begin filtering out packets that were not of interest (not the obvious cause of this problem). After continual scanning through the packets, it was apparent that there was a great deal of Cisco™ VLAN Spanning Tree Protocol (PVST) traffic running over the network. And, most of the spanning tree packets were not good ones, identifying there were significant problems on this network. Throughout the course of the packet capture, only a very small portion were spanning tree. However, when looking at just the spanning tree packets, a good majority were topology change notification packets. In filtering only on this type of spanning tree packet, it was determined that there were between 20 to 30 of these packets EACH SECOND. This is not good.

Since all the other data-communication equipment (switches) at the facility is non-Cisco™ gear, and do not operate using PVST, it was presumed that the source of the spanning tree packets were coming from the Cisco™ router at the site. However, as with any good forensic work, you need to start with the evidence you have in front of you, and work up from there to build your case. As such, I began inspecting the source MAC addresses found in the spanning tree packets, and found that 95% of the traffic was originating from two unique addresses (both of which used the Cisco™ manufacturers MAC address code). On further review, it was determined that the MAC addresses were not ones from the main Cisco™ router at all. How could that be, since all the other datacomm equipment at the facility was non-Cisco™? After looking through all the CAM tables of the each of the switches at the facility, the MAC addresses were tracked down to being on two separate sets of trunk ports (trunk 5 on switch one and trunk 6 on switch 4). I asked the client staff what was connected to these trunks, and they replied it was the IBM blade center (the ones on the TV commercials – reducing the number of servers in the computer room to just one). After a small discussion, the client staff reminded me within each blade center was an integral Cisco™ switch (that’s right, a Cisco™ switch (in lieu of a NIC) imbedded in the server’s blade center). Upon connecting to the IBM blade center console ports, the MAC addresses in the packet analysis were identified as being one port on blade one, and another on blade two.

Now that the source of the packets were identified, it was time to identify the problem. In this case, there were two separate problems. Problem One, It was obvious that the blade center was in a constant state of spanning tree reconvergence due to 20 to 30 topology changes each second. This places a severe strain on any and all equipment that is within earshot of these packets, as they are constantly dumping their MAC to port associations, and recalculating the topology tree. Luckily for the client, none of the other datacomm equipment (all their switch gear) at the facility was Cisco™ branded equipment. Since the IBM blades were using PVST, Cisco™’s proprietary spanning tree protocol, none of the datacomm equipment at the facility (with the exception of the Cisco™ router) used this for their spanning tree calculations, and in effect, simply ignored the spanning tree problems and went about the business of forwarding packets. Problem Two, even though these topology change notifications were showing significant signs of problems, it was not the cause of the missing link (or shutdown of the both the primary and redundant link in this case) on the router. In looking at the details of the PVST protocol, there are many significant and often overlooked details dealing with how the VLAN tags (or being untagged) are critical to the functioning and operational state of the links. Without going through all the minutiae, after changing the VLAN structure on the Cisco™ router (for the video security devices), it was receiving PVST packets that it deemed to be on the wrong VLAN and shutdown all ports for the VLAN’s in question (which caused all corporate traffic to stop being forwarded on both the primary and secondary links). Even though this spanning tree traffic was occurring before the addition of the video security VLAN, there was not a problem with the differences in the PVST VLAN tagging between the IBM blade center and the Cisco™ router. The problem began occurring only when the VLAN structure changed.

Problem resolved

With the assistance of the client’s server team, steps were taken to properly configure the IBM blade center, and the Cisco™ router once again behaved like expected and returned to operational status on both of its links. The Cisco™ TAC support tickets were closed, and both the network and server team staff got back to conducting normal activities.

120 servers that went from virtual to invisible

I was called by a client on a Sunday morning after scheduled power outage in the computer room the night before. The client was having trouble bringing up their VMware™ environment, which consists of 120 virtual servers. They had been attempting to bring up the virtual server environment, and each time, the systems failed to ‘come up’ on the network. They had been working on the problem for several hours, prior to calling me at 7:50 AM on Sunday. I arrived on-site and found that all of the non-VMware™ systems were fully restored to operational status after the power was turned back on (beginning at 3:00 AM). However, they attempted to get the VMware™ servers operational (using the console ports) and were unsuccessful. Thinking it was a problem with the datacomm gear in the computer room, they needed additional assistance.

After briefly consulting them about their findings, I learned that the VMware™ systems kept responding that there was a duplicate IP address in use when each of the servers were attempting to come online. The server team tried bringing many different VMware™ servers up on the network, with the same result. They even changed the IP addresses on multiple servers attempting to bring them online, to no avail (same error message on the server console each time). The strange part was 1) the servers were hard coded with the same static IP addresses since they were installed months earlier, 2) the server team attempted to ping the IP addresses that were reportedly now ‘duplicate IP addresses in use’ and found nothing, 3) if they brought any of the servers up on an isolated network switch, it would work fine even when they then connected the server back into the main switch gear after it was operational (very puzzling, and leading them to believe it was a problem with the main datacomm gear), and 4) everything had been working fine for the previous several months with these same addresses.

Identifying the problem

It was evident that a packet analyzer was needed, and the process of collecting data began. With packets pouring in by the thousands, I left the analyzer running for a few minutes prior to pausing to look at the data for clues on what was causing this problem.

When a VM system comes online (other systems do this as well), it sends an ARP packet out on the network to determine if anyone else is using the IP address that it is attempting to use. If it determines some other system is using the IP address, it will no longer use layer 3 services, and will not provision itself with an IP address (it uses 0.0.0.0 for all additional network packets).

In looking through the packet captures, I could see where the server (the VMware™ server we were initially troubleshooting) would send out an ARP packet requesting the MAC address for 192.168.0.10 (the IP address assigned to the server). A device on the network responded with the MAC address of xx:xx:xx:ce:20:cd (manufacturer code obscured), and the IP of 192.168.0.10. In looking at the manufacturer’s code portion of the MAC address, it was easy to tell that this system was not a VMware™ box.

The next step was to quickly locate this system, and isolate it from the rest of the network. The client has a server using scanning software to identify all of the machines connected to the network, by IP address. However, this scanner software could not locate the offending box. Next, ping tests were issued from multiple machines in attempts to illicit a response. The box would not respond to pings. Lastly, we went through the MAC tables for each of the main switches to determine the source port of the MAC address for this system. It was located within a few minutes, and isolated to a switch on the third floor, module #A port #1.

Problem resolved

Now that the source of the problem was identified, the switch port connecting this device was administratively shutdown and the network staff was dispatched to locate the device. It was determined that the offending device was a specialized clock that used the network to obtain time (using NTP protocol). The clock was promptly disconnected from the network. The server team was notified of the corrective action, and once again began the process of bringing up the VMware™ servers. At this time, all the VMware™ servers returned back online to full operational status with no errors.

Having a duplicate IP address on the network would cause the VMware™ servers to not come online. But that didn’t explain why none of the VMware™ systems would come up on the network, as the time clocks static IP address was different than any of the VMware™ servers. To find out why, more detailed analysis was undertaken.

Typically, when the VMware™ system sends out an ARP (for the purposes of determining who else is up in the cluster) it pads the THA (Target Hardware Address) of the ARP packet with all zeroes (00:00:00:00:00:00). In the case of when it attempts to come online, and determine if the IP address is already in use, it pads the THA with all ones (FF:FF:FF:FF:FF:FF). It was only at this time that the ARP reply from the clock sent a response saying it was indeed 192.168.0.10 (or any other IP address for that matter). Thereby causing the network layer (Layer 3) of the VMware™ server to shutdown and relinquish it’s IP address (and not allowing the server to be operational on the network). However, this is also why, when the staff attempted to illicit ping responses from the clock, it would not respond with a reply, since the THA is not all ones in the ping request. This only further exacerbated the earlier troubleshooting efforts of the IT staff. Fortunately for the client, the VMware™ systems were stable prior to this time, otherwise they would have had some very strange problems with individual servers not coming online, which would have been difficult for them to troubleshoot.

Post mortem

Based upon the analysis above, the clock was determined to have defective firmware code, where it responded to any and all gratuitous ARP packets sent out on the network, regardless of the IP address of the sender. In each case, the clock confirmed it too had the same IP address (even though it did not), and responded to the gratuitous ARP request. Due to this factor, the clock was not allowed on the network after this occurrence. Since the manufacturer was unable to correct the firmware code on the clock, the malfunctioning clock was returned for another model which receives it’s time using radio waves (non-networked). The manufacturer initially denied the problem, but later stated they had one other client reporting a similar type of problem (the other client was unable to produce the technical analysis describing why the clocks firmware was malfunctioning).