Thursday, March 30, 2006

Cisco 2811 Branch Router Install - Remote & On-site Support

This project for the bank is a small piece of a larger project and its scope is straight forward and fairly simple. The bank is a very good long-standing customer and has confidence in our company as I have installed other Cisco gear for them flawlessly. The next attempt at installing this new equipment must be successful to sustain their level of confidence in us. The problem we've encountered with installing this particular branch router seems to be an unusual one as TAC is also, at this point, stumped for a solution.

Product Small Photo

On-demand PC Remote Control

An aging 2600 series branch router is to be replaced by a more robust and sophisticated multiservice 2811XM router. This branch, as all others, has frame-relay as it's method for wide area connectivity back to the bank's core data center. The branch differs from the others in a single important regard. The difference is that not only does it perform normal banking transactions but it also has an the added role of being a DR site for the meshed frame network. It therefore has additional frame-relay configuration parameters that are not found on the bank's "regular" branch routers. The configuration of the router is a copy of the current 2600. Except for interface enumeration, i.e. Serial 0 on the old compared with Serial 0/0/0 on the new, they are configured near exactly the same. The duplicity of the configuration can especially be seen, and has been re-examined numerous times even by TAC for accuracy and completeness, in the frame-relay and routing protocol sections of the configuration.

PC Remote Control On the Web

The first attempt at installing the router was on the March 15th. With router preparation having been completed at our company's pre-installation staging and lab area prior to the 15th, the next step was to go on-site as scheduled. Accompanied by a senior member of the bank's IT staff, the new branch router was promptly cut-over at 5:00PM. After the router booted and communication test to the core and other branches were performed it was discovered that the router, although functioning with respect to communicating with the other sites, was doing so through the wrong frame-relay path. The slower backup frame-network on the second sub-interface was being used instead of the intended primary interface.

Product Small Photo

It was determined that the wrong path was in use using basic tools, like trace route, ping and some router commands. Attempts to remedy were unsuccessful and included: re-loading, power cycles, shutting the second sub-interface to force the data path to the primary PVC, re-comparing the configurations removing and re-adding security related entries in the configuration. To be certain that I had not discovered a pre-existing problem with frame network, the old 2600 router was put temporarily back in again to see if it too would only use the back-up PVC. It did not - it used the primary PVC as expected. The 2600 series was removed again and the 2811 was re-connected but without success as all efforts did not return the desired results. TAC was not involved at this point and not until the next day were they contacted. The time window in which to get the new router installed was only an hour. With time having expired the original 2600 was put back into service. The 2811 was removed and returned back to our company's lab and staging area.

Computer Remote Control on the Web

A case was opened with TAC and I described accurately the situation and provided them with information as they requested and included IOS version numbers, results of IOS commands, etc.. The only thing I could not provide was active frame-relay statistics because the router was not connected to the bank's network. TAC could not find anything wrong with the configuration. Between communications with TAC, I connected the router with another 3825 in a standard frame-relay LAB configuration. The 3825 was configured as the frame-relay switch and DCE. After completing that configuration, the primary PVC came right up on the 2811 and I was able to ping across the simulated frame-relay network to the 3825. NO CHANGES were done on the 2811. TAC said to call the carrier to verify the frame-relay switch type and they could not help any further unless I was on-site and connected to the bank's network. They only offered at this pint as a solution to change the LMI type and the encapsulation type. The next install day was scheduled for March 30th.
Online Remote Desktop Support Software
Accompanied again by a member of the bank's IT staff, the router was cut-over after 5:00PM on the 30th. It loaded and operated the same as the week before by using the backup frame-relay and not the primary. I tried the few recommended TAC solutions but they did not yield the desired effect. I added a little attempt of my own by removing the modules that were not yet needed by the bank such as the BRI and the 4 port FXS and restarting (a hale Mary pass?...perhaps). While in transit to the location of the branch, I contacted TAC to have them on the line while on site. They asked for some status data which I sent. The time window for installation was again a small one so the 2600 was put back into service. I still have that case open with TAC and continued working with them today. There is no resolution as of yet. Othe post of Cisco networking gear could be found at Computer Networking News & Reviews and at this location, Evolution of Technology & Support.

Remote Desktop Control on the Web
So at this point I'm going to attemopt to escalate the problem higher in the TAC food chain. This in hopes of getting some people involved that could really address this problem properly as it's cause is not easily explainable and out of the range first level support personel.

I'll post more when it happens.

No comments: