Author: ccie14023

TAC Tales #13: All Zeros

Posted by | Apr 25, 2018 | , |

A common approach for TAC engineers and customers working on a tough case is to just “throw hardware at it.”  Sometimes this can be laziness:  why troubleshoot a complex problem when you can send an RMA, swap out a line card, and hope it works?  Other times it’s a legitimate step in a complex process of elimination.  RMA the card and if the problem still happens, well, you’ve eliminated the card as one source of the problem. Hence, it was not an uncommon event the day that I got a P1 case from a major service provider, requeued (reassigned) after multiple RMAs.   The customer had a 12000-series GSR, top of the line back then, and was frustrated because ISIS wasn’t working. “We just upgraded the GRP to a PRP to speed the router up,” he said, “but now it’s taking 4 hours for ISIS to converge.  Why did we pay all this money on a new route processor when it just slowed our box way down?!” The GSR router is a chassis-type router, with multiple line cards with ports of different types, a fabric interconnecting them, and a management module (route processor, or RP) acting as the brains of the device.  The original RP was called a GRP, but Cisco had released an improved version called the PRP. The customer seemed to think the new PRP had performance issues,...

Read More

Where I’ve been, and what a TME is

Posted by | Apr 24, 2018 | , |

Jesse, a recent commentor, asked why I haven’t been posting much lately.  In fact, my last post was August of 2017.  Well, there are several reasons I don’t post much these days.  In part, I’m not convinced anyone is reading.  It’s nice to see a comment now and again to realize it’s not just spambots looking at SZ. The other major reason was a job change.  I moved to Cisco over two years ago, and I came in as an individual contributor (IC).  I liked to joke that I had never been so busy since…the last time I worked at Cisco.  However, as an IC, I had no idea how easy I had it. Someone got the crazy idea to make me a manager.  So now, not only do I have the Principal Technical Marketing Engineer title, I also manage a team of 10 TMEs.  The team happens to be driving Software-Defined Access, currently Cisco’s flagship product.  So, the time for blogging is a bit limited.  I’m still working on programmability in my spare time, and I’m continuing to do Cisco Live sessions at least twice a year.  My hair is turning white and I don’t think it’s just my age. That said, I cannot image a better job or place to be than this job at this time.  It’s an exciting company to work for, and an exciting...

Read More

TAC Tales #12: SACK of trouble

Posted by | Aug 26, 2017 | |

When I first started at Cisco TAC, I was assigned to a team that handled only enterprise customers.  One of the first things my boss said to me when I started there was “At Cisco, if you don’t like your boss or your cubicle, wait three months.”  Three months later, they broke the team up and I had a new boss and a new cubicle.  My new team handled routing protocols for both enterprise and service provider customers, and I had a steep learning curve having just barely settled down in the first job. A P1 case came into my queue for a huge cable provider.  Often P1’s are easy, requiring just an RMA, but this one was a mess.  It was a coast-to-coast BGP meltdown for one of the largest service provider networks in the country.  Ugh.  I was on the queue at the wrong time and took the wrong case. The cable company was seeing BGP adjacencies reset across their entire network.  The errors looked like this: Jun 16 13:48:00.313 EST: %BGP-5-ADJCHANGE: neighbor 172.17.249.17 Down BGP Notification sent Jun 16 13:48:00.313 EST: %BGP-3-NOTIFICATION: sent to neighbor 172.17.249.17 3/1 (update malformed) 8 bytes 41A41FFF FFFFFFFF 12345 Jun 16 13:48:00.313 EST: %BGP-5-ADJCHANGE: neighbor 172.17.249.17 Down BGPNotification sent Jun 16 13:48:00.313 EST: %BGP-3-NOTIFICATION: sent to neighbor 172.17.249.173/1 (update malformed) 8 bytes 41A41FFF FFFFFFFF The cause seemed to be malformed BGP packets,...

Read More

In Praise of Vendor Lock-In

Posted by | Jul 17, 2017 | |

There is one really nice thing about having a blog whose readership consists mainly of car insurance spambots:  I don’t have to feel guilty when I don’t post anything for a while.  I had started a series on programmability, but I managed to get sidetracked by the inevitable runup to Cisco Live that consumes Cisco TME’s, and so that thread got a bit neglected. Meanwhile, an old article by the great Ivan Pepelnjak got me out of post-CL recuperation and back onto the blog.  Ivan’s article talks about how vendor lock-in is inevitable.  Thank you, Ivan.  Allow me to go further, and write a paean in praise of vendor lock-in.  Now this might seem predicable given that I work at Cisco, and previously worked at Juniper.  Of course, lock-in is very good for the vendor who gets the lock.  However, I also spent many years in IT, and also worked at a partner, and I can say from experience that I prefer to manage single vendor networks.  At least, as single vendor as is possible in a modern network.  Two stories will help to illustrate this. In my first full-fledged network engineer job, I managed the network for a large metropolitan newspaper (back when such a thing existed.)  The previous network team had installed a bunch of Foundry gear.  They also had a fair amount of Cisco.  It was...

Read More

TAC Tales #11: Full up

Posted by | Mar 2, 2017 | |

No customer is happy if they have to reboot one of their Internet-facing routers periodically, and this was one of our biggest customers.  (At HTTS, they were all big customers.)  This customer had a GSR connecting to the Internet, with partial BGP routes, and he kept getting this error: %RP-3-ENCAP: Failure to allocate encap table entry, exceeded max number of entries, slot 2 1 %RP-3-ENCAP: Failure to allocate encap table entry, exceeded max number of entries, slot 2 Eventually the router would stop passing traffic and when this happened, he had to reload it.  Needless to say, he wasn’t happy. The error came with a traceback, which shows what functions the code was executing when the error was generated.  The last function was this: arp_background(0x5053d290)+0x140 1 arp_background(0x5053d290)+0x140 Well, this was obviously some sort of ARP issue.  But why was ARP causing the router to stop forwarding traffic? Looking up the error, I found that it meant the route processor was unable to allocate a rewrite entry for the slot 2 line card.  As a packet leaves the fabric of a large router like the GSR, the headers are re-written with the destination layer 2 info.  The rewrite table used for this was full.  I had the customer run a hidden command a few times, and we could see the table entries incrementing quickly: Adjacency Table has 3167 adjacencies Adjacency Table...

Read More