Author: ccie14023

TAC Tales #4: Airline Outage

I don’t advertise this blog so I’m always amazed that people even find it. I figured the least-read articles on this blog were my “TAC Tales,” but someone recently commented that they wanted to see more… Well, I’m happy to oblige. The recent events at United reminded me of a case where operations were down for one of the major airlines at Miami International Airport. It didn’t directly impact flight operations, but ticketing and baggage handling systems were down. Naturally, it was a P1 and so I dialed into the conference bridge. This airline had four Cat 6500’s acting as their core devices for the network. The four switches had vastly disparate configurations, both hardware and software. I seem to recall one of them was running a Supe 1 module, which was even old in 2007 when I took the case. There was a different software version on each of them. EIGRP was acting funny. As a TAC engineer in the routing protocols team, I absolutely hated EIGRP. EIGRP Stuck-In-Active was my nightmare case. It was always such a pain to track down the source, and meanwhile you’d have peers resetting all over the place. OSPF doesn’t do that, nor ISIS. I once got in a debate on an internal Cisco alias with some EIGRP guys. Granted, I had insulted their life’s work, but I stated that EIGRP was...

Read More

Hub and Spoke with BGP

In my previous post, we saw the theory behind hub-and-spoke VPN. We saw how H/S involves multiple VRFs with cross-importation between them, and we traced the basic flow of a route advertised from one spoke to another. Next, we are going to look at two options for configuring H/S VPNs. In this post, I will cover using BGP as the PE-CE routing protocol without independent route reflectors. In my next post, I will cover OSPF. Finally, I will return to BGP and examine the issues that come up when we use independent route reflectors with hub and spoke VPN. Review To review, let’s examine our setup. R5 is the hub PE, which has two separate links to the Hub CE, R2. These can be physically separate or different logical units on the same physical interface. In my lab, they are two physical interfaces. We have two spokes in this lab, although there is no reason you can’t use more. Each spoke has a PE and a CE router. Note: This is part of a larger lab, but I have extracted only the relevant part. This is why the router numbering does not start at 1. Also note that in my labs, the last octet of the router’s IP address equals the router number. This should help in decoding the outputs. For my initial configuration, I have MPLS enabled with...

Read More

Hub and Spoke VPN Part 1

One of the JNCIE-SP exam objectives I found difficult was hub and spoke VPN. Conceptually it’s not easy, and as is often the case, the documentation is only somewhat helpful. This series of posts is designed to walk you through the concepts of hub and spoke VPN, as well as its basic configuration using BGP, and then OSPF as the PE-CE protocol. Finally I will talk about route reflector issues when using H/S VPN. It is an important topic to master for your JNCIE exam, and if properly explained you will find it’s not as difficult as it seems. This article assumes you are familiar with basic configuration of Layer 3 MPLS VPN, including vrf import and export policies. Note: I recommend that you only use vrf import/export policies for the JNCIE lab and avoid the vrf-target command for layer 3 VPN. We’ll take a simple example topology with three sites.  Normally, if these sites are configured in a Layer 3 MPLS VPN, site 1 and site 2 are able to talk directly over the MLPS network. That is, the traffic between site 1 and site 2 is never seen by the hub site. This is, of course desirable. Routing the site-to-site traffic through a central site could produce significant delays, especially if the geographic distance between the sites is large. It also would consume bandwidth unnecessarily on the...

Read More

The joy of being an “expert”

Ah, the joys of being an “expert.”  I had forgotten what happens after you pass an exam like the JNCIE.  One of my colleagues starts grilling me on various topics for which I am unprepared, since they weren’t covered on the exam.  MX architecture, MC-LAG, MX virtual chassis, etc.  Be careful what you wish for.  The second you put “expert” on your title some people will take it as a challenge.  Of course, being in a very non-hands-on position, I am not an expert in many of the things day-to-day engineers know quite well.  I certainly know the topics on the test.  Then again, having obtained a CCIE 10 years ago, I know better than to start parading myself around as an expert on anything.  The JNCIE number goes on the blog and the bottom of my LinkedIn, but I am not putting it in my email signature.  Meanwhile, time to start reading Doug Hanks’ book on MX...

Read More

TAC Tales #3: GSR

Shortly after I went to work at TAC, my first team, which was dedicated to enterprise customers, was dissolved, and I ended up on the Routing Protocols team.  The RP team supported both enterprise and service provider customers, and I had zero experience on the SP side.  There was quite a learning curve ahead. One day my phone rang with a P1.  I dreaded P1’s.  When a P1 came in, you were thrown head-first into a potentially huge outage with no knowledge of the case in advance.  Often times a case that had been worked by another engineer got raised to P1 and you had to deal with someone else’s mess. On this particular day, the case was a line card problem on a 12000-series, or GSR.  GSR was a service provider box and I knew nothing about it.  I didn’t even think it ran IOS (it does).  I had no idea where to start.  You would think they would have given me training on every product we covered, but at HTTS, at least when I worked there, the general approach was to throw you to the wolves. So, I politely put the customer on hold and started running around the second floor of building K where HTTS is located, shouting:  “Does anyone know GSR?  Anybody around to help me?!”  Finally I stumbled across my teammate Abe in the break room...

Read More