Author: ccie14023

TAC Tales #18: All at once

Posted by | Jul 5, 2019 | |

The case came into the routing protocols queue, even though it was simply a line card crash.  The RP queue in HTTS was the dumping ground for anything that did not fit into one of the few other specialized queues we had.  A large US service provider had a Packet over SONET (PoS) line card on a GSR 12000-series router crashing over and over again. Problem Details: 8 Port ISE Packet Over SONET card continually crashing due to SLOT 2:Aug 3 03:58:31: %EE48-3-ALPHAERR: TX ALPHA: error: cpu int 1 mask 277FFFFF SLOT 2:Aug 3 03:58:31: %EE48-4-GULF_TX_SRAM_ERROR: ASIC GULF: TX bad packet header detected. Details=0x4000 1234 Problem Details: 8 Port ISE Packet Over SONET card continually crashing due to SLOT 2:Aug  3 03:58:31: %EE48-3-ALPHAERR: TX ALPHA: error: cpu int 1 mask 277FFFFFSLOT 2:Aug  3 03:58:31: %EE48-4-GULF_TX_SRAM_ERROR: ASIC GULF: TX bad packet header detected. Details=0x4000 A previous engineer had the case, and he did what a lot of TAC engineers do when faced with an inexplicable problem:  he RMA’d the line card.  As I have said before, RMA is the default option for many TAC engineers, and it’s not a bad one.  Hardware errors are frequent and replacing hardware often is a quick route to solving the problem.  Unfortunately the RMA did not fix the problem, the case got requeued to another engineer, and he…RMA’d the line card.  Again.  When that didn’t work, he...

Read More

Interviewing #2: Why do we interview?

Posted by | Jun 27, 2019 | |

In the last article on technical interviewing, I told the story of how I got my first networking job.  The interview was chaotic and unorganized, and resulted in me getting the job and being quite successful.  In this post, I’d like to start with a very basic question:  Why is it that we interview job candidates in first place? This may seem like an obvious question, but if you think about it face-to-face interviewing is not necessarily the best way to assess a candidate for a networking position.  To evaluate their technical credentials, why don’t we administer a test? Or, force network engineering candidates to configure a small network? (Some places do!)  What exactly is it that we hope to achieve by sitting down for an hour and talking to this person face-to-face? Interviewing is fundamentally a subjective process.  Even when an interviewer attempts to bring objectivity to the interview by, say, asking right/wrong questions, interviews are just not structured as objective tests.  The interviewer feedback is usually derived from gut reactions and feelings as much as it is from any objective criteria.  The interviewer has a narrow window into the candidate’s personality and achievements, and frequently an interviewer will make an incorrect assessment in either direction: By turning down a candidate who is qualified for the job.  When I worked at TAC, I remember declining a candidate who...

Read More

TAC Tales #17: Escalations

Posted by | Jun 18, 2019 | |

When you open a TAC case, how exactly does the customer support engineer (CSE) figure out how to solve the case?  After all, CSEs are not super-human.  Just like any engineer, in TAC you have a range of brilliant to not-so-brilliant, and everything in between.  Let me give an example:  I worked at HTTS, or high-touch TAC, serving customers who paid a premium for higher levels of support.  When a top engineer at AT&T or Verizon opened a case, how was it that I, who had never worked professionally in a service provider environment, was able to help them at all?  Usually when those guys opened a case, it was something quite complex and not a misconfigured route map! TAC CSEs have an arsenal of tools at their disposal that customers, and even partners, do not.  One of the most powerful is well known to anyone who has ever worked in TAC:  Topic.  Topic is an internal search engine.  It can do more now, but at the time I was in TAC, Topic could search bugs, TAC cases, and internal mailers.  If you had a weird error message or were seeing inexplicable behavior, popping the message or symptoms into Topic frequently resulted in a bug.  Failing that, it might pull up another TAC case, which would show the best troubleshooting steps to take. Topic also searches internal mailers, the...

Read More

CCIE Enterprise Infrastructure

Posted by | Jun 14, 2019 | |

There were quite a few big announcements at Cisco Live this year.  One of the big ones was the overhaul of the certification program.  A number of new certifications were introduced (such as the DevNet CCNA/CCNP), and the existing ones were overhauled.  I wanted to do a post about this because I was involved with the certification program for quite a while on launching these.  I’m posting this on my personal blog, so my thoughts here are, of course, personal and not official. First, the history.  Back when I was at Juniper, I had the opportunity to write questions for the service provider written exams.  It was a great experience, and I got thorough training from the cert program on how to properly write exam questions.  I don’t really remember how I got invited to do it, but it was a good opportunity, as a certified (certifiable?) individual, to give back to the program.  When I came to Cisco, I quickly connected with the cert program here, offering my services as a question writer. I had the training from Juniper, and was an active CCIE working on programmability.  It was a perfect fit, and a nice chance to recertify without taking the test, as writing/reviewing questions gets your CCIE renewed. As I was managing a team within the business unit that was working on Software-Defined Access and programmability, it...

Read More

Cisco Live is over! Long Live Cisco Live!

Posted by | Jun 13, 2019 | |

I think it’s fair to say that all technical marketing engineers are excited for Cisco Live, and happy when it’s over.  Cisco Live is always a lot of fun–I heard one person say “it’s like a family reunion except I like everyone!”  It’s a great chance to see a lot of folks you don’t get to see very often, to discuss technology that you’re passionate about with other like minded people, to see and learn new things, and, for us TMEs, an opportunity to get up in front of a room full of hundreds of people and teach them something.  We all now wait anxiously for our scores, which are used to judge how well we did, and even whether we get invited back. It always amazes me that it comes together at all.  In my last post, I mentioned all the work we do to pull together our sessions.  A lot of my TMEs did not do sessions, instead spending their Cisco Live on their feet at demo booths.  I’m also always amazed that World of Solutions comes together at all.  Here is a shot of what it looked like at 5:30 PM the night before it opened (at 10 AM.)  How the staff managed to clear out the garbage and get the booths together in that time I can’t imagine, but they did. My boss, Carl Solder,...

Read More

Recent Comments