Author: ccie14023

Cleared for nothing

Posted by | Feb 20, 2020 | |

I was working at Juniper when the CIO asked me to apply for a government security clearance.  There were a number of hacking attempts on our network, and a security clearance would make me eligible for briefings from the government on the nature and scope of the threats against the United States’ networks.  Being one of the few US citizens in our department, and having a security background, it made sense. I met with our “FSO”, the on-site liaison to the clearance-granting agency, in this case the Department of Defense.  I’ll call him Billy.  Billy pointed me to the government web site which housed the application, called “OPM”.  The OPM application was extensive, requiring me to input huge amounts of information about myself and my family.  It required a bit of work to track down some of the information, and when I printed the PDF copy of the application it totaled around eighty pages. One day Billy called me into his office and told me I had been awarded a secret clearance.  He let me know that I could be subject to the death penalty if I divulged any classified information.  I signed some documents, and that was it. “Don’t I get a card for my wallet or anything?” I asked Billy.  He just smiled. Shortly after getting my clearance, one of our other cleared employees brought me into...

Read More

TAC Tales #19: Butt-in-Chair

Posted by | Dec 17, 2019 | |

This one falls into the category of, “I probably shouldn’t post this, especially now that I’m at Cisco again,” but what the heck. I’ve often mentioned, in this series, the different practices of “backbone TAC” (or WW-TAC) and High Touch Technical Support (HTTS), the group I was a part of.  WW-TAC was the larger TAC organization, where the vast majority of the cases landed.  HTTS was (and still is) a specialized TAC group dedicated to Cisco’s biggest customers, who generally pay for the additional service.  HTTS was supposed to provide a deeper knowledge of the specifics of customer networks and practices, but generally worked the same as TAC.  We had our own queues, and when a high-touch customer would open a case, Cisco’s entitlement tool would automatically route their case to HTTS based on the contract number. Unlike WW-TAC, HTTS did not use the “follow the sun” model.  This meant that regular TAC cases would be picked up by a region where it was currently daytime, and when a TAC agent’s shift ended, they would find another agent in the next timezone over to pick up a live (P1/P2) case.  At HTTS, we had US-based employees only, at the time, and they had to work P1/P2 cases to resolution.  This meant if your shift ended at 6pm, and a P1 case came in at 5:55, you might be stuck...

Read More

Where does time go?

Posted by | Dec 9, 2019 | |

Two things can almost go without saying: If you start a blog, you need to commit time to writing it. When you move up in the corporate world, time becomes a precious commodity. When I started this blog several years ago, I was a network architect at Juniper with a fair amount of time on my hands.  Then I came to Cisco as a Principal TME, with a lot less time on my hands.  Then I took over a team of TMEs.  And now I have nearly 40 people reporting to me, and responsibility for technical marketing for Cisco’s entire enterprise software portfolio.  That includes ISE, Cisco DNA Center, SD-Access, SD-WAN (Viptela), and more.  With that kind of responsibility and that many people depending on me, writing TAC Tales becomes a lower priority. In addition, when you advance in the corporate hierarchy, expressing your opinions freely becomes more dangerous.  What if I say something I shouldn’t?  Or, do I really want to bare my soul on a blog when an employee is reading it?  Might they be offended, or afraid I would post something about them?  Such concerns don’t exist when you’re an individual contributor, even at the director level, which I was. I can take some comfort in the fact that this blog is not widely read.  The handful of people who stumble across it probably will not...

Read More

Before the Internet: The Bulletin Board System

Posted by | Aug 15, 2019 | |

It’s inevitable, as we get older, that we look back on the past with a certain nostalgia.  That said, I think that computing in the era when I was growing up, the 1980’s, was more fun and interesting than it is now.  Personal computers were starting to become common, but were not omnipresent as they are now.  They were quite mysterious boxes.  An error might throw you into a screen that started displaying hexadecimal with no apparent meaning.  Each piece of software had its own interface which you had to learn, since there were really no set standards.  For some, there might be a menu-driven interface.  For others there might be control keys you used to navigate.  Some programs required text commands.  Even working with devices that had only 64 Kilobytes of memory, there was always a sense of adventure. I got my start in network engineering in high school, in fact.  Computer networks as we understand them today didn’t really exist back then, in the 1980’s, except in some universities and the Defense Department.  Still, we found ways to connect computers together and get them to communicate,  the most common of which was the Bulletin Board System, or BBS. The BBS was an individual computer equipped with a modem, into which other computer users could dial.  For those who aren’t familiar with the concept of a modem, this was...

Read More

TAC Tales #18: All at once

Posted by | Jul 5, 2019 | |

The case came into the routing protocols queue, even though it was simply a line card crash.  The RP queue in HTTS was the dumping ground for anything that did not fit into one of the few other specialized queues we had.  A large US service provider had a Packet over SONET (PoS) line card on a GSR 12000-series router crashing over and over again. Problem Details: 8 Port ISE Packet Over SONET card continually crashing due to SLOT 2:Aug 3 03:58:31: %EE48-3-ALPHAERR: TX ALPHA: error: cpu int 1 mask 277FFFFF SLOT 2:Aug 3 03:58:31: %EE48-4-GULF_TX_SRAM_ERROR: ASIC GULF: TX bad packet header detected. Details=0x4000 1234 Problem Details: 8 Port ISE Packet Over SONET card continually crashing due to SLOT 2:Aug  3 03:58:31: %EE48-3-ALPHAERR: TX ALPHA: error: cpu int 1 mask 277FFFFFSLOT 2:Aug  3 03:58:31: %EE48-4-GULF_TX_SRAM_ERROR: ASIC GULF: TX bad packet header detected. Details=0x4000 A previous engineer had the case, and he did what a lot of TAC engineers do when faced with an inexplicable problem:  he RMA’d the line card.  As I have said before, RMA is the default option for many TAC engineers, and it’s not a bad one.  Hardware errors are frequent and replacing hardware often is a quick route to solving the problem.  Unfortunately the RMA did not fix the problem, the case got requeued to another engineer, and he…RMA’d the line card.  Again.  When that didn’t work, he...

Read More