I recently relocated to the US from France/Switzerland and I have been so busy the past 2 years working on that process. Yes, It is that long! 

I have been asked about career advice twice this week and I wanted to share my thoughts about it.

Networking in 2008

I think we all agree on the fact that the networking field has been very static for the past 15 years. One of the ways to provide a better network experience to the users/applications was to add more bandwidth (or invest in WAN optimization). OSPF/BGP/EIRGRP/MPLS and spanning tree haven’t changed much since 2002 right?

 
All the networking manufacturers paradigm was all about releasing new hardware that could provide more bandwidth and availability. As an engineer, you had to know networking protocols but we also had to understand specifics of networking hardware. It was very useful to understand how the 6500 Crossbar was switching packets internally. Another example was the StackWise technology: who remembers that the 3750 v2 could not locally switch without sending packets on the ring?.

Every device had a specific function in the network for example (which is still true at some point). Engineers were doing was vendors told them to do and they had to standardize their deployment (Access – Distribution – Core). It was a safe bet to design to design a network using the 3 tiers architecture mentioned previously.

 

Some networking engineers are self-educated up to a certain point and one of the ways to learn networking back in the days was to read a Cisco Press book, buy some hardware (2950 – 3600) on eBay and do some labs on your own or using a third party training company. For these engineers, the way to get a job was to climb the traditional certification pyramid (CCENT – CCNA – CCNP – CCIE). While this is still kinda relevant, the CCIE does not automatically open doors for any jobs anymore. Matt Oswalt published a quote that makes total sense “vendor certs are basically a way of putting the vendor in control of your career. On the other hand, fundamental knowledge puts YOU in control”. 

I have a dual CCIE and studied very hard to get where I am today but the journey is far from being over (hopefully). I need to be a little less focused on proprietary certification and get some open source knowledge as well. (Damn CCDE you are tempting but I need to resist !)

Linux/Python skills were definitely not mandatory in any of the job descriptions back in the days. But as you can guess it becomes more and more a requirement nowaday.

I’ve been invited to a very interesting dinner with CIOs of Fortune 100 companies recently. They are all aware of the ongoing networking transition. They admitted it was not an easy plan to embrace this evolution but they are already preparing their teams for that.  

Speaking of technologies, which technologies are we talking about? Do we need to know everything in IT? the answer is obviously “No” but it is valuable to at least understand how all the systems are interconnecting to each other.

Here is what a job description looked like back in the days (2008):

 

The need for evolution

I am doing this blog post is because our field is changing and our skills need to evolve with the networking trends. Engineers are the core of the networking industry. We all have a critical function in every organization that is willing to undertake their “business digital” transformation. We need to prepare how to evolve with the upcoming technologies.
I am willing to create a blog post series on how to tackle your own networking evolution. Please do not get me wrong, we still need to understand bits and bytes of all the networking protocols in order to provide connectivity. This statement will never go away (hopefully) and there is no working overlay if the underlay as been designed carefully. What needs to evolve is the way we are able to provision services for our customers/users/applications. When was the last time you heard that the networking team was taking too long to provide connectivity between A and B? 

 

Networking in 2014+

Short story long, network engineers have to stay relevant throughout the years. 

Today it would be a bit different, it is definitely expected to know everything that is above right  (except maybe Cisco Works and CatOS 🙂 )

Himawan Nugroho made a great Cisco Live presentation that I attended in Milan: BRKSDN-4005 – CCIE Skill transformation to SDN kungfu. The most interesting slide for me is the following one: 

 

He confirmed what I was explaining above. You still have to be an expert at traditional routing/switching but also have a broader knowledge of the following technologies:  Linux and Operating Systems, Scripting, Overlays (proprietary and standards) and network virtualization. 

Some new protocols and ways to provide network connectivity have recently emerged. Some of them are already dead (Trill anyone ?) and other are being used worldwide in different flavors (VXLAN anyone ?). 

We see plenty of blog post related to the eternal question: Should we learn how to script/code:

• Jason Edelman
• Matt Oswalt
• Kirk Byers
• Tony Mattke

My take on this is that you should be able to automate your network and most of your tasks. You should not consider going too deep (for now). We are not required to become a full-time developer.

Some of the following items you will find on this list are not necessarily new but it is something that the network engineers can’t avoid to be aware of anymore. This is by no means an exhaustive list but it gives you an indication of what the current trends are in our industry. Feel free to drop a comment if you think something valuable should be added.

.

Acquiring all of these skills do not happen overnight so I will publish quite a few blog posts about how I am preparing my own evolution. Let me know in the comments below what you liked, disliked or if you have any question.

Nic

 

One of my customer told me that one of its end user was not appearing in its CUCM database. I think it is worth to make a blogpost about it.

There are already plenty of resources on the subject (Example) but I will mainly focus on the troubleshooting section here.

There are 2 ways to configure your users on a Cisco CUCM, you can either configure them statically or you can synchronise your CUCM with your Active Directory Domain.

If you want to make sure that your AD – CUCM synchronization is working, you first need to check that the DirSync Service is activated on the CUCM Publisher:

 

Then you need to select which AD Attributes will be used as the USERID field within the CUCM. The best logical choice to me is the sAMAccountName since it will be used by the users to authenticate themselves. (Browse to System => LDAP => LDAP System)

 

Now you need the following:

• a Service Account to browse through the Active Directory Domain
• Search base : Where the CUCM will sync all the OU that are located at under the Search Base OU.
• LDAP Server Information.

 

I would advise you to use the AD “mail” field as the Directory URI CUCM field. This will be used by as a SIP URI that will be linked to the user extension. You can change this on the go if you are running 10.5.2. Otherwise you have to create a duplicate LDAP sync and then remove the old and obsolete one.

Now I can check if my users are created but I noticed that indeed, one of them was missing and I would like to understand (and fix) why !

 

If I want to troubleshoot this, I need to activate the debugs into the CUCM serviceability menu.

 

From here you have 2 options : RTMT or CLI.

I’m not an RTMT fan so I will show you the CLI way to find DirSync logs into CUCM.

The logs are located into /cm/trace/dirsync/log4j

Now you have  to search for “roy” into your debugs to find why the user is being rejected:

You can see that there is another EndUser that has the same MailID field so the CUCM does reject the synchronisation for the real user named “Roy”.

There are many other reason why the CUCM can reject a user from the synchronisation process. The most common one is when you do not enter one of the mandatory field: FirstName of LastName.

Also I recommend to read some really important LDAP Design considerations from the Cisco 10.x SRND Design Document

• Ensure that the LDAP directory attribute chosen to map into the Unified CM UserID field is unique within all synchronization agreements for that cluster.
• The LDAP attribute sn(lastname) is a mandatory attribute for LDAP Synchronization of users.
• The LDAP attribute sn(lastname) is a mandatory attribute for LDAP Synchronization of users.
• The attribute chosen as UserID must not be the same as that for any of the Application Users defined in Unified CM.

 

If you have any comments or questions, do not hesitate to post a comment.

Nicolas

In fact the title should be “My CCIE Journey – Act III” but I don’t want to use that one because I had a bad experience with the CCIE Voice lab exam 🙂

There are many (very good) links about that specific subject but I wanted to give my own opinion as well :). Here is a list (incomplete for sure) of the people that have blogged about their CCIE DC lab experience :

• Brian McGahan
• Jason Lunde
• David Varnum
• Peter Revill 
• Jeremy Waldrop
• Plowking
• Vallard
• Joel Sprague

I have shared my journey towards the CCIE RS in 2011 and I wanted to share it again with you. I passed the CCIE DC lab exam one month ago and it was tough, long, hard,arduous, baffling, difficult, exacting, exhausting, hard (yeah I already used it on purpose 🙂 ), intractable,perplexing, puzzling, strenuous, thorny, troublesome, uphill.

As soon as I failed my CCIE Voice exam, my frustration went so high and I needed a break from the Voice exam a little bit. The Data Center exams were released by Cisco and I always wanted to be involved in a Data Center infrastructure project. I immediately decided to jump into the DC field and start to climb the (infinite) ladder.

At this time my DC infrastructure background wasn’t enough to pass the CCIE DC Written, I decided to spend a year reading books and solidify my knowledge.

First and foremost the CCIE DC blueprint is like any CCIE DC, it is VERY large. As an expert that will face customers and other experts, you definitely have to dig very deep to understand what’s going on in every section of your infrastructure (Compute / Storage / Infrastructure).

In my previous CCIE Journey post I used this expression from Brian McGahan: “a CCIE journey is not a short race, it is a marathon”. 4 years after, this applies even greater today. If you have a family, you better have to have a very supportive wife/husband. My wife is the most supportive person I’ve ever met.

We had our 3rd baby 10 months ago and my daughter couldn’t sleep at night. My wife was taking care of all 3 children 24/7 while I was studying. She even stayed at my parents home for several weeks to make my study time more efficient. After all, I can say that we are both CCIE RS-DC right now :).She deserves the title as much as I do … I am pretty sure that the CCIE exam is easier than taking care of the children. What I am trying to say here, is that you have to be dedicated to this exam.

CCIE Written Preparation

I already mentioned before but I read LOTS and LOTS of books. I will give you my list very soon but first I would like to start with one of the best technical book I have read in my entire career.

Data Center Virtualization Fundamentals  written by Gustavo Santana is definitely the best Data Center book out there. If you have some Routing and Switching Skills, you probably read the very famous Routing TCP/IP Books (Volume 1 covers IGP and Volume 2 covers BGP,Multicast and IPv6). All I can say is that Santana is as awesome as Doyle. I don’t want to overemphasize but I really enjoyed every words of the book.

 

The others books are the following:

• Cisco UCS (a bit outdated but still nice to understand)

• Implementing Cisco UCS Solution 

 

• I/O Consolidation in the Data Center. (Very good regarding FCOE)

• NX-OS and Cisco Nexus Switching (Awesome Job by the authors ! Very nicely written)

 

• Using TRILL – Fabricpath – VXLAN

 

I also read some free ebooks written by EMC and IBM. To me these 2 books regarding Storage Area Networks are great free resources:

• Networked Storage Concept and Protocols
• Introduction to SAN and System Networking

I was almost ready to sit the CCIE DC Written exam but I decided to solidify all the theory I have gained throughout the year. In order to do that I gave a look at CCIE Training vendors.

I have a very good experience with all the main vendors and this is probably the most frequently asked question so far : “Which vendor did you use for your preparation”

First I never really picked up a vendor. I tend to prefer to choose an instructor. I went with INE and Micronics Training for my CCIE RS because I heard from close friends that Brian McGahan and Narbik were top notch instructors (and they are). For my voice studies, I went with IPX because Vik Malhi is the best Voice Trainer I’ve ever met (Since that time, Vik has its own training company CollabCert, you should definitely give it a try if you are interested in collaboration). So in my opinion, students should not pick a vendor, they should pick an instructor and an instructor that meets your personal requirements. Maybe McGahan, Kocharian and Malhi are not the best for you but I can tell you from my personal experience that they are the best for me.

Choose wisely ! A training vendor business is to make your studies time efficient.

I bought an All Access Path from INE and decided to enroll myself into the CCIE Data Center Written Bootcamp. If you want to have a look of the teaching style:

 The INE videos are matching all the blueprint : Nexus / Storage / UCS.

There is another useful (free) resource available for you guys: Cisco Live Portal. This place is the place to watch deep dive videos regarding every Cisco topic!  For the DC stuff there are many listed by Brian McGahan on its “how to pass the CCIE DC” blogpost.

I passed my CCIE DC written exam on my second try. It was a really tough exam …

In order to track my studies during the journey, I have used trello and I love this app. Here is an example of how I managed my tasks

CCIE LAB Preparation

The lab is a complete different story and I didn’t really relied on any vendors regarding the workbooks. I used INE and IPX for my online bootcamp but I will cover that later.

So regarding the workbooks, I didn’t really use any of them … I just did a few lab here and here from both vendor but I didn’t really like it. I just wanted to read the config guide, build the infrastructure and then run every show command I could.

For CCIE RS and Collaboration, it is very easy to host a rack in your home or at work. For the DC track, things can get more tricky since you will need a N7K (with VDCs you slice your switch into multiple virtual switches, don’t worry it is part of the blueprint 🙂 ), 2x N5K ,2x Nexus 2232 PP (in order to run FCoE), 2x MDS (9222 is my choice) and a small JBOD (I will make a separate post to show you how to build the cheapest JBOD ever 🙂 ).

INE and IPX racks can be very busy if you want to book the racks with UCS … I also recommend to use the Cisco UCS Platform Emulator on your own laptop (run on ESXi as well if you have a virtualization lab). You can do almost everything with it (except booting your favorite Operation System / Hypervisor).

My local Cisco SE (Vincent, thank you so much !) was kind enough to let me borrow 2x N5K with some FEX and  2x MDS 9222i. I have built a cheap jbod and I could test 100% of the storage feature for the lab exam.

I think the most valuable resources to practice is the Cisco Partner Education Collection .

There are so many labs and hardware there (sometimes fully booked of course) than you can spend countless hours of labs … Joel Sprague (which is an MVE [Most Valuable Engineer] I met during my studies) did a very good job by posting all the valuables labs that you can do with the Cisco PEC. I didn’t do ALL of them but the vPC / Fabricpath / UCS / N1000v are definitely mandatory … The UCS is one of the best because you can boot from SAN and the UCS is yours for 8 hours and for free.. Nothing can beat that !

 

Even if you are studying for the CCIE LAB exam and that you know that you are going to spend 8 tough hours configuring weird things, you still need to read a lot in order to configure your infrastructure.

I would recommend to read almost all the configuration guides related to the blueprint for the Nexus. For UCS and MDS, You can periodically check but there is no need to read everything like you should do for the nexus part.

I have watched both INE and IPX videos regarding the CCIE lab exam, McGahan and Rick Mur videos are perfect ! McGahan for INE was in charge of storage and Nexus while Snow was in charge for UCS.

I also attended 2 CCIE online bootcamp from INE (McGahan Again) and IPX with Jason Lunde. Both did a great job.

McGahan is definitely the big player here, his complete set of videos (Nexus – Storage – Lab Cram Session) are simply awesome. It covers way more than you need for the CCIE DC exam

Here is a preview of its DC lab cram session:

There are plenty of nice other resources that other CCIE DC have published on their own blog. Here is the 3 I used during my studies:

• Peter Revill CCIE Rants (Cover most of the blueprint)
• David Varnum Overlaid.net (Nice FabricPath blogpost !)
• Colin Lynch UCSGuru.com (Very nice FCoE blogpost !)

CCIE LAB Exam

I decided to book the CCIE the day before my vacations started because I didn’t want to go in vacations with the CCIE still in mind 🙂

So I went to Brussels on July 10th and I was very pleased by the proctor (if you read me, I would like to thank you. The experience was great). The exam is fair, it is hard but fair. There are no second guess like I had in voice. Questions were very precises and if I didn’t understand everything in the question, the task title made me clicked in my head : “Gotcha”.

You have to CAREFULLY read the tasks. If Cisco is asking for an ACL named MYCCIEDCLAB, you will not get the point if you configure it MYCCIEDCLAb. Even if your configuration is correct, they will look for the right naming convention. If you want to prevent all sorts of easy mistakes, your best weapon is the CTRL+C , CTRL+V. I can tell you this is the best thing you will ever need in the lab. Notepad is so useful as well !

During your daily job you would still do it right ? What if you want to configure vlan 100,200,300,400,500,600 in all your devices (let’s assume VTP is bad … wait a minute … it is bad .. in my opinion 🙂 ) You would open a notepad, type your commands , and paste into all devices right ?

My advice is to do the same for your CCIE Labs.

As Brian McGahan said, I did my happy dance when you see the UCS-B series booting ESXi 🙂

I finished the lab with 1 hour left. Now the critical thing to do was to stay there and look for small mistakes I could have make during these very long 8 hours. I found some and for every tasks I checked that what I did was still working and that 100% of the requirements were met.

Finally I left the building and asked the proctor when can I expect the results to be delivered. He told me : “within few hours” . I thought he was making fun of me but he was right.

I went to the airport to meet a friend from Belgium and I received the score report notification.

Was thrilled to see the results : “PASS”

The exam can be tought but again it is doable. During my studies I have met a much better DC engineer than me, he failed the exam twice 🙁 . So please be sure to read slowly and try to understand what they really want…

So what’s up to me now that I am a double CCIE. In the beginning of the post I said that I started to climb the infinite ladder, what does that really mean ? It doesn’t mean that now that I am a CCIE, I can rest and that I can live like that and that my knowledge will stay at the same level through my career. People who think they are done with learning  are wrong.

Knowledge has to be sustained ! I still have to work on every protocol if I want my knowledge to be intact. I also have to learn new emerging technologies like Dev-Ops (not new but still new to me) / ACI / NSX etc etc in order to become a better engineer !

I hope you enjoyed the blogpost and in the meantime, if you have some questions, you can leave a comment below.

 

Nicolas

Hey everyone.

 

I have just finished my upgrade to CUCM 10.5.2 and I faced an issue at the end of the ugprade.

Of course this always happen after you spent some hours waiting for the upgrade to be successful 🙂

According to the very good Cisco DocWiki, VMware Tools are specialized drivers for virtual hardware that is installed in the UC applications when they are running virtualized. It is very important that the VMware tools version running in the UC application be in sync with the version of ESXi being used.

This clearly states that it is kinda mandatory for you to install or ugprade your VMware tools after every upgrade.

Depending of the version and application you are running, there are several methods to upgrade your VMware tools:

• Method 1: Using a COP File: This is deprecated and used only for 8.0 UC servers. Definitely not something you will deal with if you are deploying a new UC infrastructure
• Method 2: Using the CLI: This method will be used if you run UCCX 8.5(1)+ or 8.5 UC Servers or CUCM IM and P version 9. This is disruptive and the server will reboot twice.
• Method 3: Upgrade from VI Client. Very easy method. Will install while the server is in production and it is not disruptive at all. 
• Method 4: Auto Upgrade during a server power cycle. Not disruptive at all and will auto upgrade.

Please bear in mind that most of the recent UC applications are compatible with either Method 3 and 4. I generally like to enable auto upgrade.

So let’s get back to our issue.

After I completed my upgrade, I went to the vCenter client to install the VMware tools but I faced a problem. It was just not working. I even rebooted my server twice but still nothing .. I still had that same issue (image below is just representing a similar issue)

 

I gave a look at the excellent support community and bug search tool… and I was lucky enough to find that I was not the only one facing that issue 🙂

Indeed there is a recent bug hitting CUCM 10.5 (and other apps I believe). CSCul78735

SElinux is preventing the VMware tools to be installed on the server. It is regulating access control security policies to the server and has been introduced in CUCM 8.6 (in fact it replaced Cisco Security Agent).

What we need to do is to bypass SElinux policies and here is
the (very complicated) procedure:

As soon as we do that, you need to install again the VMware tools and it worked for me.

 

Do not forget to enforce SElinux after you are done with the VMware tools installation.

Hope it can help you in your future upgrades 🙂

Cheers

Nicolas

Usually people are blogging on a certain topic because they want to share they knowledge with a certain protocol or product.

Today I ll take another approach with that fact and I will actually do the exact opposite. I have an issue with ISLB which allows load balancing for my iSCSI sessions. Today I will elaborate each steps needed to make it work. I have failed this configuration a LOT of time and I have followed the same steps over and over. I decided to make a blogpost about it to keep track of what I should do next time I want to configure it.

I did not play with VRRP yet but this can be an idea for a following blogpost.

The topology is the same as in my previous blog posts related to the MDS.

 

 

The difference here is that both MDS will have an iSCSI interface bound to their gigabit interface. (iscsi 1/1 mapped to gig 1/1).

 

I will start from scratch and setup the infrastructure:

The outpout above prove us that the JBOD has registered to the fabric and that VSAN 10 is running on the E port between MDS01 and MDS02. Another proof is that the FCNS commands on MDS02 has the JBOD PWWN in its database.

Now we will setup Device-alias, we will activate a test zoneset on vsan 10 because ISLB requires an already active zoneset if you want to use the auto zone feature. If you do NOT have an active zone, you will have to manually perform the zoning configuration.

 

Now we can start our ISLB configuration. Again we will first configure the infrastructure and check that both iSCSI interfaces are reachable from the L2 domain.

 

ISLB configuration can now start and you will see it is very brief:

We first need to check the IQN of our servers.

\

 

The configuration has been commited and MDS02 should have the ISLB configuration and the zoning configured on it:

All is all right here and none of the iSCSI initiator have yet logged in the fabric:

Let’s now activate debugs on both switches and try to initiate a Fabric Login from the iSCSI initiators (Server 01 first then Server 02)

 

MDS01 has performed a FLOGI onto itself on the VSAN10 and it has been mapped to interface iSCSI 1/1.

We can also see that the initiator has been correctly mapped to the JBOD

Let’s now try with server 02

Note that the MDS02 will only see 1 FLOGI and that MDS01 will see both FLOGI from its local FC Disk and from its iSCSI Initiator.

Both servers are able to map the drive and everybody is happy 🙂

Nicolas

As I mentionned at the beginning of the post, I did not played with VRRP on purpose and I will relate about that in a following blogpost 🙂