Cox's Hart on the Importance of DevOps

Whether it's moving a maintenance center from one location to another, or implementing Comcast's X1 technology at a faster rate, Cox Communications is thoroughly invested in DevOps principles.

Cox Communications Executive Vice President and Chief Technology Officer Kevin Hart started working on Cox's transformation path shortly after he joined the company five years ago.

"We had to put in some building blocks, some blocking and tackling, development methodologies, checks and balances on testing and architecture roadmaps, but now we're in that phase where we're migrating into the next-gen technology operating model leveraging DevOps and service ownership," Hart said. "So we've had this multi-year journey and development and operational improvements have been key components of our increased velocity and for improving quality, and a little bit of cost efficiency as well. It's pretty exciting."

Hart is the latest service provider executive to speak with Telco Transformation about DevOps, all of which feature the same set of questions.

Telco Transformation: What cultural transformations need to take place in order to implement a DevOps mindset across the entire workforce?

Kevin Hart: The mental model and the mind shift need to take place first. While that sounds simple, when you start to think about bringing together solutions that have a multitude of components with different functions, and different technology platforms, thinking about how they operate from an end-to-end perspective is first and foremost.

The next step is establishing a team that was historically disparate in terms of architecture, development, engineering, testing and operations into a single team with a single leadership structure. For example, we've done a couple of things over the last couple of years on our Cox.com platform where we had a multitude of different development teams that we combined into a single team. On top of that we comingled and collocated the technology team with our product and business stakeholder teams. So we were using DevOps in real-time with collocation, single leadership and a single vision. That has increased our speed to market, but it was really about creating the mindset first.

There are things you have to do around objectives, goals and metrics. You need to make sure those are aligned so that people are working in the same direction, focused on real time, focused on leveraging agile methodologies, sprints and increasing the speed to market.

The other benefit is having the different functional teams experience the challenges first hand from the other point of view. Traditionally a dev team handed off to an ops team. The ops team would take hours, days and sometimes weeks to triangulate on root cause issues depending on prolonged problems. Now those conversations are happening in the moment and in the actual development cycle. Now it's seamless in terms of changes for when we put a service into production and in terms of the support we need to make sure that service operates end-to-end.

We've also done that with our video products over the last couple of years. We used to do two to three releases a year based on a waterfall gate methodology. With DevOps we do literally hundreds of releases a year just by having those teams collocated, putting in 24/7 enhancements and functionality in production, and then mitigating any kind of negative production impact because of the collocation and communications between the teams.

It's a mindset and it takes time, particularly with where we've come from as an industry, but it has taken root now. We have DevOps across all of our key platforms; Cox.com, our end-to-end video components and our call center platform. I did a major reorganization earlier this year to combine a true DevOps team. I took the application development team, the IT operations team and the testing team and brought them under one single structure. Instead of end-to-end service owners for each of our key platforms -- voice, video, data, etc. -- we have a single team under the DevOps structure.

TT: How are employees being trained for new services and applications?

KH: We're really trying to standardize on a lot of our tools. A lot of open source components are definitely a big part of the solution, if you will. We've set up a couple of separate efforts to stand up infrastructure-as-a-service, platform-as-a-service and now we have common development ops ownership. There's a lot of training around those particular tools and platforms. We've got multi-quarter efforts where we can turn up some of these environments basically with the flip of a switch to do more real-time operations and support. Driving common, standard platforms is really a key.

We used to hire on multiple technologies, but now with the talent that is coming in with the open source (OpenStack) platforms we're able to do some on the job training and learning. They can leverage the skill sets from the open platforms and also apply that to multiple different services whether its product related, whether it's provisioning, or whether its some of our e-commerce or care tools.

So it's a combination of formal training aligning around some of the open source platforms, and then also on the job training with developers learning operational skills or vice a versa based on the nature of the way that we structure the teams.

JBoss is something we use heavily within our Cox video middleware, which is kind of the brain or orchestration layer of all that we do on the video platforms. Frankly, JBoss has enabled us to more rapidly integrate. It's integrating the X1 platform into our ecosystem for what we're calling Contour 2, and we're having great success with that platform. The DevOps work we've done around JBoss and Cox middleware was a key stepping stone to allow us to do that (integrate X1) so quickly and continue to operate that so efficiently.

TT: How are new employees being recruited?

KH: We're very lucky and fortunate to have an outstanding existing team. There's definitely a shift, even on the network side, to more software-defined solutions. We've got a lot of internal training in order to make the shift both on the IT side but also from the network side around software-as-a-service and software defined networking. We have a bunch of training going on internally.

We also have a very robust college recruitment program, which includes a co-op program and a summer internship program. In fact I just did a two day co-op and workshop a couple of weeks ago. We had 30 or so co-ops that were showcasing their new solutions that they are helping us bring to market leveraging their DevOps skill sets and some of their next-generation technology skills. That's a big part of it.

We also have our vendor relationships. We work with the largest vendors in the world by bringing in some of their lessons learned, and the skills and training around their platforms. We work very tightly with Comcast for example on things like X1. We're not recruiting from them, but we're applying training, sourcing and best practices from them.

I would say, from an industry perspective, that with the consolidation that's going on there are some good resources that are becoming available. Building a very strong, robust pipeline from the universities and building up the millennial workforce are also key parts of our strategy.

TT: What is the impact of DevOps on breaking down service silos? Did that lead to the creation of cross-disciplinary groups?

KH: This is a big one. As I said previously, it's all about the end-to-end experience. I have a weekly forum with the top technology leaders from all over the country and we go through all of the upcoming maintenance releases across the company. Every once in while there may be a service impacting issue and somebody will talk about it. I've named, I think, six service owners who are vice presidents within technology that own video, voice data and Cox Home Life end-to-end.

We use the LBGUPs (Learn, Buy, Get, Use, Pay, Service) process, which is kind of the quote to cash process, for the end-to-end lifecycle of those products and services. They own every component of that from a technology perspective. We're collocating these other functional teams so architects, engineers, operations and CPE experts are all part of a common team. They're thinking about the quality of the service from an end-to-end perspective.

Every week we do a readout on customer MPS (master production schedule), call in rate, truck roll rate and how we're enabling revenue. But we talk about it from an end-to-end service perspective, which is naturally breaking down barriers by having people take ownership and accountability for doing whatever they have to do to help that end-to-end service be the best that it can be.

It's some structure, some alignment, some awareness and some training, but it's also myself and others setting the right tone at the top for end-to-end excellence. You're going to get more rewarded as a cross-functional team than you are for just delivering your individual component. That is becoming part of our culture and mindset. We've really seen people taking more initiative and accountability. Even junior level folks are now bringing new ideas to the table so we can improve our services.

TT: Do you have some examples of how DevOps has changed, or impacted services and applications?

KH: I would say video is probably the biggest one, which I've already mentioned. Moving to a release management cycle a few years ago was a big step forward for us. Just having structure, a pipeline and a product development lifecycle. We would do quarterly releases, two or three or four a year depending on the product. But now with DevOps, with the work that we're doing on end-to-end video ownership, literally hundreds of releases are going on throughout the year. Some of them aren't major releases but rather minor enhancements or functionality, but that's a huge example.

We launched our new Contour a few months ago and we're already north of 300,000 customers. The quality, the feedback, the MPS (master production scheduling) are off the charts. It's a lot of work that we are doing behind the scenes on our DevOps model with Cox middleware, end-to-end ownership and support.

I would also say Cox.com. We did some benchmarking a few years ago and pretty much in every category of the LBGUPS model we were probably below best in class. With DevOps, with agile, the real-time development and the other work that we've done, we've actually surpassed a lot of the benchmarks in those categories. We're driving a ton of online sales and support through our e-commerce platform and that's a big testament to the team but also to this model.

There are a host of other examples around our network. Even some of the other things we've done recently. We had to move one of our maintenance technical centers here (Atlanta) because of a real estate move. We actually took a lot of the DevOps components lessons learned, things like cross-functional teams and 24/7 action items, and applied them to a physical data center move. It went flawlessly over a nine-month period. That was kind of a unique application of the DevOps model and structure. That was another creative way that we applied DevOps not only on products and services but also on our internal facilities and technical components.

It's more around a mindset. It's more around the cultural transition to understand what customers are expecting, the rate of competition and the rate of change with new technology. It's really a necessity in today's technology marketplace, so we're leveraging that along with other operating models, structures and methodologies.

For previous Q&As on DevOps, check out:

• DevOps: AT&T's Saxena on Building a 'One-Team Culture'
  
• Adopting a DevOps Mindset
  
• CenturyLink's Newkirk: DevOps Culture Puts Customers First
  
• Level 3: Embrace the DevOps Journey
  
• Accenture's Galassi on DevOps Transformation
  

— Mike Robuck, Editor, Telco Transformation