Delivering innovation faster
Comas and NBCUniversal’s Senior Director of Service Strategy and Collaboration, Charles Williams, positioned DevOps internally as a shared platform/framework for delivering innovation faster. The IBM UrbanCode platform—UrbanCode Build (application build automation software) and UrbanCode Deploy (application release automation software)—acts as the engine of DevOps, enabling NBCUniversal to combine continuous integration, continuous delivery, continuous testing, continuous feedback and continuous monitoring into one automated workflow, bridging process, culture and technology across the organization.
The UrbanCode platform supports a diverse set of IT environments and delivers comprehensive audit capabilities, which is critical to a complex enterprise, like NBCUniversal. “With the UrbanCode products we're able to provide build-as-a-service for the organization. I can reproduce any build that comes into UrbanCode at the click of a button,” says Comas. “We have a single, unified dashboard where I can see all of the activity. And we have clear records as to what was built, the binaries that relate to that build, what was done with it, how was it tested, what environment it was deployed to—all of that. Give me a date, and if you want to know what was deployed to production on that date I could tell you everything.”
NBCUniversal uses IBM Development and Test Environment Services (IDTES), developed by IBM Business Partner Skytap, in conjunction with UrbanCode Deploy to automate test environment configuration and management throughout the software development lifecycle and parallelize regression testing as part of QA automation.
In one case, NBCUniversal’s Television Group has used IDTES to reduce its regression testing from 6-8 weeks to as little as 3 hours for Compass, a large application it uses for program scheduling. Associated with Compass is an enormous regression test suite consisting of 1000 test cases. Previous to utilizing IDTES, the organization’s QA team would run these tests locally—each test run in series. Now they’ve set up an environment of 50 test machines, all pointing to the Compass QA environment on premise, with 20 tests running per machine in parallel.
Through DevOps, NBCUniversal has broken down what was previously a monolithic process into component parts, with Comas’ DevOps team taking the lead by providing the infrastructure that supports the DevOps tools, managing the tools themselves and managing the best practices in the use of those tools. The team also creates the automations for the Development teams.
Advocating for DevOps wasn’t easy at first; not everyone was receptive, so Comas and Williams had some cultural hurdles to overcome. “John and I have been talking about DevOps for a while but it was kind of falling on deaf ears. It was when we started talking about capabilities and then showing capabilities to the app owners and then into the QA environment that we started to see traction,” Williams says. “At one point I think John and I even decided to stop using the word ‘DevOps’, because it didn’t mean anything to most people. Improved speed of delivery in application development; being able to run tests repeatedly and consistently and get results fast—that’s what’s important. When we spoke in those terms we started changing the culture. And when we had a couple of opportunities to demo our agile development capabilities we started hearing, ‘Oh, so this is what DevOps is. This is what I've been wanting.’”
In that sense, tools and technology, like UrbanCode, drove culture and process change. “I think the tooling helped fortify what the culture was shifting towards; more agile development, more shift-left testing,” says Williams. “The tools underpinned the DevOps process we envisioned, which is an end-to-end workflow.”
Automating for speed and quality
NBCUniversal’s Television Group in New York stands as an example of how the organization's DevOps journey has taken shape. That particular business unit is the television distribution arm of NBCUniversal and is, in part, responsible for scheduling on-air content for all NBCU-owned stations. As noted previously, the Compass application is central to that task.
Comas and his team migrated Compass—a massive .net-based app—to the UrbanCode platform some 18 months ago. “I would describe the previous Compass process as like a ‘manu-matic’ process; they would commit their code to the repository, manually click the build button and then deploy it to an environment. Once they deployed it to an environment it was basically left to the QA team to handle,” he says. “So a week or so would go by and they'd get some results, do another build and send it back over to QA, and on and on. So the time-to-market for an application was very high—20 weeks to get a release out. One reason it took so long is that the QA team would do full-scale testing on a release that may or may not have met the business requirements.”
The Compass application team’s goal was to reduce that time by 75 percent, to five weeks. “Once we moved them to the UrbanCode platform they were able to hit that goal. In fact, we’re now looking to reduce time-to-market to two weeks,” Comas says. First, his team integrated QA automation with the Compass workflow, eliminating the need to manually execute numerous QA tests for every deployment. “We started with shift-left testing. We unit-tested the application release and did code coverage analysis on it, while also starting a small functional test where we would observe the different business requirements and create tests which would exercise those. So before the release would even get to the QA team to be fully functional and regression tested we could get immediate feedback if the release was good or not. That, in and of itself, accelerated things greatly.”
NBCUniversal’s DevOps approach also broke down silos in deployment. “Historically, releases could only take place on certain days of the week—typically Tuesday and Thursday evening—and if you missed that window you couldn't do a release until the following window,” Comas says. “So by working with the Compass Run team we can now do deployments to production much more frequently, resulting in smaller, more frequent releases. So they have much greater agility and can meet the business requirements much faster. In short, the code quality has improved, it's going faster and we're able to meet the business requirements much more rapidly.”
From a technology standpoint, Comas says the UrbanCode platform gives his team the ability to integrate a wide range of tools into one automated workflow, eliminating over-reliance on developers. “With UrbanCode, when a developer commits code without adequate unit tests, it's going to be flagged. If a developer didn't have that as part of the workflow you're not going to know that. There'd be no feedback into the process telling you the status of the application. Our Compass workflow is probably the most comprehensive automation we have, and the technology is helping us drive the application quality, improving build quality by an order of magnitude. Not only that, we’re delivering the changes the business wants much faster. Those are the ultimate goals; the reasons why we’re on this DevOps journey.”
For Williams and Comas, DevOps is about building trust and confidence and about instituting predictability, speed and quality in software delivery lifecycle. “We're providing a robust infrastructure guaranteeing a level of quality and a level of service to our Application teams,” says Comas. Indeed, the DevOps team is earning enough trust that business units are now asking them to help with larger projects. “Because we've shown that the capabilities are flexible and expandable and that they allow us to deliver on a very tight timeline we’ve seen a six-fold increase in project volume, from ten to more than 60 applications. We’re providing a path to application production that engenders a level of confidence we’ve never had before.”