Driving a Tech-Led Reimagination through DevOps at eBay

According to Mark Weinberg and Randy Shoup, eBay is too slow and lags behind industry leaders in engineering velocity. At the 2021 DevOps Enterprise Summit, Mark Weinberg, the VP of Core Product Engineering, and Randy Shoup, the VP of Engineering and Chief Architect, described how they are using DevOps to transform engineering at eBay.

Their mission is to make eBay’s software delivery velocity a competitive advantage for the company by addressing the systemic challenges that have accumulated over the years, including monolithic code, missing tools and infrastructures, poor processes, etc. Ultimately, fixing these issues will be a company-wide endeavor and not solely limited to engineering.

To begin this initiative, Weinberg and Shoup assessed the situation at eBay using value stream mapping, which flows: Plan d Develop d Deliver d Iterate. What they found were problems and bottlenecks at every stage.

At the planning stage, they found significant inter-team dependencies and conflicting priorities. Next, Development was facing slow build and test times, too much context switching, and bottlenecks created by wait states. At the delivery stage, there were staging issues, and many teams lacked an effective end-to-end, automated development pipeline. Finally, in the iteration stage, they found no end-to-end monitoring, tracking issues, and dysfunctional experimentation.

With all these challenges, Weinberg and Shoup had to figure out which to address first. They decided to focus on software delivery and development because fixing this issue would help clean up challenges both up and down the value stream.

They also needed to decide how they were going to measure if they were achieving their goals. They decided to go with the DORA metrics popularized in the State of DevOps Reports and the book Accelerate.

Based on these metrics, eBay was a medium performer overall. However, through this pilot initiative, teams began to move into the high-performing area, achieving triple deployment frequency while reducing lead times by 2.5x, and improving the change failure rate 3x. As of fall 2021, 10% of the active apps at eBay are involved in this pilot, which focuses on both short-term wins and longer capabilities. The goals include driving improvements in developer productivity, software delivery, instrumentation and monitoring, and re-architecting critical areas.

Shoup and Weinberg attribute these successes to focusing efforts on a select number of pilot domains (which cover big areas of the product, like Selling, Search, and Ads), applications within those domains, and platform tracks (which are the tools and infrastructure that support builds, CI, staging environments, and the education and training of engineers).

In the initiative, there is a collaborative effort made across the technology platform teams and the application teams. They work closely, which didn’t use to be the case at eBay. Today, senior architects are embedded in teams to advise and occasionally write critical code. There are daily standups between leaders, weekly team-of-teams meetings, weekly deep dives, and monthly operating reviews with the executive team.

As part of the pilot, the DORA four key metrics were added to the development dashboard of every application, at which point Shoup and Weinbeg iterated by adding granular viability to every deployment pipeline in order to help teams debug their processes. The teams work in a tight Plan, Do, Check, Act (PDCA) cycle. Shoup and Weinberg use the collaborative environment to steadily remove impediments by centering the initiative’s goals on conversations about what teams need from leadership in order to achieve their aims.

In addition to the previously stated quantifiable successes, the teams involved in the pilot have also reduced build, startup, and PR validation times; invested in their staging environments by using good data and reliable components; introduced automated upgrades, testing, deployment, and increased site speed; streamlined team processes and code reviews; and moved from monthly to weekly mobile releases.

The adoption of more DevOps practices also led to a culture change among teams. They look at metrics more regularly and are inspired to communicate and help each other. Teams in product organization have begun automating their own workflow. There are regular demonstrations of new tools and practices, learnings are shared between different groups, and executive support and engagement is high.

This cross-functional velocity initiative also continues to face challenges, however. Shoup and Weinberg still hope to improve overall eBay outcomes, and at times the initiative feels under-resourced while leadership feels overtaxed. Some team members are too focused on metrics and less on how those metrics support the pilot’s overall goals, and other inhibitions include fear of failure and consequences and lack of confidence in the program’s approach.

As they continue to move forward, Weinberg and Shoup hope to employ rolling planning with small, frequent, and cheap experiments. In the development phase, they want to utilize small batch sizes, fast build and testing automation, daily merges and deploys, and decoupled architecture. Success in the delivery space would include a fully automated test and deployment pipeline, one-hour commit to deploy, and iteration in production using feature flags. Lastly, in the iterate stage, there’s room for more end-to-end monitoring, tracking everywhere, and more small, cheap experiments with rapid feedback on results. The initiative will soon scale from 10% to 50% of applications.

- About The Authors

IT Revolution

Trusted by technology leaders worldwide. Since publishing The Phoenix Project in 2013, and launching DevOps Enterprise Summit in 2014, we’ve been assembling guidance from industry experts and top practitioners.

Follow IT Revolution on Social Media

• 
• 
•