This is a guest post by Curtis Hill (LinkedIn Profile)
During the last 20 years or so the practice of pairing US and European based software engineering teams with teams from all over the globe has exploded. I have worked with teams all over the world for the last 15 years and I have found it to be a wonderful experience overall from both professional and personal perspectives. That having been said when a business executive asks me what working with global teams means from a software engineering perspective I simply answer that it makes it harder. That is not to say there aren’t many valid business reasons to produce software globally. It’s just that one of them is not a reduction in engineering complexity.
Commenting on the business conditions that drove this explosion – wage arbitrage between the developed and the developing worlds, advancing communication technologies, high tech work forces popping up all over the world, etc. – is not the purpose of this paper. The business community took advantage of the conditions and software engineering teams must adjust to the new reality. If you are a software manager you are very likely to face the added complexity sooner or later. The root causes of the added complexities are obviously one or all of large physical distances between teams, large time differences and potential cultural issues including language. An education, technology and/or process methodology mismatch may exist as well. I will make no attempt here to make value judgements about the quality of the education and training or the abilities of the engineers. My experience has been that good engineers can learn most anything, bad engineers can’t and there are good and bad to be found all over this earth. No matter the relative quality of the teams and their maturity levels these root causes add complexities to any project. Far more complicated internal and external communication, unclear ownership of tasks, additional management overhead and the potential for any or all of these issues to cause strained working relationships between teams are just a few of the possible added risks/costs.
During the same 20 years or so many new schools of thought about the software development/engineering process have emerged. These came about in reaction to the abysmal performance of most software teams when measured on due date performance, quality, cost performance, customer value produced – you name it – no metrics could be found to support the idea that existing software development teams were operating smoothly. Software engineers had always used ‘plan based’ processes naturally borrowed from the more traditional engineering disciplines (electrical, structural, etc.). These plan based processes are commonly referred to as ‘waterfall’ and are still actively espoused by the Project Management Institute (PMI) among others and are in use by many practitioners. But traditional ‘plan based’ processes did not seem to work in the software world. The pace of changing requirements due to rapid market and technology twists and turns combined with the the reality that software CAN be changed at any time was a challenge not like the challenges found in those older engineering disciplines. The limitless problem set that can be addressed with software combined with a limitless solution set for even the simplest problem made this new engineering discipline unique and more complex than any engineering form that came before.
Three of these new schools of thought have gained a significant level of adoption – Agile, Lean and the Theory of Constraints (TOC). Both Agile and Lean ‘theories’ have multiple ‘real world’ implementation schemes. With the notable exception of the Agile group of concepts and accompanying methodologies these new schools of thought did not originate within the software community. Both Lean concepts and the TOC came from the manufacturing community. TOC began to be adopted and adapted by the software engineering community soon after Eli Goldratt published ‘Critical Chain’ in 1997. Lean concepts did not gain any significant adoption until very recently – post 2005. I have worked with global teams that used agile concepts (SCRUM and others) and that used Kanban (a Lean based set of ideas). I have not personally implemented TOC but I have read a good deal on the subject including Dr. Goldratt’s books and I believe some if not all of the concepts can be used effectively in a global setting.
This series of articles is an attempt to analyze these approaches and determine if and how they mitigate the risks and additional costs inherent in building software with geographically dispersed teams. Do the new development approaches ease or exacerbate these global engineering complexities? Do the global engineering complexities stretch these concepts beyond the breaking point? Is there a way to ‘configure’ the approaches so that they address some of software development’s greatest issues (the reason for these new ideas in the first place) even in a global setting? Can these new principles provide some or all of the answers needed to make global project management more productive? The first article which follows will be an analysis of the twelve agile principles proposed in the Agile Manifesto (http://agilemanifesto.org/principles.html and how each impacts a global engineering setting. The next two articles attempt to gauge the impact on global software projects of the Lean and TOC schools of thought. Throughout this series of articles we will see that the goals and the means to achieve the goals of all three new schools of thought have numerous similarities. The conclusion will propose some amalgamated approaches that my knowledge and experience tells me can be used to more effectively solve the inherent added complexities of global engineering. The conclusion will also make some proposals about implementation details but close examination of current implementation details or a detailed proposal for a new implementation approach are not in the scope of this article. Having said that, I will use some implementation examples to make points without explaining those examples in detail. There is a great deal of literature on all of the implementation schemes referenced here if more information is desired.
The Agile Manifesto and What Followed
The Agile Manifesto was published in 2001 by a group of practicing software engineers. As addressed above, they strongly believed that traditional plan based processes had not and could not be used to solve the unique complexities of software engineering. Not long after the manifesto was published numerous attempts were made to take the twelve principles espoused in the manifesto and embody them in a prescriptive process for developing software. SCRUM and Extreme Programming (XP) are two of the proposed processes that have gained a reasonable amount of popularity over the past 10+ years. Many successes using these processes have been documented. Taken as a whole the agile approaches have in many cases eased some of the traditional problems that have caused so many failed software projects – long lead times, poor quality, customer acceptance, random and rapidly arriving changes in scope, etc.
There is not a consensus in the literature on the adoption of agile processes in a global setting. Some conclude that agile processes are beneficial while others focus on the more obvious conclusion that collocated teams are more efficient than distributed teams. ‘The authors had the opportunity to compare the productivity, quality and performance of collocated agile teams with distributed agile teams and found that the collocated teams perform substantially better.’ (S1) Saying teams MUST be collocated stops the conversation and does a disservice to the body of agile concepts as a whole. One of the principles ( #6) they stand behind during this argument states – The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. The more broadly applicable statement would have added something like – ‘Each additional non-collocated team on a project adds exponentially to the communication and teamwork risks on the project. These risks must be mitigated using all possible means if a set of disperse teams are to successfully implement this principle’. By definition global engineering teams cannot possibly match the amount of face time collocated teams have with each other. For this reason there is no question that collocated teams communicate much more efficiently and effectively than do disperse teams. Attacking the communication challenges of geographically disperse teams is at the top of the list of things you must do to succeed in that environment regardless of the processes used. Every possible means of maximizing communication and teamwork must be used since the teams are not optimally located. Tenet #4 also speaks strongly to this preference except it mandates daily communication between developers and business people. This will be discussed more during the discussion of those principles ( #4 and #6). I agree with Andy McLoughlin (founder of Huddle.com) who said ‘ you have to be agile about being agile’ (S2)
This paper proposes that many of the twelve principles still bring benefits in a global development setting even if those benefits aren’t always as compelling or as easy to achieve. Another way to ask the question is this – is global development easier and more efficient using agile principles or the more traditional plan based principles? If the answer is yes then why not use agile methodologies even if global complexities water down the benefits somewhat? If agile proponents want the agile revolution to grow for the greater good of all they need to be thinking about how to adapt agile principles to the real world as opposed to standing on principle for principle’s sake. I don’t suggest that many don’t agree with me but some vocal ones seem to have cast doubt on the ability of agile concepts to scale as team sizes grow and locations proliferate.
The twelve agile principles and how global engineering impacts their adoption and effectiveness:
1) Our highest priority is to satisfy the customer
through early and continuous delivery
of valuable software.
There are several ideas inherent in this statement. One is that the longer the lead time for a software delivery the greater the possibility that the actual result will have drifted from the customer desired result. Also the positive relationship between on time deliveries and customer satisfaction is implied here. The concept of limiting work in progress (WIP) is even implied here. All of these concepts are touched on in TOC and Lean as well in one way or another. Agile uses the term ‘continuous delivery’. This term is being adopted across the software industry to mean that value, in the software world – ‘deployment of new working software’ – is being delivered or could be delivered very frequently in small increments. It’s probably not surprising the term is becoming ubiquitous since TOC and Lean speak to the concept as well in one way or another. All three schools of thought also promote the concept that ‘done means deployed’. TOC attempts to maximize throughput. In software terms throughput is – guess what – ‘deployment of new working software’. Lean primarily focuses on eliminating waste but the Kanban implementation of Lean as described in David Anderson’s book – ‘Kanban: Successful Evolutionary Change for Your Technology Business’ also has a vital ‘throughput component’. He has several mechanisms for maximizing ‘valuable’ throughput including a prioritization and escalation process. He also decouples each work item type (dev, QA, etc.) so that work items can flow through at their own ‘pace’. This will be discussed in more detail in the Lean article in this series. Interestingly the newest definitions of ‘continuous delivery’ would not consider the agile methods ‘continuous’. This agile principle advocates continuous delivery but agile implementations do not attempt continuous delivery – they prescribe time boxed delivery (iterations) for the most part.
As stated above we will see this thread of similar ideas in each school of thought throughout these articles. Adding an additional team (local or remote) to any project complicates the value chain. Potentially additional value added and non-value added steps are inserted in the value chain. Adding a remote team adds other complexities as we have said. These complexities threaten lead times and inherently add to WIP. There are communication and work flow complexities that add to the risk of work queuing adding pressure to increase the total allowable??? WIP and lengthening lead times.
This principle is one of several that absolutely apply to a global setting but are much harder to implement in that setting. This principle is, on one level, a 180 degree recoiling of the agile proponents against the plan based tendency to build systems in a monolithic manner. The project starts with a lengthy multi-level design effort, the WIP rises to a crescendo culminating in a one time, only time and final delivery. This led to multi-year software projects. Successes were few and far between. The pace and degree of changing requirements and task variability unique to software engineering exposes the unwieldiness of plan based approaches. As is going to be the case in many of the forthcoming discussions global engineering only aggravates the root problem. Can you implement a scheme that enables this principle in a global environment? Absolutely – but agility is retarded because more planning is required to create a smoothly running continuous delivery machine. There are more moving parts creating many more cross team interfaces and potential failure points, longer communication and decision paths, language and time differences, far less frequent (if any) face-to-face communication, etc. etc. requiring more planning to compensate for the risks and inefficiencies they imply.
However, agile implementation schemes such as SCRUM can help deliver on this principle even if a little more planning is required and a little agility is lost. Time boxed delivery is a better approach than one massive delivery at the end of the project for most software projects. The location or locations of the team or teams does not change that reality. You might find the SCRUM concept of daily stand-ups (another of the many implementation ideas that are also shared between the three schools of thought) to be a bit more difficult to implement in the prescribed manner – once a day, in the morning, for 15 minute and no more. However, the idea of frequent interactive sessions that provide a common understanding of the state of every task, next steps for each task and any blocking issues on the project provides all of the same advantages in a globally disperse team that it does for collocated teams. It’s just not clear that daily makes sense in every situation anymore. Maybe 30 minutes will work better. At the start of who’s day should we meet? Or should we have two daily’s per 24 hour day? These are questions that have to be answered among many others. But frequent meetings such as described above will make ‘early and continuous delivery of valuable software’ more feasible even in a global environment.
2) Welcome changing requirements, even late in
development. Agile processes harness change for
the customer’s competitive advantage.
Communicating changing requirements to a global team is more difficult whether agile principles are used or not. However, agile concepts enable timely communications around changing requirements if for no other reason than they remove the bulky change management processes that are rigorously enforced in plan based schemes. Agile implementations typically prescribe that requirements prioritization meetings be held at the beginning of each iteration. A simple requirements list in business priority order is reviewed. The known iteration velocity of the team is used to draw a line slicing off enough of the top of the list to inject into the next iteration exactly one iteration’s worth of work. In a smoothly running agile team these meetings can be completed in less than 30 minutes. If your iteration duration is two weeks then you will spend one hour a month on change management. Not bad. * Agile shifts the view of changing requirements from that of a rarely necessary evil to a desirable and welcomed reality that managed well can provide a competitive advantage. This is a sea change and in my opinion a necessary one for most software projects in the real world whether one collocated team or five globally dispersed teams are engaged in a project. Again, managing change will still be more difficult with remote teams but agile implementations try to minimize the difficulty of change management so global teams may even receive more value than collocated ones from this principle.
* – This is an oversimplification even though the process is not difficult and the gains in planning efficiency are attainable. It is not within the scope of this paper to explain individual implementation schemes (Agile or otherwise).
3) Deliver working software frequently, from a
couple of weeks to a couple of months, with a
preference to the shorter timescale.
In my opinion this can be viewed as subordinate to principle #1. It is necessary but not sufficient to implement #3 if #1 is to be achieved. This principle also implies an attempt to manage WIP. It clearly states a preference for short lead times by completing tasks each iteration and not waiting for the big bang delivery seen in planning based processes. This principle underlies the time-boxing that many agile implementations advocate. It is also related to the batch size discussions you will see in lean and TOC thinking. The quality issue is repeatedly addressed by all of these new ideas with the three related concepts – small batches, short lead times and limited WIP among others. As Eric Reis has said: ‘The biggest advantage of working in small batches is that quality problems can be identified much sooner’. (S3) Or as Jon Strickler posted on his blog – ‘ The fact that lean concepts extend into the scrum cycle by ensuring that there is a limited amount of focused work at the development team level is often overlooked. My reaction to Kanban has been similar over the past couple years as it has gained popularity. While Scrum and XP use story points or ideal man days to limit WIP, Kanban borrows a more direct tool from lean manufacturing to accomplish the same objective through the Kanban itself.'(S4)
The impact of this principle on global engineering is more or less the same as principle #1. The implementation difficulties are also similar. Here again some implementation details might change, some additional planning will be required and some agileness will be lost but it’s still a concept that will be effective even with global teams.
4) Business people and developers must work together daily throughout the project.
This is the first tenet that speaks to the preference for collocated teams. In this case it is a dictate – development teams MUST work together with the business people every day. Collocation surely improves the efficiency and effectiveness of that daily collaboration. Principle #6 below adds more weight to this concept. In many global development settings at least some of the developers are collocated with the business people. This then turns into another global communication challenge. Getting requirements from the business people into development may not be a problem but then concisely communicating those requirement to the remote teams adds a communication hop and opportunity for communication errors and communication lag times. Every possible communication channel must be explored to minimize the impact of this problem. Some structured communication must be put in place. Planned meetings should occur at all structural levels – management levels and cross team levels. It is imperative that individuals with excellent communication skills exist on BOTH sides of the communication pipe. Ad hoc communication must be strongly encouraged. Adding incentives to encourage ad hoc communication is a good idea. All communication technologies should be evaluated for the unique value each brings. Collaboration software, email, chat, social media, video conferencing etc. should be considered. This is an area that will bring opportunity for continual improvement due to advancing communication technologies. Time should be allotted for the team to research new options.
If there are no developers collocated with the business people then the requirements must be communicated directly to the remote team(s). In this setting face-to-face communication during the requirements gathering phase of the project is optimal. This means getting key people from all locations involved. The travel costs are almost always insignificant compared to the value realized. Another good time for bringing the teams together is during delivery so any issues can be communicated face-to-face. If this is not possible you are back to maximizing the effectiveness of the communication any way you can.
5) Build projects around motivated individuals.
Give them the environment and support they need,
and trust them to get the job done.
I have always thought that this agile principle while profound does not help define agility. This concept is table stakes if you want a productive engineering team regardless of the project processes employed. The trust issue is the difficult one to engender. Hopefully everybody already knew to hire motivated people and give them the tools to get their job done long before the words agile and software were ever used in the same sentence. Trust however is at the heart of why software engineering is searching for new and better ways to deliver software. The poor delivery and quality record of the industry has greatly undermined the trust many customers and business people have in software engineering teams. In my career I have spent a tremendous amount of time and energy trying to repair broken relationships between engineering teams and business people. Repeatedly missed delivery dates and, maybe worse, software that either does not work or does not match the initial requirements has caused this lack of trust. Delivering working software early and often is one of the best ways to begin to build this trust back. One of the core similarities between Agile, Lean and TOC is the concept of customer satisfaction. All of them use some mechanism to deliver software more frequently than in the old plan based methods. The common belief amongst the practitioners of all three is that reducing lead times engenders trust and improves quality.
There is also an inherent trust issue between engineering teams not just between engineering and the business or customer. How often have you seen dev and QA managers pointing fingers over a missed bug or delivery date? This behavior instantly jeopardizes the trust between the two teams. This issue is dramatically exacerbated with global teams. Many of the reasons are obvious – cultural differences, political ideologies, zero to very few personal relationships, job security concerns, etc. etc. I have seen individuals from line engineers all the way to CxO’s exhibit some level of mistrust that clearly had cultural and political undertones. Rest assured if the teams don’t believe in each other failure is guaranteed
6) The most efficient and effective method of
conveying information to and within a development
team is face-to-face conversation.
This is a corollary to #4 now adding that it would be optimal if all the development teams are also collocated. So the net of the 2 tenets is that the business people and all the development teams must be collocated for optimum efficiency and effectiveness. It is true. But we are not dealing with an optimal configuration. The main takeaway here for someone who is managing globally disperse teams is that face time is really important. Any money spent on bringing your teams together (within reason – the business purpose and the time spent must be well defined) is well spent money. Key stakeholders should get together at least quarterly and if at all possible the entire team should meet at least once a year. There should also be tactical travel for key stakeholders at critical points in projects – typically near the beginning and near the end of a project.
It is seldom possible to have face-to-face meetings at the regular tempo of the meetings proscribed by many agile implementation schemes. The cost alone is usually prohibitive to have teams come together for every sprint planning meeting or retrospective, etc. The face-to-face meetings in a global setting will have a much less granular pace than can be achieved by collocated teams. This simply means that those meetings must be structured to maximize shared information – technical, market, corporate. Teamwork enablement exercises and simple ‘getting to know each other’ time are invaluable tools in your attempt to mold a cohesive whole out of a globally disperse and culturally diverse organization.
All the other communication methods mentioned in tenet #4 above should also be used where applicable. Post pictures of all team members at every location. Include remote teams in all corporate level information dissemination meetings even if this means somebody has to stay up late or get up early. There are numerous other ideas that can be used to foment communication flow and teamwork.
7) Working software is the primary measure of progress.
This principle is on some level a reaction to the plan based use of formal estimation and project tracking mechanisms. These are examples of tasks that do not add customer value. All of the schools of thought we will discuss in these articles have something bad to say about tasks that do not add value. They each deal with it in very different ways as we will see later.
The Agile Manifesto is deliberately concise and many of it’s principles require a practitioner to understand all of the ideas inherent in the principle and to implement the ones that meet his or her current needs. This principle is certainly no exception. In plan based methods every task is estimated no matter how small and insignificant. Project tracking meetings are held on a frequent and regular basis. During those meetings engineers do their best to estimate the inestimable and predict the future. All of this non-quantifiable work is then recorded. This principle suggests that all or most of that work is a waste. Some things still have to be estimated for a number of mostly business reasons but estimations should only occur when absolutely necessary. Project tracking is far simpler in a time boxed/limited WIP environment. Agile doesn’t explicitly espouse limiting WIP but principles #1, #3 and #7 all imply it in one way or another.
8) Agile processes promote sustainable development.
The sponsors, developers, and users should be able
to maintain a constant pace indefinitely.
Agile, Lean and TOC all promote a pace or flow that is viewed as more sustainable than the plan based methods. Some of the he tenets of this ‘sustainable flow/pace’ in all three are – 1) Significant control over work input pace/flow, 2 ) Frequent deliveries of completed software, 3) Short lead times, 4) A mechanism for limiting WIP. Lean and TOC spend a good bit of time explaining and justifying these tenets. Again the Agile approach is concise but requires more interpretation.
This principle has two drivers at least. The most clearly implied principle is that engineers cannot sustain a reliable throughput if there are extreme peaks and valleys in the workload (or worse – only peaks). This is a reaction to the 80 and 100 hour weeks that many software engineers found themselves working just to deliver late anyway. The combination of a hard push with failure on the back end is one of the most demoralizing things that can happen to a team – collocated or not. The second driver, whether conscious or not, is the idea that a sustainable delivery pace/flow will ultimately produce more throughput than a workload with large variations in size and complexity. In one way or another most of the other principles speak to this very high level statement.
Global teams impact this in the same way they do many of the principles that underpin this one. Getting the pace nice and even and building a ‘software factory’ is just harder with non-collocated teams. More interface points, more communication risk and all the other things you’ve already heard five times. However, as is the case with the other principles it is still a desired approach even if it is harder to achieve.
9) Continuous attention to technical excellence
and good design enhances agility.
Technical excellence and good design were not invented by agile proponents. They are mother and apple pie to any engineer worth his or her salt. What is added here is the idea that those very desirable behaviors make an organization more agile. And they do. We’ve known for a long time that componentization, abstraction layers, design patterns, etc. make software maintenance and augmentation easier and more reliable. Easier and more reliable is agile. This is another ‘agile’ principle that is applicable to any software project anywhere or anywheres in the world.
10) Simplicity–the art of maximizing the amount
of work not done–is essential.
This is a waste elimination principle at it’s heart. Writing software, unlike building widgets, allows for an unlimited set of viable solutions. Viable here means ‘it works’. If you are making a widget the details of making that widget are set in stone before the work starts. Different software engineers can literally spend 10 to 20 to 50 times more effort to solve the same problem. The engineer that expends 50 times more effort has likely added some other ‘really neat’ stuff that the customer did not ask for. Crisp requirements, good design, developer discipline and frequent inspections are required to minimize this ‘gilding the lilly’ problem. Once again this concept is found in the other new schools of thought. For example, a lean proponent – Eric Reis – introduces the concept of ‘minimal viable product’ in his book ‘The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses’. A very similar concept to this agile principle.
This waste problem is far more likely with non-collocated teams. Inspecting is harder and requirements communication is harder. Good design and disciplined developers will still help tremendously. It may be harder in a global setting but the benefits might be even greater because of the greater likelihood of these issues occurring.
11) The best architectures, requirements, and designs
emerge from self-organizing teams.
It is harder for non-collocated teams to self-organize but it is not impossible. My opinion is that Kanban (this Lean implementation will be discussed in the Lean portion of the paper) is one of the best mechanisms to help teams self organize. Agile again is not very prescriptive here. Several examples of globally developed projects using Kanban to self-organize among other uses are detailed in David Anderson’s book – ‘Kanban: Successful Evolutionary Change for Your Technology Business’. The visual nature of the work flow and the specific card content on a Kanban board make self-organizing easier.
12) At regular intervals, the team reflects on how
to become more effective, then tunes and adjusts
its behavior accordingly.
This may be the concept most closely shared between the three schools of thought – Agile, Lean and TOC. The founders of each school of thought recognized that the problems they were addressing were both constantly morphing and too complex to solve once and for all. They saw that a software engineering process must evolve to stay viable. This is another concept that is, if anything, more important in a global setting. It speaks to agility in that adapting and changing quickly is at the heart of business agility regardless of which organization we are talking about. This principle reaches beyond a set of ideas or a business function such as engineering. This is at the heart of all human achievement. Building on the past to make things better in the future is why we have a ‘modern’ world. This is called Kaizen by the Lean group and is embodied in the fifth focusing step of the TOC.
As a software engineer if you are in a position to assist in the decision to engage globally disperse teams then it is incumbent upon you to explain the hidden costs and risks that come with global engineering such as additional management overhead, additional planning, lost agility, etc. It does not mean that your concerns will or should dominate the decision process but they are legitimate concerns that need to be considered. If the decision has already been made or in a case like the above it is determined to go forward with a global team approach then you have to decide how you are going to manage the added complexities and mitigate the added risks. If you have projects that have very low change rates, low task variability, and/or mandated planning and documentation due to governmental or business rules then a plan based approach might be right for you and having global teams likely will not be the fact that changes that decisions. If however your projects do not possess these characteristics then you should consider all of the new approaches that this paper addresses. Adopting agile concepts specifically will certainly help mitigate the complexities inherent in global engineering. The adoption process itself will be complicated by the realities of global engineering. The benefits might also be muted in some areas but in other areas agile concepts attack the very complexities inherent in global engineering.
S1. B. Cohen, M. Thias, “The Failure of the Off-shore Experiment: a
Case for Collocated Agile Teams”, AGILE ’09, 2009, pp. 251-256.
S3. Eric Reis, ‘The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses’.
S4. Jon Strickler – Agile Elements – ‘Lean, Agile – Kanban, Scrum’ May, 8, 2009