Last month, I was asked to give a second opinion on some key architectural decisions and the way they were working out in a client’s IT landscape. Such architecture assessments are some of my favorite engagements, especially when there is sufficient time to get to the bottom of things with a small team of experts. A thorough analysis, underpinned by firm evidence and fleshed out in concrete recommendations, can add tremendous value: value in terms of risk mitigation, prevention of rework (or failure) and freshly spotted opportunities.

The value of architecture assessments

Where does the added value come from? Of course, an outside reviewer can never have more depth of knowledge and experience about a system than the team that has been working on that system for months or years, no matter how experienced or erudite the reviewer is. There is, however, intrinsic value in what Daniel Kahneman calls the “outside view”.

When the same group of people make decisions together for a prolonged period of time, the quality of the decisions may start to suffer from tunnel vision. Tunnel vision is caused by the need to make progress and the desire to have harmony in the group. Time and peer pressure lead to participants not voicing, or prematurely rejecting, alternatives, and so see only information that confirms what they already thought, ignoring danger signals (confirmation bias). Asking someone from outside the team to provide a second opinion or perform an independent architecture assessment is a great way to obtain such an outside view, and combat tunnel vision as well as other forms of cognitive bias (such as “groupthink”, anchoring bias and cargo cult – see Philippe Kruchten’s presentation on cognitive biases in software architecture).

Can we quantify the value of the outside view? Almost 20 years ago, research suggested that a well-timed and well-executed architecture assessment saves 10% of a project’s costs on average. The same number probably applies to the delivery effort of epics, features and architecture runway in an agile context. It is likely the timing of independent assessments in an agile context will be different from more traditional up-front design situations. Regardless of where you are on the waterfall versus agile spectrum, independent architecture assessments should be timed when opportunity cost is low, and changes to the architecture are (still) relatively easy.

A good outside view

Providing outside views was one of the responsibilities of our “technical conscience” team at CGI (both for our own solutions and on-site for clients). For over a decade, we gathered many insights into how to add real value with our assessments. We certainly learned what not do. A  good architecture assessment is not:

• A cat and mouse game between reviewer and team
• A gate to pass or stamp of approval to obtain
• An opportunity for the reviewer to show how clever they are
• An assessment of the team’s performance
• An audit or compliance check

Avoid these mistakes, as they will significantly reduce the outside view’s added value. In our experience, most of that value comes from:

• Getting stakeholders together in a room: this is often the first time they hear each other talk about the solution or decision you’re assessing, which often leads to unexpected insights
• Uncovering the trade-offs that have been made, creating visibility of (assumptions about) priorities
• Identifying risks and opportunities when there’s still time to do something about them

As you can infer from the first bullet here, a good architecture assessment cannot be based on just a document describing the architecture. You need to talk to the team and their stakeholders to be able to fully understand the context and the drivers behind the choices that were made – ask the “why” questions.

Conclusion

Getting an “outside view” on architectural designs or decisions often yields good insights, and helps teams deal with risks and capitalize on opportunities at relatively low cost. Spot the best time, and ask someone who knows what they’re doing to assess your architecture. </begin shameless self-promotion> Let me know if I can help! </end>

Further resources: book Evaluating Software Architectures (SEI), interview with author Rick Kazman.

“[We value] working software over comprehensive documentation” features proudly on the front page of the Agile Manifesto. Further down on the page, the authors explain that they do not mean that there is no value in documentation (just less value than in the working software). The practice of documenting architectures in particular has significant positive correlations to solution success factors like predictability, customer satisfaction and technical fit. So we should not abandon the practice of architecture documentation, but we should document our architectures in a way that yields optimum business value.

Bloated architecture documentation

In the past few years, I have helped many organizations modernize their architecture way of working. At the start of these endeavors, the average architecture documents are usually quite large – 200 pages is not abnormal. The templates alone often contain dozens of pages. It didn’t use to be like that – when they started out documenting architectures 15 years or so ago, the templates were nice and small, and focused on a limited number of views, designed to show how a key set of stakeholder concerns had been addressed (such as Philippe Kruchten’s 4+1 Views). However, over the years new insights led to new sections to be added to the template. Concerns that led to business risks or disruptions required new views, such as a Security view or an Integration view. The emerging realization that architecture is a set of design decisions led to the addition of new ‘Architectural decisions’ sections. Outsourcing part of the development sometimes required yet again more detailed views.

The Architecture Document had become the single repository for all architectural knowledge related to a product. This is not a problem per se, except that these same documents were used to show stakeholders that their concerns were addressed by the architecture. Specifically, they were the vehicle for obtaining approval from business owners. Business stakeholders were expected to read, understand and approve hundreds of pages of architecture documentation. Naturally, this led to delays: the architects would spend more and more time preparing the document, and the approvers would take more and more time approving them. Leaving blank parts ‘to be completed later’ did not help, because it increased uncertainty in the stakeholders about the completeness of what they were approving. Documentation bloat had caused the architectural feedback loop to become longer and longer, and caused significant delays when starting up new initiatives.

The value of architecture documentation

It is not surprising that documentation bloat causes organizations to start questioning the added value of architecture documentation: perhaps that value is lower than the cost of the delays the document causes. So let’s see if we can answer this question: “Why were we doing this again?”

In the organizations I work with, I see three main purposes for architecture documentation. We are communicating the architecture to show stakeholders how their concerns are addressed, often in stakeholder-specific views, but also to involve them in the architectural decision process (driving the architectural feedback loop). Additionally, we are communicating the architecture “to the future” for various purposes – let’s call this architectural knowledge preservation. This leads to three distinct business goals for architecture documentation:

• Explain how stakeholder concerns are addressed (including business, DEV and OPS)
• Preserve architectural knowledge (for analysis, realization, trouble-shooting,…)
• Collaborate on architectural decision making

Each of these business goals has manifest business value, e.g. in terms of decision support, driving progress, design quality, risk and cost control.

Separate according to purpose

When we look at the audiences and timing of the three business goals identified above, we quickly see that they require very different language and rhythms in order to provide optimum value:

The insight that the language and timing requirements of these purposes are so different helps us understand that the ideal of “one document as the single repository of architectural knowledge” is not viable. If we want our business stakeholders to quickly grasp that their budget and delivery concerns are addressed, we should not send them a 200 page document and then ask their approval. If we want the development teams to easily learn how to implement our architecture, we should not bother them with exhaustive lists of business rationale behind all the underlying decisions (we should not hide that rationale from them either: sometimes it is needed to make the right development choices).  

The picture above is an example of how to separate the various rhythms and languages of architecture documentation.

Conclusion

The insights above have helped us to significantly reduce the size of the architecture documents presented to stakeholders. In one organization this helped reduce the start-up time of smaller projects from (typically) two months to two weeks – a real boost to agility and benefit to the organization.

Four years ago, Facebook changed its famous motto “Move fast and break things” to “Move fast with stable infra” (not quite as catchy?). Most people associate the original motto with disrupting established business models (see e.g. Jonathan Taplin’s book with the same name). Mark Zuckerberg himself, however, hinted that its origins lie in a particular attitude towards software engineering: “As developers, moving quickly was so important, we would even tolerate a few bugs to do it”. As time goes on, this attitude apparently does more harm than good – as even Facebook found out.

Via xkcd.com

“Move fast and break things”, in software engineering, signifies the need to quickly create functionality in the beginning of a product life cycle. It is related to the first agile principle: “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.” Functionality represents direct business value, and the “break things” part of the motto assigns lower priority to work that represents indirect business value, often called ‘enablers’.

Agile principles eight and nine, however, stress the importance of sustainable development and technical excellence, and the ability to “maintain a constant pace indefinitely”. Enablers create and maintain the architectural backbone of a product and safeguard long-term quality. An organization that neglects enablers in favor of functionality builds up technical debt; the longer enablers are postponed, the higher the debt and its impact on productivity and quality. A product or organization in technical debt cannot maintain a constant pace indefinitely.

All of the large organizations I encounter in my daily work are struggling with a burden of technical debt in one way or another. In many cases, the pressure to “move fast” has led to underfunding of enablers like ‘under the hood’ improvements and technical debt reduction. They have moved fast, and now things are broken.

How did we get here?

Discussions with architects, product owners and managers reveal some potential causes for the technical debt burden:

• ‘Regular’ modern business practices: KPIs (Key Performance Indicators) often lead to prioritizing short-term, measurable success over long-term investments.
• Overinflated stakeholder expectations: temporarily postponing enablers (often for good business reasons) leads to a steep growth in business value, delighting stakeholders. Teams then become reluctant to break the news that they cannot maintain this pace, forcing themselves to keep it up and build up more debt.
• Cargo cult: the success of Internet giants prompts organizations to adopt their methods, practices and frameworks without properly considering the differences in culture and business model (and when the methods fail, their proponents call for even more extreme implementations, because “we must have been doing it wrong” of “we were sabotaged by non-believers”).
• Misapplication of specific agile practices, such as:
  • WSJF (Weighted Shorted Job First) prioritization (specific to SAFe, the Scaled Agile Framework): due to their size and indirect business value, enablers generally cannot compete with features and functionality in WSJF prioritization. This is why SAFe advises to allocate capacity for enablers separately; ignoring this advice leads to misapplication of WSJF to enablers, and build-up of technical debt.
  • MVP (Minimum Viable Product): an initial product with just enough features to satisfy early customers, and to provide feedback for future development. It is in the nature of MVPs to ignore ‘edge cases’ (rarely occurring uses of the product) and some non-functional requirements (NFRs, e.g. limited scalability). However, edge cases and NFRs can have major impact on the architecture. Continuing development based on an MVP without properly considering (or completely rebuilding) the architecture is a misapplication of the MVP practice that leads to build-up of technical debt.

Product evolution based on a Minimum Viable Product may require rebuilding the architecture

The road ahead

During the panel discussion on the “Death of the Architect” at this year’s SATURN conference, I joked that the world is drowning in a pool of technical debt. That was a bit overdramatic, but nobody can deny that we have a problem. Not only is the technical debt burden draining businesses’ resources away from their ability to innovate, it is also causing real problems in society – that is how dependent we have become on properly working software. Architectures that do not support edge cases can even have significant ethical implications by excluding groups of clients and citizens from (sometimes essential) services (a number of cases like this are documented extensively in the Kafka Brigade’s ‘The Digital Cage’, currently only available in Dutch).

Technical debt causes real economic, societal and ethical problems. It is time we did something about it – starting by addressing the root causes mentioned above. I do not have the illusion that we can easily change the short-term focus in modern business governance practices, but we can certainly be more transparent in managing stakeholder expectations about sustainable development. We can also be smarter in applying agile practices – most of which is making sure we understand the practices and their limitations before we apply them.

Paying more attention to enablers means there will be less capacity for new things – initially, we will slow down. But maybe that’s a good thing. Grady Booch recently tweeted the question “What does software development look like in a post-agile world?” I hope we will slow down and fix things.

A few years back, I discussed the business case for reducing technical debt, and the importance of accounting for the risk exposure in that business case. However, there is another item in that business case that deserves some attention: the opportunity cost, defined by The New Oxford American Dictionary  as “the loss of potential gain from other alternatives when one alternative is chosen.”

When a Dev team spends resources and time on reducing technical debt (upgrading, refactoring, repairing), the team will produce fewer end-user stories during that time. Opportunity cost represents the business value that those end-user stories would have yielded, as a way of accounting for the scarcity of the team’s resources.

The literal term ‘opportunity cost’ is seldom heard during technical debt discussions, but it is often a major factor in deciding when to reduce the debt. Whenever a stakeholder (e.g. a product manager) says something like “Yes, we should do something about this debt, but we cannot afford to do it now”, she is probably referring to the business features that end-users are waiting for, or that have been promised before a certain deadline. In other words, the opportunity cost of reducing the technical debt – the potential gain from the alternative of delivering the business features on time – is higher than the interest on the technical debt incurred during that period.

The figure is an attempt to illustrate the opportunity cost by comparing two scenarios: in scenario 1, the technical debt is not payed back, and in scenario 2, the debt is payed back in release 1.2. The value curve at the top of the figure makes a little dip in scenario 2 (dashed line), compared to the continued growth of scenario 1. Using Philppe Kruchten’s backlog color coding, the figure shows that in scenario 1, release 1.2 introduces five new (green) user stories, while in scenario 2, there is only time for one user story because we have spent the rest of the resources on reducing the (black) technical debt. The gap between the dashed and the solid line represents the opportunity cost of reducing the technical debt. (In case you are wondering why the dashed line goes down in release 1.2, even though we are adding a user story: I always feel that existing business features in a solution are subject to some kind of value decay, due to growing expectations and demands from end-users – debatable I know, but beside the point of this blog post).

Example

A good example of opportunity cost in architectural technical debt reduction was presented to me by attendants of an RCDA Practitioner Course a few months back. In their organization, a team had been developing business process automation features for 4 years. The organization had kept track of the labor cost savings attributed to that automation effort, which amounted to 9 FTE (full time equivalent positions) per year on average. The platform the software was running on was due for a major overhaul, because it could not easily be made compliant with new European Commission regulations (most notably GDPR). During the overhaul, they would not be able to develop new features – meaning an opportunity cost equivalent to 9 FTE per year, or 0.75 FTE per month spent exclusively on the overhaul. A significant opportunity cost, but it was determined that the risk of non-compliance outweighed the opportunity cost, in favor of the overhaul.

In conclusion: if you need to draw up a complete business case for taking care of a piece of technical debt, make sure you include the opportunity cost on the costs side. This will help to facilitate a rational discussion about the impact of delaying features, putting this (architectural) choice in its business context. And while you’re at it, don’t forget to include the reduced risk exposure on the benefits side!

For architects designing complex solutions, a well-documented set of requirements can never be the sole basis of the architecture. Architects have to undestand why these specific requirements are relevant to their stakeholders, and most likely will have to ask “why?” again and again to understand the stakeholders’ real needs, so as to design the architecture that best fulfills those needs and the goals behind them.  When talking to architects about their work, I often summarize this principle as “Architecture is Context”.

The importance of context

What makes it so important for architects to understand the wider context of their solution? When we view architecture as a set of design decisions, a good measure of a decision’s architectural significance is the economic impact of that decision on the collective stakeholders affected by it (see RCDA: Architecting as a Risk- and Cost Management Discipline). So in order for architects to understand the significance of their decisions, they need to have a firm grasp of the economic context of the solution they are architecting.

A second reason architects need to understand context and background is because requirements (especially non-functional or quality attribute requirements) often turn out to cause conflicts in design decisions: for example, making certain company data available in a mobile app may be good for  productivity, but bad for security.  Architects can only make these types of trade-offs if they not only know the requirements, but also fully understand the business and technology drivers behind them (asking your stakeholders to prioritize NFRs doesn’t really help, since this quickly becomes a meaningless exercise if it is done separate from the design – see Issues Dealing with Non-Functional Requirements across the Contractual Divide).

Flavors of context

A complex solution has many types of stakeholders, each of which have their own goals and needs, contributing to the integral solution context. When focusing on economic impact, the first thing that comes to mind is the business context defined by the business stakeholders, who pay for or benefit from the solution. Architectural significance, however, is determined by a much wider range of stakeholders. Examples are operational and delivery stakeholders, for whom we need to understand the technology and project/release context, and citizens affected by the solution’s safety, security and privacy context.

Understanding context

What can architects do to improve their understanding of a solution’s context, and design architectures that better fit their stakeholders’ underlying needs?

• Talk to stakeholders. Architects who talk to their stakeholders get a better understanding of their context – well duh, talk about kicking in an open door. And yet… I still run into architects that are so focused on documentation, or on a particular subset of stakeholders, that they forget to interact with the rest. This goes both ways: being too exclusively focused on your technical stakeholders (“architects must code!”) is as bad as only being interested in “the business” side of things. Make sure you speak your stakeholders’ language – watch Jochem Schulenklopper’s SATURN 2015 talk “Why they just don’t get it” if you want to know how. Without actually talking to stakeholders, my other two tips (modeling context and trace decisions to context) lose most of their value.

• 

  Example of a system context diagram

  Modeling context. Modeling system context has been part of software design methodologies since the early days. In the 70s, the Yourdon Structured Design method involved a context diagram, which showed which external systems and users interacted with a system – a technique many (including me) still consider essential for clarifying the solution boundary and external dependencies. In fact, the first C in Simon Brown’s C4 model stands for Context, represented by just such a context diagram. UML’s use case diagrams fulfill a similar role (but much less clearly). Michael Jackson’s problem frames go a step further by modeling not just physical and logical context of the system itself, but also the wider context needed to understand the design problem to be solved.

• Trace design decisions to context. Once we have gone to all the trouble of understanding the context of the solution we are architecting, it is paramount that we record the impact of that understanding on our architecture. The perfect place to do that is in the ‘rationale’ section of our architectural decision record. Here is where we show that our decision is based on something in the real world, real concerns, goals or other drivers from our stakeholders. This not only helps us get buy-in for the decision, but also brings huge benefits if the context changes (or should I say “when the context changes”, because it usually does). If we have to revisit an architectural decision, knowing the full context of the original decision rationale makes re-appraising the trade-offs much easier. Putting it in economic terms, tracing your architectural decisions to their context lowers the cost of change – making your architecture more agile.

In conclusion, context is key in architecture, and there are several ways in which architects can improve their understanding of context and its impact. The most important tip is to talk to your stakeholders, and keep asking them “why?”. What is your take on this? Do you always draw a context diagram? How much context do you put in your architectural decision records?

One of the things architects can learn from the Agile mindset is the importance of short feedback loops. The quicker an architect receives feedback on their output, the faster they learn about its effect in their specific solution context – and better informed architects make better decisions. Architecture is a matter of reducing uncertainty by gathering knowledge and making decisions, and a shorter architecture feedback loop speeds up that uncertainty reduction, leading to better architectures. On top of this, shorter loops lead to shorter reaction times when things change, which increases agility. In this blog post, I will share some tips for shortening the architectural feedback loop.

Architectural decisions are your primary deliverable

What does your organization consider the primary architectural deliverable? Chances are it is a document with a name like Project (Start) Architecture or Software Architecture Description – let’s call it The Architecture Document. It takes weeks or even months to produce, after which time all architectural models and decisions are approved and distributed in one foul swoop – The Architecture Document version 1.0. If you are in this situation, the first step to shortening your feedback loop is to start viewing individual architectural decisions as your primary deliverable. The finer granularity of these decisions, compared to The Architecture Document, will make it much easier to speed up the feedback loop.

Continuously share concerns and decisions

Do not be afraid to share your unfinished architecture output. The sooner you share, the faster you learn. I like to let my stakeholders (both business and technical) know what I think are the most critical architectural concerns, and share decisions as soon as I am aware that they need to be taken. (One word of caution: always make sure the status of the decision-in-progress is absolutely clear to prevent people acting on them prematurely.) This gives stakeholders the opportunity to contribute to the architectural process from the start. Make this information as easy as possible to access: don’t put it in text documents, but find a low-threshold platform like a Wiki or issue tracking system.

Invite immediate feedback from stakeholders

We are often hesitant to ask for feedback on something that we ourselves do not consider to be perfect already. Architects, however, cannot afford to wait that long. Most architectures emerge from a dynamic process of frequently identifying new concerns, repeatedly finding out new facts and continuously adjusting partial decisions that interact with each other and our context. The perfect architecture is only known after the solution has been delivered (if then).

So tell your stakeholders to give you feedback when they have it, and not to wait for ‘official’ review moments like a ‘version 0.9’. Make it as easy as possible for them: enable the comments box on your Wiki, invite people to email you. If big print-outs of your architectural models adorn the team’s war-room, make sure to put a pencil on a string next to them. Make sure everyone knows you welcome feedback by asking for it at the water cooler, or on your team chat platform if you are not co-located.

Simplify your architecture documentation template

Template bloat is one of the causes of long architectural feedback loops. Organizations often lack a decent repository for architectural knowledge, and abuse their architecture documentation templates as a dumping place for all lessons learned. Diverse stakeholder concerns for all types of solutions end up getting their own sections in the template. On top of that, architects for whom The Architecture Document is the only place to store knowledge about a solution cause even more bloat. There are two things you can do to fight this document obesity:

• Create a template with sections for only the most common concerns at only the start of a solution’s lifecycle. Get rid of all ‘placeholder’ sections. Add views for other concerns only as they become significant later in the lifecycle, and only insofar as they are relevant to your context – in short, create living, minimal architecture documentation.
• Make sure you have another repository for knowledge that is not immediately relevant at the current point in the solution lifecycle. Create a wiki or library for the organization’s lessons learned and documentation plug-ins for views to be added later in the lifecycle, and find a place outside of The Architecture Document for solution-specific insights that you have gathered.

Get involved in delivery

The final tip for an effective, short feedback loop is to get involved in the delivery of your solution. If you are a software architect, coding key parts of the solution yourself is a great way to get involved. If you do not have that opportunity, get involved in integration, quality attribute testing or other architecturally significant delivery activities. You will not only become your own feedback channel, but also stimulate other delivery team members to tell you about their concerns and help improve your architecture.

In my work of coaching organizations to approach architecting in an agile way, shortening the architectural feedback loop has proven to be one of the most effective ways to improve architects’ effectivity and business value. Especially its positive effect on the scability of architecture work and the start-up time of smaller projects is quickly noticed and appreciated by business stakeholders. Let me know if the five tips in this post prove to be useful to you as well.