The Software Architect Role, Organizational Culture Fit, and Influencing Change
Organizations seeking a software “architect” usually have one of two very different ideas in mind about what an architect is and does and about where and how s/he can be effective.
Two Kinds of Architects
By far the most common idea organizations have is that an architect is just a “master builder” (the literal meaning of the Greek word “arkhi-tekton”), lead technologist, or very senior software engineer. This often results in fairly tortured, grab bag, one-size-fits-all role descriptions that may include everything from technical design responsibility to regular sprint deliverables — expectations that are inherently in conflict with one another. Good technical design requires deep, uninterrupted, strategic thought, while sprint deliverables are by their nature much more rapid and tactical. To require both from the same person is to seek a unicorn indeed.
Far less common is the idea of an experienced, hands-on philosopher and deep thinker, who combines deep technical experience with the creative imagination needed to synthesize sustainable solution designs that are functional, durable, and delightful. This kind of architect quickly recognizes patterns and commonalities and can usefully generalize basic system building blocks like business entities, processes, and decisions (or rules), machine learning models, etc by constantly probing with ontological questions like “What is this thing really?” S/he then asks questions like “Where should it go?” in order to create optimal structures, arrangements, or taxonomies with these building blocks. This kind of architect has the discipline required to discover or create proven solutions and to apply these proven solutions consistently and can communicate the nature and benefits of proposed solutions in both words and pictures. S/he is a calculated risk taker and is adept at listening carefully, empathizing with stakeholders, and creating prototypes that demonstrate important concepts to even non-technical stakeholders.
Confusion about these two very different kinds of “architects” is understandable. The hypothetical progression from engineer to very senior engineer or “architect” seems to make implicit sense, but architects and engineers actually have very different perspectives, priorities, and thought processes, all of which an organization would do well to have represented. Architects and engineers use many of the same tools and work side by side designing and creating technology. Many architects have been and to some degree still may be engineers, while some engineers may discover along their journeys that they are in fact architects, but engineers are generally not architects and vice versa.
It is vitally important not to conflate actual architects with even very senior engineers because engineers typically think in much shorter timeframes and much more tactical terms than architects, believe that the shortest distance between two points is always a straight line, often lack empathy for stakeholders (especially non-technical ones), and rarely aspire to create solutions that delight stakeholders, believing that “it works” is good enough. Engineers typically understand functionality and sometimes durability, but delight just doesn’t compute for them.
You can read more about the definition and practice of actual architecture in my article Defining Architecture.
Two Kinds of Organizations
Just as there are two kinds of “architects”, there are two kinds of organizations, and adding an actual architect to one of these two types is guaranteed to result in frustration for everyone involved.
To be clear, organizations actually exist in a number of different spectra, so there are actually many more than two kinds. One great way of classifying organizations is the Capability Maturity Model (CMM), which ranks organizations based on the way they manage their business processes. Level One organizations are chaotic, and they employ ad hoc processes that often depend on individual heroics to succeed. Subsequent levels require processes to be documented to the point that they are generally repeatable, to be defined in terms of standards, to be measured and managed quantitatively using well-established metrics, and finally to be optimized through analysis, automation, etc.
But the bottom line is that every organization chooses either to operate in the perpetual fire drill that is Level One or to be actively moving along the path to maturity, approaching Level Five, even if progress is slow — hence two basic kinds of organizations.
What Each Knows and Can Do
Chaotic organizations assume that everyone just kind of knows (or can quickly and easily discover or doesn’t really need to know or wouldn’t substantially benefit from knowing) what assets exist, where they are, what they’re for, when and how they are to be used, what everyone knows and has done/can do/is good at, what everyone else is doing and how they’re doing it, etc, thereby being susceptible to assumption and to repeatedly being penalized by not actually knowing and to becoming dependent upon “tribal” knowledge, preserved by a few “elders” of whom everyone somehow knows to inquire. I’ve found this to be true of very large, publicly traded, multi-national organizations as well as small “mom and pops” and everything in between. Despite this, these organizations are able to get things done, but timeframes, quality, cost, etc all usually suffer. Large chaotic organizations can often survive longer than small chaotic organizations due to what I call “institutional inertia”, where like a charging rhino they can continue forward long after they are no longer actually viable.
Maturing organizations create and maintain definitive lists, manifests, and query-able resources that make information and knowledge easily accessible, based on each member’s identity, role, and place in the organization. This investment enables quick asset discovery and makes accurate understanding, decision-making, staffing, efficient solution delivery, and independent analysis possible. These organizations can consistently deliver high quality solutions in relatively short timeframes.
How Each Works
Chaotic organizations assume that everyone understands and agrees on what needs to be done, how, and why. Because “everyone understands”, there is no compelling reason to write things down or to listen, read, or refer to documentation carefully, if at all. Despite ongoing, daily reminders that individual memory varies widely and fails often, decisions are discussed, made, forgotten (or inexactly recalled), discussed again, often made differently, etc. Terminology describing the same things varies widely, even within and between small teams, leading to confusion and imprecision. These organizations can figure out how to get something done but are unlikely to do the same thing in the same way they did it before. Meetings are improvised gatherings, inviting some who should attend but forgetting others, providing no agenda, outline, or meeting goals, and resulting in no defined, documented, actionable result.
Maturing organizations value and consistently invest in effective, considerate communication and work hard to establish and ensure a shared understanding by avoiding assumptions and identifying and reducing opportunities for others to mis-understand. Careful notes are taken, shared, and referred to consistently. Pictures (diagrams) are used to augment and further clarify written documentation. Crisp, consistently-referenced, and authoritative definitions (e.g. glossary entries) are created and maintained, establishing a common vocabulary. These organizations are realistic about the cost and the value of knowledge management and collaboration. Work is carefully defined, including acceptance criteria and what should happen in response to unsuccessful (or “sad” path) outcomes.
What Each Produces
Chaotic organizations are content just to get things to work acceptably (functional) so that they can move on to just getting the next thing to work acceptably, often in a completely different way. Things are “done” when stakeholder requests, objections, or complaints decrease or cease, often not because stakeholders are happy or even satisfied but because they give up or because another “fire” has broken out in another part of the organization.
Maturing organizations aspire to more than just getting things to work. They believe in and can support the value of creating solutions that are functional, that are durable, and that delight those who utilize and depend on them. They document, standardize, measure, manage, and optimize the processes used to design and deliver these solutions so that they are not only repeatable but constantly improve quality, reduce risk, and minimize finite resources like time and money.
Most of all, chaotic organizations pass up opportunities to become better at what they do every single day, despite ample evidence of the cost of being content to continue as they are. Even when lessons are learned or examples of improvement are demonstrated, these lessons don’t “stick”; the improvements are not internalized or made part of a documented, repeatable process. Members of chaotic organizations simply don’t care enough to improve or require external pressure to improve and for the most part resent and resist such pressure.
In the end, an honest accounting demonstrates the value of becoming more mature, and discipline determines those organizations that do and those that do not.
Two Kinds of Influence
As a result of all this, introducing an actual architect, who is driven to create sustainable, long-term solutions that are functional, that last, and that delight stakeholders by consistently identifying common patterns in requirements and advocating the implementation of proven solutions, into a chaotic, short timeframe, primarily tactical organization that typically lacks empathy for its non-technical stakeholders and rarely aspires to create solutions that do anything more than to more or less meet a set of relatively undocumented technical requirements is usually a recipe for frustration and wasted time and effort on both sides.
Innovation and even proven solutions to problems simply don’t “stick” in an organization unless the individuals who make up that organization really care about getting better at what they do. If the goal is simply to get something to work and then to move on to the next thing, real progress rarely occurs.
Architects are often brought in with no real authority to bring about change. Leadership often wants badly to believe that people will “do the right thing” once they’ve heard it explained, seen it demonstrated, etc, but in 20+ years as an architect and 30+ as a father of four, I’ve never actually seen people’s behavior change through influence alone (e.g. as a father, “Hey! Watch this, kids. I’m looking BOTH ways before I cross this street! That’s a ‘best practice’ you can apply in your own street crossing.”). Sadly, accountability is almost always required to bring about lasting improvements in the real world, and meaningful, effective accountability requires … you guessed it: authority.
An architect can accurately and comprehensively understand an organization’s problems and needs and write up proposals, draw diagrams, supply examples, and create prototypes and proofs of concept that explain the benefits of proven solutions, but unless the rest of the organization chooses (or is required) to look at, to engage with, and work to understand and implement these designs as intended, real, lasting solutions to difficult problems will “mysteriously” continue to elude the organization.
Why all this focus on consistency? Let’s take an example from sales, which is often about as far from engineering and architecture in an organization as possible. I’ve learned a LOT by talking with and listening to sales people, and there are a few sales people wandering the planet that can talk pretty convincingly about objects, microservices, and APIs thanks to talking with and listening to me.
Any competent sales professional will tell you that acquiring and qualifying a lead (someone who may purchase your product or service) is the most time and effort-intensive (and therefore expensive) part of the sales cycle. Once a buying decision has been made, an organization would do well to pay attention to and to cultivate that customer. But inexplicably, many do not, preferring to slowly and expensively develop, convert (sell), and then lose lead after lead in an agonizing game of attrition.
Taking approach after approach to solving what is essentially the same problem is much the same thing. Once the investment to create a proven solution has been made, using it consistently is the quickest, safest, and least expensive way to solve that particular problem. Any competent professional architect knows this and will do her/his best to influence your organization to find something that works (e.g. user story structure, microservice design, deployment automation, etc) and then stick with it.
So before you seek an architect, ask yourself (and take the time to honestly answer) the following questions:
- What kind of organization do you have?
- What do you think an architect is, i.e. which type of architect are you seeking?
- Why do you believe you need an architect? Did someone tell you you need one? You’ve noticed that everyone else has one?
- What do you expect an architect to do for your organization? Introduce change and improvement? Fix difficult and intractable problems? Lead the creation of new capabilities?
- How do you expect an architect to accomplish this? Pure influence (e.g. “Look, kids!”), influence with some authority, or exercise near complete authority in technical issues?
Your candid and realistic answers to these questions will determine whether adding an architect to your team is a success and an ongoing source of value or a frustrating waste of time for everyone involved.
