Developing Cross Organisational Cloud Solutions at Scale with AWS Service Catalog
What the problem looks like
Large organisations can often develop into isolated fragments of technical development over time. Teams working in one part may not be aware of what others are doing, even in the same building. From a technical standpoint the result is at best sub-optimal, resulting in duplication of work and reinvention of the wheel. Velocity is low as disparate teams build cloud infrastructure foundations again and again before starting on their actual projects.
This is highly damaging to any attempts at scalability of operations. At worst the results can be catastrophic, creating separate islands of technical expertise and control, with each part embedded in its own technology choices and resistant to external change and ideas. Inconsistent approaches lead to higher a maintenance burden for the business, trying to support many varying solutions rather than a few. The flexibility to react to change is diminished as more of this burden impacts, and development of fresh solutions stalls.
There are many ways of improving this disconnect, from cross-organisational stand-ups and demos to additional layers of roles designed to have broad oversight across many departments or areas. Fundamentally though there is a breakdown in the germination and sharing of successful, or unsuccessful, approaches that must be healed. Most often this is a cultural or structural issue within the business that involves people skills and organisational design. But there are also things we can do around our technical processes that encourage sharing, collaboration, and increased velocity around cloud architecture and service creation.
The Ideal World
How do we actually want this to work? Ideally we want to solve problems only once. Anytime anyone, anywhere in the organisation has the same problem to solve, they reuse one of the existing solutions and move on rapidly to add new value by working on new problems and features. As soon as something is created in one part of the organisation, it is instantly available to everyone in the organisation. We’d also like to be able to know that all the new solutions work to the same required standards, that patterns can be security accredited once and, barring significant changes, once only.
How it works
Luckily the AWS Service Catalog provides us with the backend tools that allows us to support this organisational functionality. Replicating good ideas instantly. Pulling in and pushing out solutions across an organisation in a simple and transparent way.
In other words, the Service Catalog acts as a central repository of known successful and verified patterns that have been created both inside and outside of the organisation. Anything in there can be made instantly available across all the accounts and teams in your organisation, from baseline security policy templates to full containerised service architectures. Not only are they available to be deployed in any of the organisations AWS accounts by teams using one-click, but in addition a register of what is installed, in which accounts, and at what version, is maintained by the Service Catalog. This gives complete oversight into the cloud estate of the organisation in real-time, and allows for updates to all deployed products to be pushed out to all accounts simultaneously.
Cultural change
We can see how this is not only a useful technical capability, but the functionality it provides can drive the businesses processes and culture. With the investment in a central catalog of cloud products we aim to pull in work from all over the organisation and from any teams. There is no longer a dominant central team responsible for creating templates that might not be aligned with what another team is trying to do, or being given the role of enforcing the mandated use of one catch-all platform for all use cases.
Instead the development of cloud products becomes a much more shared and collaborative affair, with a mix of different teams supplying an array of different components, re-using and creating where appropriate. We can also allow for more than one solution to a common problem to develop due to the low operational burden, allowing the best solution to rise out naturally as a result of real usage preferences over time, as illustrated by the catalog showing the deployment count of each product.
The downplaying of prescriptive solutions to challenges faced by teams helps to ease the tribalism that can form inside large organisations with many teams working in similar areas. Although there is an element of curation done centrally, and the automation of quality and security checks that allow for the continuous deployment of uploaded template. Overall the ownership of the catalog feels more shared, encouraging teams to let go of the repetition of the building blocks of their services, and feel the benefits of diving right into the more interesting and valuable higher level parts.
