The Devil is in the Detail

If architecture models are to be used to help make decisions about change in your organisation, those models must include details that reflect complexities of the real world.

The Problem

I am frequently asked what is becoming for me a classic question, “How far should I go with the detail of my model? When do you stop?”

In general, I would say the more information you can capture, the better. We do need to be able to “see the wood for the trees”, though. We don’t want to get lost in the detail – both from a point of view of the sheer volume of information and also from the perspective of capturing things that are not currently interesting.

Many models and frameworks tend to stay at quite a high level. This can be OK for some scenarios – e.g. high level design or a Powerpoint. However, detailed decision support enabled by information from the architecture model requires the detail. If this is not available then all too often, when the details come to light, they break the model.

When interacting with other parts of the organisation, architects who gloss over the details, can often be perceived as ‘arm wavers’ and can quickly lose credibility – even though the ideas themselves may be sound.

In the increasingly technologically educated world, you ignore the details at your peril. The very systems that we have to manage, as Application and Technology Architects, are becoming more and more complex. The Cloud, SaaS can provide significant benefits but the services that they provide still need to be managed as part of the Enterprise Architecture. The ‘details’ are still there but now we’re not directly in control of them. We cannot just ignore them because we have a service provider that manages how that service is delivered.

Really, we need to be able to ‘roll up’ the detail when required. Also it would be great to be able to easily come back to elements of the architecture that we didn’t initially capture to any great depth, and enrich the detail later, when we have it.

I’ve used some technology examples above, but the need to recognise the details exists just as much in Business Architecture. One of the big issues here is that we cannot ignore IT when looking at the Business Architecture. It is the reality of today that nearly every process in the organisation is supported by IT in some way.

A Way Forward

So, how can we manage the details without suffering information overload or finding ourselves in a modelling equivalent of painting the Forth Bridge?

Generally, details are missed because when we uncover them, we have nowhere relevant to capture them. So they are ignored or forgotten. If we could capture all the relevant details as we find them – and model them so that everything we know about some aspect of the enterprise can be related as appropriate to others – this would really add to the organisation’s ‘knowledge base’.

Now, modelling of this nature requires a detailed ‘language’ if we are to be able to reflect the nuances of things at the detailed level. However, this language must enable us to “black box” elements in the model – to save us from getting lost in the detail and deal with them at the right level of granularity for the activity that we’re working on right now.

The important thing about not ignoring the details is that our model needs to reflect the real world if we are to be able to make effective decisions from the knowledge we have captured. This might mean that the modelling activity or the model itself requires some complexity. We cannot hide from modelling complex things when they are truly complex in the real world.

The ability to capture the required details yet roll elements up as “black boxes” is one of the key drivers behind much of the development of the Essential Meta-Model. We started with a simple set of core concepts and introduced additional concepts incrementally to manage the detail as we found we needed it.

Don’t get lost in the details of your enterprise. Embrace them and manage them locally, in their relevant context (don’t worry about the servers if you’re looking at how your applications support your business processes!). Understand how these contexts, which are the layers and views of the Enterprise Architecture, fit together and the details will take care of themselves.

1,000 downloads achieved!

The 1,000th download of The Essential Project was achieved a couple of weeks ago, just 4 months after launch!

Reaching the 1,000 downloads so quickly has confirmed that there is a huge demand for a new breed of enterprise architecture tool, and we believe that the free, open source licensing approach we’ve taken is at the forefront of this demand, enabling SME’s as well as large global organisations to reap the benefits of enterprise architecture.

We’re also delighted in the continuing growth of the Essential Project Community, with members hailing from all over the globe and from any number of market sectors, ranging from Academia and Government Agencies to Financial Services and Healthcare.  Judging by the diverse nature of the community forum discussions, use of the tool is also wide ranging: from supporting initiatives focused on managing application and technology architecture to more business and information architecture focused activities.

Our hope on launch was that the website would grow to provide a community where ideas, successes and failures could be shared, both in the use of and extensions to The Essential Project and on Enterprise Architecture in general.  The figures to date would suggest that this is beginning to happen, and we thank you all for your interest in the project and hope that the site and the tools are proving useful to you.

As always, we welcome any help people are willing to offer to further discussions/requirements on the Essential Project forums, or with developing the Essential Project meta-model and enterprise architecture management tools.

Essential Architecture Manager and Project Portfolio Management (PPM)

I was in discussion the other day with a project manager about where Essential fits with Project Portfolio Management (PPM) – There are lots of interpretations but we talked around the PPM model described by F Warren McFarlan (See this article for a very brief overview Putting the Project Puzzle Together).

If you consider all the factors that influence a project portfolio, which generally comprise both business and technology elements, it often gets trickier to define the tangible impacts and the value as you move towards the technology part of the discussion.  Knowing that improving process X allows you to cut resource is only half the story; it becomes much more compelling if you also know that it means you can remove application Y and reduce the server costs and storage costs associated with it.

We think that when you consider the questions that you are trying to answer from the technology side, the route to the answers become clearer (and what you need to capture into Essential becomes evident).  We came up with a list of ‘IT’ questions which we thought should feed in to the project portfolio debate:

  • What processes does an application support? If we change the process then what is the impact?
  • What technology does an application use? If we change or remove an application then what technology does it sit on, what storage does it use, is it replicated across environments?
  • Where do we have common application services?  Are we leveraging what we have got, or rebuilding something from the ground up?

There are lots more questions we could get into, but as a base we thought this would give us enough to start to build more compelling cases to feed into the PPM initiative.  We may tie costs to some of these, some of which may be a best guess, but some of which would be accurate.  We think that eventually we could end up getting the enterprise architecture tightly-coupled with the PPM processes, which will no doubt help with demonstrating the value of the EA effort.

We also decided that to start with we would focus on the areas where we knew the business already had project ideas, and we would start to populate Essential around those project ideas and the questions we came up with.

It will be interesting to see where we get to with this.  If anyone has other experiences or ideas then do let us know.

Continued Success of the Essential Project: Open Source Enterprise Architecture Management Tools

The interest in the Essential Project is continuing and we’re fast approaching 1,000 downloads of the Essential enterprise architecture management tool. We should hit this milestone in the next few weeks and we will issue a press release to celebrate this achievement.
In looking at the stats, we’ve found that we have a large number of people who are using the site on a regular basis. The chart below is from Google Analytics and it shows the number of times that people have re-visited the site, so you can see that, for example, 607 people have visited the site between 26 and 50 times.

Google Analytics Essential Project repeat visitors

We guesstimate that we have at least 1,600 active users which is a good community of users so early in the life of the Essential Project. It’s great to see that the Essential Project site and enterprise architecture tools are proving to be of use to lots of people, and we do appreciate the feedback that we get. We would also welcome any help people are willing or able to offer to further discussions/requirements on the Essential Project forums or with developing the Essential Project meta-model and enterprise architecture management tools.

If something’s hard to do, then it’s not worth doing – Homer Simpson

The above quote from Homer Simpson could be applied to a lot of people when they consider embarking on Enterprise Architecture for their organisation.  Luckily, I’m not one of them!
In principle Enterprise Architecture isn’t hard at all.  I was trying to explain what I do to a friend of mine who is a builder/runs a swimming school and he couldn’t really understand the issues.  It seemed inconceivable to him that an organisation could have little understanding of the applications they are running and all the links between them; or no understanding across the organisation of the business strategy and what changes this would require from the business and IT communities; or be running two applications that actually do the same thing; or have two processes that achieve the same result.
This, of course, is symptomatic of the difference between a small business and a large global organisation, but it highlights that in principle Enterprise Architecture is a no brainer.  You simply need to understand the processes that your people are going to perform to achieve your business strategy, the applications that will assist them, the information that the people and applications will require and the underlying technology that will allow the applications to run.  Once you have this understanding, making adjustments, whether to streamline your business or because of a change in your strategy, is easy.
Of course, in practice ‘doing’ EA is never easy, and the ‘understanding’ element is only part of the equation.  Even if you understood all the elements listed above, achieving the ‘control’ required to manage change is still a significant challange.
Nonetheless, contrary to Homer’s advice, even though doing EA is hard, it is definitely worth doing.

How can organisations afford the luxury of enterprise architecture in the middle of a global recession?

The current downturn is forcing companies and government agencies across the world to prune their IT budgets and shelve major business change projects. In this adverse economic climate enterprise architecture will often be regarded by management as an overhead activity of little relevance to the cost-cutting challenges they face. There are two main reasons why this attitude can prevail:

  1. Enterprise architecture is frequently associated with the design of the future operating model that will result from planned major changes in business processes. When times are hard, such long-term interests usually take a back seat and quick-fix initiatives take priority.
  2. By its very name ‘enterprise architecture’ implies building a comprehensive view of the entire business operating model. Organisations in survival mode will be more interested in surgical measures focused on selected areas of the business, than in the contemplation of elegant, all-embracing business models.

As a consequence, enterprise architecture activities may be seen as unimportant and easily expendable. Skills and knowledge built up over many years may then be lost as expert practitioners are laid off or else redeployed to take on work seen as having greater urgency.

The Essential Project Team believes such an outcome can be counterproductive, as enterprise architecture has a potentially vital role to play in enabling organisations to handle the challenges of an economic downturn. The key to this is to ensure that enterprise architecture activities are focused on what the business regards as high priority.

To address adverse trading conditions, the enterprise architect first needs to concentrate on the current state, rather than the future business operating model. Then the focus should be specifically on those areas where rationalisation of existing assets and business process simplification are likely to pay dividends in the short term. The Essential Architecture Manager has been designed to enable promising areas to be selected and modelled without the need to cover the entire spectrum of business and IT capabilities. And that modelling should be carried out only to the level of detail that is necessary to enable appropriate management decisions to be made.

We know that many organisations could make significant short-term savings simply by rationalising their technology platforms – for example, by eliminating redundant servers. The media company that provided the original beta test site for the Essential Architecture Manager during 2008 opted to use the package in the first instance to model and record its current technology assets. This exercise revealed inefficiency, duplication and obsolescence in a number of areas, and it highlighted possibilities for early savings to be made.
A similar case can be made for organisations using tools such as the Essential Architecture Manager to record details of current IT applications and the business activities they support. This can reveal scope for rationalisation and simplification, which can lead in turn to significant benefits – for example, savings in licence fees and software maintenance charges. While working with a financial services company, one member of the Essential project Team used such a targeted architectural approach to demonstrate that a range of new financial products could be supported by the firm’s existing systems: this obviated the need for a proposed new suite of applications.

In short, now is the time for smart enterprise architects to lift up the drain covers, rather than scan the distant horizons. Those who are sensitive to current business priorities will hopefully then still be around to contribute to and support a more strategic agenda when better times return.

Training in the Essential Project

We’ve been asked several times about training.  Obviously training can cover a broad range of topics.  In considering what type of training we might offer, we have thought about what would be useful to the people that are using, or will be using, the Essential Architecture Manager and tried to stick to our aim of offering things that are effective and add value.

We don’t think we should offer training that covers best practices in Enterprise Architecture or modelling techniques, but short courses, with a hands-on practical approach, that show how to use the Essential Project tools to achieve specific business goals, with clear value.  We’ve had a quick brainstorm of ideas based on what people have asked for and where we see Essential delivering value in the current economic climate.  The courses we think would be of use are:-
1.    Business Operating Model Planning and Control
2.    Application Portfolio Management
3.    Technology Infrastructure Management
4.    Managing Project Architecture Dependencies within Programmes
5.    Strategic Enterprise Architecture Definition and Transition Planning
6.    Essential Viewer Custom Report Development
7.    Essential Architecture Manager Administration

We think these would be 1 or 2 day courses, with a clear business case for each course to help justify the training costs.

We are keen to gauge interest in training, both in the types of courses that we have identified and also in the delivery method and location.  Are the courses the right ones or are there others we should be looking to deliver?  What would be useful?  If you think we’ve got it wrong or can give us more details on your views, please feel free to add a comment to this blog.

Thanks in advance for taking the time to do this.

Developing The Essential Project Through Community

One of the main hopes we had for the project when we launched was that the community of Essential users would become part of the development process for the project. In this blog, I explain how you can contribute.

Developing The Essential Project Through Community

It’s only been just over a month since we launched the Essential Project and we’re very pleased and excited in the response. One of the main hopes we had for the project when we launched was that the community of Essential users would become part of the development process for the project. Already we’ve had some some very interesting requests, through the forums, for things that we haven’t encountered before but can handle through small extensions to the Essential Meta Model and Essential Viewer reports.

Why Contribute?

As the Essential Project tools are open-source, anyone is free to make any extensions that they require. We are very keen that as many extensions as possible are put back into the Project and shared with the community – to help make sure that the Essential Project has the coverage that is required.

By contributing your extensions and enhancements back to the Project, it not only drives the Project forward but means that any extensions that you need will be brought into and managed by the Project as it moves forward.

How will this work?

Obviously, we will need some sense of order to how the contribution process works. The Essential Project Team provide governance of the Essential Meta Model and co-ordinate requests for enhancements, taking other similar requests into account before designing and delivering them. Likewise, contributions of completed components, e.g. a new analysis report or meta model extension, can be shared by contributors through the Essential Project website. When a meta model extension is received, the Team will produce the update pack to make it simple for other community members to introduce the extension to their Essential installation.

However, we are really keen that requests for additional capabilities (modelling, reporting or software extensions) are developed with the community to make sure that as many different perspectives on the same requirement are considered. To make this work, we would like to operate a process in the spirit of the approach taken for the Java Community Process and run this via the forums, in the Enhancements and Suggestions forum.

What is the process?

The main thrust of the process – called the Essential Community Process (or ECP) is about capturing requirements and reviewing the proposed solution with as wide a group as possible. Although in many cases (especially in the early stages of the Project) the Essential Project Team will be the ones developing the solution, the process does not assume this  – it could be any member of the community.

The process can be iterative and will operate as follows:

  • Request
  • Discuss and gather further requirements from community
  • Design proposed solution
  • Circulate to the community for defined period of time
  • Implement
  • Test within the community
  • Review tests and approve
  • Release update


Clearly there’s a balance to strike between taking a controlled approach and the time it takes to produce the answer to the request. However, I think that we should be able to run the process such that the time taken is proportionate to the scope and complexity of the requirement being met (in some cases, perhaps only days). The important thing is to open up the gathering of requirements, design and testing activities to the community while making sure that the results fit in with the overall Essential Meta Model, can be taken forward and are easily shared.

We’re already looking at how we should capture software (or any other) licensing models and instances of these models, and we’d like to know more about requirements for organisation policy and compliance. If you have any thoughts on any requirements that you have in these areas, please let us know by taking part in this forum.

Sample Repository Coming (Very) Soon…

If you’ve been keeping an eye on the forums you’ll know that we’re aiming to get the sample repository complete by this Friday 24th April. Sorry for the delay, we had some EAS client work we had to finish which meant we couldn’t dedicate as much time to this as we’d hoped. So, it should be Friday and it is a very simple example based on a fictional financial services company. We’ll look to extend it over time and also will look for people to give us ideas on where to extend it to help show how things are modelled.

On another note, we’ve reached the milestone of 500 downloads which is great to see. It’ll be nice to see how people are using it. If you’d be willing to share a bit about what you’re doing, how you’ve found using the tools, then let us know and we’ll publish some experiences.

Manual Processes and ‘Non-Information’ Technology

How do we ensure that the enterprise architecture truly covers the enterprise? Many architects find that many EA frameworks are very oriented towards managing the IT aspects of the architecture and do not provide enough support for the non-IT things.

How do we ensure that the enterprise architecture truly covers the enterprise?

Many architects find that many EA frameworks are very oriented towards managing the IT aspects of the architecture and do not provide enough support for the non-IT things. I suspect that there’s an element of “the squeaky wheel gets the oil” here as managing and delivering IT systems has been and continues to be one of the most tricky activities that organisations do. However, I don’t think it helps to separate IT-focussed enterprise architecture initiatives from the broader EA discipline, rather the former is a part of the latter. We designed the Essential Meta Model to provide the constructs for capturing knowledge about all aspects of the enterprise (including the extended enterprise), where Information Technology is just one class of technologies that are used to support how the organisation functions.

Enterprise Architecture is about understanding and managing how the organisation works. This understanding enables us to make the right decisions both tactically and strategically in whatever issue is at hand. What information about the organisation do we need to capture? At the core of this understanding are people; processes; information that people need to perform the process; applications to support or automate the execution of the processes; and technology to support those applications. Pretty much all other knowledge about the enterprise refers to or operates on these core concepts. The last two things I mentioned (applications and technology) do not have to be Information Technology things. I’ll come back to these in a moment.

Is the knowledge we need just about what we can see in the enterprise? No, we need 3 main views of this knowledge, which I’ll describe in the terms we use in Essential Meta Model:

  • Conceptual. This defines ‘The What’, enables us to capture the requirements and very importantly, gives us a semantic grounding for all of the other elements that we will capture at the following views that operate at a lower level of abstraction.
  • Logical. This defined the ‘The How’ and is where we design how we see the enterprise working – either now or in the future.
  • Physical. This defines ‘The Where’ and ‘The Who’ and is where we capture what’s actually there in the organisation, such as individuals, teams, data stores, deployments of applications and technology instances (e.g. servers).

With these three views, we can then understand things like: how many ways do we do the same thing? Is there a good reason for that or should we be standardising?

OK, so this sounds all very much IT-oriented, and in many ways, the physical is particularly tricky for IT architecture for a number of reasons. First a start, physical applications don’t really exist as such in the physical world. The digital media arena demonstrates this, where physical instances of the product can be produced instantly and for no cost. Many applications – especially mash-ups and composites are run-time configurations of a distributed set of software components and other applications. What are the physical ‘bits’ (pardon the pun) of one of these?

We’ve had a lot of success with capturing and managing the physical technology architecture in the last year or so, taking clustered, virtual and distributed server architectures into account. However, we had to make sure we didn’t take any shortcuts with how these things really exist in the physical world.

So information technology things are well catered for but what about manual processes or processes that are performed by machines or by people using machines? How do we capture these in our meta model? Well, a manual process is performed by people we’ve captured in the physical view of the business layer. It’s likely that information is required to perform the process and we can capture that. (Note, ‘information’, not ‘data’). If it’s a manual process, surely it has no supporting applications defined for it. Or does it? Actually, the lack of information here cannot be used safely to make assumptions like this. Rather, we take exactly the same approach with ‘manual’ processes as we take with any other and capture the supporting applications and the technology used to support/deliver that application.

Processes can still be supported by ‘applications’ that are not delivered by Information Technology. For example, we might have a paper-based form that is used to support the process of capturing customer information. This form is the application and the paper is the supporting technology, which is stored in a filing cabinet instead of a database. Similarly, if we need to capture that a milling machine is involved in the process of manufacturing a widget, that machine is part of the technology architecture of the application (the behaviour of the control system of the machine) that supports that manufacturing process.

If someone described doing a manual process to me, I would ask “and what do you use to do the process?” Even if the answer is pen and paper or “I talk to the customer, using a script”. From the answers to these questions we get an understanding that we can choose to characterise as “manual”.

I don’t think that these examples are stretches of the terms ‘application’ and ‘technology’ if you think conceptually about what these are doing. As I mentioned earlier, the organisation is about people performing processes to meet the organisation’s objectives – to produce products, provide services etc. – and there are applications that support the doing of the processes (or could fully-automate the process) and these applications are supported by technologies. It doesn’t matter if this involves Information Technologies or any other Technologies.

The point is that manual is a function of what supporting things you use to do the process. With the exception of fully-automated processes, most processes -including those supported by applications – are driven manually. Indeed, many so-called “manual” processes are still supported by office productivity tools.

Although this involves more modelling, it’s far more accurate to capture processes in this way than adding a “this is manual” attribute to the process. This gives us the objective knowledge in the model on which to later on base inference and derive knowledge from, long after the process was captured.

There are many other elements of knowledge that architects of all types need to capture about the enterprise in order to support their architectural processes. We are already working on expanding what we call the EA Support layer of the Essential Meta Model. The things in here refer to or operate on the elements in the Core meta model and so far we’ve found that this approach has elegantly handled all our requirements to date. But we’d love to hear about the sorts of things that you need to have in there.