Manual Processes and ‘Non-Information’ Technology

How do we ensure that the enterprise architecture truly covers the enterprise? Many architects find that many EA frameworks are very oriented towards managing the IT aspects of the architecture and do not provide enough support for the non-IT things.

How do we ensure that the enterprise architecture truly covers the enterprise?

Many architects find that many EA frameworks are very oriented towards managing the IT aspects of the architecture and do not provide enough support for the non-IT things. I suspect that there’s an element of “the squeaky wheel gets the oil” here as managing and delivering IT systems has been and continues to be one of the most tricky activities that organisations do. However, I don’t think it helps to separate IT-focussed enterprise architecture initiatives from the broader EA discipline, rather the former is a part of the latter. We designed the Essential Meta Model to provide the constructs for capturing knowledge about all aspects of the enterprise (including the extended enterprise), where Information Technology is just one class of technologies that are used to support how the organisation functions.

Enterprise Architecture is about understanding and managing how the organisation works. This understanding enables us to make the right decisions both tactically and strategically in whatever issue is at hand. What information about the organisation do we need to capture? At the core of this understanding are people; processes; information that people need to perform the process; applications to support or automate the execution of the processes; and technology to support those applications. Pretty much all other knowledge about the enterprise refers to or operates on these core concepts. The last two things I mentioned (applications and technology) do not have to be Information Technology things. I’ll come back to these in a moment.

Is the knowledge we need just about what we can see in the enterprise? No, we need 3 main views of this knowledge, which I’ll describe in the terms we use in Essential Meta Model:

  • Conceptual. This defines ‘The What’, enables us to capture the requirements and very importantly, gives us a semantic grounding for all of the other elements that we will capture at the following views that operate at a lower level of abstraction.
  • Logical. This defined the ‘The How’ and is where we design how we see the enterprise working – either now or in the future.
  • Physical. This defines ‘The Where’ and ‘The Who’ and is where we capture what’s actually there in the organisation, such as individuals, teams, data stores, deployments of applications and technology instances (e.g. servers).

With these three views, we can then understand things like: how many ways do we do the same thing? Is there a good reason for that or should we be standardising?

OK, so this sounds all very much IT-oriented, and in many ways, the physical is particularly tricky for IT architecture for a number of reasons. First a start, physical applications don’t really exist as such in the physical world. The digital media arena demonstrates this, where physical instances of the product can be produced instantly and for no cost. Many applications – especially mash-ups and composites are run-time configurations of a distributed set of software components and other applications. What are the physical ‘bits’ (pardon the pun) of one of these?

We’ve had a lot of success with capturing and managing the physical technology architecture in the last year or so, taking clustered, virtual and distributed server architectures into account. However, we had to make sure we didn’t take any shortcuts with how these things really exist in the physical world.

So information technology things are well catered for but what about manual processes or processes that are performed by machines or by people using machines? How do we capture these in our meta model? Well, a manual process is performed by people we’ve captured in the physical view of the business layer. It’s likely that information is required to perform the process and we can capture that. (Note, ‘information’, not ‘data’). If it’s a manual process, surely it has no supporting applications defined for it. Or does it? Actually, the lack of information here cannot be used safely to make assumptions like this. Rather, we take exactly the same approach with ‘manual’ processes as we take with any other and capture the supporting applications and the technology used to support/deliver that application.

Processes can still be supported by ‘applications’ that are not delivered by Information Technology. For example, we might have a paper-based form that is used to support the process of capturing customer information. This form is the application and the paper is the supporting technology, which is stored in a filing cabinet instead of a database. Similarly, if we need to capture that a milling machine is involved in the process of manufacturing a widget, that machine is part of the technology architecture of the application (the behaviour of the control system of the machine) that supports that manufacturing process.

If someone described doing a manual process to me, I would ask “and what do you use to do the process?” Even if the answer is pen and paper or “I talk to the customer, using a script”. From the answers to these questions we get an understanding that we can choose to characterise as “manual”.

I don’t think that these examples are stretches of the terms ‘application’ and ‘technology’ if you think conceptually about what these are doing. As I mentioned earlier, the organisation is about people performing processes to meet the organisation’s objectives – to produce products, provide services etc. – and there are applications that support the doing of the processes (or could fully-automate the process) and these applications are supported by technologies. It doesn’t matter if this involves Information Technologies or any other Technologies.

The point is that manual is a function of what supporting things you use to do the process. With the exception of fully-automated processes, most processes -including those supported by applications – are driven manually. Indeed, many so-called “manual” processes are still supported by office productivity tools.

Although this involves more modelling, it’s far more accurate to capture processes in this way than adding a “this is manual” attribute to the process. This gives us the objective knowledge in the model on which to later on base inference and derive knowledge from, long after the process was captured.

There are many other elements of knowledge that architects of all types need to capture about the enterprise in order to support their architectural processes. We are already working on expanding what we call the EA Support layer of the Essential Meta Model. The things in here refer to or operate on the elements in the Core meta model and so far we’ve found that this approach has elegantly handled all our requirements to date. But we’d love to hear about the sorts of things that you need to have in there.

What are the Essential Project Team up to?

If you’ve been keeping an eye on the forums you’ll know that we’re in the process of putting together a sample repository.  The sample will take a fictional investment management company as an example and show a thin slice through all the meta-model layers to demonstrate the linkages between the different architecture concepts.  We chose an investment management company as it’s an area we know well, but also because we think it is broad enough to be understood without having to be a domain expert.  We’re currently populating the model and are expecting to get it out in the next couple of weeks – time permitting.
Something we’ve been asked a number of times via the forums is for a pictorial representation of the meta model, showing the links within and between layers.  We have been working on this and it is nearing completion so we hope to have it up on the site within a week or so, again, time permitting.
With many people trying out Essential now, it would be great to hear views on it, and ideas as to where the tool, the website or the documentation could be improved.  Please do let us have your views on the Forums.

Where have all the graphical models gone?

Essential Architecture Manager takes a different approach to modelling your organisation’s enterprise architecture.

Essential Architecture Manager takes a rather different approach to modelling your organisation’s enterprise architecture. There are many benefits of this approach but the way of working may seem unusual – or even limiting – to architects who are more familiar with more graphical modelling tools.

Knowledge capture

The key thing to get to grips with in Essential Architecture Manager is that modelling your organisation is about capturing information about it in a structured way that builds a detailed knowledge base that can then be used to answer questions about the enterprise.

The repository in Essential Architecture Manager isn’t a repository of diagrams or of diagram elements. Rather the repository is a knowledge base of instances of elements of the Essential Meta Model. To capture this information, you simply complete the forms in modelling environment, Protege, which adds this new information to the knowledge base.

Far from being a low-tech solution, the form-based approach to capturing the information is very useful. We often found that with purely-graphical tools, we were building simple spreadsheet forms or the like to load groups of new artefacts into the repository. The Protege forms give us this straight away.

Where it makes sense, to, simple graphical forms are provided to make the capture of the information about your organisation as easy as possible, e.g defining business process flows, or technology architectures. However, these use simple graphical elements that help focus the user on capturing the information rather than which colours to use or how to lay out the diagram.

The benefits of this approach are that we don’t need to worry about particular diagramming notations, e.g. UML, to capture information. Simply complete the forms. This means that in many cases, the information can be captured even without an understanding of how the meta model works or any particular notation.

Don’t model the report – separating capture from reporting and viewing

The Essential Meta Model has been designed to represent detailed knowledge about enterprise architectures so that we can ask questions of it – many of which are answered by derived or inferred information.

We’ve consciously separated the capture of the information – modelling – from the reporting and analysis of the model in the knowledge base. By doing this, we avoid common scenarios where models are created that answer only the specific question that the architect had in mind when capturing the information – what we call modelling the report.

The two components of Essential Architecture Manager, the Essential Modeller [Protege] and Essential Viewer make the separation of the capture from the reporting and analysis.

Notation agnostic

Essential Viewer gives us the capability to present views of the model in any notation that we need – potentially multiple representations for different stakeholders. However, the real power of it is the ability to perform complex analysis of the modelled enterprise architecture and present this in forms that can be consumed by stakeholders in the format that is best for them.

So, what about graphical models?

It’s in Essential Viewer that the “graphical models” are to be found, dynamically created in the right notation for your requirements.

Essential Project Launch Success

The launch of The Essential Project has far exceeded our expectations in terms of number of visitors and number of downloads.

To date we have had over 1700 visits, with nearly 800 unique visitors to the website, and there have been about 200 downloads of the Essential Tool. The most popular pages on the site, apart from the home page, appear to be the downloads page, getting started tutorial, meta model documentation and the forums.

We realise it is early days and many people may still be investigating the tool, but we would love to hear any views on it, and where we can improve either the tool, the website or the documentation we have provided. If anyone has time, please do send feedback through the forums.

Identify the problems your EA needs to solve

In my opinion, when introducing enterprise architecture into an organisation the most crucial aspect is the identification of the problem you need it to solve.  And to do this effectively I believe that you need to clearly understand the aims and objectives of the business at that time.

We often find the EA team sitting within the IT organisation, but the team still needs to ensure that the identification of need is business led and that the EA is not used simply to document technical IT issues.

For example, it may be that the most important objective for the business at a point in time is to reduce costs to remain competitive in its pricing.  Enterprise Architecture can meet this need in a number of ways depending on the company in question, but one way will be to capture the current state in the application architecture layer to allow consolidation of applications and thereby achieve cost reductions.

However, this would be a wholly inappropriate response if the business need were to expand into new markets.  In this case the Enterprise Architecture should be identifying common processes and services that could be shared across divergent businesses to aid speed of change and agility.

This is at the crux of the reason that, to date, many companies are not satisfied with their EA initiative.  All too often the EA will follow a prescribed, often IT based, root; lets capture the current state first; lets create an application catalogue; lets sort out our integration architecture; without first determining what the business needs it to do.

Once the link to business need is in place the EA can begin to provide what the business needs and positive results will be quickly seen.

Once the business need is identified, it’s time to consider the tools, people and processes needed to support it.  In terms of processes we have identified two major contributing factors to a successful EA, ‘understanding’ and ‘control’.

By ‘understanding’, we mean having a coherent view of the business and IT assets of your enterprise readily available and accessible in order to support well-informed tactical and strategic decision-making.  As the amount of information to pull together is often large, tools with the right information captured will give the understanding part of the equation.

By ‘control’, we mean having appropriate processes and techniques in place to manage and co-ordinate changes to these assets in line with these decisions.

In general, many of the problems that an EA will solve will be IT related, but the link with the business is imperative to ensure that the problem is the most relevant at that time and, crucially, that the EA Team are seen as making a real difference to the business in a relatively short time space.

How can we exploit existing architectural information?

Even if you are just starting out with your enterprise architecture initiatives, it is highly likely that you already have architectural information about your organisation captured in some form or other. How can we exploit the existing architectural assets with Essential Architecture Manager?

In a bit of contrast to Jason’s last posting, I’ve taken a more technical perspective this time.

Even if you are just starting out with your enterprise architecture initiatives, it is highly likely that you already have architectural information about your organisation captured in some form or other. How can we exploit the existing architectural assets with Essential Architecture Manager? Can we automatically load these assets into the Essential Architecture Manager repository? And if we can do that, can we get information out of Essential Architecture Manager to be loaded into some of our existing systems?

These are questions that we quickly encountered when we started using Essential Architecture Manager in real organisations.

Potential Approaches

Since Essential Architecture Manager is built on Protege, we started to explore the tabs that were already available for bringing external data sources into your Protege project, e.g. the very useful DataMaster.

It is often the case that existing sources of information lack the structure of a formal meta model such as that in Essential and when they do, there are bound to be meta concepts that do not directly map to the Essential Meta Model.

We therefore find ourselves with a typical application integration mapping and transformation scenario. In order to import the information from the existing source, we may have to combine or split elements from the source information in order to map it to the target. We may also need to create inferred elements from the source information.

The existing plugins for Protege lacked this mapping and transformation capability, and so at first, where an import was only needed as part of the initial start up of the initiative, we found that we could construct transformation scripts, e.g. XSLT, to generate entries for the .PINS file of the Protege project and then paste them manually into the file – an approach known as ‘PINS hacking’ in the Protege community.

While this solved some immediate problems, we identified a number of problems with this approach, such as importing into projects using a database backend (there’s no .PINS file to hack!), creating new, unique instance IDs in Protege, and of course on-going synchronisation between the Essential repository and updates to the external information sources.

The Solution

It was clear to us from this point that in order to reliably and predictably import information into Essential Architecture Manager, we needed to be interacting directly with Protege through its API and not through its underlying data store – good practice for any data integration really. This way, Protege could take care of creating unique instance IDs,  defining relationships between imported artefacts etc.

We found that through the Script Tab, we could “drive” Protege, via its API, in a way that effectively automated the steps that you would do if you were using the front-end GUI. This way, we knew that Protege could properly manage the integrity of the repository. We just needed an effective way of turning the source information into Protege scripts that we could run and we’d have the basis of our solution.

We’ve now been using what we’re calling the Essential Integration Server for several months to synchronise the Essential Architecture Manager repository with an external configuration management suite. You may have noticed that every class in the Essential Meta Model has a slot called ‘external_repository_instance_reference’. This is used by Essential Integration Server scripts to synchronise individual assets between Essential Architecture Manager and one or more external information sources. An instance in Essential can have multiple external references and we can combine information from multiple sources to build a more complete picture of an architectural asset in the Essential repository.

The mapping between the existing information source and the Essential Meta Model is put together by defining an XSLT (or any similar approach) that writes the import script. We are building a library of useful Python script functions that help with things like creating instances (or returning a reference to it, if it already exists in the repository) or building certain more complex relationships. Using the script language is fairly straight-forward and is almost as productive as a more graphical integration tool – mainly because we have the power of the full Protege API plus a rich scripting language that enables us to handle any import / integration scenario. The Essential Integration Server provides a web-based user interface for running these XSLT scripts on the source data and producing the resulting Protege scripts automatically. This is particularly useful when you are running the import on a regular on-going basis.

However, the Essential Integration Server is not quite fully automated and that’s why we haven’t released it yet in the same way as the other components. Currently, it does everything except run the scripts for you in Protege’s Script tab. That’s the manual step that we are working to automate at the moment.

Getting information out of Essential

I’ve described in some detail how we recommend that existing information is imported or synchronised into Essential Architecture Manager. How about getting the information that is in Essential out to be used by other systems?

Essential Viewer already provides that capability. Rather than producing a ‘report’ that renders HTML, you can simply build a report that produces, XML, CSV files or whatever your systems need. We’ve used this with a lot of success and because it is run from within Essential Viewer, your target system (or your integration environment) can request this extract via HTTP as a REST-style web service.

Complete solution coming soon

The solution for getting information into and out of Essential Architecture Manager is there. It needs a little more work to make importing fully automated but what we have today is being used in anger on a regular basis in a real organisation right now.

If you need to get your existing information into Essential right now, let us know and we’d be more than happy to supply the current version of Essential Integration Server and to help you build the mapping to the Essential Meta Model.

PowerPoint and Excel for Architecture Modelling; Why Not?

Whether planning, designing or executing IT change, there are plenty of situations that require discovery and analysis of architecture related information.

Often, when a piece of work is viewed as a one-off exercise (e.g. IT solution architecture design, architecture reviews), we see a prevalence of architecture information captured using any combination of Word documents, Excel spreadsheets, presentation slides and Visio diagrams as project outputs.

In the short term, these formats appear to serve their purpose; they provide a means of capturing architecture elements and relationships for both analysis and communication. But why is it that over time, they ultimately become inefficient and cumbersome when used as a means of retaining architectural knowledge?

What You See is What You Get: If you want a new perspectives on the information captured, then you will usually find yourself drawing another diagram or designing another spreadsheet. Over time, maintaining consistency across these different views of your architecture becomes increasingly difficult.

The Nth Dimension: Here, we’re referring to the challenges associated with capturing complex multidimensional inter-relationships using office productivity tools. Even when armed with the rows, columns, formulas and scripts afforded by a spreadsheet, it would certainly be a non-trivial exercise to map the applications, underlying technology, information exchanged and business processes involved in, for example, an organisations’s global integration architecture?

Manual Meta-Model: Even if you do your best to ensure that common terms are used across your spreadsheets, documents and diagrams, the fact is that these formats lack any meaningful ability to share fine-grained information elements without some form of programming. In other words, it is left up to the project or architecture team to manually enforce a standardised, shared meta-model for consistent semantics. Not a very scalable approach.

Repository-based tools to the rescue then? Well, they are certainly capable of addressing most of the maintainability and scalability issues associated with using office productivity and drawing tools for architecture modelling. However, I still often find myself asking the question:

“Why do even the most experienced architects (with access to sophisticated repository-based tools), fail to resist the temptation to go back to good old PowerPoint, Excel and Visio”

I don’t believe there is a simple answer to this question, but I ‘m pretty sure that beyond familiarity, it is simplicity and ease of use that draws us back to them; characteristics that are not always the first to spring to mind when working with repository tools.