How can we exploit existing architectural information?

Even if you are just starting out with your enterprise architecture initiatives, it is highly likely that you already have architectural information about your organisation captured in some form or other. How can we exploit the existing architectural assets with Essential Architecture Manager?

In a bit of contrast to Jason’s last posting, I’ve taken a more technical perspective this time.

Even if you are just starting out with your enterprise architecture initiatives, it is highly likely that you already have architectural information about your organisation captured in some form or other. How can we exploit the existing architectural assets with Essential Architecture Manager? Can we automatically load these assets into the Essential Architecture Manager repository? And if we can do that, can we get information out of Essential Architecture Manager to be loaded into some of our existing systems?

These are questions that we quickly encountered when we started using Essential Architecture Manager in real organisations.

Potential Approaches

Since Essential Architecture Manager is built on Protege, we started to explore the tabs that were already available for bringing external data sources into your Protege project, e.g. the very useful DataMaster.

It is often the case that existing sources of information lack the structure of a formal meta model such as that in Essential and when they do, there are bound to be meta concepts that do not directly map to the Essential Meta Model.

We therefore find ourselves with a typical application integration mapping and transformation scenario. In order to import the information from the existing source, we may have to combine or split elements from the source information in order to map it to the target. We may also need to create inferred elements from the source information.

The existing plugins for Protege lacked this mapping and transformation capability, and so at first, where an import was only needed as part of the initial start up of the initiative, we found that we could construct transformation scripts, e.g. XSLT, to generate entries for the .PINS file of the Protege project and then paste them manually into the file – an approach known as ‘PINS hacking’ in the Protege community.

While this solved some immediate problems, we identified a number of problems with this approach, such as importing into projects using a database backend (there’s no .PINS file to hack!), creating new, unique instance IDs in Protege, and of course on-going synchronisation between the Essential repository and updates to the external information sources.

The Solution

It was clear to us from this point that in order to reliably and predictably import information into Essential Architecture Manager, we needed to be interacting directly with Protege through its API and not through its underlying data store – good practice for any data integration really. This way, Protege could take care of creating unique instance IDs,  defining relationships between imported artefacts etc.

We found that through the Script Tab, we could “drive” Protege, via its API, in a way that effectively automated the steps that you would do if you were using the front-end GUI. This way, we knew that Protege could properly manage the integrity of the repository. We just needed an effective way of turning the source information into Protege scripts that we could run and we’d have the basis of our solution.

We’ve now been using what we’re calling the Essential Integration Server for several months to synchronise the Essential Architecture Manager repository with an external configuration management suite. You may have noticed that every class in the Essential Meta Model has a slot called ‘external_repository_instance_reference’. This is used by Essential Integration Server scripts to synchronise individual assets between Essential Architecture Manager and one or more external information sources. An instance in Essential can have multiple external references and we can combine information from multiple sources to build a more complete picture of an architectural asset in the Essential repository.

The mapping between the existing information source and the Essential Meta Model is put together by defining an XSLT (or any similar approach) that writes the import script. We are building a library of useful Python script functions that help with things like creating instances (or returning a reference to it, if it already exists in the repository) or building certain more complex relationships. Using the script language is fairly straight-forward and is almost as productive as a more graphical integration tool – mainly because we have the power of the full Protege API plus a rich scripting language that enables us to handle any import / integration scenario. The Essential Integration Server provides a web-based user interface for running these XSLT scripts on the source data and producing the resulting Protege scripts automatically. This is particularly useful when you are running the import on a regular on-going basis.

However, the Essential Integration Server is not quite fully automated and that’s why we haven’t released it yet in the same way as the other components. Currently, it does everything except run the scripts for you in Protege’s Script tab. That’s the manual step that we are working to automate at the moment.

Getting information out of Essential

I’ve described in some detail how we recommend that existing information is imported or synchronised into Essential Architecture Manager. How about getting the information that is in Essential out to be used by other systems?

Essential Viewer already provides that capability. Rather than producing a ‘report’ that renders HTML, you can simply build a report that produces, XML, CSV files or whatever your systems need. We’ve used this with a lot of success and because it is run from within Essential Viewer, your target system (or your integration environment) can request this extract via HTTP as a REST-style web service.

Complete solution coming soon

The solution for getting information into and out of Essential Architecture Manager is there. It needs a little more work to make importing fully automated but what we have today is being used in anger on a regular basis in a real organisation right now.

If you need to get your existing information into Essential right now, let us know and we’d be more than happy to supply the current version of Essential Integration Server and to help you build the mapping to the Essential Meta Model.

PowerPoint and Excel for Architecture Modelling; Why Not?

Whether planning, designing or executing IT change, there are plenty of situations that require discovery and analysis of architecture related information.

Often, when a piece of work is viewed as a one-off exercise (e.g. IT solution architecture design, architecture reviews), we see a prevalence of architecture information captured using any combination of Word documents, Excel spreadsheets, presentation slides and Visio diagrams as project outputs.

In the short term, these formats appear to serve their purpose; they provide a means of capturing architecture elements and relationships for both analysis and communication. But why is it that over time, they ultimately become inefficient and cumbersome when used as a means of retaining architectural knowledge?

What You See is What You Get: If you want a new perspectives on the information captured, then you will usually find yourself drawing another diagram or designing another spreadsheet. Over time, maintaining consistency across these different views of your architecture becomes increasingly difficult.

The Nth Dimension: Here, we’re referring to the challenges associated with capturing complex multidimensional inter-relationships using office productivity tools. Even when armed with the rows, columns, formulas and scripts afforded by a spreadsheet, it would certainly be a non-trivial exercise to map the applications, underlying technology, information exchanged and business processes involved in, for example, an organisations’s global integration architecture?

Manual Meta-Model: Even if you do your best to ensure that common terms are used across your spreadsheets, documents and diagrams, the fact is that these formats lack any meaningful ability to share fine-grained information elements without some form of programming. In other words, it is left up to the project or architecture team to manually enforce a standardised, shared meta-model for consistent semantics. Not a very scalable approach.

Repository-based tools to the rescue then? Well, they are certainly capable of addressing most of the maintainability and scalability issues associated with using office productivity and drawing tools for architecture modelling. However, I still often find myself asking the question:

“Why do even the most experienced architects (with access to sophisticated repository-based tools), fail to resist the temptation to go back to good old PowerPoint, Excel and Visio”

I don’t believe there is a simple answer to this question, but I ‘m pretty sure that beyond familiarity, it is simplicity and ease of use that draws us back to them; characteristics that are not always the first to spring to mind when working with repository tools.