Data Lens

You may have noticed from our site that the Data Lens is in beta.  It’s a lens that we’ve developed because we’ve been continually told that people don’t have control of their data.

In our EA consulting, we have seen:

  • Organisations that were unwittingly reporting incorrect MI figures because data was inaccurate or incomplete
  • Projects that intended to master and duplicate data that already existed in the organisation
  • Inconsistency in what people thought certain data was
  • Differing views on where data was sourced from
  • Projects repeating the same data collection work, asking the same questions again

The Data Lens looks to address this by bringing transparency and coherence to your data estate.  It is aimed at supporting the demands of people wanting to use data, such as:

  • Data Lake or Analytics efforts, which need to know information such as where data is sourced from, what terms are used for the same data, e.g. client and customer, how good the data is in terms of quality and completeness, etc.
  • Platform projects need to know where data masters exist, where data flows, how data is transformed, etc.
  • Any data rationalisation project needs to know where master sources of data exist, where duplication exists and how data is used.
  • Plus, Data Scientists need to understand the sources of data available for their analysis

The lens addresses these needs by providing a number of views and tools.

The Data Definition views provide data definitions, summaries and dynamically produced data models.

The Data Architecture Analysis views are geared towards you understanding sources of data, data flows, where duplication exists, etc.

Data Management is where the lens excels.  You are able to understand data quality across a number of criteria and see sources of data.  The Quality Dashboard shows the quality of the key data required to support your strategic objectives and business capabilities, and also the initiatives impacting that data.  This allows you to identify where your data initiatives may need to be focused to improve your business data output and enable your strategy.  The Data Quality Analysis page lets you pick the data you need and it then shows you where to source it from, plus the quality, completeness and accuracy of that data. This is really useful if you are using the data for other purposes, e.g. MI reporting or analytics. The data dashboard provides and summary view of your data which you can drill down into.

We see the Data Lens acting as the bridge between the tools that are more focused on the physical data layer, and which typically meet the needs of the technical teams but not the business users or the data scientists.  Equally, where you have conceptual data in a tool, the lens can act as the bridge to the physical data, removing the gap between the conceptual and physical layers, bringing context and meaning to the data.

The lens is currently in beta but we are allowing organisations to register an interest and we would love to get any feedback on the lens.

Essential View Loaders

We’ve come across a number of Essential users who have said they’d rather not spend time writing the import specifications to import data and anything to speed the import of lots of data to get the out-of-the box views working quicker would be helpful. In response, we’ve created some view loaders that will populate key elements of the views. This means that you can focus on gathering the data required and then just run the importer to load the data into your repository.

We’ve priced the loaders at a point that is low, so you won’t need to blow the budget on them, and we will be using the revenue to fund the Essential Project work. We’re intending to build more view loaders in the coming weeks and months, and would welcome feedback on which views you’d like us to prioritise. We’re also due a new dot release of Essential soon, which has some meta model changes and some updated views that we’ll be offering view loaders alongside.

We’re working some other things at the moment, although if you keep an eye on the site you can probably work out what, but more about that in the next few months.

IRM UK EA Conference – Outsourcing and EA

I presented a session on Outsourcing and EA at the IRM EA conference last week; specifically how, as Enterprise Architects, we are in a prime position to ensure that outsourcing deals are both created and run effectively as we are in the unique position of having the knowledge and understanding of both the business and IT across the entire enterprise.  We likened EA’s to the Spartans in the battle of Thermopylae who held off an army of (allegedly) a million men for seven days with only 300 warriors – primarily because they understood and had a map of the landscape.  (Unfortunately they were betrayed and slaughtered after a week – hopefully the analogy doesn’t stretch that far!).

Research by both Gartner and AT Kearney suggests that around 1/3rd of outsource initiatives fail.  We discussed how use of our architecture knowledge and artefacts can mitigate the risks of failure and how EA can be used to bring greater success.  We touched on our work to help organisations use EA and Essential together to reduce the outsource transition time (from idea to completed transition to a new provider) from a typical 18-24 months to 6-9 months, which addresses a key concern raised by the FCA.  We showed some examples of how Essential has been used to support such initiatives across a number of organisations.

The conference itself was very interesting and it seems to me that EA is really coming of age – there were many talks showing how EA is used in organisations to provide real and concrete benefit to senior managers.

If you would like a copy of the presentation then drop me an e-mail at the info at address.

The Data Visualisation Tools Arena

At a demo recently we were asked how Essential fits alongside data visualisation tools such as QlikView and Tableau.  Visualisation and BI environments are hot topics at the moment, with a lot of commentary appearing about them, so we thought we should share our view of how we see Essential playing in this field.

In simple terms, we see the split being between the quantative and qualitative view.  The quantative view gives you the numbers that tells you where your issues are and the qualitative view then gives you the information behind the numbers that allow you make informed decisions to resolve the issues.

The data visualisation tools’ key strength are in addressing the quantitative data requirements, so they are great at taking data and transforming it into meaningful statistics that can be used to formulate facts and uncover patterns in data.  If I have questions about how many interfaces there are between my applications, how many meet the SLAs, how many move client data etc., the BI visualisation tools are perfect.  Tableau will even provide some rudimentary views of connections.

On the other hand, whilst we do provide some quantitative views with Essential, its real strength is at the other end of the spectrum in providing the qualitative information.  So, if you want to understand the underlying reasons behind the data relationships then Essential can provide the insight and detail to explain why and how things are as they are.  For example, whilst the quantitative information tells you how many interfaces you have, it doesn’t tell you how those interfaces relate, what they are responsible for, what they eventually impact, etc. Essential will give you the view to show, for example, which are your critical interfaces – which applications and processes fail if a specific feed fails etc.  This information is less about statistics and more about knowledge.

Between the two types of tool you can get a powerful view of your IT environment, but you need to know where to get best value from each and, of course, understand your requirements as Essential can be a source for the quantitative as well as the qualitative in some cases.

Business Architecture Trends

First of all I would like to wish all our Essential users a happy and prosperous 2013!

Everyone on the Essential Project Team is very excited about the forthcoming year and the plans that we have for adding to and improving Essential and the services we offer around it – but that is for another blog or maybe another section of the website – for now I wanted to talk about Business Architecture.

We’ve noticed a lot of discussion regarding business architecture at the moment, both in blogs and discussions and also in the traffic we see to the Essential Project; over the last two months five of our top ten page visits were to tutorials related to business architecture and the business architecture tutorial home page had three times as many visits as the application and technology tutorial home page and six times as many visits as the Info and data tutorials home page.  Another interesting point is that this almost exactly mirrors the introduction to a blog I wrote 3 years ago, substituting business architecture for business capability modelling.

So there is evidently a huge desire within the EA community to focus on business architecture, which I think is great as EA isn’t really EA without the BA.  It is imperative to understand the objectives, drivers and principles of your business if you are to be effective in your EA efforts.  Without these, and the ability to demonstrate to your stakeholders how all your initiatives – IT or not – are supporting the business objectives and drivers, you are missing the potential to make a real difference to your organisation.

I think there are two important strands here.  The first is in actually supporting key business stakeholders by providing them with specific targeted information which allows them to make informed decisions in a timely and effective manner.  The nature of the information required will vary from stakeholder to stakeholder and organisation to organisation.  The key, as an architect, is in understanding these needs and understanding the analysis that needs to be undertaken to inform the decision, thus understanding what information to give to your stakeholder.  The second is in communicating how effective the business architecture, and ultimately the enterprise architecture, is in supporting the business using key performance metrics.

We are seeing many more of our clients bring activities that may have once been viewed as pertinent only to the application and technology layers into the business layer.  Recent projects have involved us working with clients who are looking to understand, measure and report on the maturity of their business capabilities, benchmarking them against industry standards and internally desired maturity measures.  We are also working with clients that want to identify standard processes that support business capabilities and to measure and report on their implementation and exceptions.

Another trend we have noticed is that EA Teams are becoming more aware of the need to communicate their success in supporting the business to key stakeholders, with an increase in the number of EA Teams that want to measure and report on the effectiveness of their operation providing, for example, quarterly dashboards tracking progress against strategic objectives.

Essential Architecture Manager has a large number of views that support the business architecture out of the box, from Business Reference Models that allow you to navigate through the layers of your reference model to understand the underlying architectures, to the Business Objective/Project Footprint that shows the footprint of current change against the business objectives defined, overlaid onto the capability model, to Business Roadmaps which show the progression of the architecture from the current to the future state.  In addition, as the meta model is extensive, we have been able to quickly and easily create client specific dashboards enabling EA Teams to effectively report their progress.

We would be really interested in feedback from any members of the community outlining their current initiatives in the Business Architecture layer so we can further develop Essential and ensure that it continues to support the requirements of the community.


Never let a serious crisis go to waste!

At a recent conference run by the Enterprise Architecture Specialist Group of the British Computer Society (BCS), a panel of the speakers was asked how a newly constituted EA unit could best establish its credibility. The question came from a member of the audience who had recently taken charge of such a unit, and who was keenly aware of the dangers of being regarded by the rest of the IT function as a superfluous, “ivory tower” operation.

This concern resonated with the Essential Project Team’s experience. All too often we hear of cases where newly-formed enterprise architecture teams, with the best of intentions, have spent months documenting business processes, information, applications and technologies at an unnecessarily fine level of detail. Given this, it is hardly surprising when they find themselves disbanded as a result of ill-considered cost cutting initiatives.The BCS conference panel’s responses all boiled down to three simple imperatives that we think are appropriate not only for newly formed EA units, but also for well-established ones:

Keep it simple!

Make it relevant!

Deliver timely value!

Every organization has issues that keep senior people awake at night, and enterprise architecture can often provide the means to address these. Sometimes the opportunities lie purely in the IT domain. An example of this was a media company with a complex IT landscape that was faced with an urgent need to reduce costs. Capturing the basic information on the firm’s applications and technology platforms and producing the relevant EA reports enabled a small task force within ten weeks to identify $30m of savings through elimination of duplicate applications and redundant technologies. More often the opportunities are business-related, and these are the ones that generally win the most brownie points for the EA unit. Several years ago the merger of two chemical companies looked to be a marriage made in heaven from a traditional due diligence perspective. However, a rapidly conducted, high level view of the business architecture of the two firms revealed fundamental incompatibilities between their respective operating models. This caused a re-think of the viability of the deal. A more recent example is that of a pharmaceutical company whose legal department had an urgent need to track the movement of certain critical information across national boundaries. In all these cases, the EA function responded quickly by capturing sufficient information to meet the requirement.

The Essential Metamodel and the Essential Architecture Manager have been designed in such a way that cases like this can be handled with minimal administrative effort. And the captured information can of course be retained for other uses after the crises have passed.

Essential Information and Data Pack

It’s been a couple of months since we released the Information and Data Pack and I thought it would be useful to take a more detailed look at what is in this extension to the Essential Meta Model and what it can do for us.


Firstly, thanks to our community members who have given us some feedback and found a couple of small bugs in there. We’ve started a forum thread to catch any more issues as we find them but will be releasing a patch and new version of the pack in the coming weeks – we wanted to make sure we had as many issues addressed in a single update as possible. If you’ve found any issues in there, please let us know in this forum.


This optional extension pack is a major extension to the Information layer of the meta model but although there are some tweaks of the core meta model elements, this is mostly an extension to what was already there.


We have added a number of important meta classes for managing Data elements and the relationships that these have to Application, Information and Business elements, enabling the resulting models to support Enterprise Data Management and Enterprise Information Management activities. More about that later in this blog.


One of the most important concepts in the pack is the separation between Information and Data. We have defined strong semantics for what is Information and what is Data, so that there is a clear and consistent framework for capturing and managing these elements of your Information and Data architecture.


We’ve based this on the commonly used “Information is Data in context” definition. Data elements have the same meaning and value regardless of the context in which they are used, whereas Information elements differ depending on the context. e.g. Product data means the same things regardless of whether we are looking it in the context of Stock, Sales, Manufacturing. That’s not to say that we only have one Product data element in our architecture, we can define as many variants of how Product data is managed in our organisation as we need – in terms of the attributes and relationships. e.g. Product Item data might have a different set of attributes to a Bill of Materials data object.


In contrast, Information takes data and uses it in a particular context. e.g. Stock Volume by Location might use data about Products at particular locations in the organisation and would have a different value for a single Product depending on the Location.


This separation of the Information and Data fits neatly into how we need to manage these things in the real world. Data combined and used to produce the Information that we need.


Naturally, we’ve used the Conceptual, Logical and Physical abstractions for the new Data metaclasses, covering the WHAT, HOW and WHERE for data elements. In addition to adding these new meta classes, we have created powerful relationship classes to them that enable us to clearly understand how Information and Data is being used by Applications and Business Processes. Some of this might seem a bit complex at first glance but this is due to the contextualised nature of these relationships. What we’ve created are constructs that enable us to understand what Information Applications use and create and in the context of that, which data elements are used, specifically, to deliver that Information – and in that context what are the CRUD for the Information and the Data. We believe that having these types of contextual relationships is a uniquely powerful capability of Essential Architecture Manager but we haven’t added these just because we can but because without them we cannot accurately or even reliably understand what is really going on in our organisation with respect to Information and Data.


So, what kind of things can we do with the Information and Data Pack?

We have designed the meta model to support Information Management and Data Management activities managing all the dependencies that exist between information and data elements and also how these are used by Application elements and how they are produced by Applications.


More specifically, in combination with the Views that are supplied out-of-the-box, we can understand where issues exist with our Master Data Management solutions, e.g.

  • where are we mastering Product data?
  • how is this data being supplied to our Applications?
  • how does this compare to how data should be supplied to our applications (according to our MDM policies)?


As you will have come to expect, the supplied Views are easy-to-consume, hyperlinked perspectives that are designed to give valuable insights to a wide range of stakeholders – not just Information and Data professionals.


We can browse the Data catalogue, drill into individual subjects, see what Objects we have in each subject and how each of these objects is defined. Further drill-downs can take us to views that should how this Object is provided to the relevant applications in the enterprise and from where.


Based on the detailed CRUD information that we can capture in the model, we can easily produce CRUD matrices from a wide variety of perspectives, e.g. CRUD of Data Subjects by Business Capability, CRUD of Data Objects by Business Process – both of which are derived from the details in the model, which means that these are automatically updated as the content of the model is updated.


One of the most powerful Views is also one of the simplest. We find that providing a browser-based, easy-to-access place to find accurate and detailed definitions for all the Information and Data elements – and most importantly in easy-to-understand terms – is a capability that is quickly valued, in particular by non-technical stakeholders. Deliberately, there are no ERDs or UML class diagrams in these definitions. Rather, we can explore the catalogue of the Information and Data elements in a highly hyperlinked environment, providing an automatically generated data (and information) dictionary to the organisation.


In that context, we’ve introduced some nice little features that will be included across all the Views such as the ability to identify the content owner of particular elements or groups of elements. This means that if we see that the content on a View is incorrect, we can click a link and send an email to that owner to let them know that things have changed or that there’s an error in the content.


We recognise that while most of the meta model concepts for the Information and Data pack are straightforward, there are some more complex areas, in particular in the definition of relationships. As always, we’ve worked hard to keep these as simple as possible but the reality of how Information and Data is used and produced is (or at least can be!) complex and we need to able to capture and manage these complexities. However, the capabilities and results are worth the investment in understanding how to use these. For example, the way the Information to Application (and in that context, to Data) relationships work, enable us to understand how a packaged application has been configured in terms of how the out-of-the-box data structures are being used. This means that we can understand where Product Codes are being stored in Location Code tables, for example, and this is the kind of scenario where the ‘devil is in the detail’ and fine-grained things like this can become the source of a lot of larger-scale issues in the architecture.


We’ve already had some excellent feedback on the pack and based on demands from real-world use of Essential Architecture Manager, we are now looking to extend the granularity of the application-to-data relationships to include data attributes, rather than just data objects. You might not always need to go to that level of detail but if you do, the capability will be there – and this will be designed to enable you to model at asymmetrical levels of granularity.


Although it’s currently an optional extension, the Information and Data pack will be incorporated into the next baseline version of the meta model. We think that the game-changing capabilities of the Information and Data Pack are a vital part of the enterprise architecture model and so it is natural that this extension become part of the core meta model.


What about OWL?

OWL is W3C standard for defining ontologies for the Semantic Web. When people find out that we’re using Protege in Essential Architecture Manager, they often ask why we haven’t used OWL. So, what about OWL?

When people find out that we’re using Protege as a part of Essential Architecture Manager, a common question is whether we will, or why we don’t currently, use OWL.

OWL is an open-standard language from W3C for describing ontologies – the Web Ontology Language. This uses RDF (another XML language for describing resources that is often used to capture the contents of a repository) to store the Protege repository – both classes and individuals (as instances are known in Protege OWL).

There’s a lot of work currently going on in the OWL space, developing standard ontologies for particular domains, producing advanced reasoners to use the ontologies and so on.

The Protege Project is heavily involved in supporting these ontology initiatives and indeed, currently Protege 4 only supports OWL ontologies. I had some concerns about this but have been assured by the Protege Team that Protege Frames is still very much part of the Protege roadmap and the continued development of Protege 3 demonstrates this.

So, why not use OWL for Essential Architecture Manager?

This is a question we asked ourselves several years ago and we took some time to understand which approach best supported our requirements. The advice on the Protege Project website was a key part of making this decision – our requirements very neatly matched the description of the Frames-based ontology.

Some seem to contend that unless you are using OWL, you are not working on an ontology. This is not the case. It’s the semantics, the types (classes) of things, the actual things and the relationships between the classes and things that are what makes an ontology.

With Essential Architecture Manager, we are not trying to create a universal or standard ontology for the Enterprise Architecture of all organisations. Rather, by using Essential Architecture Manager, you are building an ontology about your organisation. We find it helpful to think about the Essential Meta Model as a partially-populated ontology, like the trunk and branches of a tree, and by modelling your enterprise in Essential, you are adding the leaves to the tree.

In this way, everyone who uses Essential is creating a different ontology. However, they are using the same language to build or complete that ontology. The Essential Meta Model provides a Domain Specific Language for Enterprise Architecture that users of Essential are able to use to build the ontology of their enterprise architecture.

Using the same DSL means that details of these ontologies can be shared, exchanged, reasoned on, etc. using a shared set of tools and shared semantics.

It’s the instances of the ontology that are specific to each ‘enterprise’ but the classes and types – the meta model – that are ‘standard’ across all Essential deployments.

We realise that many existing users of Protege are getting great benefits from using OWL but in addition to the better ‘fit’ of Frames to our knowledge base requirements, the simplicity and ease of use of Protege Frames is very important to the Essential Project.

By supplying an extensive class hierarchy in the Essential Meta Model, users of the Essential Project are somewhat different from many typical Protege users in that they do not need to start with Class modelling. Rather, they pick up the Class hierarchy and start creating instances. The Protege Forms make this very clear and straight-forward, so users are able to get on with ‘adding the leaves’ to the ontology. There is no need to know anything about ontologies or ontology development to use and get the benefits of Essential Architecture Manager.

This also means that we can clearly and easily separate the development of the Essential Meta Model class-hierarchy from the modelling activity – creating instances of those classes. Indeed, we normally hide the Classes tab in a multi-user configuration to reinforce this separation. The main activities in using Essential are about creating and relating instances. Designing and creating new classes is something to be approached when the out-of-the-box meta model needs to be extended – not as part of the day-to-day use of Essential.

By focussing on describing the details (the Instances) of the organisation – current, planned or future – rather than on how to describe (the Classes) the organisation, the benefits of enterprise architecture can be realised very quickly. During our consulting engagements, we’ve seen clients spend months defining their own detailed meta model before getting started on using this meta model. By picking up the Essential Meta Model – and extending it where and when required – we’ve seen clients gain real benefits of EA and EA tools in the same timeframe that others are still defining their meta model.

For those who are interested in OWL and Essential, Protege provides the capability to export the Essential Architecture Manager repositories in OWL format, preserving the class-hierarchy and the properties of these classes. This export could provide a starting point for those who are interested in using the Essential Meta Model ontology as a starting point for OWL-based activities.

I’d be really interested to hear about anyone’s experience doing just this.

Exploiting information that you already have about your organisation

Whether you are just starting out with a new EA modelling solution or already have a powerful repository, everyone has information about their organisation in many other places in many other forms. Exploit this existing information.

Many organisations embarking on EA modelling already have a wealth of information in a variety of disparate forms, such as presentations, drawing tools and most commonly in spreadsheets. Rather than re-key, or re-model all this work, shouldn’t we be able to exploit this existing information – lifting it directly – by importing it into Essential Architecture Manager?

Over the last few years, we have explored a variety of options for importing existing information into Protege (and therefore Essential Architecture Manager) and we realised that the only safe and reliable approach was to do this using the Protege API to add, remove and update information in the knowledge base in effectively the same was as the graphical user interface and let Protege take care of all the integrity issues as it does when you work with the forms. Using the Protege API is at the heart of the Essential integration tools.

The new Essential Integration tab plugin for Protege and Essential Architecture Manager simplifying the one-off ‘data load’ of things like your spreadsheets but also the on-going synchronisation of important data that is maintained in other systems. We’ve had some very good experience of doing this sort of synchronisation with configuration management databases and virtual server environment configurations.

This new tab is an evolution of the ‘skunk-works’ release of the Integration Server. We have made it into a tab to make it easier for both stand-alone and multi-user use (both modes are supported) and to make it easier to install and add this capability to your Essential Architecture Manager. The process of running an import or synchronisation is now much simpler than it was with the Integration Server.

The concept of the integration server will remain but will be re-worked to provide a means of automating imports and on-going synchronisations from other sources, e.g. on a scheduled daily basis. However, the new integration tab will be the most commonly used mechanism for importing information into Essential en-masse.

Getting information out of Essential to share with other systems is simply a matter of creating a suitable view for Essential Viewer, maybe an XML document or a CSV etc. With the contribution from Essential Project Community member, Mathwizard, this can be saved directly as an export file from Essential Viewer, using your browser.

So how to do you go about importing your spreadsheets, XML documents etc. using the integration tab?

As you might expect, XML is the ideal format for integrating the source information and the integration tab expects source data to be in XML format, so it is often easiest to work with an XML export of your source information.

The first step is to work out how this existing information will be represented in Essential. Unfortunately, there is no magic solution to this. You need to define how your source information maps to the Essential Meta Model and with this understanding define the transform that is required.

By the way, creating relationships and relationship classes during imports can be particularly tricky as the source information often does not represent the relationships in a way that can be mapped to the relationships in the Essential Meta Model. In many cases, it can be best to focus on importing the instances and then completing the relationships in Protege via the forms.

There are two approaches to defining the transform that I’ll explore in a moment. We intend to produce and share a library of transforms from common source formats into the baseline Essential Meta Model.

Currently, we have a transform for importing the XML representation of an Essential repository that Essential Viewer uses. This means that you can import elements from other Essential repositories, out of the box. Also, our previous experience with certain configuration management databases and virtual server environments mean that we can easily share transforms to import Technology Nodes and Technology Instances into Essential.

The two approaches to transforming your existing source information into the Essential Meta Model are:

  1. Transform you existing source to XML using the Essential Viewer XML schema. Then import this resulting XML into Essential using the out-of-the-box Essential Repository transform. This approach is useful for once-off data loads from existing sources such as spreadsheets where it’s most important to get the main instances into Essential rather than complex relationships.
  2. Define your own transform file and use this to have the integration tab transform your source information and import it into Essential. Although this may seem to be a more complex exercise than approach 1, you have more flexibility in defining how your source information maps to the Essential Meta Model. A how-to guide for writing transforms is being written at the moment that explains how the transforms work and the supporting tools that are available, e.g. a library of script functions that wrap the Protege API to support things like on-going synchronisation of instances in Essential with instances in other, external repositories.

We would be very happy to help anyone who needs to construct their own custom transform via the forums or even to undertake commissions to build custom transforms as required.

And of course, if you’ve built some transforms that you’d like to share with the Community, we would really like to add these to the Share area of the site.

Happy exploiting of your existing information!