Data Lens

You may have noticed from our site that the Data Lens is in beta.  It’s a lens that we’ve developed because we’ve been continually told that people don’t have control of their data.

In our EA consulting, we have seen:

  • Organisations that were unwittingly reporting incorrect MI figures because data was inaccurate or incomplete
  • Projects that intended to master and duplicate data that already existed in the organisation
  • Inconsistency in what people thought certain data was
  • Differing views on where data was sourced from
  • Projects repeating the same data collection work, asking the same questions again

The Data Lens looks to address this by bringing transparency and coherence to your data estate.  It is aimed at supporting the demands of people wanting to use data, such as:

  • Data Lake or Analytics efforts, which need to know information such as where data is sourced from, what terms are used for the same data, e.g. client and customer, how good the data is in terms of quality and completeness, etc.
  • Platform projects need to know where data masters exist, where data flows, how data is transformed, etc.
  • Any data rationalisation project needs to know where master sources of data exist, where duplication exists and how data is used.
  • Plus, Data Scientists need to understand the sources of data available for their analysis

The lens addresses these needs by providing a number of views and tools.

The Data Definition views provide data definitions, summaries and dynamically produced data models.

The Data Architecture Analysis views are geared towards you understanding sources of data, data flows, where duplication exists, etc.

Data Management is where the lens excels.  You are able to understand data quality across a number of criteria and see sources of data.  The Quality Dashboard shows the quality of the key data required to support your strategic objectives and business capabilities, and also the initiatives impacting that data.  This allows you to identify where your data initiatives may need to be focused to improve your business data output and enable your strategy.  The Data Quality Analysis page lets you pick the data you need and it then shows you where to source it from, plus the quality, completeness and accuracy of that data. This is really useful if you are using the data for other purposes, e.g. MI reporting or analytics. The data dashboard provides and summary view of your data which you can drill down into.

We see the Data Lens acting as the bridge between the tools that are more focused on the physical data layer, and which typically meet the needs of the technical teams but not the business users or the data scientists.  Equally, where you have conceptual data in a tool, the lens can act as the bridge to the physical data, removing the gap between the conceptual and physical layers, bringing context and meaning to the data.

The lens is currently in beta but we are allowing organisations to register an interest and we would love to get any feedback on the lens.

Essential Cloud – Available Now

Today marks a step change in the life of The Essential Project as we move to Public Preview of Essential Cloud, the final step before General Release. A cloud offering has been at the top of the Essential Community request list for some time and we have combined the best of the Essential Project with a cloud based service to provide additional enterprise capabilities. As well as all the benefits of Essential Open Source, Essential Cloud offers a comprehensive security interface covering both the instances in the repository and the viewer, a user-friendly, browser-based data capture interface extended to include tablet and mobile access, an enhanced viewer environment and single sign-on support via SAML. As this is a cloud service, technical support is automatically included as are platform updates, to ensure that you can keep up with the latest Essential developments with none of the hassle.

In line with our focus on value, Essential Cloud will be a low-cost option, with an annual subscription covering access to both the modeller and the viewer for unlimited users. We are not utilising a seat-based license model as the feedback from the Essential Community and our clients is that the key to an effective architecture initiative, one that provides value to the business, is to enable the users to own and update key aspects of the architecture, i.e. those areas that do not require modelling expertise, such as dates, ownership, governance models and so on. This spreads the load of keeping organisational information up to date and enables architects to focus on business value rather than being distracted with managing routine updates. A seat-based license model does not fit with this approach as the costs quickly become prohibitive; we would rather an organisation’s investment in EA is used to build out their architecture than pay for licenses.

To support this new model further, we are working with our user groups to design new data capture mechanisms that will provide business users with easy access and enable them to update information without having to understand the detail of the meta model or architecture modelling techniques. We already have some early prototypes, and we see this as an important way of enabling EA to continue to provide value to the business.

This is an exciting step in the broadening of the Essential platform, but we do want to assure you that we remain fully committed to Essential Open Source. This will continue to evolve in parallel with Essential Cloud and, crucially, the meta model will remain shared so both platforms will benefit from all advancements as well as the ability to move easily between Cloud and Open Source. Going forward we see the Essential Community consisting of both Open Source and Cloud users. We greatly value the contribution made by the community and we will continue to look to them to help us evolve the Essential proposition to ensure it remains at the forefront of knowledge driven decision support.

We have created an overview video showing Essential Cloud’s capabilities and we will also be holding a series of Webex’s where we will provide a demonstration of Essential Cloud and hold a Q&A session.

If you are interested in the Webex or Public Preview sign up here.

The Public Preview benefits are, of course, in addition to the existing benefits that are provided across both Essential Cloud and Open source:

  • Over 100 out of the box views focused on analysis, road mapping and decision making
  • Ontology based meta model for an entire organisation, with the ability to support other EA Frameworks
  • Import and export of data via unique excel import utility, with fast start view loaders, and APIs to integrate with existing data sources
  • Access to business focused lenses providing dedicated support for key areas such as Application Portfolio Management, Data Management, Strategic Resource Optimisation

If you are new to Essential then, Essential Cloud aside, one of our most exciting recent developments is the addition of add-on business focused lenses.

The lenses have been in our mind since 2010 when Jon Carter wrote a blog titled ‘Welcome to the View Store’, suggesting the concept of an app store for Essential. We were staggered by the interest and up take of Essential and so our early focus was on developing the tool functionality, but now we have made our earlier vision a reality.  Our business outcome focused lenses consist of a series of dashboards and views that respond to specific business problems, supported by everything you need to light up the views – data capture spreadsheets, import specifications, process documentation and videos.

The lenses have provided an ideal means for us to partner with organisations outside of the usual EA arena, allowing us to extend the use of Essential to cover different aspects of an organisation. For example, we have partnered with a strategic resource specialist to create a strategic resource optimisation lens which enables organisations to ensure they have the right staffing resources in place to meet the future demands of the business, such as the skills to enable digital business expansion. We have a couple of packs on offer now: Application Portfolio Management and Strategic Resource Management, and we will be expanding the offering shortly to include Data Management, and then, over time, we plan to move into many additional areas such as M&A, Outsourcing Support, Financial Regulations, etc.

We also have a set of low-cost View Loaders that provide the templates to do bulk data capture into Essential Cloud or Open Source. So, if you want to get bulk data into Essential quickly, the loaders can speed this up.

If you would like to develop your own pack to put on the view store, or talk to us about an idea for joint development of a pack, please contact us.

Essential View Loaders

We’ve come across a number of Essential users who have said they’d rather not spend time writing the import specifications to import data and anything to speed the import of lots of data to get the out-of-the box views working quicker would be helpful. In response, we’ve created some view loaders that will populate key elements of the views. This means that you can focus on gathering the data required and then just run the importer to load the data into your repository.

We’ve priced the loaders at a point that is low, so you won’t need to blow the budget on them, and we will be using the revenue to fund the Essential Project work. We’re intending to build more view loaders in the coming weeks and months, and would welcome feedback on which views you’d like us to prioritise. We’re also due a new dot release of Essential soon, which has some meta model changes and some updated views that we’ll be offering view loaders alongside.

We’re working some other things at the moment, although if you keep an eye on the site you can probably work out what, but more about that in the next few months.

Configuring the Server Memory Settings for Essential

One of the most common issues with setting up Essential is getting the memory configuration of the various components configured correctly. As the complexity of the Essential meta-model and of Essential Viewer has increased then so have the memory resources required by the server to support them.

Right now, our current recommendation for a server is a multi-core processor such as an i5/i7 or Xeon equivalent and more importantly plenty of RAM. 4GB is a minimum but 8GB is more practical. You’ll struggle to use the Import Utility and Viewer together on a system with only 4GB of RAM. Assuming you’re running all the components on the same server (which is perfectly fine and can yield great performance) then here’s how we’d allocate the RAM across the main components…

  • Tomcat running Essential Viewer – 2GB RAM
  • Tomcat running Essential Import Utility – 2GB RAM
    • We’d install the Essential Import Utility on a separate instance running on different port e.g. 9080 as it improves stability and performance
    • If running both on a single instance then allocate 4GB RAM to Tomcat
  • Protege – 1.5GB RAM
  • If running a Database configuration we’ll ensure there’s about 1GB for that
  • We need some memory for the OS to run smoothly so about 1GB for that

This adds up to about 7.5GB. In reality, you’ll rarely use all that RAM simultaneously however this configuration is one we’ve used countless times with excellent performance.

So, now you’ve got plenty of RAM then how do you configure the components to use that.

First up, make sure you’re using the 64bit versions of all your components. If you’re running 32bit versions, you’ll max out a 1.5GB which will work whilst the repository is small but will cause you problems later on.


On Windows:

  1. Start Protege. Go to File->Preferences->Protege.lax
  2. Update the row for the property ‘
  3. This is set in bytes, so set this to 2048000000 for installs with the 64-bit Java environment.
  4. Click OK
  5. Restart Protege

On Mac:
If you run Protege on a Mac by double clicking an icon, you need to edit the Info.plist file that is hidden within that icon. Right click the icon (or ^-click for one button mouses) and click “show package contents”. A new finder window will come up. Double click “Contents” and then “Info.plist”. Traverse down the tree as follows: “Root” –> “Java” –> “VMOptions”. In VMOptions edit the -Xmx line to indicate the correct memory usage, e.g. 2048M. Note that this can be specified in megabytes by using the ‘M’ value.

For example, here are my settings:

Save the changes that you’ve made and restart Protege for these to take effect.

This principle also applies to the Protege server. If you have not already, update the ‘run_protege_server.bat’ / ‘’ file to increase the maximum memory JVM option as follows by setting the -Xmx parameter:

For Unix / Mac / Linux:

MAX_MEMORY=-Xmx2048M -XX:MaxPermSize=512m

On 64-bit Windows platforms (with the 64-bit Java installation):

set MAX_MEMORY=-Xmx2048M

On 32-bit JVMs on 64/32-bit Windows, there’s a limit to how much memory can be allocated:

set MAX_MEMORY=-Xmx1536M


Tomcat / Essential Viewer / Essential Import Utility

The memory settings for the Tomcat that is running the Essential Viewer should also be set to around 2GB for 64-bit Java environments.

On Windows

If you are running Tomcat as a Windows service, you can set the upper memory limit using the tomcat8w.exe program. You’ll find this either in the start menu or in the install folder of Tomcat. This will pop-up a configuration panel.

  • Select the ‘Java’ tab and then set the parameter for the Maximum memory pool to 2048
  • Click Apply
  • restart Tomcat for these settings to take effect.

On Mac

If running the Viewer Tomcat on a Mac / Linux platform, you can set these using the ‘’ file in <TOMCAT INSTALL>/bin and set the CATALINA_OPTS variable, e.g.:

export CATALINA_OPTS=”-Xms128m -Xmx2048m -XX:MaxPermSize=512m”
export JAVA_OPTS=”-Djava.awt.headless=true”

If this file doesn’t exist then simply create a new text file and save it as with these lines in it.

Again, you must restart Tomcat for these settings to take effect.


If things aren’t working as expected, then the Log files are your friends. The Protege log is in the Protege install folder under logs and is called protege_###.log. The Tomcat log is in the Tomcat install folder under logs and is called catalina.out. What you’re looking for is anything that mention “memory” or “heap”. If you’re seeing these errors then you haven’t properly configured the settings.

As always, you can post your questions on the Essential Forums at and we’ll answer as quickly as we can. Don’t forget to use the search too as there are over five years of posts and there’s a good chance your question has been answered before.

Once you’ve got these settings right, you should have many years of stability and performance from your Essential Install. If you’re still having problems though and would like some professional support then contact EAS via the Services menu for more information on how we can help.

Harnessing Conceptual Structures to Expose Organizational Dynamics

Alex and I have just had a paper published in the International Journal of Conceptual Structures and Smart Applications (IJCSSA), where we discuss how the practical application of Conceptual Structures lags far behind the theory. We introduce how Essential toolset can address this and provide a platform that can be used to support the real-world application of Conceptual Structures to explore and expose the dynamics of any organization.

The article is available at:


Securing the Viewer

The Essential Viewer can be configured to require that any user first authenticate themselves before accessing the requested View.

In the open-source release, the access control is relatively coarse-grain – can the user access the Viewer or not? – but this helps to keep things simple to configure and manage. We can connect Viewer to an LDAP directory such as Active Directory so that users can use their normal login and password to access the Viewer.

Continue reading “Securing the Viewer”

Surf’s up! Essential catches the Forrester Wave™

Enterprise Architecture Solutions Ltd (EAS), sole sponsor of the Essential Project and developer of the Essential Architecture Manager software, has been identified by Forrester Research as one of the ten most significant providers of enterprise architecture management suites (EAMS).  In this year’s Forrester Wave™ covering EAMS, the research firm has reported extensively on those selected software providers based on their breadth of EA coverage (business/technology orientation and strategy/projects focus), as well as on their ability to serve four common EA objectives:

  • Manage cost and complexity
  • Ensure high-quality and efficient business solutions
  • Set technology strategy, innovate and govern
  • Business/IT planning and alignment

Based on a thorough evaluation of the Essential Architecture Manager Version 4, Forrester rated EAS as a “strong performer”, which they define as being able to demonstrate strong functionality in supporting most or all EA objectives.

For further information please contact Forrester Research Inc., and request the report entitled “The Forrester Wave™: EA Management Suites”, Q2 2013.

The Essential Project team are thrilled to be included in the latest Forrester Wave™. We have long believed that the Essential Architecture Manager offers a compelling EA solution for all sizes and types of organisation, but inclusion in the Forrester Wave™ is a great reflection on the maturity of our offering. We have some powerful innovations planned for 2013, so rest assured that we are not resting on our laurels.

Finally, we would like to say thank you to the international Essential Community for helping us to evolve and shape the Essential software platform.

Never let a serious crisis go to waste!

At a recent conference run by the Enterprise Architecture Specialist Group of the British Computer Society (BCS), a panel of the speakers was asked how a newly constituted EA unit could best establish its credibility. The question came from a member of the audience who had recently taken charge of such a unit, and who was keenly aware of the dangers of being regarded by the rest of the IT function as a superfluous, “ivory tower” operation.

This concern resonated with the Essential Project Team’s experience. All too often we hear of cases where newly-formed enterprise architecture teams, with the best of intentions, have spent months documenting business processes, information, applications and technologies at an unnecessarily fine level of detail. Given this, it is hardly surprising when they find themselves disbanded as a result of ill-considered cost cutting initiatives.The BCS conference panel’s responses all boiled down to three simple imperatives that we think are appropriate not only for newly formed EA units, but also for well-established ones:

Keep it simple!

Make it relevant!

Deliver timely value!

Every organization has issues that keep senior people awake at night, and enterprise architecture can often provide the means to address these. Sometimes the opportunities lie purely in the IT domain. An example of this was a media company with a complex IT landscape that was faced with an urgent need to reduce costs. Capturing the basic information on the firm’s applications and technology platforms and producing the relevant EA reports enabled a small task force within ten weeks to identify $30m of savings through elimination of duplicate applications and redundant technologies. More often the opportunities are business-related, and these are the ones that generally win the most brownie points for the EA unit. Several years ago the merger of two chemical companies looked to be a marriage made in heaven from a traditional due diligence perspective. However, a rapidly conducted, high level view of the business architecture of the two firms revealed fundamental incompatibilities between their respective operating models. This caused a re-think of the viability of the deal. A more recent example is that of a pharmaceutical company whose legal department had an urgent need to track the movement of certain critical information across national boundaries. In all these cases, the EA function responded quickly by capturing sufficient information to meet the requirement.

The Essential Metamodel and the Essential Architecture Manager have been designed in such a way that cases like this can be handled with minimal administrative effort. And the captured information can of course be retained for other uses after the crises have passed.

Essential Upgrade

Following on from Sarah’s last blog that announced what’s coming, I wanted to let you know that we’re in the final stages of packaging Version 3 of Essential Architecture Manager.

The big focus in this version is a complete overhaul of Essential Viewer to make it more easily customised in terms of look and feel – e.g. to tailor it to your intranet styling, improve performance, error handling and to make it simpler to control.

However, version 3 introduces a much extended and improved meta model – based on our own practical experience of using Essential in the real world and of course the contributions of the Essential Project Community. Many of the ECPs that we have been running have now been enhanced and included in the version 3 baseline meta model.

How do we upgrade to version 3?

We’re really excited about the power that version 3 provides but how do we upgrade all those users who have already been enjoying earlier versions?

Upgrading Essential Viewer is quite straightforward as it is delivered as a WAR package. Obviously, any custom Views that have been created will need to be migrated but it is easy to deploy the Version 3 Viewer alongside existing version 2 instances as we transition any custom views.

Migrating the meta model is a little more complex but we have created a number of tools to help make this consistent and more straightforward.

Protege manages the meta model and the model via the Classes and Instances, respectively. These are managed separately (e.g. the PONT and PINS files) as are the form configurations which control the layout, widgets, labels and tool-tips.

This is generally a good thing but it means that the safest and most consistent way to make automated changes to the repository is via the Protege API and doing so means that all the checks and balances that the GUI applies can also be exploited. This also means that we (and you) can make changes and enhancements very easily via the Protege GUI and then encode those as a script of API calls that can be applied to any repository. In layman’s terms, this means we can build upgrade packs to help ease the migration.

We are currently in the process of completing the upgrade pack that will take existing repositories based on v1.2 of the meta model to version 3 and this will be released soon after the Essential Architecture Manager version 3 is launched.

Preparing to upgrade

The idea and intention of the upgrade packs is that they will gracefully upgrade any existing repository. Any extensions that you might have made to the baseline meta model, e.g. additional slots / attributes on a class, additional classes will be preserved as part of the upgrade.

We always recommend that you take an ‘extend’ approach to any customisations or additional features that you might want. Changes to the names or purpose of an out-of-the-box slot – or changing the name of a class may be lost or have unexpected results. If you’ve lost track of what you might have changed, it might be worth comparing your repository to the version 3 baseline repository using the ‘diff’ function of the “Prompt Tab” in Protege.

Many of the forms have had a re-layout to improve the user interface – and in particular to move slots that we have deprecated (served notice of retirement) to the bottom of the form.

Where possible, we have tried to migrate the forms sympathetically but given that one of the powers of Protege is that it is very easy to customise the form layout, we cannot really make any safe assumptions about what a form might look like. There will be cases where the upgrade ‘resets’ the layout to the version 3 baseline form. If you have made changes, you will need to review the layout of such forms – especially if you have additional slots on the class.

It is common to find that you are working in specific areas of the model (rather than the whole model) for particular modelling activities and this means that reviewing the form layouts can be done on an ‘as required’ basis.

How do apply the update packs?

I have described how the update packs take the form of scripts that drive the Protege API but how do we apply them? Protege does not have a plugin for managing this sort of programmatic update of the classes, instances and forms.

Currently, we provide packs as Python scripts that are executed in the “Script Console Tab” that is provided in the standard Protege install. We have tried to package this up as simply as possible, but this is still a bit technical and requires some care when running the scripts.

To improve and simplify this process, we are developing a new Protege Tab Plugin to make updating the repository more like updating your iPhone or iPad!

We’ve seen how the use of the installer has (generally) simplified the install process and we want to provide the same capability for adding new packs of classes – and importantly instances (e.g. standard enumerations) to the repository. So, this plugin will have applications beyond just upgrading the meta model.

Options for migrating to Version 3

Are there any options for how to migrate your existing Essential repository to the version 3 platform? It is possible to make use of the Essential Integration Tab and import an XML snapshot of your repository (the same snapshot that Essential Viewer uses) into an empty version 3 baseline repository. I have used this approach myself to migrate one of our demo repositories before applying further update packs.

However, if you have made any changes to the meta model, this may not be straight-forward and the instance migration packs would still need to be applied.

For this reason, I would recommend using the upgrade pack to apply all the changes required to your existing repository, classes and instances. As with any toolset like this, you should take a backup (using the Protege Project->Archive capability) of your repository before starting to apply the upgrade. And of course, the Essential Project Team will be happy to help with any tricky migrations, perhaps due to some customisations, and the option of commercial support is always available.

Version 3 has so much that is new and improved, that we’re sure that this will be an Essential Upgrade! 😉



The Essential Roadmap

We’ve been kinda quiet on this blog since the summer, not because we all took the autumn off, but because we’ve been working feverishly behind the scenes reviewing the Essential roadmap and preparing a major new release which we are planning to make available early in 2012.

We will be releasing Essential Architecture Manager v3, which includes new versions of the meta model, the software platform and the viewer as well as updated training modules – I did say we’d been busy!

The roadmap below gives you a taster of what is in v3 and what we are planning beyond this (click on the roadmap to view a larger image):-

There’s a lot of new and, we think, exciting stuff in here.  We will include full details when we release it, but to whet your appetite ………….

The Strategy Management area has been completely overhauled to provide support for defining and managing Architecture States (current and future – by layer or strategic project as required) and for defining and managing architectural roadmaps, with milestones and timelines, to demonstrate how the organisation will move from one architectural state to another with supporting Strategic Plans.  In addition the Change Management area has been updated to allow change programmes, projects and activities to be defined which detail the strategic plans that will deliver the roadmap transitions.

An example Roadmap view:-

Information quality and security are key to any organisation and, due to much demand, we have enhanced both areas.  Data Standards Management now provides the ability to manage Data Quality across the organisation, whilst Security Management allows an organisation to define its Security Architecture and its relationship to the resources in the organisation.  In addition to the usual application of these areas, we have been able to resolve other issues for organisations, such as providing the required regulatory views for movement of data across country borders, with a long term solution that is quick and easy to implement by simply adding to the information artefacts already captured.

Some example Data Management and Security views:-

There are also a number of other smaller additions to the meta model which provide benefit across the repository as a whole, for example, Data Governance (a peer of Data Standards Management in Data Management); Taxonomy (enabling user-defined classifications to be applied to any element); Geography (for including geographical location information with elements such as Site) and Synonym (for any element in the model).

We are always looking at how we can provide more benefit and we have developed a new and innovative solution to help organisations get information into the repository as quickly and efficiently as possible, in support of our view that capturing the data is the means to the end (views/reports/analysis of the information) rather than the end itself.  We have developed a Spreadsheet Export and Import facility which allows the user to define a spreadsheet from within Essential, incorporating a series of worksheets and including data from the existing repository if required.  This is then exported to excel and allows almost anyone within the organisation to capture the required information by simply filling in the spreadsheet; no modelling knowledge is required for this task.  The information in the spreadsheet is then imported directly into Essential, populating the repository and creating the required links and relationships between the captured artefacts.  This is not only quick,  it also allows the data capture to be completed by more and cheaper resources; the person with the EA knowledge is then freed up to concentrate on the important tasks.

We’ve be using this in beta for a few months,and it is proving hugely effective at capturing large amounts of quality data quickly.


All the existing training modules have been updated to take account of v3 and new modules have been created for the new areas, all of which are now readily available.

Finally, you have probably noticed from the screenshots that we also have a brand new Essential Viewer. We’ve taken the time to think about what users, especially business users, really want from the reports  and have created a look and feel that’s clean, friendly and focused. It also features active filters to show or hide elements on a page which we think are a great way to simplify the content on a page without building a whole new view. There are also some less obvious but powerful features such as dynamic linking. We’ve always been proud of the ability to navigate the architecture – moving up, down and across to discover more about your organisation – but previously each link only ever went to one place. In future, any link on a page can be configured to access any number of relevant views. For example, clicking on a Data Subject can take you to a Data Subject Summary, a Data Subject to Role matrix, a Data Subject to Application Matrix, etc. There’s nothing to learn, no training required, you click on a link and click where you want to go. It’s powerful and effortless.  For those who develop their own views, you can easily take advantage of this with just a few extra lines in your code. We’ve also developed a whole new CSS framework for Essential Viewer that helps you build great looking views for you organisation quickly. Finally, we know how important it is for these tools to feel like part of your own organisation and we have provided the ability to re-brand the viewer. You can add your own logos, colour schemes, fonts and portal name. This kind of branding used to take a lot of effort but now can be achieved extremely quickly. And of course, you can use the new viewer on your iPad or iPhone…

As always, these changes are driven by our clients and community, as are the next Essential developments.  We are always keen to hear your views and suggestions, please feel free to comment.