In the past we at Silwood have often referred to the ERP and CRM solutions which are the target for our metadata discovery product, Safyr®, as “Black Boxes” in terms of their underlying data models.
In this context we are using one of the Oxford Dictionary definitions of a Black Box which is: “A complex system or device whose internal workings are hidden or not readily understood”.
I was reminded of this when talking to a senior representative of a rapidly growing data governance and lineage software company recently who referred to the problems businesses have (and which they help to solve) which are associated with “data in a dark room”.
By this he meant that organisations continue to struggle to exploit the huge amounts of data that they collect or acquire because of the challenges they experience in managing, understanding and governing that data. He went on to emphasise the opportunities which are being wasted because those organisations cannot easily perform tasks using data which should be straightforward.
This could be in trying to visualise what is affected when business imperatives require a change to process or data. Another instance might how to locate and then analyse the risks associated with Personal Data for GDPR compliance.
We certainly see these, and other examples of organisations not maximising the return from their investments by being unable to leverage their ERP and CRM data which they store in systems from SAP, Oracle, Microsoft and increasingly Salesforce.
So why is it that, despite the advances made in data warehouse, data integration and migration, data governance, master data and other software products, customers find it so difficult, expensive, time consuming and risky to incorporate the data from these packages whilst implementing these solutions?
The reason lies in the nature of their metadata: the data model which underpins these systems. For example to deliver an effective data migration or data warehouse project, it is essential that the architects and developers understand the data structures with which they are working.
For many systems e.g. home grown applications, files, machine data and some packages it is a relatively easy task to figure out their data models. Many data modeling and information management software products have the capability to reverse engineer their metadata which usually contains information that makes sense to the analyst or architect. In additional the data models are relatively limited in size and complexity.
Compare this with the metadata in an SAP packaged application. For a start the data model will contain over 90,000 tables. Apart from the size and innate complexity, the tables and field names in the database System Catalog are so opaque that they mean nothing and are of no value to an data analyst or architect.
Most ERP and CRM systems have also been customised at the data model level so, even if documentation was available it is likely to be inaccurate.
These characteristics mean that the standard tools provided by the information and data management software vendors will not be of much value. This is because they only access the information in the System Catalog or provide template based solutions which then need to be compared to what has been implemented to identify differences.
The package vendors do provide some tools for those technical specialists who work with their products at that level, however they are of very limited value for data folk.
As a result organisations are reliant on internal specialists, expensive external consultants, internet search and other manual approaches to solve this problem.
So how do you see inside the Black Box and illuminate your application’s data model and make use of the information it contains? The rich metadata is there – it is just that it is not accessible in any effective way without a product designed for that purpose.
This is what we do here at Silwood. We provide a specialist software product called Safyr which shines a light on ERP and CRM data by giving access to the full data model, quickly, as implemented and accurately.
Most importantly it then provides analysts and architects working on data projects with the tools to search and find the information they need using business terms.
As a result they are able to take control of that part of the project which requires intelligence about the data model and share the results with the other tools being used in the program.
Customers enjoy faster implementation, less rework and improved trust in the data by the business.