The theme at Enterprise Data World 2016 was “transforming to a data driven business” and Silwood Technology were delighted to be sponsoring the event again that year.
The process of becoming a data driven business is a broad subject that covers a multitude of disciplines, for example, data governance, enterprise information management, master data, business intelligence and analytics as well as data architecture, data quality and more.
I was musing on this topic and how it related to us at Silwood in the context of our specialist area.
This is the enabling our customers and partners to accelerate project delivery by giving them the ability to access and discover the metadata which make up the data models which underpin their large, complex packages applications through the use of our product, Safyr. In this way they could find and understand the data structures of the systems and solutions which create or provide the data for the business.
Without this it is almost impossible to derive enough meaning from the data to deliver the necessary integration, analytics or governance for transformation.
To find a deeper connection however, it seemed reasonable to try to understand what is meant by “Data Driven Business”. A quick internet search actually left me not much the wiser, mainly because the term seems to be used to fit the given circumstances. This is in contrast to it being a phrase which has a widely accepted meaning, against which the degree to which products, services and solutions fit are measured.
Perhaps a alternative term could be Data-driven decision management as per Techtarget’s definition. They suggest that this approach is taken to try to gain competitive advantage or a significant improvement in for example, productivity. They quote a study from the MIT Center for Digital Business which found “that organizations driven most by data-based decision making had 4% higher productivity rates and 6% higher profits”.
For me I think that in many cases, data has had the potential to drive businesses and organisations to the extent that it has been able to since businesses started recording transactions.
Whether in the past decisions have been taken on the back of what the data suggests of course is an entirely different matter and probably the topic of a whole new blog article.
Now however, there is a difference.
There is more data available from more sources, there are much smarter methods for accessing, moving, analysing and understanding it.
There are also ways of automating decision making based on the data through the use of algorithms, triggers and programming.
Perhaps ultimately, the definition of a truly data driven business will be one where there is little or no manual, human intervention at all!
Metadata represents the data foundation layer of an application, whether that is a large package such as one from SAP, an in house developed application or database, a social media feed such as Twitter, data coming from sensors and logs or even unstructured information.
In some cases the term metadata can also represent the data structures which are used by an integration, master data, data warehouse or other ‘intermediary’ platform.
To enable a data driven business it would make sense therefore to be able to access, understand and use the metadata, or underlying data models, in all source applications in order to make the task of moving, integrating, analyzing, mastering and governing data easier to achieve.
Not having that information is like having cupboards or a fridge full of unlabelled food items and then trying to create a meal. It might be exciting, however the results are variable and to achieve something tasty is likely to take longer than necessary. Further it would also be almost impossible to repeat.
Finding the metadata from many of the sources typically found in an IT ecosystem can be relatively easy. There may be documentation or relevant data models available on the internet. There may be specialists who can find and understand it for you and there are data modelling, metadata repository and other software tools which can deliver that information quite easily and quickly in a meaningful way. This would typically be the case for smaller application packages or in house developed systems and for external sources where the data structures are well known and publicized.
When it comes to large complex and often highly customized packages from vendors such as SAP, Oracle and increasingly Salesforce however these tools and techniques are simply not adequate.
There are several reasons for this. Firstly, the data models are extremely large. For example SAP Business Suite has over 90,000 tables, so even before tables are added and extended to reflect the customer’s needs it is impossible to navigate and find what you are looking for using the methods outlined above in any reasonable timeframe.
Secondly, for most packages the rich metadata about table and field descriptions, relationships, etc., is not stored in the database System Catalogue so even if a third party tool can reverse engineer it or connect at that level, there is very little there of any use to a data professional. The useful information is stored in the Data Dictionary tables.
Thirdly, there are no tools, except for Safyr, which address this problem. The main packaged application vendors for the most part ignore this pressing need and leave customers to find their own solutions such as using internal technical resources, external consultants, templates or (often outdated) documentation, or even internet searches or guesswork.
Information and Data Management software makers do refer to metadata, however it is normally the metadata which is at the heart of their own solution rather than the metadata outside the confines of their own system.