It is because you need to know where the data that flows through these complex applications lives and what it means. Often this is in the context of analytics for decision making, or for reporting, or for data migration or integration.
Understanding and being able to use metadata effectively is also critical for the effective implementation of data intelligence platforms which include data governance, data catalog, data lineage, data management and data quality applications.
If you cannot be certain that you are using the right data or of its provenance, then business confidence in its accuracy will be lower. And the data you are using for decision making may not be correct.
And for Enterprise Resource Planning applications the difficulties of working with their metadata can have a significant impact on your ability to deliver critical data projects effectively.
We didn’t create this problem. However, we at Silwood Technology have spent the last 20+ years developing software which helps data teams to cut through the size, complexity, inaccessibility and opaque nature of ERP metadata in order to improve confidence in data and reduce the cost and effort of this work.
Firstly, it is important to acknowledge that these applications cover a wide range of business activities and that means that they will, by necessity, have a large and complex data model (metadata). For example, an SAP S/4HANA system has over 100,000 tables and a million attributes.
It is highly unlikely that you will need all the tables from the source ERP application for your project. Therefore, the size of the data model alone means that you will face a challenge in merely being able to navigate and locate those specific objects which you need.
This may be because there may be significant numbers that are simply not used, or because they are not relevant for the specific use case on which you are engaged.
Further, these data models are commonly customized to meet the particular business needs of a customer, which adds to your challenge, because it means that documentation and templates for example are of minimal value.
Unfortunately, to add to the problem, the application’s database system catalog only contains technical (or ‘physical’) names for tables, attributes, and other objects. In many ERP’s these would be of only very limited value to the data analyst or business user. In addition, their system catalogs do not contain the information which allows you to see how tables are related to each other. All of this means that trying to find and use metadata by reverse engineering it from the RDBMS itself is of little to no value.
A final challenge for the data specialist is that these packaged applications provide little in the way of metadata discovery tools specifically designed for them. Of course, there are software tools for the technical specialist – however these are rarely of any benefit if you need to be able to make use of the data model and do not have specific application knowledge.
The most common method in our experience is to ask technical applications specialists who may have access to application vendors tools to find and deliver the required metadata. This work often has to be allocated and then scheduled by the relevant team which can cause delay. Then the results need to be checked, and possibly reworked, before the information can be provisioned into the target system, for example a data catalog platform, which can present a further challenge in itself.
Other methods we have come across include:
- Manual inspection of the metadata through other application functions (e.g. Help in SAP
- Refer to documentation (assuming it is up to date and caters for customizations)
- Use templates (do these reflect the same application version and customizations that you are running)
- Guesswork
- Internet search
Our customers and partners use our product Safyr as an important component which supports improvement in business confidence in data and accelerates the delivery of critical data intensive projects (e.g. data governance, data catalog and lineage, data analytics, data migration) through quicker and more accurate metadata discovery for ERP’s.
They often report a significant productivity gain, usually around 90%, in the area in which they use Safyr when compared to traditional tools and techniques mentioned above.
They use Safyr to perform three tasks. The first is to harvest business and technical metadata (as implemented) from the source ERP applications Safyr supports. These include packages from vendors such as SAP, Microsoft, Salesforce and Oracle. Because of the importance of being able to access and use business and well as technical names for metadata objects and the relationships between them, Safyr goes to the data dictionaries of these applications for this more valuable metadata rather than the database system catalog.
Many of these systems have tens, or hundreds, of thousands of tables so it is vital for the data team to be able to use a specialist tool which is designed to support them in finding exactly what they need – quickly and easily and without needing any specialist application knowledge.
This is the second task which is to identify the key metadata objects needed for the project or business use case. Safyr users can search for, discover, and subset groups of objects (tables, related tables with both business and technical information) before curating them into what are called “Subject Areas”.
Finally, these Subject Areas can be shared for ingestion by leading data catalog products such as Collibra, Alation, Precisely, Purview, Quest and others; as well as in data modeling tools including SAP PowerDesigner, and other platforms.
Daniela Zelmer of Gewobag said
As a non-SAP specialist, comprehending the intricacies of SAP data can be challenging. However, Safyr has made this process significantly easier and more efficient. Its capability to extract metadata from our SAP and SAP BW system has been invaluable in providing us with clear insights into our data landscape.
As I said in the introduction; “We didn’t create this problem” – we have however helped many customers to solve it. If you would like to learn more and explore how you could benefit from improved ERP metadata discovery processes please contact us for an introductory call, or download an evaluation copy of Safyr