If your organisation is considering, or already implementing a data catalog, data governance or enterprise metadata management platform and you are an SAP customer, you may think that SAP and data catalogs don’t mix. And, if you have ERP or CRM packages from vendors other than SAP – maybe from Oracle or Microsoft, then the thrust of this blog will also apply to your organisation.
Assembling the metadata from each of the key systems used by your organization is not only essential for your data catalog, it is critical to its success.
SAP applications are unlike almost any other data source when it comes to loading their metadata into your data catalog.
The characteristics of SAP’s data landscape means it is simply not practical or effective to scan the application database or use an API to connect to the metadata and it all into your new data environment. And, in any case this would not be of any value to you because the application database system catalog has no ‘business friendly’ metadata you can work with.
So, it might seem counter-intuitive that SAP would be an early candidate for being one of the data sources to be ‘catalogued’ – after all, it’s hard to get at the metadata, and there is a huge amount of it.
However, think of it this way:
- If you have SAP (or an equivalent) then it is probably the main application used by your organization, or a major one at least. So, starting with a key system like SAP is going to give you a very large ‘first bite’ at the elephant-eating task you will need to undertake to find and catalog all your data sources.
- Given the size of SAP, whilst it is unlikely that you will need all if its metadata, you will probably need a significant amount – perhaps tens of thousands of tables. Therefore, you will be testing the ability of your chosen catalog to ingest and manage industrial quantities of metadata from the beginning. Better to find out if you have performance challenges up-front, rather than 6 months into the project
- More than likely, if SAP is your primary System of Record, then many other systems will interface to it in some way. If part of your task is to document interfaces for data lineage, then it would seem strange to not start with the main system
- SAP is probably one of the main systems your Catalog users experience on a day-to-day basis, so you will get an early win from the Users by populating the catalog with data elements that they are most familiar with
Believe it or not, this is achievable – quickly, effectively and accurately. However, it does depend on three factors:
- Having an effective way to mine the ‘business friendly’ SAP metadata
- Being able to choose what you need – most users will want to do some curation of the metadata before loading into the catalog
- Your catalog’s ability to ingest the mined and curated SAP metadata
If you take away only one message from this blog it is this: put your catalog vendor or implementation partner in a chair, shine a bright light in their eyes and insist that they tell you how they will support your project by helping you to derive value from your data catalog by incorporating SAP metadata. And don’t take “Oh we can get round to SAP once you’ve done some of the easier systems” or “your SAP team can help you with this” for an answer.