The Agile (SAP) Datawarehouse

As of today, a lot of companies are transforming into an Agile organization. Did you notice this at your company already? Probably yes, because being more flexible as a company results in a better response to the needs of customers. And we all know how that (ideally) translates to revenue and profits.

The issue with the adaption of Agile methodologies is that most organizations have processes (and therefore people) in place that do not cooperate with the Agile philosophy. Transforming such organizations is quite a difficult job. Luckily specialized Agile consultants can facilitate in bridging this gap

There is one thing most experts tend to forget which is that theory is not always aligned with reality. Particularly in an SAP Datawarehouse environment that consists of BW4HANA, HANA Enterprise and a Data Lake. How can we simplify this environment and make it a more Agile (SAP) Datawarehouse? Hereby four subjects you should think about when using Agile methodologies with respect to Business Intelligence.

Virtualization versus reusability

Virtualizing data models is a trend that happens more often. Within traditional BI (talking about BW) data is transformed and staged through different layers. Each layer physically stores a representation of the data. By the means of virtualizing we reduce the number of layers and therefore also the data footprint of the Data warehouse. New technologies support these virtualizing principles. As an example, SAP BW4HANA has artifacts like ODS Views and HANA CompositeProviders that enables the developer to expose, join and transform data more virtually without the need of staging.

These new techniques facilitate Agile Development. The downside is that when adapting these new virtualizing techniques developers tend to focus on the short-term goals (finishing stories; higher velocity) instead of focusing on long-term goals (reusability of data models). With the consequences that after a couple of sprints (especially when you do cross-domain developments) stovepipes have been built whereas proper integration layers are still missing. Resulting in rework and negative impact on existing data models.

How to tackle this? Be sure that during poker sessions the estimations are based on a broader perception than the current sprint. Sometimes it is even wiser to just stage data when it serves a broader function. It all comes back to having  1) a clear data strategy 2) solid guidelines and 3) a bird’s-eye view on the required solution.

Documentation: not what, but why

When you talk about implementing changes or doing a full-scale BI project, in the end everything must be documented. The main purpose is to safeguard the supportability of the build solution by a service organization and to keep the knowledge in-house in case an external resource is leaving.

Nevertheless, documenting a solution is not the most enjoyable thing to do. So what happens in real life, is that (BI) developers tend to take the easy way and write down things that are already available within the systems. This includes copying of code or using system generated documentation. The problem is this only answers the question on what has been built and not why.  What would you like to achieve with the technical documentation? Is it necessary to document developments that are simple? Think of acquisition layers, one-to-one mappings or simple queries. Or is it better to focus on data models that actually adds intelligence to the data?

By focusing more on the why question you prevent unnecessary time spent on irrelevant models that do not require any documentation. Instead this time can be spent on improving the quality of documentation of complex data models. This might be a mind-shift, since a lot of organizations still want to have full control on everything. But eventually it will improve the overall quality of changes and making an organization (and thus the data warehouse) more Agile.

Proven technologies vs Innovation

Business Intelligence is a field that is changing rapidly. New innovation techniques and tools come and go. Some new possibilities provide an answer on earlier technical challenges. Adapting these new techniques (early) might be interesting at first but cautious with it since it could have a negative impact on the stability of the system and even on the velocity of the team.  When we talk about innovation this can be new products, processes, techniques, guidelines or even methodologies. One thing they have in common is that they all introduce a new way of working.

Introducing a new way of working goes (in most cases) hand in hand with uncertainty. Uncertainty in terms of system availability (How does the new product perform on the system?), team members (Is my team capable of using this new technique? Are they certified?) and even the end users (Do they require new training material?).

So should you incorporate innovations in an Agile environment? The answer is Yes. Agile developments are excellent for adapting and trying out new things (think big, start small). The only down-side is that occasionally using innovative products (even though they might sound solid) cannot fulfill a need completely and result in a solution that is not working properly. In such cases solid traditional techniques would have worked since they have a proven track record.

The bottom line is,  there should be a right balance between using and adapting innovation. But play it safe and always be critical on what to choose, because even though using new techniques sounds seductive at first, proven old-style techniques might work better in the long term (results in less rework).

Supportability: Begin with the end in mind

Every development, change or project should eventually be maintained by a support organization. In theory this means that also support activities should be visible within an Agile organization. In practice hardly any time is reserved for this. This is a shame! Because with appropriate maintenance on data models or its definition, Agile teams are able to respond more efficiently (and thus quicker) on new requirements.

There are multiple ways of improving this. Real cases exist where support activities are incorporated for a specific percentage within a sprint. But also sprints where the team is fully dedicated to improve the operations are no exception. One term that is mentioned quite a lot is DevOps. This Agile method actually bridges the gap between software development and the operation of it. A DevOps team is fully responsible for the run of and maintenance of a specific development (for example data model), hence the motto is “we build it, we run it”.

The above requires organizational change. We all know that this is not easy to implement. But what if we just gave more attention on defining stories and have a broader view on the required deliverables. Try to not only incorporate solely business (or technical) requirements, but take the support perspective into account too. A few examples on questions that should be answered during the story definition that really improves the supportability could be: Can a data model be reloaded easily? What is the impact on the business when data is not loaded? What are the critical elements of the data model that might cause incidents in the future? How can we improve the performance of the data loads?

In the end it all comes down to having data models that are fast, stable, easily changeable and maintainable. Such models go hand in hand with the verb Agile and do require less or no rework at all when new requirements should be incorporated.

Wrap-up

All topics mentioned should trigger you to decide which way to go. Either way at a certain point these topics will pop-up during retrospectives. Why not do anything with it pro-actively before start building your Agile Datawarehouse?  At McCoy & Partners we can assist you in making the right choices. Feel free to contact us anytime!