Our architecture included a number of pre-defined design patterns that allowed for faster development that supports agile more directly. These included design patterns for:
- Loading raw data (incremental, full, flat file, manual input, process push)
- Loading dimensions (type 1 and 2)
- Loading detailed fact tables
- Loading consolidated / summary tables
Reuse of design patterns supports agile development in many ways. It speeds development of similar features, minimizes reinvention, and enables new team members to be productive faster.
There are a lot of options when developing a data flow and data architecture for your data warehouse. These are just a few examples of ways you can design the architecture to support incremental, agile development. As a result of our architecture design:
- Most refactoring / reloading has been of a scale that can be completed quickly.
- Design patterns provide guidance for new development and speed the orientation of new team members.
- Full history in the Data Lake supports historical reloads, "as was" queries, experimentation, research, prototyping, and also trouble shooting the source OLTP systems.
- The Data Lake has adapted to multiple loading patterns, including direct push from information producers. This is an architectural pattern we developed that enables some very innovative data management practices (more to come on this topic in future articles).
To sum up, there are many benefits to having a predefined data warehouse architecture. Some of these include:
- Provides an organizing framework - the architecture draws the lines on the map in terms of what the individual components are, how they fit together, who owns what parts, and priorities.
- Improved flexibility and maintenance - allows you to quickly add new data sources, and add / modify data from existing sources.
- Faster development and reuse - warehouse developers are better able to understand the data warehouse process, data base contents, and business rules more quickly.
- Coordinated parallel efforts - multiple, relatively independent efforts have a chance to converge successfully.
All of these benefits also allow you to leverage agile development more readily.
Additional steps in building this foundational approach to agile data warehouse development include:
- Ensuring solid testing and tools
- Implementing a robust data quality program
- Giving the development team the ability to self manage their agile development approach, incorporating continuous improvement
I will cover these remaining steps in the next few upcoming articles.
Source: CIO US
Sign up for CIO Asia eNewsletters.