Optimisation of ETL design
Initial situation
- a central data warehouse exists primarily for the purposes of effective planning, decision-making, reporting of financial results and consolidation of key information from the bank's operations systems
- within BASEL II implementation, the daily volume of processed data increased by 100%
- the period of duration of ETL processes exceeded the defined period of ten hours
- exceeding of the time for processing incremental data had fatal impacts on the function of the bank's analytical processes, which are implemented above the data warehouse and are very important for the day-to-day operation of the bank's dependent organisational units
Project goals
- overall evaluation of the technical solution, analysis of the causes behind the non-standard length of daily processing and auditing of the existing Business Intelligence solution
- design of a solution to eliminate bottlenecks
Project results
- detailed analysis of the system and applied transformation algorithms
- description of detected bottlenecks in the system and proposals for eliminating them
- designed solution that reduces the running of batch ETL processes to approx. 70% of the original period, supports easy development and facilitates administration of the entire Business Intelligence environment
- solution based on the assumption of a 50% increase of the volume of processed data

