Case Study
High performance data loading
Ability to load and process up to 500.000 files per day, corresponding to about 80GB of raw data per day
Challenge
- Cope with Real-time requirements for loading of large data files, being able to load and process up to 500.000 files per day, that correspond to about 80GB of raw data per day.
Major problems addressed:
- loading of data files with different data sets;
- data processing capabilities;
- real time performance monitoring;
- limit values checking;
- data quality verification;
- high volumes of data to be loaded and process each 15 minutes that can grow to
- 80 GB per day and up to 500.000 files per day;
- Being able to load and process files in less than one second using several parallel streams.
Solution
- The solution involved an innovative architecture, in which the data is kept and processed in the database layer.
- The application layer is responsible for controlling the execution of the database operations, but does not process data directly. In this way it was possible to reduce the communication between application and database to a minimum, assigning all the data processing to the database, and achieving superior performance.
- The loading of data from the files to the database is performed through the External Table mechanism available in Oracle 10g, and all the remaining data processing is realized though the execution of SQL statements, profiting from all advanced SQL functionality provided by Oracle, as for example multi-table insert and merge. All data loading and processing is performed based on the configuration defined using the administration GUI. The administration GUI allows, among others, data source definition, data target (database) definition, easy mapping definition, data quality rules definition, management of the active loaders, configuration consistency check and data files validation.
Benefit
- Greater efficiency and performance enabling faster access to information, in almost real-time.
Client / Partner
Product & Service
Case Studies
-
High performance data loading
Ability to load and process up to 500.000 files per day, corresponding to about 80GB of raw data per day
