Home >> Products >> DataMigrator Change Data Capture (CDC) for Real-Time Data Integration

DataMigrator Change Data Capture (CDC) for Real-Time Data Integration

DataMigrator Change Data Capture

iWay Software's DataMigrator is a fully automated extract, transform, and load (ETL) tool that simplifies the creation and maintenance of data warehouses and operational data stores. An intuitive, easy-to-use interface enables fast end-to-end ETL process creation involving heterogeneous data structures across disparate computing platforms. iWay DataMigrator adapters provide access to virtually any relational or legacy data source. DataMigrator uses these adapters to read from the data sources and copy data to a data warehouse or other database for analysis.

DataMigrator Change Data Capture (CDC) adds a real-time capability using database logs to read only the changes (inserts, updates, and deletions) made to tables in any of the major relational databases and delivers those changes to DataMigrator.

With shrinking batch windows and the need for up-to-date information, traditional batch-processing is sometimes unable to handle today's business needs. There is often too much data to do a full refresh of the data warehouse every night. At the same time, business analysts want their analytical databases to have the most current information. These needs can be addressed by using CDC to provide access to database changes almost as soon as they happen.

All the major relational databases log changes made to database tables for error recovery. While these logs (or journals) can be used to detect changes, every database uses a unique format. On IBM Mainframes there is nonative logging, so a subsystem is a provided that stores records in a logstream using standard VSAM journaling (JRNAD).

DataMigrator CDC makes database logs from disparate databases available in a common format so that they can be used as a data source. By using CDC data sources instead of full source tables, processing time can be greatly reduced, freeing up batch windows. Processing the changes as they occur lets a data warehouse provide near real-time access to operational data.

DataMigrator CDC is completely integrated and all configuration is done from DataMigrator's Data Management Console. A polling interval (how often to check for changes) can be specified as often as once a minute. A timeout interval (how long to keep checking) can also be specified.

Once a CDC source is configured, it can be read using SQL and used like other data sources. A Data Flow (ETL job) built for an initial load can be easily converted to perform CDC processing.

Components

  • DataMigrator Server. Extracts data from static or CDC data sources, applies transformations, and loads target tables or files
  • DataMigrator CDC. Enables the logs or journals from relational databases or VSAM to be read and makes the changed data available to the DataMigrator server for processing
  • iWay Data Adapters. Provide access to relational and legacy data sources on local and remote servers, and are available for virtually any enterprise data source
  • Data Management Console. Provides a graphical user interface for configuring the server and adapters, creating metadata and building data and process flows for ETL tasks. Provides a single console for all management tasks on any DataMigrator server

Supported CDC Data Sources

  • IBM zSeries: DB2, VSAM
  • IBM iSeries: DB2/400
  • Windows, UNIX, and Linux: IBM UDB, ORACLE 10, 11, MS SQL Server 2008

Benefits

  • Improve timeliness of information in a data warehouse
  • Dramatically reduce ETL run time by only processing changes
  • Avoid impact on source systems with a non-intrusive log-scraping technology
  • Simplify development efforts by using a CDC system completely integrated with ETL
  • Leverage existing investment in iWay DataMigrator and/or WebFOCUS