Building a Near Real-Time Data Platform for a Major Provincial Bank

Project Overview: Meeting the Demands of Modern Data

A major provincial bank sought to establish a state-of-the-art near real-time data platform. The core objectives were to achieve:

• Low Latency: Ensuring data is available for processing and analysis with minimal delay.
• High Throughput: Handling large volumes of data efficiently.
• Loose Coupling: Designing a system where components are independent and can be updated without affecting others.
• High Availability: Guaranteeing continuous operation and data accessibility.
• Scalability: Building a platform capable of growing with increasing data loads and analytical needs.

This ambitious project spanned from February 2019 to October 2019, aiming to transform the bank’s capabilities in data collection, transmission, processing, analysis, mining, presentation, and subscription.

Our Role and Solution: Engineering Excellence with GoldenGate and Go

Our team was entrusted with key responsibilities in the platform’s development:

Strategic Data Integration with GoldenGate

• Planning and Design: We meticulously planned and designed the architecture for near real-time data collection, transmission, and filtering. This involved moving data from various relational database systems to a central big data platform.
• GoldenGate Implementation: Leveraging the Oracle GoldenGate framework, we established robust and efficient data pipelines. This ensured reliable and timely data synchronization, crucial for the near real-time nature of the platform.

Caption: Conceptual architecture of the near real-time data platform, highlighting data flow and key technologies.

Intelligent Operations & Maintenance with Go

• Backend Management System: A critical component was the development of a sophisticated backend management program using the Go programming language.
• Comprehensive Functionality: This system provided a suite of tools for:
  • Monitoring: Real-time tracking of data flows and system health.
  • Configuration Management: Easy and dynamic configuration of data pipelines and processes.
  • Automated Backups: Ensuring data safety and recoverability.
  • Performance Statistics: Collecting and analyzing metrics to optimize performance.
  • Diagnostic Information Gathering: Facilitating rapid troubleshooting and issue resolution.
• Intelligent O&M: The Go-based system enabled intelligent operations and maintenance across numerous data streams, significantly improving efficiency and reducing manual intervention.

Outcomes and Impact

The successful implementation of the near real-time data platform empowered the major provincial bank with:

• Enhanced data processing capabilities with significantly reduced latency.
• A highly scalable and available infrastructure ready for future growth.
• Streamlined and intelligent operations, minimizing downtime and operational costs.
• Improved data accessibility for timely analysis, reporting, and decision-making.