Bridging the Gap: Effectively Managing Data in the Era of Digital Apps and Legacies

Our approach involves adding a new computing layer for CDC, which enhances the process of data integration. Despite this addition, our approach still offers the advantage of seamless integration with legacy applications, without the need for any additional computing resources or modifications to the existing codebase. Moreover, the data streaming process can be easily configured to capture a wide range of data events, including inserts, updates (considering the before and after state), and deletion indications. This flexibility enables comprehensive data synchronization, ensuring that all relevant data changes are efficiently captured and streamed in real-time. With our approach, we can read data once and make it available for reuse multiple times in real-time without causing any crashes or disruptions to the legacy systems. On the other hand, our approach allows flexibility and respects the diverse paces and methodologies of different squads within the organization. Some squads solely consume data from our data stream, while others may create their own replicas for local persistence, such as our CRM solution. However, in all cases, data is available for consumption in real-time, eliminating the need for batch processing against the legacies that used to require hours to make data available to end users. This shift in mindset also triggers a chain reaction, extending to digital apps themselves becoming part of this new paradigm. These apps can now trigger events, which are efficiently handled by microservices and middleware at an asynchronous level, enabling seamless integration with other applications. This eliminates the need for time-consuming batch processing. On the other hand, it is crucial to emphasize the significance of peopleware in this context. Adopting an event-driven architecture requires a shift in the architectural mindset when designing and developing digital apps in the background. It also necessitates strong collaboration between squads, going beyond the requirements of traditional process and data integration. The interdependence among the applications developed in this environment relies on near real-time processing, making it essential for each squad to understand the behaviors of applications developed by other squads in order to effectively handle errors and ensure proper event directionality. At the same time, despite the numerous advantages of this technique, our project is not without its challenges. Configuring data streaming in both the publisher and consumer sides is not a straightforward task and requires careful attention. It also involves understanding how to effectively utilize cloud offerings. For example, squads must familiarize themselves with concepts such as topics and queues rather than solely relying on development frameworks. Mishandling these concepts can lead to issues and bugs, such as record duplication. Therefore, having a strong architecture team is crucial to support the overall transformation that comes with adopting this new mindset. Last but not least, adopting an event-driven architecture also brings positive implications for advanced analytics and AI. By leveraging the data streaming approach, these technologies can effectively consume real-time data, enabling valuable insights, AI calibration, and the development of predictive models. Our company has embraced this new approach, leveraging it to deliver exceptional solutions to our students. Additionally, it has facilitated the modernization of our legacy systems, which play a key role in our IT ecosystem. This modernization enables us to reuse integrations without the need for peer-to-peer middleware development for each project. Our technology department assumes a leading role in delivering business solutions with an accelerated time-to-market, benefiting from the speed and efficiency offered by this approach.