The Software Architects' Newsletter August 2023

[InfoQ Live Logo] The Software Architects' Newsletter August 2023 [View in browser]( Welcome to the InfoQ Software Architects' Newsletter! Each month, we bring you essential news and experience from industry peers on emerging patterns and technologies. This month, we focus on "event-driven architecture". This topic's technologies, patterns, and practices span the entire "diffusion of innovation" graph in our 2023 [Software Architecture and Design InfoQ Trends Report](. We see event-driven architecture (EDA) being adopted by the late majority and a related topic, "correctly built distributed systems", being adopted by the early majority. Key challenges remain, including "designing for resilience" and incorporating "workflow and decision automation platforms" into IT systems. We believe both topics are currently at the early adoption stage. News Grab Reduces Traffic Cost for Kafka Consumers on AWS to Zero Grab took advantage of the ability of Apache Kafka consumers to connect to the broker node in the same availability zone (AZ) introduced in Kafka 2.3 and reduced the traffic cost on AWS to zero for reconfigured consumers. The change has [substantially reduced overall infrastructure costs]( for running Apache Kafka on AWS. Related to this topic, Thomas Betts recently sat down with Lily Mara, Engineering Manager at OneSignal, for an InfoQ podcast, "[Building a Reliable Kafka Data Processing Pipeline with Lily Mara](". New CloudWatch Metrics for AWS Lambda Asynchronous Invocations AWS recently added [three new Amazon CloudWatch metrics]( for AWS Lambda: AsyncEventsReceived, AsyncEventAge, and AsyncEventsDropped, to monitor the performance of asynchronous event processing. Earlier, Lambda users had little visibility into processing asynchronous requests and relied on Lambda service teams to resolve processing delays that led to inefficiencies in asynchronous event processing. With the added three metrics, developers will, according to the company, have better visibility into their asynchronous invocations and can track the events sent to Lambda. Grammarly Replaces Its In-House Data Lake with Databricks Platform Using Medallion Architecture [Rafal Gancarz](, an InfoQ editor, recently covered how [Grammarly adopted the medallion architecture]( while migrating from their in-house data lake, storing Parquet files in AWS S3, to the Delta Lake lakehouse. The company created a new event store for over 6000 event types from 40 internal and external clients and, in the process, improved data quality and reduced the data-delivery time by 94%. The team chose the [Databricks Platform]( as their new data platform, which consisted of [Apache Spark]( for data processing, [Delta Lake]( for storage, and [Unity Catalog]( for data governance. The team used a medallion architecture to organize active data tables and backfill historical data from the old data lake. This data design pattern, known as "multi-hop" architecture, logically organizes data in the lakehouse by creating three distinct layers (bronze, silver, and gold), each improving data quality and structure. Cadence 1.0: Uber Releases Its Scalable Workflow Orchestration Platform Uber released a major version of its workflow orchestration platform named Cadence after six years in development. Uber and other companies use [Cadence]( to build stateful services at scale using native programming languages. The team targets usability, observability, and efficiency improvements for subsequent releases. Cadence is an open-source platform for workflow orchestration. Like other systems of that type, it helps to handle complex stateful workflows at scale with efficiency and reliability in mind. Unlike similar platforms, workflows are defined directly in programming languages like Java and Go (officially supported), Python, or Ruby (supported by the community). Case Study A Case for Event-Driven Architecture with Mediator Topology In a recent InfoQ article, [Sergii Kram](, lead software architect at iDeals, shared a business case where he used [event-driven architecture with a Mediator topology]( and several interesting implementation details, such as elastic scalability, reliability, and durable workflows. All systems were built using Kubernetes, KEDA, AWS, and .NET technologies. In the iDeals product, users upload files to later share and review online as part of due diligence processes. But behind the scenes, everything is much more complicated. Each file must be processed, converting it to a basic format, and optimizing it to view on browsers, generating previews, determining the language and recognizing text on images, collecting metadata, and other operations. The files include documents, pictures, technical drawings, archives (.zip), and videos. Sometimes the product can receive hundreds of thousands of files uploaded daily, and sometimes there are days without activity. Still, users generally want to start collaborating on a file as soon as possible after uploading it. So an architecture was needed that would scale elastically and be cost-effective. Kram and his team chose an event-driven architecture with the Mediator topology pattern. A special service called "Event Mediator", is internally named "Orchestrator". It receives the initial message to process the file and executes a file-processing script called a "workflow". The workflow is a declarative description of what must be done with a particular file as a set of discrete steps. Each step type is implemented as a separate stateless service. In pattern terms, they are called Event Processors or "Converters". The end result created was a highly scalable system that is easy to extend, modify, and test, with good observability and cost-effectiveness. Kram stated that they did have some failures and incidents. Most were related to defects in third-party libraries appearing only in edge cases under heavy load. As responsible engineers, they tried their best to contribute diagnostic information and fixes, and they were very grateful to the OSS maintainers for their support and responsiveness. One lesson learned was to have easily configurable limits so that when something goes wrong, it is easy to reduce the load, let the system stabilize, recover, and continue operations under degraded throughput while working on the fix. This content is an excerpt from a recent InfoQ article by Sergii Kram, "[A Case for Event-Driven Architecture with Mediator Topology](". To get notifications when InfoQ publishes content on these topics, follow "[Event-Driven Architecture](", "[Distributed Systems](", and "[Asynchronous Programming](" on InfoQ. Missed a newsletter? You can find all of the [previous issues]( on InfoQ. SPONSORED [Architecting Distributed Transactional Applications (By O'Reilly)]( Learn how to build efficient, elastically scaling, multi-region applications using this blueprint that walks through options, approaches, and best practices for both the application and persistence layers. [Download now]( - sponsored by Cockroach Labs Upcoming Events QCon: For practitioners, by practitioners [Uncover the latest practices in frontend, emerging AI and ML trends, how to become a Staff+ Engineer and more]( At QCon San Francisco (Oct 2-6) learn from world-class software practitioners from Monzo, Google, Netflix, CapitalOne, UC Berkeley, NY Times & more on the most important topics you need to know about right now in an ever-changing environment. You'll leave with actionable insights you can take back to your team. [Explore the speakers](. [1-day tickets are available now for $1,190. Early bird pricing is available until September 5.]( About InfoQ Senior software developers rely on the InfoQ community to keep ahead of the adoption curve. One of the main reasons software architects and engineers tell us they keep coming back to InfoQ is because they trust the information provided and selected by their peers. We’ve been helping software development teams adopt new technologies and practices for over 15 years through InfoQ articles, news items, podcasts, tech talks, trends reports, and QCon software development conferences. We hope you find this newsletter useful. If not, you can unsubscribe using the link below. [Unsubscribe]( Forwarded email? Subscribe and get your own copy. [Subscribe](   Follow InfoQ on: [Twitter]( [Facebook]( [Youtube]( [LinkedIn]( You have received this email because you subscribed to "The Architects' Newsletter". To stop receiving the Architects' Newsletter, please click the following link: [Unsubscribe]( - - - C4Media Inc. (InfoQ.com), 705-2267 Lake Shore Blvd. West, Toronto, Ontario, Canada, M8V 3X2