The Software Architects' Newsletter September 2023

[InfoQ Live Logo] The Software Architects' Newsletter September 2023 [View in browser]( Welcome to the InfoQ Software Architects’ Newsletter! Each month, we bring you essential news and experience from industry peers on emerging patterns and technologies. This month, we again focus on "event-driven architecture". This topic's technologies, patterns, and practices span the entire "diffusion of innovation" graph in our 2023 [Software Architecture and Design InfoQ Trends Report](. We see event-driven architecture (EDA) being adopted by the late majority and a related topic, "correctly built distributed systems", being adopted by the early majority. Key challenges remain in this space, including "designing for resilience" and incorporating "workflow and decision automation platforms" into IT systems. We believe both topics are currently at the early adoption stage. News Delivery Hero Implements Event-Driven Architecture to Handle Baemin Growth Baemin, a South Korean food delivery service owned by Delivery Hero, successfully navigated the challenges of rapid customer member growth by [moving from a monolithic architecture to a more flexible event-driven]( microservices-based system. As a recent Delivery Hero blog post shared, a cornerstone of the Baemin system is an event-driven architecture. The architecture's standout features include the meticulous separation of concerns through layered event subscribers and the use of event storage for reliability. Utilizing a common Relational Database Management System (RDBMS) and employing the Transactional Outbox Pattern, Baemin has built a robust framework that's scalable and dependable. Getting to Know Dapr with Mark Fussell and Yaron Schneider In a recent episode of the InfoQ podcast, Thomas Betts talked to Mark Fussell and Yaron Schneider about the [Distributed Applications Runtime, Dapr](. In the latest InfoQ Architecture and Design Trends Report, Dapr is part of the early adopter ideas of design for portability and cloud-bound applications. Dapr provides APIs that abstract away the infrastructure details for modern applications, providing secure, best-practice implementations that work across any cloud, framework, and language. Reddit Adopts Server-Driven UI for Its New Feed Architecture Across Mobile Apps Reddit [reworked its feeds' functionality]( in the iOS mobile app and introduced it to the Android app. In both cases, they used the Server-Driven UI (SDUI) as their communication approach, which allows localized content layout changes without mobile app releases. In 2022, the company committed to making Reddit easier to use, and since then, several improvements have been made, including better posting experience, searchable comments, and updated moderator tools. This year's primary focus is on new and improved feeds with better user experience. The team has taken on building a new feeds platform from scratch. The move was primarily dictated by the pains of working with previous feeds architecture. LinkedIn's Open-Source "iris-message-processor" Achieves 86.6x Faster Escalation Management Speeds LinkedIn developed a new open-source service called "[iris-message-processor](" to enhance the performance and reliability of its existing Iris escalation management system. The service significantly improves processing speeds, being ~4.6x faster under average loads and ~86.6x faster under high loads than its predecessor. "iris-message-processor" is a fully distributed replacement for the previous iris-sender subprocess, allowing for more concurrent processing and direct sending of messages to appropriate vendors. It enables increased horizontal scaling and reduces demands on the existing database by not using it as a message queue. Grab Reduces Traffic Cost for Kafka Consumers on AWS to Zero Grab took advantage of the ability of Apache Kafka consumers to connect to the broker node in the same availability zone (AZ) introduced in Kafka 2.3 and reduced the traffic cost on AWS to zero for reconfigured consumers. The change has [substantially reduced overall infrastructure costs]( for running Apache Kafka on AWS. In a related QCon London presentation shared on InfoQ this month, Rosemary Wang, engineer and developer advocate at HashiCorp, discussed moving "[From Cloud-Hosted to Cloud-Native](". Sponsored [ScyllaDB]( [SQL to NoSQL Architectural Differences]( When and how to migrate data from SQL to NoSQL are matters of much debate. It can certainly be a daunting task, but when your SQL systems hit architectural limits or your cloud provider expenses skyrocket, it’s probably time to consider a move. Read this white paper to learn about the architectural differences between SQL and NoSQL, the tradeoffs between flexibility, scale and cost, and considerations for successful SQL to NoSQL migrations. Learn more about this topic in the eBook "[SQL to NoSQL: Architecture Differences and Considerations for Migration](", sponsored by ScyllaDB Case Study Building Kafka Event-Driven Applications with KafkaFlow [KafkaFlow]( is an open-source framework by [FARFETCH](. It helps .NET developers working with Apache Kafka to create event-driven applications. KafkaFlow lets developers easily set up "Consumers" and "Producers". The simplicity makes it an attractive framework for businesses seeking efficiency and robustness in their applications. Why use KafkaFlow? KafkaFlow provides an abstraction layer over the Confluent .NET Kafka client. It does so while making it easier to use, maintain, and test Kafka consumers and producers. Imagine you need to build a Client Catalog for marketing initiatives. You will need a service to consume messages that capture new Clients. Once you start laying out your required service, you’ll notice that existing services are inconsistent withhow they consume messages. It's common to see teams struggling and often solving simple problems such as graceful shutdowns. You've figured out that you have four different implementations of a JSON serializer across the organization, just to name one of the challenges. What functionality does KafkaFlow provide? Adopting a framework like KafkaFlow simplifies the process and can speed up the development cycle. KafkaFlow has a set of features designed to enhance the developer experience: - Middlewares: KafkaFlow allows developers to create middlewares to process messages, enabling more control and customization of the Kafka consumer/producer pipeline. - Handlers: Introduces the concept of message handlers, allowing developers to forward message processing from a topic to a message-type dedicated handler. - Deserialization Algorithms: Offers a set of Serialization and Deserialization algorithms out-of-the-box. - Multi-threaded Consumers: Provides multi-threading with message order guaranteed, helping to ensure optimal use of system resources. - Administration API and Dashboard: Provides API and Dashboards to manage Consumers and Consumer groups, with operations such as pausing, resuming, or rewinding offsets, all at runtime. - Consumer Throttling: Provides an easy way to bring priorities to topic consumption. The full-length version of this article explores each of these features in detail. Exploring the Future of KafkaFlow Currently, KafkaFlow provides a robust, developer-friendly abstraction over Kafka that simplifies building real-time data processing applications with .NET. However, like any active open-source project, it’s continually evolving and improving. Given the [project's current trajectory](, we might anticipate several developments. For instance, KafkaFlow could further enhance its middleware system, providing more control and flexibility over message processing. We might also see more extensive administrative APIs, giving developers greater control over their Kafka clusters. Being extensible by design, we can expect the KafkaFlow community to grow, leading to more contributions, innovative features, extensions, and support. As more developers and organizations adopt KafkaFlow, we’re likely to see an increase in learning resources, tutorials, case studies, and other community-generated content that can help new users get started and existing users get more from the library. This content is an excerpt from a recent InfoQ article by Guilherme Ferreira, "[Building Kafka Event-Driven Applications with KafkaFlow](". To get notifications when InfoQ publishes content on these topics, follow "[Event-Driven Architecture](", "[Distributed Systems](", and "[Asynchronous Programming](" on InfoQ. Missed a newsletter? You can find all of the [previous issues]( on InfoQ. Sponsored [ScyllaDB]( [NoSQL Database Trends Report: Expectation vs. Reality]( A recent DZone survey assessed how the reality of working with relational and different types of NoSQL databases compared to the team's original expectations. Download this report to learn which NoSQL database types seemed better in theory than in practice, NoSQL database trends through adoption and use cases, how CAP theorem tradeoffs are shifting year over year, and more. Learn more about this topic in the report "[NoSQL Database Trends Report: Expectation vs. Reality](", sponsored by ScyllaDB Upcoming events QCon: For practitioners, by practitioners [AI, ML, FinTech and data engineering tracks announced for QCon London 2024 (8-10 April).]( Learn from 75+ senior software leaders across [15 technical topics]( including; AI, machine learning, FinTech, frontend, mobile, programming languages, software architecture, GenAI, LLMs, performance engineering, leadership, teams, APIs, modern software, data engineering, cloud-native [and more](. Launch pricing expires Oct 12. [Book now and save £600](. About InfoQ Senior software developers rely on the InfoQ community to keep ahead of the adoption curve. One of the main reasons software architects and engineers tell us they keep coming back to InfoQ is because they trust the information provided and selected by their peers. We’ve been helping software development teams adopt new technologies and practices for over 15 years through InfoQ articles, news items, podcasts, tech talks, trends reports, and QCon software development conferences. We hope you find this newsletter useful. If not, you can unsubscribe using the link below. [Unsubscribe]( Forwarded email? Subscribe and get your own copy. [Subscribe](   Follow InfoQ on: [Twitter]( [Facebook]( [Youtube]( [LinkedIn]( You have received this email because you subscribed to "The Architects' Newsletter". To stop receiving the Architects' Newsletter, please click the following link: [Unsubscribe]( - - - C4Media Inc. (InfoQ.com), 705-2267 Lake Shore Blvd. West, Toronto, Ontario, Canada, M8V 3X2