Data Engineering Round-Up

A Collection of Top Software DevOps News, Articles, Presentations April 2023 [InfoQ] Data Engineering Round-Up Sponsored by [Cockroach Labs] [Latest Content](#latest-content), [Top Viewed Content](#top-viewed-content), [Top News](#news), [Top Articles](#top-articles), [Top Presentations](#top-presentations-and-interviews) In this special newsletter we bring you up to date on all the new content and news related to Data Engineering on InfoQ. We are also maintaining a portal page for this content on InfoQ at: [(. [] Latest Content on InfoQ [Meta AI’s Large Language Model with 10x Fewer Parameters]( (news, Mar 18, 2023) [ChatGPT Now Available in Preview on Microsoft’s Azure OpenAI Service]( (news, Mar 15, 2023) [Zero-Copy In-Memory Sharing of Large Distributed Data: V6d]( (news, Mar 14, 2023) [The Wonders of Postgres Logical Decoding Messages]( (articles, Mar 06, 2023) [Google Uses AutoML to Discover More Efficient AI Training Algorithm]( (news, Mar 21, 2023) [Architecting Distributed Transactional Applications (By O'Reilly)]( Learn how to build efficient, elastically scaling, multi-region applications using this blueprint that walks through options, approaches, and best practices for both the application and persistence layers. [Download now](. Sponsored content [Architecting Distributed Transactional Applications]( [] Top Viewed Content on InfoQ [OpenAI Unveils a Powerful, Cost-Effective, and User-Friendly Embedding Model]( (news, Dec 26, 2022) [Deep Learning Pioneer Geoffrey Hinton Publishes New Deep Learning Algorithm]( (news, Jan 10, 2023) [Microsoft Unveils VALL-E, a Game-Changing TTS Language Model]( (news, Jan 27, 2023) [Google AI Unveils Muse, a New Text-to-Image Transformer Model]( (news, Jan 13, 2023) [AWS Announces the General Availability of Amazon Omics]( (news, Dec 04, 2022) [] Top News [Google Unveils MusicLM, an AI That Can Generate Music from Text Prompts]( Google researchers have introduced MusicLM, an AI model that can generate high-fidelity music from text. MusicLM creates music at a constant 24 kHz throughout a number of minutes by modeling the conditional music generating process as a hierarchical sequence-to-sequence modeling problem. [AI Developers Release Open-Source Implementations of ChatGPT Training Algorithm]( AI research groups LAION and CarperAI have released OpenAssistant and trlX, open-source implementations of reinforcement learning from human feedback (RLHF), the Algorithm used to train ChatGPT. Independent AI developer Phil Wang has also open-sourced his own implementation of the algorithm. [OpenAI Releases Conversational AI Model ChatGPT]( OpenAI released ChatGPT, a conversational AI model based on their GPT-3.5 language model (LM). ChatGPT is fine-tuned using Reinforcement Learning from Human Feedback (RLHF) and includes a moderation filter to block inappropriate interactions. [Architecting for Scale (By O'Reilly)]( Learn techniques for building systems that can handle huge quantities of traffic, data, and demand—without affecting the quality your customers expect. Architects, managers, directors in engineering and operations organizations will learn how to build applications at scale that run more smoothly. [Download now](. Sponsored content [Architecting for Scale]( [OpenAI is Rolling out ChatGPT Plus as a Premium Service]( OpenAI has announced the release of ChatGPT Plus, a premium version of its well-liked ChatGPT chatbot. The new service intends to give users a premium experience that includes ChatGPT access during peak hours, priority access to new features and upgrades, and quicker response times. [GitHub Enhanced Copilot with New AI Model and Security-Oriented Capabilities]( GitHub Copilot adopted a new AI model which is both faster and more accurate than the previous one, says GitHub. Additionally, GitHub has started using AI to detect vulnerabilities in Copilot suggestion by blocking insecure coding patterns in real-time. [] Top Articles [Design Pattern Proposal for Autoscaling Stateful Systems]( In this article, author discusses the challenges in auto-scaling stateful storage systems and proposes a design solution to scale up (vertical) and scale out (horizontal) with minimum configuration. [article]( [How I Contributed as a Tester to a Machine Learning System: Opportunities, Challenges and Learnings]( Testers usually struggle to understand ML-based systems and explore what contributions they can make. This is a journey of assuring quality of ML-based systems as a tester. [article]( [DynamoDB Data Transformation Safety: from Manual Toil to Automated and Open Source]( The open source utility Dynamo Data Transform was built to simplify and build safety and guardrails into data transformation for DynamoDB based systems. [article]( [Moving towards a Future of Testing in the Metaverse]( This article describes the metaverse concept, explores key engineering challenges and quality concerns, and walks through technological advances in AI and software testing to mitigate the challenges. [article]( [Understanding and Debugging Deep Learning Models: Exploring AI Interpretability Methods]( ML interpretability refers to a user's ability to explain decisions made by an ML system. In this article, author discusses several methods with use case details on LIME and Shapley Values. [article]( [Foundations of Scalable Systems (By O'Reilly)]( In these complimentary chapters from O’Reilly, you will explore the essential ingredients of designing scalable solutions, including replication, state management, load balancing, and caching. [Download the three free chapters now](. Sponsored content [Foundations of Scalable Systems]( [] Top Presentations [Resilient Real-Time Data Streaming across the Edge and Hybrid Cloud]( Kai Waehner explores different architectures and their trade-offs for transactional and analytical workloads. Real-world examples include financial services, retail, and the automotive industry. [Kai Waehner]( [Data Versioning at Scale: Chaos and Chaos Management]( Einat Orr discusses several technologies that version large data sets, the use cases they support and the technology developed to best support those use cases. [Einat Orr]( [Open Machine Learning: ML Trends in Open Science and Open Source]( Omar Sanseviero discusses the trends in the ML ecosystem for Open Science and Open Source, the power of creating interactive demos using Open Source libraries and BigScience. [Omar Sanseviero]( [Orchestrating Hybrid Workflows with Apache Airflow]( Ricardo Sueiras discusses how to leverage Apache Airflow to orchestrate a workflow using data sources inside and outside the cloud. [Ricardo Sueiras]( [Taming the Data Mess, How Not to Be Overwhelmed by the Data Landscape]( Ismaël Mejía reviews the current data landscape and discusses both technical and organizational ideas to avoid being overwhelmed by the current lack of consolidation of the data engineering world. [Ismaël Mejía]( [Connect with InfoQ on Twitter]( [Connect with InfoQ on Facebook]( [Connect with InfoQ on LinkedIn]( [Connect with InfoQ on Youtube]( You have received this message because you are subscribed to the “Special Reports Newsletter”. To stop receiving this email, please click the following link: [Unsubscribe]( C4Media Inc. (InfoQ.com), 2275 Lake Shore Boulevard West, Suite #325, Toronto, Ontario, Canada, M8V 3Y3