Data Science Insider: November 11th, 2023

In This Week’s SuperDataScience Newsletter: AI Reshapes Engineering. A Data Scientist's Binary Analysis Alternative. Mastering PySpark Unit Testing with unittest and pytes. Neural Networks Embrace Human-Like Thinking. Wearable AI Revolution Unveiled. Cheers, - The SuperDataScience Team P.S. Have friends and colleagues who could benefit from these weekly updates? Send them to [this link]( to subscribe to the Data Science Insider. --------------------------------------------------------------- [AI Reshapes Engineering]( brief: Naveen Zutshi discusses the transformative impact of AI on engineering careers in this illuminating Forbes article. As generative AI tools automate coding tasks, focus shifts to deep data and domain knowledge, along with essential soft skills. The demand for skills beyond programming includes strong problem-solving, analytical thinking, and adaptability. Zutshi emphasizes the importance of deep data science expertise over technical prowess and highlights the evolving role of prompt engineers in an AI-driven organization. The article explores the changing landscape of coding bootcamps and the need for education that aligns with the demands of AI-driven enterprises. For data scientists, this shift towards deep data science expertise and soft skills underscores the importance of adapting to the changing demands of the industry. Why this is important: The emergence of prompt engineering and the integration of no-code solutions highlight the dynamic nature of the field, requiring data scientists to stay abreast of evolving technologies and skill sets. [Click here to learn more!]( [A Data Scientist's Binary Analysis Alternative]( brief: In this Towards Data Science article, Akif Mustafa explores Probit regression as an alternative to logistic regression for binary outcome analysis. The Probit model employs the cumulative distribution function of the normal distribution to model the relationship between a binary outcome variable and independent variables. The article provides an intuitive example involving weight and depression status, explaining the estimation of parameters and the mathematical structure behind Probit regression. A comparison with logistic regression reveals similarities, with Probit being less sensitive to extreme values. While logistic regression is commonly preferred in practice, having knowledge of Probit regression enhances the data scientist's toolkit, enabling them to choose the most appropriate model for different scenarios. Why this is important: The distributed nature of Spark makes unit testing particularly critical, helping identify issues that may not be visible until code runs on large datasets. [Click here to read on!]( [Mastering PySpark Unit Testing with unittest and pytes]( In brief: This Medium article explores the importance of unit testing in modern Data Engineering, focusing on creating unit tests for PySpark applications using Python's unittest and pytest libraries. The author provides code examples, discusses the advantages and disadvantages of each library, and emphasizes the significance of unit testing for data professionals. Unit testing ensures individual components function as intended, minimize bugs, detect regressions, serve as documentation, facilitate refactoring, and build stakeholder trust. In the context of PySpark applications, unit testing becomes crucial due to Spark's distributed nature. The article is useful to data scientists as it compares unittest and pytest, concluding that both are excellent choices, and suggests starting with unittest for Python newcomers. Why this is important: The distributed nature of Spark makes unit testing particularly critical, helping identify issues that may not be visible until code runs on large datasets. [Click here to discover more!]( [Neural Networks Embrace Human-Like Thinking]( In brief: Neural networks have achieved a breakthrough in mimicking human-like thinking, displaying "systematic compositionality," a key aspect of human intelligence. Published in Nature, the study challenges a decades-long debate in cognitive science, asserting that with training, neural networks can acquire human-like abilities previously considered beyond their scope. The research, led by Brenden Lake of New York University and Marco Baroni of Pompeu Fabra University, introduces the meta-learning for compositionality (MLC) method, enabling neural networks to match or surpass human performance in tasks involving rule application and comprehension of written instructions. The researchers also set MLC against two neural network-based models from OpenAI in a series of tests and found both MLC and humans performed far better than OpenAI models. Why this is important: This advancement in systematic compositionality, holds significant implications for data scientists working with AI models, bringing AI closer to human-like thinking, and enhancing the ability to combine known concepts in new ways. [Click here to see the full picture!]( [Wearable AI Revolution Unveiled]( In brief: US startup Humane is set to unveil its AI Pin, a revolutionary $699 wearable smartphone sans screen, featuring a $24 monthly subscription. Leaked documents obtained by The Verge reveal the device's details ahead of the official launch. The Pin, a square magnetic device, attaches to clothing, doubling as a battery pack with swappable batteries. It runs on a Humane-branded T-Mobile network, integrating Microsoft and OpenAI AI models. With a voice-centric interface and innovative green laser projection, the Pin aims to be a standalone device, challenging the smartphone paradigm. The Humane Subscription offers cell data, cloud storage, and unlimited AI queries. The operating system, Cosmos, suggests a seamless AI-driven experience akin to ChatGPT's plugin system. Why this is important: The device's reliance on voice interaction, gesture control, and AI features like language translation and email summarization reflects evolving user interfaces and the expanding role of AI in daily life. [Click here to see the full picture!]( [Super Data Science podcast]( In this week's [SuperDataScience]( [Podcast]( episode, Dr. Blake Richards discusses the world of AI and human cognition. Learn about the essence of intelligence, the ways AI research informs our understanding of the human brain, and discover the potential future scenarios where AI and humanity might intersect. [Click here to find out more!]( --------------------------------------------------------------- What is the Data Science Insider? This email is a briefing of the week's most disruptive, interesting, and useful resources curated by the SuperDataScience team for Data Scientists who want to take their careers to the next level. Want to take your data science skills to the next level? Check out the [SuperDataScience platform]( and sign up for membership today! Know someone who would benefit from getting The Data Science Insider? Send them [this link to sign up.]( # # If you wish to stop receiving our emails or change your subscription options, please [Manage Your Subscription]( SuperDataScience Pty Ltd (ABN 91 617 928 131), 15 Macleay Crescent, Pacific Paradise, QLD 4564, Australia