Jupyter AI, EDA with ChatGPT, CRUD with Pinecone & Responsible AI Toolbox

PaLM 2 & Duet AI for Google Cloud – an AI-powered collaborator                                                                                                                                                                                                                                                                                                                                                                                                                 [Open in app]( or [online]() [Jupyter AI, EDA with ChatGPT, CRUD with Pinecone & Responsible AI Toolbox]( PaLM 2 & Duet AI for Google Cloud – an AI-powered collaborator May 12   [Share](   👋 Hey, "The best models are created when data scientists and domain experts collaborate." - [Carla Gentry, Chief Data Scientist at Analytical-Solution]( Sharing knowledge among data scientists and domain experts is crucial to create effective models. Collaborating and incorporating responsible AI technology can help develop innovative solutions that meet the needs of the market while achieving accuracy, reduced bias, and ethical alignment.   This week's resources in [DataPro#43]( focus on a range of topics related to data analysis and artificial intelligence. One of the central themes is the [Responsible AI Toolbox]( which provides techniques for developing ethical AI models. Another topic is the [use of large language models to generate robot task plans]( which can improve automation and efficiency in various industries. Additionally, we cover a [hosted TensorBoard experience]( [Google I/O event]( [Rust Polars]( and [time series data analysis with sARIMA and Dash](. I hope these topics offer valuable ideas and knowledge for anyone interested in the field of data science and AI. Get ready for a productive learning experience!   If you’re interested in sharing ideas to foster the growth of the data community, then this survey is for you. Consider sharing your thoughts and get a FREE bestselling Packt ebook as PDF. Jump on in!  [TELL US WHAT YOU THINK]( Cheers, Merlyn Shelley Editor in Chief, Packt This Week’s Key Highlights: - [Generative AI extensions for Jupyter]( - [Exploratory Data Analysis with ChatGPT]( - [CRUD with Pinecone]( - [Duet AI for Google Cloud – an AI-powered collaborator]( - [Few-Shot Object Learning in Robotic Environments]( Latest Research on GitHub - [jupyterlab]( A user-friendly and powerful way to explore generative AI models in notebooks.  - [Arize-ai]( Phoenix provides MLOps insights at lightning speed with zero-config observability for model drift, performance, and data quality.  - [BrewLLM]( LLMOps tools to build, chain, evaluate and deploy prompts for GPT and other models.  - [getmetal]( Motorhead is a memory and information retrieval server for LLMs.  - [ajndkr]( Ship production-ready [LangChain]( projects with [FastAPI]( - [log10-io]( Python client library for managing your LLM data in one place.  - [bentoml]( Online Inference API for NLP Transformer models - summarization, text classification, sentiment analysis and more.  - [dpiras]( CosmoPower-JAX is an extension of the [CosmoPower]( framework to emulate cosmological power spectra in a differentiable way.  [Pledge your support]( Industry Insights AWS ML - [Amazon SageMaker with TensorBoard: An overview of a hosted TensorBoard experience:]( Amazon SageMaker with TensorBoard allows data scientists to visualize and debug deep learning models during the training process. This capability is integrated with [SageMaker]( training jobs and domains, providing users access to TensorBoard data and helping them perform model debugging tasks with visualization plugins. This post demonstrates how to set up a training job with TensorBoard in SageMaker using the SageMaker Python SDK, explore training output and delete unused applications.  - [Announcing new Jupyter contributions by AWS to democratize generative AI and scale ML workloads:]( AWS has released open-source features for data scientists and machine learning developers to enhance productivity and user experience in the Jupyter Notebook platform. [Jupyter AI]( an open-source project, brings generative AI functionalities, that help generate, debug, explain, answer questions on source code, and can even populate entire notebooks from natural language prompts. Jupyter AI offers magic commands and a chat UI in JupyterLab, and users can perform tasks with natural language prompts and insert AI-generated responses. AWS aims to improve the platform for the open-source community through this project. Jupyter AI is an official open-source initiative of Project Jupyter.  Google AI & ML - [Google Cloud advances generative AI at I/O: new foundation models, embeddings, and tuning tools in Vertex AI]( Google announced the release of [PaLM 2]( a new language model with improved multilingual, reasoning, and coding capabilities, alongside three new foundation models available in Vertex AI. These models can be accessed via API, Generative AI Studio, or deployed to a data science notebook. Google is transforming how organizations interact with AI in the cloud through its expanding toolset and the availability of these models for all industries and technical expertise levels. Some models are available to Google Cloud account holders in preview, while others are available through a [trusted tester program](. Google also highlights its commitment to taking a responsible approach to AI guided by its [AI Principles]( - [Introducing Duet AI for Google Cloud – an AI-powered collaborator:]( Google has launched Duet AI, an always-on AI collaborator that offers assistance to users with all skill levels on building secure, scalable applications. [Duet AI]( uses state-of-the-art generative-AI foundation models and is personalized and intent-driven, offering expert guidance. Google aims to build a cloud platform that is more human-centered, holistic, and helpful with responsible AI at the heart of the experience and believes that the future of developer productivity is more targeted personalized help.  Just for Laughs! What do you call a machine learning model that is always right?   A biased model!  Understanding Core Concepts Responsible AI Toolbox – By [Sina Fakhraee]( [Balamurugan Balakreshnan]( [Megan Masanz]( One of the biggest challenges we face in data science is understanding what the model does. For example, if the algorithms we use are all black boxes, it’s not that easy to know how the decisions are made. To discern how our algorithms make decisions, we can make use of responsible AI. This will give us the opportunity to explain the model’s decisions, find the features that contribute to the prediction, do error analysis on the dataset, and also ensure fairness in the dataset.  Microsoft recently developed a Responsible AI Toolbox that encompasses interpretability, fairness, counterfactual analysis, and causal decision-making through three dashboards: a fairness dashboard, an error analysis dashboard, and an interpretability dashboard.  The Toolbox is available here: [( Responsible AI dashboard  To leverage the Toolbox, we will create a model to apply the Toolbox to. Let’s look at the process of creating and analyzing responsible AI.  Let’s go through the steps to create a model in the Azure ML service:  - Go to the Azure ML Studio UI.  - Start the compute instance.  - Click on Notebook in the Author section.  - Create a new notebook with Python 3.8 with Azure ML as the kernel.  - Name the notebook RAIDashboard.ipynb.  - Now, we need to install or upgrade libraries. Only install libraries if they aren’t already on your system. At the time of writing, Python 3.8 with the Azure ML kernel was used, and it already has raiwidgets installed. [Read more here ...](   The above content is extracted from the newly released book, "[Azure Machine Learning Engineering]( authored by [Sina Fakhraee]( [Balamurugan Balakreshnan]( [Megan Masanz]( and published in Jan 2023. To get a glimpse of the book's contents, make sure to read the [free chapter provided here]( or if you want to unlock the full Packt digital library free for 7 days, [try signing up now]( To learn more, click on the button below.  [DISCOVER FRESH CONCEPTS & KEEP READING!]( Latest Developments in LLMs & GPTs - [Language models can explain neurons in language models:]( The article discusses an automated approach to interpretability research using GPT-4 to produce natural language explanations of [neuron behavior](. This approach aims to understand individual components' (neurons and attention heads) functionality in neural networks with tens or hundreds of billions of parameters. The proposed approach is part of the third pillar of the author's approach to [alignment research]( which scales with the pace of AI development.  - [ProgPrompt: Generating Situated Robot Task Plans Using Large Language Models:]( In task planning, large language models can be used to score potential actions or generate action sequences, but current methods have limitations. The authors here propose a programmatic LLM prompt structure that enables plan generation across different environments, robot capabilities, and tasks by prompting the LLM with program-like specifications of available actions and objects. The approach achieves state-of-the-art success rates in VirtualHome household tasks and is deployed on a physical robot arm for tabletop tasks.  - [FewSOL: A Dataset for Few-Shot Object Learning in Robotic Environments:]( The Few-Shot Object Learning (FewSOL) dataset is introduced for object recognition with a few images per object. It includes real-world objects captured with RGBD images, object segmentation masks, poses, and attributes. Synthetic images generated using 3D object models augment the dataset. The dataset is used to investigate few-shot object classification and joint object segmentation and few-shot classification with state-of-the-art methods for few-shot learning and meta-learning, revealing room for improvement in few-shot object recognition in robotic environments.  [Pledge your support]( Find Out What’s New - [Rust Polars: Unlocking High-Performance Data Analysis:]( [Polars]( is a popular open-source library in the data science and machine learning community that enables data manipulation, preparation, and analysis. It's not tied to a specific programming language and comes with various functionalities for manipulating complex datasets. The article introduces the fundamental data structure in Polars, Series, and explores querying and modifying Series. This knowledge will be essential when working with Series and learning about Polars' data frames and efficient data input/output operations.  - [Version Control Your ML Model Deployment with Git using Modelbit:]( Version control is crucial for data teams, especially when deploying models, as it enables teams to identify changes and diagnose issues that arise during deployment. It allows teams to work on the same codebase and improve models without interfering with each other's work, ensuring replicable and credible results. Git functionalities are important for data teams, and Modelbit can streamline model deployment with Git. However, the author suggests that Modelbit may be less intimidating and more streamlined than Heroku for both experienced and beginner data teams.  - [Exploratory Data Analysis with ChatGPT:]( This article demonstrates an example of exploratory data analysis (EDA) run by ChatGPT, covering various stages of EDA, outputs, and limitations. The analysis uses a dataset sample from [Common Crawl]( a vast collection of web [crawl data used]( to train language models like LLMs. The author hosts the [sample dataset on Kaggle]( and the analysis is run on [Google Colab](. The article also touches on the future of LLMs in analytics and their significance in training and analyzing large datasets.  - [CRUD with Pinecone:]( Vector databases have become a popular solution for handling large-scale, high-dimensional data due to their ability to perform efficient similarity searches and support complex data structures. [Pinecone]( is an increasingly popular vector database solution among developers and data scientists. Vector databases emphasize the importance of spatial relationships between data points, enabling high-performance, accurate search results in applications that demand quick identification of similar items within a dataset. The code for this article is [available here.]( - [Time Series Data Analysis with sARIMA and Dash:]( This article discusses the usefulness of sARIMA models in time-series data analysis and presents a web application built using Python and Plotly Dash to help users analyze their data and fit an optimal sARIMA model for predictions. The [live app]( and [GitHub repository]( are also shared for easy access.  See you next time! As a GDPR-compliant company, we want you to know why you’re getting this email. The _datapro team, as a part of Packt Publishing, believes that you have a legitimate interest in our newsletter and its products. Our research shows that you opted-in for email communication with Packt Publishing in the past and we think your previous interest warrants our appropriate communication. If you do not feel that you should have received this or are no longer interested in _datapro, you can opt out of our emails by clicking the link below.   [Like]( [Comment]( [Restack](   © 2023 Copyright © 2022 Packt Publishing, All rights reserved. Our mailing address is:, Packt Publishing Livery Place, 35 Livery Street, Birmingham, West Midlands B3 2PB United Kingdom [Unsubscribe]() [Start writing]()