Using Machine Learning Pipeline Execution Activity in Azure Data Factory (2024-05-13)

SQLServerCentral Newsletter for May 13, 2024 Problems displaying this newsletter? [View online](. [SQL Server Central]( Featured Contents - [Using Machine Learning Pipeline Execution Activity in Azure Data Factory]( - [Read Data Stored in a Lake Database using Azure Synapse Analytics]( - [From the SQL Server Central Blogs - Use a slicer to filter a visual based upon a measure in Power BI]( - [From the SQL Server Central Blogs - Friday Basics: the CIA Triad]( - [T-SQL Fundamentals]( Question of the Day - [The Query Traceflag]( The Voice of the DBA  Crazy Interviews In my career, I've had a lot of interviews. There were times when I wanted to change positions, and I'd interview with many companies. I thought interviews were good practice for finding jobs, so I would schedule some even if I wasn't that interested in the job. It gave me practice answering questions, and a few times I was even surprised and became interested in the position. These days there are a lot of people looking for jobs. Many tech companies laid off a lot of staff in 2023 and are still letting some go in 2024. There are also plenty of non-tech companies who have downsized their staffs, for a variety of reasons. As you search for a job, you might find yourself challenged in new ways during an interview. I ran across a story recently on some of the [nightmare processes companies are putting candidates through](. I think this is mostly high-tech companies, but a lot of interview silliness filters down to many other companies over time. I've had interviews where I was challenged by a group to answer questions. I've had day-long interviews with a variety of tech and business people. I've taken tests before a conversation with others, and I've even been asked to whiteboard my way through a problem presented to me. In all those sessions, I've enjoyed the challenge and found it a good way to learn about how the team thinks and how I would interact with them. I can often ask questions to help me learn about the company, position, and staff. I haven't been asked to spend days building something, nor had assignments to take home or have double-digit numbers of interviews. I get that there are a lot of people that embellish their resumes, and probably a lot of candidates that may not have contributed a lot to their former team, but asking someone to invest hours and hours on creating something just for a chance at a job seems like overkill. There isn't a good way to evaluate someone in a short time spent with them. I think interviews give you an idea of base knowledge, but not how someone might work in a team, how they collaborate, or even how they might respond under pressure. While an interview creates pressure, working under a deadline is a different type of pressure. I wish a coding test existed to evaluate someone's skills, but academic quiz questions are different than working with a specification from a client. I think some of the ways people interview others aren't well thought through and don't help us hire better. Recommendations from people you know are the best way to find candidates, but you might not have those when you have an opening. I do think that presenting some real situations from your job, and having people collaborate with you to solve problems is the best way to go about things. Even better if you can actually work with them on an instance or in an IDE, but that is hard to set up. I don't know how I'd feel about a job that gave me 10 or more hours of homework just to get to the interview. I hope I never find out. Steve Jones - SSC Editor [Join the debate, and respond to today's editorial on the forums](   Featured Contents [SQLServerCentral Article]( [Using Machine Learning Pipeline Execution Activity in Azure Data Factory]( Sucharita Das from SQLServerCentral This article shows how one can run a machine learning activity in an Azure Data Factory (ADF) pipeline. [External Article]( [Read Data Stored in a Lake Database using Azure Synapse Analytics]( Additional Articles from MSSQLTips.com In this article, we look at how to read data stored in a lake database using Azure Synapse Analytics to help reduce overall Azure costs. [Blog Post]( From the SQL Server Central Blogs - [Use a slicer to filter a visual based upon a measure in Power BI]( Meagan Longoria from Data Savvy Have you ever wanted to filter a visual by selecting a range of values for a measure? You may have found that you cannot populate a slicer with a... [Blog Post]( From the SQL Server Central Blogs - [Friday Basics: the CIA Triad]( K. Brian Kelley from Databases – Infrastructure – Security In information security (INFOSEC), there several foundational concepts and principles. One of the ones that’s introduced almost immediately is called the CIA triad or the Information Security Triad. While... [T-SQL Fundamentals]( [T-SQL Fundamentals]( Additional Articles from SQLServerCentral Master Transact-SQL's fundamentals, and write correct, robust code for querying and modifying data with modern Microsoft data technologies, including SQL Server 2022, Azure SQL Database, and Azure SQL Managed Instance.   Question of the Day Today's question (by Steve Jones - SSC Editor):  The Query Traceflag  Think you know the answer? [Click here]( and find out if you are right.    Yesterday's Question of the Day (by Steve Jones - SSC Editor) Finding Marks I have marked a few transactions in my code. How can I find out which marks were stored in a transaction log? Answer: These are stored in the dbo.logmarkhistory table in the msdb database Explanation: Marks are stored in the dbo.logmarkhistory table in the msdb database. Ref: logmarkhistory - [ [Discuss this question and answer on the forums](    Database Pros Who Need Your Help Here's a few of the new posts today on the forums. To see more, [visit the forums](. --------------------------------------------------------------- SQL Server 2016 - Administration [What does it mean when Query Store Tracked Queries shows the same Plan Id twice?]( - I'm investigating a query which sometimes performs badly and, when I locate one of the query instances in Query Store and look at it with Tracked Queries, it shows the same Plan Id twice. What does this mean? Obviously, I can't force a plan, since it's the same plan it's already using! SQL Server 2019 - Administration [can I monitor auto stats updates through extended events?]( - We have pretty big tables very sensitive to stats with many updates. Our systems are complex with many places requiring explicit statistics updates and we want to monitor this to investigate potential SAMPLE updates overwriting FULLSCAN updates. We managed to catch all such updates through extended events by filtering on sp_statement_starting and sql_statement_starting but we […] [Access Issues]( - We have a windows account that currently has access to half of the sql server machines.(Both RDP and SQLAccess) The rest of the servers we do not have both RDP and SQLAccess. Does this mean when our accounts were created in AD we were not assigned to all relevant security groups which allows us to […] SQL Server 2019 - Development [Blocking question]( - Hi all, We have a SSIS package that does 2 things in parallel: moves data from one partition to another within the same table. First, it truncates destination partition, then selects from source partition and inserts to destination one. But my select process is blocking truncate process, and truncate in turn is blocking insert to […] [a function that acts like both a scalar and table valued function]( - Hi, i hope i'm doing this question justice. A peer of mine likes the idea of using user defined table types. I think mostly because he doesnt want to pass 84 data values representing his "row" to either procs or functions 2 or 3 levels deep. He also likes the idea of inserting into tables […] [How to check for data loss after rollback]( - Hello SSC, I am not a DBA,. I have some DBA knowledge, but my education is all programming. I had a production deadlock that took over 30 hours to rollback. I am concerned about data loss, but I have no idea how to troubleshoot this. I have a bunch of logs from the DBA's, so […] Amazon AWS and other cloud vendors [AWS RDS for SQL Server - Backup to/Restore from s3 - Multiple Account Scenario]( - Hi All, I need a sanity check, and hoping someone here can provide it.... Environment: Prod Instance - AWS ec2 instance running SQL Server (2022) in "Prod Account" s3 Bucket in "Backups Account" Dev Instance - AWS ec2 instance running SQL Server (2022) in "Dev Account" Use Case: Need to be able to execute weekly […] Reporting Services [Report definition has an invalid target namespace]( - I'm hoping someone here has encountered this before and was able to fix it. I've got a client with SQL Server 2014, they also use SSRS and asked me to create a new report to get automatically run every week. I created the report on my computer using Report Designer but when they tried to […] General [How to create a baseline using counters to analyze server performance?]( - I am currently working on creating a baseline for our server, using various counters to analyze performance. The server exclusively hosts SQL Server. Could anyone here share their experiences or methodologies on how to create such a baseline and use it for performance analysis? Powershell [Use of variable issue]( - All, I'm trying to set up some code in a SQL agent job. The following code works as expected: $sqlConn = New-Object System.Data.SqlClient.SqlConnection $sqlConn.ConnectionString = "" $sqlcommand = $sqlConn.CreateCommand() $sqlcommand.CommandText="Select filename from dbo.ssisfiles" $DataAdapter = new-object System.Data.SqlClient.SqlDataAdapter $SqlCommand $dataset = new-object System.Data.Dataset $DataAdapter.Fill($dataset) foreach ($Row in $dataset.Tables[0].Rows) { if (Test-Path -Path "c:\as_demo.txt") { Throw "File […] Integration Services [Query about GAC (Global Assembly Cache)]( - Hi everyone, So I have a SSIS package that performs some importing/exporting from MySQL database. This SSIS package uses code written in Script Task C#. To make connectivity to MySQL server, I have used MySql.Data Nuget package. When I installed this Nuget package I had to manually add "MySql.Data.dll" file into GAC otherwise there were […] SQL Server 2022 - Administration [Contained AGs user sync]( - Hello everybody, I've read that contained AGs create copys of the system dbs which are synced between the replicas. As far as I know during the setup process of the contained AG the existing sysadmins are added to the contained AG master copy. But I havn't found any info about created sysadmins after the contained […] SQL Server 2022 - Development [SSIS upgrade]( - I have about 40 packages that are dtsx files that were created with SQL 2016. It appears that I need to either install SNAC (which is decprecated / not recommended) or change them all to sql ole db. Is there an easy way to change to sql ole db?   [How best to identify one (and only one) default row in a table?]( - We have a table with a numbers of rows in it. (Not that many rows, but it's the concept I'm interested in.) One of these, and only one, can be selected by a user as the default row. The obvious way to support this requirement is to add an IsDefault bit, not null, with a […] [More fun with Recursive CTEs]( - Instead of dealing with courses this time, I'm trying to sort out how to model a recipe problem. (It's bread, so it's not crazy complex.) Oh wait, here... have some table definitions: USE [Bakery] GO /* Sorry, this started in Abscess */ CREATE TABLE [dbo].[Ingredient]( [IngredientID] [int] IDENTITY(1,1) NOT NULL, [IngredientName] [varchar](255) NOT NULL, [UnitPrice] […]   [RSS Feed]( This email has been sent to {EMAIL}. To be removed from this list, please click [here](. If you have any problems leaving the list, please contact the webmaster@sqlservercentral.com. This newsletter was sent to you because you signed up at SQLServerCentral.com. ©2019 Redgate Software Ltd, Newnham House, Cambridge Business Park, Cambridge, CB4 0WZ, United Kingdom. All rights reserved. webmaster@sqlservercentral.com  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -