Ingest and Retrieve data from an ADX Cluster Using Python (2024-05-06)

SQLServerCentral Newsletter for May 6, 2024 Problems displaying this newsletter? [View online](. [SQL Server Central]( Featured Contents - [Ingest and Retrieve data from an ADX Cluster Using Python]( - [Snake draft sorting in SQL Server, part 2]( - [From the SQL Server Central Blogs - Cloud Databases Made Simple: Creating Your First AWS RDS Instance]( - [From the SQL Server Central Blogs - Tired of Tedious SQL Queries? AI Tools Can Automate The Boring Parts]( - [Deciphering Data Architectures]( Question of the Day - [The Backup File Extension]( The Voice of the DBA  The DBAccountant One of the things that I've found with cloud computing services is that the people involved with managing these resources sometimes get asked to become financial accountants. I saw an interesting post from SQL Rod asking about this new task as something all of us might need to consider a part of our job in the modern world. He asks if we are [Techouncants or Accountechs?]( This isn't in the sense of being the Financial DBA trying to manage costs in the cloud, though that is part of his post. It's more about making smart financial decisions. Certainly, as more workloads move to the cloud, and they are for many of us, we likely need to keep an eye on costs, usage, and tuning. However, there are other choices. When I had to spec servers, we wanted to get something with room to grow, but not the top-of-the-line most expensive option. I wanted something engineered well for the next few years, and certainly not under-engineered. Usually because adding resources was hard in those days. Virtualization has made things easier, but often there are still some limits. Still, the goal is matching resources to workload with room to burst if needed. At the same time, are there other choices we can make? Rod asks about choosing the appropriate edition, the HA/DR strategy, tooling, and more. Can we be efficient with our use of money, time, and results, trying to neither overengineer nor under-engineer? The whole post has a lot of "it depends" in it, as the decisions on what you should choose are something you have to think about and arrive at a decision that balances costs with other factors. It can be clearer often to choose one thing over another, but it's not always crystal clear which choice is better. I like his hardware analogy in the post. I'm not a fan of just throwing hardware at a situation, but in some cases, that's the best solution. The cloud makes it easier to see where costs are. As a Redgate customer put it recently, "tuning queries becomes more important in the cloud as we can see exactly what each one costs." That can be true for your SQL Server VMs, but it's often not seen as important when the costs are a one-time spend for your hardware. However, Jeff Moden might argue that you should be tuning queries even without chargebacks or a cloud bill. It will save time for your users, which is always valuable. I agree, since I think we have no shortage of work and time is our most valuable resource. Save it when you can for the most people. That often means a big part of being a DBAccountant is being able to show, lead, or somehow get others to write better code early. Ensure everyone (or automated tests) knows how to view execution plans, learns to use them, and tries to write more efficient code the first time. Steve Jones - SSC Editor [Join the debate, and respond to today's editorial on the forums](   Featured Contents [SQLServerCentral Article]( [Ingest and Retrieve data from an ADX Cluster Using Python]( Sucharita Das from SQLServerCentral Learn how to query data in Azure Data Explorer using Python. [External Article]( [Snake draft sorting in SQL Server, part 2]( Additional Articles from SimpleTalk In my previous post, I showed how to borrow a snake draft concept from fantasy football, or a packing technique from the shipping industry, to distribute different portions of a workload to run in parallel. From the SQL Server Central Blogs - [Cloud Databases Made Simple: Creating Your First AWS RDS Instance]( epivaral from SQL Guatemala I recently started full time learning of Amazon Web Services (AWS). I found that AWS's Relational Database Service (RDS) is really user-friendly. RDS simplifies many of the complicated aspects of... [Blog Post]( From the SQL Server Central Blogs - [Tired of Tedious SQL Queries? AI Tools Can Automate The Boring Parts]( Tracy McKibben from RealSQLGuy - Helping You To Become A SQL Hero If you've ever wrestled with complex SQL code, you know the drill. Hours spent on a single query, eyes glazing over at the endless lines of commands, that sinking... [Deciphering Data Architectures]( [Deciphering Data Architectures]( Additional Articles from SQLServerCentral Data fabric, data lakehouse, and data mesh have recently appeared as viable alternatives to the modern data warehouse. These new architectures have solid benefits, but they're also surrounded by a lot of hyperbole and confusion. This practical book provides a guided tour of these architectures to help data professionals understand the pros and cons of each.   Question of the Day Today's question (by Steve Jones - SSC Editor):  The Backup File Extension I run this command in SQL Server 2022. What is the extension of the backup file? BACKUP DATABASE HerdofTwo TO DISK = 'HerdOfTwo_20240501' Think you know the answer? [Click here]( and find out if you are right.    Yesterday's Question of the Day (by Steve Jones - SSC Editor) Two Table Hints What happens when I run this code: SELECT p.ProductName , p.ProductCategory FROM dbo.Product AS p WITH (NOLOCK, TABLOCK); Answer: This returns an error about conflicting hints. Explanation: This returns an error, as you can only have one locking hint in a query. If the hints conflict, you get this error: Msg 1047, Level 15, State 1, Line 4 Conflicting locking hints specified. Ref: Table Hints - [ [Discuss this question and answer on the forums](    Database Pros Who Need Your Help Here's a few of the new posts today on the forums. To see more, [visit the forums](. --------------------------------------------------------------- SQL Server 2017 - Administration [Data Archival: 1 table (Database A) to 1 table (Database B)]( - I'm currently in the process of building an archival & reporting solution and just want to run through a bunch of scenarios and determine what would be best in my situation. I can easily achieve this multiple ways but I don't know if this is the "right way" or the "best" method My objective is […] [Principal owns a service?]( - This is one I haven't seen before. I'm trying to drop a database user and got the below message. The login doesn't own any roles or schemas or other objects and everything I google on the error comes up with "principal owns a schema" which is not what this error is. When I try googling […] [Degraded Performance in 2017 vs 2008 R2 with inserts]( - We're finally upgrading from SQL Server 2008 R2 Enterprise to SQL Server 2017 Standard and have moved our test database to the new server. Both VMs are on similar hardware with 4 vCPUs. Of course, given the chasm of versions we expected much better performance. Unfortunately, that's not the case. Our inserts on the 2008 […] SQL Server 2016 - Development and T-SQL [SQL Server Semantic Search (Find Text in External Files)]( - Dear community, I am trying to search word documents (docx) for a particular drug name and return the file name of a successful match. I am in the testing phase and with the assistance of Daniel Calbimonte article (link below) I have a better understanding on how to achieve this. The issue I am faced […] [Slow update on temp table]( - I have a job running each night starting at 4:30AM and usually finishing at around 6:30AM or shortly thereafter doing some number crunching. Occasionally, this job slows down taking much longer...today it ran until about 8:45AM. This statement, normally would take a second or two...the table is about 250,000 rows with 8 columns...today it took […] SQL Server 2019 - Administration [Backing up and Restoring a 3 terabyte database takes over 5 hours]( - I have a 3 terabyte database that is taking over 5 hours to backup and restore. I'm utilizing multiple files to do the backup. We are on an Azure VM and backing up to Azure Storage. Just wondering if there is anything else I can do to speed up the backup restore of this database. […] [Table Partitioning - remove null range]( - hi guys,  I have a partitioned table AM_ITM_INSTNC_MSS and I want to get rid of MYDW_TPData2015 filegroup, how can I achieve that? [Backup of encrypted databases failing]( - I've had some backups of my encrypted databases failing with the error "BACKUP 'DBName' detected an error on page (10:12345) in file 'D:\path\DBName_File.ndf' The databases were originally encrypted on a production server and restored to a DEV environment, which may be relevant. All the databases on the DEV server are encrypted, but only two were […] SQL Server 2019 - Development [Create Copy of DB in same server.]( - Hi I want to create a copy of a DB on the same server. The account that creates the copy DB should NOT be able to read from the original DB but have full rights to the copy DB. Cannot use a snapshot, Changes will be made to the copy and the copy will be […] Reporting Services [SSRS Expression IIF Zero then ... Got #Error]( - Hello, Need some help for SSRS. The column % expression is =iif(sum(Fields!ColumnA.Value) = 0 or sum(Fields!ColumnB.Value) = 0,0,sum(Fields!ColumnB.Value) / sum(Fields!ColumnA.Value)) But I am getting #Error if ColumnA is zerro and no error if ColumnA and B had no value. Please advise. Thank you.   Analysis Services [Issues backing up Analysis Services databases - silent failures]( - I have a PowerShell script that I use to automate the backup of Analysis Services databases - allows for handling new/dropped databases, dynamic filenames, etc. However, on some occasions the backup fails but no output is reported stating why. I get a success message when it works but nothing when it fails. I've tried adding […] Strategies and Ideas [Do I need a recursive query for this? bill of materials (sort of)]( - I am working on a data warehouse project for a bakery. For the most part, the basic structure is like this: Purchase--(M,1)--Ingredient--(1,M)--Recipe--(M,1)--Product with one exception the "Recipe" part works fine. (IngredientKey, ProductKey, Weight). The part I'm having trouble with is that one ingredient (starter) consists of other ingredients (kinda like a subassembly, if you wanted […] [Do I need a recursive query for this? bill of materials (sort of)]( - I am working on a data warehouse project for a bakery. For the most part, the basic structure is like this: Purchase--(M,1)--Ingredient--(1,M)--Recipe--(M,1)--Product with one exception the "Recipe" part works fine. (IngredientKey, ProductKey, Weight). The part I'm having trouble with is that one ingredient (starter) consists of other ingredients (kinda like a subassembly, if you wanted […] Integration Services [Hardcoded filter vs filter by parameter]( - I am trying to set a date filter on several large transaction tables to limit the years loaded. When I use a hardcoded filter like: DATEFROMPARTS ( DATEPART(yyyy, GETDATE()) - 4, 1, 1 ) to get running year and previous 4 years the dataflow starts reading data immediately and data is loaded in 10 seconds. […] Hardware [Disk (SCSI) Controllers - Parallel Disk I/O]( - Hey For SQL Server VMs I use the max number of SCSI controllers supported by the relevant hypervisor. But for the first time in a loooong time I am looking at a physical implementation. The most logical thing I can think of is to have multiple disk controllers and place each SQL disk on a […]   [RSS Feed]( This email has been sent to {EMAIL}. To be removed from this list, please click [here](. If you have any problems leaving the list, please contact the webmaster@sqlservercentral.com. This newsletter was sent to you because you signed up at SQLServerCentral.com. ©2019 Redgate Software Ltd, Newnham House, Cambridge Business Park, Cambridge, CB4 0WZ, United Kingdom. All rights reserved. webmaster@sqlservercentral.com  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -