How to Delete Large Amounts of Data (2024-08-30)

SQLServerCentral Newsletter for August 30, 2024 Problems displaying this newsletter? [View online](. [SQL Server Central]( Featured Contents - [How to Delete Large Amounts of Data]( - [Microsoft Fabric: Using Lakehouse data in Real Time Dashboards]( - [Is it still worth it to sponsor PASS Summit?]( - [From the SQL Server Central Blogs - Python: How to use secrets stored in azure key vault in your Pulumi project]( - [The Definitive Guide to Azure Data Engineering: Modern ELT, DevOps, and Analytics on the Azure Cloud Platform]( Question of the Day - [Synapse Analytics History III]( The Voice of the DBA  Trying New Technology I had someone ask me about [DuckDB](. Would I think that's a good choice for a database? I don't really know. From their blog and some online research, maybe, but it's also a minority player in a niche space. I had a chat recently with someone that had implemented [ArangoDB]( a graph database. Why that and not Neo4J I asked them? Someone at the company had tried the database and recommended it. Not a bad reason, as I think experience with tech is important, but it's not the most important thing. As I've aged, and maybe matured, I think less about the ability of a technology to work and more about the ability of a technology to be maintained over time. Not by me, but by everyone in my organization. Not everyone, but can anyone working on our staff learn and use it, including the future employees we haven't yet hired. There seem to be no shortage of new niche technologies. I have a few newsletters I subscribe to, and I see new projects and new solutions appearing every day. New tools, utilities, frameworks, even databases. Some of these might be amazing, and incredibly useful, but will they exist in a few years? In fact, that's a question I ask myself about plenty of Microsoft technologies that appear. Will they really be around in 5 years? Long-term, or at least medium-term, supportability is important. I also worry about the training and learning required for new technology. I've seen companies that adopt too many products in their tech stack and it becomes hard to hire experienced people. Even if we hire smart people that can learn, we have a lot to teach them. The more we need to teach, the slower they are to be productive. It can be even slower for us to trust them to work independently, especially in a crisis. I think that most organizations should limit the number of technologies they use. This could be frameworks, languages, and more, including databases. Don't add something new just because a developer, DBA, or even executive likes it. Certainly, be careful about changing technologies when the change isn't adding value to your organization. Every change has costs, every new advantage contains a disadvantage, and every additional thing creates training requirements. Some people might pick things up quickly, easily, and during their off hours. That person might be you, but how many others will be able to do that? Not many. That's been my experience. The world is full of average people, by definition. While the average level (skill, capability experience, etc.) at your organization might be higher than our industry, over time, that will change. As our organizations grow, and as we change staff, we often become more average. Our choices, and methodologies, our architecture, and more must survive the average employee, not the high performing ones. Every organization ought to limit tech choices. There ought to be a process and way to add new technologies, and employees ought to be able to submit a request, make a case, and have others decide if taking on a new technology makes sense. If so, great, but do so carefully. I like seeing new technologies built and adopted, but I also try not to just adopt the latest shiny things. Experiment, in a time-boxed fashion, and make decisions when appropriate, consciously because the benefits outweigh the costs. And not just slightly outweigh the costs, but substantially. In all likelihood whoever proposes the new tech isn't thinking about the downside, and there will always be more downsides than you can see right now. Steve Jones - SSC Editor [Join the debate, and respond to today's editorial on the forums](   Featured Contents [SQLServerCentral Article]( [How to Delete Large Amounts of Data]( Jim Youmans-439383 from SQLServerCentral As a DBA, we are often tasked with maintenance work on the database. Often this includes making sure that old data is purged from the tables. This is especially true with logging tables. Sometimes multiple applications write to the same logging tables, and they can grow very quickly. Let’s say we have a logging table […] [External Article]( [Microsoft Fabric: Using Lakehouse data in Real Time Dashboards]( Additional Articles from SimpleTalk Real-Time dashboards are a great feature in Real Time Intelligence experience to monitor our data. However, by default ... [Technical Article]( [Is it still worth it to sponsor PASS Summit?]( Additional Articles from SQLServerCentral Conference planning is underway. Several folks have asked me whether I think the PASS conference is still worth exhibiting. I thought an article would be best to break down my thoughts. I’m in an interesting position, being a marketing leader who also specializes in SQL Server and doesn’t work for Microsoft. I’ve worked directly for several companies in the ecosystem, so I know how difficult reaching the DBA market has been post-pandemic. [Blog Post]( From the SQL Server Central Blogs - [Python: How to use secrets stored in azure key vault in your Pulumi project]( Adetokunbo Ige from Adetokunbo Ige Software Engineers are excited to create Infrastructures just like DevOps Engineers. However, other IaC tools like Terraform and CloudFormation, relies on domain-specific Languages (DSLs) or JSON/YAML templates to create... [The Definitive Guide to Azure Data Engineering: Modern ELT, DevOps, and Analytics on the Azure Cloud Platform]( [The Definitive Guide to Azure Data Engineering: Modern ELT, DevOps, and Analytics on the Azure Cloud Platform]( Site Owners from SQLServerCentral Build efficient and scalable batch and real-time data ingestion pipelines, DevOps continuous integration and deployment pipelines, and advanced analytics solutions on the Azure Data Platform. This book teaches you to design and implement robust data engineering solutions using Data Factory, Databricks, Synapse Analytics, Snowflake, Azure SQL database, Stream Analytics, Cosmos database, and Data Lake Storage Gen2.   Question of the Day Today's question (by Steve Jones - SSC Editor):  Synapse Analytics History III What was the originally name of the appliance that eventually became Azure Synapse Analyics? Think you know the answer? [Click here]( and find out if you are right.    Yesterday's Question of the Day (by Steve Jones - SSC Editor) Converting Integers The max integer value is 2,147,483,647. What is returned from this code? SELECT 2147483649 / 2 AS Result2; Answer: 1073741824.500000 Explanation: If there are integer constants greater than the max (2,147,483,647), they are convered to the decimal data type. Ref: Convert integer data - [ [Discuss this question and answer on the forums](    Database Pros Who Need Your Help Here's a few of the new posts today on the forums. To see more, [visit the forums](. --------------------------------------------------------------- SQL Server 2017 - Administration [Does a Query Lock to the Table_Name or Table_ID]( - SQL 2017 ENT SCENARIO: Table_1 has a Sch-M lock on during an intended updating process. A QUERY is executed against Table_1 and is queued up, waiting for the Sch-M lock to release. While the QUERY is queued up on Table_1, Table_1 is renamed to Table_1_OLD A newly designed Table_2 is then renamed to Table_1 QUESTION: […] SQL Server 2016 - Development and T-SQL [Help with to generate Data pattern Sql help]( - Need to generate a from and to From to 50 -99 100 -299 300 -- I tried to do the logic ,but its working as 1-50 51-100 101 -  FinalCTE AS ( Select ,FROMQUANTITY AS S_FROMQUANTITY ,ROW_NUMBER() OVER(PARTITION BY C.po_line_id ORDER BY C.FROMQUANTITY ASC) AS A ,CASE WHEN CAST(C.FROMQUANTITY AS NUMERIC(18,7)) >= 1.0 THEN […] [Losing Saved Connections in SSMS 2016]( - Microsoft SQL Server Management Studio 13.0.16106.4 Hi , I am used to having saved connections in the Connect to Server popup. For some reason my SQL Server management studio 2016 is not saving them anymore. Please help me get saved connections back or get the connections to save again.  SQL Server 2019 - Administration [SQL Server Availability Group Database Backup Missing Data]( - I have a production SQL2019 Enterprise SQL Server which hosts an Availability Group with 12 small to medium databases in it. This is an OLTP server and there is no database over 200gb. The AG only has 2 nodes and is used for H/A when patching. Both nodes are onPrem so the Availability Mode is […] SQL Server 2019 - Development [ssis dashboard suddenly missing from reports option under catalog/ssisdb]( - hi i rt clicked ssisdb today under integration services catalog , chose reports and didnt see the dashboard option under my regular id. my id has sa. so i connected with my admin which also has sa and did see it. losing permissions would have occurred in the last 24 hrs. i saw in one […] [checking on some restrictions related to functions]( - hi i'm helping a friend on his creation of a function.  here are the issues... he's getting an error setting nocount on. i told him that is only a proc thing as far as i know.  is that correct? 2. he's getting an error setting xact abort on.  i was a little surprised by this. […] SQL Azure - Development [Hybrid Cloud Server Integration with SQLServerCentral’s SQL Server 2022]( - Hello everyone, I’m currently exploring the use of hybrid cloud servers and am focusing on integrating them with SQLServerCentral’s SQL Server 2022. While hybrid cloud environments offer flexibility, managing and optimizing them with SQL Server 2022 presents several challenges. Here are some specific areas where I’m seeking advice: Integration Best Practices: What are the best […] Powershell [dump to csv -- image column]( - If there is an Image Column in mssql table and I wanted to use export-csv to dump this info to file how do you handle that column to be able to import that csv to another DB. Thanks. SQL Server 2022 - Administration [Alias woes on SQL Server 2022]( - Our company is moving to the cloud. I have SQL Developer Edition 2022 on Windows Server 2022 Datacenter setting up our QA SharePoint Farm.. I've created a named instance running over port 55055 using TCP/IP. THis is what I have in my on premises SQL server that we set up about 10 years ago. The […] [Db level user is not having execute permission no db_owner]( - Dear all, we have a strange situation Db level [ user ]is not having execute permission also not having db_owner , having ddl read and write but in audit logs it is showing the same user had action_id/EX permission in audit logs how can this is possible. [SQL Server 2022 Memory Consumption]( - I have migrated to SQL Server 2022 from SQL Server 2014 and after migration I have observed that it is consuming more memory than before. Memory is going almost 80-90%. We have same configuration, there is not memory caping on SQL Server 2022 and there was no caping on SQL Server 2014. We have same […] SQL Server 2022 - Development [SSIS package project connections don't work on upgrade]( - We're upgrading from SQL Server 2016 to 2022. The SSIS package project level connections don't work as expected. One queries the server as an anonymous logon instead of the domain account associated with the SQL Server Agent. One even affects both the original server and the new one to which it is pointed. I have […] [2022 azure vm cant run same ssis pkg we run on prem]( - Hi, we set up an environment on our new azure 2022 vm with a cloned "prod target warehouse", ssis project clone from prod, cloned sql agent etc etc. amongst other things the ssis points at a new erp's sales data we are testing with in just one agent step/ssis project execution for that 2019 std […] [ssis 32 bit vs 64, debug and non debug]( - Hi i am looking in vs2022 at my imported 2022 dev azure vm based ssis project. And vs2022 at my 2019 non azure non vm (i think) based production ssis project. The idea is to compare their bittages after i got an error on the dev box basically saying the 64 bit connector wasnt installed […] [Strange(?) performance problems]( - I am debugging an NHibernate generated SQL_query for performance problem and came up with something strange. The query is a SELECT statement that reads from a view. I've tracked the problem to this part of the query: AND (myView.A LIKE (@p7+'%') OR myView.B=@p8 OR myView.C=@p9) If I run the query with only condition at a […]   [RSS Feed]( This email has been sent to {EMAIL}. To be removed from this list, please click [here](. If you have any problems leaving the list, please contact the webmaster@sqlservercentral.com. This newsletter was sent to you because you signed up at SQLServerCentral.com. ©2019 Redgate Software Ltd, Newnham House, Cambridge Business Park, Cambridge, CB4 0WZ, United Kingdom. All rights reserved. webmaster@sqlservercentral.com  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -