Full vs. Incremental Loads – Data Engineering with Fabric (2024-04-17)

SQLServerCentral Newsletter for April 17, 2024 Problems displaying this newsletter? [View online](. [SQL Server Central]( Featured Contents - [Full vs. Incremental Loads – Data Engineering with Fabric]( - [Get the most out of SQL Server Agent logs]( - [From the SQL Server Central Blogs - SQL Quickstart Guide by Walter Shields – Book Review & Summary]( - [From the SQL Server Central Blogs - RTO and RPO are myths unless you’ve tested recovery]( - [T-SQL Fundamentals]( Question of the Day - [Restoring the Resource Database]( The Voice of the DBA  Missing the Office Recently I traveled to visit a customer who has an in-the-office culture. They have multiple large buildings outside a major US city and almost all their employees (7000+) live nearby and are expected to be in the office the whole week. More senior people can opt for 4 10-hour shifts rather than 5 8-hour shifts, but with few exceptions, they have people in the office. I hadn't seen that in a long time. Almost every customer is mostly remote or some level of hybrid (usually 2-3 days a week in the office). What's more, they have an open culture, with rows of desks for teams and spaces between the rows for managers and directors. No cubes! Everyone below a senior VP/C-level is at a desk. There are conference rooms and smaller quiet pods, but mostly everyone works in wide open spaces. From developers to sales to finance to human resources. It's both chaotic and loud, but also refreshing. I also loved seeing so many people wearing company logos on t-shirts, jackets, hats, and more. The environment reminds me of my time at JD Edwards. We had cubicles, but mostly open ones with half walls to the side and one full wall at the back where two cubes met. However, it was a very strong culture of comradery and closeness that I hadn't seen at any large company before or since: until now. Teams might be close in some places I've worked, but the entire company at JD Edwards felt like a family of people who were working together. We didn't always get along and argued, but we were all united. It was a great feeling. I miss that. Redgate is a great company, and I enjoy my co-workers, but I miss going to an office and seeing everyone inside it. Most people come to the office once or twice a week, but I miss my visits when most everyone was there every day. I've worked remotely for over twenty years, but I do still miss the office some days. Certainly my visits to the various Redgate offices, while good, have me wishing I saw more people there every day. I'm heading there tomorrow, and I hope I see lots of people. I don't know that mandating everyone in the office is a better choice, but I do know there are good things about gathering groups of people together. Strong leadership, empathy, a clear vision, and a balanced workload are important for any organization, but especially important if you want everyone in the office these days. This customer did a great job with it, and I think they would still have a strong culture with remote work, but their choice works, and I'm a little jealous. Steve Jones - SSC Editor [Join the debate, and respond to today's editorial on the forums](   Featured Contents [SQLServerCentral Article]( [Full vs. Incremental Loads – Data Engineering with Fabric]( John Miner from SQLServerCentral Learn how to perform full and incremental loads in Fabric with a little SparkSQL. [External Article]( [Get the most out of SQL Server Agent logs]( Additional Articles from SimpleTalk If you haven’t migrated your workloads to a managed database platform yet, you’re probably still relying on SQL Server Agent for various maintenance and other scheduled tasks. Most of the time, these processes just work. But when it’s time to troubleshoot, it can be cumbersome to get to the root of some problems. [Blog Post]( From the SQL Server Central Blogs - [SQL Quickstart Guide by Walter Shields – Book Review & Summary]( Tracy McKibben from RealSQLGuy - Helping You To Become A SQL Hero In today’s world, data is everywhere. Companies collect it, governments analyze it, and it influences everything from what products we buy to how we vote. But all that data... From the SQL Server Central Blogs - [RTO and RPO are myths unless you’ve tested recovery]( K. Brian Kelley from Databases – Infrastructure – Security I’ve watched teams spend a lot of time on backup strategy. They plan out the full, differential, and log backups to ensure they can successfully meet the recovery point... [T-SQL Fundamentals]( [T-SQL Fundamentals]( Additional Articles from SQLServerCentral Master Transact-SQL's fundamentals, and write correct, robust code for querying and modifying data with modern Microsoft data technologies, including SQL Server 2022, Azure SQL Database, and Azure SQL Managed Instance.   Question of the Day Today's question (by Steve Jones - SSC Editor):  Restoring the Resource Database How can I restore a version of the Resource Database in SQL Server 2022? Think you know the answer? [Click here]( and find out if you are right.    Yesterday's Question of the Day (by Steve Jones - SSC Editor) Express In-Memory OLTP Can I create In-Memory OLTP tables in SQL Server Express? Answer: In SQL Server 2016 and later editions. Explanation: Express includes the ability to create In-Memory OLTP tables. It's possible this is in 2014, but that documentation is retired. Ref: Edition Scale limits - [ [Discuss this question and answer on the forums](    Database Pros Who Need Your Help Here's a few of the new posts today on the forums. To see more, [visit the forums](. --------------------------------------------------------------- SQL Server 2016 - Administration [Compare instance performance before and after a change]( - I'm about to make some changes to a set of related databases (yes, the change is tested in QA and it does improve the performance) and would like to get some numbers comparing the real-world, production performance before and after the changes. I don't need to dig to the database level or even lower, the […] SQL Server 2019 - Administration [Problems Connecting to Sybase using ODBC Drivers]( - Hi All, I have recently migrated a sql database from an old server to a new one. This database is populated from an old sybase database. I have made the required changes to the SSIS job using visual studio and runs there fine. I have setup the ODBC DSN connectors and they test successfully. However […] SQL Server 2019 - Development [automate sql installation in a 2-node windows failover cluster]( - Can you recommend best blog or approach or scripts to automate sql installation in a 2-node windows failover cluster? [Create list of random, mutually exclusive selection(s) from the same table]( - The TEMP table '#project_types' has 3 columns: 1) a pk, 2) the 'project_type' unique name, and 3) 'is_combinable' which is a 0/1 boolean (INT because legacy...). The job spec for this part (of the larger project specification) says: "randomly select a representative distribution of 1 value (if not combinable) or 1-3 values (if combinable) and […] [gut check dba's recommendation to increase log to 200gig]( - Hi , log files have always blown my mind. I'll try to explain simply what prompted this post and then ask the community for direction. we run a 2019 std sql agent job daily early morning mostly made up of 12 steps each responsible for ingesting to our wh sales facts and dimensions from 12 […] [ssis - same data source's revenue column violates constraints]( - Hi, i've babysat our 3am , 12 erp source (one step per source) ssis pkg/subpkgs based sqlagent job for months now at my new job. Twice now in the last week or so one sales view column "Revenue" from one specific erp has brought the job down. The reason ssis says it is failing is […] SQL Azure - Development [Tunning or alternative to PIVOT]( - Hi everyone, we have a table containing kind of key-value pairs, which are the columns of some entities. This is how master data is ingested in the platform. Then a query does some joins, filtering and finally a PIVOT to only output a row by entity, with all their columns denormalized. The query plan looks […] Integration Services [SSIS package not parsing statement with exec (SP) with resultset]( - I have a package that loops through a bunch of SQL Servers and executes a stored procedure that uses temp tables. It was build in 2008 and I am now moving it to 2016 but having issues getting it to work in 2008 the OLE DB sql was simply "uspSystem_GetDatabaseInfo" That was giving me grief […] [SSIS Execute Package Utility Issue]( - Hi everyone, So I have a ssis package that performs various operations like import, export etc based on values provided in config files. Today I have added some more code in the script task that downloads a pdf file, converts it to excel format and then reads this file and perform ETL operations. It loads […] [KingsWaySoft Premium Service Lookup Error: Must Specify atleast one output colum]( - Hi Experts, I have developed one SSIS package which watches a remote folder for .csv files and read them and then get lookup id where its required using KingsWaySoft Premium Service Lookup and then finally map values to CRM attributes in CRM Destination component. Now this all works smoothly when I run in Local system. […] Design Ideas and Questions [EF Schema Deployment]( - Just looking for some background on if anything has changed around "code first" for building schema. I've always preferred designing first, then building the schema (usually in a database project) and handling changes to schema that way. I'm in the position now where there is a project where it appears the developers picked EF and […] [Designing two tables considering following scenario.]( - I want to design two tables considering the following scenario. Consider the image below:   I’m moving blue boxes (which shows up on screen after Show Options the button is clicked) from left to right (on the grid) after clicking Move Text Content! button. Here are the steps I follow: Select one option from […] SQL Server 2022 - Administration [Always On-manual failover - warning on possible data loss]( - Hi, I'm testing basic alwayson with SQL Server 2022 Standard on two servers (primary and secondary replica with an old Adventureworks database). The idea is to allow sql clients to continue to operate while one server is down for updating and rebooting (monthy windows and sql server updates). I'm testing the manual failover: Although the […] SQL Server 2022 - Development [Contained availability groups error]( - Hello, I am testing the setup of contained availability groups. I am getting some strange errors while creating contained availability groups both through wizzard and with TSQL. I started with creating a contained availability group with the wizzard on MSSMS. Setup: I clicked the contained checkbox. The databases Meet prerequesites (FULL backup was made beforehand). […] [Seeding Contained availability groups]( - /    [RSS Feed]( This email has been sent to {EMAIL}. To be removed from this list, please click [here](. If you have any problems leaving the list, please contact the webmaster@sqlservercentral.com. This newsletter was sent to you because you signed up at SQLServerCentral.com. ©2019 Redgate Software Ltd, Newnham House, Cambridge Business Park, Cambridge, CB4 0WZ, United Kingdom. All rights reserved. webmaster@sqlservercentral.com  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -