Advantages of Kappa architecture in the Modern Data Stack (2023-06-19)

SQLServerCentral Newsletter for June 19, 2023 Problems displaying this newsletter? [View online](. [SQL Server Central]( Featured Contents - [Advantages of Kappa architecture in the Modern Data Stack]( - [ISJSON enhancements in SQL Server for valid JSON format for a Value, Array or Object]( - [Overcoming Organization Challenges in Cloud Migration: A Webinar for Senior IT Leaders]( - [Understanding Power BI Security Options - Row-Level, Column-Level and Object-Level Security]( - [From the SQL Server Central Blogs - Power BI – Empowering Users to Disable Tooltips]( - [From the SQL Server Central Blogs - Flyway with Multiple Folders]( - [Azure SQL Revealed: A Guide to the Cloud for SQL Server]( Question of the Day - [The JSON Table]( The Voice of the DBA  Highly Skilled The world of work has been changing quite a bit over the last few years. The pandemic appeared to be a boon to technology workers, both by allowing us to work remotely, but also taking advantage of our skills to deploy more technology throughout the world. At the same time, late 2022 and 2023 have seen many companies pull back and cut their staff, including technical staff. It seems that in mid-2023, we have both a demand for technology workers and a lot of people looking for technology jobs. A recent 2022 report on working in technology has some interesting insights. It is [summarized at ZDNet]( but you can see the [original report here](. The highlights from the report are that quite a few top performers left their jobs, recruitment doesn't work well, and finding top talent is hard. These aren't that surprising to many of us that actually write code or manage systems, but it is nice to see that founders and executives are reporting this. For much of my career, I do think many executives have taken advantage of tech workers, pressing them to get more and more done with fewer resources. Even though many people were paid well (though not all), they may not have enjoyed their jobs. I've had a lot of great positions, but over the years, I've met a surprising number of people who wouldn't recommend others follow them into their career choice. The pandemic made it easier for people to switch jobs since everyone was remote. That might not have benefitted lots of people, but it certainly helped top talent. If you're one of those people, you likely got a better job, with better compensation, and hopefully, more interesting work to you. For others, we might still be looking for something new (or worried about the security of our current position). One of the main parts of the report talks about the problems with the traditional recruitment model. First, it takes too long, an average of 4 months to fill technical roles. There is pressure to speed that up, while also understanding that candidates will need some training and re-skilling or up-skilling to meet an organization's particular needs. That would seem to be good for many candidates, but I think this report is biased to startups and newer organizations, not mature ones. I do think the impressions you make early are important. A candidate needs to ensure that they have technical skills. You should be able to solve problems but also think on your feet. Respond to the interviewer and showcase critical thinking. A developer should be able to answer general questions, like how to find duplicates in a table or sort data that is aggregated. You don't need to solve all of [Itzik's query problems]( but you ought to know some of them. I'd expect you to know the basics of this job. If you're unsure, post a note on the forums and ask people what things they need to know to do their jobs. However, apart from the technical skills, you need good soft skills. A strong candidate will communicate well, display confidence, relate to the interview(ers), and build some rapport. They will look to ask questions that seek more information about what is asked, but without being confrontational or disagreeable. There's a skill to debating a point rather than arguing with someone. The ability to get along with others and find synergy is important. When others think adding you to the team makes the entire team better, you stand a good chance of getting an offer. If you're unsure of whether you present a good impression, get a few friends or co-workers to interview you and give you honest feedback. Maybe even ask a friend who hires people to mock interview for a position. Learning to present a better impression will go a long way toward increasing your chances of getting hired. Especially if firms are trying to make quicker decisions. Maybe the big change from the report is 80% of people surveyed say they are willing hire someone without a college degree. Whether they actually would to that, whether their HR or other policies allow this, and whether they convince existing employees to do this remains to be seen. However, many of us know that a college degree isn't a good predictor of whether someone can do any job well. I hope future hiring trends confirm that. Steve Jones - SSC Editor [Join the debate, and respond to today's editorial on the forums](   Featured Contents [SQLServerCentral Article]( [Advantages of Kappa architecture in the Modern Data Stack]( anishninan from SQLServerCentral A comparison of the lambda and kappa architectures for real time ingestion and analysis of big data. [External Article]( [ISJSON enhancements in SQL Server for valid JSON format for a Value, Array or Object]( Additional Articles from MSSQLTips.com SQL Server supports working with JSON data and provides many different functions that can be used. SQL Server 2022 has expanded the ability to check if the JSON format is valid using the ISJSON function which we will cover in this article. [External Article]( [Overcoming Organization Challenges in Cloud Migration: A Webinar for Senior IT Leaders]( Additional Articles from Redgate Is your team looking to modernize and migrate your data but have questions about the complex organizational challenges? This webinar panel on June 29th covers all aspects of data modernization and migration that senior IT leaders should be aware of, including organization culture, communication, and politics. Why not send it to your boss? [Technical Article]( [Understanding Power BI Security Options - Row-Level, Column-Level and Object-Level Security]( Steve Jones - SSC Editor from SQLServerCentral In this article, learn how to control access levels to various data and parts of Power BI reports using row-level security as well as column and object-level security. [Blog Post]( From the SQL Server Central Blogs - [Power BI – Empowering Users to Disable Tooltips]( DataOnWheels from DataOnWheels Let’s set the scene. You’ve built a wonderful, useful, and descriptive report for your stakeholders with a variety of tooltips that offer deeper insights. They love the tooltips, and... [Blog Post]( From the SQL Server Central Blogs - [Flyway with Multiple Folders]( Steve Jones - SSC Editor from The Voice of the DBA A customer asked about how they could organize their migration scripts in different ways to manage them and worried about it being complex. I decided to test a few... [Azure SQL Revealed]( [Azure SQL Revealed: A Guide to the Cloud for SQL Server]( Site Owners from SQLServerCentral Access detailed content and examples on Azure SQL, a set of cloud services that allows for SQL Server to be deployed in the cloud. This book teaches the fundamentals of deployment, configuration, security, performance, and availability of Azure SQL from the perspective of these same tasks and capabilities in SQL Server. This distinct approach makes this book an ideal learning platform for readers familiar with SQL Server on-premises who want to migrate their skills toward providing cloud solutions to an enterprise market that is increasingly cloud-focused.   Question of the Day Today's question (by Steve Jones - SSC Editor):  The JSON Table Which JSON related function returns a table? Think you know the answer? [Click here]( and find out if you are right.    Yesterday's Question of the Day (by Steve Jones - SSC Editor) Contained AG MSDB Naming I have a SQL Server 2022 Contained Availability Group named "FinanceAG". What will be the name of the contained msdb that is included in this AG? This is the name that an admin will see in SSMS if not connected with a login used in the contained AG. Answer: FinanceAG_msdb Explanation: When you use a contained AG, the system creates a new master and msdb with the names of _master and _msdb. In this case, the names will be FinanceAG_msdb and FinanceAG_master. Ref: What is a Contained Availability Group? - [ [Discuss this question and answer on the forums](    Database Pros Who Need Your Help Here's a few of the new posts today on the forums. To see more, [visit the forums](. --------------------------------------------------------------- SQL Server 2017 - Administration [Availability Group database went into 'Restoring' state]( - Dear Friends, Please advise why would one of my Availability Group databases go into 'Restoring' state. Thanks in advance.  [SSIS DB]( - I was thinking to add SSIS db to Always on AG group. So during failover does anything need to be done differently in order to work packages? SQL Server 2016 - Administration [Database copy issue through - COPY DB WIZARD]( - I have a production server with SQL server 2016 version (13.0.5026.0) and a database with 25 GB of Data and 525 tables in it. Now, I want to transfer this production DB to my development environment with SQL server 2016 version (13.0.5026.0). I am using DB copy wizard for DB copy to the new instance […] SQL Server 2016 - Development and T-SQL [Help with a SQL Unpivot]( - I am trying to use unpivot to extract some data, however, the results are getting messed up on at least one record. By this I mean it is out of order which it should be CLNUM: MODEL: CLSTATUS: CLSTATDATE: CLSTOP: Here is the query any help or advice would be appreciated. WITH ClientsWithOpenMattersCTE (ClientNo, HowMany) […] SQL Server 2019 - Administration [Reset launchpad service to use default account]( - I have an issue with SQL Server Launchpad service. Initially it wouldn't start (configured to use "NT Service\MSSQLLaunchpad$") but on changing it to the domain account which runs the SQL Server service it starts without issue. I'd like to change it back to the default to see if it still works after doing so but […] [Can't Restore DB]( - hi everyone I need to get a copy of a stored proc from backup b/c the one I am using in my current instance of the db isn't correct. I am not able to restore the db. What am I doing wrong? The name of the db is OptionsDB and I want to keep the […] [partitioning a huge table]( - Hi folks I would like to get your views about partitioning a huge table in sql server .Around on size of 100 gb.Beofre i present this idea to upper mnmgt i would like to have your inputs .I am of view that since there is only 1 dedicated disk for data files (mdf/ndf) there is […] [Access db]( - Access db wanting to move  sql server, how much control does business have over table management? SQL Server 2019 - Development [How Can I Speed This Query Up?]( - Hi everyone My SQL query is close to 1000 lines with 17 separate calcs (ie sub-queries). The entire query takes almost 3 minutes to run. This is way too long. I figured out the problem area. It turns out that 70% of the time is spent on 2 subqueries. Each is the same except that […] [puzzeled with "exec sp_cursor 180150007,40,1"]( - I've noticed in one of our applications a cursor uses operation 40 and that operation is very slow. This handles a table with 62 rows with images ( most < 500KB , but some (5) > 2MB ) MS Docs sp_cursor describe optype, but I cannot actually get my interpretation around as to why this […] [Not wanting to use a Cursor, but...]( - Ok, need someone to talk me off the cursor ledge on this one. I have some train stop data that I need to accumulate the number of riders for, over a time when the train is heading in one direction, on a specific day. The data is all stored on a yearly basis, so I […] Reporting Services [Changing a datasource using T-SQL Script]( - I've been looking for a way to change multiple data sources for reports to a different data source. Is their a T-SQL Script that can accomplish this? Integration Services [You have an error in your SQL syntax; check the manual that corresponds]( - I am having great trouble creating a data pipeline using MariaDB connector 3 with SSIS/Mysql connector 8. I can view the tables, but it gives me this error: "TITLE: Microsoft Visual Studio ------------------------------ ERROR [42000] [ma-3.1.18][10.4.24-MariaDB]You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the […] [Iterate Through Tables, Using a Control Table w/ a List of Tables to Process?]( - I'm trying to create an SSIS package that will query a master control table [TableList], which contains 2 columns (SourceTable and TargetTable), then load those tables from a source server to a target server. It "runs", but although the variables with the table names seem to update for each iteration, it looks like the table […] SQL Server 2022 - Development [SQL Website Development: A Comprehensive Guide to Building Dynamic Web Applicati]( - Introduction to SQL Website Development What is SQL? Importance of SQL in web development Understanding Dynamic Web Applications Definition and examples Benefits of dynamic web applications SQL Database Management Systems Introduction to DBMS Popular SQL DBMS options Setting Up the Environment Choosing a programming language Installing a web server and database Creating the Database Structure […]   [RSS Feed]( This email has been sent to {EMAIL}. To be removed from this list, please click [here](. If you have any problems leaving the list, please contact the webmaster@sqlservercentral.com. This newsletter was sent to you because you signed up at SQLServerCentral.com. ©2019 Redgate Software Ltd, Newnham House, Cambridge Business Park, Cambridge, CB4 0WZ, United Kingdom. All rights reserved. webmaster@sqlservercentral.com  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -