Causes of Data Quality Issues (MSSQLTips)

Before data is introduced into a system rules need to be established to make sure everyone is following the same guidelines. It doesn't help if you put strict data entry rules in place and then get sloppy with data imports. As data passes through many processes, it gets difficult to employ the same measures and rules in each place unless clear business rules are defined. Here are a few things that we can do as data professionals to ensure the data stays as clean as possible. NULL values. One of the problems you will often face is that the data is not complete. Databases that don't enforce the need for data elements to be entered allow for the data to be easily skipped. Sure it is possible to clean up data after the fact, but enforcing strict database rules eliminates the need to have to fill in the blanks later on. Default values. Another problem is the use of default values. Was there a conscious effort by someone to enter the real value or did they just use the default to make it easier to get the record into the system. How reliable is this data on the backend when it comes time for some action to be taken with the data. Think through the need for default values and understand if this will lead to better or worse data quality. Too many values or options. Having too many options is also a sure way to get inconsistencies in the data. Is it easier to just pick something at the top of the list or are the options meaningful enough that the data will be used in an appropriate manner. Relying on users for accurate data entry. The use of predefined values could provide for better data versus free form entry. Free form data entry is great for notes and to add additional commentary, but for all critical data elements it is best to have a predefined list of values to make manual entry easier as well as to make automated processes conform to the same set of rules. Partner or third party data issues. Getting data from third party sources is an easy way to corrupt a lot of data all at once. Automation is necessary, but data cleanliness is key before introducing foreign data into your systems. Checks and balances need to be part of the automation process to ensure that the quality meets or exceeds the thresholds before new data is allowed into the system. Inconsistent rules from system to system. Systems don't live in silos anymore. There is way too much data that is valuable in many different systems and the need to combine and cross reference data from multiple systems is now the norm. By establishing different rules in each system it gets quite difficult to combine data in a meaningful way. There needs to be a thought process in place where you think through not only how the quality of the data impacts this one system, but how it could impact all systems. Lack of rules and planning. This is probably the most obvious cause. Systems aren't built just to collect and have perfect data, they are built to meet a business need. The problem with this is that if a system is thrown together to meet an immediate need, there are often many steps that are skipped just to meet a deadline. This may be true for a brand new system or for updates that need to be introduced. Always keep data quality in mind for whatever changes need to be introduced. Did you know that Melissa has been delivering data quality solutions for over 35 years and has had solutions for SQL Server for over 15 years? These solutions include data profiling, matching, reference sets, cleansing, enriching and more for 240 plus countries around the globe. Learn more at [Melissa.com](. So what rules do you follow for data quality? Stay tuned for the next installment. Edgewood Solutions LLC | PO Box 682, Wilton, NH 03086 [Unsubscribe {EMAIL}]( [Update Profile]( | [Our Privacy Policy]( | [Constant Contact Data Notice]( Sent by newsletter@mssqltips.com