Who's Responsible for Data Quality (MSSQLTips)

You've probably heard the saying "garbage in, garbage out" many times during your career as a data professional. Whether you are the person that designs the databases or applications, the person that bulk loads data into systems, the person that directly inputs data or a consumer of the data in applications, analytics or reporting, we all have a part in the quality of the data. If the data being entered into systems is not consistent and therefore cannot be trusted, the people that deal with the data for reports and analytics spend a lot of time massaging and fixing data anomalies to make sure the information that is produced is as useful as possible. If data issues are not corrected, you run the risk of producing output that is erroneous and therefore questionable. So how do you solve this quality of data issue? Whose responsibility is it that the data in systems is accurate, valid and complete? As data moves through the various components of a system, measures should be put in place for each time data is modified. This can be programmatic or manual processes, but the key to consistency is that all modes of data change are done consistently throughout all stages. How can this be done? The burden of data quality falls on every person or system that interacts with the data as follows: - Data Modeler / Architect - before systems are built, these people need to understand the purpose of the system so they can provide clear guidelines as to how data is stored and the business rules around the data. - Database Administrator - these people need to take the guidelines and implement a database solution that adheres to what needs to be put in place. - Database Developer - database level code should be in place to make sure the data adheres to the requirements and rules that have been outlined. - Application Developer - application level code needs to use techniques that only allow for valid data input to eliminate erroneous data entry as much as possible. - BI Professional - when data issues are identified, all parties that are part of the lifecycle of a system need to be notified so new measures can be put in place. - Management - identify data rules that need to be put in place. Make sure the rules are not constantly changing. Don't allow for shortcuts to systems to meet a deadline and jeopardize the integrity of the data. - Business User - use the systems as they were intended. Don't take shortcuts or use parts of the application for things they were not intended. Try to avoid building your own systems for shortcomings in applications and if you do create department level systems make sure data integrity continues to be part of the process. Unfortunately, when database applications are built, most of the above is followed when the initial system is released, the problems arise as the system is used and requirements change during the life of the database application. Some changes require major updates and this is often where shortcuts come into play. Management might be more concerned about meeting a deadline than making sure changes can be supported and allow for the data rules to be followed. Another common scenario is where sideline applications are built to support requirements that were never part of an application, but the data becomes so critical it needs to be meshed with other systems data, but since strict guidelines were not put in place the data is often not reliable and there is a need for a lot of data rework. In addition to the requirements changes, there are often data changes that need to be put in place. Consolidation of systems often pose issues where there is a need to merge different data sets, data stored in systems may become stale and not reliable. The reliance on outside data sources may also play a role in creating data inconsistencies. As you are probably well aware, there is not a simple solution to ensuring data quality throughout. The only simple answer would be a static system that never changes, which in most cases does not provide a viable business solution. Did you know that Melissa offers tools for various types of data professionals. This includes [Unison]( for business users, data quality components for [SSIS]( for DBAs and Developers as well as a full scale [API]( for business applications, check out their solutions to assist in your data quality. So what causes data quality issues? Stay tuned for the next installment. Edgewood Solutions LLC | PO Box 682, Wilton, NH 03086 [Unsubscribe {EMAIL}]( [Update Profile]( | [Our Privacy Policy]( | [Constant Contact Data Notice]( Sent by newsletter@mssqltips.com