Kill Your Darlings - Managing Poor-Quality Data in Data Migration
At some point, every database will face the issue of poor-quality data, and you’ll need to decide whether to keep, enhance, or remove it. This often comes up during data migration, such as when moving to a new CRM with stricter requirements for minimum fields.
Table of Contents
So how do you start fixing poor-quality data?
Before deciding what to do with poor-quality data, first try assessing its current state. Data quality metrics help determine whether your questionable customer information should be kept, enhanced, or discarded. While these metrics may differ slightly depending on your industry, we’ve got a few of the most critical data quality metrics:
- Accuracy: How correct is the data? For example, does the customer address match their actual residence, or are phone numbers up to date?
- Completeness: Are all necessary fields (First Name, Middle Name, Last Name, Phone Number, and more) populated?
- Consistency: Does the data remain uniform across all systems and platforms? Is “John Doe” still “John Doe” For example, if a customer’s name is “John Doe” in one system, it shouldn’t appear as “J. Doe” in another. Inconsistent data often signals the need for deduplication and normalization.
- Timeliness: How recent is the data? Outdated data may no longer be relevant and could lead to incorrect insights. For example, leads from five years ago that haven’t been updated are likely no longer accurate.
- Relevance: Is the data still useful for the business? Data that doesn’t serve a business purpose or assist in decision-making is often better to archive or delete.
Evaluate your data with these points in mind. It should help paint a clearer picture of its quality, helping you decide whether to cleanse, enhance, archive, or delete it. And when you use tools like DataTools Kleber, you can automate parts of this evaluation, allowing for faster and more accurate assessment.
What are your options?
- Fill in the gaps: You could add generic information to meet the new CRM’s minimum requirements. However, this doesn’t enhance the quality of your data.
- Use third-party services: Engage a third-party provider to append missing phone numbers, addresses, or other details.
- Archive the old data: Maintain a legacy copy of the old CRM and archive the data for future use.
A great tool to consider is DataTools Kleber, which can help clean up your address fields using advanced parsing and normalise phone numbers with just a click. These tools can significantly speed up the process of data normalisation.
Could I just delete poor quality data instead?
Now, if you’re considering deleting old data, think carefully! One safe strategy is to delete data based on timeframes tied to your business cycles. For example, if leads from over five years ago don’t meet your data requirements, it might be time to let them go. Similarly, consider appending or analysing older customer data before deciding to delete it.
Ultimately, the decision to keep, archive, or delete data depends on your organization’s needs and what you’re comfortable with. There’s no single right answer—just what makes sense for your business.
