How to find the missing data

Missing data is a common challenge in data science that can significantly impact the quality of analyses and the validity of conclusions drawn from datasets. Understanding the types of missing data, their causes, and effective handling techniques is crucial for data scientists. Here’s an overview of missing data in data science.

Types of Missing Data

1. Missing Completely at Random (MCAR)
   MCAR occurs when the likelihood of a data point being missing is completely independent of both observed and unobserved data. This means that the missingness does not depend on any characteristics of the individuals or items involved. For example, if a survey respondent accidentally skips a question due to a formatting error, that missing response is MCAR. When data is MCAR, analyses performed on the remaining data are unbiased, but true MCAR situations are rare in practice13.
2. Missing at Random (MAR)
   In this case, the probability of a data point being missing is related to observed data but not to the missing values themselves. For instance, if higher-educated individuals are less likely to report their income in a survey, the missingness is related to education level (an observed variable), not directly to income (the missing value). While MAR allows for some statistical techniques to estimate missing values based on available information, it still requires careful consideration24.
3. Missing Not at Random (MNAR)
   MNAR occurs when the missingness is related to the unobserved value itself or other unmeasured variables. For example, individuals with high stress levels may choose not to disclose their stress levels due to stigma. In this scenario, the reason for missing data is directly tied to the nature of the data itself, making it challenging to handle appropriately15.

Causes of Missing Data

Missing data can arise from various sources:

• Nonresponse: Participants may skip questions in surveys or drop out of studies.
• Data Entry Errors: Mistakes during data collection or entry can lead to missing values.
• Attrition: Participants leaving a longitudinal study can result in incomplete datasets.
• Systematic Bias: Certain groups may be less likely to respond due to specific characteristics or circumstances.

Handling Missing Data

There are several strategies for addressing missing data:

1. Deletion Methods
   • Complete Case Analysis (Listwise Deletion): Removes all observations with any missing values. This method can lead to biased results if the missingness is not MCAR.
   • Available Case Analysis (Pairwise Deletion): Only removes cases with missing values for specific analyses, allowing for more data utilization compared to listwise deletion.
2. Imputation Methods
   • Single Imputation: Replaces missing values with a single value such as the mean or median of available observations. While simple, this method can introduce bias.
   • Multiple Imputation: Involves creating several different plausible datasets by imputing values multiple times and then combining results. This method accounts for uncertainty around the imputed values and is generally more robust than single imputation.
3. Model-Based Approaches
   Techniques such as Maximum Likelihood Estimation (MLE) and Bayesian methods can be used to estimate parameters while accounting for missingness without directly imputing values.
4. Using Algorithms That Handle Missing Data
   Some machine learning algorithms can handle missing values inherently without requiring imputation or deletion.