lec3

Here is the comprehensive extraction of all technical keywords, nouns, verbs, formulas, and visual elements presented in a structured key-value table, followed by an in-depth summary of the entire lecture.

---

1. Key-Value Extraction Table (Keywords, Technical Nouns, Verbs & Visuals)

Technical Term / Keyword / Action (Noun/Verb)	Preprocessing Category	Key-Value Attributes / Technical Definition	Context, Examples & Visual Representation (from Text & Images)
Data Preprocessing (Noun / Verb)	Overall Framework	Role: Transforming raw, unclean data into an understandable, clean format for mining/ML.	Lecture title; pipeline foundational step.
Data Validation (Noun / Verb)	Validation	Definition: Automated check ensuring input data is sensible, valid, and structurally complete.	Slide 3, 5; ensures system input matches expectations.
Presence Check (Noun / Verb)	Validation	Condition: Validates that data exists and attributes/columns are complete.	Example: A data file must have exactly 8 columns.
Type Check (Noun / Verb)	Validation	Condition: Checks whether a field matches its designated primitive type.	Example: age must be strictly numeric.
Length Check (Noun / Verb)	Validation	Condition: Enforces string character length constraints.	Example: Address string length $\le$ 100 characters.
Range Check (Noun / Verb)	Validation	Condition: Restricts numerical or temporal values within bounded intervals.	Example: Month integer value must fall within [1–12].
Format Check (Noun / Verb)	Validation	Condition: Validates pattern compliance via regular structures.	Examples: Date format "DD-MM-YYYY", Email pattern XX@YY.ZZ.
Data Verification (Noun / Verb)	Verification	Definition: Cross-checking typed data against external logical/business rules.	Checking truthfulness, existence, or transactional clearance (Slide 6).
Business Rules (Noun)	Verification	Rules: Context-dependent business logic used for accuracy verification.	Pinging email address with confirmation link; checking if User ID exists; credit card purchase validation.
Data Cleaning (Noun / Verb)	Cleaning	Definition: Resolving missing values, smoothing noise, handling outliers, removing duplicates, and fixing inconsistencies.	Described as the “number one problem in data processing” (Slide 7).
Missing Data (Noun)	Cleaning	Definition: Unrecorded attribute values across tuples.	Causes: Sensor malfunction, misunderstanding, deleted inconsistencies, unrecorded historical shifts (Slide 8).
Listwise Deletion (Noun / Verb)	Missing Data	Action: Complete Case Analysis; drops the entire row/record if any attribute is missing.	Table (Slide 10): Row 2, 4, 5 crossed out due to missing gender or result. Code: df.dropna().
Pairwise Deletion (Noun / Verb)	Missing Data	Action: Available Case Analysis; uses available values for specific bivariate/multivariate tests, keeping partial rows.	Table (Slide 10): Retains rows, computing statistics only on non-null pairs.
Data Imputation (Noun / Verb)	Missing Data	Action: Systematically replacing missing values with estimated figures.	Automatic replacement strategies (Slide 11).
Global Constant (Noun)	Imputation	Value: Fixed category/flag (e.g., "unknown").	Creates a distinct pseudo-class for missing values.
Central Tendency Imputation (Noun / Verb)	Imputation	Formula/Metric: Replaces nulls with Mean ($\mu$), Median, or Mode.	Smarter variant: Imputing conditional mean belonging strictly to the same target class (Slide 11).
Forward Fill (ffill) (Noun / Verb)	Imputation	Direction: Propagates last known valid value forward in sequence.	Slide 12 diagram: Apple price 0.8 propagated downward to index 1 and 2; code: df.fillna(method='ffill').
Backward Fill (bfill) (Noun / Verb)	Imputation	Direction: Propagates subsequent valid value backward in sequence.	Slide 12 diagram: Mango price 1.2 at index 3 propagated to index 4; code: df.fillna(method='bfull').
K-Nearest Neighbors Imputation (KNNImputer) (Noun / Verb)	Imputation	Mechanism: Imputes missing feature based on mean/weighted values of $K$ nearest data points.	Slide 13–14: Scikit-learn implementation (from sklearn.impute import KNNImputer, n_neighbors=2).
Regression Imputation (Noun / Verb)	Imputation	Mechanism: Models missing attribute as a dependent target variable predicted from other features.	Fits linear/logistic model over non-null rows to predict null entries.
Most Probable Value (Noun)	Imputation	Mechanism: Probabilistic/symbolic inference using Bayesian inference or Decision Trees.	Infers replacement based on conditional probability distributions.
Machine Learning Pipeline (Noun)	Pipeline Workflow	Definition: Chaining data preprocessing transformers with predictive estimators.	Slide 14: Pipeline(steps=[('imputer', SimpleImputer(strategy='mean')), ('classifier', LogisticRegression())]).
Noisy Data (Noun)	Cleaning	Definition: Erroneous variance, corruption, or unwanted artifacts in data.	Types: Outliers, random input errors, format mismatches, irrelevant attributes, duplicates (Slide 15).
Data Smoothing (Noun / Verb)	Cleaning	Definition: Filtering noise out of continuous signals or ordered sequences.	Techniques: Moving average, exponential smoothing, bin smoothing (Slide 16).
Moving Average (MA) (Noun / Verb)	Smoothing	Action: Averages sliding time-window slices to dampen volatility.	Chart (Slide 16): Compares Original curve vs MA(3) (short-term), MA(5) (medium), MA(7) (long-term smoother trend).
Exponential Smoothing (Noun / Verb)	Smoothing	Action: Applies exponentially decreasing weighted averages over past observations.	Prioritizes recent observations while discounting older historical entries.
Equal-Width (Distance) Binning (Noun / Verb)	Smoothing / Discretization	Formula: $W=(B-A)/N$ Properties: Divides domain into $N$ equal intervals; uniform grid.	Slide 17–18: Prone to outlier skew; Example intervals: [-, 10), [10, 20), [20, +).
Equal-Depth (Frequency) Binning (Noun / Verb)	Smoothing / Discretization	Properties: Divides sorted data into $N$ intervals containing an equal count of samples.	Slide 17–19: Good scaling; Sorted price: [4, 8, 9, 15], [21, 21, 24, 25], [26, 28, 29, 34].
Smoothing by Bin Means (Noun / Verb)	Smoothing	Action: Replaces every item in a bin with the arithmetic mean of that bin.	Slide 19: Bin 1 becomes [9, 9, 9, 9], Bin 2 becomes [23, 23, 23, 23], Bin 3 becomes [29, 29, 29, 29].
Outlier Detection & Removal (Noun / Verb)	Cleaning	Definition: Identifying points that diverge abnormally from the general distribution.	Methods: Statistical ($Z$-score, IQR), ML (Regression residuals, DBSCAN), Visual (Boxplots, Scatterplots) (Slide 20).
Valid Outlier vs. Noisy Outlier (Noun)	Cleaning	Distinction: Distinguishing legitimate real-world extremes from measurement artifacts.	Valid: High CEO salary; Noisy: Human Age = 200 or GPS coordinates plotting inside the ocean (Slide 7, 15, 21).
Low-Pass Filter (Noun / Verb)	Signal Processing	Action: Attenuates high-frequency fluctuations while passing low-frequency fundamental trend.	Slide 21: Removes jitter from oscillating continuous signals.
Data Filtering (Noun / Verb)	Data Selection	Action: Narrowing dataset records/columns according to boolean conditional predicates.	Slice/subset operation: selecting target columns, records where ID = 5, or records from today (Slide 23).
Data Sorting (Noun / Verb)	Data Ordering	Action: Alphabetizing or ordering records based on hierarchical sort keys.	Multi-column rule: Sort the least important column first, progressing to most important last (Year $\to$ Month $\to$ Week).
Data Integration (Noun / Verb)	Integration	Definition: Merging data from disparate heterogeneous sources into a single coherent store.	Solves schema integration, entity identification, and structural conflict (Slide 25).
Schema Integration (Noun / Verb)	Integration	Challenge: Mapping disparate schema attributes representing the same conceptual field.	Slide 25: Matching A.cust-id with B.cust-#.
Entity Identification Problem (Noun)	Integration	Challenge: Recognizing identical real-world entities across differing representations.	Slide 25: Resolving "Bill Clinton" to "William Clinton".
Data Value Conflict (Noun)	Integration	Challenge: Divergence in values for identical entities due to alternate metrics or units.	Resolving metric units (e.g., cm/kg) vs British imperial units (e.g., inches/lbs).
Data Transformation (Noun / Verb)	Transformation	Definition: Converting data into functional formats suitable for downstream modeling.	Operations: Aggregation, skewness reduction (log transform), normalization, feature construction (Slide 27).
Data Skewness (Noun)	Data Distribution	Definition: Asymmetric distribution of data violating Gaussian normality assumptions.	Negative skew: $\text{Mean}<\text{Median}<\text{Mode}$ (left tail); Positive skew: $\text{Mean}>\text{Median}>\text{Mode}$ (right tail) (Slide 28).
Log Transformation (Noun / Verb)	Transformation	Function: $y=\log (x)$ or $\log (1+x)$ (np.log1p).	Compresses right-skewed tails into symmetrical bell curves (Slide 30, 33).
Min-Max Normalization (Noun / Verb)	Normalization	Formula: $v^{\prime }=\frac{v-{\min }_A}{{\max }_A-{\min }_A}({\text{new\_max}}_A-{\text{new\_min}}_A)+{\text{new\_min}}_A$	Maps income $73,600fromrange‘[$12,000, $98,000]‘to‘[0.0,1.0]‘,yielding$0.716$ (Slide 31).
Z-Score Normalization (Noun / Verb)	Standardization	Formula: $v^{\prime }=\frac{v-{\mu }_A}{{\sigma }_A}$	Scales to mean 0, variance 1. Income $73,600with$\mu=54,000, \sigma=16,000$yields$Z = 1.225$ (Slide 31).
StandardScaler (Noun)	Python Transformer	Class: Scikit-learn scaler centering data to zero mean and unit variance.	Assumes normal distribution; distorted by skewed inputs and outliers (Slide 32).
MinMaxScaler (Noun)	Python Transformer	Class: Scikit-learn transformer bounded typically between $[0,1]$.	Highly sensitive to extreme outliers (Slide 32).
RobustScaler (Noun)	Python Transformer	Class: Scikit-learn scaler utilizing Median and Interquartile Range (IQR).	Immune to outlier distortion and skewed distributions (Slide 32).
Data Reduction (Noun / Verb)	Reduction	Definition: Reducing volume or dimensionality while preserving analytical integrity.	Mitigates storage, memory, and computational bottlenecks (Slide 34).
Dimensionality Reduction (Noun / Verb)	Reduction	Action: Pruning attribute space via Wavelet Transforms, PCA, or Feature Selection.	Slide 35: Eliminates redundant and irrelevant features.
Principal Component Analysis (PCA) (Noun / Verb)	Dimensionality Reduction	Mechanism: Orthogonal linear projection maximizing variance across orthogonal components.	Reduces dimensions while retaining maximum information (Slide 35).
Feature Selection (Noun / Verb)	Reduction	Action: Identifying the minimal subset of attributes sufficient for modeling.	Guided by statistical tests and Information Gain (Decision Trees) (Slide 36).
Redundant Attributes (Noun)	Reduction	Definition: Attributes that duplicate existing information (detected via correlation analysis).	Example: Purchase price and sales tax paid (Slide 37).
Irrelevant Attributes (Noun)	Reduction	Definition: Features providing zero predictive or analytical utility.	Example: Student ID when predicting GPA (Slide 37).
Clustering for Reduction (Noun / Verb)	Numerosity Reduction	Action: Compressing records into cluster representatives (centroids, diameters).	Replaces thousands of data points with cluster centroids (Slide 39–40).
Sampling (Noun / Verb)	Numerosity Reduction	Definition: Drawing a small subset $s$ to represent entire dataset $N$ in sub-linear time.	Enables heavy mining algorithms to scale to massive volumes (Slide 41).
SRSWOR (Noun)	Sampling	Type: Simple Random Sampling Without Replacement.	Selected object is permanently removed from the sampling population pool (Slide 42–43).
SRSWR (Noun)	Sampling	Type: Simple Random Sampling With Replacement.	Selected object is returned to the pool, allowing duplicate draws (Slide 42–43).
Stratified Sampling (Noun / Verb)	Sampling	Type: Partitioning population into non-overlapping strata and sampling proportionally.	Overcomes bias caused by skewed datasets (Slide 42, 44).
Data Discretization (Noun / Verb)	Discretization	Definition: Transforming continuous quantitative features into categorical interval buckets.	Unsupervised vs. Supervised; Top-down (Split) vs. Bottom-up (Merge) (Slide 46).
Encoding / Decoding (Noun / Verb)	Discretization	Process: Encoding converts descriptive strings to compact numeric/alphanumeric keys; Decoding reconstructs the original strings.	Severity example (Slide 47): Critical $\to 3$, Major $\to 2$, Minor $\to 1$, Info $\to 0$.
Granularity (Noun)	Discretization	Concept: The level of detail represented by intervals or categorical groupings.	Trade-off: High coding efficiency vs. information coarsening (Slide 48).
Coordinate Precision (Noun)	Discretization / Geospatial	Metric: Mapping decimal places of Latitude/Longitude to physical distance scales.	Table (Slide 49): 0 decimals = country ($111\text{ km}$); 4 decimals = parcel ($11.1\text{ m}$); 8 decimals = tectonic plates ($1.1\text{ mm}$).
Concept Hierarchy Generation (Noun / Verb)	Hierarchy Generation	Definition: Organizing low-level detailed attributes into hierarchical conceptual levels.	Schema DAGs: Address $\to$ ZipNo $\to$ City $\to$ Country; Numerical: Age $\to$ youth, adult, senior (Slide 50).
Automatic Hierarchy Generation (Noun / Verb)	Hierarchy Generation	Heuristic: Ordering nominal attributes based on unique distinct value cardinality.	Attribute with highest distinct count is placed at bottom (Slide 52: Street [674,339] $\to$ City [3,567] $\to$ State [365] $\to$ Country [15]).
Know Your Data (KYD) (Noun / Verb)	EDA / Preprocessing	Definition: Deep exploratory analysis of data characteristics, distributions, and flaws prior to modeling.	Slide 55: Essential preliminary phase to prevent model bias and guide preprocessing.
Cardinally & Unique Values (Noun)	KYD / EDA	Metric: Count of distinct values per attribute column.	Differentiates continuous attributes from high/low-cardinality categorical attributes (Slide 56).
Correlation Matrix (Noun)	KYD / EDA	Tool: Pairwise correlation computation across all numerical features.	Slide 59: Heatmap showing Age vs Spending ($-0.15$), Income vs Spending ($-0.07$).
K-Means Clustering (Noun / Verb)	Preprocessing / ML	Algorithm: Unsupervised partitioning algorithm grouping data into $K$ centroids.	Slide 60: 3D customer segmentation based on Age, Annual Income, and Spending Score (4 clusters: $N=62,42,45,51$).

---

2. Comprehensive Lecture Summary

This lecture covers the complete engineering and analytical lifecycle of Data Preprocessing within modern Big Data Analytics and Machine Learning workflows. Raw real-world data is invariably incomplete, noisy, inconsistent, and high-dimensional; direct ingestion into machine learning algorithms leads to biased models and degraded predictive accuracy. The lecture systematically deconstructs preprocessing into seven fundamental pillars:

Data Ingestion & KYD
       │
       ├──► 1. Validation & Verification (Ensuring structural & business integrity)
       ├──► 2. Data Cleaning (Handling Missing Data & Noisy/Outlier Data)
       ├──► 3. Data Filtering & Sorting (Focusing & structuring views)
       ├──► 4. Data Integration (Schema matching, entity resolution & deduplication)
       ├──► 5. Data Transformation (Log scaling, Min-Max & Z-Score Normalization)
       ├──► 6. Data Reduction (Dimensionality, numerosity, histograms, clustering, sampling)
       └──► 7. Discretization & Concept Hierarchies (Binning, encoding, taxonomic abstraction)

---

I. Data Validation and Data Verification

• Data Validation: A deterministic, rule-based process ensuring that ingested values are structurally sound and computationally sensible. Standard validation gates include:
  • Presence Checks (verifying non-null attributes and complete schema column counts),
  • Type Checks (confirming numeric, string, or boolean types),
  • Length Checks (e.g., character bounds $\le 100$),
  • Range Checks (e.g., month bounded within $[1,12]$),
  • Format Checks (regex validation for standard timestamps like DD-MM-YYYY and email strings).
• Data Verification: Cross-references the entered data against higher-level external systems, database state, or contextual business rules. Examples include checking whether a User ID or Product ID exists in database tables, verifying billing details during credit card authorization, or sending email verification loops.

---

II. Data Cleaning: Missing and Noisy Data

Data cleaning is highlighted as the primary challenge in preprocessing:

1. Missing Data:
   • Causes: Hardware failure, schema changes, inconsistent input deleted retroactively, or user refusal to provide sensitive inputs (e.g., income).
   • Deletion:
     • Listwise Deletion (Complete Case Analysis): Drops the entire row if a single value is missing. This introduces severe sample bias if data is not missing completely at random.
     • Pairwise Deletion (Available Case Analysis): Uses all complete attribute pairs for specific calculations while retaining incomplete records for others.
   • Imputation Techniques:
     • Global Constants (e.g., tagging missing items as "unknown").
     • Central Tendency (imputing global or class-conditional mean, median, or mode).
     • Sequential Methods (Forward Fill ffill and Backward Fill bfill for time-series).
     • Predictive/Distance-based Imputation ($K$-Nearest Neighbors via KNNImputer, Linear/Logistic Regression imputation, and Bayesian Decision Trees).
     • Pipelines: Encapsulating imputers within sklearn.pipeline.Pipeline to avoid data leakage during training/testing splits.
2. Noisy Data & Smoothing:
   • Noise manifests as outliers, typos, measurement variances, or duplicate records.
   • Smoothing Methods:
     • Moving Averages (MA): Dampening high-frequency noise in temporal data using window sizes (e.g., MA(3), MA(5), MA(7)).
     • Exponential Smoothing: Decreasing historical weight decays.
     • Bin Smoothing: Partitioning sorted values into equal-width or equal-depth bins, subsequently replacing bin members with the bin mean or boundaries.
   • Outlier Detection: Distinguishes valid outliers (e.g., legitimate high CEO compensation) from corrupt outliers (e.g., Age = 200 or GPS coordinates falling into the ocean). Methods include statistical thresholds ($Z$-score, Interquartile Range / IQR), clustering algorithms (DBSCAN), regression residuals, and low-pass filtering.

---

III. Data Filtering and Sorting

• Filtering: Subsetting and narrowing data views to focus exclusively on relevant cohorts (e.g., isolating specific identifiers, temporal windows like date = today, or omitting non-essential feature columns).
• Sorting: Ordering datasets using single or compound keys. When performing multi-column sorting, the correct technical sequence is to sort the least important column first and the most important column last to establish prioritized hierarchies (e.g., Week $\to$ Month $\to$ Year).

---

IV. Data Integration and Redundancy Resolution

• Integration Challenges:
  • Schema Integration: Harmonizing disparate naming conventions and metadata (e.g., cust-id $\equiv$ cust-#).
  • Entity Identification: Mapping real-world entities under differing typographical aliases (e.g., "Bill Clinton" $\equiv$ "William Clinton").
  • Data Value Conflicts: Reconciling different unit measurements (Metric vs. Imperial scales).
• Redundancy Resolution: Redundant attributes (such as storing both unit price and sales tax, or derivable revenue totals) inflate memory usage. They are flagged using correlation analysis and removed to prevent multicollinearity and speed up model training.

---

V. Data Transformation: Skewness and Normalization

• Skewed Data Handling:
  • Many statistical estimators assume Gaussian normality. Positively skewed data ($\text{Mean}>\text{Median}>\text{Mode}$) and negatively skewed data ($\text{Mean}<\text{Median}<\text{Mode}$) induce model prediction bias, distort variance, and cause node workload imbalance in distributed systems like Apache Spark or Hadoop.
  • Log Transformation (np.log1p) stabilizes high variance and converts long-tailed distributions into near-normal distributions.
• Feature Scaling / Normalization:
  • Min-Max Normalization: Scales values to a fixed range (typically $[0,1]$): $v^{\prime }=\frac{v-\min }{\max -\min }(\text{new\_max}-\text{new\_min})+\text{new\_min}$ Limitation: Sensitive to extreme outliers.
  • Z-Score Standardization: Centers data to zero mean with unit variance: $v^{\prime }=\frac{v-\mu }{\sigma }$ Limitation: Assumes Gaussian underlying distributions.
  • RobustScaler: Standardizes features using the Median and IQR ($Q_3-Q_1$), preventing outliers from skewing the scaling scale.

---

VI. Data Reduction Strategies

To combat computational bottlenecks, data reduction condenses volume and dimensionality while preserving analytical integrity:

1. Dimensionality Reduction: Eliminates irrelevant and redundant features. Methods include:
   • Wavelet Transforms and Principal Component Analysis (PCA).
   • Feature Selection: Identifying the minimal sufficient feature subset via statistical hypothesis tests or Information Gain metrics.
2. Numerosity / Volume Reduction:
   • Histograms & Binning: Storing frequency summaries instead of raw observations.
   • Clustering: Replacing massive instance clouds with compact centroid vectors and cluster radii.
   • Sampling: Generating representative subsets in sub-linear time. Uses Simple Random Sampling Without Replacement (SRSWOR), With Replacement (SRSWR), and Stratified Sampling (which partitions data into strata to preserve minority classes within skewed distributions).
   • Data Compression: Lossy and lossless encoding schemes.

---

VII. Data Discretization, Coding, and Concept Hierarchies

• Discretization: Converts continuous variables into distinct categorical intervals via top-down (splitting) or bottom-up (merging) algorithms (supervised or unsupervised).
• Coding (Encoding/Decoding): Replaces long string descriptions with compact numerical or alphanumeric codes (e.g., incident severity: Critical $\to 3$, Major $\to 2$, Minor $\to 1$, Info $\to 0$). This accelerates computation, minimizes storage, and simplifies validation, but must be balanced against loss of information granularity.
• Geospatial Discretization: Latitude and longitude precision tables demonstrate that truncation controls granularity—from global regions ($0$ decimal places $\approx 111\text{ km}$) down to specialized structural/tectonic surveys ($8$ decimal places $\approx 1.1\text{ mm}$).
• Concept Hierarchy Generation:
  • Constructs Directed Acyclic Graphs (DAGs) and taxonomies to group low-level primitives into abstract concepts (e.g., Address $\to$ Zip $\to$ City $\to$ Country; or numeric age into youth, adult, senior).
  • Automatic Generation: Can be inferred automatically by counting unique attribute cardinalities. Attributes exhibiting the highest number of distinct values are positioned at the deepest, most granular leaf levels of the hierarchy (e.g., Street [$674,339$] $\to$ City [$3,567$] $\to$ State [$365$] $\to$ Country [$15$]).

---

VIII. “Know Your Data” (KYD) and Exploratory Case Study

Before training machine learning algorithms, practitioners perform KYD (Exploratory Data Analysis) to inspect data quality and guide feature engineering. The KYD lifecycle covers seven primary activities:

1. Data Overview (dimensions, types, sample head records),
2. Summary Statistics (mean, median, mode, spread, skewness),
3. Missing Value Audits (null counts, patterns of missingness),
4. Cardinality Checks (unique values per attribute),
5. Outlier Detection (boxplots, IQR, $Z$-score),
6. Correlation Analyses (Pearson/Spearman for numerical features, Chi-square for categorical features),
7. Visualizations (histograms, scatterplots, correlation heatmaps).

Case Study Synthesis: A synthetic retail customer dataset ($N=200$) with features Age ($18–69$), Annual Income (mean \approx \50,000$),‘SpendingScore‘($1–99$),and‘Gender‘demonstratestheseprinciplesinpractice.Summarystatisticsrevealnomissingvalues,awideincomedistributionwith2extremeoutliers,andweakpairwisecorrelations(Agevs.SpendingScore:$-0.15$;Incomevs.SpendingScore:$-0.07$).

Finally, applying 3D K-Means Clustering across Age, Annual Income, and Spending Score reveals four distinct, actionable behavioral cohorts:

• Cluster 0 ($N=62$ customers),
• Cluster 1 ($N=42$ customers),
• Cluster 2 ($N=45$ customers),
• Cluster 3 ($N=51$ customers).

This illustrates how end-to-end preprocessing directly powers downstream unsupervised machine learning and business segmentation.

---

Here is an exam-oriented study package designed specifically for written mid-terms. It skips coding syntax and focuses on pipeline block diagrams you can draw on paper, high-yield short-answer questions, step-by-step calculation problems, and long-answer scenario questions.

---

Part 1: High-Yield Pipeline & Block Diagrams

(Practice drawing these simple ASCII block diagrams on paper—professors love giving points for clean diagrams!)

1. The Complete End-to-End Data Preprocessing Pipeline

[ Raw Heterogeneous Data ]
           │
           ▼
┌─────────────────────────────────────────────────────────────┐
│ 1. Data Validation & Verification                           │
│    • Validation: Structure/Type/Format/Range checks        │
│    • Verification: Cross-check with Business Rules         │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│ 2. Data Cleaning                                            │
│    • Missing Data: Listwise/Pairwise Deletion vs. Imputation│
│    • Noisy Data: Smoothing (MA, Binning), Outlier Removal   │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│ 3. Data Integration                                         │
│    • Schema matching, Entity resolution, Redundancy removal │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│ 4. Data Transformation                                      │
│    • Log Transformation (reduce skewness)                   │
│    • Normalization (Min-Max, Z-Score)                       │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│ 5. Data Reduction & Discretization                          │
│    • Dimensionality (PCA, Feature Selection)                │
│    • Numerosity (Clustering, Stratified Sampling)           │
│    • Discretization & Concept Hierarchies                   │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ▼
[ Clean, Transformed Data Ready for ML / Analytics ]

---

2. Missing Data Decision Flowchart

                 [ Missing Value Detected ]
                             │
            Is the class label itself missing?
                     /               \
                 (Yes)               (No)
                  /                     \
       [ Ignore Tuple / Row ]     Is dataset small or rows
                                  irreplaceable?
                                   /           \
                               (Yes)           (No, large data)
                                /                 \
                     [ Imputation ]           [ Deletion ]
                     /      │     \             /       \
            Time-series  Tabular Complex     Listwise   Pairwise
                 │          │       │        (Drop row) (Drop for
            (ffill/bfill) (Mean/   (KNN /                specific
                          Median/ Regress)               test)
                           Mode)

---

3. Skewness Distribution Comparison (Must Know for Quick Sketching!)

  Negative Skew (Left-tailed)         Normal Distribution         Positive Skew (Right-tailed)
        Mean < Median < Mode          Mean = Median = Mode            Mode < Median < Mean

              /\                              /\                              /\
             /  \                            /  \                            /  \
            /    \                          /    \                          /    \
           /      \                        /      \                        /      \
      ____/        \                  ____/        \____                  /        \____
      ▲     ▲    ▲                         ▲                              ▲    ▲     ▲
    Mean Median Mode              Mean = Median = Mode                  Mode Median Mean

---

4. Concept Hierarchy Trees (Taxonomy)

          [ Country ]                         [ Year ]
               ▲                                ▲
               │                                │
       [ State / Province ]                 [ Quarter ]
               ▲                                ▲
               │                                │
            [ City ]                         [ Month ]
               ▲                                ▲
               │                                │
            [ Street ]                       [ Day ]
 (High Cardinality / Specific)       (Low Cardinality / Specific)

---

Part 2: High-Yield Short-Answer Q&A (2–4 Marks Each)

Q1: What is the exact difference between Data Validation and Data Verification?

• Data Validation: An internal, automated check ensuring the input format and syntax are sensible and complete for the computer (e.g., Age is numeric, date is "DD-MM-YYYY", length $\le 100$).
• Data Verification: Checking entered data against external facts or business rules to ensure real-world truthfulness and accuracy (e.g., sending an activation email to verify it exists, checking if User ID exists in the database, verifying a credit card transaction).

Q2: Contrast Listwise Deletion vs. Pairwise Deletion. What is the major trade-off?

• Listwise Deletion (Complete Case Analysis): If any attribute in a row is missing, the entire row is discarded.
  • Trade-off: Simple and unbiased if data is Missing Completely at Random (MCAR), but wastes large amounts of valid data.
• Pairwise Deletion (Available Case Analysis): Only discards missing values for specific bivariate analyses (e.g., correlation between feature A and B), preserving that row when analyzing feature B and C.
  • Trade-off: Retains more data, but sample sizes vary across tests, which can introduce mathematical inconsistencies.

Q3: Explain the difference between a “Valid Outlier” and a “Noisy Outlier” with examples.

• Valid Outlier: A legitimate, real-world extreme data point that reflects true reality. It should not be deleted blindly (e.g., the multi-million dollar salary of a company CEO in an employee salary dataset).
• Noisy Outlier: An error caused by measurement failure, sensor fault, or input typo that corrupts the data (e.g., a person’s Age = 200, or GPS trajectory points appearing inside the ocean due to GPS signal loss).

Q4: What is the difference between Redundant Attributes and Irrelevant Attributes?

• Redundant Attributes: Features that duplicate information already present in other attributes (e.g., storing both Product Price and Sales Tax Paid when tax is a fixed percentage; or Birth Date and Age).
• Irrelevant Attributes: Features that provide zero predictive or analytical value for the specific problem at hand (e.g., using Student ID to predict a student’s GPA).

Q5: When should you use RobustScaler instead of StandardScaler or MinMaxScaler?

• MinMaxScaler maps data into $[0,1]$, but extreme outliers compress all normal data into a tiny sub-range.
• StandardScaler relies on the sample Mean and Standard Deviation, both of which are heavily distorted by outliers and skewed data.
• RobustScaler uses the Median and Interquartile Range (IQR), which are resistant to outliers. Therefore, RobustScaler is preferred when the dataset contains significant outliers or heavy skewness.

Q6: Why does Skewed Data cause major performance bottlenecks in distributed big data engines (e.g., Hadoop, Apache Spark)?

• Distributed engines partition data across worker nodes using hash keys.
• If data is heavily skewed on a key, one partition becomes disproportionately massive (Data Skew / Straggler problem).
• One worker node is forced to process 90% of the volume while all other nodes sit idle, completely breaking distributed parallel processing.

Q7: State the rule used to automatically generate Concept Hierarchies for nominal data. What is the major exception?

• Rule: Calculate the number of distinct values (cardinality) per attribute. The attribute with the highest distinct count is placed at the lowest, most detailed level of the hierarchy (e.g., Street [674,339] $\to$ City [3,567] $\to$ State [365] $\to$ Country [15]).
• Exception: Temporal / Calendar attributes. In calendar data, a year has fewer distinct values in a typical dataset than months or days, but Year is conceptually higher than Month or Day.

Q8: Distinguish between SRSWOR, SRSWR, and Stratified Sampling.

• SRSWOR (Without Replacement): Once a data item is selected, it is removed from the population pool; an item can only appear once.
• SRSWR (With Replacement): Selected items are returned to the pool; an item can be sampled multiple times.
• Stratified Sampling: The population is divided into distinct non-overlapping sub-groups (strata), and random samples are drawn proportionally from each stratum. This guarantees minority classes are represented in skewed datasets.

---

Part 3: Step-by-Step Calculation / “Show Your Work” Questions

Problem 1: Binning Methods & Smoothing by Bin Means

Given raw data: [4, 8, 9, 15, 21, 21, 24, 25, 26, 28, 29, 34] ($N=12$)

Task A: Partition into 3 Equal-Frequency (Equi-Depth) Bins and Smooth by Bin Means

1. Calculate items per bin: $\text{Depth}=\frac{\text{Total items}}{\text{Number of bins}}=\frac{12}{3}=4\text{ items per bin}$
2. Partition into bins:
   • $\text{Bin 1}=[4,8,9,15]$
   • $\text{Bin 2}=[21,21,24,25]$
   • $\text{Bin 3}=[26,28,29,34]$
3. Compute Bin Means:
   • $\text{Mean}(\text{Bin 1})=\frac{4+8+9+15}{4}=\frac{36}{4}=9$
   • $\text{Mean}(\text{Bin 2})=\frac{21+21+24+25}{4}=\frac{91}{4}=22.75\approx 23$
   • $\text{Mean}(\text{Bin 3})=\frac{26+28+29+34}{4}=\frac{117}{4}=29.25\approx 29$
4. Smoothed Data:
   • $\text{Bin 1}\to [9,9,9,9]$
   • $\text{Bin 2}\to [23,23,23,23]$
   • $\text{Bin 3}\to [29,29,29,29]$

Task B: Partition into 3 Equal-Width Bins

1. Find Min and Max: $A=\min =4$, $B=\max =34$, $N=3$.
2. Calculate Bin Width ($W$): $W=\frac{B-A}{N}=\frac{34-4}{3}=\frac{30}{3}=10$
3. Define Intervals:
   • $\text{Interval 1}:[4,14)$
   • $\text{Interval 2}:[14,24)$
   • $\text{Interval 3}:[24,34]$
4. Allocate items:
   • $\text{Bin 1 }[4,14):[4,8,9]$
   • $\text{Bin 2 }[14,24):[15,21,21]$
   • $\text{Bin 3 }[24,34]:[24,25,26,28,29,34]$

---

Problem 2: Min-Max Normalization

Formula: $v^{\prime }=\frac{v-{\min }_A}{{\max }_A-{\min }_A}\times ({\text{new\_max}}_A-{\text{new\_min}}_A)+{\text{new\_min}}_A$

Scenario:

• Attribute Income ranges from \min = \12,000$to$\max = $98,000$.
• We want to normalize v = \73,600$intotherange$[0.0, 1.0]$.

Calculation: $v^{\prime }=\frac{73,600-12,000}{98,000-12,000}\times (1.0-0.0)+0.0$ $v^{\prime }=\frac{61,600}{86,000}\times 1.0=0.7162\approx 0.716$

---

Problem 3: Z-Score Standardization

Formula: $v^{\prime }=\frac{v-{\mu }_A}{{\sigma }_A}$

Scenario:

• For the same Income attribute, sample mean \mu = \54,000$andstandarddeviation$\sigma = $16,000$.
• Standardize v = \73,600$.

Calculation: $v^{\prime }=\frac{73,600-54,000}{16,000}=\frac{19,600}{16,000}=1.225$ (Interpretation: The value lies $1.225$ standard deviations above the average income).

---

Part 4: Comprehensive Long-Answer / Scenario Questions (6–10 Marks)

Long Answer 1: Data Integration & Redundancy

Question:

A commercial company acquires a competitor and needs to merge their customer databases. Describe the three primary challenges they will encounter during Data Integration, and explain how correlation analysis helps resolve attribute redundancy.

Model Answer:

1. Schema Integration:
   • Problem: Different database systems use different column names for the exact same entity.
   • Example: System A labels customer numbers as cust_id, whereas System B labels them as customer_# or client_code. Resolving metadata mappings is required.
2. Entity Identification Problem:
   • Problem: Recognizing that two slightly different text strings from different sources correspond to the same real-world person or object.
   • Example: "William Clinton" in Database 1 and "Bill Clinton" in Database 2 represent the same customer. String matching, deduplication, and fuzzy matching are needed.
3. Data Value Conflicts:
   • Problem: The same real-world attribute is recorded using different units, currencies, or representations.
   • Example: Heights stored in inches vs. centimeters; weights in pounds (British imperial) vs. kilograms (metric); or prices in USD vs. THB.
4. Handling Redundancy via Correlation Analysis:
   • Redundant attributes slow down model convergence and waste memory.
   • By computing the Pearson correlation coefficient for numeric attributes (or Chi-square for categorical attributes), we identify pairs of attributes with near-perfect correlation (e.g., $r\approx 1.0$).
   • Example: If Annual Income and Income Tax Paid correlate at $0.99$, one is a derivable duplicate of the other. The analyst safely removes one feature to eliminate redundancy.

---

Long Answer 2: Missing Data Strategy Formulation

Question:

You are provided with a 1,000,000-row tabular dataset for loan approval prediction. The target variable is Approved (Yes/No). During inspection, you notice missing values in:

1. Approved (missing in 500 rows)
2. Age (missing in 15% of records)
3. Monthly Income (missing in 3% of records)

Propose and justify an optimal cleaning strategy for each feature.

Model Answer:

• 1. Missing Approved (Target Class Label):
  • Strategy: Ignore the tuple / Listwise deletion (drop the 500 rows).
  • Justification: In supervised classification, imputing target labels introduces false assumptions and heavy model bias. Since 500 rows represent only $0.05\%$ of 1,000,000 records, deleting them causes negligible data loss while preserving label integrity.
• 2. Missing Monthly Income (Numeric feature, 3% missing):
  • Strategy: Class-Conditional Median or Regression / KNN Imputation.
  • Justification: Income distributions are notoriously skewed by high earners. Using the median (or predicting income via regression from Education and Job Title) avoids skewing the data, whereas a global mean would be distorted by outliers.
• 3. Missing Age (Numeric feature, 15% missing):
  • Strategy: KNN Imputation or Predictive Inference (Decision Tree/Bayesian) within a processing pipeline.
  • Justification: 15% missingness is too large for deletion (would discard 150,000 records). Imputing a single constant or overall mean would drastically reduce feature variance. KNN imputation borrows the age of individuals with the most similar income, loan amounts, and credit profiles, preserving underlying multidimensional relationships.

---

Long Answer 3: “Know Your Data” (KYD) / Exploratory Data Analysis Protocol

Question:

Explain what “Know Your Data” (KYD) is, why it must precede model training, and outline the 7 fundamental activities a data scientist must conduct during this phase.

Model Answer:

• Definition & Importance:

  • Know Your Data (KYD) is the systematic process of deeply understanding the structure, quality, distributions, and characteristics of a dataset before applying transformations or machine learning models.
  • Why it matters: Machine learning algorithms assume specific data distributions. Skipping KYD leads to garbage-in/garbage-out results, undetected skewness, model bias, and invalid conclusions drawn from noise.

• The 7 Key Activities of KYD:

  1. Data Overview: Inspect dataset dimensions (number of rows/columns), feature data types (numerical, categorical, datetime, text), and view the first few records (df.head()).
  2. Summary Statistics: Compute measures of central tendency (mean, median, mode), spread (variance, standard deviation, min, max, range), and skewness to see if features are normally distributed.
  3. Missing Value Audit: Quantify the count and percentage of nulls per column and evaluate patterns of missingness (random vs. systematic).
  4. Unique Values & Cardinality: Count distinct values per column to differentiate between low-cardinality categorical features (e.g., Gender) and high-cardinality or continuous identifiers.
  5. Outlier Detection: Use statistical methods ($Z$-scores, IQR thresholds) and visual inspection (box plots) to detect extreme values.
  6. Correlation Analysis: Evaluate pairwise feature dependencies using Pearson/Spearman coefficients (numerical) or Chi-Square tests (categorical) to identify colinear or irrelevant features.
  7. Visualizations: Plot histograms, boxplots, scatter plots, and correlation heatmaps to visualize distributions, clusters, and anomalies.

---