lec2

Key-Value Technical Extraction Table

The following table extracts the keywords, technical nouns, verbs, tools, standards, and concepts found across the text, diagrams, screenshots, and schemas of the lecture.

Category / Domain	Keyword / Technical Term	Grammatical / Functional Type	Contextual Definition & Key-Value Details
Foundations (DIKW)	Data	Noun	Raw facts and figures representing events or objects; exists without context or meaning; forms: text, numbers, symbols.
Foundations (DIKW)	Information	Noun	Processed data structured with context and relationships; reveals meaning. Formula: $\text{Data}+\text{Context}=\text{Information}$.
Foundations (DIKW)	Knowledge	Noun	Synthesized information combined with rules/interpretation; used to make decisions and solve problems. Formula: $\text{Knowledge}=\text{Information}+\text{Rules}$.
Foundations (DIKW)	Wisdom	Noun	Integrated, actionable insight into human behavior and external factors; reveals fundamental principles and optimal actions.
Foundations (DIKW)	Decisions / Action	Noun / Verb	The purposeful direction and business movement executed from derived wisdom (e.g., launching targeted discount campaigns).
Data Provenance	Internal Data	Noun	Data communicated and generated within the boundary of an organization’s own systems or environment.
Data Provenance	External Data	Noun	Data originating outside the enterprise system (e.g., third-party feeds, open data, social platforms).
Data Provenance	Direct Data	Noun	Data collected specifically for the intended primary operational purpose (e.g., Call Detail Records collected for telecom billing).
Data Provenance	Indirect Data	Noun	Secondary data repurposing; data collected for one purpose and analyzed for another (e.g., telecom call records used to model dynamic population mobility).
Data Typology	Qualitative (Categorical)	Noun / Classification	Non-numeric data describing traits, qualities, and groupings.
Data Typology	Nominal Data	Noun	Categorical values with no inherent order (e.g., color, profession, nationality).
Data Typology	Ordinal Data	Noun	Categorical values from an ordered set or ranking (e.g., military rank, academic title, test letter grades).
Data Typology	Quantitative	Noun / Classification	Measurable, numerical values.
Data Typology	Discrete Data	Noun	Countable, finite values that cannot be meaningfully divided (e.g., headcount of students in a classroom, number of pets).
Data Typology	Continuous Data	Noun	Real numbers that can be infinitely subdivided along a continuous scale (e.g., distance, temperature, weight).
File Formats	Structured Files	Noun / Classification	Tabular files with rigid predefined schemas: CSV (comma delimiter), TSV (tab delimiter \t), Microsoft Excel (.xls, .xlsx), Fixed-length text.
File Formats	Fixed-Length Text	Noun / Format	Flat files where every data column starts and ends at explicit character indexes across all lines.
Fixed-Length Specs	VariableWidths	Image Attribute	Column widths defined in MATLAB example: 10 (LastName), 7 (Gender), 4 (Age), 26 (Location), 7 (Height), 7 (weight), 7 (Smoker).
Fixed-Length Specs	DataTypes	Image Attribute	Data types mapped per column: 'char', 'categorical', 'double', 'logical'.
File Formats	Semi-Structured Files	Noun / Classification	Self-describing data with hierarchical tags/keys: JSON, XML, YAML.
File Formats	JSON	Noun / Format	JavaScript Object Notation; lightweight, fast payload transmission over REST APIs; natively supports numbers, strings, objects, and Boolean arrays.
File Formats	XML	Noun / Format	eXtensible Markup Language; tag-based hierarchical structure used with SOAP webservices; supports rich validation, dates, namespaces, and binary attachments.
File Formats	XML Schema (XSD)	Noun / Standard	Structural schema language (xs:schema, xs:element, xs:complexType, xs:sequence, xs:attribute) enforcing strict typing (xs:string, xs:positiveInteger, xs:decimal) and constraints (use="required").
File Formats	YAML	Noun / Format	Human-readable data serialization language commonly used for configuration and semi-structured logs.
File Formats	Unstructured Data	Noun / Classification	Data without a predefined data model: plain text (.txt), system logs, PDFs, Word docs, audio, video, images.
Big Data Storage	Apache Parquet	Noun / Format	Open-source columnar storage file format optimized for fast querying and analytics in Apache Spark.
Big Data Storage	ORC	Noun / Format	Optimized Row Columnar format; high-performance columnar storage designed for Apache Hadoop and Hive workloads.
Big Data Storage	SequenceFile	Noun / Format	Binary flat-file format consisting of key-value pairs; heavily utilized in native Hadoop MapReduce pipelines.
Streaming Sources	Kafka Topics	Noun / Platform	Distributed, append-only log streams for publishing and subscribing to real-time events.
Streaming Sources	Apache Flume	Noun / Platform	Distributed service for collecting, aggregating, and moving large amounts of streaming log data into Hadoop.
Streaming Sources	IoT Streams	Noun	Continuous event streams emitted by telemetry devices and physical sensor networks.
Data Management	Data Management	Noun / Process	The end-to-end discipline of collecting, organizing, protecting, and accessing data assets to drive productivity, efficiency, and business decisions.
Business Value	Productivity	Noun / Metric	Enables workers to locate, interpret, and relay critical operational information without friction.
Business Value	Cost Efficiency	Noun / Metric	Eliminates redundant research, duplicate processing, and unnecessary storage overhead.
Business Value	Operational Nimbleness	Noun / Metric	Allows rapid corporate adaptability to market fluctuations and competitor moves.
Business Value	Security & Risk Mitigation	Noun / Process	Shields corporate data from external cyber-attacks, exfiltration, and unauthorized access.
Business Value	Reduced Data Loss	Noun / Process	Guarantees business continuity through regular backups, disaster recovery, and data durability.
Business Value	Accurate Decisions	Noun / Metric	Provides decision-makers with unified, up-to-date, single-source-of-truth datasets.
Lifecycle	Data Lifecycle Management (DLM)	Noun / Framework	Systematic policy framework that oversees data across all stages of its existence to maximize its useful life.
DLM Phase 1	Acquisition	Verb / Stage	Gathering raw data automatically or manually from internal applications or external APIs.
DLM Phase 2	Storage	Verb / Stage	Persisting ingested data in secure cloud or on-premises repositories with backup policies.
DLM Phase 3	Processing	Verb / Stage	Cleansing, enriching, transforming, normalizing, and performing analytical computation.
DLM Phase 4	Usage	Verb / Stage	Consuming data for business intelligence, machine learning, operational workflows, and decision support.
DLM Phase 5	Disposal	Verb / Stage	Secure deletion, purging, or regulatory archival once retention windows expire.
Data Architecture	Data Architecture	Noun / Blueprint	Defines the structural blueprint of an enterprise’s physical/logical data assets and its end-to-end data pipeline flow.
Architecture Tiers	Data Sources Layer	Architectural Tier	Operational applications (OLTP, ERP, CRM), flat files, cloud APIs (ODATA, REST), and social media feeds.
Architecture Tiers	Operational Data Exchange	Architectural Tier	Staging layer holding raw and uncurated data on unified platforms (SQL or Data Lake).
Architecture Tiers	Modern Data Warehouse	Architectural Tier	Governed, cleansed, enriched, and historical record repositories optimized for enterprise queries.
Architecture Tiers	Semantic Models	Architectural Tier	Domain-specific, governed analytical business logic (e.g., Marketing, Sales, Finance, Manufacturing).
Architecture Tiers	BI & Analytics Layer	Architectural Tier	Self-Service BI (Power BI, Tableau, Qlik) alongside Advanced Analytics (Machine Learning, Data Science).
Data Modeling	Data Model	Noun / Concept	Conceptual abstraction defining how data is organized, stored, and related within distributed storage systems.
Model Traits	Scalability	Noun / Feature	Capability to scale horizontally across multi-node distributed clusters to handle petabyte-scale datasets.
Model Traits	Schema-on-Read	Noun / Pattern	Flexible ingestion where unstructured/semi-structured data is parsed during execution rather than prior to storage.
Model Traits	Schema-on-Write	Noun / Pattern	Rigid relational structure where schema must be predefined before any record is committed to disk.
Model Traits	Performance & Fault Tolerance	Noun / Metric	Architectural optimization for low query latency, parallel processing, and resilience against node failure.
Modeling Types	Relational Model	Noun / Type	Traditional tabular model based on rows and columns with relational keys (e.g., PostgreSQL, Oracle, MySQL).
Modeling Types	Dimensional Model	Noun / Type	De-normalized schema models (Star schema, Snowflake schema) utilized in OLAP and Data Warehouses.
Modeling Types	Document Model	Noun / Type	Stores hierarchical, semi-structured documents (e.g., JSON/BSON in MongoDB, Couchbase).
Modeling Types	Key-Value Store	Noun / Type	Fast lookup database storing values against unique hash keys (e.g., Redis, Amazon DynamoDB).
Modeling Types	Column-Family Store	Noun / Type	NoSQL model storing data partitioned by column families for sparse, wide matrices (e.g., Apache Cassandra, HBase).
Modeling Types	Graph Database	Noun / Type	Stores highly connected networks using explicit nodes, edges, and properties (e.g., Neo4j, social graph analysis).
Modeling Types	Distributed Data Modeling	Noun / Pattern	Partitioning, sharding, and replicating data across clusters while optimizing for data locality.
Data Discovery	Data Catalog	Noun / System	Searchable inventory of enterprise data assets indexed with contextual metadata.
Catalog Metadata	Metadata Fields	Attributes	Dataset name, publisher, maintainer, email, public access level, spatial bounds, program codes, data dictionaries.
Standards	ISO 8601 Accrual Periodicity	Standard / Notation	Standardized recurrence intervals: R/P10Y (Decennial), R/P4Y (Quadrennial), R/P1Y (Annual), R/P2M (Bimonthly), R/P3.5D (Semiweekly), R/P1D (Daily), R/P2W (Biweekly).
Data Integration	ETL / Integration	Verb / Process	Extract, Transform, and Load; unifying disparate data from siloed applications into a coherent data pool.
Integration Tools	Integration Ecosystem	Software Platform	AWS Glue, Apache NiFi, Azure Data Factory, Informatica, SAS, Qlik Replicate, IBM Information Server, Oracle Data Integrator, SAP BusinessObjects, Microsoft SSIS, Talend, Alteryx, RapidMiner.
Cloud Integration	AWS Glue	Cloud Service	Fully managed, serverless cloud ETL engine for scheduling, crawling, transforming, and loading datasets.
AWS Glue Components	Glue Crawlers	Noun / Service	Automated scanners that inspect data stores (S3, RDS, Redshift) to infer schemas and populate the catalog.
AWS Glue Components	Glue Data Catalog	Noun / Service	Centralized metadata repository storing schema definitions, partitions, and Hive DDL statements.
AWS Integration	Amazon Query & Analytics	Ecosystem	Amazon Athena (serverless SQL), Redshift Spectrum (S3 querying), QuickSight (BI), EMR (managed Spark/Hadoop).
Data Science Tools	RapidMiner	Software Platform	Visual workflow platform for ETL, preprocessing, predictive modeling, machine learning, and validation.
RapidMiner Workflow	Process Operators	Verbs / Operators	Visual pipeline steps: Read CSV, Select Attributes, Append, Numerical to Polynominal, Filter Examples, Write CSV.
Data Governance	Master Data Management (MDM)	Noun / Practice	Enterprise-wide discipline ensuring the consistency, accuracy, and stewardship of shared core business entities.
MDM Domains	Master Data Domains	Noun / Entities	Customers (Employees, Patients, Suppliers), Products (BOM, Assets, Equipment), Locations (Branches, Facilities, Stores), Others (Accounts, Contracts, Policies).
MDM Taxonomy	Master Data	Noun / Type	Core shared enterprise nouns that give context to business activities (e.g., Customer, Product, Store).
MDM Taxonomy	Transaction Data	Noun / Type	Ephemeral records representing operational events and transactions (e.g., Order ID, order line items, quantity, timestamp).
MDM Taxonomy	Reference Data	Noun / Type	Static lookup datasets used for categorization and standardization (e.g., Currency codes, Baht-to-USD rates, postal codes).
Data Characteristics	Physical Characteristics	Noun / Attributes	Data footprint/size, regulatory compliance demands, institutional ownership, and sensitivity/classification.
Data Characteristics	Volume	Noun / Metric	Total scale of data managed and processed across active online systems and offline historical archives.
Data Characteristics	Retention Period	Noun / Metric	Legally or operationally mandated duration data must persist before disposal (e.g., 30 days vs. 5 years).
Data Characteristics	Transaction Boundaries	Noun / Concept	Scopes of atomic work that dictate when and where application state and consistency must be committed.
System Latency	Real-Time Performance	Noun / Latency	Sub-second latency; processing and responding in under 500 milliseconds (e.g., web response targets).
System Latency	Near Real-Time	Noun / Latency	Latency spanning one to two seconds; human-perceived instantaneous response.
System Latency	Delayed / Batch Time	Noun / Latency	Latency spanning seconds, hours, or scheduled recurring intervals (e.g., daily, weekly, monthly batches).
Data Volatility	Volatility	Noun / Concept	The frequency and rate at which data values change or overwrite over time.
Data Volatility	Static Data	Noun / Type	Immutable, chronological, event-driven data adhering to Write-Once Read-Many (WORM); consumes terabytes (e.g., web logs, POS logs, bank ledgers).
Data Volatility	Dynamic Data	Noun / Type	Continually mutating, state-oriented data managed in normalized RDBMS; requires strict ACID compliance and frequent read/write operations.
Future Topics	Storage & Infrastructure	Architecture Topics	Cloud Storage, Ephemeral vs. Persistent Storage, SAN (Storage Area Network), NAS (Network-Attached Storage), RAID arrays, Data Security, Data Governance, and Data Pipelines.

---

Comprehensive Lecture Summary

1. The Epistemological Hierarchy: DIKW & Decision Making

The lecture establishes the hierarchical continuum connecting raw inputs to executive action:

• Data: Unprocessed, objective signals and raw figures that exist without surrounding context (e.g., uncontextualized strings and numbers such as "M", "12", "211092").
• Information: Data infused with contextual meaning and associative relationships (e.g., mapping "M" as Gender, "12" as Age, and "211092" as Birthdate).
• Knowledge: The synthesis of structured information with analytical frameworks, business rules, and cognitive evaluation to identify patterns and understand why events occur.
• Wisdom & Decision: Actionable insight that anticipates future outcomes and identifies optimal principles (what is best), culminating in purposeful execution (what action to take).

In a retail analytics scenario, this flows from transaction logs (Data) $\to$ sales spike correlation with rainy days (Information) $\to$ combining weather forecasts with promotion response rates (Knowledge) $\to$ anticipating customer purchasing habits during storms (Wisdom) $\to$ automatically dispatching umbrella discount coupons to mobile devices in rain-affected regions (Decision).

2. Typologies, Taxonomies, and Origins of Data

• Data Provenance: Data is segmented into Internal vs. External (based on whether it originates within internal corporate networks or outside environments) and Direct vs. Indirect (based on primary purpose of collection versus secondary analytical reuse, such as mining cellular call billing logs to determine dynamic urban population densities).
• Statistical Classifications:
  • Qualitative (Categorical): Nominal (unordered sets, e.g., nationalities, professions) and Ordinal (ranked scales, e.g., academic ranks, performance tiers).
  • Quantitative (Numerical): Discrete (countable integers, non-divisible, e.g., headcount) and Continuous (infinitely divisible real values, e.g., distances, weights).

3. Data File Formats and Serialization

The technical mechanisms for storing data encompass four major categories:

• Structured Formats: Delimited files (CSV using commas, TSV using tab separators \t), spreadsheets, and fixed-length flat files (where columns are strictly partitioned by character offsets, explicit field widths, and strong data typing: character, numeric double, categorical, logical).
• Semi-Structured Formats: Self-describing, human-readable data:
  • JSON: Lightweight, minimal syntax optimized for high-throughput web transmission via RESTful APIs.
  • XML: Schema-enforced hierarchic markup used in enterprise SOAP architectures. It is strictly validated using XML Schemas (XSD), which enforce sequences, elements, attributes, nullability, and explicit data types (xs:string, xs:positiveInteger, xs:decimal).
  • YAML: Clean, indentation-delimited format suited for configuration.
• Unstructured Formats: Natural language documents, server log files, audio, video, and image binaries.
• Big Data Analytical Formats: Distributed systems use specialized formats including Apache Parquet (column-oriented storage engine for Apache Spark), ORC (Optimized Row Columnar, built for Hadoop and Hive performance), and SequenceFiles (binary key-value serializations for MapReduce).

4. Enterprise Data Architecture and Management

• Core Value Drivers: Data Management encompasses the institutional disciplines required to acquire, organize, govern, and utilize data. It drives six operational benefits: increasing employee productivity, reducing cost overhead by eliminating duplicated research, enabling market nimbleness, minimizing security vulnerabilities, preventing catastrophic data loss via backup policies, and ensuring accurate, data-driven decisions.
• Architectural Blueprints: Multi-tiered data pipelines unify ingestion and consumption:
  1. Data Sources: Disparate transactional engines (OLTP, ERP, CRM), flat files, and web APIs.
  2. Operational Data Exchange: Central landing zone utilizing SQL engines or massive distributed Data Lakes for raw staging.
  3. Modern Data Warehouse: Cleaned, normalized, governed, and enriched historical tables.
  4. Semantic Models: Governed business logic layered for functional domains (Sales, Marketing, HR, Finance).
  5. Analytics & Consumption: Dashboards (Power BI, Tableau, Qlik) and advanced statistical computing (Machine Learning, predictive models).

5. Data Lifecycle Management (DLM) and Cataloging

• The DLM Framework: Maximizes the business utility of data across five sequential phases:
  1. Acquisition: Manual or programmatic ingestion from internal/external channels.
  2. Storage: Cloud-based or on-premises persistence with automated backup policies.
  3. Processing: Data cleansing, normalization, deduplication, and transformation.
  4. Usage: Operational analytics, dashboarding, and decision-making.
  5. Disposal: Secure purging or regulatory cold archival once retention expires.
• Data Catalogs: Centralized metadata inventories that index corporate datasets to ensure searchability, compliance, and transparency. Datasets are tagged with governance metadata (licensing, maintainer emails, unique identifiers, spatial scopes) and update intervals structured around the ISO 8601 Accrual Periodicity standard (e.g., R/P1Y for Annual, R/P1D for Daily, R/P2W for Biweekly).

6. Distributed Data Modeling Paradigms

Data models must satisfy scalability (horizontal node scaling for petabytes), flexibility (schema-on-read vs. schema-on-write), and performance:

• Relational: Standard tables with strict relational integrity, common in transactional systems.
• Dimensional: Star and snowflake schemas optimized for fast OLAP aggregations in data warehouses.
• Document: Hierarchical collections (MongoDB, Couchbase) mapping directly to JSON application objects.
• Key-Value: High-performance, low-latency hash maps (Redis, DynamoDB).
• Column-Family: Wide-column models (Cassandra, HBase) optimized for massive distributed writes.
• Graph: Edge-node graph structures (Neo4j) tracking interconnected networks and relationships.
• Distributed Modeling Needs: Multi-node systems demand deliberate partition keys, data locality optimization, cross-node replication, and built-in fault tolerance.

7. Data Integration, Cloud ETL, and Analytics

• Integration Engines: Overcome enterprise data silos through coordinated ETL pipelines. The modern stack includes tools like Informatica, Talend, Apache NiFi, Azure Data Factory, SSIS, and AWS Glue.
• Serverless Ingestion (AWS Glue): Employs automated Glue Crawlers to infer schema definitions from raw objects stored across S3, RDS, or Redshift, synchronizing them into the Glue Data Catalog using Hive DDL definitions. These catalogs are immediately queried by downstream engines like Amazon Athena, Redshift Spectrum, EMR, and QuickSight.
• Visual Data Prep & ML Pipelines: Tools such as RapidMiner operationalize workflows through directed acyclic graphs (DAGs) of operators—from ingestion (Read CSV) to transformation (Select Attributes, Append, Numerical to Polynominal, Filter Examples) and eventual persistence (Write CSV).

8. Master Data Management (MDM)

MDM establishes a single, verified master data service across core corporate domains:

• Master Data: The foundational, persistent nouns of an organization (Customers, Products, Stores, Facilities, Employees).
• Transaction Data: The transient, event-driven records representing operational activity linking master entities together (e.g., an Order capturing that Customer A bought Product B at Store C for 300 Baht).
• Reference Data: Static reference sets used for categorization and conversion (e.g., lookup tables for state codes, ISO country codes, currency conversions).

9. Operational Data Characteristics: Latency and Volatility

• Latency Spectrum:
  • Real-time: Sub-second response ($\le 500\text{ ms}$).
  • Near real-time: 1–2 second perceived immediacy.
  • Delayed/Batch: Scheduled recurring execution (daily, weekly, monthly cycles).
• Volatility Classifications:
  • Static Data: Append-only, chronological event records following a Write-Once, Read-Many (WORM) pattern (e.g., web logs, POS logs, bank debit histories, stock trading streams); grows rapidly into terabytes.
  • Dynamic Data: Continually mutating, state-driven data stored in normalized RDBMS that must enforce continuous data integrity via frequent atomic read/write operations.