14. Data Analytics: From Raw Data to Statistical Insight

Analyzing quantitative data is a multi-step process. Raw data cannot simply be fed into statistical formulas. Researchers must first explore, clean, and understand their dataset before choosing the correct statistical tools to apply.

---

Step 1: Exploring, Organizing, and Preprocessing Data

Before any formula is applied, raw data — which initially arrives as a disorganized mass of surveys, logs, or sensor readings — must be structured and cleaned. Statistical algorithms blindly calculate whatever data they receive ("garbage in, garbage out"), so this phase is critical.

1.1 Exploring and Organizing Datasets

• Data Coding: Converting responses into machine-readable format. For example, converting "Male" and "Female" into 0 and 1, or grading a Likert scale from "Strongly Disagree" (1) to "Strongly Agree" (5). Qualitative data may be grouped into overarching themes.
• Data Entry & Tabulation: Structuring coded data into a grid (Excel spreadsheet, CSV, or SPSS data matrix) where:

- Each row represents a single participant, observation, or transaction.

- Each column represents a specific variable or feature (e.g., Age, Test Score).

• Exploratory Data Analysis (EDA): Using visual and quantitative tools to understand the data's shape before applying strict formulas.

- Visualization: Histograms, scatter plots, box plots, and heatmaps highlight anomalies, distributions, and obvious trends.

- Initial Descriptive Stats: Calculating mean, median, and standard deviation to get a quick first impression.

1.2 Data Preprocessing (Cleaning the Data)

Data preprocessing prevents skewed or biased results by resolving inconsistencies in the dataset.

• Handling Missing Values: Researchers must decide:

- Deletion: Removing the entire participant row (listwise deletion) or just the missing field (pairwise deletion). Reduces dataset size.

- Imputation: Estimating the missing value using averages (mean/median imputation) or advanced predictive algorithms (e.g., K-Nearest Neighbors).

• Identifying and Handling Outliers: An outlier is a data point that drastically deviates from the norm (e.g., a recorded age of 145). Outliers must be investigated — they can be removed, capped, or transformed depending on whether they are errors or genuine data.
• Data Transformation and Discretization: Heavily skewed data may require a log transformation to normalize the distribution, making it suitable for standard statistical tests. Discretization converts continuous data (e.g., Exact Age) into categorical bins (e.g., Age Groups: 18–25, 26–35).

1.3 Data Scaling and Reduction

• Feature Scaling: Bringing all variables into a comparable range so no single variable dominates the analysis.

- Normalization: Scales data to a [0, 1] range.

- Standardization: Centers data around a mean of 0 with a standard deviation of 1.

• Data Reduction: Reducing the volume of data while preserving its analytical value. Principal Component Analysis (PCA) combines highly correlated features into fewer dimensions, making analysis faster and less noisy.

---

Step 2: Evaluating Data Characteristics

Once the data is clean, researchers classify what kind of data they are holding before choosing a formula:

• Continuous vs. Discrete Data: Is the data measured on an infinitely divisible scale (continuous, like time or weight), or counted in distinct whole numbers (discrete, like the number of people in a room)?
• Scale of Measurement: Data generally falls into one of four scales, each permitting different mathematical operations:

- Nominal: Basic categories without order (e.g., Male/Female, Red/Blue).

- Ordinal: Categories with a strict ranking, but unequal or unclear intervals (e.g., 1st place, 2nd place).

- Interval: Ordered ranks with equal distances between them, but no true zero (e.g., Celsius temperature).

- Ratio: All the above, plus an absolute true zero (e.g., physical distance, money, weight).

---

Step 3: Choosing the Goal of the Algorithm

Depending on the research subproblem, researchers choose from specific branches of statistics.

Descriptive Statistics

Used strictly to summarize and describe the data at hand.

• Calculating Central Tendency: Finding the mathematical "middle" of the dataset.

- Mean (\mu or \bar{x}): \frac{\Sigma x_i}{N} — Sum of all values divided by total number of values.

- Median: The exact middle value when the data is sorted in order.

- Mode: The most frequently occurring value in the dataset.

• Calculating Variability: Discovering how spread out the data is from the center.

- Range: Max - Min

- Variance (\sigma^2): \frac{\Sigma (x_i - \mu)^2}{N} — The average of squared differences from the Mean.

- Standard Deviation (\sigma): \sqrt{\sigma^2} = \sqrt{\frac{\Sigma (x_i - \mu)^2}{N}} — The spread in the same unit as the original data.

• Calculating Correlation: Measuring the degree of association between two variables.

- Pearson's Correlation Coefficient (r): r = \frac{\Sigma (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\Sigma (x_i - \bar{x})^2 \Sigma (y_i - \bar{y})^2}} — Measures linear correlation between -1 and +1.

Inferential Statistics

Used to make predictions or decisions that extend beyond the immediate dataset.

• Estimating Population Parameters: Using a small representative sample to infer values for an entire population.

- Confidence Interval (for mean): \bar{x} \pm Z \frac{\sigma}{\sqrt{n}} — Calculates the margin of error around the true population mean.

• Testing a Null Hypothesis: Calculating whether there is a statistically significant relationship between variables.

- Z-Test / T-Test: t = \frac{\bar{x} - \mu}{\frac{s}{\sqrt{n}}} — Determines if there is a significant difference between the means of two groups.

---