Data Splitting & Preprocessing in Python | ML Preprocessing Course | DataSoSi
Email: info@datasosi.com
Home » Python 4 » Data Splitting & Preprocessing
Python IV: Data Preprocessing
Python 4 · Machine Learning Preparation

Data Splitting & Preprocessing for Machine Learning

Learn the workflow that makes machine-learning results trustworthy. Build clean, leakage-safe training and test data before model building begins.

8 units · 8 individual unit capstones · 2 end-to-end capstones · Google Colab/Anaconda · interactive final capstone · CiAI certificate

Original price was: USD$50.00.Current price is: USD$15.99. /month

Self-paced online access · Python + scikit-learn · project-based learning

Enroll in Python IV: Data Preprocessing
View curriculum ↓

Designed for learners who understand Python data analysis and EDA and are ready to prepare data correctly for machine learning.

â‘ 

Leakage-Safe Thinking

Learn why splitting comes first and why every learned preprocessing decision must come from training data only.

â‘¡

8 Unit Capstones

Complete one focused capstone for every unit so each major preprocessing decision is practised independently.

â‘¢

Notebook Capstone

Complete an end-to-end loan-default preprocessing assessment in Google Colab or Anaconda/Jupyter.

â‘£

Final Interactive Capstone

Bring the complete eight-unit workflow together in a guided DataSoSi project environment.

Learn Preprocessing as a Machine Learning Workflow

Machine learning models do not learn from raw data. They learn from data that has been split, cleaned, encoded, transformed and prepared in the correct order. The course teaches that order—and the reasoning behind it—so learners understand when a preprocessing step belongs, why it is necessary, and how to apply it without leaking information from the test set.

The rule that runs through the entire course

Any preprocessing step that learns from data must be fit on the training data only, then applied to the test set. The test set represents future unseen data and must not influence preprocessing decisions.

1. Define X + ySeparate predictors and target
2. Split firstTrain/test + stratification
3. Preprocess trainImpute · encode · select · scale
4. Evaluate honestlyCross-validate then open test once

Core preprocessing skills

  • Detect and prevent data leakage
  • Define features and target variables
  • Classify numerical, categorical, ordinal and boolean data
  • Perform stratified train-test splitting
  • Handle missing data with appropriate imputation
  • Encode nominal and ordinal categories
  • Create useful engineered features

Machine-learning preparation skills

  • Remove near-constant and redundant features
  • Rank features with statistical and model-based methods
  • Understand feature extraction and PCA
  • Measure and address class imbalance
  • Use class weights, oversampling and SMOTE appropriately
  • Select and fit scalers correctly
  • Use StratifiedKFold and sklearn Pipeline

Data Splitting & Preprocessing — 8 Units

One workflow runs through the entire course. Each unit adds another step until learners can build a complete, reproducible, leakage-safe preprocessing pipeline.

Understand data leakage, how test-set information can contaminate training decisions, and why train/validation/test separation is fundamental to reliable machine learning.

  • What leakage is and why it matters
  • Train, validation and test roles
  • Common leakage patterns
  • Why splitting must happen before preprocessing

Separate the feature matrix from the target, classify variable types correctly, and recognize numeric traps where a number should actually be treated as a category.

  • Create X and y
  • Numerical vs categorical variables
  • Ordinal and boolean variables
  • Numeric traps and data inspection

Perform reproducible train-test splits, use stratification when the target is imbalanced, and verify that class balance is preserved in both sets.

  • train_test_split()
  • test_size and random_state
  • stratify=y
  • Verify split sizes and class balance

Detect missing values, choose an imputation strategy, compare simple and advanced approaches, and fit imputers on training data only.

  • Mean, median and mode imputation
  • SimpleImputer
  • KNNImputer
  • MICE / iterative imputation
  • fit_transform on train, transform on test

Encode nominal, ordinal and boolean data appropriately, avoid leakage in target-based encoding, and engineer new variables such as ratios, flags and transformations.

  • One-hot encoding
  • Ordinal / label encoding
  • Leave-one-out target encoding
  • Boolean conversion
  • Feature engineering with ratios and flags

Remove weak or redundant features, rank predictors using several methods, and understand when feature extraction such as PCA is useful.

  • VarianceThreshold
  • Correlation filtering
  • SelectKBest
  • Lasso regularisation
  • Random Forest feature importance
  • Principal Component Analysis (PCA)

Understand why accuracy can mislead on imbalanced targets and compare class weighting, oversampling and synthetic resampling techniques.

  • Class distribution and class ratios
  • class_weight='balanced'
  • RandomOverSampler
  • SMOTE and SMOTE+ENN
  • Tomek Links and SMOTETomek
  • Resample training data only

Choose the right scaler, evaluate stability with stratified cross-validation, and combine the full preprocessing sequence into a reproducible sklearn Pipeline.

  • StandardScaler
  • MinMaxScaler
  • RobustScaler
  • StratifiedKFold
  • cross_val_score
  • End-to-end sklearn Pipeline

8 Individual Unit Capstones

Every unit ends with its own applied capstone. Learners practise one major decision at a time before combining all eight stages in the two end-to-end capstones.

Unit Capstone 1

Spot Data Leakage

Audit a workflow, identify leakage risks and explain why splitting must happen before learned preprocessing steps.

  • Leakage diagnosis
  • Train/test boundaries
  • Reasoning before code
Unit Capstone 2

Define Features & Target

Separate X and y, classify feature types and identify variables that should not be treated according to their apparent storage type.

  • Feature matrix
  • Target definition
  • Column-type map
Unit Capstone 3

Split & Verify

Create a reproducible stratified train-test split and verify that dataset size and target balance are preserved.

  • 80/20 split
  • Stratification
  • Class-balance checks
Unit Capstone 4

Impute Missing Data

Diagnose missingness, choose an appropriate strategy and fit the imputer on training data without contaminating the test set.

  • Missing-value audit
  • Median/mean/mode choices
  • Leakage-safe fit/transform
Unit Capstone 5

Encode & Engineer

Convert categorical and boolean variables into model-ready form and engineer a new feature that captures useful signal.

  • One-hot encoding
  • Ordinal handling
  • Feature engineering
Unit Capstone 6

Select Useful Features

Compare variance, correlation and model-based importance methods to decide which predictors should remain.

  • Variance filtering
  • Correlation checks
  • Feature importance
Unit Capstone 7

Handle Class Imbalance

Measure the minority-class problem and choose between class weighting, oversampling or SMOTE without changing the test set.

  • Class ratio
  • Resampling choices
  • Train-only balancing
Unit Capstone 8

Scale & Cross-Validate

Choose a scaler, run stratified cross-validation and demonstrate a repeatable end-to-end preprocessing workflow.

  • Scaler selection
  • StratifiedKFold
  • Pipeline thinking

2 End-to-End Capstones

After the eight focused unit capstones, learners complete two larger assessments: one independently in a notebook environment and one as the final integrated DataSoSi project.

Capstone 1 · Google Colab or Anaconda/Jupyter

Loan Default Prediction — End-to-End Preprocessing Assessment

Work independently with a new loan-default dataset and apply the entire preprocessing workflow from leakage audit through cross-validation and professional reporting.

200 applicationsNew capstone dataset
27% defaultIntentionally imbalanced target
8 sectionsUnits 1–8 integrated
NotebookGoogle Colab or Anaconda/Jupyter
  • Detect the leakage column LoanApprovedDate
  • Identify the RiskRating numeric trap
  • Split with stratification before preprocessing
  • Impute missing income safely
  • Encode categorical and boolean variables
  • Engineer a Loan-to-Income feature
  • Run feature-selection checks and handle imbalance
  • Scale, cross-validate and write a 400–600 word decision report
Capstone 2 · Final Interactive DataSoSi Capstone

Build the Complete Leakage-Safe Preprocessing Workflow

Complete the full sequence again in the interactive project environment. The final capstone reinforces not just which Python commands to use, but the order of operations and the reasoning behind each decision.

8 unitsFull course integrated
1 workflowRaw data to model-ready data
VerificationCheck each major decision
AI TutorHints, explanations and debugging support
  • Separate X and y and protect the holdout set
  • Handle missing values, encoding and feature engineering
  • Select features and address class imbalance
  • Scale numerical variables correctly
  • Use cross-validation before touching the test set
  • Explain why each preprocessing choice is leakage-safe

Tools You Will Use

PythonCore programming language
PandasData inspection and preparation
NumPyNumerical operations
scikit-learnPreprocessing, CV and pipelines
imbalanced-learnSMOTE and resampling
Colab / AnacondaNotebook capstone environment

Continue Your Python Data Science Path

Once your data is split and preprocessed correctly, the next step is to build models, evaluate how well they perform and interpret what the results mean.

Next Course

Building, Evaluating and Interpreting Models

Move from model-ready data into model training, performance evaluation, comparison and interpretation.

Explore Building, Evaluating and Interpreting Models →

Preprocessing is the bridge between EDA and modelling

Exploratory analysis helps you understand the data. Preprocessing converts those findings into a reproducible, leakage-safe modelling dataset. The next course uses that prepared data to build and evaluate machine-learning models.

Data Splitting & Preprocessing — Course Details

Format100% self-paced, online
PriceUS$15.99/month
Instructional curriculum8 units
Individual unit capstones8 — one for each unit
End-to-end capstones2
Total applied assessments10 capstone/project assessments
Notebook capstoneGoogle Colab or Anaconda/Jupyter · Loan Default
Final capstoneInteractive DataSoSi preprocessing workflow
Core toolsPython, Pandas, NumPy, scikit-learn, imbalanced-learn
CertificateCanadian Institute of Artificial Intelligence (CiAI)

Frequently Asked Questions

You will learn how to move from a raw dataset to clean, correctly split and model-ready training and test data. The course covers leakage prevention, X/y definition, stratified splitting, missing-data imputation, encoding, feature engineering, feature selection and extraction, class imbalance, scaling, cross-validation and sklearn pipelines.

The course includes 8 individual unit capstones, one for each unit, plus 2 end-to-end capstones. That gives learners 10 substantial applied assessments across the course.

The notebook capstone uses a 200-application loan-default dataset with a 27% default rate. Learners work independently through eight sections covering leakage, data splitting, missing data, encoding, feature engineering, feature selection, imbalance handling, scaling, cross-validation and a written decision report.

The final DataSoSi capstone integrates the entire eight-unit workflow in one interactive project. Learners move from feature/target definition and splitting through preprocessing, imbalance handling, scaling and validation while explaining why each decision is correct and leakage-safe.

Leakage allows information from outside the training process to influence the model and can produce unrealistically strong performance. The course therefore reinforces the rule that learned preprocessing decisions must be fit on training data and only then applied to the test set.

No. The course emphasizes understanding the workflow and the reasoning behind each decision. Learners should understand when a technique belongs, why it is necessary and how to apply it safely; exact syntax can be supported by documentation and AI coding tools.

Continue with Building, Evaluating and Interpreting Models, where you use prepared datasets to train, compare, evaluate and interpret machine-learning models.

The certificate of completion is issued by the Canadian Institute of Artificial Intelligence (CiAI).

Meet the Creator of the Course

Learn from a senior scholar and practitioner working across machine learning, AI, analytics and applied decision problems.

Dr. Mark Lokanan

Dr. Mark Lokanan

Professor · Senior Machine Learning Engineer at Vedia · Senior Machine Learning & AI Scholar

Dr. Mark Lokanan is a Professor at Royal Roads University, a Senior Machine Learning Engineer at Vedia, and a senior machine-learning and AI scholar. His work applies machine learning, deep learning and artificial intelligence to fraud, financial crime, analytics and complex real-world decision problems. His teaching focuses on making Python, data science and AI practical and accessible to professionals and adult learners.

Royal Roads University Profile →Vedia →
© 2026 DataSoSi | Canadian Institute of Artificial Intelligence (CiAI). All Rights Reserved. · Facebook · YouTube
Data Splitting & Preprocessing Assistant
Ask me about the 8 units, 8 unit capstones, 2 end-to-end capstones, Google Colab/Anaconda, leakage-safe preprocessing, certificate, creator, or next course.