Knowledge Graph — Coursera Notes › Academic disciplines › Information Technology / Computer Science › Artificial Intelligence › Machine Learning & Data

Data preprocessing

concept · part of Machine Learning & Data

The step of transforming raw data into a format suitable for machine learning, including scaling and encoding.

Feature values are scaled to a range between 0 and 1 using MinMaxScaler to ensure optimal neural network performance. The dataset is split into 80% training and 20% testing sets.

• Removing noise: punctuation, special characters, emojis (depending on use case)
• Standardizing text: converting to lowercase, removing irrelevant characters
• Handling stop words: common words like 'the', 'is', 'in', 'and', 'of', 'to' that carry little meaning and are often removed to focus on content-rich words.

Missing values can be handled by imputation, e.g., filling with median for numerical columns. Duplicate entries should be removed to ensure data consistency.

for column in df.select_dtypes(include=['float64', 'int64']).columns:
 df[column].fillna(df[column].median(), inplace=True)
df.drop_duplicates(inplace=True)

Standardization scales numerical features to have zero mean and unit variance using StandardScaler from scikit-learn. This prevents any single feature from dominating the model.

scaler = StandardScaler()
numeric_features = df.select_dtypes(include=['float64', 'int64']).columns
df[numeric_features] = scaler.fit_transform(df[numeric_features])

Categorical features are converted into numerical format using one-hot encoding via pd.get_dummies(df, drop_first=True). This ensures compatibility with machine learning algorithms.

df = pd.get_dummies(df, drop_first=True)

Outliers can be detected using Z-score analysis. Data points with Z-score >= 3 are considered outliers and removed to maintain dataset integrity.

z_scores = np.abs(stats.zscore(df.select_dtypes(include=['float64', 'int64'])))
df = df[(z_scores < 3).all(axis=1)]

Logarithmic transformation (e.g., np.log1p) is applied to reduce skewness in skewed features, making the data more suitable for modeling.

df['income_log'] = np.log1p(df['income'])

The dataset is split into training and testing sets using train_test_split from scikit-learn, typically with test_size=0.2 and random_state=42 for reproducibility. This ensures unbiased evaluation of model performance.

X = df.drop('target', axis=1)
y = df['target']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

Inside Data preprocessing (10)

• Data cleaning — The process of handling missing values, outliers, and inconsistencies in a dataset.
• Outliers — Data points that deviate significantly from other observations, identified via Z-score or IQR.
• CSV — CSV is a common file format for storing tabular data, often used in data preprocessing and analysis.
• Data normalization — TensorFlow normalizes data by dividing pixel values by 255.0 to get values in [0,1].
• fit — A method that learns preprocessing parameters from data, e.g., LabelEncoder learns the mapping from original to encoded labels.
• MinMaxScaler — A scaling method that transforms features to a fixed range, typically [0, 1].
• Missing values — Data points that are absent in a dataset, handled by removal or imputation.
• Normalization — Scaling data to a standard range, including Min-Max scaling and Z-score standardization.
• One-hot encoding — A technique to convert categorical variables into binary columns for machine learning models.
• transform — A method that applies a learned transformation to a DataFrame, returning a prepared DataFrame.

Connections

• Uses One-hot encoding
• Uses Scikit-learn
• Prerequisite of Amazon Neptune ML
• Related to Missing values
• Related to Outliers
• Related to Normalization
• Related to Data transformation

This is the text view of an interactive 3D knowledge graph — open this page with JavaScript enabled to explore it visually.