Foundations of Data Science unit V

 

UNIT-V

Data Cleaning and Preparation

Explain missing data and methods to handle missing values in Pandas?

Handling Missing Data:

• In real-world datasets, some values may be missing (NaN / None).
• Pandas provides several tools to detect, remove, or fill missing data.

1. Detecting Missing Data:

• isnull() → Detects missing values (gives True/False).
• notnull() → Opposite of isnull() (non-missing).

Example:

import pandas as pd
import numpy as np

data = {"Name": ["Ravi", "Sita", "John", "Ravi"],
        "Age": [20, 25, 22, 20],
        "Marks": [None, 50, 60, None]}

df = pd.DataFrame(data)

print(df.isnull())
print(df.notnull())

---

2. Removing Missing Data

dropna() → Removes rows or columns with missing values.

Example:

# Remove rows with missing values
print(df.dropna())

# Remove columns with missing values
print(df.dropna(axis=1))

---

3. Filling Missing Data

fillna(value) → Replace missing values with a fixed value.

method='bfill' → Forward fill (previous value is carried forward).

method='ffill' → Backward fill (next value is used).

fillna(df.mean()) → Replace with column mean/median.

Example:

# Replace missing Age with 0
print(df.fillna(0))

# Forward fill
print(df.fillna(method="ffill"))

# Replace with mean of column
df["Age"] = df["Age"].fillna(df["Age"].mean())

---

4. Checking for Any Missing Values

print(df.isnull().sum())   # Count missing values in each column

---

Explain Data Transformation ?

Removing Duplicates

When working with large datasets, the same row or value may appear more than once. Pandas provides simple methods to detect and remove duplicate data.

1. Detecting Duplicates

• duplicated() → Checks whether a row is a duplicate of a previous one.
• Returns True/False for each row.

Example:

import pandas as pd

data = {"Name": ["Ravi", "Sita", "John", "Ravi"],
        "Age": [20, 25, 22, 20]}

df = pd.DataFrame(data)

print(df.duplicated())

Output:

0    False
1    False
2    False
3     True    # duplicate of first row

2. Removing Duplicates

drop_duplicates() → Removes duplicate rows.

Example:

print(df.drop_duplicates())

3. Removing Duplicates Based on Specific Columns

• Keep only unique values for given columns.

Example:

# Remove duplicates only based on "Name"
print(df.drop_duplicates(subset=["Name"]))

4. Keeping First or Last Duplicates

By default, drop_duplicates() keeps the first occurrence.

Use keep="last" to keep the last occurrence.

Use keep=False to drop all duplicates.

Example:

print(df.drop_duplicates(keep="first"))  # keep first (default)

print(df.drop_duplicates(keep="last"))   # keep last

print(df.drop_duplicates(keep=False))    # drop all duplicates

---

Explain Transforming Data Using a Function or Mapping ?

When working with data, sometimes we need to transform values into a new form. In Pandas, this can be done using:

• Functions
• Mapping
• replace()

Functions

Mapping (Dictionary or Series)

1. Using a Function

We can apply a function to transform data.

Example:

import pandas as pd

data = {"Name": ["Ravi", "Sita", "John"],
        "Salary": [25000, 30000, 40000]}

df = pd.DataFrame(data)

# Increase salary by 10%
df["Salary"] = df["Salary"].apply(lambda x: x * 1.10)

print(df)

Output:

   Name   Salary
0  Ravi   27500.0
1  Sita   33000.0
2  John   44000.0

Here, the function increased all salaries by 10%.

2. Using Mapping (Dictionary or Series)

We can map values from one set to another.

Example:

# Dictionary Mapping
dept_map = {"HR": "Human Resource",
            "IT": "Information Technology",
            "Finance": "Finance"}

df["Dept"] = df["Dept"].map(dept_map)
print(df)

Here, names were mapped to departments using a dictionary.

3. Using replace() for Transformation

Another way to transform values is replace().

Example:

df["Dept"] = df["Dept"].replace(
    {"HR": "Human Resource",
     "IT": "Information Technology"}
)

print(df)

Output:

     Name             Dept
0    Ravi    Human Resource
1    Sita    Information Technology
2    John    Finance

---

Write a Replacing Values in Pandas ?

When working with data, sometimes we need to replace old values with new ones. In Pandas, this can be done using the replace() function.

1. Basic Value Replacement

We can replace a single value with another.

Example:

import pandas as pd

data = {"Name": ["Ravi", "Sita", "John"],
        "Dept": ["HR", "IT", "IT"]}

df = pd.DataFrame(data)

# Replace "IT" with "Information Technology"
df["Dept"] = df["Dept"].replace(
    {"IT": "Information Technology"}
)

print(df)

2. Replacing Multiple Values

We can replace many values at once using a dictionary.

df["Dept"] = df["Dept"].replace(
    {"HR": "Human Resource",
     "IT": "IT Dept"}
)

print(df)

3. Replacing with Lists

We can also replace multiple values with new ones using lists.

Example:

df["Name"] = df["Name"].replace(
    ["Ravi", "Sita"],
    ["Ravindra", "Seeta"]
)

print(df)

Output:

   Name             Dept
0  Ravindra     Human Resource
1  Seeta        IT Dept
2  John         IT Dept

4. Replacing Using Regular Expressions

If values follow a pattern, we can use regex.

df["Dept"] = df["Dept"].replace(
    to_replace="IT.*",
    value="Information Tech",
    regex=True
)

print(df)

Output:

     Name             Dept
0    Ravindra     Human Resource
1    Seeta        Information Tech
2    John         Information Tech

---

What are the Detecting and Filtering Outliers and Explain?10M                                                                                             IMP

Outliers:

• Outliers are values that are very different from the rest of the data.
• They can be too high or too low compared to the majority.
• Outliers may occur due to errors (wrong data entry) or natural variations.

Example

If most students scored between 40–80, but one student scored 5 or 100, these are outliers.

---

1. Detecting Outliers

(a) Using Mean and Standard Deviation

If a value is too far from the mean, it can be considered an outlier.

import pandas as pd

data = pd.DataFrame({
    "Marks": [45, 50, 48, 47, 100, 49, 46, 5]
})

mean = data["Marks"].mean()
std = data["Marks"].std()

# Find outliers: values more than 2 std away from mean
outliers = data[
    (data["Marks"] < mean - 2*std) |
    (data["Marks"] > mean + 2*std)
]

print("Outliers:\n", outliers)

Output:

Outliers:
   Marks
4    100
7      5

(b) Using Interquartile Range (IQR)

• IQR = Q3 − Q1
• Outliers are values below (Q1 - 1.5*IQR) or above (Q3 + 1.5*IQR).

Q1 = data["Marks"].quantile(0.25)
Q3 = data["Marks"].quantile(0.75)

IQR = Q3 - Q1

outliers = data[
    (data["Marks"] < (Q1 - 1.5*IQR)) |
    (data["Marks"] > (Q3 + 1.5*IQR))
]

print("Outliers:\n", outliers)

---

2. Filtering Outliers

Once detected, we can remove them from the dataset.

# Remove outliers using IQR method
filtered_data = data[
    (data["Marks"] >= (Q1 - 1.5*IQR)) &
    (data["Marks"] <= (Q3 + 1.5*IQR))
]

print("Filtered Data:\n", filtered_data)

Output:

Filtered Data:
   Marks
0     45
1     50
2     48
3     47
4     49
5     46

---

3. Handling Outliers

• Instead of deleting, sometimes we:
  • Cap values at the minimum and maximum allowed values.
  • Transform data → Apply log or square root to reduce effect.
  • Keep them → If they are real and meaningful (e.g., a rich billionaire in income dataset).

---

Explain abput PLOTTING WITH PANDAS ?

• Pandas has built-in plotting functions (based on Matplotlib) that allow quick visualization of data.
• We can call .plot() on Series or DataFrame to create different types of plots.

1. Line Plots

• Default plot type in Pandas.
• Useful for showing trends over time (continuous data).

Example:

import pandas as pd

import matplotlib.pyplot as plt

data = pd.Series([1, 3, 5, 7, 9])
data.plot(kind="line")
plt.show()

---

2. Bar Plots

• Used to compare categories.
• Vertical (default) or horizontal (kind="barh").

Example:

df.plot(
    x="Subject",
    y="Marks",
    kind="bar"
)

plt.show()

---

3. Histograms

• Show the distribution of numerical data.
• Data is divided into bins (intervals).

Example:

import numpy as np

data = pd.Series(np.random.randn(1000))

data.plot(kind="hist", bins=20)

plt.show()

---

4. Density Plots (KDE – Kernel Density Estimation)

• Smooth curve version of histogram.
• Shows probability distribution of data.

Example:

data.plot(kind="density")

plt.show()

---

5. Scatter (Point) Plots

• Show relationship between two numerical variables.
• Each point = one observation.

Example:

df.plot(
    x="Height",
    y="Weight",
    kind="scatter"
)

plt.show()