Data Cleaning Techniques with Examples and Outputs 🧹

1. Stop Words Removal 🗑️

Purpose:
Removes common words like "the," "is," and "and" to reduce noise and focus on meaningful content.

Code Example:

import nltk
from nltk.corpus import stopwords

nltk.download('stopwords')

def remove_stopwords(text):
    stop_words = set(stopwords.words('english'))
    words = text.split()
    filtered_words = [word for word in words if word.lower() not in stop_words]
    return ' '.join(filtered_words)

text = "This is an example sentence with some common words."
cleaned_text = remove_stopwords(text)
print(cleaned_text)

Output:

example sentence common words.

Why It Helps:

• ✅ Reduces text size for faster processing.

• ✅ Improves focus on keywords during retrieval.

2. Special Character Removal ✂️

Purpose:
Removes punctuation, HTML tags, and symbols that add clutter without contributing meaning.

Code Example:

import re
import string

def clean_text(text):
    # Remove HTML tags
    text = re.sub(r'<.*?>', '', text)
    # Remove special characters
    text = ''.join(char for char in text if char not in string.punctuation)
    return text

text = "Hello! <b>This</b> is a <i>test</i>."
cleaned_text = clean_text(text)
print(cleaned_text)

Output:

Hello This is a test

Why It Helps:

• ✅ Ensures cleaner input for embeddings.

• ✅ Improves text consistency across data.

3. Text Normalization 🔤

Purpose:
Standardizes text by converting to lowercase and applying stemming (reducing words to their root form).

Code Example:

import nltk
from nltk.stem import PorterStemmer
nltk.download('punkt')

def normalize_text(text):
    text = text.lower()
    stemmer = PorterStemmer()
    words = nltk.word_tokenize(text)
    stemmed_words = [stemmer.stem(word) for word in words]
    return ' '.join(stemmed_words)

text = "Running runs quickly. Cats are playing."
normalized_text = normalize_text(text)
print(normalized_text)

Output:

run run quickli . cat are play .

Why It Helps:

• ✅ Makes text uniform (lowercase).

• ✅ Ensures similar words are grouped (run, running → "run").

4. Fact-Checking and Updating Information 📚

Purpose:
Ensures retrieved data is accurate and up-to-date by integrating APIs or knowledge graphs.

Example Workflow: (Conceptual)

• Use APIs like WolframAlpha or Google Knowledge Graph for verification.

• Implement checks for date references or version numbers to update content.

Pseudo-Code Example:

def fact_check(text):
    # Example API request (replace with an actual service)
    result = external_api_call(text)  
    if not result['valid']:
        return "Fact-check failed."
    return text

response = fact_check("The capital of France is Paris.")
print(response)

Output:

The capital of France is Paris.

Why It Helps:

• ✅ Avoids hallucinations (false answers) from the model.

• ✅ Ensures trustworthiness in generated responses.

5. Domain-Specific Cleaning 🔬

Purpose:
Applies cleaning methods customized to specific fields (e.g., medical, legal, financial).

Example for Medical Terms:

medical_terms = ["mg", "ml", "dose", "tablet"]

def remove_unwanted_terms(text):
    words = text.split()
    filtered_words = [word for word in words if word.lower() not in medical_terms]
    return ' '.join(filtered_words)

text = "Take 10 mg dose twice a day."
cleaned_text = remove_unwanted_terms(text)
print(cleaned_text)

Output:

Take 10 twice a day.

Why It Helps:

• ✅ Keeps data relevant for specialized models.

• ✅ Avoids filtering out domain-specific terms unintentionally.

Key Takeaways 📝

1. Stop Words Removal reduces noise by focusing only on important words.

2. Special Character Removal cleans up unnecessary clutter like HTML tags and symbols.

3. Text Normalization standardizes words for consistency and efficiency.

4. Fact-Checking ensures the information is accurate and trustworthy.

5. Domain-Specific Cleaning targets field-related terms for precision.

Would you like to see extended versions for any of these techniques? 😊