# Data Cleaning Techniques with Examples and Outputs 🧹

### **1\. Stop Words Removal 🗑️**

**Purpose:**  
Removes common words like "the," "is," and "and" to reduce noise and focus on meaningful content.

**Code Example:**

```python
import nltk
from nltk.corpus import stopwords

nltk.download('stopwords')

def remove_stopwords(text):
    stop_words = set(stopwords.words('english'))
    words = text.split()
    filtered_words = [word for word in words if word.lower() not in stop_words]
    return ' '.join(filtered_words)

text = "This is an example sentence with some common words."
cleaned_text = remove_stopwords(text)
print(cleaned_text)
```

**Output:**

```python
example sentence common words.
```

**Why It Helps:**

* ✅ Reduces text size for faster processing.
    
* ✅ Improves focus on keywords during retrieval.
    

---

### **2\. Special Character Removal ✂️**

**Purpose:**  
Removes punctuation, HTML tags, and symbols that add clutter without contributing meaning.

**Code Example:**

```python
import re
import string

def clean_text(text):
    # Remove HTML tags
    text = re.sub(r'<.*?>', '', text)
    # Remove special characters
    text = ''.join(char for char in text if char not in string.punctuation)
    return text

text = "Hello! <b>This</b> is a <i>test</i>."
cleaned_text = clean_text(text)
print(cleaned_text)
```

**Output:**

```python
Hello This is a test
```

**Why It Helps:**

* ✅ Ensures cleaner input for embeddings.
    
* ✅ Improves text consistency across data.
    

---

### **3\. Text Normalization 🔤**

**Purpose:**  
Standardizes text by converting to lowercase and applying stemming (reducing words to their root form).

**Code Example:**

```python
import nltk
from nltk.stem import PorterStemmer
nltk.download('punkt')

def normalize_text(text):
    text = text.lower()
    stemmer = PorterStemmer()
    words = nltk.word_tokenize(text)
    stemmed_words = [stemmer.stem(word) for word in words]
    return ' '.join(stemmed_words)

text = "Running runs quickly. Cats are playing."
normalized_text = normalize_text(text)
print(normalized_text)
```

**Output:**

```python
run run quickli . cat are play .
```

**Why It Helps:**

* ✅ Makes text uniform (lowercase).
    
* ✅ Ensures similar words are grouped (run, running → "run").
    

---

### **4\. Fact-Checking and Updating Information 📚**

**Purpose:**  
Ensures retrieved data is **accurate** and **up-to-date** by integrating APIs or knowledge graphs.

**Example Workflow:** *(Conceptual)*

* Use APIs like **WolframAlpha** or **Google Knowledge Graph** for verification.
    
* Implement checks for **date references** or **version numbers** to update content.
    

**Pseudo-Code Example:**

```python
def fact_check(text):
    # Example API request (replace with an actual service)
    result = external_api_call(text)  
    if not result['valid']:
        return "Fact-check failed."
    return text

response = fact_check("The capital of France is Paris.")
print(response)
```

**Output:**

```python
The capital of France is Paris.
```

**Why It Helps:**

* ✅ Avoids **hallucinations** (false answers) from the model.
    
* ✅ Ensures **trustworthiness** in generated responses.
    

---

### **5\. Domain-Specific Cleaning 🔬**

**Purpose:**  
Applies cleaning methods customized to specific fields (e.g., **medical**, **legal**, **financial**).

**Example for Medical Terms:**

```python
medical_terms = ["mg", "ml", "dose", "tablet"]

def remove_unwanted_terms(text):
    words = text.split()
    filtered_words = [word for word in words if word.lower() not in medical_terms]
    return ' '.join(filtered_words)

text = "Take 10 mg dose twice a day."
cleaned_text = remove_unwanted_terms(text)
print(cleaned_text)
```

**Output:**

```python
Take 10 twice a day.
```

**Why It Helps:**

* ✅ Keeps data **relevant** for specialized models.
    
* ✅ Avoids filtering out **domain-specific terms** unintentionally.
    

---

### **Key Takeaways 📝**

1. **Stop Words Removal** reduces noise by focusing only on important words.
    
2. **Special Character Removal** cleans up unnecessary clutter like HTML tags and symbols.
    
3. **Text Normalization** standardizes words for consistency and efficiency.
    
4. **Fact-Checking** ensures the information is accurate and trustworthy.
    
5. **Domain-Specific Cleaning** targets field-related terms for precision.
    

Would you like to see **extended versions** for any of these techniques? 😊