1. df.head(): Previewing Your Data

The df.head() method provides a quick way to preview the first few rows of your DataFrame. This is invaluable for understanding the structure of your data and verifying its contents.

import pandas as pd

data = {'Name': ['Alice', 'Bob', 'Charlie', 'David'],
        'Age': [25, 30, 22, 28],
        'City': ['New York', 'London', 'Paris', 'Tokyo']}

df = pd.DataFrame(data)

print(df.head(2))  # Display the first 2 rows

2. df.tail(): Inspecting the End of Your Data

Similar to df.head(), the df.tail() method lets you view the last few rows of your DataFrame. This is particularly useful for checking if your data is complete or if there are any unexpected patterns at the end of your dataset.

import pandas as pd

data = {'Name': ['Alice', 'Bob', 'Charlie', 'David'],
        'Age': [25, 30, 22, 28],
        'City': ['New York', 'London', 'Paris', 'Tokyo']}

df = pd.DataFrame(data)

print(df.tail(2))  # Display the last 2 rows

3. df.info(): Understanding Your Data’s Structure

The df.info() method provides a comprehensive overview of your DataFrame’s metadata. It tells you the data types of each column, the number of non-null values, and the memory usage of your DataFrame. This information is crucial during the initial exploratory data analysis phase, as it helps you understand the nature of your data and identify potential issues.

import pandas as pd

data = {'Name': ['Alice', 'Bob', 'Charlie', 'David'],
        'Age': [25, 30, 22, 28],
        'City': ['New York', 'London', 'Paris', 'Tokyo']}

df = pd.DataFrame(data)

df.info()

Output:

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 4 entries, 0 to 3
Data columns (total 3 columns):
 #   Column  Non-Null Count  Dtype 
---  ------  --------------  ----- 
 0   Name    4 non-null      object
 1   Age     4 non-null      int64 
 2   City    4 non-null      object
dtypes: int64(1), object(2)
memory usage: 224.0+ bytes

4. df.describe(): Exploring Data Distributions

The df.describe() method generates descriptive statistics for the numerical columns of your DataFrame. It provides you with essential information about the distribution of your data, such as the count, mean, standard deviation, minimum, and maximum values. This method is invaluable for understanding the central tendency and variability of your data.

import pandas as pd

data = {'Name': ['Alice', 'Bob', 'Charlie', 'David'],
        'Age': [25, 30, 22, 28],
        'City': ['New York', 'London', 'Paris', 'Tokyo']}

df = pd.DataFrame(data)

print(df.describe())

Output:

             Age
count   4.000000
mean   26.250000
std     3.535534
min    22.000000
25%    23.750000
50%    26.500000
75%    28.250000
max    30.000000

5. df.fillna(): Handling Missing Data

Missing data (represented as NaN) is a common problem in real-world datasets. The df.fillna() method provides a powerful way to address this issue by replacing missing values with a specified value or by using a method like the mean of a column. This ensures that your data is complete and ready for analysis.

import pandas as pd

data = {'Name': ['Alice', 'Bob', 'Charlie', 'David'],
        'Age': [25, 30, 22, None],
        'City': ['New York', 'London', 'Paris', 'Tokyo']}

df = pd.DataFrame(data)

df['Age'] = df['Age'].fillna(df['Age'].mean())  # Replace missing age with mean
print(df)

6. df.drop_duplicates(): Eliminating Duplicates

Duplicate rows can skew your analysis and produce inaccurate results. The df.drop_duplicates() method helps you maintain data integrity by removing duplicate rows, ensuring that each row represents a unique observation.

import pandas as pd

data = {'Name': ['Alice', 'Bob', 'Charlie', 'Alice'],
        'Age': [25, 30, 22, 25],
        'City': ['New York', 'London', 'Paris', 'New York']}

df = pd.DataFrame(data)

df = df.drop_duplicates(subset=['Name', 'Age'])  # Drop duplicates based on 'Name' and 'Age'
print(df)

7. df.astype(): Converting Data Types

The df.astype() method allows you to convert column data types to ensure compatibility with mathematical operations or other analysis tasks. For instance, you might need to change non-numeric data to a float data type for numerical analysis.

import pandas as pd

data = {'Name': ['Alice', 'Bob', 'Charlie', 'David'],
        'Age': ['25', '30', '22', '28'],
        'City': ['New York', 'London', 'Paris', 'Tokyo']}

df = pd.DataFrame(data)

df['Age'] = df['Age'].astype(int)  # Convert 'Age' column to integers
print(df)

8. df.clip(): Handling Outliers

Outliers can disproportionately affect statistical analysis. The df.clip() method allows you to handle outliers by limiting values to a specified range. This helps to prevent extreme values from skewing your results and ensures a more accurate analysis.

import pandas as pd

data = {'Name': ['Alice', 'Bob', 'Charlie', 'David'],
        'Age': [25, 30, 22, 50],  # David has an outlier age
        'City': ['New York', 'London', 'Paris', 'Tokyo']}

df = pd.DataFrame(data)

df['Age'] = df['Age'].clip(lower=18, upper=40)  # Clip ages between 18 and 40
print(df)

9. df.groupby(): Aggregating Data by Groups

The df.groupby() method is a cornerstone of data analysis. It allows you to aggregate data by grouping rows based on the values of one or more columns. This is extremely useful for calculating summary statistics within each group, such as means, sums, or counts.

import pandas as pd

data = {'Name': ['Alice', 'Bob', 'Charlie', 'David'],
        'Age': [25, 30, 22, 28],
        'City': ['New York', 'London', 'Paris', 'Tokyo'],
        'Salary': [60000, 75000, 55000, 80000]}

df = pd.DataFrame(data)

grouped = df.groupby('City')['Salary'].mean()  # Calculate mean salary by city
print(grouped)

Output:

City
London       75000.0
New York     60000.0
Paris        55000.0
Tokyo        80000.0
Name: Salary, dtype: float64

10. df.apply(): Applying Custom Functions

The df.apply() method allows you to apply custom functions to your DataFrame, either row-wise or column-wise. This is particularly useful for complex data transformations and calculations that cannot be easily achieved using built-in functions.

import pandas as pd

data = {'Name': ['Alice', 'Bob', 'Charlie', 'David'],
        'Age': [25, 30, 22, 28],
        'City': ['New York', 'London', 'Paris', 'Tokyo']}

df = pd.DataFrame(data)

def age_category(age):
    if age < 25:
        return 'Young'
    elif age < 35:
        return 'Adult'
    else:
        return 'Senior'

df['Age Category'] = df['Age'].apply(age_category)  # Apply custom function to create a new column
print(df)

11. df.merge(): Combining DataFrames

The df.merge() method is similar to SQL joins. It allows you to combine two DataFrames based on common columns or indices. This is crucial for integrating data from multiple sources, such as merging customer information with sales records.

import pandas as pd

customer_data = {'CustomerID': [1, 2, 3],
                'Name': ['Alice', 'Bob', 'Charlie']}
customer_df = pd.DataFrame(customer_data)

order_data = {'CustomerID': [1, 2, 4],
            'OrderDate': ['2023-01-15', '2023-02-10', '2023-03-05']}
order_df = pd.DataFrame(order_data)

merged_df = pd.merge(customer_df, order_df, on='CustomerID')  # Merge on 'CustomerID'
print(merged_df)

12. df.concat(): Concatenating DataFrames

The df.concat() method allows you to concatenate DataFrames along a particular axis, either vertically (rows) or horizontally (columns). This is useful for combining datasets that share a common structure, such as appending new data to an existing DataFrame.

import pandas as pd

df1 = pd.DataFrame({'A': [1, 2], 'B': [3, 4]})
df2 = pd.DataFrame({'A': [5, 6], 'B': [7, 8]})

concatenated_df = pd.concat([df1, df2], ignore_index=True)  # Concatenate vertically
print(concatenated_df)

13. df.pivot_table(): Creating Pivot Tables

Pivot tables are powerful tools for summarizing data. The df.pivot_table() method allows you to create pivot tables that analyze the relationship between variables. You can specify the columns for rows, columns, and values, and the method will calculate the desired aggregation (e.g., mean, sum) for each cell.

import pandas as pd

data = {'Name': ['Alice', 'Bob', 'Charlie', 'David'],
        'Age': [25, 30, 22, 28],
        'City': ['New York', 'London', 'Paris', 'Tokyo'],
        'Salary': [60000, 75000, 55000, 80000]}

df = pd.DataFrame(data)

pivot_table = pd.pivot_table(df, values='Salary', index='City', columns='Age')  # Create pivot table
print(pivot_table)

Output:

Age          22     25     28     30
City                                        
London       NaN     NaN     NaN  75000
New York  60000.0   NaN     NaN     NaN
Paris      55000.0   NaN     NaN     NaN
Tokyo       NaN     NaN  80000.0   NaN

14. df.resample(): Time-Series Analysis

The df.resample() method is essential for time-series analysis. It allows you to resample your data to a different frequency, such as converting daily data to monthly data. This is crucial for understanding temporal trends in your data, such as seasonal patterns or growth over time.

import pandas as pd

data = {'Date': pd.to_datetime(['2023-01-01', '2023-01-02', '2023-01-03', '2023-01-04']),
        'Sales': [100, 120, 150, 180]}

df = pd.DataFrame(data).set_index('Date')

resampled_df = df['Sales'].resample('D').mean()  # Resample to daily frequency (mean aggregation)
print(resampled_df)

Output:

Date
2023-01-01    100.0
2023-01-02    120.0
2023-01-03    150.0
2023-01-04    180.0
Freq: D, Name: Sales, dtype: float64

15. df.unstack(): Reshaping MultiIndex DataFrames

MultiIndex DataFrames can be complex to analyze. The df.unstack() method allows you to transform a MultiIndex DataFrame by unstacking one of the levels. This makes it easier to visualize and analyze complex data structures by reorganizing the data into a more readable format.

import pandas as pd

data = {'Year': [2022, 2022, 2023, 2023],
        'Quarter': ['Q1', 'Q2', 'Q1', 'Q2'],
        'Sales': [100, 150, 120, 180]}

df = pd.DataFrame(data).set_index(['Year', 'Quarter'])

unstacked_df = df.unstack()  # Unstack the 'Quarter' level
print(unstacked_df)

Output:

       Sales      
Quarter    Q1   Q2
Year              
2022    100  150
2023    120  180

16. df.pipe(): Chaining Functions Together

The df.pipe() method allows you to chain functions together in a clean and readable way. You pass the DataFrame as the first argument to a custom function, and the function can then perform a sequence of operations on the DataFrame and return the result. This simplifies complex operations and maintains a clear coding style.

import pandas as pd

data = {'Name': ['Alice', 'Bob', 'Charlie', 'David'],
        'Age': [25, 30, 22, 28],
        'City': ['New York', 'London', 'Paris', 'Tokyo']}

df = pd.DataFrame(data)

def process_data(df):
    df['Age Category'] = df['Age'].apply(lambda age: 'Adult' if age >= 18 else 'Child')
    return df.groupby('Age Category').size()

result = df.pipe(process_data)  # Chain functions using pipe
print(result)

Output:

Age Category
Adult    4
dtype: int64