225 Pandas Interview Questions and Answers (2026)

A DataFrame is a 2D labeled table, and Pandas offers many constructors depending on the source of your data: in-memory Python objects, NumPy arrays, or external files.

• From a dictionary: Keys become column names, values become columns: pd.DataFrame({'a': [1,2], 'b': [3,4]})

• From a list of dicts or list of lists: List of dicts: each dict is a row. List of lists: pass columns= to name columns.

• From a NumPy array: pd.DataFrame(np.arange(6).reshape(2,3)) with optional index= and columns=.

• From a Series or dict of Series: Each Series becomes a column, aligned on its index.

• From external files: Readers like pd.read_csv(), pd.read_excel(), pd.read_sql(), pd.read_json().

• Empty then populate: pd.DataFrame() and assign columns later (slower, avoid growing row by row).

A Series is a one-dimensional labeled array in Pandas: it holds a sequence of values of a single dtype along with an associated index that labels each element.

• One-dimensional and homogeneous: Backed by a NumPy array, so all values share one dtype (with object as a fallback for mixed data).

• Has an index: Defaults to a RangeIndex (0,1,2,...) but can be custom labels, enabling label-based lookup.

• Building block of a DataFrame: Each column of a DataFrame is a Series sharing the DataFrame's index.

• Supports vectorized ops and alignment: Arithmetic aligns on the index automatically, and it exposes methods like .mean(), .value_counts().

The core difference is dimensionality: a Series is a single labeled column of one dtype, while a DataFrame is a 2D table of multiple columns (each itself a Series) sharing a common row index.

• Dimensions: Series is 1D (one axis: the index). DataFrame is 2D (row index + columns).

• Dtype: A Series is homogeneous (one dtype). A DataFrame can hold a different dtype per column.

• Indexing: Series is labeled by index only. DataFrame is labeled by both index and column names.

• Relationship: Selecting a single column from a DataFrame (df['col']) returns a Series.

Modern Pandas has two primary data structures: the 1D Series and the 2D DataFrame. A third, the Panel (3D), existed historically but has been removed.

• Series (1D): A labeled one-dimensional array of a single dtype.

• DataFrame (2D): A labeled table of rows and columns, where each column is a Series.

• Index: Not a data container itself, but a fundamental structure (Index, MultiIndex) that labels the axes of both.

• Panel (deprecated): A former 3D structure, removed in Pandas 0.25; use MultiIndex DataFrames or xarray instead.

A Series is a single one-dimensional labeled array of one dtype, whereas a DataFrame is a two-dimensional table made up of multiple Series columns that share a common row index.

• Shape: Series has one axis; DataFrame has two (rows and columns).

• Data types: Series holds a single dtype; a DataFrame can mix dtypes across columns.

• Access: df['col'] returns a Series; a DataFrame needs both row and column selectors (.loc[], .iloc[]).

• Think of it as: A DataFrame is essentially a dict of Series aligned on one index.

Yes: a Series is Pandas' one-dimensional labeled array, an ordered sequence of values of a single dtype paired with an index that labels each value.

• Two aligned parts: The values (a NumPy-backed array) and the index (labels for each value).

• Homogeneous: All elements share one dtype, falling back to object if mixed.

• Label-aware: You can look up by label (s['a']) or position (s.iloc[0]), and operations align on the index.

A Series can be created from most 1D containers: a Python list, a NumPy array, a dict, or a scalar, optionally with a custom index.

• From a list or NumPy array: pd.Series([1,2,3]) gets a default RangeIndex; pass index= for custom labels.

• From a dictionary: Keys become the index and values become the data: pd.Series({'a': 1, 'b': 2}).

• From a scalar: pd.Series(5, index=['a','b','c']) broadcasts the value across the given index.

• From a DataFrame column: Selecting df['col'] returns a Series.

Use the .to_numpy() method (preferred in modern Pandas) to extract the underlying data as a NumPy ndarray, dropping the index and column labels.

• Preferred: df.to_numpy(): Explicit and recommended over the older .values attribute.

• Legacy: df.values: Still works but less predictable in edge cases; prefer .to_numpy().

• Watch the resulting dtype: With mixed column dtypes, NumPy upcasts to a common type (often object); pass dtype= to force one.

• Labels are lost: The index and column names are not carried into the array.

A Series is a one-dimensional labeled array; a DataFrame is a two-dimensional labeled table (a collection of Series sharing one index). Both carry axis labels, which is what separates them from raw NumPy arrays.

• Series (1D):

  • A single column of values with an associated index (labels).

  • Holds one dtype for all its values.

• DataFrame (2D):

  • A table with a row index and named columns.

  • Each column is a Series and can have its own dtype (mixed types across columns).

• Relationship:

  • Selecting one column (df['col']) returns a Series; the DataFrame is essentially a dict-like container of aligned Series.

  • Operations align on labels automatically, not just position.

You pass the data to the pd.DataFrame() constructor, and Pandas infers rows and columns from the structure you give it: dict keys become columns, list-of-lists become rows, and NumPy arrays become a 2D block.

• From a dict of columns: Keys are column names, values are the column data; the most common pattern.

• From a list of lists / 2D array: Each inner list is a row; supply columns= to name them.

• From a list of dicts (records): Each dict is a row; union of keys becomes columns, missing keys fill with NaN.

• From a NumPy array: Positional data; add index= and columns= for labels.

Use the pd.Series() and pd.DataFrame() constructors: they infer structure from the input, so a 1D input builds a Series and a 2D/tabular input builds a DataFrame.

• Series:

  • From a list or array: values with a default integer index.

  • From a dict: keys become the index labels.

• DataFrame:

  • Dict of lists/arrays: keys are columns.

  • List of dicts (records): each dict is a row.

  • 2D array or list of lists: positional rows and columns.

  • For record arrays, pd.DataFrame.from_records() is also available.

.loc selects by label (index/column names), while .iloc selects by integer position. Use .loc when you know the labels and .iloc when you know the numeric position.

• .loc (label-based):

  • Indexes by the actual index/column labels, e.g. df.loc['2020', 'sales'].

  • Slices are inclusive of the end label.

  • Accepts boolean masks, which is why filtering uses it.

• .iloc (position-based):

  • Indexes by 0-based integer position, e.g. df.iloc[0, 2].

  • Slices are exclusive of the end, like normal Python.

• When to use each:

  • Meaningful labels or boolean filtering: .loc.

  • Positional access (first N rows, iterating by offset): .iloc.

The core distinction is label versus position: .loc indexes using the index and column labels, and .iloc indexes using integer offsets regardless of what the labels are.

• .loc is label-based:

  • End of a slice is inclusive: df.loc[2:5] includes label 5.

  • Supports boolean arrays for filtering.

• .iloc is integer-position-based: End of a slice is exclusive: df.iloc[2:5] gives positions 2, 3, 4.

• Gotcha: If the index is itself integers, .loc[2] looks up label 2, not the 3rd row: they can diverge.

.isin() returns a boolean mask indicating whether each element is contained in a given set of values. It is the vectorized way to test membership against multiple values at once, used mostly for filtering rows.

• What it returns: A boolean Series/DataFrame of the same shape, True where the value is in the passed collection.

• When to use it:

  • Filtering rows matching any of several values: cleaner than chaining multiple == with |.

  • Negate with ~ to exclude values ("not in").

• Notes: Accepts lists, sets, Series, or dicts; matching is by value and dtype-sensitive.

Boolean masking selects data by passing a Series/array of True/False values the same length as the axis: Pandas keeps only the rows (or elements) where the mask is True.

• Build the mask with a vectorized comparison: An expression like df['age'] > 30 returns a boolean Series aligned to the index.

• Apply it by indexing: df[mask] or df.loc[mask] returns only the True rows; alignment is by index, not position.

• Combine conditions with bitwise operators: Use &, |, ~ (not and/or) and wrap each term in parentheses because of operator precedence.

• Why it works: vectorized and fast: Comparisons run in C over the whole array, avoiding Python loops.

Select one column with a string key (returns a Series) and multiple columns with a list of names (returns a DataFrame).

• Single column: df['col'] returns a Series; df.col (attribute access) also works but fails for names with spaces or that clash with methods.

• Multiple columns: df[['a','b']] takes a list and returns a DataFrame; note the double brackets.

• Single column as a DataFrame: df[['a']] (list of one) keeps it 2D instead of collapsing to a Series.

• Label-based alternative: df.loc[:, ['a','b']] is explicit and pairs well with row selection.

loc[] selects by label (index/column names) while iloc[] selects by integer position, and their slicing behavior differs at the boundary.

• loc is label-based:

  • Uses index and column labels: df.loc[5, 'name'] where 5 is a label value.

  • Slices are inclusive of the end label: df.loc['a':'c'] includes 'c'.

  • Accepts boolean masks.

• iloc is position-based:

  • Uses 0-based integers regardless of the labels: df.iloc[0, 2].

  • Slices are exclusive of the end (like Python): df.iloc[0:3] gives rows 0,1,2.

• Common trap: With an integer index, loc still matches label values while iloc matches positions; they can return different rows.

loc indexes by the meaning of the labels while iloc indexes by physical integer position, so the right choice depends on whether you're keying off names or ordinal location.

• Label-based .loc:

  • Selects on index and column labels, accepts boolean masks, and its slices are end-inclusive.

  • Best when data has meaningful labels (dates, IDs, names).

• Position-based .iloc:

  • Selects on 0-based integer positions, end-exclusive slices, like NumPy arrays.

  • Best for 'first N rows', 'last column', or when labels are irrelevant.

• Both take row and column selectors: df.loc[rows, cols] and df.iloc[rows, cols]; both can also set values in place.

• Key gotcha: They diverge when the index is integers that aren't in row order (e.g. after sorting or filtering).

Boolean indexing filters rows by passing a boolean Series aligned to the DataFrame's index: rows where the value is True are kept and the rest dropped.

• Create the boolean Series: Any vectorized comparison (==, >, .isin(), .str.contains()) produces one True/False per row.

• Index with it: df[mask] or the more explicit df.loc[mask]; use df.loc[mask, 'col'] to filter and select columns together.

• Alignment matters: The mask is matched by index label, so a misaligned mask can give unexpected NaNs or errors.

• Combine and negate: Use &, |, and ~ with parentheses to build compound masks.

Pandas is an open-source Python library for working with structured, tabular data: it provides fast, expressive data structures that make cleaning, transforming, and analyzing data straightforward. It is built on top of NumPy and is a cornerstone of the Python data-analysis ecosystem.

• Core data structures:

  • Series: a labeled one-dimensional array.

  • DataFrame: a labeled two-dimensional table of columns, like a spreadsheet or SQL table.

• Primary use cases:

  • Data cleaning and preparation (handling missing values, reshaping, type conversion).

  • Exploratory analysis and aggregation (grouping, pivoting, summarizing).

  • Reading and writing many formats: CSV, Excel, SQL, JSON, Parquet.

• Key features:

  • Automatic data alignment on a labeled index.

  • Powerful groupby for split-apply-combine workflows.

  • Built-in handling of missing data (NaN) and time-series tools.

  • Vectorized operations for speed instead of Python loops.

Pandas is a Python library for data manipulation and analysis, giving you fast, flexible data structures for working with labeled tabular and time-series data. It is the go-to tool for loading, cleaning, transforming, and analyzing structured datasets in Python.

• What it provides:

  • The Series (1D) and DataFrame (2D) structures with labeled axes.

  • Built on NumPy for vectorized, efficient operations.

• What it is used for:

  • Reading/writing data from CSV, Excel, SQL, JSON, and more.

  • Cleaning and reshaping: filtering, handling missing values, merging, and pivoting.

  • Aggregation and summarization via groupby and pivot tables.

  • Time-series analysis with date/time indexing and resampling.

• Why it matters: it is a foundational tool that feeds into visualization and machine-learning libraries, making it central to the Python data workflow.

Pandas is a Python library for fast, expressive manipulation of labeled tabular and time-series data, built on top of NumPy.

• Core data structures: Series (1-D labeled array) and DataFrame (2-D labeled table) with an explicit Index.

• Label-based alignment: Operations align on index/column labels automatically, so arithmetic across objects just works.

• Rich I/O: Readers/writers for CSV, Excel, JSON, SQL, Parquet, and more (read_csv, to_parquet).

• Data cleaning and reshaping: Missing-data handling (NaN, fillna, dropna), plus merge, pivot, melt.

• Group-by and time series: Split-apply-combine via groupby, and first-class datetime indexing, resampling, and rolling windows.

Pandas is an open-source Python library for data analysis and manipulation, providing labeled, table-like data structures built on top of NumPy.

• Two primary structures:

  • Series: a one-dimensional labeled array (like a single column).

  • DataFrame: a two-dimensional labeled table of columns, each with its own dtype.

• Purpose: Loading, cleaning, transforming, aggregating, and analyzing structured data.

• Ecosystem role: The de facto standard for data wrangling in Python, integrating with NumPy, matplotlib, and scikit-learn.

Pandas' key benefits are concise, vectorized data handling with labeled structures, broad I/O support, and powerful cleaning, reshaping, and aggregation tools.

• Expressive, less code: Filtering, grouping, and joining that would take loops become one-line vectorized calls.

• Labeled data with alignment: Named rows/columns and automatic index alignment reduce positional bugs.

• Missing-data handling: Native NaN support with fillna, dropna, interpolate.

• Flexible I/O: Read/write CSV, Excel, SQL, JSON, Parquet with a consistent API.

• Analysis power: groupby, pivot tables, merges, and rich time-series tools (resampling, rolling windows).

• Performance: Operations run in optimized C/NumPy under the hood rather than Python loops.

Pandas is better than raw lists/dicts because it stores columns as typed, contiguous arrays and exposes vectorized, labeled operations, giving both speed and clarity that manual Python collections can't match.

• Speed: Vectorized operations run in C over NumPy arrays, far faster than Python-level loops over lists.

• Conciseness: Filtering, aggregating, and joining are single expressions instead of nested loops and comprehensions.

• Built-in structure: Labeled indexes, per-column dtypes, and alignment come free; with dicts you'd track all of that manually.

• Data quality tools: Missing-value handling, type coercion, and duplicate removal are built in.

• Caveat: For tiny or purely non-tabular data, plain dicts/lists can be simpler and lighter.

NumPy provides homogeneous, position-indexed n-dimensional arrays for numerical computing; Pandas builds on it to add labeled axes, heterogeneous columns, and higher-level data-analysis tools.

• Data types:

  • NumPy: one dtype for the whole array.

  • Pandas: each column can have its own dtype (mixed types in one table).

• Indexing: NumPy is positional; Pandas adds meaningful row/column labels and alignment.

• Abstraction level: NumPy targets math on arrays and matrices; Pandas targets tabular data wrangling (groupby, merge, I/O, missing data).

• Relationship: They complement each other: Pandas uses NumPy arrays internally, and you can drop to .values/.to_numpy() for heavy numeric work.

Use pd.read_csv() to load a CSV into a DataFrame and df.to_csv() to write one back out.

• Reading: read_csv accepts a path or URL and many options: sep, header, index_col, usecols, dtype, parse_dates, na_values.

• Writing: Commonly pass index=False to avoid writing the row index as a column.

• Large files: Use chunksize to iterate in pieces, or usecols/dtype to limit memory.

Use pd.read_excel(), pointing it at the file path (or file-like object) and, optionally, the sheet to load. It returns a DataFrame (or a dict of DataFrames if you request multiple sheets).

• Install an engine: Pandas needs a backend: openpyxl for .xlsx, xlrd for legacy .xls.

• Basic call: df = pd.read_excel("file.xlsx") reads the first sheet by default.

• Common arguments:

  • sheet_name: name, index, list, or None (returns all sheets as a dict).

  • header, usecols, skiprows, dtype, parse_dates work much like in read_csv().

These options let you control typing, date handling, column selection, missing-value recognition, and indexing up front, which improves correctness, memory, and load speed rather than fixing things after the read.

• dtype: Specify column types explicitly to avoid inference and shrink memory (e.g. category for repeated strings).

• parse_dates: Parse given columns into datetime64 during the read.

• usecols: Load only the columns you need, cutting memory and time.

• na_values: Treat custom sentinels like "-" or "unknown" as NaN.

• index_col: Set one or more columns as the index directly on load.

You sort by values with sort_values() and by the index (labels) with sort_index(). Both work on DataFrames and Series and return a new object by default.

• sort_values():

  • Series: sorts the data directly.

  • DataFrame: pass by= one or more column names.

• sort_index(): Orders by row labels, or by column labels with axis=1.

• Common options:

  • ascending=False for descending; na_position= controls NaN placement.

  • inplace=True to modify in place instead of returning a copy.

Use describe(), which returns count, mean, std, min, quartiles, and max for numeric columns. Pass include='all' to also summarize non-numeric columns.

• Default behavior: Summarizes only numeric columns: count, mean, std, min, 25%/50%/75%, max.

• Extending it:

  • include='all' adds object/categorical stats like unique, top, freq.

  • include= / exclude= filter by dtype; percentiles= customizes the quantiles shown.

Sort a DataFrame with sort_values() to order by column data, or sort_index() to order by row/column labels. You can sort by multiple columns and control direction per column.

• By values:

  • df.sort_values(by='col') for one column.

  • Pass a list to by= for multi-key sorting; ascending can be a matching list of booleans.

• By labels: df.sort_index() orders rows; add axis=1 to sort columns alphabetically.

• Useful flags: kind= chooses the sort algorithm; na_position= controls NaN order; ignore_index=True resets the index.

sort_values() orders rows by the data in one or more columns (by= plus default axis=0), and can also order columns by a row's values with axis=1.

• Sort rows by column values (default):

  • by= names the column(s); a list sorts hierarchically (first key, then ties broken by next).

  • ascending accepts a bool or a list of bools matching by.

• Sort columns by row values: Set axis=1 and let by= reference an index label (a row).

• Behavior notes:

  • Returns a new DataFrame unless inplace=True.

  • NaNs go last by default; change with na_position='first'.

df.info() prints a concise technical summary of a DataFrame: its structure, column types, non-null counts, and memory footprint, making it the go-to first look at a dataset.

• What it reports:

  • The index type and range (row count).

  • Each column's name, non-null count, and dtype.

  • A tally of dtypes and total memory usage.

• Why it's useful:

  • Quickly spot missing data (non-null count < row count).

  • Catch wrong dtypes (e.g. numbers stored as object).

• Useful args: memory_usage='deep' for accurate object-column memory, and show_counts to force per-column counts on large frames.

head(), tail(), and sample() return a small subset of rows for quick inspection: the first rows, the last rows, and random rows respectively.

• head(n=5): First n rows; the default sanity check on structure and values.

• tail(n=5): Last n rows; useful for time series or to see how data ends.

• sample(n=... or frac=...):

  • Random rows: less biased than head/tail if data is ordered.

  • Set random_state for reproducibility; frac samples a fraction, replace=True allows bootstrapping.

df.corr() computes pairwise correlation coefficients between the numeric columns of a DataFrame, returning a symmetric correlation matrix.

• How it works:

  • Considers only numeric columns; non-numeric ones are excluded.

  • Returns a matrix where entry (i,j) is the correlation of columns i and j; the diagonal is 1.

  • Handles missing data pairwise (uses rows where both values are present).

• method options:

  • 'pearson' (default): linear correlation.

  • 'spearman': rank-based, captures monotonic relationships.

  • 'kendall': rank correlation for ordinal data.

• Caveats: measures association, not causation; Pearson only captures linear links and is sensitive to outliers. Use Series.corr(other) for a single pair.

DataFrame.plot() is a convenience wrapper around Matplotlib that lets you create common charts directly from a DataFrame or Series without writing boilerplate plotting code.

• What it provides:

  • Chart types via kind= ('line', 'bar', 'hist', 'box', 'scatter', 'area', 'pie') or accessor methods like df.plot.bar().

  • Automatically uses the index for the x-axis and column names as labels/legend.

• Relation to Matplotlib:

  • It delegates to Matplotlib under the hood and returns a Matplotlib Axes object.

  • You can pass an existing axis with ax= and further customize with Matplotlib calls (titles, limits, then plt.show()).

• Backend is configurable (pd.options.plotting.backend) to use Plotly or others instead of Matplotlib.

Access the df.columns attribute, which returns an Index object holding the column labels.

• df.columns: Returns an Index (array-like) of the column names.

• Convert as needed: df.columns.tolist() or list(df.columns) for a plain Python list.

• Also useful: iterating over df.columns directly works, and you can assign to it to rename all columns at once.

pd.get_dummies() converts categorical columns into binary (0/1) indicator columns, one per category, which is exactly what most ML models need for nominal features.

• Basic usage: pd.get_dummies(df, columns=['color']) replaces color with columns like color_red, color_blue.

• Avoid the dummy trap: drop_first=True drops one category to prevent perfect multicollinearity in linear models.

• Handling nulls: dummy_na=True adds a column flagging missing values instead of ignoring them.

• Caveat: Column set is data-dependent: for train/test consistency use reindex or scikit-learn's OneHotEncoder which remembers categories.

groupby() splits a DataFrame into groups sharing common key values so you can compute per-group statistics, analogous to SQL's GROUP BY.

• Creates a lazy grouping: Returns a DataFrameGroupBy object; computation runs only when you apply a method.

• Supports flexible keys: One column, multiple columns (creating a MultiIndex), a function, or a pd.Grouper for time-based bins.

• Enables many operations: Aggregation (mean, sum), transformation, filtering, and iteration over groups.

• Common tuning: as_index=False keeps keys as columns; dropna=False retains NaN groups.

groupby() splits a DataFrame into groups by one or more keys, applies a function to each group, and combines the results: it is the workhorse for aggregating and summarizing data by category.

• What it does:

  • Groups rows sharing a key value, then lets you aggregate (sum, mean), transform, or filter per group.

  • Returns a lazy GroupBy object; nothing computes until you call a method on it.

• Common usage:

  • Single key: df.groupby('col'); multiple keys give a MultiIndex result.

  • Select columns to aggregate to avoid computing on unneeded ones.

• Tip: use as_index=False (or .reset_index()) to keep group keys as columns rather than an index.

Pivoting turns long data into a wide layout by choosing which column becomes the row index, which becomes the columns, and which fills the values.

• Core parameters:

  • index: column(s) forming the new row index.

  • columns: column whose unique values become new columns.

  • values: column(s) filling the cells.

• Which method: pivot() for unique combinations; pivot_table() when aggregation is needed.

• The inverse operation is melt() (wide back to long).

Split-apply-combine is the mental model behind groupby(): pandas splits the data into groups, applies a function to each independently, then combines the results into a single output.

• Split: Partition rows into groups by the key(s); creates the GroupBy object.

• Apply: Run a function per group: aggregation (reduces to one value), transformation (same shape), or filtration (keeps/drops groups).

• Combine: Stitch group results back together; the shape depends on which apply type you used.

• Why it matters: knowing which apply type you want (agg, transform, filter) tells you the output shape before you run it.

get_dummies() converts categorical values into one-hot (indicator) columns: each distinct category becomes a new 0/1 column, so models can consume the categories numerically.

• What it produces:

  • One column per category, value 1 where the row had that category and 0 otherwise.

  • Works on a Series, or on a DataFrame's object/category columns.

• Key arguments:

  • drop_first=True: drops one category to avoid the dummy-variable trap (collinearity) in linear models.

  • prefix: names the generated columns.

  • dummy_na=True: adds an explicit column for NaN.

• Caveat: it doesn't remember categories, so train and test sets can produce mismatched columns; OneHotEncoder from scikit-learn is safer for pipelines.

pandas mirrors SQL's core relational operations: merge() is the analog of a JOIN (combine tables on keys) and groupby() is the analog of GROUP BY (split rows by key, then aggregate).

• merge() ≈ JOIN:

  • on / left_on/right_on set the join keys (the ON clause).

  • how='inner'|'left'|'right'|'outer' map directly to SQL join types.

• groupby() ≈ GROUP BY:

  • Follows split-apply-combine: group by key, apply an aggregate, combine results.

  • df.groupby('k')['x'].sum() ≈ SELECT k, SUM(x) FROM t GROUP BY k.

• Other parallels:

  • query()/boolean masks ≈ WHERE; filtering a grouped result ≈ HAVING.

  • concat() ≈ UNION.

• Difference: pandas keeps intermediate objects in memory and preserves an index, whereas SQL runs in the database engine and is set-oriented.

reset_index() moves the current index back into regular column(s) and replaces it with a default integer RangeIndex: it's how you flatten an index after operations that produced one.

• Typical uses:

  • After groupby or pivot, to turn group keys back into columns for a tidy, flat table.

  • After filtering/sorting, to get clean contiguous 0..n-1 row labels.

  • To collapse a MultiIndex into columns.

• Key parameters:

  • drop=True: discard the old index instead of keeping it as a column.

  • level=...: reset only specific levels of a MultiIndex.

• Inverse operation: set_index() does the opposite: promotes a column to the index.

An Index is the immutable, labeled axis of a Series or DataFrame: an ordered set of labels used to identify and align rows (and columns), acting like a row-key.

• It labels the axis: Every Series has one index; a DataFrame has a row index plus a column index.

• It is immutable: You can't modify labels in place; you assign a new index or use .rename(). Immutability makes it safe to share across objects.

• It powers alignment and lookup: Label-based selection via .loc[] and automatic alignment in operations both rely on it.

• It is typed and can be specialized: Variants like RangeIndex, DatetimeIndex, and MultiIndex optimize for their data.

Indexes give DataFrame rows meaningful labels that drive selection, alignment, joining, and grouping, turning raw positional data into keyed, self-describing data.

• Identification and lookup: Label-based access with .loc[] is clearer and often faster than positional access.

• Automatic alignment: Operations between objects align on the index, preventing accidental positional mixing.

• Joins and merges: .join() and merge(..., left_index=True) use the index as the key.

• Performance: A sorted or unique index enables fast label lookups; a hierarchical index encodes grouping structure.

• Not always needed: For purely positional work you can rely on the default RangeIndex.

The Index is the labeled axis that names each row (and each column) of a pandas object; its purpose is to identify, align, and efficiently look up data by label rather than by position.

• What it is: An immutable, ordered array-like of labels attached to an axis.

• What it's for:

  1. Fast label-based selection (.loc[]).

  2. Automatic data alignment across objects.

  3. Keys for joins, grouping, and reshaping.

• Specialized types: DatetimeIndex for time series, CategoricalIndex, and MultiIndex for hierarchy.

Boolean masking filters rows by passing a Series of True/False values (aligned to the index) into df[...]: pandas keeps only the rows where the mask is True.

• Build the mask with a vectorized comparison: e.g. df['age'] > 30 returns a boolean Series the same length as the frame.

• Apply it: df[mask] or df.loc[mask] selects matching rows; use .loc to also pick columns.

• Combine conditions with bitwise operators: Use &, |, ~ (not and/or), and wrap each condition in parentheses due to operator precedence.

• Helpful mask builders: isin(), between(), str.contains(), and isna() produce masks for common filters.

A RangeIndex is a memory-efficient integer index representing a range of values (start, stop, step), and it's the default because a fresh DataFrame has no natural row labels, so pandas assigns 0, 1, 2, ... implicitly.

• It's a lazy, computed index: It stores only start/stop/step rather than a full array, so it uses almost no memory regardless of length.

• Why it's the default: Every DataFrame needs an index; auto-numbering rows is the simplest, cheapest choice when none is specified.

• It behaves like a normal Int64 index for selection: Label-based .loc and positional .iloc happen to coincide until you filter or reorder rows.

• It can convert to a regular index: Operations like dropping rows or set_index() replace it; reset_index(drop=True) restores a clean RangeIndex.

set_index() promotes one or more columns to become the DataFrame's row index, while reset_index() does the reverse: it moves the current index back into a column and restores a default RangeIndex.

• set_index(): Turns a column into the index for label-based lookups and alignment; can build a MultiIndex from several columns.

• reset_index(): Pushes the index back as a regular column and reassigns 0, 1, 2, ...; use drop=True to discard it instead of keeping it as a column.

• Common pairing: reset_index() is frequently used after groupby() to flatten grouped keys back into columns.

• Both return a copy by default: Pass inplace=True to modify in place (though assigning the result is generally preferred).

astype() converts a Series or DataFrame to a specified dtype, returning a new object (it does not mutate in place) and raising if the conversion isn't valid.

• What it does:

  • Casts values to the target type: df["a"].astype("int64"), astype("category"), astype("string").

  • Accepts a dict to cast columns selectively: df.astype({"a": "int32", "b": "category"}).

• Key behaviors:

  • Strict by default: invalid casts (e.g. "abc" to int) raise ValueError; use pd.to_numeric(..., errors="coerce") when you need lenient conversion.

  • Can lose precision or overflow silently (float to int truncates, downcasting can wrap).

  • Cannot cast a NumPy int64 with NaN directly; go to nullable Int64 instead.

• vs alternatives: Use dedicated converters (to_numeric, to_datetime) for parsing/coercion; astype() is for known, clean conversions.

Use a lambda in Pandas when you need a small, one-off transformation inline, most commonly passed to apply(), map(), agg(), or assign(), where defining a named function would be overkill.

• Good fits:

  • Quick per-element or per-row logic: df["c"].apply(lambda x: x.strip().lower()).

  • Custom aggregations in groupby().agg() or column creation in assign().

• When to prefer a named function: Multi-line logic or reuse: a def is clearer and testable.

• Caveat: A lambda inside apply() is still a Python loop; if a vectorized equivalent exists, use it instead.

They are namespaced accessors on a Series that expose vectorized string (.str) and datetime (.dt) methods, so you transform an entire column at once instead of looping.

• .str (text columns):

  • Use for cleaning and parsing text: .str.lower(), .str.contains(), .str.extract(), .str.split().

  • Skips NaN and supports regex.

• .dt (datetime columns):

  • Use to pull out or reshape date parts: .dt.year, .dt.day_name(), .dt.date, .dt.floor('H').

  • Column must be datetime dtype (pd.to_datetime() first).

• When to use: Any time you'd otherwise write .apply() with a string or date function: the accessor is faster and cleaner.

Both rename columns, but rename() is selective and safe (map only the labels you specify, returns a new object), while assigning to df.columns replaces the entire column index at once and must match the width exactly.

• rename(): partial, mapping-based:

  • Pass a dict: df.rename(columns={'a':'x'}); unmentioned columns are unchanged.

  • Can rename index labels too, accepts a function, and errors gracefully.

  • Returns a copy unless inplace=True.

• df.columns = [...]: full replacement:

  • You must supply a label for every column, in order; wrong length raises a ValueError.

  • Fast and clean when relabeling all columns at once.

• Use rename() for a few targeted changes; assign df.columns when replacing the whole header.

The axis parameter tells a method which direction to operate along: axis=0 ('index') moves down the rows (operating per column), and axis=1 ('columns') moves across the columns (operating per row).

• axis=0 (default for most reductions):

  • Collapses rows: df.sum(axis=0) returns one value per column.

  • Think 'apply the function along the vertical axis'.

• axis=1:

  • Collapses columns: df.sum(axis=1) returns one value per row.

  • Used for df.drop(cols, axis=1) or row-wise apply.

• Mental model to avoid confusion:

  • The axis you name is the one that gets consumed/aggregated, so axis=0 gives a result indexed by columns.

  • A few methods (like concat) use axis to choose the direction of joining rather than reduction.

Handling missing data in pandas follows three stages: detect where nulls are, decide whether to drop them, or fill them with sensible values. The right choice depends on how much data is missing and why.

• Detecting:

  • isna()/notna() return boolean masks; df.isna().sum() counts nulls per column.

  • Missing is represented by NaN, NaT for datetimes, or pd.NA.

• Dropping:

  • dropna() removes rows (or columns with axis=1) containing nulls.

  • Tune with how='all', thresh=n, or subset=[...].

• Filling:

  • fillna() with a constant, a statistic (mean/median), or a method (ffill/bfill).

  • interpolate() estimates values from neighbors for ordered numeric/time series.

• Guidance: drop only when nulls are few or rows are unusable; impute when preserving samples matters and you can justify the fill value.

Pandas represents missing data with sentinel values and provides a consistent set of methods to detect and handle them; most computations skip nulls by default rather than failing.

• Representation:

  • NaN (a float) for numeric gaps, NaT for datetimes, and pd.NA for the newer nullable dtypes.

  • Because NaN is a float, integer columns with nulls are upcast to float (unless using nullable Int64).

• Detection: isna(), notna(), and aggregations like df.isna().sum().

• Handling: Remove with dropna() or fill with fillna()/interpolate().

• Behavior in operations: Reductions like sum and mean skip NaN by default (skipna=True); alignment during arithmetic introduces NaN where labels don't match.

Handle nulls in three steps: identify them, decide whether to remove or fill based on context, then apply the matching method (dropna(), fillna(), or interpolate()).

• Understand what counts as missing:

  • Pandas represents nulls as NaN, None, or NaT (for datetimes).

  • Detect with isna()/notna().

• Remove: dropna() for rows/columns you can afford to lose; tune with how, thresh, subset.

• Fill:

  • fillna() with a constant, a statistic, or a directional fill.

  • interpolate() for ordered numeric data.

• Watch practicalities:

  • Most methods return a new object; assign back or use inplace=True.

  • Convert placeholder strings (e.g. "NA", "?") to real NaN first, via na_values in read_csv or replace().

Detect with isna(), drop with dropna(), and fill with fillna() (or interpolate()): the three core operations of missing-data handling.

• Detect:

  • df.isna() (or isnull(), an alias) for a boolean mask.

  • df.isna().sum() for per-column counts; df.isna().any() to flag affected columns.

• Drop:

  • df.dropna(axis=0) removes rows, axis=1 removes columns.

  • Refine with subset to only consider certain columns.

• Fill:

  • df.fillna(value) for constants or statistics.

  • df.ffill()/df.bfill() for directional filling; df.interpolate() for estimated values.

They control the direction and threshold of removal: axis picks rows vs columns, how decides whether any or all values must be missing, and thresh sets a minimum count of non-null values to keep.

• axis (direction): axis=0 (default) drops rows; axis=1 drops columns.

• how (condition):

  • how='any' (default) drops if at least one value is missing.

  • how='all' drops only if every value is missing.

• thresh (threshold):

  • Keeps rows/columns having at least N non-null values.

  • It overrides how (if thresh is given, how is ignored).

• Related: subset limits which labels are checked for missingness.

ffill and bfill copy an existing neighboring value into the gap (backward or forward in position), while interpolate computes a new value between neighbors based on their trend.

• ffill (forward fill):

  • Propagates the last valid observation downward; good for carrying forward a known state.

  • Leading NaNs stay NaN (nothing precedes them).

• bfill (backward fill): Pulls the next valid value upward; trailing NaNs stay NaN.

• interpolate (estimation):

  • Fills with values along a line/curve between the two surrounding points, producing a gradient rather than a repeat.

  • Best for continuous numeric or time series data where values change smoothly.

• Choosing: Use directional fills for step-like/state data; use interpolation when the quantity trends between measurements.

Set the how argument of merge() (or join()) to choose which rows are kept when keys don't match on both sides.

• how='inner' (default for merge): Keeps only keys present in both frames (the intersection).

• how='left': Keeps all left rows; unmatched right columns become NaN.

• how='right': Mirror of left: keeps all right rows.

• how='outer': Keeps all keys from both (the union), filling missing sides with NaN.

• Helpers:

  • indicator=True adds a _merge column showing left_only/right_only/both, great for auditing joins.

  • join() takes the same how but defaults to the index and how='left'.

The how= argument decides which keys appear in the result: keep only matches, or keep unmatched rows from one or both sides.

• inner (default): Keeps only keys present in both frames (the intersection).

• left: Keeps all rows from the left frame; unmatched right columns become NaN.

• right: Mirror of left: keeps all rows from the right frame.

• outer: Keeps keys from both (the union); fills missing sides with NaN.

• cross: Cartesian product of all rows (no keys); every left row pairs with every right row.

suffixes= controls the labels appended to overlapping (non-key) column names that exist in both frames, so they don't collide in the result.

• Default is ('_x', '_y'): Left frame's duplicate column gets _x, right's gets _y.

• Only applies to overlapping columns that are not join keys.

• Customize for clarity:

  • e.g. suffixes=('_left', '_right') makes provenance obvious.

  • Passing None for a side keeps that side's names unchanged (and errors if it would collide).

Use duplicated() to flag repeat rows and drop_duplicates() to remove them; both let you target specific columns via subset and control which copy to keep via keep.

• Identify:

  • df.duplicated() returns a boolean Series marking rows that repeat an earlier row.

  • df.duplicated().sum() gives a quick count of duplicates.

• Scope with subset: Compare only chosen columns, e.g. subset=["id"], instead of the whole row.

• Handle with keep: keep="first" (default), keep="last", or keep=False to flag/drop all copies.

• Remove: df.drop_duplicates() returns a cleaned copy; it does not modify in place unless inplace=True.

Use value_counts() on the column: it returns each unique value paired with how many times it occurs, sorted by frequency by default.

• df["col"].value_counts():

  • Returns a Series indexed by category, values = counts, descending order.

  • Excludes NaN unless you pass dropna=False.

• Useful options:

  • normalize=True gives proportions instead of raw counts.

  • sort=False keeps natural order rather than by frequency.

• Related: df.groupby("col").size() achieves the same count when you want grouped output.

Call drop_duplicates(), which returns a new DataFrame with repeated rows removed; tune subset and keep to control what is compared and which copy survives.

• Basic use: df.drop_duplicates() removes rows identical across all columns.

• subset: Restrict the comparison to specific columns, e.g. subset=["email"].

• keep: "first" / "last" keep one copy; False drops every duplicated row entirely.

• Persisting the change: Reassign (df = df.drop_duplicates()) or pass inplace=True; consider reset_index(drop=True) afterward.

duplicated() identifies duplicates by returning a boolean mask, while drop_duplicates() acts on them by returning a DataFrame with the duplicates removed. One reports, the other removes.

• duplicated():

  • Returns a boolean Series (same length as the DataFrame): True where a row repeats an earlier one.

  • Great for inspecting, counting (.sum()), or filtering duplicates to view them.

• drop_duplicates():

  • Returns a DataFrame with duplicate rows removed.

  • Essentially df[~df.duplicated()] under the hood.

• Shared parameters: Both accept subset and keep, so they behave consistently.

unique() returns the actual distinct values, while nunique() returns just the count of them: one gives the list, the other gives the number.

• unique():

  • Returns a NumPy array of the distinct values, in order of appearance.

  • Includes NaN as a value; Series method only.

• nunique():

  • Returns an integer count of distinct values.

  • Excludes NaN by default; pass dropna=False to include it.

  • Works on a DataFrame too, giving a per-column count.

• Rule of thumb: Want the values, use unique(); want how many, use nunique().

Both count occurrences of unique values, but normalize=False (the default) returns raw integer counts while normalize=True returns their relative frequencies (proportions summing to 1).

• Default (counts):

  • Returns absolute frequency per value, sorted descending by default.

  • Good for seeing sample size and cardinality.

• normalize=True (proportions):

  • Divides each count by the total, so values sum to 1.0 (multiply by 100 for percentages).

  • Ideal for comparing category distributions across differently sized groups.

• Shared behavior: Both exclude NaN unless you pass dropna=False.

A Timedelta represents a duration, and Pandas offers several constructors to build them from strings, numbers, or Python objects.

• From a string: pd.Timedelta('2 days 3:00:00') parses human-readable durations.

• From value + unit keywords: pd.Timedelta(days=1, hours=6) or pd.Timedelta(5, unit='h').

• From subtracting datetimes: Subtracting two Timestamp objects yields a Timedelta.

• Vectorized creation: pd.to_timedelta(['1 day', '2 days']) builds a TimedeltaIndex from a list or Series.

Use pd.to_datetime() to parse strings into Pandas datetime objects; it handles single values, lists, and whole columns.

• Basic conversion: pd.to_datetime(df['col']) infers common formats automatically and returns a datetime64 Series.

• Specify format for speed and correctness:

  • format='%Y-%m-%d' avoids ambiguity and parses much faster than inference.

  • Use dayfirst=True for day/month/year strings.

• Handling bad data: errors='coerce' turns unparseable values into NaT instead of raising.

• At read time: pd.read_csv(..., parse_dates=['col']) parses during loading.

A Timedelta is Pandas' representation of a duration or difference between two points in time: the equivalent of Python's datetime.timedelta but with nanosecond precision and vectorized support.

• What it stores: An elapsed time (days, hours, minutes, down to nanoseconds), not an absolute moment.

• How it arises: Subtracting two Timestamp values produces a Timedelta.

• How it's used:

  • Add or subtract it from a Timestamp to shift dates.

  • A column of them has dtype timedelta64[ns].

• Related types: a collection of them forms a TimedeltaIndex.

A DatetimeIndex is an index of timestamps that gives a DataFrame time series superpowers (slicing by date, resampling), and pd.date_range() generates evenly spaced dates to build one.

• What DatetimeIndex enables:

  • Partial-string indexing like df['2024-01'] to select a month.

  • Time-aware operations: .resample(), .asfreq(), .shift().

• Creating with date_range():

  • Give any two of start, end, periods, plus a freq.

  • Frequency strings: 'D' daily, 'B' business days, 'H' hourly, 'M' month-end.

Avoid it by never assigning through chained indexing: use a single .loc[] or .iloc[] call for combined selection-and-assignment, and make an explicit .copy() whenever you want a standalone subset to modify separately.

• Use single-step label/position indexing: df.loc[mask, 'col'] = value instead of df[mask]['col'] = value.

• Copy deliberately: subset = df[df.x > 0].copy() when the subset is a new object you intend to edit.

• Enable Copy-on-Write: pd.options.mode.copy_on_write = True in pandas 2.x removes the ambiguity and the warning altogether.

• What not to do: Don't just set mode.chained_assignment = None to mute it: that hides a genuine correctness risk.

Both return the underlying data as a NumPy array, but .to_numpy() is the modern, recommended method with explicit control, while .values is the older attribute whose behavior is less predictable with extension dtypes.

• .values (legacy): Returns a NumPy array, but for extension types (nullable Int64, Categorical) it may return something non-NumPy or a surprising dtype.

• .to_numpy() (preferred):

  • Explicit and consistent; accepts dtype= and na_value= to control conversion.

  • Documented as the recommended way since Pandas 0.24.

• Guidance: Prefer .to_numpy() in new code for clarity and control over dtype/NA handling.

query() filters rows using a string expression evaluated against the columns, giving a compact, readable alternative to boolean masking.

• Syntax and readability: df.query('age > 30 and city == "NY"') avoids repeating df[...] and lets you use and/or instead of &/|.

• Reference Python variables with @: df.query('age > @threshold') injects an outer variable into the expression.

• When to use it: Great for complex multi-condition filters where the mask syntax gets noisy; can also be faster on large frames via numexpr.

• Caveats: Column names with spaces need backticks; expressions are strings, so no IDE checking or autocomplete.

For a single condition, index directly with the boolean mask; for multiple, combine masks with bitwise operators (parenthesized) or use query() for readability, keeping everything vectorized.

• Single condition: df[df['x'] > 0]: one vectorized comparison, minimal overhead.

• Multiple conditions: df[(df['x'] > 0) & (df['y'] < 5)]: use &/| and wrap each term, since and/or don't work on Series.

• Efficiency tips:

  • Use .isin() instead of chained | equality checks; use .between() for ranges.

  • query() can outperform on large frames and reads cleaner for many conditions.

  • Avoid Python loops or .apply() row-wise for filtering: they defeat vectorization.

Use .where() and .mask() when you want to keep the object's shape but replace values that fail (or meet) a condition, rather than dropping rows like boolean indexing does.

• .where(cond) keeps where True: Values where cond is False are replaced (default NaN, or a supplied other).

• .mask(cond) is the inverse: Replaces values where cond is True; it's .where(~cond).

• When to reach for them:

  • Conditional replacement while preserving shape and index (e.g. cap outliers, blank out invalid entries).

  • Contrast with df[mask], which removes rows entirely.

Use .at/.iat when you need to access or set a single scalar value: they are optimized for that one job and are faster than .loc/.iloc, which are built to handle slices, lists, and boolean masks.

• .at and .iat are scalar-only accessors:

  • .at is label-based (like .loc), .iat is integer-position-based (like .iloc).

  • They bypass the overhead of alignment and indexing logic, so lookups in tight loops are noticeably faster.

• Use .loc/.iloc for anything beyond a single cell: Ranges, multiple columns, boolean masks, or fancy indexing all require .loc/.iloc.

• Rule of thumb: single cell by exact row and column, reach for .at/.iat; otherwise use .loc/.iloc.

.query() lets you filter a DataFrame using a string expression instead of a boolean mask, making complex row-selection conditions more readable and concise.

• It evaluates a boolean expression against columns: Column names are referenced directly as variables, so df.query('age > 30 and city == "NY"') replaces a longer bracketed mask.

• When it is useful:

  • Readability: chained conditions avoid repeating the DataFrame name for every clause.

  • Reference outside variables with the @ prefix, e.g. df.query('age > @threshold').

  • For large frames it can be faster because it can use the numexpr engine.

• Caveats:

  • Column names with spaces or special characters need backticks.

  • The string is dynamically evaluated, so avoid untrusted input.

.where() and .mask() conditionally keep or replace values based on a boolean condition, returning an object of the same shape rather than dropping rows: they are inverses of each other.

• .where(cond, other) keeps values where the condition is True: Where cond is False, values are replaced by other (default NaN).

• .mask(cond, other) is the opposite: It replaces values where cond is True and keeps them where False.

• Shape is preserved: Unlike boolean indexing (df[mask]), which removes rows, these return the same-sized structure with substitutions, which is useful for cleaning or capping values.

DataFrame.filter() selects rows or columns by their labels (names), not by their contents: it filters on the index/column names, whereas boolean filtering selects rows based on the actual data values.

• filter() works on labels along an axis:

  • items=: keep exactly these labels.

  • like=: keep labels containing this substring.

  • regex=: keep labels matching a pattern.

  • axis= chooses columns (default) or rows.

• Boolean filtering works on values: df[df['age'] > 30] keeps rows whose data satisfies a condition; it inspects cell contents, not names.

• Rule of thumb: choosing which columns/rows by name, use .filter(); choosing which rows by their data, use a boolean mask.

A NumPy array is a homogeneous, N-dimensional numeric array with no labels, while a Pandas DataFrame is a labeled, two-dimensional table whose columns can each hold a different data type. Pandas is built on NumPy but adds labels, mixed types, and higher-level data operations.

• Data types:

  • NumPy arrays are homogeneous: every element shares one dtype.

  • DataFrame columns can be heterogeneous: strings, ints, dates side by side.

• Labeling and indexing:

  • Arrays are accessed purely by integer position.

  • DataFrames have a named index and column labels, enabling automatic alignment and label-based access with .loc.

• Functionality:

  • Pandas adds high-level operations: groupby, joins, missing-data handling, and I/O to CSV/SQL.

  • NumPy is lower-level and geared toward fast numerical/matrix computation.

• Relationship: a DataFrame's underlying values are often NumPy arrays, so they interoperate closely.

Pandas was created (by Wes McKinney) to give Python a fast, flexible tool for real-world tabular data: raw Python is too slow and manual, and NumPy lacks labels, heterogeneous columns, and missing-data handling.

• Problem with raw Python: Lists and dicts require hand-written loops for filtering, joining, and aggregation: verbose, error-prone, and slow.

• Problem with NumPy alone:

  • Arrays are homogeneous and position-indexed only, so mixed-type columns and named rows/columns are awkward.

  • No native missing-data or heterogeneous-column support.

• What Pandas adds:

  • Labeled axes with automatic alignment, per-column dtypes, and built-in NaN handling.

  • High-level, vectorized operations (groupby, merge, pivot) that replace loops with concise, C-backed calls.

A DataFrame is essentially a set of NumPy arrays (one or more per dtype) wrapped with labeled row and column indexes plus a rich analysis API.

• Underlying storage: Column values are held in NumPy arrays, so numeric operations get the same vectorized C speed.

• What Pandas adds:

  • Labeled Index on both axes with automatic alignment.

  • Heterogeneous columns (different dtype per column).

  • Missing-data (NaN) semantics, and high-level ops like groupby, merge, and I/O.

• Interop: Convert freely with df.to_numpy() and pd.DataFrame(array).

These parameters control how read_csv() parses columns, types, dates, missing values, and how much it loads at once: setting them correctly avoids costly type inference and reduces memory.

• dtype: Force column types (e.g. {"id": "int32", "code": "category"}) to save memory and skip inference.

• parse_dates: Convert columns to datetime64 at load time instead of parsing later.

• usecols: Read only a subset of columns: less memory and faster I/O.

• chunksize: Return an iterator of DataFrames so you process a huge file in pieces.

• na_values: Define extra strings to treat as NaN (e.g. "N/A", "missing").

• index_col: Use a column (or columns) as the DataFrame index directly.

• Others worth knowing: sep, header, names, nrows, skiprows, and encoding.

The core trade-off is universality and readability (CSV) versus performance, compactness, and type safety (Parquet). CSV is the lowest-common-denominator text format; Parquet is a binary columnar format built for analytics.

• CSV strengths:

  • Human-readable, editable in any tool, and needs no extra dependency.

  • Easy to append and stream line by line.

• CSV weaknesses:

  • No stored schema: dtypes and dates are re-inferred each read.

  • Larger on disk and slower to parse.

• Parquet strengths:

  • Columnar + compressed: smaller files, faster reads, and column pruning via columns=.

  • Preserves dtypes exactly, including categoricals and timezones.

• Parquet weaknesses: Binary (not human-readable) and requires pyarrow or fastparquet.

• Rule of thumb: Use CSV for small files and interchange; use Parquet for large or repeatedly-read analytical data.

CSV is a simple, universal, human-readable text format; Parquet is a compressed, columnar binary format optimized for analytical performance and type fidelity. The choice depends on interoperability needs versus scale and speed.

• Storage layout: CSV is row-oriented text; Parquet is column-oriented binary, which enables per-column compression and selective reads.

• Size: Parquet is typically several times smaller due to columnar compression and encoding.

• Speed: Parquet reads/writes far faster at scale; CSV parsing text is expensive.

• Schema/types: Parquet embeds a typed schema; CSV stores only strings, so types must be inferred.

• Portability: CSV is readable everywhere with no tooling; Parquet needs a compatible library.

• Bottom line: CSV for portability and small data; Parquet for large-scale, performance-critical analytics.

Passing chunksize makes read_csv() return an iterator (a TextFileReader) that yields DataFrames of N rows at a time, so you process a file larger than memory piece by piece instead of loading it all at once.

• Bounded memory: Only one chunk is in memory at a time, so peak usage stays roughly proportional to chunksize, not file size.

• Incremental aggregation: Compute partial results per chunk (counts, sums, filters) and combine at the end.

• Streaming pipelines: Good for filtering or transforming then writing each chunk out (e.g. to Parquet or a database).

• Caveat: Operations needing the whole dataset (global sort, exact median) are harder: you must aggregate across chunks or use a tool like Dask.

Use pd.read_sql() to run a query (or read a table) into a DataFrame and DataFrame.to_sql() to write a DataFrame to a table. Both need a database connection, typically a SQLAlchemy engine.

• Create a connection: Use create_engine() from SQLAlchemy; Pandas accepts the engine as its con argument.

• read_sql():

  • Accepts a SQL query string or a table name; read_sql_query() and read_sql_table() are the explicit variants.

  • Supports params= for safe parameterized queries and chunksize= to iterate large results.

• to_sql():

  • if_exists controls behavior: "fail", "replace", or "append".

  • index=False avoids writing the DataFrame index as a column; chunksize and method="multi" speed up large inserts.

• Best practice: Parameterize queries instead of string-formatting to prevent SQL injection.

read_parquet and read_feather load data from binary columnar formats (Parquet and Arrow/Feather), which are far faster and more space-efficient than CSV and preserve dtypes. Choose them over read_csv whenever you control the file format and care about speed, size, or type fidelity.

• read_parquet:

  • Reads Apache Parquet: compressed, columnar on-disk format (needs pyarrow or fastparquet).

  • Supports column pruning (columns=) and predicate pushdown, so you read only what you need.

  • Great for long-term storage and analytics on large datasets.

• read_feather:

  • Reads the Arrow IPC (Feather) format: optimized for very fast read/write, minimal serialization overhead.

  • Ideal for temporary interchange between processes or caching intermediate results.

• Why prefer them over CSV:

  • They store dtypes (int, float, datetime, categorical) so nothing is re-inferred or lost.

  • Much smaller files and much faster I/O than text-based CSV.

  • CSV still wins for human readability and universal tool compatibility.

pivot() reshapes long data into wide by spreading unique values of a column into new columns, while melt() collapses wide columns back into long key/value rows. They are conceptual inverses: melt undoes pivot and vice versa.

• pivot() (long to wide):

  • Takes index, columns, and values: each unique value in columns becomes its own column.

  • Errors on duplicate index/column pairs; use pivot_table() to aggregate those.

• melt() (wide to long): Uses id_vars (columns to keep) and value_vars (columns to unpivot) into variable and value columns.

• How they invert: melt a wide table on an id column, then pivot back on that id and the variable column to recover the original shape.

Both split a continuous variable into discrete bins, but pd.cut bins by value ranges (equal-width or explicit edges) while pd.qcut bins by quantiles so each bin holds roughly the same number of observations.

• pd.cut:

  • Pass an integer for that many equal-width bins, or a list of explicit edges.

  • Use labels= to name bins (e.g. low/medium/high); returns a Categorical.

  • Good when the boundaries themselves are meaningful (age brackets, score thresholds).

• pd.qcut:

  • Pass an integer for that many quantiles (e.g. 4 for quartiles) or a list of quantile edges.

  • Bins have equal counts but unequal widths.

  • Good for ranking into deciles/quartiles; may fail on duplicate edges (use duplicates='drop').

The core difference is how bin edges are chosen: cut() divides by value range (equal-width or explicit edges), while qcut() divides by quantiles so each bin has roughly equal counts.

• cut():

  • Equal-width bins based on the value range; bin populations can be very uneven.

  • Accepts custom edges, so boundaries are predictable and meaningful.

• qcut():

  • Equal-frequency bins based on quantiles; widths vary with the distribution.

  • Useful for percentile ranking (quartiles, deciles).

• Quick rule: Care about the value ranges: use cut(). Care about equal group sizes: use qcut().

rank() assigns a numerical rank to each value in a Series or DataFrame column, and its method parameter controls how ties (equal values) are resolved.

• Basic behavior:

  • Ranks default to ascending (smallest value gets rank 1); set ascending=False to reverse.

  • NaNs get NaN rank by default; na_option can push them to top/bottom.

• Tie-breaking via method:

  • 'average' (default): ties get the mean of the ranks they span.

  • 'min' / 'max': ties all get the lowest / highest rank in the group.

  • 'first': ranks assigned in order of appearance, breaking ties by position.

  • 'dense': like 'min' but ranks increase by 1 with no gaps between groups.

• Common uses: percentile ranking (pct=True), leaderboards, and ranking within groups via groupby().rank().

nlargest(n, columns) and nsmallest(n, columns) return the n rows with the largest or smallest values in the given column(s), and they are typically faster and clearer than sorting the whole object then slicing.

• What they do:

  • Return the top/bottom n rows already ordered, on a Series or DataFrame column.

  • keep handles ties: 'first', 'last', or 'all' (may return more than n).

• Vs. sort + slice:

  • sort_values().head(n) fully sorts the data (O(n log n)); nlargest uses a partial selection that only tracks the top n, which is cheaper when n is small relative to size.

  • More readable and expresses intent directly.

• Note: for very small data the difference is negligible; the win shows on large frames with small n.

Columns are cheap to add or drop because a DataFrame is column-oriented, while adding rows is expensive: the preferred pattern is to collect rows and build the frame once rather than growing it incrementally.

• Adding columns:

  • df['new'] = values for simple assignment (aligns on index).

  • df.assign(new=...) returns a copy (good for method chaining).

  • df.insert(loc, name, values) to place a column at a specific position.

• Deleting columns/rows:

  • df.drop(columns=[...]) or df.drop(index=[...]) returns a new frame (or inplace=True).

  • del df['col'] or df.pop('col') remove a column in place.

• Adding rows (performance-sensitive):

  • pd.concat([df, new_rows]) appends, but each call copies the whole frame (O(n) each time).

  • Appending in a loop is O(n²): instead collect dicts/lists and build once with pd.DataFrame(records).

  • The deprecated df.append() was removed; use concat.

• Key takeaway: mutating shape returns copies unless in place, and repeated growth reallocates memory, so batch your changes.

Split-apply-combine is the mental model behind groupby(): split the data into groups by key, apply a function to each group independently, then combine the results into a new structure.

1. Split:

   • Partition rows into groups based on one or more keys (df.groupby('col')).

   • No computation happens yet: it produces a lazy GroupBy object.

2. Apply: Run a function on each group: aggregation (sum), transformation (zscore), or filtering.

3. Combine: Stitch the per-group results back together into a Series or DataFrame, indexed by the group keys.

The output shape depends on the apply step: aggregation collapses each group to one row, transformation returns the original shape, filtering drops whole groups.

Split-apply-combine is the strategy Pandas uses to summarize data by category: it splits rows into groups, applies a computation to each group, and combines the outputs back together.

• Split: df.groupby(key) defines the grouping without computing anything eagerly.

• Apply: Aggregate with .agg(), reshape same-length results with .transform(), or select groups with .filter().

• Combine: Results are assembled into a Series/DataFrame keyed by group labels.

• Why it matters: It replaces manual loops over categories with vectorized, readable operations, mirroring SQL's GROUP BY.

agg() accepts a list, dict, or named-aggregation form to compute several summaries at once, either on the same column or different columns.

• List of functions: g['x'].agg(['mean', 'sum', 'max']) produces one column per function.

• Dict per column: g.agg({'x': 'sum', 'y': 'mean'}) applies different aggregations to different columns.

• Named aggregation (preferred): Gives flat, readable output column names and avoids a confusing MultiIndex header.

melt() unpivots a wide DataFrame into long (tidy) form: it turns columns into rows, producing one variable column and one value column while keeping chosen identifier columns.

• Key parameters:

  • id_vars: columns to keep fixed as identifiers.

  • value_vars: columns to unpivot (defaults to all non-id columns).

  • var_name and value_name: rename the resulting variable/value columns.

• Why use it: Tidy long format is easier to filter, group, and plot; it is the inverse of pivot().

Both reshape long data into a wide, spreadsheet-style layout, but pivot() only rearranges and requires unique index/column pairs, while pivot_table() aggregates duplicates using a function.

• pivot():

  • Pure reshape: no aggregation. Raises ValueError if an index/column combination is duplicated.

  • Use when each cell is uniquely determined.

• pivot_table():

  • Aggregates duplicate entries via aggfunc (default 'mean').

  • Supports fill_value, margins (totals), and multiple aggregations.

• Rule of thumb: unique cells use pivot(); possible duplicates or need for summarizing use pivot_table().

They differ by what the applied function returns and thus the output shape: agg() reduces each group to a scalar, transform() returns a like-indexed result the same length as the input, and filter() keeps or drops whole groups.

• agg(): One row per group. Example: df.groupby('k')['x'].agg('sum').

• transform():

  • Broadcasts the group result back to the original rows: same length as input.

  • Ideal for group-wise normalization or filling: df['x'] - df.groupby('k')['x'].transform('mean').

• filter():

  • Takes a predicate on each group and returns the subset of original rows from groups that pass.

  • Example: keep groups with more than 5 rows via lambda g: len(g) > 5.

Named aggregation lets you produce clearly named output columns by mapping each new column to a (source column, function) pair, avoiding messy MultiIndex column names.

• Syntax: Pass keyword arguments where the key is the output name and the value is pd.NamedAgg(column, aggfunc) or a tuple ('col', 'func').

• Benefits: Flat, readable column names; multiple aggregations on the same or different columns in one call.

The key distinction is aggregation: pivot() is a lossless reshape that fails on duplicate index/column pairs, whereas pivot_table() collapses duplicates with an aggregation function.

• Duplicates:

  • pivot() raises if a cell maps to more than one value.

  • pivot_table() summarizes them with aggfunc.

• Extra features of pivot_table(): fill_value for missing cells, margins=True for row/column totals, and a list of aggfuncs.

• Performance: pivot() is lighter when you know cells are unique.

melt() reshapes wide data to long (tidy) format by unpivoting columns into two columns: one holding the former column names (variable) and one holding their values (value).

• Key parameters:

  • id_vars: columns to keep fixed as identifiers.

  • value_vars: columns to unpivot (default: all others).

  • var_name / value_name: rename the output columns.

• When to reshape wide to long:

  • Tidy data for plotting libraries (e.g. seaborn) that expect one observation per row.

  • When many columns really represent the same variable measured under different labels (e.g. monthly columns).

  • It is the inverse of pivot().

They reshape data between columns and index levels: stack() pivots columns into a new inner row-index level (wide to long), and unstack() pivots an index level back into columns (long to wide).

• stack(): columns become rows:

  • Moves the innermost column level down into a new innermost row-index level, producing a longer, narrower object (often a Series).

  • By default drops missing combinations (dropna=True).

• unstack(): rows become columns:

  • Takes a row-index level and spreads it across columns, making the result wider.

  • Pick which level with unstack(level=-1) or a level name.

• They are inverses: stack() then unstack() (same level) round-trips the shape.

• Common use: reshape after a multi-level groupby() to turn one grouping key into columns for easier reading or plotting.

Long (tidy) data has one observation per row with variables as columns; wide data spreads one variable's values across multiple columns. Reshape to long for analysis and grouping, to wide for presentation and cross-variable comparison.

• Tidy/long format:

  • Each variable is a column, each observation a row (e.g. columns date, metric, value).

  • Best for groupby, aggregation, filtering, and most plotting libraries.

• Wide format:

  • One row per entity with each category in its own column (like a spreadsheet).

  • Good for human-readable tables and row-wise comparison.

• How to convert:

  • Wide to long: melt() (or stack()).

  • Long to wide: pivot() / pivot_table() (or unstack()).

• Rule of thumb: reshape to long to compute, reshape to wide to display.

crosstab() is a convenience function for computing a frequency (contingency) table between two or more arrays/Series; it's essentially a specialized pivot_table() whose default aggregation is counting.

• crosstab():

  • Takes array-like inputs directly (not just a DataFrame), e.g. pd.crosstab(df.a, df.b).

  • Defaults to counting occurrences of each row/column combination.

  • normalize=True gives proportions instead of counts.

• pivot_table():

  • Operates on a DataFrame with named columns and defaults to aggfunc='mean'.

  • More general: aggregates a values column with any function.

• Relationship:

  • crosstab() is a thin wrapper over pivot_table(); both support margins for totals.

  • Use crosstab() for quick frequency counts, pivot_table() for aggregating value columns.

aggfunc chooses how values are aggregated within each cell, and margins adds row/column subtotals (an "All" total) to the table.

• aggfunc:

  • Defaults to 'mean'; accepts a name, a function, or a list (e.g. ['sum', 'count']).

  • Can be a dict mapping each values column to its own function.

  • A list produces a hierarchical column index (one block per function).

• margins:

  • margins=True appends totals computed with the same aggfunc across each row and column.

  • margins_name renames the label (default 'All').

  • Note the margin is the aggregate of the raw data, not a sum of the cells (matters for mean).

A GroupBy object is iterable: looping over it yields (key, sub_dataframe) pairs, letting you handle each group with custom logic that doesn't fit a single aggregation.

• How to iterate:

  • for key, group in df.groupby('col'): key is the group label, group is that subset as a DataFrame.

  • With multiple keys the key is a tuple.

  • get_group(key) fetches a single group without looping.

• When it's useful:

  • Side effects: writing each group to a separate file, plotting, or logging.

  • Complex per-group logic that isn't a clean aggregation.

• Caveat: prefer vectorized agg(), transform(), or apply() for computation; manual iteration is slower and should be a last resort.

Data alignment means Pandas matches values by index and column labels (not positions) before operating, and broadcasting means a lower-dimensional object is stretched to match a higher-dimensional one across those labels.

• Alignment is label-based:

  • When you add two Series/DataFrames, Pandas aligns on the union of labels first, then operates on matched pairs.

  • Labels present in one object but not the other produce NaN (unless you supply a fill).

• Broadcasting stretches shapes:

  • A scalar applies to every element; a Series applies across a DataFrame's rows (matching on columns) by default.

  • Use axis=0 to broadcast a Series down the rows (matching on the index instead).

• Practical benefit:

  • You rarely reindex manually: operations "just work" on differently ordered or partially overlapping data.

  • Watch for surprise NaNs from mismatched labels; use methods like add(other, fill_value=0) to control them.

Arithmetic between Pandas objects aligns operands on their index (and column) labels first, computes on matched labels, and yields NaN wherever labels don't overlap: position is irrelevant.

• Union of labels: The result index is the union of both operands' indexes, sorted, regardless of original order.

• Non-overlapping labels become NaN: Any label missing from either side has no partner, so the operation returns NaN there.

• Method form controls fills: Operators like + have no fill option, but a.add(b, fill_value=0) substitutes for missing labels before aligning.

• DataFrame case: Alignment happens on both axes (index and columns) simultaneously.

An Index is the immutable, labeled axis of a Series or DataFrame: it names the rows (and columns), enabling fast lookups, automatic alignment, and meaningful joins/groupings.

• What it is:

  • An ordered, array-like set of labels attached to an axis; immutable so it can be safely shared and hashed.

  • Specialized types exist: RangeIndex, DatetimeIndex, MultiIndex.

• Purpose:

  • Identify and select data by label via .loc rather than by position.

  • Drive automatic alignment in arithmetic, concat, and join.

• How it facilitates operations:

  • A unique/sorted index enables hash- or search-based lookups instead of scans.

  • Time-series slicing, reindex, and resampling all rely on the index being the semantic key.

Use a MultiIndex when your data has more than one natural key per axis (e.g. (country, year) or (ticker, date)): it lets you represent higher-dimensional data in a 2-D frame and slice, aggregate, and reshape along each level.

• When to reach for it:

  • Panel/hierarchical data where rows are identified by a combination of keys.

  • After a multi-key groupby, whose result is naturally a MultiIndex.

• Advantages:

  • Partial indexing: select all rows for one outer key with .loc['USA'].

  • Cross-section selection with .xs(level=...) on an inner level.

  • Easy reshaping between long and wide via stack()/unstack().

  • Aggregation per level with groupby(level=...).

• Caveats:

  • Sort with sort_index() for efficient/unambiguous slicing.

  • Syntax is heavier; flatten with reset_index() when a plain frame is easier.

A MultiIndex is a hierarchical index with two or more levels on a single axis, so each row (or column) is keyed by a tuple of labels: it lets a 2-D DataFrame represent higher-dimensional, grouped data.

• Structure:

  • Each label is a tuple like ('USA', 2020); levels have names and order (outer to inner).

  • Create it with set_index(['a','b']), pd.MultiIndex.from_tuples/from_product, or a multi-key groupby.

• Selecting data:

  • Partial key: .loc['USA'] returns all inner rows.

  • Full key: .loc[('USA', 2020)].

  • Cross-section on an inner level: .xs(2020, level='year').

• Reshaping and aggregating:

  • stack()/unstack() move levels between rows and columns.

  • groupby(level=...) aggregates by a chosen level.

• Best practice: Call sort_index() so slicing is fast and avoids ambiguity warnings.

MultiIndexing (hierarchical indexing) lets an axis have multiple levels of labels, so you can represent higher-dimensional data in a 2D DataFrame and index by combinations of keys.

• Multiple levels per axis: Each row (or column) is keyed by a tuple, e.g. ('2023', 'Q1').

• Created several ways: Via set_index() on multiple columns, groupby() on multiple keys, or pd.MultiIndex.from_tuples().

• Selection uses tuples and slicers: Use .loc[('a', 'b')], .xs() for cross-sections, or pd.IndexSlice for partial slicing.

• Reshaping ties in: .stack() and .unstack() move levels between the row and column axes.

Reindexing conforms a Series or DataFrame to a new set of labels: it keeps values whose labels match, inserts missing labels (filled with NaN or a fill rule), and drops labels not in the new index.

• .reindex() changes labels, not just order: New labels appear as missing; absent labels are dropped.

• Fill options: fill_value for a constant, or method='ffill'/'bfill' for ordered (e.g. time) data.

• Works on either axis: Pass index= and/or columns= to realign rows and columns.

• Common uses: Aligning multiple objects to a shared index, or forcing a complete date range.

A MultiIndex is a hierarchical index with two or more levels, letting a single axis carry compound keys so you can model multi-dimensional data (like grouped or panel data) within a 2D structure.

• Structure: Each label is a tuple across ordered levels (e.g. country, then year).

• When to use it: Naturally grouped/nested data, results of a multi-key groupby(), or pivoted tables where columns also become hierarchical.

• Working with it: Partial selection with .loc['level0'], cross-sections with .xs(), reshape with .unstack().

• Caveat: Keep it sorted (.sort_index()) for efficient and unambiguous slicing; deep nesting can hurt readability.

The category dtype stores a column as a small set of unique values (the categories) plus integer codes pointing to them, which saves memory and speeds up operations on columns with many repeats: it's an efficient encoding of the raw labels, whereas one-hot encoding expands those labels into separate 0/1 columns.

• How it's stored: An array of integer codes plus a lookup of unique categories, so repeated strings aren't stored over and over.

• Why use it: Big memory savings for low-cardinality columns and faster groupby/sort operations; supports ordered categories for meaningful comparisons.

• How to create it: df['col'].astype('category'), optionally with pd.CategoricalDtype(categories=..., ordered=True).

• Relation to one-hot encoding:

  • category is a single compact column; one-hot (pd.get_dummies()) turns categories into multiple binary columns for models that need numeric, non-ordinal features.

  • Use category for storage/analysis efficiency; use one-hot when an ML algorithm requires the expanded form.

Pandas represents categorical data with the category dtype, which stores a small set of unique values (categories) once and encodes each element as an integer code pointing into them, saving memory and enabling ordered semantics.

• How it works internally:

  • A Categorical holds two arrays: the unique categories and integer codes that index them.

  • Repeated string values are stored as small integers instead of full copies.

• Advantages:

  • Memory: huge savings when a column has many repeats but few distinct values.

  • Performance: faster groupby, sorting, and comparisons over integer codes.

  • Ordered categories: set ordered=True to support logical comparisons like < / > (e.g. low < medium < high).

• Trade-offs:

  • Little benefit (or overhead) when cardinality is high (few repeats).

  • Adding values not already in categories requires add_categories first.

Choosing the right dtype matters because it directly controls how much memory a column uses and how fast vectorized operations run: the wrong dtype (often object) forces Python-level loops and bloated storage.

• Memory footprint:

  • Downcasting int64 to int32/int16 or float64 to float32 can halve memory.

  • Low-cardinality strings as category instead of object cut memory dramatically.

• Performance:

  • Numeric dtypes live in contiguous NumPy arrays, enabling fast vectorized/C-level operations.

  • object columns store Python object pointers, so operations fall back to slow per-element Python.

• Correctness and semantics:

  • Dates as datetime64 enable date arithmetic and .dt accessors; as strings they don't.

  • Nullable integer dtype (Int64) keeps integers integral even with missing values.

• Practical check: use df.info(memory_usage='deep') and pd.to_numeric(..., downcast=...) to inspect and optimize.

Convert repetitive, low-cardinality columns to the category dtype with astype('category'): Pandas then stores the distinct values once and represents each row as a compact integer code, cutting memory and speeding up group and sort operations.

• How to create/handle it:

  • df['col'] = df['col'].astype('category') or define a CategoricalDtype for explicit order.

  • Manage categories via .cat.add_categories, .cat.remove_unused_categories, .cat.rename_categories.

• Memory benefit:

  • Each value becomes a 1 to 2 byte code instead of a full Python string object.

  • Biggest gains when rows >> unique values (e.g. a country column across millions of rows).

• Performance benefit:

  • groupby, value_counts, and sorts operate on integer codes.

  • Ordered categories enable meaningful comparisons and sorting.

• When NOT to use it:

  • High-cardinality columns (mostly unique) give no savings and add overhead.

  • Frequent addition of brand-new values requires managing categories.

Strings default to object because Pandas historically had no native string type and NumPy has no variable-length string dtype, so it stores each string as a pointer to a Python str object; datetimes should be explicitly converted with pd.to_datetime to get the efficient datetime64 dtype.

• Why strings are object:

  • NumPy arrays are fixed-width; variable-length text doesn't fit, so Pandas boxes each string as a Python object.

  • This means string operations are Python-level and slower; use the .str accessor for vectorized-style methods.

  • Modern Pandas offers a dedicated string dtype (astype('string')) that is more explicit and supports pd.NA.

• Handling datetime conversion:

  • pd.to_datetime(df['col']) parses strings into datetime64[ns].

  • Pass format= for speed and correctness, and errors='coerce' to turn unparseable values into NaT.

  • After conversion, use the .dt accessor (.dt.year, .dt.dayofweek) and date arithmetic.

The category dtype is a Pandas type for columns that take a limited set of distinct values: it stores those unique values once and encodes rows as integer codes, similar to an enum or a database lookup table.

• What it stores: A categories index of unique values plus integer codes mapping each row.

• Why use it:

  • Memory: repeated labels become small integers instead of duplicated objects.

  • Speed: groupby and sorting run on codes.

  • Semantics: ordered categories (ordered=True) allow meaningful </> comparisons for ranks like sizes or grades.

• Best fit: low-cardinality columns like status, gender, country, or ordinal ratings.

Categorical data in Pandas is data drawn from a fixed, finite set of values (categories), represented by the category dtype which encodes each value as an integer code into a stored list of unique categories.

• Two kinds:

  • Nominal: no inherent order (e.g. color, city).

  • Ordinal: meaningful order (e.g. low/medium/high), set with ordered=True.

• Working with it:

  • Create with astype('category') or pd.Categorical(...).

  • Access category metadata via the .cat accessor (.cat.categories, .cat.codes).

• Why it matters:

  • Saves memory and speeds up aggregations for low-cardinality columns.

  • Preserves order semantics for correct sorting and comparison of ordinal data.

  • Signals intent: this column is a fixed set, not free text.

Pandas dtypes are mostly built on NumPy (int64, float64, bool, datetime64) plus Pandas-specific ones (category, nullable Int64/string); object is the fallback because it holds arbitrary Python objects when nothing more specific fits.

• Main dtypes:

  • Numeric: int64, float64 (and smaller widths like int32).

  • bool for true/false.

  • datetime64[ns] and timedelta64[ns] for time data.

  • category for low-cardinality labels.

  • Nullable extension types: Int64, boolean, string (support pd.NA).

• Why object is a fallback:

  • It stores an array of pointers to Python objects, so it can hold anything (strings, mixed types, lists).

  • Used when no homogeneous native dtype applies (notably text, or a mixed-type column).

  • Cost: operations are Python-level, not vectorized, so it's slower and more memory-heavy.

• Takeaway: prefer specific dtypes (category, string, datetime64) over object wherever possible for speed and memory.

Use category for columns with relatively few distinct values repeated many times (low cardinality): it stores each unique value once and represents rows as small integer codes, cutting memory and speeding up group and comparison operations.

• How it saves memory:

  • Internally it's two parts: categories (the unique values) and codes (int8/int16 pointers into them).

  • A million-row string column of 5 distinct values becomes 5 stored strings plus a million tiny integers.

• How it speeds things up:

  • groupby, value_counts, and joins operate on integer codes rather than full strings.

  • Ordered categoricals enable fast </> comparisons and logical sorting.

• When NOT to use it:

  • High-cardinality columns (mostly unique values): overhead of categories can exceed savings.

  • Adding values outside the category set requires add_categories first, so it's less convenient for frequently changing data.

Downcasting shrinks a DataFrame by casting numeric columns to the smallest dtype that safely holds their range and converting low-cardinality object columns to category, often reducing memory by large factors.

• Numeric downcasting:

  • Use pd.to_numeric(s, downcast="integer") or downcast="float" to pick the smallest fitting type (int64 to int8, float64 to float32).

  • Choose signed vs unsigned based on whether values can be negative.

• Object/string downcasting: Convert repetitive text columns with astype("category") for the biggest wins.

• How to decide safely:

  • Inspect min/max before casting so you don't overflow or truncate.

  • Watch float32 precision loss; only downcast floats when precision allows.

  • Read big files smaller from the start with the dtype= argument in read_csv.

memory_usage() reports the bytes consumed by each column (and the index); passing deep=True makes it account for the actual contents of object dtypes rather than just the pointer array.

• What it returns:

  • A Series of byte counts per column plus an Index entry; sum it for the total.

  • df.info(memory_usage="deep") gives the same info in a summary.

• Why deep=True matters:

  • By default an object column only counts the 8-byte pointers, not the Python string objects they reference, badly understating real usage.

  • deep=True follows those pointers to measure the true footprint (much slower but accurate).

• Practical use: Use it to find which columns to downcast or convert to category/string.

An ordered categorical is a category dtype whose categories have a defined ranking, so pandas treats them as having a meaningful order (like low < medium < high) for comparisons and sorting.

• How to create one:

  • Use pd.Categorical(values, categories=[...], ordered=True) or CategoricalDtype(categories=[...], ordered=True).

  • The order is set by the category list order, not alphabetical.

• What it enables:

  • Element-wise comparisons like s > "medium" work and respect the defined ranking.

  • sort_values() and min/max follow the logical order instead of lexical order.

• Contrast with unordered:

  • An unordered categorical raises TypeError on </> comparisons; only == is allowed.

  • Change ordering later with cat.as_ordered() or cat.reorder_categories().

Both transform data, but they differ in scope: map() is a Series-only method that applies an element-wise mapping, while apply() works on both Series (element-wise) and DataFrames (along an axis, receiving whole rows or columns).

• map() (Series only):

  • Element-wise substitution; accepts a function, a dict, or a Series for lookup-style mapping.

  • Unmatched keys become NaN when using a dict/Series.

• apply() (Series or DataFrame):

  • On a Series: applies a function to each element.

  • On a DataFrame: passes each column (axis=0) or row (axis=1) to the function, so it can aggregate or combine columns.

• Rule of thumb: use map() for simple element-wise mapping on one Series; use apply() when you need row/column-level logic.

Vectorization means operating on entire arrays at once using Pandas/NumPy's C-level implementations, instead of looping element-by-element in Python. It is preferred because it is dramatically faster and more concise.

• Why it's fast:

  • The loop runs in compiled C over contiguous memory, avoiding per-element Python interpreter overhead.

  • Benefits from CPU-friendly bulk operations and, for NumPy dtypes, no per-object boxing.

• Why loops are slow:

  • iterrows() creates a Series per row (expensive, and can coerce dtypes); itertuples() is faster but still a Python loop.

  • apply() is a thin loop wrapper: convenient but not truly vectorized.

• Practical guidance:

  • Reach for arithmetic, boolean masks, .str, .dt, np.where, and .isin() before writing a loop.

  • Loops are acceptable only for genuinely non-vectorizable, row-dependent logic or tiny data.

Use .map() for element-wise mapping on a Series, .applymap() for element-wise operations across every cell of a DataFrame, and .apply() when you need a function over a Series or over whole rows/columns. All three are essentially Python-level loops, so they are slower than vectorized operations.

• .map() (Series): Best for value substitution via a dict/Series or a simple per-element function.

• .applymap() (DataFrame, element-wise): Applies one function to every cell; use for uniform formatting/typing (renamed to DataFrame.map() in newer Pandas).

• .apply() (Series or DataFrame): Use when logic spans a whole row (axis=1) or column (axis=0), or to return an aggregate.

• Performance:

  • Ranking (roughly slowest to fastest): apply(axis=1) < applymap()/map() with a function < vectorized/NumPy.

  • Prefer vectorized expressions; use these methods only when no vectorized equivalent exists.

Use df.apply(func, axis=1) when a computation needs several columns of the same row together and can't be expressed with vectorized column operations. It's convenient but slow, because it materializes each row as a Series and runs your function in a Python loop.

• Good use cases:

  • Complex conditional logic combining multiple columns that's awkward to vectorize.

  • Prototyping or one-off transforms where clarity matters more than speed.

• Performance cost:

  • Each row is boxed into a Series (allocation + dtype coercion), so it's often orders of magnitude slower than column math.

  • Scales poorly: cost grows linearly with rows, with heavy per-row overhead.

• Faster alternatives: Use np.where, np.select, boolean masks, or column arithmetic instead.

You can iterate rows with iterrows() or itertuples(), but both should be a last resort behind vectorization. The key difference is that iterrows() yields each row as a Series, while itertuples() yields lightweight named tuples that are much faster.

• iterrows():

  • Yields (index, Series) pairs; convenient label access but slow.

  • Building a Series per row can upcast dtypes (e.g. ints become floats) since a row mixes column types.

• itertuples():

  • Yields namedtuples; access via row.col_name, typically much faster and preserves dtypes.

  • Use index=False to drop the index; odd/duplicate column names may be renamed positionally.

• Preference: avoid both when possible; if you must loop, prefer itertuples().

They differ by the object they operate on and their granularity: map() is element-wise on a Series, applymap() is element-wise on every cell of a DataFrame, and apply() runs a function over a Series' elements or over a DataFrame's rows/columns.

• map(): Series, element-wise: Accepts a function, dict, or Series; ideal for value lookups and substitutions.

• applymap(): DataFrame, element-wise: Applies one function to every individual cell; now aliased as DataFrame.map() in recent versions.

• apply(): Series or DataFrame, axis-aware: On a DataFrame it hands your function a whole column or row, so it can reduce or combine values, not just transform single cells.

• Memory aid: map = Series cells, applymap = DataFrame cells, apply = Series/DataFrame with axis control.

.str and .dt are accessors that expose vectorized string and datetime methods on a Series, letting you apply element-wise operations to a whole column without Python loops.

• .str for string (object/StringDtype) Series:

  • Mirrors Python str methods across the column: s.str.lower(), .str.strip(), .str.contains(), .str.split(), .str.replace().

  • Handles regex, slicing (s.str[:3]), and NaN safely (skips missing values).

• .dt for datetime/timedelta Series:

  • Exposes datetime components and methods: .dt.year, .dt.month, .dt.dayofweek, .dt.hour, .dt.floor(), .dt.strftime().

  • Requires a datetime dtype; convert first with pd.to_datetime().

• Why use them: They are vectorized and concise, replacing .apply() with per-element Python functions.

Generally no: explicit iteration is slow and un-idiomatic in Pandas, so reach for it only when a computation genuinely cannot be vectorized. Prefer built-in vectorized operations first.

• Why iteration is discouraged:

  • Per-row Python overhead makes it orders of magnitude slower than vectorized code.

  • iterrows() returns each row as a Series, which can silently upcast dtypes (e.g. ints to floats).

  • You should never mutate the frame while iterating over it.

• If you must iterate, keep in mind:

  • itertuples() is much faster than iterrows() and preserves dtypes (returns namedtuples).

  • Iteration should be read-only; collect results in a list and build a new column at the end.

• Alternatives (in order of preference):

  1. Native vectorized operators and methods (.str, .dt, arithmetic, np.where).

  2. groupby/merge/map for aggregation and lookups.

  3. .apply() only when logic can't be expressed vectorially.

Vectorization is applying an operation to an entire array/Series at once, letting Pandas and NumPy execute the loop in compiled C code rather than in Python, one element at a time.

• What it means in practice:

  • Operators and functions are element-wise over aligned data: df['a'] + df['b'] adds whole columns in one call.

  • Index/label alignment happens automatically before the operation.

• Why it's fast: The iteration lives in optimized C over contiguous typed memory, avoiding interpreter overhead per element.

• Common vectorized tools: Arithmetic and comparison operators, boolean masking, np.where, .str/.dt accessors, and aggregation methods.

• Caveat: .apply() is not truly vectorized: it still calls Python per element.

Vectorized operations run in compiled C over whole arrays, while apply() and iterrows() execute Python per row/element, making them slower and more verbose for the same result.

• Performance:

  • Vectorized code avoids per-row interpreter overhead and object creation, often 10x-100x faster.

  • apply() is essentially a loop with function-call overhead; iterrows() also boxes each row into a Series.

• Correctness/clarity:

  • Vectorized expressions are concise, handle index alignment, and preserve dtypes.

  • iterrows() can upcast dtypes across mixed columns.

• When apply() is still fine: Genuinely non-vectorizable logic, or calling a library that only takes scalars; prefer it over an explicit loop, but after native operations.

.pipe() lets you insert your own function into a method chain, passing the DataFrame through as an argument so custom transformations read left-to-right like built-in methods.

• What it does: df.pipe(f, *args) is equivalent to f(df, *args) but keeps the fluent chain intact.

• Why it's useful:

  • Avoids nesting like f(g(h(df))), which reads inside-out; chaining reads in execution order.

  • Encourages small, testable, reusable transformation functions.

  • Reduces intermediate variables and accidental mutation.

• Tip for target position: If the DataFrame isn't the first argument of your function, pass a tuple: df.pipe((f, 'arg_name'), ...).

assign() returns a new DataFrame with one or more added or overwritten columns, making it the chain-friendly way to create columns without mutating the original.

• What it does:

  • Accepts column=value keyword pairs; each value can be an array or a callable receiving the current frame.

  • Returns a copy, leaving the source untouched.

• Why it's useful in chaining:

  • Fits inline in a pipeline where df['x'] = ... (in-place assignment) can't be used.

  • The callable form references the frame as it exists at that step, so it sees earlier chained changes and avoids naming a temp variable.

  • Can define multiple columns in one call; later ones can build on earlier ones (Python 3.6+ order preserved).

replace() substitutes specified values with other values anywhere in a Series or DataFrame, regardless of whether they are missing. It is broader than fillna() (which only targets NaN) and different from map() (which transforms every element of a Series via a mapping or function).

• replace(): value-for-value substitution:

  • Works on any values, not just nulls: df.replace(-999, np.nan) or df.replace({'yes':1,'no':0}).

  • Supports lists, dicts, and regex; leaves non-matched values untouched.

• fillna(): only fills missing values: Targets NaN/NaT only, with a scalar, dict, or method like ffill.

• map(): element-wise transform (Series only): Applies a dict/function to every element; unmatched keys become NaN, unlike replace() which keeps them.

• Rule of thumb: replace() for selective swaps, fillna() for nulls, map() for a full remap of a column.

explode() transforms each element of a list-like value in a column into its own row, replicating the other column values. It is used to flatten nested/collection data into a tidy long format.

• What it does:

  • A cell holding a list [a, b] becomes two rows, one per element, with the index duplicated.

  • Empty lists or NaN produce a single row with NaN.

• When to use it:

  • Normalizing columns that store lists (tags, multiple values per record) before grouping or joining.

  • Often follows a split: df['tags'].str.split(',') then explode('tags').

fillna() replaces nulls with a value you supply or a directional copy (ffill/bfill), while interpolate() computes new estimated values from surrounding data points, respecting order and spacing.

• fillna(): explicit substitution:

  • Scalar, dict, or Series; or method='ffill' to carry the last known value forward.

  • Does not consider the trend between points, just copies or fills a constant.

• interpolate(): computed estimates:

  • Default 'linear' draws a straight line between known neighbors; also 'time', 'polynomial', 'spline'.

  • Best for ordered numeric or time-series data where values change smoothly.

• Choose fillna() for categorical or flat fills, interpolate() when trend continuity matters.

They both fill missing values, but fillna() inserts a value you define (constant or propagated), whereas interpolate() derives values mathematically from neighboring data, making it order-aware.

• Core difference:

  • fillna() is a substitution: it never looks at the numeric trend, only fills with what you give it.

  • interpolate() is estimation: it assumes an underlying pattern (linear by default) between known points.

• Data suited to each:

  • fillna() works on any dtype, including strings and categories.

  • interpolate() is for ordered numeric or time-series data; on unordered rows results are meaningless.

• Edge behavior: By default interpolate() fills interior gaps but leaves leading NaN, since it has no prior point to interpolate from.

Choose based on how much data is missing, why it's missing, and whether the column is critical: drop when missingness is small or rows/columns are unusable, impute when you can't afford to lose data and a reasonable estimate exists.

• Drop when the loss is acceptable:

  • Use df.dropna() when only a few rows are affected or a column is mostly empty (drop it with axis=1).

  • Risk: dropping introduces bias if missingness is not random.

• Impute when data is valuable:

  • Numeric: fill with mean/median (median resists outliers) via fillna().

  • Categorical: fill with mode or a sentinel like "Unknown".

  • Ordered/time series: ffill, bfill, or interpolate().

• Decide with a rule of thumb:

  • Inspect proportion missing per column with df.isna().mean() before deciding.

  • Fit imputation on training data only to avoid leakage.

Detect nulls with isna()/notna(), then handle them by dropping (dropna()) or filling (fillna(), directional fills, or interpolate()) depending on the data's structure.

• Detect:

  • df.isna() returns a boolean mask; df.isna().sum() counts per column.

  • df.info() shows non-null counts quickly.

• Drop: df.dropna() removes affected rows (or columns with axis=1).

• Fill with a value: df.fillna(0) or a statistic like df.fillna(df.mean()).

• Directional fill: ffill() propagates the last valid value forward; bfill() pulls the next valid value back.

• Interpolate: df.interpolate() estimates values from surrounding data (linear by default), best for ordered/numeric series.

Most Pandas aggregations skip missing values by default (skipna=True), so NaN is excluded from the computation rather than poisoning the result: set skipna=False to force propagation.

• Default behavior:

  • sum(), mean(), min(), etc. ignore NaN and compute over the remaining values.

  • mean() divides by the count of non-null values, not the total length.

• Forcing propagation: skipna=False makes any NaN in the data return NaN for the aggregate.

• Edge cases:

  • sum() of an all-NaN series is 0.0, while mean() of it is NaN.

  • count() always excludes NaN; use it to see how many values actually contributed.

  • In groupby, missing values in the grouping key are dropped by default (control with dropna=False).