refactor: migrate exploratory data analysis examples to polars

[ugohuche]

No description provided.

[brentomagic]

Thanks for taking on this larger EDA migration — diamonds/Enum, group_by+pivot, is_null() for cancelled flights, and skimpy usage are in the right direction. Before merge, the notebook still reads as a partial conversion: a lot of code is polars while much of the teaching text (and one key example) is still pandas/R.

Blocking

1. Unusual values: prose and code disagree
   Option 2 recommends replacing unusual values with missing (pd.NA / null) while keeping rows. The following cell does option 1 instead:

   diamonds2=diamonds.filter(~((pl.col("y") <3) | (pl.col("y") >20)))

   That drops rows. Please implement the recommended approach, e.g.:

   diamonds2=diamonds.with_columns(
   pl.when((pl.col("y") <3) | (pl.col("y") >20))
   .then(None)
   .otherwise(pl.col("y"))
   .alias("y")
   )

   Also update the markdown examples (diamonds.loc[...], pd.NA) to polars.

2. Prerequisites list a non-existent package
   Text asks for polars-profiling, which isn’t a real/installable package here (and isn’t used in the notebook). Drop it or replace with whatever tool you actually intend (e.g. ydata-profiling, if supported).

3. Large leftover pandas/R teaching text
   Code and prose are out of sync in several places; readers will follow the wrong API:

   • “We pluck them out with pandas”
   • is.na() (R) next to is_null() code
   • pd.crosstab() prose above a group_by/pivot cell
   • Section still titled “pandas built-in tools for EDA” / df.info() while showing describe()
   • Crosstab styling prose still describes pd.crosstab() + background_gradient() but code builds a polars pivot and only uses .to_pandas().style...
   • “Exploratory Plotting with polars” heading, then “pandas has some built-in plotting…”
   • Exercises still say “together with pandas”

   Please do a full prose pass so every claim matches the polars code (or honestly document intentional to_pandas() bridges for styling/plotting).

Medium

4. Deprecated polars APIs (raise DeprecationWarning today):
   • pivot(..., columns=...) → on=...
   • .melt(id_vars=...) → .unpivot(index=...)
5. Flights load — prefer pl.read_csv(url) over pd.read_csv + pl.from_pandas (pandas isn’t needed here).
6. max_counts cell — markdown still talks as if this is dataframe styling (max_counts()); the code just takes column maxima. Align text and intent (or restore a styling example via to_pandas().style.highlight_max() and say so).
7. Kernelspec display_name: ".venv" — local env noise; prefer a generic Python 3 kernel name.

Nit / process

8. Commit author metadata is Your Name <you@example@example.com> — please configure git user.name/email before further commits on this branch.

Happy to re-review once the unusual-values example and the prose pass are fixed.

[brentomagic]

Review notes

Thanks for the EDA migration — core polars conversions for diamonds, grouping/pivots, and cancelled-flight flags look good. Before merge:

Blocking

1. Unusual values example is wrong — prose says replace with missing; code filters rows away. Use with_columns(pl.when(...).then(None)...) and update the pandas markdown snippets.
2. Remove polars-profiling from prerequisites (not a real package / unused).
3. Prose pass needed — many cells still teach pandas/R (pd.crosstab, is.na(), “pandas built-in tools”, etc.) while code is polars.

Medium

• pivot(columns=) → on=; melt → unpivot
• Load flights with pl.read_csv
• Fix max_counts teaching text vs code
• Drop .venv kernelspec display name

Full review is on the PR as changes requested.

[meshackamadi]

Summary

• Unusual Values Fix (Blocking)
• Prerequisites Cleanup (Blocking)
• Deprecated Polars APIs
• Direct CSV Loading
• Prose Pass (Blocking & Medium)
• Metadata Cleanup

File Modified

• Exploratory-data-analysis.ipynb