Suggested improvements for the Polars user guide book

[MrPowers]

The Polars user guide book is awesome. Great job making Polars approachable to newbies like me!

I reproduced a lot of the examples in the polars-fun repo so people have easy access to Jupyter notebooks where they can run these computations too.

Here are some of the issues / discussion points on the user guide:

Getting Started

This code snippet has a small typo (change scan_scv to scan_csv). The snippet doesn't work on my computer when the typo is fixed.

df = (pl.scan_scv("https://j.mp/iriscsv")
      .filter(pl.col("sepal_length") > 5)
      .groupby("species")
      .agg(pl.all().sum())
      .collect()
)

The error is "RuntimeError: Any(Io(Os { code: 2, kind: NotFound, message: "No such file or directory" }))" which I don't understand.

Expressions

Here's a code snippet in the guide:

df = df[
    [
        pl.col("names").n_unique().alias("unique_names_1"),
        pl.col("names").unique().count().alias("unique_names_2"),
    ]
]
print(df)

My personal preferences are to avoid bracket notation and avoid reassigning df. Perhaps we could use this snippet:

res_df = df.select(
    [
        pl.col("names").n_unique().alias("unique_names_1"),
        pl.col("names").unique().count().alias("unique_names_2"),
    ]
)
print(res_df)

Feel free to push back on this one.

Indexing

I don't really understand this section. I'd prefer to rewrite this section with queries that are run on an example DataFrame, similar to the Polars expressions section.

The Polars design decision to not have an index seems like one of the main differentiators from Pandas. I'd like to expand upon "They are not needed. Not having them makes things easier. Convince me otherwise" cause it seems so important.

Here are some specific questions I'd find interesting:

• setting a Pandas index makes certain queries faster. Can Polars perform just as fast as Pandas, even when the index isn't set?
• Pandas indexes make things more complicated. How can we demonstrate this with code snippets?

Other points to hit

I have other points to hit, but will save them for another PR. Let me know if you find this helpful. I'm really excited about this project and want to help make it as accessible and compelling as possible for new users to try!

[ritchie46]

Thanks for your suggestions. Additions to the polars user guide are most welcome, and I often struggle to keep information up to date, or fail to understand which information is most valuable. So please help. ;)

Any PRs can be made on the polars-book repo itself. Same counts for issues: https://github.com/pola-rs/polars-book

s code snippet has a small typo (change scan_scv to scan_csv). The snippet doesn't work on my computer when the typo is fixed.

I think this is a bug. The scan_csv cannot work on internet data, (as we cannot lazily scan over http). So we should read eagerly instead:

df = (pl.read_csv("https://j.mp/iriscsv")
      .lazy()
      .filter(pl.col("sepal_length") > 5)
      .groupby("species")
      .agg(pl.all().sum())
      .collect()
)

My personal preferences are to avoid bracket notation and avoid reassigning df. Perhaps we could use this snippet:

I totally agree. The bracket notation was from a time when we were still experimenting with the API. Now using select is idiomatic polars. I think it is far more readable! So yes, Could you make a PR for your proposed snippet?

Indexing
I don't really understand this section. I'd prefer to rewrite this section with queries that are run on an example DataFrame, similar to the Polars expressions section.

The reason for this section is to explain that because polars does not have an index, accessing a polars dataframe is simpler than in pandas. We don't need loc, at, iloc to disambiguate indices.

However, in idiomatic polars, you don't need to index as much as you need in polars, as you'd be using the expression API much more. Maybe I should have clarified that in the start of that section?

setting a Pandas index makes certain queries faster. Can Polars perform just as fast as Pandas, even when the index isn't set?

On a micro benchmark pandas probably wins if the index is pre-set (Maybe it doesn't, polars joins are really fast). But indices are only useful if they can be used. If you join on a column you don't have an index, the index is precomputed in vain. When you do a groupby or join operation you create a new table, again needing to recompute the index.

So on a macro level, you often need to recompute indices and I believe often in vain, so polars just computes them when they are needed. This saves RAM and likely compute.

Pandas indexes make things more complicated. How can we demonstrate this with code snippets?

Pandas indexes are often used as a means to modify data instead of only to improve performance. Often when I read

some_pandas_query().reset_index().some_other_operation() the index determines how the final dataframe looks. I often find this very counter-intuitive. But I really don't know much about pandas indexing, I think we could show some multi-index kung-fu of pandas and then show how we would do it in polars.

I have other points to hit, but will save them for another PR. Let me know if you find this helpful. I'm really excited about this project and want to help make it as accessible and compelling as possible for new users to try!

Keep em coming! 😄

[MrPowers]

Thank you for the detailed responses, started updating the book here: pola-rs/polars-book#108