Understanding what to do with a joined table in interactive mode, plus feedback about the documentation

[astrojuanlu]

I'm trying to follow the Aggregating and joining data tutorial with my own data and I'm a bit frustrated about joined tables. When I get to this point:

countries_and_gdp = countries.inner_join(gdp,
                                         predicates=countries['iso_alpha3'] == gdp['country_code'])
countries_and_gdp[countries, gdp]

The line countries_and_gdp[countries, gdp] looks really weird, and it's not explained in the paragraphs below. The tutorial then proceeds to do a very similar join without really explaining what countries_and_gdp[countries, gdp] is supposed to mean, or what other options are available.

By skimming other parts of the documentation, and especially https://ibis-project.org/ibis-for-sql-programmers/#join-projection, I understood that I should be able to pick what columns do I want from a joined table. However, if I try to do any combination of countries_and_gdp[countries, gdp.any_column], I get an annoying IbisError: Table operation is not yet materialized. On the other hand, if I instead pick a subset of columns like countries_and_gdp[countries, gdp["col_name_1", "col_name_2"]], I get RelationError: The expression "[MASSIVE_BLOB]" does not fully originate from dependencies of the table expression..

On a more general note, I think the choice of using interactive mode for the tutorial and then assuming it's turned off for all the rest of the documentation makes for a very poor reader experience, since it's very difficult to bridge the simple (but incomplete) steps from the tutorial with the in-depth (but overwhelming) guides from the "Ibis for SQL Programmers". The rest of the resources are not really useful for me as a beginner, since the User Guide seems to cover some explanations (as per the Diátaxis framework) that I am not interested in at the moment, and the "Execution Backends" are reference material (again following the aforementioned division).

[cpcloud]

Hi @astrojuanlu!

Sorry to hear you're frustrated. I agree with everything you've said here. Let's try to suss out some concrete follow-ups.

Does it make sense to have a nav section corresponding to each of the sections of the Diataxis framewwork?

I think most or all of the documentation pieces clearly fit in one of these categories, so it wouldn't be too much effort to move them around a bit.

To address some specific points/frustrations:

I get an annoying IbisError: Table operation is not yet materialized

Yes, this has been annoying and confusing to many users. We've removed this limitation and the corresponding materialize() API. This change will be available in the next release, 3.0.0.

I get RelationError: The expression "[MASSIVE_BLOB]" does not fully originate from dependencies of the table expression.

The MASSIVE BLOB part is also a source of frustration for some folks. We don't have any concrete plans to improve the repr, but that's only because no one has been vocal about it or taken interest in improving it. It seems like we should address this sooner rather than later.

I think the choice of using interactive mode for the tutorial and then assuming it's turned off for all the rest of the documentation makes for a very poor reader experience.

Agreed. What do you think about interactive mode for every tutorial, and non-interactive for reference, how-to, and explanation?

[astrojuanlu]

Hi @cpcloud , thanks for the quick reply! Really appreciate it.

Does it make sense to have a nav section corresponding to each of the sections of the Diataxis framework?

Absolutely! Several projects are already formally adopting variations of this 4-pane structure, see for example NumPy, FPM

Yes, this has been annoying and confusing to many users. We've removed this limitation and the corresponding materialize() API. This change will be available in the next release, 3.0.0.

I'm relieved that I'm not the only one then. Looking forward to trying out the next version.

Agreed. What do you think about interactive mode for every tutorial, and non-interactive for reference, how-to, and explanation?

Makes sense. Another option could be to have a switch that the docs reader could flip, like this:

https://sphinx-inline-tabs.readthedocs.io/

(similar to how tabs are used in https://ibis-project.org/#abstract-over-sql-dialects)

In any case, maybe the last part of the tutorial could include some steps that teach how to transition to non-interactive mode, so one is ready to read the rest of the documentation by the end of the tutorial.

[astrojuanlu]

To follow up on some of the topics I brought up, in light of the new Ibis 3.0 release:

api: materialize is removed. Joins with overlapping columns now have suffixes.

I haven't tried the other points yet, but as soon as I do I will either continue the conversation here or open issues for specific problems.