Best practice for third party libraries and packages

[lorentzenchr]

If one writes a library/package that does some data mangling, and one wants to be able to switch backends in the future, ibis seems to be a perfect fit.

The documentation mainly focuses on direct/human users. Are there recommendations for the usage of and development with ibis in a 3rd party library?

To give concrete examples:

1. One needs to open a connection to some backend. Should a library create such a connection when loading the library (import mylibrary) and then use it inside of all of its classes and functions? In that case, the connection would be a global (and internal) variable.
2. Say a function def myfun(data) receives some in-memory data, creates a temporary table, calulcates and returns a single number. Should the function then first register the temporary table (create_table, read_in_memory or read_pandas ...) and then drop_table before returning?

[cpcloud]

Hi @lorentzenchr!

To answer your questions:

Should a library create such a connection when loading the library (import mylibrary) and then use it inside of all of its classes and functions?

Almost certainly no, if only to avoid debugging headaches associated with holding a complex stateful object like a database connection in a module global variable. Additionally, with such a connection it's not clear that you could ever have more than one instance of the corresponding backend.

Say a function def myfun(data) receives some in-memory data, creates a temporary table, calulcates and returns a single number. Should the function then first register the temporary table (create_table, read_in_memory or read_pandas ...) and then drop_table before returning?

I think that's hard to answer without more context about the use case/application. We recently added a .cache() method that may be the right tool for your use case: it won't drop the temp table when the function exits, but it does drop all temp tables created by .cache() when your session ends (depending on what session means for the backend).

[lorentzenchr]

@cpcloud Thanks for having had a look. I‘ve to say, I‘m now even more confused.
Let‘s say my library offers a public function that a user or another library might call:

def fun(df):
    # For the time being, we assume a pandas df and use pandas API,
    # but would love to use ibis
    df["new"] = 1
    return df

How could a library then use ibis that seems to need a connection, e.g. the default con to duckdb?

[cpcloud]

I think I made some assumptions in previous answer that were unwarranted.

Let's start over.

Here are some questions that I think will help answer your questions. The more concrete the examples you can give, the better we can help.

1. Are you building a new 3rd party backend?
2. Are you providing ibis as an API to end users as part of something else?

It would be helpful if you could describe what the end goal of what you're doing is!

[lorentzenchr]

Are you building a new 3rd party backend?

No.

2. Are you providing ibis as an API to end users as part of something else?

No, as far as I can tell.

It‘s more like seaborn. To be concrete, this question arose while developing https://github.com/lorentzenchr/model-diagnostics. Meanwhile, I decided to go with polars.

[cpcloud]

Got it.

I would definitely put your use case in a third category: APIs that use ibis's expressions API as a library and not necessarily for interactive analytics. visidata using the vdsql extension falls squarely into this bucket as well.

Here are some high-level best practices with ibis:

1. Avoid digging around in ibis.expr.* for the APIs you need. There's likely a way to do what you need with a more public (and probably more generic) API
2. Use to_x methods for turning ibis expressions into other things: to_pandas(), to_pyarrow(), to_csv(), to_parquet()
3. Use ibis.memtable for querying in-memory objects like pyarrow tables and pandas dataframes

I'm sure others have thoughts on these and other best practices. @gforsyth @jcrist @lostmygithubaccount probably have thoughts here.

Let's consider putting these into our docs once folks have commented.

[lostmygithubaccount]

@cpcloud is it worth distinguishing "Ibis as an API" and "Ibis as a SDK"? I would think they are similar and agree with your point above -- library developers should largely be using the public API, but they'll probably want at least one level lower (e.g. getting access to the database connection as asked for in #6636 and similar requests)

there's probably a better way for us to message "Ibis as an API for data practitioners" versus "Ibis as a SDK for library developers", where there will obviously be a lot of overlap between the usage and people