Too much diagnostics for quick commands

[dmpetrov]

Description

I run a diff for two very small datasets. The output is a bit overwhelming 😅 I'm not sure I need any diagnostic for query that run instantly (few seconds).

We need some smarter way of reporting the progress. Like - report it only if a query take more than let say 5 seconds.

>>> diff.show()
Processed: 1 rows [00:00, 387.29 rows/s]
Generated: 3 rows [00:00, 1252.65 rows/s]
Processed: 1 rows [00:00, 295.87 rows/s]
Generated: 3 rows [00:00, 937.76 rows/s]
Processed: 1 rows [00:00, 418.76 rows/s]
Generated: 3 rows [00:00, 1354.46 rows/s]
Processed: 1 rows [00:00, 370.78 rows/s]
Generated: 3 rows [00:00, 1222.00 rows/s]
Cleanup: 7 tables [00:00, 3716.00 tables/s]
    file file file    file file      file                      file     file score
  source path size version etag is_latest             last_modified location
0     s1   p1    0       1   e1         1 1970-01-01 00:00:00+00:00     None     1
1     s2   p2    0       1   e1         1 1970-01-01 00:00:00+00:00     None     2
2     s3   p3    0       1   e1         1 1970-01-01 00:00:00+00:00     None     3

Here I'd like to see the final result only like:

>>> diff.show()
    file file file    file file      file                      file     file score
  source path size version etag is_latest             last_modified location
0     s1   p1    0       1   e1         1 1970-01-01 00:00:00+00:00     None     1
1     s2   p2    0       1   e1         1 1970-01-01 00:00:00+00:00     None     2
2     s3   p3    0       1   e1         1 1970-01-01 00:00:00+00:00     None     3

[dmpetrov]

It's also related to clean up message. I see this message in almost any command which is annoying 🙂

Cleanup: 2 tables [00:00, 4454.92 tables/s]

[mattseddon]

We could change the Tqdm progress bars so that they disappear once completed. Jump to near the end of the following videos to see what I'm talking about:

Current state:

Screen.Recording.2025-01-06.at.3.38.50.pm.mov

Dropping progress bar on completion:

Screen.Recording.2025-01-06.at.3.37.45.pm.mov

Note: Even if we don't want to do this for all progress bars we could enable that option for cleanup progress.

[dmpetrov]

@mattseddon good idea! I still have a feeling that we need more. But this looks like a great first step.

[mattseddon]

We can reopen if the above change wasn't enough