Empty-named files / directories need special care

[dmpetrov]

Description

There is a bucket that contains directory examples with files. It looks like this directory was created in cloud console UI - it has a empty-name record for this directory as far as I can see. This is a common way of creating empty dirs in such storages.

To handle this correctly, we need to treat these empty-named files in a special way - they should look like dirs. This will help prevent issues when working with files and folders in the bucket.

Issue 1. When I list this directory I see only this empty-name record. It looks like we need to handle these empty files / directories in a special way.

from datachain import DataChain

dc = DataChain.from_storage("gs://mybucket/examples").save("myds")
dc.show()
print("Files: ", dc.count())

                         file      file file              file              file  \
                       source      path size           version              etag
0  gs://mybucket  examples    0  1735506238856683  COu7/cbwzYoDEAE=
       file                             file     file
  is_latest                    last_modified location
0         1 2024-12-29 21:03:58.859000+00:00     None
Files:  1

The result is expected if I list it as a dir:

from datachain import DataChain

dc = DataChain.from_storage("gs://mybucket/examples/").save("myds")
dc.show()
print("Files: ", dc.count())

                          file                                               file  \
                        source                                               path
0   gs://mybucket  examples/videos/HoldingPen_...
1   gs://mybucket  examples/videos/HoldingPen_...
...
Files:  171

Issue 2. System signals seem broken for this special file / dir

dc = DataChain.from_storage("gs://mybucket/examples").save("myds")
dc.shuffle().show()

datachain.lib.signal_schema.SignalResolvingError: cannot resolve signal name 'sys.rand': is not found

Version Info

0.8.3
Python 3.10.8

[shcheklein]

Relevant link: https://cloud.google.com/storage/docs/objects#simulated-folders

These directories indeed were created using console UI.