transferManager does not create subfolders when you uploadDirectory ?

[stant]

Describe the bug

when I uploadDir with subfolders, it uploads all the files but does not create folder entries in S3. I know S3 is just 1 big bucket with no real folders, but should it create folder entries or does it not? can we get that option?

I am expecting them when I get list and then since I don't get them I don't display them in my file manager.
missing folder for thirdfolder/linux/ thirdfolder/mac/ thirdfolder/windows/ ...

aws web screen still shows folders, so maybe it is pulling out folder names from this same output and "displaying" folders it figures out along the way? that is more work. folders explicitly created show firstfolder/ secondfolder/ but xfer manager does not create any subfolders.

listAllObjectsAsync():796 entered listAllObjectsAsync()
listAllObjectsAsync():807 S3Object name firstfolder/
listAllObjectsAsync():807 S3Object name firstfolder/New Folder/
listAllObjectsAsync():807 S3Object name firstfolder/New Folder/whatever/
listAllObjectsAsync():807 S3Object name firstfolder/New Folder/whatever/heythere.txt
listAllObjectsAsync():807 S3Object name firstfolder/New Folder/whatever/tryit.txt
listAllObjectsAsync():807 S3Object name secondfolder/
listAllObjectsAsync():807 S3Object name secondfolder/folder2nd3/
listAllObjectsAsync():807 S3Object name secondfolder/folder2nd3/folder1234/
listAllObjectsAsync():807 S3Object name secondfolder/folder2nd3/folder1234/zip.groovy
listAllObjectsAsync():807 S3Object name thirdfolder/
listAllObjectsAsync():807 S3Object name thirdfolder/README.txt
listAllObjectsAsync():807 S3Object name thirdfolder/diskFree.groovy
missing thirdfolder/linux/
listAllObjectsAsync():807 S3Object name thirdfolder/linux/duSelected.groovy
listAllObjectsAsync():807 S3Object name thirdfolder/linux/git-status.groovy
listAllObjectsAsync():807 S3Object name thirdfolder/linux/grepSelectedFiles.groovy
listAllObjectsAsync():807 S3Object name thirdfolder/linux/pasteAsLinkFiles.groovy
listAllObjectsAsync():807 S3Object name thirdfolder/linux/tar.groovy
listAllObjectsAsync():807 S3Object name thirdfolder/mac/duSelected.groovy
listAllObjectsAsync():807 S3Object name thirdfolder/mac/grepSelectedFiles.groovy
listAllObjectsAsync():807 S3Object name thirdfolder/mac/pasteAsLinkFilesMac.groovy
listAllObjectsAsync():807 S3Object name thirdfolder/mac/tar.groovy
listAllObjectsAsync():807 S3Object name thirdfolder/renameFiles.groovy
listAllObjectsAsync():807 S3Object name thirdfolder/runCommandOnSelectedFiles.groovy
listAllObjectsAsync():807 S3Object name thirdfolder/searchAndSaveToFile.groovy
listAllObjectsAsync():807 S3Object name thirdfolder/setLogLevels.groovy
listAllObjectsAsync():807 S3Object name thirdfolder/watchFolders.groovy
listAllObjectsAsync():807 S3Object name thirdfolder/watchFolders2.groovy
listAllObjectsAsync():807 S3Object name thirdfolder/watchFolders3.groovy
listAllObjectsAsync():807 S3Object name thirdfolder/windows/findStrGrep.groovy
listAllObjectsAsync():807 S3Object name thirdfolder/zip.groovy

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

if I am uploading a folder with subfolders, I would like folder entries, besides I think the first one, to be created also.
missing thirdfolder/linux/
missing thirdfolder/mac/ ...

Current Behavior

no errors reported

Reproduction Steps

public void uploadDirectory( String locFile, String bucketName, final String rmtFileArg ) 
    {
    try {
        String rmtFile = rmtFileArg.replace( "\\", "/" );
        if ( ! rmtFile.endsWith( "/" ) ) rmtFile = rmtFile + "/";
        logger.info( "AwsS3 uploadDirectory locFile =" + locFile + "=   to bucketName =" + bucketName + "   to rmtFile =" + rmtFileArg + "=" );

        DirectoryUpload directoryUpload = transferManager.uploadDirectory(UploadDirectoryRequest.builder()
                .source(Paths.get(locFile))
                .bucket(bucketName)
                .s3Prefix( rmtFile )
                .followSymbolicLinks( true )
                .build());
//        DirectoryUpload directoryUpload = transferManager.uploadDirectory(
//                bucketName,
//            rmtFile, new File(locFile), true ); //recursive);  Cannot Use. maybe is V1?
//    // loop with Transfer.isDone()
//    XferMgrProgress.showTransferProgress(xfer);
//    // or block with Transfer.waitForCompletion()

        CompletedDirectoryUpload completedDirectoryUpload = directoryUpload.completionFuture().join();
        completedDirectoryUpload.failedTransfers()
                .forEach(fail -> logger.fine("Object [" + fail.toString() + "] failed to transfer" ));

Possible Solution

give me a method option to create subfolders as found.

Additional Information/Context

No response

AWS Java SDK version used

software.amazon.awssdk:s3:2.29.29 xfermgr same aws-crt:0.33.5

JDK version used

java version "17.0.5" 2022-10-18 LTS Java(TM) SE Runtime Environment (build 17.0.5+9-LTS-191) Java HotSpot(TM) 64-Bit Server VM (build 17.0.5+9-LTS-191, mixed mode, sharing)

Operating System and version

manjaro linux 6.12.1-4 manjaro 64-bit

[RanVaknin]

Hi @stant ,

I've re-read this issue a few times, and I'm not sure I understand the expected behavior.

Like you mentioned, folders in S3 do not really exist. S3 is a blob storage and any keys with a "/" delimiter will appear as if they are in a directory only in the console. When you write that "missing thirdfolder/linux/" it's because there is no folder named thirdfolder/linux/, its just a prefix you attached to one of your objects so that the S3 console could display that in a directory tree-like fashion.

S3 transfer manager is just a high level interface wrapping simple API operations like putObject and getObject. It wont upload zero byte files to mimic a directory structure because there are no directories in S3. The reason why you are seeing firstfolder is because you manually created it as its own S3 object. But for your other files its not the case.

From the S3 documentation:

In Amazon S3, buckets and objects are the primary resources, and objects are stored in buckets. Amazon S3 has a flat structure instead of a hierarchy like you would see in a file system. However, for the sake of organizational simplicity, the Amazon S3 console supports the folder concept as a means of grouping objects. The console does this by using a shared name prefix for the grouped objects.

And

Important
When you create a folder in Amazon S3, S3 creates a 0-byte object with a key that's set to the folder name that you provided. For example, if you create a folder named photos in your bucket, the Amazon S3 console creates a 0-byte object with the key photos/. The console creates this object to support the idea of folders.

Thanks,
Ran~

[stant]

For example, if you create a folder named photos in your bucket, the Amazon S3 console creates a 0-byte object with the key photos/. The console creates this object to support the idea of folders.

Yes. Right. You Can create a folder like 0-byte object so support the idea of folders.
My question is that transfer mgr uploadDirectory does not seem to do this. Can we get an argument that will tell it to do this for us?

and I don't know what you mean by S3 console creates a 0-byte object. I don't think it really does. Just acts like there is one because it has a subobject?

Per your question, I was hoping uploadDir() would create :
thirdfolder/linux
thirdfolder/mac
thirdfolder/windows
whereever a folder object was needed, otherwise I have to figure it out.

Thanks,
Stan

[RanVaknin]

Hi @stant ,

and I don't know what you mean by S3 console creates a 0-byte object. I don't think it really does. Just acts like there is one because it has a subobject?

Sorry, let me clarify. The AWS Console only makes it seem like there are directories when looking inside of an S3 bucket. But those are just visual, not actual directories.

As discussed, S3 is fundamentally a flat object store that doesn't have a true folder hierarchy. While the S3 console creates 0-byte objects to represent folders for visualization purposes, we believe this should remain a UI-level abstraction rather than being handled by the SDK's data transfer utilities.

If you need folder-like organization in your application, you have a 2 options:

1. Parse the object keys and derive the folder structure programmatically (as the console does)
2. Explicitly create 0-byte objects for folders before uploading if needed

The TransferManager will continue to focus on its core responsibility of efficient file transfer operations.

Thanks for your understanding.

Best regards,
Ran~

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

[stant]

Thanks Ran for the info.