How django minio storage Large file uploading?

[c0d33py]

Has anyone successfully implemented large file uploads using the 'django-minio-storage' package? I'm encountering some challenges with the configuration and would appreciate any insights into optimizing the upload process and handling large files effectively in Django.

[thomasf]

For large files it probably would be wise to drop down to the raw minio python client and generate a presigned upload URL/post policy that a web browser can use to upload directly to minio without passing all the data through Python at all.

[c0d33py]

Thanks for you response. everything has been working fine. However, I've run into a bit of a confusion when it comes to saving the uploaded file data to the default storage.

In my current implementation, I have a method _save_uploaded_file_data that should ideally save the uploaded file data to the default storage after it has been successfully uploaded to MinIO using a presigned URL. Here's the relevant part of the code:

models.py
class FileModel(models.Model):
    uploaded_file = models.FileField(upload_to='uploads)

minio.py
class MinioUploader:
    def __init__(self, file):
        self.client = default_storage.client
        self.bucket_name = settings.MINIO_STORAGE_MEDIA_BUCKET_NAME
        self.file = file
        self.put_presigned_url = self._get_put_presigned_url()

    def _get_put_presigned_url(self):
        return self.client.presigned_put_object(
            self.bucket_name,
            self.tus_file.filename,
            expires=timedelta(minutes=2),
        )

    def upload_file(self, file_path):
        file_data = self._read_file_data(file_path)
        response = self._upload_to_presigned_url(file_data)

        if response.status_code == 200:
            self._save_uploaded_file_data(file_data)

        return response

    def _read_file_data(self, file_path):
        with open(file_path, 'rb') as file:
            file_data = file.read()
        return file_data

    def _upload_to_presigned_url(self, file_data):
        headers = {
            'Cache-Control': settings.MINIO_STORAGE_MEDIA_OBJECT_METADATA.get(
                'Cache-Control'
            ),
        }

        response = requests.put(
            self.put_presigned_url,
            data=file_data,
            headers=headers,
        )
        return response

    def _save_uploaded_file_data(self, file_data):
        uploaded_file_name = self.file.filename
        uploaded_file_path = f"{settings.MEDIA_URL}{uploaded_file_name}"

        # Save the uploaded file data to a local file
        with default_storage.open(uploaded_file_path, 'wb') as file:
            file.write(file_data)

        print(f"Uploaded file data saved: {uploaded_file_path}")

While this code seems straightforward, I'm unsure about the correctness of my approach. Specifically, I'm not entirely confident about how to correctly save the uploaded file data to the default storage (which might be local storage, for example).

Pls could provide some guidance or insight into the correct way to handle this scenario, I would greatly appreciate it. Any tips, code snippets, or best practices related to this situation would be extremely helpful.