Adding GCS dependency for backend and prompt service #1106
Conversation
backend/pyproject.toml
Outdated
| @@ -33,6 +33,7 @@ dependencies = [ | |||
| "social-auth-app-django==5.3.0", # For OAuth | |||
| "social-auth-core==4.4.2", # For OAuth | |||
| "unstract-sdk~=0.56.0rc4", | |||
| "gcsfs==2024.6.0", | |||
There was a problem hiding this comment.
@harini-venkataraman won't we need the dependencies for Azure and S3 if this is needed? @gaya3-zipstack @hari-kuriakose
There was a problem hiding this comment.
@gaya3-zipstack I got a doubt. Shouldn't we be adding this in the SDK?
There was a problem hiding this comment.
@ritwik-g Adding it in SDK is a baggage for SDK as we are only using fsspec APIs. The integrator can decide what filesystem is required and cann the dependencies accordingly.
Eg, tools need minio and not GCS. Hence adding GCS in SDK will be unnecessary for tools.
However we could use a selective way to install SDK like explicitly mention what we need to be added. We could take it up later....
There was a problem hiding this comment.
Okay but let's make sure to add the Azure and S3 dependencies as well.
There was a problem hiding this comment.
s3 and azure is not used in backend yet, and we can add it when we make the integration.
There was a problem hiding this comment.
@harini-venkataraman @gaya3-zipstack we will be requiring it as soon as we release it to production for on-prem customers. This is not a separate requirement. S3, GCS and Azure storage support for all 3 needs to be present. Testing wise we can test it later. So my suggestion would be to take care of this so that the effort to make it on-prem ready will be minimal. Where ever we need google storage we will need the S3 and Azure as well there.
There was a problem hiding this comment.
Any way this is what I feel. If you think it is better to proceed without the same please go ahead.
for more information, see https://pre-commit.ci
|
|
| @@ -14,12 +14,12 @@ dependencies = [ | |||
| "google-cloud-secret-manager==2.16.1", | |||
| "google-cloud-storage==2.9.0", | |||
| # Filesystem connectors | |||
| "s3fs[boto3]==2024.6.0", # For Minio | |||
| "s3fs[boto3]", # For Minio | |||
There was a problem hiding this comment.
@gaya3-zipstack @harini-venkataraman I think we should pin it to a major version while allowing the minor and patch version selection. Not mentioning any version would not be a good idea.
| "PyDrive2[fsspec]==1.15.4", # For GDrive | ||
| "oauth2client==4.1.3", # For GDrive | ||
| "dropboxdrivefs==1.4.1", # For Dropbox | ||
| "boxfs==0.2.1", # For Box | ||
| "gcsfs==2024.6.0", # For GoogleCloudStorage | ||
| "gcsfs", # For GoogleCloudStorage |
There was a problem hiding this comment.
@gaya3-zipstack @harini-venkataraman I think we should pin it to a major version while allowing the minor and patch version selection. Not mentioning any version would not be a good idea.
What
Adding dependency version needed for Backend and prompt service.
...
Why
This will be needed while the integrator uses fsspec. Corresponding file storage dependency is to be added.
...
How
Pinned the dependency in toml.
...
Can this PR break any existing features. If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)
No, these are additions of dependency
...
Database Migrations
Not applicable.
...
Env Config
Not applicable.
...
Relevant Docs
Related Issues or PRs
Not applicable.
...
Dependencies Versions
Not applicable.
...
Notes on Testing
In backend, this is already present as a transitive dependency through connectors.
Pinned the version.
...
Screenshots
Checklist
I have read and understood the Contribution Guidelines.