[FEAT] Remote File Storage for Prompt Studio #827

harini-venkataraman · 2024-11-05T17:58:18Z

What

PR under progress and will be pushing changes itratively.
Includes the following,

Common Util to include FS SDK APIs into prompt studio
Prompt Studio specific utility methods for Upload and fetch content methods.
Used FS APIs for Indexing
Delete method for Prompt Studio
...

Why

To support application for k8 deployement
...

How

Including implementation of FS APIs from SDK.
...

Can this PR break any existing features. If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)

Most of the pieces are under feature flag and deprecations, but since it is a breaking change, we can expect regressions in corner cases.
...

Database Migrations

Not applicable.
...

Env Config

Not applicable.
...

Related Issues or PRs

Notes on Testing

Screenshots

Run and indexing - with feature flag enabled
Indexing

Checklist

I have read and understood the Contribution Guidelines.

…unstract into feature/remote_storage

backend/file_management/file_management_helper.py

backend/utils/file_storage/helpers/prompt_studio_file_helper.py

backend/utils/file_storage/common_utils.py

backend/utils/file_storage/helpers/prompt_studio_file_helper.py

backend/utils/file_storage/helpers/common_file_helper.py

unstract/core/tests/file_storage/test_fs_permanent.py

unstract/core/pyproject.toml

backend/utils/file_storage/helpers/common_file_helper.py

backend/prompt_studio/prompt_studio_core_v2/prompt_studio_helper.py

backend/utils/file_storage/helpers/prompt_studio_file_helper.py

backend/utils/constants.py

for more information, see https://pre-commit.ci

gaya3-zipstack

Looks good.

backend/utils/file_storage/helpers/common_file_helper.py

platform-service/pyproject.toml

gaya3-zipstack

Added comments to revert changes on platform-service sdk upgrade.

gaya3-zipstack

Looks fine

for more information, see https://pre-commit.ci

github-actions · 2024-12-02T11:49:01Z

filepath	function	$$\textcolor{#23d18b}{\tt{passed}}$$	SUBTOTAL
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_logs}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_cleanup}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_cleanup\_skip}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_client\_init}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_get\_image\_exists}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_get\_image}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_get\_container\_run\_config}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_get\_container\_run\_config\_without\_mount}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_run\_container}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$		$$\textcolor{#23d18b}{\tt{9}}$$	$$\textcolor{#23d18b}{\tt{9}}$$

sonarqubecloud · 2024-12-02T11:49:32Z

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

gaya3-zipstack and others added 12 commits October 22, 2024 12:23

Add more tests

9374a79

Commit pdm.lock changes

5a2dbd0

Check in with improvements

3a1b94b

Merge branch 'feature/remote_storage' of https://github.com/Zipstack/…

945b215

…unstract into feature/remote_storage

Commit pdm.lock changes

a90e2ed

Add permanent storage here

83e32b6

Merge branch 'feature/remote_storage' of https://github.com/Zipstack/…

ffdaf65

…unstract into feature/remote_storage

Commit pdm.lock changes

f6e2bb8

Add skeleton for temporary fs

c64cb0a

Merge branch 'feature/remote_storage' of https://github.com/Zipstack/…

d070bb1

…unstract into feature/remote_storage

Initial commit for Prompt studio file storage changes

107476b

Prompt studio changes

d4ee2a8

harini-venkataraman requested review from hari-kuriakose, tahierhussain, kirtimanmishrazipstack and gaya3-zipstack November 5, 2024 17:58

harini-venkataraman marked this pull request as ready for review November 5, 2024 17:59

chandrasekharan-zipstack reviewed Nov 6, 2024

View reviewed changes

kirtimanmishrazipstack and others added 2 commits November 12, 2024 16:55

Merge branch 'main' into feat/remote-storage-prompt-studio

d146896

Commit pdm.lock changes

7381cf2

kirtimanmishrazipstack requested changes Nov 12, 2024

View reviewed changes

backend/utils/file_storage/helpers/common_file_helper.py Outdated Show resolved Hide resolved

kirtimanmishrazipstack and others added 2 commits November 13, 2024 15:20

Merge branch 'main' into feat/remote-storage-prompt-studio

eb0e1aa

Commit pdm.lock changes

6eb9ef0

kirtimanmishrazipstack requested changes Nov 13, 2024

View reviewed changes

backend/utils/file_storage/helpers/common_file_helper.py Outdated Show resolved Hide resolved

kirtimanmishrazipstack requested changes Nov 13, 2024

View reviewed changes

backend/utils/file_storage/helpers/common_file_helper.py Outdated Show resolved Hide resolved

kirtimanmishrazipstack and others added 5 commits November 19, 2024 23:14

Merge branch 'main' into feat/remote-storage-prompt-studio

e852d70

Commit pdm.lock changes

1007977

Handling legacy file path

ebf5412

Remove redundant tests

b068cea

Remove redundant tests

7af6b78

harini-venkataraman added 3 commits November 26, 2024 13:00

Remove unused exceptions

2b716cc

Revert unused changes

dd2c671

Enable feature flag for indexing

76f4d21

gaya3-zipstack reviewed Nov 26, 2024

View reviewed changes

backend/prompt_studio/prompt_studio_core_v2/prompt_studio_helper.py Outdated Show resolved Hide resolved

gaya3-zipstack reviewed Nov 26, 2024

View reviewed changes

backend/prompt_studio/prompt_studio_core_v2/prompt_studio_helper.py Outdated Show resolved Hide resolved

gaya3-zipstack reviewed Nov 26, 2024

View reviewed changes

backend/prompt_studio/prompt_studio_core_v2/prompt_studio_helper.py Outdated Show resolved Hide resolved

Commit pdm.lock changes

59479a1

chandrasekharan-zipstack reviewed Nov 26, 2024

View reviewed changes

backend/utils/file_storage/helpers/prompt_studio_file_helper.py Show resolved Hide resolved

backend/utils/constants.py Outdated Show resolved Hide resolved

harini-venkataraman and others added 5 commits November 26, 2024 14:00

Adding feature flags to dynamic indexer

a8341bf

[pre-commit.ci] auto fixes from pre-commit.com hooks

c9c83db

for more information, see https://pre-commit.ci

Addition of doc strings

e9b7111

Changing imports for Feature Flags

8ad905c

[pre-commit.ci] auto fixes from pre-commit.com hooks

eb412f4

for more information, see https://pre-commit.ci

chandrasekharan-zipstack approved these changes Nov 26, 2024

View reviewed changes

gaya3-zipstack approved these changes Nov 26, 2024

View reviewed changes

kirtimanmishrazipstack reviewed Dec 2, 2024

View reviewed changes

backend/utils/file_storage/helpers/common_file_helper.py Outdated Show resolved Hide resolved

kirtimanmishrazipstack reviewed Dec 2, 2024

View reviewed changes

backend/utils/file_storage/helpers/common_file_helper.py Outdated Show resolved Hide resolved

kirtimanmishrazipstack approved these changes Dec 2, 2024

View reviewed changes

gaya3-zipstack reviewed Dec 2, 2024

View reviewed changes

platform-service/pyproject.toml Outdated Show resolved Hide resolved

gaya3-zipstack requested changes Dec 2, 2024

View reviewed changes

harini-venkataraman and others added 2 commits December 2, 2024 14:31

Reverting SDK to rc1

e12caae

Commit pdm.lock changes

85a2df6

gaya3-zipstack approved these changes Dec 2, 2024

View reviewed changes

gaya3-zipstack and others added 3 commits December 2, 2024 17:12

Merge branch 'main' into feat/remote-storage-prompt-studio

f2f1620

Roll back pdm lock file changes

1e362d8

[pre-commit.ci] auto fixes from pre-commit.com hooks

9781f5e

for more information, see https://pre-commit.ci

gaya3-zipstack merged commit 6a0a856 into main Dec 2, 2024
6 checks passed

gaya3-zipstack deleted the feat/remote-storage-prompt-studio branch December 2, 2024 12:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FEAT] Remote File Storage for Prompt Studio #827

[FEAT] Remote File Storage for Prompt Studio #827

harini-venkataraman commented Nov 5, 2024 •

edited

Loading

gaya3-zipstack left a comment

gaya3-zipstack left a comment

gaya3-zipstack left a comment

github-actions bot commented Dec 2, 2024

sonarqubecloud bot commented Dec 2, 2024

[FEAT] Remote File Storage for Prompt Studio #827

[FEAT] Remote File Storage for Prompt Studio #827

Conversation

harini-venkataraman commented Nov 5, 2024 • edited Loading

What

Why

How

Can this PR break any existing features. If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)

Database Migrations

Env Config

Related Issues or PRs

Notes on Testing

Screenshots

Checklist

gaya3-zipstack left a comment

Choose a reason for hiding this comment

gaya3-zipstack left a comment

Choose a reason for hiding this comment

gaya3-zipstack left a comment

Choose a reason for hiding this comment

github-actions bot commented Dec 2, 2024

sonarqubecloud bot commented Dec 2, 2024

Quality Gate passed

harini-venkataraman commented Nov 5, 2024 •

edited

Loading