Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[da-vinci][dvc] Flag enablement for automatic subscription of partitions #1332

Open
wants to merge 35 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
d282196
Initial commit
kristyelee Nov 20, 2024
e7aca2e
Merge branch 'linkedin:main' into kristy_lee/650_davinci
kristyelee Nov 20, 2024
ec82b2c
Test Commit
kristyelee Nov 25, 2024
5fa6390
Test Commit
kristyelee Nov 25, 2024
6601afd
Test unsubscription of partitions
kristyelee Nov 26, 2024
5b18fe9
Rewrite unsubscription of partitions
kristyelee Nov 26, 2024
e5e6046
Merge branch 'linkedin:main' into kristy_lee/650_davinci
kristyelee Nov 26, 2024
ef49d50
Modify Partition Removal
kristyelee Nov 26, 2024
0059955
Partition adjustment in bootstrapping [DaVinciBackend]
kristyelee Nov 27, 2024
b32d787
Merge branch 'linkedin:main' into kristy_lee/650_davinci
kristyelee Nov 29, 2024
1fa3eba
[dvc] Partition Difference Calculation and Update to Partition Subscr…
kristyelee Dec 2, 2024
321fb40
Integration Test [In Writing]
kristyelee Dec 4, 2024
52a557c
Partition Removal
kristyelee Dec 5, 2024
c5c9f16
Integration Test [In Writing]
kristyelee Dec 5, 2024
b520ab8
Integration Test [In Writing]
kristyelee Dec 6, 2024
262bbc6
Integration Test [In Writing]
kristyelee Dec 9, 2024
699c200
[dvc] Partition Difference Calculation and Update to Partition Subscr…
kristyelee Dec 9, 2024
002d55c
[dvc] Partition Difference Calculation and Update to Partition Subscr…
kristyelee Dec 9, 2024
fd9fa02
[dvc] Config flag enabled for subscribing on disk partitions automati…
kristyelee Dec 11, 2024
69d619a
[dvc] Config flag enabled for subscribing on disk partitions automati…
kristyelee Dec 12, 2024
ac32242
[dvc] Config flag enabled for subscribing on disk partitions automati…
kristyelee Dec 16, 2024
9554248
[dvc] Config flag enabled for subscribing on disk partitions automati…
kristyelee Dec 16, 2024
7b20b28
[dvc] Config flag enabled for subscribing on disk partitions automati…
kristyelee Dec 16, 2024
9bfe418
[dvc] Config flag enabled for subscribing on disk partition automatic…
kristyelee Jan 9, 2025
7ba15f0
[dvc] Config flag enabled for subscribing on disk partition automatic…
kristyelee Jan 12, 2025
b194128
Merge branch 'linkedin:main' into kristy_lee/650_davinci
kristyelee Jan 13, 2025
a0ba24b
[dvc] Config flag enabled for subscribing on disk partitions automati…
kristyelee Jan 13, 2025
b28feda
Merge branch 'linkedin:main' into kristy_lee/650_davinci
kristyelee Jan 21, 2025
2af0a75
Merge branch 'linkedin:main' into kristy_lee/650_davinci
kristyelee Jan 24, 2025
6f34d1b
[dvc] Config flag enabled for subscribing on disk partitions automati…
kristyelee Jan 27, 2025
15f2fa1
Unit test
kristyelee Jan 27, 2025
4cb909f
Unit test
kristyelee Jan 29, 2025
c278353
Unit test
kristyelee Jan 30, 2025
aee0931
Unit test
kristyelee Jan 31, 2025
0e82518
[dvc] Config flag for subscribing on disk partitions automatically. U…
kristyelee Feb 1, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,7 @@ public DaVinciBackend(
String pid = Utils.getPid();
String instanceSuffix =
configLoader.getCombinedProperties().getString(PUSH_STATUS_INSTANCE_NAME_SUFFIX, (pid == null ? "NA" : pid));
// Current instance name.
String instanceName = Utils.getHostName() + "_" + instanceSuffix;

// Fetch latest update schema's protocol ID for Push Status Store from Router.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,13 @@ synchronized CompletableFuture<Void> subscribe(
// Recreate store config that was potentially deleted by unsubscribe.
config.store();
}

ComplementSet<Integer> unassignedPartitionSet = ComplementSet.newSet(subscription);
kvargha marked this conversation as resolved.
Show resolved Hide resolved
// Set of partitions that are not included in the submitted set of partitions.
unassignedPartitionSet.removeAll(partitions);

subscription.addAll(partitions);
unsubscribe(unassignedPartitionSet);

if (daVinciFutureVersion == null) {
trySubscribeDaVinciFutureVersion();
Expand Down
Loading