[Enhancement] Introduce TxnStateCache for merge commit sync mode #55001

banmoy · 2025-01-13T06:32:56Z

Why I'm doing:

This is the first PR of merge commit sync mode optimization #54995. Introduce TxnStateCache on BE side. You can see #54995 for details

What I'm doing:

Fixes #issue

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

trueeyu · 2025-01-13T07:02:22Z

be/src/util/dynamic_cache.h

@@ -360,7 +377,7 @@ class DynamicCache {
        return ret;
    }

-    mutable std::mutex _lock;


Why use bthread::Mutex?

because the cache may be accessed in the brpc request

be/src/runtime/batch_write/txn_state_cache.cpp

be/src/common/config.h

be/src/runtime/batch_write/batch_write_mgr.cpp

be/src/runtime/batch_write/txn_state_cache.h

kevincai · 2025-01-15T07:37:35Z

be/src/runtime/batch_write/txn_state_cache.cpp

+        return Status::ServiceUnavailable("Transaction state cache is stopped");
+    }
+    TxnStateDynamicCacheEntry* entry = create_if_not_exist ? cache->get_or_create(txn_id, 1) : cache->get(txn_id);
+    if (create_if_not_exist) {


going to set the txn id every time even the entry already exists because of create_if_not_exist = true.

yes. DynamicCache does not support to accept parameters for constructor when creating an entry, so use this way as a workaround

be/src/runtime/batch_write/txn_state_cache.cpp

be/src/runtime/batch_write/txn_state_cache.h

be/src/common/config.h

be/src/runtime/batch_write/batch_write_mgr.cpp

be/src/runtime/batch_write/txn_state_cache.cpp

sonarqubecloud · 2025-01-15T10:55:55Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

…55001 (#55071) This is the second PR of merge commit sync mode optimization #54995. Introduce TxnStateDispatcher on FE side. You can see #54995 for details Signed-off-by: PengFei Li <[email protected]>

…tarRocks#55001 (StarRocks#55071) This is the second PR of merge commit sync mode optimization StarRocks#54995. Introduce TxnStateDispatcher on FE side. You can see StarRocks#54995 for details Signed-off-by: PengFei Li <[email protected]>

Signed-off-by: PengFei Li <[email protected]>

github-actions · 2025-01-16T09:14:52Z

[FE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2025-01-16T09:14:58Z

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2025-01-16T09:25:40Z

[BE Incremental Coverage Report]

✅ pass : 403 / 455 (88.57%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	be/src/runtime/batch_write/batch_write_mgr.h	0	1	00.00%	[53]
🔵	be/src/service/internal_service.cpp	0	5	00.00%	[1282, 1283, 1286, 1290, 1291]
🔵	be/src/http/action/update_config_action.cpp	1	6	16.67%	[333, 334, 335, 336, 338]
🔵	be/src/runtime/batch_write/isomorphic_batch_write.cpp	26	33	78.79%	[404, 426, 427, 439, 440, 441, 453]
🔵	be/src/util/dynamic_cache.h	24	28	85.71%	[306, 321, 342, 344]
🔵	be/src/runtime/batch_write/txn_state_cache.h	19	22	86.36%	[130, 131, 132]
🔵	be/src/runtime/batch_write/batch_write_mgr.cpp	26	29	89.66%	[269, 270, 271]
🔵	be/src/runtime/batch_write/txn_state_cache.cpp	305	329	92.71%	[36, 114, 122, 140, 172, 173, 217, 221, 268, 269, 270, 282, 283, 301, 302, 304, 316, 317, 335, 398, 467, 468, 469, 470]
🔵	be/src/runtime/exec_env.cpp	2	2	100.00%	[]

…rRocks#55001) Signed-off-by: PengFei Li <[email protected]>

…kport #55001) (#55150) Signed-off-by: PengFei Li <[email protected]>

…kport #55001) (#55149) Signed-off-by: PengFei Li <[email protected]>

banmoy requested review from a team as code owners January 13, 2025 06:32

wanpengfei-git added the PROTO-REVIEW label Jan 13, 2025

wanpengfei-git requested a review from a team January 13, 2025 06:33

mergify bot assigned banmoy Jan 13, 2025

banmoy mentioned this pull request Jan 13, 2025

Optimize merge commit sync mode #54995

Open

trueeyu reviewed Jan 13, 2025

View reviewed changes

trueeyu reviewed Jan 14, 2025

View reviewed changes

be/src/runtime/batch_write/txn_state_cache.cpp Show resolved Hide resolved

trueeyu reviewed Jan 14, 2025

View reviewed changes

be/src/common/config.h Show resolved Hide resolved

wyb reviewed Jan 14, 2025

View reviewed changes

be/src/runtime/batch_write/batch_write_mgr.cpp Outdated Show resolved Hide resolved

banmoy force-pushed the txn_state_cache branch 4 times, most recently from 3a9b0ee to d761fda Compare January 15, 2025 01:32

trueeyu previously approved these changes Jan 15, 2025

View reviewed changes

kevincai reviewed Jan 15, 2025

View reviewed changes

wyb reviewed Jan 15, 2025

View reviewed changes

kevincai reviewed Jan 15, 2025

View reviewed changes

be/src/runtime/batch_write/txn_state_cache.cpp Show resolved Hide resolved

kevincai reviewed Jan 15, 2025

View reviewed changes

be/src/runtime/batch_write/txn_state_cache.cpp Outdated Show resolved Hide resolved

banmoy dismissed trueeyu’s stale review via b64bf9d January 15, 2025 10:30

banmoy force-pushed the txn_state_cache branch 2 times, most recently from 0e8b88f to 55a3ea1 Compare January 15, 2025 10:48

banmoy force-pushed the txn_state_cache branch from 55a3ea1 to f47c46a Compare January 15, 2025 11:17

wanpengfei-git removed the PROTO-REVIEW label Jan 15, 2025

banmoy added 13 commits January 16, 2025 13:28

Add BE implementation

b9fe4da

Signed-off-by: PengFei Li <[email protected]>

[BE] Fix config

c6bf462

Signed-off-by: PengFei Li <[email protected]>

BE add txn state poller

64aab38

Signed-off-by: PengFei Li <[email protected]>

fix compile

5eced3b

Signed-off-by: PengFei Li <[email protected]>

BE skip poll

cd8a1fa

Signed-off-by: PengFei Li <[email protected]>

enable trace log for test

b6343af

Signed-off-by: PengFei Li <[email protected]>

Improve log

9c646f0

Signed-off-by: PengFei Li <[email protected]>

rename some configure

5ac7149

Signed-off-by: PengFei Li <[email protected]>

remove unused parameters of update_transaction_state

740b9d6

Signed-off-by: PengFei Li <[email protected]>

Fix comments

3bdfd48

Signed-off-by: PengFei Li <[email protected]>

DynamicCache use template Lock

ffeb0b0

Signed-off-by: PengFei Li <[email protected]>

Support to clean txn state

c6fc6a5

Signed-off-by: PengFei Li <[email protected]>

fix test

b098460

Signed-off-by: PengFei Li <[email protected]>

banmoy dismissed kevincai’s stale review via b098460 January 16, 2025 05:29

banmoy force-pushed the txn_state_cache branch from b959b0a to b098460 Compare January 16, 2025 05:29

kevincai approved these changes Jan 16, 2025

View reviewed changes

wyb approved these changes Jan 16, 2025

View reviewed changes

wyb enabled auto-merge (squash) January 16, 2025 09:26

trueeyu approved these changes Jan 16, 2025

View reviewed changes

wyb merged commit 5d145b0 into StarRocks:main Jan 16, 2025
50 checks passed

banmoy added a commit to banmoy/starrocks that referenced this pull request Jan 16, 2025

[Enhancement] Introduce TxnStateCache for merge commit sync mode (Sta…

83df20d

…rRocks#55001) Signed-off-by: PengFei Li <[email protected]>

banmoy mentioned this pull request Jan 16, 2025

[Enhancement] Introduce TxnStateCache for merge commit sync mode (backport #55001) #55149

Merged

24 tasks

banmoy added a commit to banmoy/starrocks that referenced this pull request Jan 16, 2025

[Enhancement] Introduce TxnStateCache for merge commit sync mode (Sta…

7eaf4ff

…rRocks#55001) Signed-off-by: PengFei Li <[email protected]>

banmoy mentioned this pull request Jan 16, 2025

[Enhancement] Introduce TxnStateCache for merge commit sync mode (backport #55001) #55150

Merged

24 tasks

wyb pushed a commit that referenced this pull request Jan 16, 2025

[Enhancement] Introduce TxnStateCache for merge commit sync mode (bac…

faf0e38

…kport #55001) (#55150) Signed-off-by: PengFei Li <[email protected]>

wyb pushed a commit that referenced this pull request Jan 16, 2025

[Enhancement] Introduce TxnStateCache for merge commit sync mode (bac…

8224e71

…kport #55001) (#55149) Signed-off-by: PengFei Li <[email protected]>

github-actions bot added the 3.4-merged label Jan 16, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement] Introduce TxnStateCache for merge commit sync mode #55001

[Enhancement] Introduce TxnStateCache for merge commit sync mode #55001

banmoy commented Jan 13, 2025 •

edited

Loading

trueeyu Jan 13, 2025

banmoy Jan 13, 2025

kevincai Jan 15, 2025

banmoy Jan 15, 2025

sonarqubecloud bot commented Jan 15, 2025

github-actions bot commented Jan 16, 2025

github-actions bot commented Jan 16, 2025

github-actions bot commented Jan 16, 2025

[Enhancement] Introduce TxnStateCache for merge commit sync mode #55001

[Enhancement] Introduce TxnStateCache for merge commit sync mode #55001

Conversation

banmoy commented Jan 13, 2025 • edited Loading

Why I'm doing:

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

trueeyu Jan 13, 2025

Choose a reason for hiding this comment

banmoy Jan 13, 2025

Choose a reason for hiding this comment

kevincai Jan 15, 2025

Choose a reason for hiding this comment

banmoy Jan 15, 2025

Choose a reason for hiding this comment

sonarqubecloud bot commented Jan 15, 2025

Quality Gate passed

github-actions bot commented Jan 16, 2025

[FE Incremental Coverage Report]

github-actions bot commented Jan 16, 2025

[Java-Extensions Incremental Coverage Report]

github-actions bot commented Jan 16, 2025

[BE Incremental Coverage Report]

file detail

banmoy commented Jan 13, 2025 •

edited

Loading