refactor: system transactions #17406

mustafauzunn · 2025-01-16T07:34:37Z

Description:

The final PR to support the new format of GossipEvent where transactions will be passed as repeated bytes

Related issue(s):

Fixes #16665

Notes for reviewer:

Checklist

Documented (Code comments, README, etc.)
Tested (unit, integration, etc.)

Signed-off-by: Mustafa Uzun <[email protected]>

…5-system-transaction-refactoring # Conflicts: # platform-sdk/swirlds-platform-core/src/main/java/com/swirlds/platform/eventhandling/DefaultTransactionPrehandler.java # platform-sdk/swirlds-platform-core/src/main/java/com/swirlds/platform/state/SwirldStateManager.java # platform-sdk/swirlds-platform-core/src/main/java/com/swirlds/platform/state/TransactionHandler.java

Signed-off-by: Mustafa Uzun <[email protected]>

codecov · 2025-01-16T07:50:16Z

Codecov Report

Attention: Patch coverage is 55.33981% with 46 lines in your changes missing coverage. Please review.

Project coverage is 68.87%. Comparing base (311c068) to head (d567b00).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...-app/src/main/java/com/hedera/node/app/Hedera.java	0.00%	18 Missing ⚠️
...om/swirlds/platform/metrics/AddedEventMetrics.java	20.00%	7 Missing and 1 partial ⚠️
...om/swirlds/platform/pool/TransactionPoolNexus.java	28.57%	5 Missing ⚠️
.../com/swirlds/platform/builder/PlatformBuilder.java	0.00%	4 Missing ⚠️
.../swirlds/platform/system/events/UnsignedEvent.java	50.00%	3 Missing ⚠️
.../java/com/swirlds/platform/cli/DiagramCommand.java	0.00%	2 Missing ⚠️
...irlds/platform/system/transaction/Transaction.java	33.33%	1 Missing and 1 partial ⚠️
.../demo/consistency/ConsistencyTestingToolRound.java	0.00%	1 Missing ⚠️
...re/src/main/java/com/swirlds/platform/Browser.java	0.00%	1 Missing ⚠️
...vent/validation/DefaultInternalEventValidator.java	0.00%	1 Missing ⚠️
... and 1 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #17406      +/-   ##
============================================
- Coverage     68.91%   68.87%   -0.05%     
+ Complexity    22667    22663       -4     
============================================
  Files          2613     2613              
  Lines         97892    97916      +24     
  Branches      10156    10157       +1     
============================================
- Hits          67463    67438      -25     
- Misses        26613    26662      +49     
  Partials       3816     3816

Files with missing lines	Coverage Δ
...ows/standalone/impl/StandaloneDispatchFactory.java	`98.41% <100.00%> (-0.05%)`	⬇️
...swirlds/platform/builder/ApplicationCallbacks.java	`100.00% <ø> (ø)`
...m/components/consensus/DefaultConsensusEngine.java	`88.23% <100.00%> (+0.73%)`	⬆️
...ction/system/SystemTransactionExtractionUtils.java	`93.75% <ø> (-6.25%)`	⬇️
...java/com/swirlds/platform/event/PlatformEvent.java	`87.61% <100.00%> (-1.77%)`	⬇️
...form/event/creation/tipset/TipsetEventCreator.java	`81.87% <ø> (ø)`
...wirlds/platform/event/hashing/PbjStreamHasher.java	`93.10% <100.00%> (+1.79%)`	⬆️
...ent/resubmitter/DefaultTransactionResubmitter.java	`100.00% <ø> (ø)`
.../swirlds/platform/pool/DefaultTransactionPool.java	`57.14% <100.00%> (ø)`
...com/swirlds/platform/state/SwirldStateManager.java	`96.61% <100.00%> (-0.06%)`	⬇️
... and 17 more

... and 11 files with indirect coverage changes

codacy-production · 2025-01-16T07:50:31Z

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ -0.01% (target: -1.00%)	✅ 77.66%

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (`3e8fa1f`)	97675	71128	72.82%
Head commit (`50733eb`)	97689 (+14)	71124 (-4)	72.81% (-0.01%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#17406)	94	73	77.66%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings Change summary preferences

_{Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more}

IvanKavaldzhiev · 2025-01-16T12:08:18Z

...form-core/src/main/java/com/swirlds/platform/eventhandling/DefaultTransactionPrehandler.java

@@ -91,6 +91,8 @@ public DefaultTransactionPrehandler(
    public List<ScopedSystemTransaction<StateSignatureTransaction>> prehandleApplicationTransactions(
            @NonNull final PlatformEvent event) {
        final long startTime = time.nanoTime();
+        final List<ScopedSystemTransaction<StateSignatureTransaction>> scopedSystemTransactions = new ArrayList<>();
+        Consumer<ScopedSystemTransaction<StateSignatureTransaction>> consumer = scopedSystemTransactions::add;


I think this field can be final

IvanKavaldzhiev · 2025-01-16T12:09:03Z

...dk/swirlds-platform-core/src/main/java/com/swirlds/platform/pool/DefaultTransactionPool.java

@@ -47,8 +45,8 @@ public DefaultTransactionPool(@NonNull final TransactionPoolNexus transactionPoo
    @Override
    public void submitSystemTransaction(@NonNull final StateSignatureTransaction payload) {
        Objects.requireNonNull(payload);
-        transactionPoolNexus.submitTransaction(
-                new EventTransaction(new OneOf<>(TransactionOneOfType.STATE_SIGNATURE_TRANSACTION, payload)), true);
+        Bytes payloadBytes = StateSignatureTransaction.PROTOBUF.toBytes(payload);


This should be final

Signed-off-by: Mustafa Uzun <[email protected]>

github-actions · 2025-01-20T09:38:19Z

Node: HAPI Test (Token) Results

28 files 2 errors 26 suites 15s ⏱️
197 tests 0 ✅ 0 💤 197 ❌
304 runs 0 ✅ 0 💤 304 ❌

For more details on these parsing errors and failures, see this check.

Results for commit a754bcb.

♻️ This comment has been updated with latest results.

Signed-off-by: Mustafa Uzun <[email protected]>

…ring' into 16665-system-transaction-refactoring

Signed-off-by: Mustafa Uzun <[email protected]>

github-actions · 2025-01-20T14:55:09Z

Node: HAPI Test (Smart Contract) Results

117 files 2 errors 115 suites 36m 3s ⏱️
169 tests 169 ✅ 0 💤 0 ❌
1 072 runs 1 072 ✅ 0 💤 0 ❌

For more details on these parsing errors, see this check.

Results for commit b64fc96.

♻️ This comment has been updated with latest results.

Signed-off-by: Ivan Kavaldzhiev <[email protected]>

…ring' into 16665-system-transaction-refactoring

IvanKavaldzhiev · 2025-01-21T08:35:31Z

...-platform-core/src/main/java/com/swirlds/platform/system/transaction/TransactionWrapper.java

    /** The hash of the transaction */
    private Bytes hash;
+    /** The bytes of the transaction */
+    private Bytes transaction;


This field should be removed and the Bytes put inside EventTransaction

IvanKavaldzhiev · 2025-01-21T08:35:40Z

...-platform-core/src/main/java/com/swirlds/platform/system/transaction/TransactionWrapper.java

-        this.payload = EventTransaction.newBuilder()
-                .applicationTransaction(payloadBytes)
-                .build();
+        this.transaction = payloadBytes;


I think we should revert this change

Signed-off-by: Ivan Kavaldzhiev <[email protected]>

… transaction format Signed-off-by: Ivan Kavaldzhiev <[email protected]>

github-actions · 2025-01-21T12:46:17Z

Node: HAPI Test (Time Consuming) Results

4 files 1 errors 3 suites 22m 7s ⏱️
17 tests 17 ✅ 0 💤 0 ❌
18 runs 18 ✅ 0 💤 0 ❌

For more details on these parsing errors, see this check.

Results for commit 50733eb.

Signed-off-by: Ivan Kavaldzhiev <[email protected]>

Signed-off-by: Mustafa Uzun <[email protected]>

github-actions · 2025-01-21T13:13:45Z

Node: HAPI Test (Restart) Results

5 files 1 errors 4 suites 0s ⏱️
4 tests 0 ✅ 0 💤 4 ❌
8 runs 0 ✅ 0 💤 8 ❌

For more details on these parsing errors and failures, see this check.

Results for commit dcaeb9c.

github-actions · 2025-01-21T13:13:52Z

Node: HAPI Test (Node Death Reconnect) Results

2 tests 0 ✅ 0s ⏱️
2 suites 0 💤
3 files 2 ❌
1 errors

For more details on these parsing errors and failures, see this check.

Results for commit dcaeb9c.

codacy-production · 2025-01-21T13:19:29Z

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ -0.04% (target: -1.00%)	✅ 57.28%

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (`311c068`)	97675	71128	72.82%
Head commit (`d567b00`)	97699 (+24)	71103 (-25)	72.78% (-0.04%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#17406)	103	59	57.28%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings Change summary preferences

_{Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more}

Signed-off-by: Mustafa Uzun <[email protected]>

lpetrovic05 · 2025-01-21T13:39:03Z

...irlds-platform-core/src/main/java/com/swirlds/platform/builder/PlatformComponentBuilder.java

@@ -1030,7 +1030,7 @@ public PlatformComponentBuilder withTransactionPool(@NonNull final TransactionPo
    @NonNull
    public TransactionPool buildTransactionPool() {
        if (transactionPool == null) {
-            transactionPool = new DefaultTransactionPool(blocks.transactionPoolNexus());
+            transactionPool = new DefaultTransactionPool(blocks.transactionPoolNexus(), blocks.stateLifecycles());


this dependency should be removed

lpetrovic05 · 2025-01-21T13:41:29Z

...-sdk/swirlds-platform-core/src/main/java/com/swirlds/platform/metrics/AddedEventMetrics.java

-            if (transaction.isSystem()) {
-                numSysTrans++;
-                sysSize += transaction.getSize();
-                avgBytesPerTransactionSys.update(transaction.getSize());
-            } else {
+            // TODO: how to know if a transaction is a system transaction?
+            if (isNewFormat) {
                numAppTrans++;
-                appSize += transaction.getSize();
-                avgBytesPerTransaction.update(transaction.getSize());
+                appSize += transaction.getBytesSize();
+                avgBytesPerTransaction.update(transaction.getBytesSize());
+            } else {
+                if (transaction.isSystem()) {
+                    numSysTrans++;
+                    sysSize += transaction.getSize();
+                    avgBytesPerTransactionSys.update(transaction.getSize());
+                } else {
+                    numAppTrans++;
+                    appSize += transaction.getSize();
+                    avgBytesPerTransaction.update(transaction.getSize());
+                }


simplify to just track all transactions equally, remove metric for system ones

lpetrovic05 · 2025-01-21T13:42:21Z

...dk/swirlds-platform-core/src/main/java/com/swirlds/platform/pool/DefaultTransactionPool.java

-        transactionPoolNexus.submitTransaction(
-                new EventTransaction(new OneOf<>(TransactionOneOfType.STATE_SIGNATURE_TRANSACTION, payload)), true);
+        final Bytes payloadBytes = stateLifecycles.encodeSystemTransaction(payload);
+        transactionPoolNexus.submitTransaction(payloadBytes, true, true);


this transformation should be done by a wiring transformer

lpetrovic05 · 2025-01-21T13:44:08Z

...-sdk/swirlds-platform-core/src/main/java/com/swirlds/platform/pool/TransactionPoolNexus.java

+    public synchronized boolean submitTransaction(
+            @NonNull final Bytes transaction, final boolean priority, final boolean isSystem) {


priority & isSystem are always equal, can be simplified

lpetrovic05 · 2025-01-21T13:46:06Z

...-sdk/swirlds-platform-core/src/main/java/com/swirlds/platform/pool/TransactionPoolNexus.java

-        if (TransactionUtils.getLegacyTransactionSize(transaction) > maximumTransactionSize) {
+        if (TransactionUtils.getLegacyTransactionSize(appTransaction) > maximumTransactionSize) {


@lpetrovic05 create a ticket for this cleanup

lpetrovic05 · 2025-01-21T13:46:16Z

...form-sdk/swirlds-platform-core/src/main/java/com/swirlds/platform/state/StateLifecycles.java

+    default Bytes encodeSystemTransaction(@NonNull final StateSignatureTransaction transaction) {
+        throw new IllegalStateException("Invoke the method on the appropriate SwirldMain implementation!");
+    }


should be removed

lpetrovic05 · 2025-01-21T13:53:28Z

...swirlds-platform-core/src/main/java/com/swirlds/platform/system/transaction/Transaction.java

+    Bytes getTransactionBytes();
+


there should be a single method call to get the transaction

lpetrovic05 · 2025-01-21T13:55:05Z

...swirlds-platform-core/src/main/java/com/swirlds/platform/system/transaction/Transaction.java

@@ -39,14 +39,18 @@ public sealed interface Transaction permits ConsensusTransaction {
    @Deprecated
    EventTransaction getTransaction();


seems we dont need this method anymore

lpetrovic05 · 2025-01-21T14:01:31Z

...swirlds-platform-core/src/main/java/com/swirlds/platform/system/transaction/Transaction.java

+    int getBytesSize();
+


there should be a single size method

lpetrovic05 · 2025-01-21T14:03:16Z

...form-sdk/swirlds-platform-core/src/main/java/com/swirlds/platform/util/TransactionUtils.java

    /**
     * Check if a transaction is a system transaction.<br>
     * This is a convenience method that delegates to {@link #isSystemTransaction(OneOf)}.
     *
     * @param transaction the transaction to check
     * @return {@code true} if the transaction is a system transaction, {@code false} otherwise
     */
-    public static boolean isSystemTransaction(@NonNull final EventTransaction transaction) {
-        return isSystemTransaction(transaction.transaction());
+    // TODO: Refactor this method to work properly


lpetrovic05 · 2025-01-21T14:06:59Z

...core/src/testFixtures/java/com/swirlds/platform/test/fixtures/event/TestingEventBuilder.java

+     * <p>
+     * If set to true, transactions will be encoded in the new format of Bytes. If set to false, the deprecated OneOf format will be used.
+     */
+    private boolean useNewTransactionFormat = false;


this should be reverted

Signed-off-by: Mustafa Uzun <[email protected]>

Signed-off-by: Ivan Kavaldzhiev <[email protected]>

…ring' into 16665-system-transaction-refactoring

…ing them to TransactionPool Signed-off-by: Ivan Kavaldzhiev <[email protected]>

…ion-refactoring

Signed-off-by: Ivan Kavaldzhiev <[email protected]>

mustafauzunn added 4 commits January 14, 2025 13:53

refactor: addapt callback consumer logic

d14f8ea

Signed-off-by: Mustafa Uzun <[email protected]>

refactor: transaction pools

1bb1957

Signed-off-by: Mustafa Uzun <[email protected]>

refactor: after main

39c9d26

Signed-off-by: Mustafa Uzun <[email protected]>

mustafauzunn added the Platform Tickets pertaining to the platform label Jan 16, 2025

mustafauzunn added this to the v0.59 milestone Jan 16, 2025

mustafauzunn self-assigned this Jan 16, 2025

IvanKavaldzhiev reviewed Jan 16, 2025

View reviewed changes

mustafauzunn added 10 commits January 16, 2025 14:43

refactor: EventTransaction with Bytes

f253eb3

Signed-off-by: Mustafa Uzun <[email protected]>

refactor: adapt logic with Bytes

1053d02

Signed-off-by: Mustafa Uzun <[email protected]>

refactor: adapt logic for hasher

e907097

Signed-off-by: Mustafa Uzun <[email protected]>

refactor: adapt logic for hasher

ec0d2cd

Signed-off-by: Mustafa Uzun <[email protected]>

refactor: code

40b0940

Signed-off-by: Mustafa Uzun <[email protected]>

refactor: code

ba61264

Signed-off-by: Mustafa Uzun <[email protected]>

style: formatting

e989d5d

Signed-off-by: Mustafa Uzun <[email protected]>

refactor: code

b22154b

Signed-off-by: Mustafa Uzun <[email protected]>

refactor: code

df15758

Signed-off-by: Mustafa Uzun <[email protected]>

refactor: code

31c1566

Signed-off-by: Mustafa Uzun <[email protected]>

mustafauzunn and others added 4 commits January 20, 2025 12:42

refactor: code

48b8acf

Signed-off-by: Mustafa Uzun <[email protected]>

Merge remote-tracking branch 'origin/16665-system-transaction-refacto…

56e00cd

…ring' into 16665-system-transaction-refactoring

refactor: code

a71ba3c

Signed-off-by: Mustafa Uzun <[email protected]>

style: formatting

85c594e

Signed-off-by: Mustafa Uzun <[email protected]>

mustafauzunn and others added 3 commits January 20, 2025 16:57

Merge branch 'main' into 16665-system-transaction-refactoring

6d321a8

fix: unit tests - IssDetectorTests

996073c

Signed-off-by: Ivan Kavaldzhiev <[email protected]>

Merge remote-tracking branch 'origin/16665-system-transaction-refacto…

d9f42bb

…ring' into 16665-system-transaction-refactoring

IvanKavaldzhiev reviewed Jan 21, 2025

View reviewed changes

IvanKavaldzhiev added 2 commits January 21, 2025 11:24

refactor: extract properly StateSignatureTransaction from Transaction

b64fc96

Signed-off-by: Ivan Kavaldzhiev <[email protected]>

test: adapt IssDetectorTests to work with both the deprecated and new…

50733eb

… transaction format Signed-off-by: Ivan Kavaldzhiev <[email protected]>

IvanKavaldzhiev and others added 2 commits January 21, 2025 14:59

nit: fix warnings and clean up code

22bbbe3

Signed-off-by: Ivan Kavaldzhiev <[email protected]>

style: formatting

dcaeb9c

Signed-off-by: Mustafa Uzun <[email protected]>

refactor: code

a754bcb

Signed-off-by: Mustafa Uzun <[email protected]>

mustafauzunn requested a review from lpetrovic05 January 21, 2025 13:31

lpetrovic05 reviewed Jan 21, 2025

View reviewed changes

mustafauzunn and others added 7 commits January 21, 2025 16:35

refactor: code

f359089

Signed-off-by: Mustafa Uzun <[email protected]>

style: formatting

4206348

Signed-off-by: Mustafa Uzun <[email protected]>

nit: revert SystemTransactionExtractionUtils changes

891ca14

Signed-off-by: Ivan Kavaldzhiev <[email protected]>

Merge remote-tracking branch 'origin/16665-system-transaction-refacto…

0dc59f3

…ring' into 16665-system-transaction-refactoring

refactor: use wiring to encode StateSignatureTransactions before pass…

9a0503c

…ing them to TransactionPool Signed-off-by: Ivan Kavaldzhiev <[email protected]>

Merge remote-tracking branch 'origin/main' into 16665-system-transact…

a601c16

…ion-refactoring

nit: fix build

d567b00

Signed-off-by: Ivan Kavaldzhiev <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: system transactions #17406

refactor: system transactions #17406

mustafauzunn commented Jan 16, 2025

codecov bot commented Jan 16, 2025 •

edited

Loading

codacy-production bot commented Jan 16, 2025 •

edited

Loading

IvanKavaldzhiev Jan 16, 2025

mustafauzunn Jan 17, 2025

IvanKavaldzhiev Jan 16, 2025

mustafauzunn Jan 17, 2025

github-actions bot commented Jan 20, 2025 •

edited

Loading

github-actions bot commented Jan 20, 2025 •

edited

Loading

IvanKavaldzhiev Jan 21, 2025

IvanKavaldzhiev Jan 21, 2025

github-actions bot commented Jan 21, 2025

github-actions bot commented Jan 21, 2025

github-actions bot commented Jan 21, 2025

codacy-production bot commented Jan 21, 2025 •

edited

Loading

lpetrovic05 Jan 21, 2025

IvanKavaldzhiev Jan 21, 2025

lpetrovic05 Jan 21, 2025

lpetrovic05 Jan 21, 2025

IvanKavaldzhiev Jan 21, 2025

lpetrovic05 Jan 21, 2025

lpetrovic05 Jan 21, 2025

lpetrovic05 Jan 21, 2025

IvanKavaldzhiev Jan 21, 2025

lpetrovic05 Jan 21, 2025

lpetrovic05 Jan 21, 2025

lpetrovic05 Jan 21, 2025

lpetrovic05 Jan 21, 2025

lpetrovic05 Jan 21, 2025

		public synchronized boolean submitTransaction(
		@NonNull final Bytes transaction, final boolean priority, final boolean isSystem) {

		if (TransactionUtils.getLegacyTransactionSize(transaction) > maximumTransactionSize) {
		if (TransactionUtils.getLegacyTransactionSize(appTransaction) > maximumTransactionSize) {

		@@ -39,14 +39,18 @@ public sealed interface Transaction permits ConsensusTransaction {
		@Deprecated
		EventTransaction getTransaction();

refactor: system transactions #17406

Are you sure you want to change the base?

refactor: system transactions #17406

Conversation

mustafauzunn commented Jan 16, 2025

codecov bot commented Jan 16, 2025 • edited Loading

Codecov Report

codacy-production bot commented Jan 16, 2025 • edited Loading

Coverage summary from Codacy

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jan 20, 2025 • edited Loading

Node: HAPI Test (Token) Results

github-actions bot commented Jan 20, 2025 • edited Loading

Node: HAPI Test (Smart Contract) Results

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jan 21, 2025

Node: HAPI Test (Time Consuming) Results

github-actions bot commented Jan 21, 2025

Node: HAPI Test (Restart) Results

github-actions bot commented Jan 21, 2025

Node: HAPI Test (Node Death Reconnect) Results

codacy-production bot commented Jan 21, 2025 • edited Loading

Coverage summary from Codacy

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jan 16, 2025 •

edited

Loading

codacy-production bot commented Jan 16, 2025 •

edited

Loading

github-actions bot commented Jan 20, 2025 •

edited

Loading

github-actions bot commented Jan 20, 2025 •

edited

Loading

codacy-production bot commented Jan 21, 2025 •

edited

Loading