PHOENIX-7485: Add metrics for time taken during mutation plan creation and execution #2061

sanjeet006py · 2025-01-23T09:21:41Z

Summary of changes:

Added metrics to capture time taken in mutation plan creation and execution for upserts and deletes.
As the mutation plan creation and execution for a single upsert/delete can complete in less than 1 milli-second so, capturing metrics in Nano seconds. With PHOENIX-7484, it was observed that even mutation plan creation and execution is taking less than a milli second per call. If a regression is introduced in mutation plan creation and execution such that they start taking 1 milli-second per call then regression per call is not much in milli-seconds but at a scale of 2M rows being upserted/deleted it's huge. So, better to track at finer granularity to identify sub milli-second regression which can manifest as huge regression for end users.
Added support for computing elapsed time in nano seconds in EnvironmentEdgeManager and EnvironmentEdge.
Added metrics to track time taken by executeMutation call in nano seconds. There are already metrics available to track time taken by upserts and deletes in executeMutation call but they track at milli seconds granularity. If we change those to track at nano seconds granularity then this will be a breaking non-backward compatible change.

sanjeet006py · 2025-01-27T16:54:39Z

@virajjasani @tkhurana @jpisaac can you please review the PR. Thanks

tkhurana · 2025-01-27T17:56:22Z

@sanjeet006py Using nanoseconds resolution for measuring calls which themselves take less than 1ms adds considerable measurement overhead. AFAIK each call to System.nanosecond itself takes ~25- 30ns so for 2 calls you are adding double overhead. Now if the operation itself takes < 1 ms let us say 600ns you are adding a 10% measurement overhead.

https://www.javaadvent.com/2019/12/measuring-time-from-java-to-kernel-and-back.html

sanjeet006py · 2025-01-27T19:40:24Z

@sanjeet006py Using nanoseconds resolution for measuring calls which themselves take less than 1ms adds considerable measurement overhead. AFAIK each call to System.nanosecond itself takes ~25- 30ns so for 2 calls you are adding double overhead. Now if the operation itself takes < 1 ms let us say 600ns you are adding a 10% measurement overhead.

https://www.javaadvent.com/2019/12/measuring-time-from-java-to-kernel-and-back.html

Thanks, it was a great article. I got two key takeaways from the article:

Minimize the number of calls to measure time.
Measuring time in millis and nanos has roughly same overhead.

I was of understanding that by measuring mutation plan creation and execution only, I am already minimizing amount of calls being made to track time. I have seen time taken by mutation plan creation and execution is of the order of 5-15 micro seconds (thus, using nano seconds clock). Should we just track time taken by a single executeMutation call and not track at further granularity? Though for executeMutation call we should track at nano seconds granularity.

tkhurana · 2025-01-27T23:06:27Z

Have you looked at the time taken by the execution of mutationPlan for upsert select on a table which has large number of rows and client side deletes which first require rows to be scanned on the server to determine which rows to be deleted ? I am pretty sure their execution times will be much larger.

sanjeet006py · 2025-01-28T04:08:46Z

Have you looked at the time taken by the execution of mutationPlan for upsert select on a table which has large number of rows and client side deletes which first require rows to be scanned on the server to determine which rows to be deleted ? I am pretty sure their execution times will be much larger.

I haven't benchmarked this use case explicitly but I agree that execution of mutation plan will be time taking in this due to needing to scan first to prepare row mutations. I benchmarked upsert value and that is pretty fast for a single call (5-15 micro seconds) thus, decided to measure elapsed time in nano seconds.

d-c-manning · 2025-01-28T19:27:01Z

phoenix-core/src/it/java/org/apache/phoenix/monitoring/PhoenixMetricsIT.java

@@ -487,7 +495,7 @@ public void testMetricsForUpsert() throws Exception {
            String t = entry.getKey();
            assertEquals("Table names didn't match!", tableName, t);
            Map<MetricType, Long> p = entry.getValue();
-            assertEquals("There should have been sixteen metrics", 16, p.size());
+            assertEquals("There should have been sixteen metrics", 22, p.size());


either remove the text, or update the text

Suggested change

assertEquals("There should have been sixteen metrics", 22, p.size());

assertEquals("There should have been 22 metrics", 22, p.size());

d-c-manning · 2025-01-28T19:28:53Z

phoenix-core/src/it/java/org/apache/phoenix/monitoring/PhoenixLoggingMetricsIT.java

        // Hence mutation metrics are not expected during connection close
        loggedConn.close();
        assertTrue("Mutation write metrics are not logged for " + tableName2,
-                mutationWriteMetricsMap.size() == 0);
+                mutationWriteMetricsMap.size() > 0);


the text for assertTrue, and the comment above, should be updated to reflect the new reality.

d-c-manning · 2025-01-28T19:29:31Z

phoenix-core-client/src/main/java/org/apache/phoenix/util/PhoenixRuntime.java

@@ -1660,4 +1660,12 @@ public static long getCurrentScn(ReadOnlyProps props) {
        String scn = props.get(CURRENT_SCN_ATTRIB);
        return scn != null ? Long.parseLong(scn) : HConstants.LATEST_TIMESTAMP;
    }
+
+    public static long convertTimeInNsToMs(long value) {


these duplicate EnvironmentEdge#convertNsToMs - can we share?

d-c-manning · 2025-01-29T06:38:59Z

phoenix-core/src/it/java/org/apache/phoenix/monitoring/PhoenixLoggingMetricsIT.java

@@ -156,9 +156,9 @@ public void testPhoenixMetricsLoggedOnClose() throws Exception {
        // Autocommit is turned off by default
        // Hence mutation metrics are not expected during connection close


this line 157 comment is still here though (are mutation metrics now expected during connection close?

Thanks. Yes, now when auto-commit is off then we do expect mutation metrics to be there on connection close() as we are now recording metrics for executeUpdate() call also.

Do you think this could be an issue?

I don't fully understand the scenario and the impact. The test makes me think that before, without a call to commit or setAutoCommit(true), then an executeUpdate is insufficient to make any writes, and therefore you have no mutation metrics. But now, even without a commit, we have some write metrics. But what are those metrics? Is it just the time taken for the mutation planning? Should the test confirm that those are the only metrics present?

Otherwise no, it does not seem to be an issue.

I don't fully understand the scenario and the impact.

The test seems to be asserting the metrics are being populated on calling Connection.close() only when they should be populated and is testing two cases: for reads and for writes. Before my change, no metrics were expected for writes on calling Connection.close() when commit() has not been called (explicitly/implicitly). But now we do expect metrics to be populated for writes even if commit() is not called and these metrics are for time taken in mutation plan creation and execution. So, time taken by mutation planning (plan creation + execution) is also being populated in MutationMetrics and will be available even if commit() is not called.

Should the test confirm that those are the only metrics present?

Sure, I can add this assertion also. Thanks

sanjeet006py · 2025-01-29T14:54:07Z

@tkhurana based on the response in here, please let me know if there is a concern with the overall change in this PR. Thanks

Sanjeet Malhotra added 3 commits January 23, 2025 01:24

Add mutation metrics

2c6e94c

Add test coverage

5b7bf61

Minor nits

2bc4b03

sanjeet006py closed this Jan 23, 2025

sanjeet006py reopened this Jan 23, 2025

Fix tests

cc4bf18

virajjasani requested review from virajjasani, tkhurana and jpisaac January 23, 2025 17:19

Fix tests

3e1be4f

d-c-manning reviewed Jan 28, 2025

View reviewed changes

Address David's comments

089f7b0

d-c-manning reviewed Jan 29, 2025

View reviewed changes

Sanjeet Malhotra added 3 commits January 29, 2025 12:16

Extension of David's comments

f28a768

Address David's comments

1120150

Address David's comments

36ee9d4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PHOENIX-7485: Add metrics for time taken during mutation plan creation and execution #2061

PHOENIX-7485: Add metrics for time taken during mutation plan creation and execution #2061

sanjeet006py commented Jan 23, 2025

sanjeet006py commented Jan 27, 2025

tkhurana commented Jan 27, 2025 •

edited

Loading

sanjeet006py commented Jan 27, 2025

tkhurana commented Jan 27, 2025

sanjeet006py commented Jan 28, 2025 •

edited

Loading

d-c-manning Jan 28, 2025

d-c-manning Jan 28, 2025

d-c-manning Jan 28, 2025

d-c-manning Jan 29, 2025 •

edited

Loading

sanjeet006py Jan 29, 2025

sanjeet006py Jan 29, 2025

d-c-manning Jan 29, 2025

sanjeet006py Jan 29, 2025 •

edited

Loading

sanjeet006py Jan 29, 2025

sanjeet006py commented Jan 29, 2025

	assertEquals("There should have been sixteen metrics", 22, p.size());
	assertEquals("There should have been 22 metrics", 22, p.size());

		@@ -156,9 +156,9 @@ public void testPhoenixMetricsLoggedOnClose() throws Exception {
		// Autocommit is turned off by default
		// Hence mutation metrics are not expected during connection close

PHOENIX-7485: Add metrics for time taken during mutation plan creation and execution #2061

Are you sure you want to change the base?

PHOENIX-7485: Add metrics for time taken during mutation plan creation and execution #2061

Conversation

sanjeet006py commented Jan 23, 2025

sanjeet006py commented Jan 27, 2025

tkhurana commented Jan 27, 2025 • edited Loading

sanjeet006py commented Jan 27, 2025

tkhurana commented Jan 27, 2025

sanjeet006py commented Jan 28, 2025 • edited Loading

d-c-manning Jan 28, 2025

Choose a reason for hiding this comment

d-c-manning Jan 28, 2025

Choose a reason for hiding this comment

d-c-manning Jan 28, 2025

Choose a reason for hiding this comment

d-c-manning Jan 29, 2025 • edited Loading

Choose a reason for hiding this comment

sanjeet006py Jan 29, 2025

Choose a reason for hiding this comment

sanjeet006py Jan 29, 2025

Choose a reason for hiding this comment

d-c-manning Jan 29, 2025

Choose a reason for hiding this comment

sanjeet006py Jan 29, 2025 • edited Loading

Choose a reason for hiding this comment

sanjeet006py Jan 29, 2025

Choose a reason for hiding this comment

sanjeet006py commented Jan 29, 2025

tkhurana commented Jan 27, 2025 •

edited

Loading

sanjeet006py commented Jan 28, 2025 •

edited

Loading

d-c-manning Jan 29, 2025 •

edited

Loading

sanjeet006py Jan 29, 2025 •

edited

Loading