pcm-sensor-server errors: "PCM Error: someone has corrupted custom counter" #500

ballegre · 2023-02-02T18:34:31Z

ballegre
Feb 2, 2023

Built using latest master. Disabled PCM_USE_PERF and see this error on 2 different platforms so far (Broadwell and Haswell).

[root@n-0026-1-d ~]# ./pcm-sensor-server --reset

=====  Processor information  =====
Linux arch_perfmon flag  : yes
Hybrid processor         : no
IBRS and IBPB supported  : yes
STIBP supported          : yes
Spec arch caps supported : no
Max CPUID level          : 20
CPU model number         : 79
Number of physical cores: 28
Number of logical cores: 56
Number of online logical cores: 56
Threads (logical cores) per physical core: 2
Num sockets: 2
Physical cores per socket: 14
Last level cache slices per socket: 14
Core PMU (perfmon) version: 3
Number of core PMU generic (programmable) counters: 4
Width of generic (programmable) counters: 48 bits
Number of core PMU fixed counters: 3
Width of fixed counters: 48 bits
Nominal core frequency: 2400000000 Hz
IBRS enabled in the kernel   : no
STIBP enabled in the kernel  : no
Package thermal spec power: 120 Watt; Package minimum power: 58 Watt; Package maximum power: 240 Watt;

Socket 0: 2 memory controllers detected with total number of 4 channels. 2 QPI ports detected. 0 M2M (mesh to memory) blocks detected. 0 HBM M2M blocks detected. 0 EDC/HBM channels detected. 2 Home Agents detected. 0 M3UPI blocks detected.
Socket 1: 2 memory controllers detected with total number of 4 channels. 2 QPI ports detected. 0 M2M (mesh to memory) blocks detected. 0 HBM M2M blocks detected. 0 EDC/HBM channels detected. 2 Home Agents detected. 0 M3UPI blocks detected.
Initializing RMIDs

 Disabling NMI watchdog since it consumes one hw-PMU counter. To keep NMU watchdog set environment variable PCM_KEEP_NMI_WATCHDOG=1 (this reduces the core metrics set)
WARNING: Core 22 IA32_PERFEVTSEL1_ADDR is not zeroed 1114304
 Zeroed PMU registers
Socket 0
Max QPI link 0 speed: 19.2 GBytes/second (9.6 GT/second)
Max QPI link 1 speed: 19.2 GBytes/second (9.6 GT/second)
Socket 1
Max QPI link 0 speed: 19.2 GBytes/second (9.6 GT/second)
Max QPI link 1 speed: 19.2 GBytes/second (9.6 GT/second)
Starting plain HTTP server on http://localhost:9738/
PCM Error: someone has corrupted custom counter 1 on core 22 expected value 4393170 value read 1114304
PCM Error: someone has corrupted custom counter 1 on core 22 expected value 4393170 value read 1114304
PCM Error: someone has corrupted custom counter 1 on core 22 expected value 4393170 value read 1114304
PCM Error: someone has corrupted custom counter 1 on core 22 expected value 4393170 value read 1114304
PCM Error: someone has corrupted custom counter 1 on core 22 expected value 4393170 value read 1114304
PCM Error: someone has corrupted custom counter 1 on core 22 expected value 4393170 value read 1114304
PCM Error: someone has corrupted custom counter 1 on core 22 expected value 4393170 value read 1114304
^ChandleSignal: signal 2 caught.
handleSignal: closing socket 75
Stopping HTTPServer
Cleaning up PMU:
Cleaning up
 Zeroed PMU registers
 Zeroed uncore PMU registers
 Freeing up all RMIDs
 Re-enabling NMI watchdog.
handleSignal: exiting with exit code 1...
[root@n-0026-1-d ~]# timed out waiting for input: auto-logout

Answered by ballegre

Feb 3, 2023

There's an entire development team behind the OS I am testing on. I will pursue an answer to the question about anything programming the PMU.

If I cannot resolve sharing the PMU issue then it seems the perf_event API is the way to go. This is what I see from pcm on the haswell system with master build using the perf_event API:

[root@n-0004-1-d ~]# ./pcm 1 -r -nsys

 Intel(r) Performance Counter Monitor ($Format:%ci ID=%h$)



=====  Processor information  =====
Linux arch_perfmon flag  : yes
Hybrid processor         : no
IBRS and IBPB supported  : yes
STIBP supported          : yes
Spec arch caps supported : no
Max CPUID level          : 15
CPU model number         : 63
Number of physical…

View full answer

rdementi · 2023-02-03T08:38:56Z

rdementi
Feb 3, 2023
Maintainer

does it work if you don't use PCM_USE_PERF variable? Could you share the output if PCM_USE_PERF is not set?

1 reply

ballegre Feb 3, 2023
Author

To be clear. The output above is with -DPCM_USE_PERF removed:

+       #set (PCM_COMMON_FLAGS "${PCM_COMMON_FLAGS} -Wextra -DPCM_USE_PERF")
+       set (PCM_COMMON_FLAGS "${PCM_COMMON_FLAGS} -Wextra")

I suppose you would like to know what happens when PCM_USE_PERF is set? The short answer is it works on broadwell.

The long answer gets more complicated. We are a multi-platform vendor and need to run pcm on many platforms for perf regression testing. So far I have evaluated 3: broadwell, haswell, and skylake. The output above is custom built pcm-sensor-server on the broadwell platform. I would like to use the same pcm-sensor-server binary on all platforms.

I originally ran pcm-sensor-server using the pre-built binaries on broadwell. This worked well so I began to deploy on the other platforms and noticed that L2 cache hits, local and remote memory BW metrics from pcm-sensor-server were zero (on haswell and skylake). So in an attempt to work around I cloned master and built with defaults that have -DPCM_USE_PERF set. Same results with master: works on broadwell, broke on haswell and skylake.

I spent some time studying code, pcm, CMake, cpp, etc. I set env varaible "PCM_NO_PERF=1" and things looked better on haswell. The zero metrics now had real values but still see the "corrupted custom counter" error (on both broadwell and haswell). At this point I decided to recompile with -DPCM_USE_PERF removed and use this binary for further evaluation (and started this discussion).

So, I'm looking for a way to get a single pcm-sensor-server that works on all platforms. An attempt to summarize:

pre-built pcm-sensor-server: works on broadwell, broke on haswell and skylake
latest master pcm-sensor-server: same results as above
latest master built with no perf pcm-sensor-server: works on all platforms but gets continuous "corrupted custom counter" errors

Appreciate any help and happy to do what I can to assist.

rdementi · 2023-02-03T16:58:46Z

rdementi
Feb 3, 2023
Maintainer

thank you for the explanation. haswell, broadwell, skylake do not support the same set of metrics. If I correctly remember Haswell does not support LMB and RMB memory bandwidth and Skylake does not support L2HIT by default. You can check by running the main pcm binary (not pcm-sensor-server) and see what metrics it prints on each platform.

When you use PCM_NO_PERF=1 then the perf_event API is not used and PCM programs PMU directly such that when other clients try to use PMU through perf kernel API they don't know that PCM is using PMU already and corrupt the PMU counter programming (the errors you see). If everyone is using perf_event API then the access is coordinated and no conflicts occur.

PCM Error: someone has corrupted custom counter 1 on core 22 expected value 4393170 value read 1114304

I decoded the value 1114304 from your log: someone is programming PMU to collect INST_RETIRED.ANY_P event for core 22 in parallel when you are running PCM. Do you have an idea what could it be?

0 replies

ballegre · 2023-02-03T17:46:45Z

ballegre
Feb 3, 2023
Author

There's an entire development team behind the OS I am testing on. I will pursue an answer to the question about anything programming the PMU.

If I cannot resolve sharing the PMU issue then it seems the perf_event API is the way to go. This is what I see from pcm on the haswell system with master build using the perf_event API:

[root@n-0004-1-d ~]# ./pcm 1 -r -nsys

 Intel(r) Performance Counter Monitor ($Format:%ci ID=%h$)



=====  Processor information  =====
Linux arch_perfmon flag  : yes
Hybrid processor         : no
IBRS and IBPB supported  : yes
STIBP supported          : yes
Spec arch caps supported : no
Max CPUID level          : 15
CPU model number         : 63
Number of physical cores: 24
Number of logical cores: 48
Number of online logical cores: 48
Threads (logical cores) per physical core: 2
Num sockets: 2
Physical cores per socket: 12
Last level cache slices per socket: 12
Core PMU (perfmon) version: 3
Number of core PMU generic (programmable) counters: 4
Width of generic (programmable) counters: 48 bits
Number of core PMU fixed counters: 3
Width of fixed counters: 48 bits
Nominal core frequency: 2300000000 Hz
IBRS enabled in the kernel   : no
STIBP enabled in the kernel  : no
Package thermal spec power: 120 Watt; Package minimum power: 61 Watt; Package maximum power: 240 Watt;

INFO: Linux perf interface to program uncore PMUs is present
Socket 0: 2 memory controllers detected with total number of 4 channels. 2 QPI ports detected. 0 M2M (mesh to memory) blocks detected. 0 HBM M2M blocks detected. 0 EDC/HBM channels detected. 2 Home Agents detected. 0 M3UPI blocks detected.
Socket 1: 2 memory controllers detected with total number of 4 channels. 2 QPI ports detected. 0 M2M (mesh to memory) blocks detected. 0 HBM M2M blocks detected. 0 EDC/HBM channels detected. 2 Home Agents detected. 0 M3UPI blocks detected.
Initializing RMIDs


 Resetting PMU configuration
 Zeroed PMU registers
 Closed perf event handles
Trying to use Linux perf events...
Successfully programmed on-core PMU using Linux perf
Socket 0
Max QPI link 0 speed: 19.2 GBytes/second (9.6 GT/second)
Max QPI link 1 speed: 19.2 GBytes/second (9.6 GT/second)
Socket 1
Max QPI link 0 speed: 19.2 GBytes/second (9.6 GT/second)
Max QPI link 1 speed: 19.2 GBytes/second (9.6 GT/second)

Detected Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz "Intel(r) microarchitecture codename Haswell-EP/EN/EX" stepping 2 microcode level 0x46

 EXEC  : instructions per nominal CPU cycle
 IPC   : instructions per CPU cycle
 FREQ  : relation to nominal CPU frequency='unhalted clock ticks'/'invariant timer ticks' (includes Intel Turbo Boost)
 AFREQ : relation to nominal CPU frequency while in active state (not in power-saving C state)='unhalted clock ticks'/'invariant timer ticks while in C0-state'  (includes Intel Turbo Boost)
 L3MISS: L3 (read) cache misses 
 L2MISS: L2 (read) cache misses (including other core's L2 cache *hits*) 
 L3HIT : L3 (read) cache hit ratio (0.00-1.00)
 L2HIT : L2 cache hit ratio (0.00-1.00)
 L3MPI : number of L3 (read) cache misses per instruction
 L2MPI : number of L2 (read) cache misses per instruction
 READ  : bytes read from main memory controller (in GBytes)
 WRITE : bytes written to main memory controller (in GBytes)
 LOCAL : ratio of local memory requests to memory controller in %
LLCRDMISSLAT: average latency of last level cache miss for reads and prefetches (in ns)
 L3OCC : L3 occupancy (in KBytes)
 TEMP  : Temperature reading in 1 degree Celsius relative to the TjMax temperature (thermal headroom): 0 corresponds to the max temperature
 energy: Energy in Joules


 Core (SKT) | EXEC | IPC  | FREQ  | AFREQ | L3MISS | L2MISS | L3HIT | L2HIT | L3MPI | L2MPI |   L3OCC | TEMP

   0    0     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan      240     47
   1    0     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan      768     47
   2    0     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan        0     48
   3    0     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan      240     46
   4    0     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan    19440     43
   5    0     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan        0     45
   6    0     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan       96     43
   7    0     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan       96     47
   8    0     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan      192     49
   9    0     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan        0     44
  10    0     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan       96     48
  11    0     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan        0     49
  12    1     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan     3936     49
  13    1     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan      960     48
  14    1     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan       96     49
  15    1     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan        0     49
  16    1     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan       48     51
  17    1     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan        0     48
  18    1     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan      192     47
  19    1     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan       48     47
  20    1     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan        0     49
  21    1     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan       96     47
  22    1     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan        0     51
  23    1     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan    10752     47
  24    0     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan       48     47
  25    0     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan       48     47
  26    0     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan        0     48
  27    0     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan      624     46
  28    0     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan     5232     43
  29    0     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan      960     45
  30    0     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan       96     43
  31    0     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan     1008     47
  32    0     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan      480     49
  33    0     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan       48     44
  34    0     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan        0     48
  35    0     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan       96     49
  36    1     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan      144     49
  37    1     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan      144     48
  38    1     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan        0     49
  39    1     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan       96     49
  40    1     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan       48     51
  41    1     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan       96     48
  42    1     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan      480     47
  43    1     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan      144     47
  44    1     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan      192     49
  45    1     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan      240     47
  46    1     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan     7968     51
  47    1     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan       48     47
---------------------------------------------------------------------------------------------------------------
 SKT    0     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan    29808     40
 SKT    1     0.00   -1.00   0.00    -1.00       0        0      0.00    0.00  -nan  -nan    25728     42
---------------------------------------------------------------------------------------------------------------

Intel(r) QPI traffic estimation in bytes (data and non-data traffic outgoing from CPU/socket through QPI links):

               QPI0     QPI1    |  QPI0   QPI1  
---------------------------------------------------------------------------------------------------------------
 SKT    0       57 M     57 M   |    0%     0%   
 SKT    1       63 M     63 M   |    0%     0%   
---------------------------------------------------------------------------------------------------------------
Total QPI outgoing data and non-data traffic:  241 M
MEM (GB)->|  READ |  WRITE | LOCAL | CPU energy | DIMM energy | LLCRDMISSLAT (ns)| UncFREQ (Ghz)|
---------------------------------------------------------------------------------------------------------------
 SKT   0     0.43     0.20   96 %      66.04      14.58          88.74             2.99
 SKT   1     0.09     0.05   53 %      42.70      14.30         107.28             2.92
---------------------------------------------------------------------------------------------------------------
       *     0.51     0.25   89 %     108.74      28.88          91.20             2.95

12 replies

ballegre Feb 9, 2023
Author

My latest pcm master works on the Skylake system. I also believe the original worked before this build as well.

Detected Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz "Intel(r) microarchitecture codename Skylake-SP" stepping 4 microcode level 0x2006c0a

The pcm-sensor-server has zreo counters for local/remote memory bandwidth and l2 cache hits. Perhaps the fix was not applied to pcm-sensor-server.

[root@n-0074-1-a ~]# ./pcm-sensor-server --reset -d

=====  Processor information  =====
[root@n-0074-1-a ~]# Linux arch_perfmon flag  : yes
Hybrid processor         : no
IBRS and IBPB supported  : yes
STIBP supported          : yes
Spec arch caps supported : yes
Max CPUID level          : 22
CPU model number         : 85
Number of physical cores: 40
Number of logical cores: 80
Number of online logical cores: 80
Threads (logical cores) per physical core: 2
Num sockets: 2
Physical cores per socket: 20
Last level cache slices per socket: 20
Core PMU (perfmon) version: 4
Number of core PMU generic (programmable) counters: 3
Width of generic (programmable) counters: 48 bits
Number of core PMU fixed counters: 3
Width of fixed counters: 48 bits
Nominal core frequency: 2000000000 Hz
IBRS enabled in the kernel   : no
STIBP enabled in the kernel  : no
The processor is not susceptible to Rogue Data Cache Load: no
The processor supports enhanced IBRS                     : no
Package thermal spec power: 125 Watt; Package minimum power: 65 Watt; Package maximum power: 264 Watt;

INFO: Linux perf interface to program uncore PMUs is present
Socket 0: 2 memory controllers detected with total number of 6 channels. 3 UPI ports detected. 2 M2M (mesh to memory) blocks detected. 0 HBM M2M blocks detected. 0 EDC/HBM channels detected. 0 Home Agents detected. 3 M3UPI blocks detected.
Socket 1: 2 memory controllers detected with total number of 6 channels. 3 UPI ports detected. 2 M2M (mesh to memory) blocks detected. 0 HBM M2M blocks detected. 0 EDC/HBM channels detected. 0 Home Agents detected. 3 M3UPI blocks detected.
Initializing RMIDs

 Disabling NMI watchdog since it consumes one hw-PMU counter. To keep NMU watchdog set environment variable PCM_KEEP_NMI_WATCHDOG=1 (this reduces the core metrics set)
 Closed perf event handles
Trying to use Linux perf events...
Successfully programmed on-core PMU using Linux perf
Socket 0
Max UPI link 0 speed: 23.3 GBytes/second (10.4 GT/second)
Max UPI link 1 speed: 23.3 GBytes/second (10.4 GT/second)
Max UPI link 2 speed: 21.5 GBytes/second (9.6 GT/second)
Socket 1
Max UPI link 0 speed: 23.3 GBytes/second (10.4 GT/second)
Max UPI link 1 speed: 23.3 GBytes/second (10.4 GT/second)
Max UPI link 2 speed: 21.5 GBytes/second (9.6 GT/second)
Starting plain HTTP server on http://localhost:9738/

[root@n-0074-1-a ~]# curl -s http://localhost:9738/metrics | grep -i local | head
Local_Memory_Bandwidth{socket="0",core="0",thread="0",source="core"} 0
Local_Memory_Bandwidth{socket="0",core="0",thread="1",source="core"} 0
Local_Memory_Bandwidth{socket="0",core="1",thread="0",source="core"} 0
Local_Memory_Bandwidth{socket="0",core="1",thread="1",source="core"} 0
Local_Memory_Bandwidth{socket="0",core="2",thread="0",source="core"} 0
Local_Memory_Bandwidth{socket="0",core="2",thread="1",source="core"} 0
Local_Memory_Bandwidth{socket="0",core="3",thread="0",source="core"} 0
Local_Memory_Bandwidth{socket="0",core="3",thread="1",source="core"} 0
Local_Memory_Bandwidth{socket="0",core="4",thread="0",source="core"} 0
Local_Memory_Bandwidth{socket="0",core="4",thread="1",source="core"} 0
setsockopt failed while setting timeout value, Socket operation on non-socket
[root@n-0074-1-a ~]# curl -s http://localhost:9738/metrics | grep -i remote | head
Remote_Memory_Bandwidth{socket="0",core="0",thread="0",source="core"} 0
Remote_Memory_Bandwidth{socket="0",core="0",thread="1",source="core"} 0
Remote_Memory_Bandwidth{socket="0",core="1",thread="0",source="core"} 0
Remote_Memory_Bandwidth{socket="0",core="1",thread="1",source="core"} 0
Remote_Memory_Bandwidth{socket="0",core="2",thread="0",source="core"} 0
Remote_Memory_Bandwidth{socket="0",core="2",thread="1",source="core"} 0
Remote_Memory_Bandwidth{socket="0",core="3",thread="0",source="core"} 0
Remote_Memory_Bandwidth{socket="0",core="3",thread="1",source="core"} 0
Remote_Memory_Bandwidth{socket="0",core="4",thread="0",source="core"} 0
Remote_Memory_Bandwidth{socket="0",core="4",thread="1",source="core"} 0
setsockopt failed while setting timeout value, Socket operation on non-socket
[root@n-0074-1-a ~]# curl -s http://localhost:9738/metrics | grep -i l2_cache | head
L2_Cache_Misses{socket="0",core="0",thread="0",source="core"} 6821890
L2_Cache_Hits{socket="0",core="0",thread="0",source="core"} 0
L2_Cache_Misses{socket="0",core="0",thread="1",source="core"} 145972
L2_Cache_Hits{socket="0",core="0",thread="1",source="core"} 0
L2_Cache_Misses{socket="0",core="1",thread="0",source="core"} 454119
L2_Cache_Hits{socket="0",core="1",thread="0",source="core"} 0
L2_Cache_Misses{socket="0",core="1",thread="1",source="core"} 162019
L2_Cache_Hits{socket="0",core="1",thread="1",source="core"} 0
L2_Cache_Misses{socket="0",core="2",thread="0",source="core"} 399098
L2_Cache_Hits{socket="0",core="2",thread="0",source="core"} 0

opcm Feb 10, 2023
Collaborator

So far I can confirm that pcm command works on my Haswell system and continues to work on my Broadwell system. I am working on getting access to Skylake.

I noticed a few other things.

The error message that is the subject of this discussion has gone away with PCM_NO_PERF=1.

this is not expected. I assumed a different perfmon agent will still corrupt counter 1 if perf_event API is not used.

The pcm command no longer shows L2HIT column on Haswell (present on Broadwell).

The issue was that PCM used an L2 hit event which is impacted by errata: torvalds/linux@e979121
I have removed this L2 hit metric on Haswell and other impacted cpu generations. I will find a replacement event and restore the metric if possible.

The pcm-sensor-server continues show zero count on local/remote memory bandwidth and l2 cache hits.

L2 cache hits: see above, same issue.
local/remote memory bandwidth: these metrics are relying on MBM technology which is not supported in Haswell (there are a few rare Haswell SKUs which still support) and Skylake CPU generations.

Just noticed the pcm-numa command. Looking at the cpp code now to see how that works.

pcm-numa occupies counters which are already used for L3/L2 cache stats.

Local and remote memory bandwidth is very useful in detecting and evaluating changes in NUMA placement behavior. Do you have any suggestions on what can be used as a substitute?

I would suggest to look into QPI/UPI data bandwidth (=data traffic between the sockets) and put it into relation to memory bandwidth. If the QPI/UPI traffic is relatively high then NUMA placement behavior is suboptimal.

opcm Feb 10, 2023
Collaborator

My latest pcm master works on the Skylake system. I also believe the original worked before this build as well.
Detected Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz "Intel(r) microarchitecture codename Skylake-SP" stepping 4 microcode level 0x2006c0a
The pcm-sensor-server has zreo counters for local/remote memory bandwidth and l2 cache hits. Perhaps the fix was not applied to pcm-sensor-server.

local/remote memory bandwidth: see above. Skylake does not support MBM technology.

Number of core PMU generic (programmable) counters: 3

L2 cache hits: you are hitting a different errata on Skylake which reduces the number of available counters such that pcm needs to reduce the scope of supported metrics on Skylake (L2 cache hit not supported):
https://cdrdv2-public.intel.com/604224/604224%20Performance-Monitoring-Impact-of-TSX-Memory-Ordering-Issue.pdf

pcm utility supports -force-rtm-abort-mode option as a work-around to restore the metric. I can implement this option for pcm-sensor-server too

ballegre Feb 10, 2023
Author

Thanks for the detailed replies. Appreciate you sharing this info. Really helps us.

Odd about the "corrupted custom counter" error going away. I can switch back and forth between the old build and latest build to reproduce the issue. As promised, I asked our development team if anyone knew of a component accessing PMU or perf API and there were no definitive replies. I'm fine with this outcome, not a problem.

Thanks for the suggestion using QPI/UPI as a proxy for remote memory accesses. I can use this.

You mention a couple of possible fixes: the l2 cache hits and pcm-sensor-server supporting -force-rtm-abort-mode. Will you be opening issues that we can track?

Fell free to mark this discussion answered. Or I can do it if you prefer. Thanks.

rdementi Feb 13, 2023
Maintainer

you are welcome

You mention a couple of possible fixes: the l2 cache hits and pcm-sensor-server supporting -force-rtm-abort-mode. Will you be opening issues that we can track?

both fixes are now in the master branch.

Fell free to mark this discussion answered. Or I can do it if you prefer. Thanks.

please mark as answered if you think your questions have been addressed

ballegre · 2023-02-03T18:02:53Z

ballegre
Feb 3, 2023
Author

And I don't have access to a skylake system now so easier to debug on haswell. I can get one if we need to move forward on this issue. Thanks.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pcm-sensor-server errors: "PCM Error: someone has corrupted custom counter" #500

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments 13 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

pcm-sensor-server errors: "PCM Error: someone has corrupted custom counter" #500

ballegre Feb 2, 2023

Replies: 4 comments · 13 replies

rdementi Feb 3, 2023 Maintainer

ballegre Feb 3, 2023 Author

rdementi Feb 3, 2023 Maintainer

ballegre Feb 3, 2023 Author

ballegre Feb 9, 2023 Author

opcm Feb 10, 2023 Collaborator

opcm Feb 10, 2023 Collaborator

ballegre Feb 10, 2023 Author

rdementi Feb 13, 2023 Maintainer

ballegre Feb 3, 2023 Author

ballegre
Feb 2, 2023

Replies: 4 comments 13 replies

rdementi
Feb 3, 2023
Maintainer

ballegre Feb 3, 2023
Author

rdementi
Feb 3, 2023
Maintainer

ballegre
Feb 3, 2023
Author

ballegre Feb 9, 2023
Author

opcm Feb 10, 2023
Collaborator

opcm Feb 10, 2023
Collaborator

ballegre Feb 10, 2023
Author

rdementi Feb 13, 2023
Maintainer

ballegre
Feb 3, 2023
Author