Add bench.ps1 and customize run and exporters

[nietras]

@joelverhagen a few proposed changes to benchmark running now that args are forwarded. I do not know if you prefer the old way if so I can change to whatever you prefer. This change means csv file output is more directly usable e.g. in Excel. As output is sorted and units only defined in column names.

Changed run parameters since Sep and others are quite fast now and below 200 ms for a run, this can be tweaked to whatever you like. Currently, show casing how one can filter and only run Sep methods by --filter *.Sep* allowing for incremental updates.

Method,Job,AnalyzeLaunchVariance,EvaluateOverhead,MaxAbsoluteError,MaxRelativeError,MinInvokeCount,MinIterationTime,OutlierMode,Affinity,EnvironmentVariables,Jit,LargeAddressAware,Platform,PowerPlanMode,Runtime,AllowVeryLargeObjects,Concurrent,CpuGroups,Force,HeapAffinitizeMask,HeapCount,NoAffinitize,RetainVm,Server,Arguments,BuildConfiguration,Clock,EngineFactory,NuGetReferences,Toolchain,IsMutator,InvocationCount,IterationCount,IterationTime,LaunchCount,MaxIterationCount,MaxWarmupIterationCount,MemoryRandomization,MinIterationCount,MinWarmupIterationCount,RunStrategy,UnrollFactor,WarmupCount,LineCount,Mean [ms],Error [ms],StdDev [ms],Gen0,Gen1,Gen2,Allocated [KB]
Sep_MT,Job-LGJPSJ,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 8.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,Default,Default,Default,11,5,Default,5,3,Default,16,Default,1000000,309.0,35.37,25.58,16000.0000,15000.0000,1000.0000,268686.44
Sep,Job-LGJPSJ,False,Default,Default,Default,Default,Default,Default,11111111111111111111111111111111,Empty,RyuJit,Default,X64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 8.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,Default,Default,Default,Default,11,5,Default,5,3,Default,16,Default,1000000,682.5,24.98,18.06,18000.0000,17000.0000,3000.0000,266674.98

Example console output

[joelverhagen]

Wow, that's an amazing delta in mean between multi and single threaded! Nice work!

[joelverhagen]

This will take a long time to run I think on my PC, but I guess that's fine. We'll see how it goes next time I update the blog. The original SimpleJob parameters were a nice balance between stability and fast runtime. I don't feel strongly.

[nietras]

I actually just tweaked the params based on some more running. I underestimated how slow some of the entries are... I'll submit another PR 😅

[nietras]

Wow, that's an amazing delta in mean between multi and single threaded! Nice work!

Thanks, it should be close to 2x but with server gc is even more. For the floats bench I have in Sep I am seeing up to 7x speedup for multi-threaded. The parallel implementation is extremely efficient. 🚀 Note that those specific numbers are on dotnet 8.