Support DocumentDB by replacing certain MongoDB operations

[kwx4957]

What is changed

ChaosCenter

• Add pre-commit check for aws documentDB

Auth Server

• Refactor CreatePaginationStage and add int return values
• $facet to $group and $project
  • Calculate total and retrieve data by creating an array using $push with $$ROOT, then slice the data with $slice
• The result struct changed no longer returns an array

GraphQL Server

• DB schema changed from array to int
• Export and add common function for CreatePaginationStage
• $lookup multiple join seperate each stage $lookup and $project etc

5 Types

• $facet to $count
• $facet to $group
  • Use $switch to count data based on specific cases in groupbyTotal
• $facet to $group and $project
  • Calculate total and retrieve data by creating an array using $push with $$ROOT, then slice the data with $slice
• $facet to $group, $project, and $unwind
• $bucket to $switch
  • Calculate avg_resiliency_score

Test Code

• chaos_experiment_run
  • Changed data type from array to integer

AWS documentDB e2e test result

I ran the e2e test on aws documnetdb instead of mongodb, and got the following results

The Google Drive link below is videos of the e2e test.
https://drive.google.com/drive/folders/1_Rs8TPVlOg7KKDCRqzlgtePDO9Kp4I0O?usp=drive_link

Run simple Experiment

i ran simple chaosexperiment using k6. this is the result.

Related proposal: #4812
Related issue: #4459

[namkyu1999]

Can you check a backend-checks?

[kwx4957]

yes, i will check

[kwx4957]

i find a error when get ListExperimentRun, i will fix it

[Saranya-jena]

Hi @kwx4957 can you perform a benchmarking test to check how these changes have affected the API latency? You can use a mongo compass to run the mongo pipelines and check the latency.

[Saranya-jena]

Otherwise the changes look good to me! 🚀

[kwx4957]

@Saranya-jena Thank you for informing me about the tool I wasn't aware of. yes I will run a benchmarking test.

[S-ayanide]

Hi @kwx4957 were you able to do a benchmark test for the same!

[kwx4957]

@S-ayanide sorry, it's my first benchmark test. it took a time. As soon as possible I will upload the result

[kwx4957]

@Saranya-jena @S-ayanide thanks for waiting. This is the result. I configured a pipeline for each API in mongodb compass, and then used the explain function to measure the values at the bottom.

executionTimeMillis is the overall time for execution. This time is not only the time that the query runs, but it includes the time it takes to generate/pick the execution plan.

executionTimeMillisEstimate is the time for execution.

Each collections data: 500,000

executionTimeMillis

	query name	Before query( avg: 10 cases )	After query( avg: 10 cases)	Diff
Project	GetProjectsByUserID	189.7	163.1	+16%	not used index
ExperimentRun	GetExperimentRun	3.8	2.7	+40%
ExperimentRun	ListExperimentRun	2.8	3.1	-10%
Experiment	GetExperiment	236.2	179.9	+31%
Experiment	getWfRunDetails	0.4	0.4
Experiment	GetExperimentStats	697.9	342.6	+100%	not used index
Experiment	ListExperiment	10.2	3	+304%
Infra	GetInfra	163.1	152.2	+7%
Infra	GetInfraStats	245.4	181.1	+35%	not used index
Infra	ListInfras	802.7	513.4	+56%	not used index
Environment	ListEnvironments	0.1	0.7	-700%
ChaosProbes	GetProbeReference	316.7	258.5	+22%

executionTimeMillisEstimate

	query name	Before query( avg: 10 cases )	After query( avg: 10 cases)	Diff
Project	GetProjectsByUserID	6.8	157.8	-2320%
ExperimentRun	GetExperimentRun	0	0
ExperimentRun	ListExperimentRun	0	0
Experiment	GetExperiment	0	0
Experiment	getWfRunDetails	0	0
Experiment	GetExperimentStats	0	0
Experiment	ListExperiment	7.1	187.9	-2646%
Infra	GetInfra	0	0
Infra	GetInfraStats	10.4	180.1	- 1731%
Infra	ListInfras	5.5	3.3	+66%
Environment	ListEnvironments	0	0
ChaosProbes	GetProbeReference	0	0