Enforce information that can verify the validity of the bug.

[insuyun]

Hi, all. I am Insu Yun, an assistant professor at KAIST. First of all, I would like to thank you for your great work. I have also been experiencing similar issues and was planning to conduct similar research. Your research is more thorough than I had anticipated and is truly excellent.

I would like to add some instructions to the guideline I am considering. I would appreciate it if you can share your opinions about this.

2. Identify suitable targets for the evaluation
+ - Do not use targets that are not being actively developed

I still see many JavaScript studies using ChakraCore as their target. Unfortunately, ChakraCore has been barely maintained, with only about 20 commits applied since 2022. I believe we need to work on targets that are actively maintaining.

+ 3.1.1.5. specify whether the bug has been reported, confirmed, is a duplicate (exists in the latest version but has already been reported by other people), or has been patched.
+ 3.1.1.6 provide a tracker for the bug to validate it (e.g., a bug report ID or commit hash if it has been patched).

As you mentioned in the paper, we need to be be possible to evaluate the bugs. Thus, when reporting a new bug in the paper, we need sufficient information to track the bug. Therefore, I believe the above things should be enforced.

Thank you in advance.

[mu00d8]

Hi Insu,

Thank you for your kind words. Let me first apologize for my late response, I haven't seen this issue before. I had failed to properly 'watch' this repo, so I did not get any emails. Sorry!

+ - Do not use targets that are not being actively developed

I agree; only issue I see is that this might conflict with people using intentionally old versions for comparability purposes, for example. Maybe we can word this along the lines of:

- Do not use targets that are not being actively developed; in cases where such targets are required, explain your choice

What do you think?

+ 3.1.1.5. specify whether the bug has been reported, confirmed, is a duplicate (exists in the latest version but has already been reported by other people), or has been patched.
+ 3.1.1.6 provide a tracker for the bug to validate it (e.g., a bug report ID or commit hash if it has been patched).

Sounds good to me, this is important context everybody should provide.

Thanks for chiming in and raising these excellent points.