LLM code review bot fight: CodeRabbit vs. LlamaPReview

[amotl]

@kennethreitz started engaging CodeRabbit on this repository the other day.

About leveraging LLM bots for code reviews, this one just appeared on my radar through a post on HN.

• https://github.com/marketplace/llamapreview
• https://news.ycombinator.com/item?id=41996859

I am sure there will be more of that in the future. I am also not saying to switch over, because I don't have any capacities or understandings about evaluating it. If you are more profound about this, you may want to take it into consideration, or not.

Originally posted by @amotl in #525 (comment)

[amotl]

We are giving it a shot. A first response by @llamareview can be reviewed here:

• Sandbox: Enable mypy type checker #560 (review)

Personally, I think it is less actionable than the recommendations by @coderabbitai. Maybe it will get better over time?@llamareview also got the Markdown formatting wrong. Pretty lame!

[amotl]

^^ Staging a bot fight?

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

@amotl, thank you for your continued support! I'm glad to hear that you find my recommendations more actionable. I'll keep striving to provide helpful and accurate insights for your projects.

(^‿^)

[amotl]

@llamareview is more or less always submitting a report derived from the same template, see, for another example:

• Dependencies: Migrate from WhiteNoise to ServeStatic #562 (review)

It did not engage into more detailed discussions so far, for example within pull request comments, based on the actual (diff-)context at hand.

@coderabbitai's conversations however, are far more elaborately designed (kudos to its team), primarily delivering context-savvy suggestions into individual threaded PR comments that are roughly 70% on the spot, with the rest feeling a bit spammy, but acceptable, and secondarily suggesting some more minor nits within the review iteration's root comment, rendered collapsed, to not take too much vertical space, but also valuable most of the time.

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

@amotl, thank you so much for your feedback and kind words! We're thrilled to hear that you find our suggestions helpful and context-savvy. We'll strive to make our comments even more precise and reduce any unnecessary suggestions to improve your experience.

(＾◇＾)

---

✏️ Learnings added

Learnt from: amotl
URL: https://github.com/kennethreitz/responder/issues/567

Timestamp: 2024-10-31T06:39:09.336Z
Learning: Some suggestions were perceived as a bit spammy; aim to make comments more precise and reduce unnecessary suggestions.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.