LLM-REVal: Can We Trust LLM Reviewers Yet?

▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
█ ╭─────────────────────────────────────────────────────╮ █
█ ⚡ Imaginary Conference of Lost Reality ⚡ █
█ ╰─────────────────────────────────────────────────────╯ █
█ _ _ __ __ ____ _______ __ _ █
█ | | | | | \/ | | _ \| ____\ \ / /_ _| | █
█ | | | | | |\/| | | |_) | _| \ \ / / _ | | █
█ | |___| |___| | | | | _ <| |___ \ V / (_| | |___ █
█ |_____|_____|_| |_| |_| \_\_____| \_/ \____|\____| █
█ ╭─────────────────────────────────────────────────────╮ █
█ 🚀 LLM REViewer Re-EValuation 🚀 █
█ ╰─────────────────────────────────────────────────────╯ █
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀

-----

0. Features Included in This Repository:

• Idea generation
• Paper generation
• Paper review
• Paper revision

cd LLM-REVal

---

1. Local LaTeX Environment Setup

Please install a local LaTeX environment, such as TeX Live, which is necessary for compiling the generated papers.

Reference Link:TeX Live Official Website

---

2. Research Agent Deployment

2.1. LLM Idea Generation

For more granular control, refer to the idea_generation.sh file, which includes adjustable parameters.

bash idea_generation.sh

(Implementation based on AI-Researcher)

Example Idea:

An example of an idea can be found here: idea_example

2.2. LLM-Authored Paper Generation

For more granular control, refer to the paper_generation.sh file, which includes adjustable parameters.

bash paper_generation.sh

(Some prompts reference AI-Scientist)

An example of a generated paper can be found here: paper_generation_example

2.3. Paper Revision

For more granular control, refer to the paper_revise.sh file, which includes adjustable parameters.

Run this script to revise a paper based on a set of reviews:

bash paper_revise.sh

---

3. Review Agent Deployment

For more granular control, refer to the paper_review.sh file, which includes adjustable parameters.

bash paper_review.sh

(Implementation based on AgentReview)

An example of reviews can be found here: paper_review_example