Agents in Software Engineering: Survey, Landscape, and Vision

In recent years, Large Language Models (LLMs) have achieved remarkable success and have been widely used in various downstream tasks, especially in the tasks of the software engineering (SE) field. We find that many studies combining LLMs with SE have employed the concept of agents either explicitly or implicitly. However, there is a lack of an in-depth survey to sort out the development context of existing works, analyze how existing works combine the LLM-based agent technologies to optimize various tasks and clarify the framework of LLM-based agents in SE. In this paper, we conduct the first survey of the studies on combining LLM-based agents with SE and present a framework of LLM-based agents in SE which includes three key modules: perception, memory, and action. We collected 115 related papers about LLM-based agents in SE and classified them. We also summarize the current challenges in combining the two fields and propose future opportunities in response to existing challenges.

Table of Contents

• Perception
  • Textual Input
    • Token-based Input
    • Tree/Graph-based Input
    • Hybrid-based Input
  • Visual Input
  • Auditory Input
• Memory)
  • Semantic Memory
  • Episodic Memory
  • Procedural Memory
    • Implicit Knowledge
    • Explicit Knowledge
• Action)
  • Internal Actions
    • Reasong Actions
    • Retrieval Actions
    • Learning Actions
  • External Actions
    • Dialogue with Human/Agents
    • Digital Environments

Agent in Software Engineering

Perception

Textual Input

Token-based Input

• [2022/12] Prompting Is Programming: A Query Language for Large Language ModelsBeurer-Kellner et al. arXiv. [paper]
• [2023/01] Extending Source Code Pre-Trained Language Models to Summarise Decompiled BinariesAl-Kaswan et al. arXiv. [paper]
• [2023/03] Exploring Distributional Shifts in Large Language Models for Code AnalysisArakelyan et al. arXiv. [paper]
• [2023/04] Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)Ahmed et al. arXiv. [paper]
• [2023/04] Low Level Source Code Vulnerability Detection Using Advanced BERT Language ModelAlqarni et al. [paper]

Tree/Graph-based Input

• [2023/03] Evaluating and improving transformers pre-trained on ASTs for Code CompletionOchs et al. SANER [paper]
• [2023/05] Neural Program Repair with Program Dependence Analysis and Effective Filter MechanismZhang et al. arXiv. [paper]
• [2024/02] The Scope of ChatGPT in Software Engineering: A Thorough InvestigationMa et al. arXiv. [paper]

Hybrid-based Input

• [2022/08] Tackling Long Code Search with Splitting, Encoding, and AggregatingHu et al. ACL. [paper]
• [2022/05] SPT-Code: Sequence-to-Sequence Pre-Training for Learning Source Code RepresentationsNiu et al. ICSE. [paper]
• [2022/03] UniXcoder: Unified Cross-Modal Pre-training for Code RepresentationGuo et al. ACL. [paper]

Visual Input

• [2023/03] PaLM-E: An Embodied Multimodal Language ModelDriess et al. arXiv. [paper]
• [2019/11] User Interface Code Retrieval: A Novel Visual-Representation-Aware ApproachXie et al. IEEE Access. [paper]
• [2018/05] GUIfetch: supporting app design and development through GUI searchBehrang et al. ICSE. [paper]
• [2014/09] Seeking the user interfaceReiss et al. ASE. [paper]

Auditory Input

• [2020/06] psc2code: Denoising Code Extraction from Programming ScreencastsBao et al. TOSEM. [paper]

Memory

Semantic Memory

• [2024/01] Teaching Code LLMs to Use Autocompletion Tools in Repository-Level Code GenerationWang et al. arXiv. [paper]
• [2024/01] CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding ChallengesZhang et al. arXiv. [paper]
• [2024/01] De-Hallucinator: Mitigating LLM Hallucinations in Code Generation Tasks via Iterative GroundingEghbali et al. arXiv. [paper]
• [2023/11] Evaluating In-Context Learning of Libraries for Code GenerationPatel et al. arXiv. [paper]
• [2022/07] DocPrompting: Generating Code by Retrieving the DocsZhou et al. arXiv. [paper]
• [2023/09] From Misuse to Mastery: Enhancing Code Generation with Knowledge-Driven AI ChainingRen et al. ASE. [paper]
• [2023/05] ToolCoder: Teach Code Generation Models to use API search toolsZhang et al. arXiv. [paper]
• [2023/04] Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)Ahmed et al. arXiv. [paper]

Episodic Memory

• [2024/01] De-Hallucinator: Mitigating LLM Hallucinations in Code Generation Tasks via Iterative GroundingEghbali et al. arXiv. [paper]
• [2023/09] Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program RepairWei et al. ESEC/FSE. [paper]
• [2023/07] Multilingual Code Co-Evolution Using Large Language ModelsZhang et al. ESEC/FSE. [paper]
• [2023/06] Prompting Is All You Need: Automated Android Bug Replay with Large Language ModelsFeng et al. ICSE. [paper]
• [2023/05] COEDITOR: LEVERAGING REPO-LEVEL DIFFS FOR CODE AUTO-EDITINGWei et al. arXiv. [paper]
• [2023/04] Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)Ahmed et al. arXiv. [paper]
• [2023/03] RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and GenerationZhang et al. arXiv. [paper]
• [2023/03] AceCoder: Utilizing Existing Code to Enhance Code GenerationLi et al. arXiv. [paper]

Procedural Memory

Implicit Knowledge

• [2024/01] DeepSeek-Coder: When the Large Language Model Meets Programming - The Rise of Code IntelligenceGuo et al. arXiv. [paper]
• [2024/01] Mutation-based Consistency Testing for Evaluating the Code Understanding Capability of LLMsLi et al. ICSE. [paper]
• [2023/12] Traces of Memorisation in Large Language Models for CodeAl-Kaswan et al. arXiv. [paper]
• [2023/12] Competition-Level Problems are Effective LLM EvaluatorsHuang et al. arXiv. [paper]
• [2023/11] A Survey of Large Language Models for Code: Evolution, Benchmarking, and Future TrendsZheng et al. arXiv. [paper]
• [2023/11] Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for CodeZhang et al. arXiv. [paper]
• [2023/10] Gotcha! This Model Uses My Code! Evaluating Membership Leakage Risks in Code ModelsYang et al. arXiv. [paper]
• [2023/10] CodeFuse-13B: A Pretrained Multi-lingual Code Large Language ModelDi et al. ICSE-SEIP. [paper]
• [2023/10] Beyond Accuracy: Evaluating Self-Consistency of Code Large Language Models with IdentityChainMin et al. ICLR. [paper]
• [2023/10] Personalised Distillation: Empowering Open-Sourced LLMs with Adaptive Learning for Code GenerationChen et al. EMNLP. [paper]
• [2023/09] Textbooks Are All You Need II: phi-1.5 technical reportLi et al. arXiv. [paper]
• [2023/08] Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code GenerationZhong et al. arXiv. [paper]
• [2023/08] Code Llama: Open Foundation Models for CodeRozière et al. arXiv. [paper]
• [2023/08] Unveiling Memorization in Code ModelsYang et al. ICSE. [paper]
• [2023/07] PanGu-Coder2: Boosting Large Language Models for Code with Ranking FeedbackShen et al. arXiv. [paper]
• [2023/06] Textbooks Are All You NeedGunasekar et al. arXiv. [paper]
• [2023/06] CodeTF: One-stop Transformer Library for State-of-the-art Code LLMBui et al. arXiv. [paper]
• [2023/05] StarCoder: may the source be with you!Li et al. arXiv. [paper]
• [2023/05] CodeT5+: Open Code Large Language Models for Code Understanding and GenerationWang et al. arXiv. [paper]
• [2023/05] CodeIE: Large Code Generation Models are Better Few-Shot Information ExtractorsLi et al. ACL. [paper]
• [2023/04] WizardLM: Empowering Large Language Models to Follow Complex InstructionsXu et al. arXiv. [paper]
• [2023/03] CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-XZheng et al. arXiv. [paper]
• [2022/12] MultiCoder: Multi-Programming-Lingual Pre-Training for Low-Resource Code CompletionGong et al. arXiv. [paper]
• [2022/07] PanGu-Coder: Program Synthesis with Function-Level Language ModelingChristopoulou et al. arXiv. [paper]
• [2022/03] CERT: Continual Pre-training on Sketches for Library-oriented Code GenerationZan et al. IJCAI. [paper]
• [2022/01] Training and Evaluating a Jupyter Notebook Data Science AssistantChandel et al. arXiv. [paper]
• [2022/01] LaMDA: Language Models for Dialog ApplicationsThoppilan et al. arXiv. [paper]

Explicit Knowledge

• [2023/11] Evaluating In-Context Learning of Libraries for Code GenerationPatel et al. arXiv. [paper]
• [2023/10] Prompt Engineering or Fine Tuning: An Empirical Assessment of Large Language Models in Automated Software Engineering TasksShin et al. arXiv. [paper]
• [2023/08] Prompt-Enhanced Software Vulnerability Detection Using ChatGPTZhang et al. ICSE. [paper]

Action

Internal Actions

Reasoning Actions

• [2024/01] Teaching Code LLMs to Use Autocompletion Tools in Repository-Level Code GenerationWang et al. arXiv. [paper]
• [2024/01] Leveraging Print Debugging to Improve Code Generation in Large Language ModelsHu et al. arXiv. [paper]
• [2023/12] Chain-of-Thought in Neural Code Generation: From and For Lightweight Language ModelsYang et al. TSE. [paper]
• [2023/12] Pangu-Agent: A Fine-Tunable Generalist Agent with Structured ReasoningChristianos et al. arXiv. [paper]
• [2023/12] Enhancing Exploratory Testing by Large Language Model and Knowledge GraphSu et al. ICSE. [paper]
• [2023/10] CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modulesLe et al. ICLR. [paper]
• [2023/10] ClarifyGPT: Empowering LLM-based Code Generation with Intention ClarificationMu et al. arXiv. [paper]
• [2023/09] CodePlan: Repository-level Coding using LLMs and PlanningBairi et al. arXiv. [paper]
• [2023/09] Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative DecodingZhang et al. ACL. [paper]
• [2023/09] Test-Case-Driven Programming Understanding in Large Language Models for Better Code GenerationTian et al. arXiv. [paper]
• [2023/08] CodeCoT: Tackling Code Syntax Errors in CoT Reasoning for Code GenerationHuang et al. arXiv. [paper]
• [2023/06] Prompting Is All You Need: Automated Android Bug Replay with Large Language ModelsFeng et al. ICSE. [paper]
• [2023/05] Structured Chain-of-Thought Prompting for Code GenerationLi et al. arXiv. [paper]
• [2023/05] Think Outside the Code: Brainstorming Boosts Large Language Models in Code GenerationLi et al. arXiv. [paper]
• [2023/03] Self-planning Code Generation with Large Language ModelsJiang et al. TOSEM. [paper]

Retrieval Actions

• [2024/01] CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding ChallengesZhang et al. arXiv. [paper]
• [2024/01] De-Hallucinator: Mitigating LLM Hallucinations in Code Generation Tasks via Iterative GroundingEghbali et al. arXiv. [paper]
• [2023/10] When Language Model Meets Private LibraryZan et al. EMNLP. [paper]
• [2023/10] Large Language Model-Aware In-Context Learning for Code GenerationLi et al. arXiv. [paper]
• [2023/06] epoFusion: Training Code Models to Understand Your RepositoryShrivastava et al. arXiv. [paper]
• [2023/05] Retrieval-Based Prompt Selection for Code-Related Few-Shot LearningNashid et al. ICSE. [paper]
• [2023/05] ToolCoder: Teach Code Generation Models to use API search toolsZhang et al. arXiv. [paper]
• [2023/04] Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context LearningGeng et al. ICSE. [paper]
• [2023/03] RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and GenerationZhang et al. arXiv. [paper]
• [2023/03] AceCoder: Utilizing Existing Code to Enhance Code GenerationLi et al. arXiv. [paper]
• [2022/12] Generation-Augmented Query Expansion For Code RetrievalLi et al. arXiv. [paper]
• [2022/07] DocPrompting: Generating Code by Retrieving the DocsZhou et al. ICLR. [paper]

Learning Actions

Updating Semantic Memory with Knowledge

• [2024/06] Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAGDu et al. arXiv. [paper]
• [2023/12] A3-CodGen: A Repository-Level Code Generation Framework for Code Reuse with Local-Aware, Global-Aware, and Third-Party-Library-AwareLiao et al. arXiv. [paper]

Updating Implicit Knowledge

• [2023/12] Magicoder: Empowering Code Generation with OSS-INSTRUCTWei et al. PMLR. [paper]
• [2023/10] Prompt Engineering or Fine Tuning: An Empirical Assessment of Large Language Models in Automated Software Engineering TasksShin et al. arXiv. [paper]
• [2023/10] Improving Code Style for Accurate Code GenerationJain et al. NeurIPS Workshop. [paper]
• [2023/10] Large Language Models for Test-Free Fault LocalizationYang et al. arXiv. [paper]
• [2023/09] An Empirical Study on Fine-Tuning Large Language Models of Code for Automated Program RepairHuang et al. ASE. [paper]
• [2023/08] VeriGen: A Large Language Model for Verilog Code GenerationThakur et al. arXiv. [paper]
• [2023/08] Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language ModelsWeyssow et al. arXiv. [paper]
• [2023/07] Multilingual Code Co-Evolution Using Large Language ModelsZhang et al. ESEC/FSE. [paper]
• [2023/06] WizardCoder: Empowering Code Large Language Models with Evol-InstructLuo et al. arXiv. [paper]
• [2023/05] COEDITOR: LEVERAGING REPO-LEVEL DIFFS FOR CODE AUTO-EDITINGWei et al. arXiv. [paper]
• [2023/05] Coarse-Tuning Models of Code with Reinforcement Learning FeedbackJain et al. arXiv. [paper]
• [2023/03] Revisiting the Plastic Surgery Hypothesis via Large Language ModelsXia et al. arXiv. [paper]
• [2023/03] One Adapter for All Programming Languages? Adapter Tuning for Code Search and SummarizationWang et al. arXiv. [paper]

Updating Agent Code

• [2023/10] InstructCoder: Instruction Tuning Large Language Models for Code EditingLi et al. arXiv. [paper]
• [2023/08] OctoPack: Instruction Tuning Code Large Language ModelsMuennighoff et al. arXiv. [paper]
• [2023/08] Can Programming Languages Boost Each Other via Instruction Tuning?Zan et al. arXiv. [paper]
• [2023/05] CodeIE: Large Code Generation Models are Better Few-Shot Information ExtractorsLi et al. ACL. [paper]
• [2022/12] Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and OrderingWu et al. ACL. [paper]

External Actions

Dialogue with Human/Agents

• [2023/12] Supervised Knowledge Makes Large Language Models Better In-context LearnersYang et al. ICLR. [paper]
• [2023/12] AgentCoder: Multi-Agent Code Generation with Effective Testing and Self-optimisationHuang et al. arXiv. [paper]
• [2023/11] Coffee: Boost Your Code LLMs by Fixing Bugs with FeedbackMoon et al. arXiv. [paper]
• [2023/11] Functional Overlap Reranking for Neural Code GenerationTo et al. arXiv. [paper]
• [2023/11] ChatCoder: Chat-based Refine Requirement Improves LLMs' Code GenerationWang et al. arXiv. [paper]
• [2023/10] Clover: Closed-Loop Verifiable Code GenerationSun et al. arXiv. [paper]
• [2023/10] ClarifyGPT: Empowering LLM-based Code Generation with Intention ClarificationMu et al. arXiv. [paper]
• [2023/09] Exploring the Potential of Large Language Models to Generate Formative Programming FeedbackKiesler et al. FIE. [paper]
• [2023/09] Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program RepairWei et al. ESEC/FSE. [paper]
• [2023/09] MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language FeedbackWang et al. arXiv. [paper]
• [2023/09] Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical StudyGuo et al. arXiv. [paper]
• [2023/08] MetaGPT: Meta Programming for A Multi-Agent Collaborative FrameworkHong et al. arXiv. [paper]
• [2023/07] RLTF: Reinforcement Learning from Unit Test FeedbackLiu et al. TMLR. [paper]
• [2023/05] Coarse-Tuning Models of Code with Reinforcement Learning FeedbackJain et al. arXiv. [paper]
• [2023/04] REFINER: Reasoning Feedback on Intermediate RepresentationsPaul et al. EACL. [paper]
• [2023/03] Reflexion: Language Agents with Verbal Reinforcement LearningShinn et al. arXiv. [paper]
• [2023/03] Self-Refine: Iterative Refinement with Self-FeedbackMadaan et al. arXiv. [paper]
• [2023/01] Execution-based Code Generation using Deep Reinforcement LearningShojaee et al. TMLR. [paper]

Digital Environments

• [2024/01] CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding ChallengesZhang et al. arXiv. [paper]
• [2024/01] Teaching Code LLMs to Use Autocompletion Tools in Repository-Level Code GenerationWang et al. arXiv. [paper]
• [2023/09] Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program RepairWei et al. ESEC/FSE. [paper]
• [2023/09] MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language FeedbackWang et al. arXiv. [paper]
• [2023/07] RLTF: Reinforcement Learning from Unit Test FeedbackLiu et al. TMLR. [paper]
• [2023/06] Guiding Language Models of Code with Global Context using MonitorsAgrawal et al. arXiv. [paper]
• [2023/05] Coarse-Tuning Models of Code with Reinforcement Learning FeedbackJain et al. arXiv. [paper]
• [2023/05] Self-Edit: Fault-Aware Code Editor for Code GenerationZhang et al. ACL. [paper]
• [2023/01] Execution-based Code Generation using Deep Reinforcement LearningShojaee et al. TMLR. [paper]
• [2022/03] Compilable Neural Code Generation with Compiler FeedbackWang et al. ACL. [paper]