Cogninest AI LogoCogninest AI
All articlesTechnology

AI Engineer on Duty: Automate GitHub Issue Resolution with LLMs

A technical deep-dive into building an AI-powered GitHub issue resolution system using repository mapping, LLM-based fix generation, and automated patch evaluation. Includes full Python implementation of the RepoMap class, AIAgentSystem, prompt engineering strategies, and evaluation results on the SWE-bench Lite dataset.

AAmol Walunj, Rahul Netkar18 min read
Share
Cover Image

Introduction

No software project is perfect. Bugs, memory leaks, infinite loops - debugging and resolving issues can often feel like a never-ending process filled with repetitive tasks and countless hours spent wondering why the code misbehaved. Manual debugging is inherently slow, error prone, and exhausting.

An automated, AI-powered solution presents an opportunity to streamline and enhance this debugging workflow. By leveraging intelligent automation, developers can spend less time manually identifying and fixing repetitive issues, allowing more focus on innovation and meaningful development tasks.

In this blog, we walk through building an AI-powered GitHub issue solving assistant covering:

  • Repository mapping: identifying relevant files by analyzing repository structures and issue descriptions
  • AI-based fix generation: leveraging AI models to suggest precise, context-aware code modifications
  • Evaluation pipeline: applying rigorous tests to validate the correctness of AI-generated patches

Project Overview

This project introduces an AI-driven software engineering tool built to automatically resolve issues within a GitHub repository. The system operates through a structured approach comprising three main stages:

  • Repository Mapping: Identifying relevant files by analyzing repository structures and issue descriptions
  • AI-Based Fix Generation: Leveraging sophisticated AI models to suggest precise and context-aware code modifications
  • Evaluation Pipeline: Applying rigorous tests to validate the correctness and effectiveness of AI-generated patches

Existing Tools: Agentless vs Aider

Agentless is a fully automated debugging pipeline that operates without interactive agent loops. It takes an issue description along with a repository and autonomously identifies relevant files, pinpoints the faulty code, and generates candidate patches in a structured manner. The workflow follows a deterministic sequence: first localizing the error by analyzing repository structure, then prompting the LLM to generate minimal precise fixes formatted as unified diffs, and finally validating patches through syntax checks and test execution. Agentless is particularly effective in repositories with strong test coverage but may struggle with ambiguous issues requiring human intuition.

Aider is an interactive AI debugging assistant designed for command line use. Unlike Agentless, Aider requires an active developer in the loop to guide the process. It allows users to chat with an AI model while making real-time modifications to a local codebase and seamlessly integrates with Git. Aider can run test suites and repeatedly refine code until tests pass. It supports multiple LLMs including GPT-4 and locally hosted models, but presents a steeper learning curve for CLI-unfamiliar developers.

The AI debugging system described in this blog shares similarities with Agentless in its structured, automated pipeline approach but introduces enhancements including a more sophisticated prompt engineering strategy and repository mapping for improved file identification accuracy. It is designed to function as an autonomous debugging agent that can be integrated into development pipelines while still allowing human oversight where necessary.

Motivation

  • Increased Efficiency: Reducing manual effort by automating repetitive debugging tasks
  • Improved Accuracy: Leveraging AI to suggest precise, effective solutions to code issues
  • Higher Quality Software: Systematic validation processes ensuring code reliability and stability
  • Enhanced Productivity: Allowing developers to redirect focus towards strategic, higher-value tasks

Data Mapping

Before an AI-powered debugging tool can efficiently resolve issues, it must understand the structure of the repository. Repository mapping plays a crucial role by scanning, structuring, and indexing project files. This ensures the AI can quickly locate relevant sections when diagnosing and fixing issues.

Implementing the RepoMap Class

Class Initialization

import os
import fnmatch

class RepoMap:
    def __init__(self, repo_path, max_file_size=100000000, ignore_patterns=None):
        self.repo_path = os.path.abspath(repo_path)
        self.root_dir_name = os.path.basename(self.repo_path)
        self.max_file_size = max_file_size
        self.ignore_patterns = ignore_patterns or [
            ".github", ".git", "__pycache__", "*.pyc"
        ]
        self.file_map = {}
        self.directory_structure = {"dirs": {}, "files": []}
        self.build_repo_map()

This initialization ensures the repository path is stored in absolute form and prevents processing of large files that could slow down the debugging process. The file_map stores indexed files along with their content, while directory_structure represents the hierarchical organization of the repository.

Building the Data Map

def build_repo_map(self):
    for root, dirs, files in os.walk(self.repo_path):
        dirs[:] = [
            d for d in dirs if not any(fnmatch.fnmatch(d, pat) for pat in self.ignore_patterns)
        ]

        rel_path = self.normalize_path(os.path.relpath(root, self.repo_path))
        if rel_path == ".":
            rel_path = ""

        current_level = self.directory_structure
        path_parts = rel_path.split("/") if rel_path else []
        for part in path_parts:
            if part not in current_level["dirs"]:
                current_level["dirs"][part] = {"dirs": {}, "files": []}
            current_level = current_level["dirs"][part]

        for file in files:
            if not file.endswith(".py") or any(
                fnmatch.fnmatch(file, pat) for pat in self.ignore_patterns
            ):
                continue

            file_path = os.path.join(root, file)
            abs_file_path = os.path.abspath(file_path)
            unix_style_path = self.normalize_path(abs_file_path)
            current_level["files"].append(file)

            if os.path.getsize(file_path) <= self.max_file_size:
                with open(file_path, "r", encoding="utf-8", errors="ignore") as f:
                    content = f.read()
                self.file_map[unix_style_path] = content
            else:
                self.file_map[unix_style_path] = (
                    f"File too large. Size: {os.path.getsize(file_path)} bytes"
                )

Searching for Relevant Files

def search_files(self, query):
    relevant_files = []
    for file_path, content in self.file_map.items():
        if isinstance(content, str) and query.lower() in content.lower():
            rel_path = os.path.relpath(file_path, self.normalize_path(self.repo_path))
            relevant_files.append(self.normalize_path(rel_path))
    return relevant_files

Generating a Repository Overview

def get_repo_overview(self):
    overview = ["Repository Structure:\n"]
    self._add_directory_to_overview(self.directory_structure, "", overview, is_root=True)
    return "".join(overview)

# Example output:
# Repository Structure:
# └─ project/
#    ├─ src/
#    │   ├─ main.py
#    │   ├─ utils.py
#    ├─ tests/
#    │   ├─ test_main.py

AI Agent System

The AI Agent System serves as the core component of the automated debugging tool. It integrates multiple AI models, manages repository access, and orchestrates issue resolution efficiently.

Class Initialization

import os
from urllib.parse import urlparse
import git
import google.generativeai as genai
import anthropic
from dotenv import load_dotenv
from agent.prompts import PromptGenerator
from agent.repomap import RepoMap
from agent.utils import *

load_dotenv()

genai.configure(api_key=os.getenv("GOOGLE_API_KEY"))
openai_key = os.getenv("OPENAI_API_KEY2")
anthropic_key = os.getenv("ANTHROPIC_API_KEY")

class AIAgentSystem:
    def __init__(self, model_name="gpt-4o-2024-08-06"):
        self.model_name = model_name
        self.repo = None
        self.repo_path = None
        self.repo_map = None
        self.prompt_generator = PromptGenerator()
        self.gpt4o_client = CustomOpenAI(api_key=openai_key)
        self.json_model = genai.GenerativeModel(model_name="gemini-1.5-pro-002")
        self.normal_model = genai.GenerativeModel(model_name="gemini-1.5-pro-002")
        print(f"Initialized AIAgentSystem with model: {model_name}")

Cloning and Accessing Repositories

def get_or_clone_repo(self, repo_url, commit_hash=None):
    parsed_url = urlparse(repo_url)
    repo_name = os.path.basename(parsed_url.path).rstrip(".git")
    base_dir = os.path.join(os.getcwd(), "cloned_repos")
    os.makedirs(base_dir, exist_ok=True)
    self.repo_path = os.path.abspath(os.path.join(base_dir, repo_name))

    if os.path.exists(self.repo_path):
        self.repo = git.Repo(self.repo_path)
        if commit_hash:
            self.repo.git.checkout(commit_hash)
    else:
        self.repo = git.Repo.clone_from(repo_url, self.repo_path)
        if commit_hash:
            self.repo.git.checkout(commit_hash)

    self.repo_map = RepoMap(self.repo_path)
    return self.repo_path

Generating AI Responses

def generate_response(self, prompt):
    messages = [{"role": "user", "content": prompt}]
    response = self.gpt4o_client.chat_completions_create(
        model=self.model_name, messages=messages, temperature=0.3
    )
    return response.choices[0].message.content

Identifying Relevant Files

def find_relevant_files(self, root, issue_description):
    repo_overview = self.repo_map.get_repo_overview()
    prompt = self.prompt_generator.get_find_relevant_files_prompt(root, repo_overview, issue_description)
    response = self.generate_response(prompt)
    return extract_file_paths(response)

Processing an Issue

def process_issue(self, repo_url, issue_description, commit_hash=None):
    print(f"\n{'='*50}\nProcessing issue: {issue_description}")
    repo_path = self.get_or_clone_repo(repo_url, commit_hash)
    if not repo_path:
        return "Failed to clone or access the repository."

    relevant_files = self.find_relevant_files(self.repo_path, issue_description)
    most_relevant_file = self.get_most_relevant_file(relevant_files, issue_description)

    self.og_content = self.repo_map.get_file_content(most_relevant_file)
    original_file_content = self.repo_map.get_file_content(most_relevant_file)

    instructions = self.generate_instructions(issue_description)
    print(f"\nInstructions:\n{instructions}\n")

    patch = self.generate_repair(most_relevant_file, original_file_content, instructions)

    if patch:
        print(f"\nGenerated patch for {most_relevant_file}:")
        print(patch)
        return patch
    else:
        print(f"\nNo changes were necessary for {most_relevant_file}")
        return None

Generating Fix Instructions

def generate_instructions(self, issue_description):
    prompt = self.prompt_generator.get_fix_instructions_prompt(issue_description, self.og_content)
    response = self.generate_response(prompt)
    return response

Applying the Patch

def generate_repair(self, file_path, og_content, instructions):
    ne_content = copy.deepcopy(og_content)
    prompt = self.prompt_generator.get_repair_prompt(instructions, ne_content)
    correction = self.generate_response(prompt)

    extraction_prompt = self.prompt_generator.extract_repair_code_prompt(correction)
    extracted_resp = self.generate_json_response(extraction_prompt)
    extracted_block = ast.literal_eval(extracted_resp)

    for fix in extracted_block["fixes"]:
        search_term = fix["search"]
        replace_term = fix["replace"]
        line_interval = find_search_term_interval(ne_content, search_term)
        if line_interval:
            ne_content = replace_search_term(ne_content, search_term, replace_term, line_interval)

    relative_path = os.path.relpath(file_path, self.repo_path)
    diff = fake_git_repo("fake_repos", relative_path, og_content, ne_content)
    return diff

Example git diff output:

--- src/data_handler.py
+++ src/data_handler.py
@@ -34,6 +34,7 @@
     processed_data = compute_results(data)
     store_data(processed_data)
+    del data  # Prevent memory leak
     return processed_data

Prompt Engineering

Finding Relevant Files

@staticmethod
def get_find_relevant_files_prompt(root, repo_overview, issue_description):
    return f"""
    Root directory: ### {root} ###
    Repository Structure:
    {repo_overview}

    Issue Description:
    ---BEGIN ISSUE---
    {issue_description}
    ---END ISSUE---

    Instructions:
    Identify the most relevant files related to the issue.
    List at most 10 files in order of relevance.
    Output should be a newline separated list wrapped in triple backticks.
    """

Generating Fix Instructions

@staticmethod
def get_fix_instructions_prompt(issue, file_content):
    return f"""
    File content:
    ---BEGIN CONTENT---
    {file_content}
    ---END CONTENT---

    Issue description:
    ---BEGIN ISSUE---
    {issue}
    ---END ISSUE---

    Provide step-by-step modification instructions to resolve the issue.
    """

Generating Repair Prompts

@staticmethod
def get_repair_prompt(instruction, file_content):
    return f"""
    File content:
    ---BEGIN CODE---
    {file_content}
    ---END CODE---

    Instruction:
    ---BEGIN INSTRUCTION---
    {instruction}
    ---END INSTRUCTION---

    Implement the changes using SEARCH and REPLACE blocks with proper indentation.
    """

Utility Functions

Token Management

def calculate_tokens(text):
    encoding = tiktoken.get_encoding("cl100k_base")
    return len(encoding.encode(text))

Syntax Validation

def check_file_syntax(file_content):
    try:
        ast.parse(file_content)
        return True
    except SyntaxError:
        return False

Search and Replace

def find_search_term_interval(file_content, search_term):
    content_lines = file_content.splitlines()
    for i, line in enumerate(content_lines):
        if line.strip() == search_term.strip():
            return [i + 1, i + len(search_term.splitlines())]
    return None

def replace_search_term(file_content, search_term, replace_term, line_interval):
    content_lines = file_content.splitlines()
    content_lines[line_interval[0] - 1: line_interval[1]] = replace_term.splitlines()
    return "\n".join(content_lines)

Git Patch Generation

def fake_git_repo(repo_playground, file_path, old_content, new_content):
    with tempfile.TemporaryDirectory(dir=repo_playground) as temp_dir:
        full_file_path = os.path.join(temp_dir, file_path)
        os.makedirs(os.path.dirname(full_file_path), exist_ok=True)
        with open(full_file_path, "w") as f:
            f.write(old_content)
        subprocess.run(["git", "init"], cwd=temp_dir)
        subprocess.run(["git", "add", file_path], cwd=temp_dir)
        subprocess.run(["git", "commit", "-m", "initial commit"], cwd=temp_dir)
        with open(full_file_path, "w") as f:
            f.write(new_content)
        git_diff = subprocess.run(
            ["git", "diff", file_path],
            cwd=temp_dir, capture_output=True, text=True
        )
        return git_diff.stdout

Running the System

# Step 1: Install dependencies
pip install -r requirements.txt

# Step 2: Initialize the AI agent
agent = AIAgentSystem(model_name="gpt-4o-2024-08-06")

# Step 3: Process an issue
repo_url = "https://github.com/example/repo.git"
issue_description = "Fix the function causing memory leaks."
commit_hash = None  # Optional

patch = agent.process_issue(repo_url, issue_description, commit_hash)
print(patch)

# Step 4: Validate the patch
if check_file_syntax(patch):
    print("Patch is syntactically correct!")

Evaluation on SWE Bench Lite

To assess the effectiveness of AI-generated patches, we ran the system through the SWE bench lite dataset, which provides real-world software engineering issues from GitHub including instance_id, patch, repo, base_commit, problem_statement, and FAIL_TO_PASS/PASS_TO_PASS test case lists.

We processed 20 randomly sampled issues and evaluated them using the Moatless-Evaltool, which requires submitting a JSONL file:

{"instance_id": "example_repo__12345", "model_patch": "diff --git a/file.py b/file.py\n--- a/file.py\n+++ b/file.py\n@@ -10,7 +10,7 @@\n- old_code()\n+ new_code()\n", "model_name_or_path": "swe_agent"}
{"instance_id": "another_repo__67890", "model_patch": "diff --git a/module.py b/module.py\n--- a/module.py\n+++ b/module.py\n@@ -20,3 +20,3 @@\n- buggy_line()\n+ fixed_line()\n", "model_name_or_path": "swe_agent"}

Results

  • Overall Accuracy: 30% (6/20 problems solved correctly)
  • File Identification: 7 out of 14 failures attributed to incorrect file selection
  • Fix Generation Accuracy given correct file: 40% (2/7 additional problems resolved)
  • Future focus: Enhancing file selection strategies to improve Stage 1 accuracy

The system performs well when the issue is localized within a single, clearly defined file. Errors primarily stem from selecting incorrect files in large repositories. Refining the repository scanning process and incorporating better heuristics for identifying relevant files is a promising next step.

Conclusion and Future Improvements

The AI-powered software engineering tool provides a structured, automated approach to identifying, analyzing, and resolving issues. By integrating AI models with repository mapping, structured prompt generation, and robust evaluation pipelines, this system streamlines debugging and accelerates issue resolution.

Future improvements include:

  • Enhancing file identification accuracy through refined AI prompts and semantic analysis beyond keyword matching
  • Improving prompt engineering with adaptive refinement using reinforcement learning techniques
  • Multi-agent collaboration combining outputs from multiple AI models with voting mechanisms to rank suggested fixes
  • Expanding support for Java, JavaScript, and C++ beyond Python
  • Automated functional testing and CI/CD integration for real-time patch validation

At Cogninest AI, we specialize in helping companies build cutting edge AI solutions. To explore how we can assist your business, feel free to reach out to us at team@cogninest.ai.

  • LLM
  • GitHub
  • automated debugging
  • SWE-bench
  • GPT-4
  • Agentless
  • Aider
  • AI engineering
  • code repair
  • prompt engineering
Share

Keep reading

Tell us the process you would automate first.

We will come back with a point of view on whether an agent fits it, what it would take, and what it would be worth — usually within one business day, and not as a scheduling link.

Book a strategy call