Turn a GitHub repo into a single text file for LLM-friendly input (repost)

Stephen Turner

20 mins ago

[This article was first published on Getting Genetics Done, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

< mark>Reposted from the original at https://blog.stephenturner.us/p/github-repo-to-text-for-llm-input.

—

If you use ChatGPT, Claude, or even some local model through Ollama or HuggingFace Assistants, you’ll know that the chat interface makes it challenging to feed in an entire repo like a Python or R package, because functions, tests, etc. can be scattered across many files throughout a repo. Here I’ll demonstrate how to turn an entire GitHub repo into a single text file for LLM-friendly input, with R and Python packages as examples.

GitHub Repo to Text Converter

This GitHub Repo to Text Converter can help here.

Web app: repo2txt.simplebasedomain.com
Source: github.com/abinthomasonline/repo2txt

This little app takes a URL to a GitHub repo, lets you select which files in the repo directory structure to include, and creates a single plain text file you can feed into an LLM to ask for explanations, usage, or anything else you’d like to ask about the codebase. You can download or copy this text to the clipboard to later paste into an LLM of your choosing.

The tool runs entirely in the browser. You can also provide a personal access token if you want to access private repositories (again, securely, since everything runs in the browser).

Demo: Python package with GPT-4o

I recently demonstrated writing a CLI app using Click from a Cookiecutter template in Python. It’s a simple app that tells you how much caffeine remains in your system based on how much you consume and when you’ll go to bed. It’s on GitHub at stephenturner/caffeinated.

< picture style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgb(59 130 246 / 0.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 #0000; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 #0000; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 #0000; --tw-shadow: 0 0 #0000; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0;">< source srcset_temp="https://substackcdn.com/image/fetch/w_140,h_140,c_fill,f_webp,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F03154a6a-d796-4511-922f-c1391c266d53_3416x2444.png" style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgb(59 130 246 / 0.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 #0000; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 #0000; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 #0000; --tw-shadow: 0 0 #0000; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0;" type="image/webp">

Build a Python CLI with Click+Cookiecutter

Stephen Turner

Nov 10

Read full story< svg class="lucide lucide-arrow-right" fill="none" height="16" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" stroke="currentColor" viewbox="0 0 24 24" width="16" xmlns="http://www.w3.org/2000/svg">< path d="M5 12h14">< path d="m12 5 7 7-7 7">

Here I’m pasting the repo URL, https://github.com/stephenturner/caffeinated into the app. The tool is smart enough to recognize that this is a Python package, and I probably want the __init__.py, __main__.py, other package .py files, and the unit tests in tests/ that I’ll use with pytest.

After hitting “Generate Text File” you’ll see that it generates a single plain text file, first listing out the directory structure of the selected files, then listing out the contents of each file separately. Here’s a preview. My __init__.py is empty (just to signal that this is a package), and the cli.py file is truncated here.

Directory Structure:

└── ./
    ├── caffeinated
    │   ├── __init__.py
    │   ├── __main__.py
    │   └── cli.py
    └── tests
        └── test_caffeinated.py


---
File: /caffeinated/__init__.py
---


---
File: /caffeinated/__main__.py
---

from .cli import caffeinated

if __name__ == "__main__":
    caffeinated()


---
File: /caffeinated/cli.py
---

import click
import math
from datetime import datetime
from importlib.metadata import version, PackageNotFoundError

### TRUNCATED ###

Once you have this, you can easily paste this into an LLM of your choice to chat with the codebase. Here, I pasted this into GPT-4o with the leading prompt: “Explain this code to me, how to use it what it does.” GPT-4o returned all of the text and code below as a single response.

First, the directory structure.

GPT-4o explaining the package’s directory structure.

Next, an explanation of __main__.py, cli.py, and the tests.

GPT-4o explaining the key files in this repo.

Next, a demonstration of how to use it. Note above that I did not select the README to add to my text file. GPT-4o divined this usage information from the code itself, rather than regurgitating the README.

GPT-4o explaining example output from this package.

And finally, a simple explanation of the (minimal) test suite.

GPT-4o explaining how to test the package with pytest.

Demo: R package with Claude 3.5 Sonnet

This also works on an R package. In my last job I wrote a package to assist with containerizing R packages with a usethis-like interface. It’s called pracpac (practical R packaging [with Docker]), and you can read more in the paper or on the GitHub repo. Here I’ll take the code from pracpac repo and pull out the relevant R source files.

Generating a single text file from an R package repo.

This time I asked Claude 3.5 Sonnet to tell me more about this package.

Claude 3.5 Sonnet explaining the pracpac R package.

It’s just HTML+JavaScript

If you look at the source code you’ll see that this is a simple implementation using just HTML and JavaScript. Which means you can download the repo and open the index.html, or if you’d prefer a vanity URL, you can fork the repo and set up GitHub pages to serve from your main branch (e.g. stephenturner.github.io/repo2txt).

Getting Genetics Done by Stephen Turner is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

To leave a comment for the author, please follow the link and comment on their blog: Getting Genetics Done.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Build a Python CLI with Click+Cookiecutter

Related