GitHub Parser

Extracts information from one or more GitHub pull requests or commits and formats it into structured markdown files in a single output folder. You can extract everything, or selectively pull just one section (for example, only the feedback/comments). A backend PR and its frontend PR can be parsed in one run: their files, feedback and extracted documents are grouped as Backend, then Frontend.

Highlights

• Works with pull requests and commits
• Parses several PRs or commits into one folder, grouped by repository category (Repositories.txt), so a backend PR and its frontend PR read Backend, then Frontend
• Works with public and private repositories
• Selective extraction: choose any of code, feedback
• Handles paginated GitHub API responses for files and comments
• Configurable ignore rules through Ignore.txt
• Configurable extra extraction rules through SeparateExtractionList.txt
• Configurable sections through Categories.txt (by path) and Repositories.txt (by repository)
• Masks passwords, pass phrases, tokens, API keys and connection strings in extracted config files
• Writes a manifest.json describing the sources and every extracted file, for importers
• Generates a desktop output folder for the selected PR or commit
• Produces diff-style markdown for both main code output and separate extraction files
• Shows markdown document changes as word-level diffs, so a one-word edit in a long paragraph stays readable
• Captures every review verdict and comment, including comments on ignored files

Requirements

• Python 3.8+ (Python 3.12 recommended)
• requests 2.31.0
• GitHub personal access token for private repos (recommended for all usage to avoid low rate limits)

Installation

Conda

conda env create -f GitHubParser.yml
conda activate GitHubParser

Note:

• GitHubParser.yml is portable and does not include a machine-specific prefix.

pip

python -m pip install requests==2.31.0

Authentication

For private repositories (and to avoid low rate limits), authenticate with a token. For security the token is never accepted as a command-line argument - it would leak into shell history and process listings. Provide it in one of two ways:

• Set the GITHUB_TOKEN environment variable, or
• Enter it at the secure, no-echo prompt the script shows for a private repo.

The script checks each repository's visibility through the GitHub API, so you are never asked whether a repository is private. A public repository runs with no token question; the token prompt appears only for a private repository when GITHUB_TOKEN is not set. If the API cannot answer (rate limit or network error), the script falls back to asking.

PowerShell:

$env:GITHUB_TOKEN = "your_token_here"

CMD:

set GITHUB_TOKEN=your_token_here

Create token: https://github.com/settings/tokens

Scopes:

• private repos: repo
• public-only access: public_repo

Usage

Recommended (cross-platform)

python GitHubParser.py

Run with no arguments and the script:

1. Prompts for the PR or commit URL(s); several are separated by spaces or commas
2. Checks whether each repository is public or private, and prompts for a token only when one is private and GITHUB_TOKEN is not set
3. Prompts for which sections to extract (default: all)

The token is typed with no echo; once entered, the prompt line shows one * per character.

Output is coloured: progress in the terminal's normal text colour, success green, skipped or missing items yellow, failures red. Colour is used only in a terminal (not when the output is piped or redirected) and works in Windows Terminal, PowerShell and cmd on Windows 10+, macOS and Linux. Set NO_COLOR=1 to turn it off, or FORCE_COLOR=1 to keep it when piping.

Example URLs:

https://github.com/owner/repo/pull/123
https://github.com/owner/repo/commit/<sha>

A commit URL may include a #diff-<hash> fragment (a link to a specific file on the page); it is ignored.

Command-line flags

All prompts can be skipped with flags. Anything not provided falls back to an interactive prompt.

# Extract only feedback (comments) from a commit
python GitHubParser.py --only feedback https://github.com/owner/repo/commit/<sha>

# Extract code + feedback from a PR of a public repo, no prompts
python GitHubParser.py --sections code,feedback --public https://github.com/owner/repo/pull/123

# Private repo, token from the environment
python GitHubParser.py --private --only feedback <url>

# A backend PR and its frontend PR in one run: one folder, grouped Backend / Frontend
python GitHubParser.py --private https://github.com/owner/api/pull/21 https://github.com/owner/ui/pull/19

Flag	Description
url (positional)	PR or commit URL(s). Several URLs go into one folder, grouped by repository (see Several PRs in one run). Prompted if omitted.
-s, --sections, --only	Comma-separated: code, feedback, or all. Alias: comments = feedback. Prompted if omitted (default all).
--private / --public	Skip the automatic public/private check.

There is deliberately no token flag: the token comes only from GITHUB_TOKEN or the secure prompt (see Authentication).

Windows helper scripts

This repository also includes:

• GitHubParser.bat
• GitHubParser.ps1

Important:

• GitHubParser.bat reads machine-specific values from GitHubParser.config.bat.
• Update GitHubParser.config.bat before first use.

The GitHubParser PowerShell function exposes the same options as the Python CLI and forwards them through GitHubParser.bat to GitHubParser.py, so it can be driven directly from PowerShell in the same manner. Anything omitted falls through to the interactive prompt.

# Interactive
GitHubParser

# Usual run: visibility is detected, a token is asked for only if the repo is private
GitHubParser -Url "https://github.com/owner/repo/pull/123"

# Extract only feedback from a commit
GitHubParser -Only feedback https://github.com/owner/repo/commit/<sha>

# Code + feedback from a public PR, no prompts
GitHubParser -Sections code,feedback -Public https://github.com/owner/repo/pull/123

# Private repo: prompt (no echo) for a token, never placing it on the command line
GitHubParser -Private -Token (Read-Host -AsSecureString) -Only feedback <url>

# A backend PR and its frontend PR in one run
GitHubParser -Private https://github.com/owner/api/pull/21 https://github.com/owner/ui/pull/19

Parameter	Description
-Url (positional)	One or more PR or commit URLs. Several URLs go into one folder, grouped by repository. Prompted if omitted.
-Sections / -Only	Comma-separated: code, feedback, or all. Alias: comments = feedback. Accepts an array (code,feedback).
-Token	GitHub token as a SecureString (e.g. (Read-Host -AsSecureString)). Injected via the GITHUB_TOKEN environment variable for the child process only, then cleared - never passed as an argument.
-Private / -Public	Skip the automatic public/private check.

Launcher config

GitHubParser.config.bat contains these editable values:

• conda_prefix: full path to your conda environment, for example C:\\Users\\your-user\\miniconda3\\envs\\GitHubParser
• conda_base: full path to your conda installation, for example C:\\Users\\your-user\\miniconda3
• python_script: full path to GitHubParser.py on your machine

The batch launcher uses this config file to:

• call %conda_base%\\Scripts\\activate.bat %conda_base%
• run conda activate %conda_prefix%
• execute python %python_script% %* (any arguments passed to GitHubParser.bat are forwarded to the Python CLI)

Output

Output is written to a folder on your desktop, named by source:

<Desktop>/PR_123_org_repo/          # for a pull request
<Desktop>/Commit_d974808_org_repo/  # for a commit (short SHA)
<Desktop>/PR_21&19_org_repo/        # for several PRs (named after the first repository)

Files written (only for the sections you requested):

• Code.md (section code)
• Feedback.md (section feedback)
• manifest.json (always): the sources (repository, number, title, author, branches, state, category) and every separately extracted file with its section, group and output path
• Reports/Ignore_Report.txt (only when code is requested)
• Reports/SeparateExtraction_Report.txt (only when code is requested)

Files matched by SeparateExtractionList.txt are written to a subfolder named after their section, mirroring their path in the repository (leading dots are dropped from folder names, so .github becomes github):

PR_123_org_repo/
  Code.md
  Feedback.md
  MD/
    README.md.md
    docs/setup.md.md
  Config/
    src/Api/appsettings.json.md
  Dependencies/
    src/Api/MyProject.csproj.md
    web/package.json.md
  CI/
    github/workflows/build.yml.md
  Reports/
    Ignore_Report.txt
    SeparateExtraction_Report.txt

Running the parser again for the same PR or commit overwrites files of the same name; files from an earlier run that are no longer produced stay in the folder.

Several PRs in one run

Give several URLs and they are written to one folder, PR_21&19_org_repo (the identifiers joined with &, named after the first repository). Every source gets a group: its repository's category from Repositories.txt, or its repository name when it is not listed. Two sources that would share a label (two PRs of one repository) get the PR number appended.

• Code.md: one ## <group> per source, so ## Backend then ## Frontend
• Feedback.md: one ## <group> per source holding its ### Reviews, ### Conversation and ### Comments (per-file headings become ####)
• Separately extracted files: the group is the folder after the section, so two repositories with the same path cannot collide: MD/Backend/CHANGELOG.md.md, MD/Frontend/CHANGELOG.md.md
• Reports and manifest.json label every file with its group

PR_21&19_org_api/
  Code.md                       # Backend, then Frontend
  Feedback.md                   # Backend, then Frontend
  manifest.json
  Config/Backend/src/Api/appsettings.json.md
  MD/Backend/CHANGELOG.md.md
  MD/Frontend/CHANGELOG.md.md
  Reports/...

Configuration

Ignore.txt

Controls which files are left out entirely: they are not in Code.md and not extracted separately. It does not filter feedback: review comments on ignored files still appear in Feedback.md.

The starter list ignores lock files, generated code (migrations, *.Designer.cs, *.g.cs, snapshots), build output and bundles (dist/, bin/, *.min.js, *.map), vendored code (node_modules/, vendor/) and binary assets (images, fonts, archives).

Pattern behavior (shared by all three pattern files):

• One glob pattern per line
• Lines starting with # are comments
• Optional negation with !pattern
• Last matching rule wins
• * matches across folders, so *.md matches docs/setup.md too
• A leading **/ also matches at the repository root, as in .gitignore: **/*.md matches CHANGELOG.md and docs/setup.md

SeparateExtractionList.txt

Controls which changed files get their own markdown diff file instead of a place in Code.md. A [Section] header names the output subfolder for the patterns below it; when several sections match, the last matching line wins.

[MD]
**/*.md

[Config]
**/appsettings*.json
**/*.yml

Starter sections:

• MD: markdown documents
• Config: appsettings, *.config, .env.example, YAML, Dockerfile, docker-compose files
• Dependencies: package.json, *.csproj, Directory.*.props, requirements*.txt, pyproject.toml, go.mod, Cargo.toml, pom.xml, Gradle build files
• CI: .github/workflows/*, Azure Pipelines and GitLab CI files (listed last, so workflow YAML lands here rather than in Config)

Categories.txt

Controls the sections of Code.md, with the same [Section] syntax: a file goes to the section of its last matching pattern, and to Other when nothing matches. Sections appear in alphabetical order. The starter file puts ui/, frontend/ and JavaScript/TypeScript/JSX/Vue files under Frontend, and api/, C#/Java/Python/Go files and Localization/ resources under Backend (listed second, so it wins when both match).

These rules are for monorepos and unlisted repositories. A repository listed in Repositories.txt skips them: all its files get that section.

Repositories.txt

Gives a whole repository a section, with the same [Section] syntax. Patterns are matched case-insensitively against owner/repo; a pattern without a / is matched against the repository name alone, and the last matching line wins.

[Frontend]
*-ui
*-frontend

[Backend]
*-api
owner/MyProject

A listed repository's files all go under that section in Code.md, whatever their paths. In a run with several PRs, the section is also the group of the repository's feedback and separately extracted files (see Several PRs in one run). Leave a monorepo unlisted so Categories.txt splits it by path.

Secrets in config files

Separately extracted files of the Config section have the values of secret-looking keys masked as *** before they are written: keys containing password, passphrase, secret, token, apikey/api_key, clientsecret, privatekey, connectionstring, accountkey or sharedaccesskey, in JSON pairs, YAML pairs, XML attributes, .env lines and Key=value; parts of connection strings. The key stays, so the diff still shows what changed. Code snippets of comments on those files are masked the same way, and Reports/SeparateExtraction_Report.txt and manifest.json say how many values each file had masked.

.env and .env.*, secrets*.json, local.settings.json and *.tfvars files are in the Config section, so they are masked too. Key and certificate files (*.pem, *.key, *.pfx, *.p12, *.jks, *.keystore, *.snk, id_rsa*, id_ecdsa*, id_ed25519*) are in Ignore.txt: they are never written out, only listed in the Ignore report.

Secrets anywhere

Credentials with a recognisable format are masked as *** in every file the script writes, Code.md and Feedback.md included: GitHub tokens (ghp_, gho_, ghu_, ghs_, ghr_, github_pat_), AWS access key IDs, Slack tokens, Google API keys, Stripe live keys, OpenAI/Anthropic keys, JWTs, and the body of private key blocks. A token hard-coded in source or pasted into a review comment is caught this way; the Saved: line says how many were masked.

Safety

The script only reads from GitHub (it sends GET requests to api.github.com and nothing else), and only writes inside its own output folder on the Desktop. The token is sent only to api.github.com, is never printed, never written to a file and never taken as a command-line argument.

Output format details

Code.md and separate extraction files

Code diffs are rendered in markdown code blocks with old/new sections when patch data is available.

Files matched by SeparateExtractionList.txt are not included in Code.md; they are written only to separate extracted markdown files.

When GitHub does not provide textual patch data for a file (binary files, very large diffs, renames without changes), the file keeps its heading with a note saying so, both in Code.md and in separate extraction files.

Markdown files

Markdown documents are prose, so they get a layout of their own:

• A new file is shown as the document itself.
• Each change in a modified file is a diff block with one line of context around it (context lines are cut at 120 characters):
  • + added line, - removed line
  • ! line edited in place, with the edit marked as [-removed-]{+added+} and unchanged stretches shortened to …, so a one-word change in a long paragraph stays visible
• Changes that only touch whitespace (re-padded tables, re-indented lists, re-flowed text) are left out and listed in one note with their line numbers.
• An added line identical to the line next to it is flagged as a possible accidental duplicate, a common leftover of merge conflicts resolved by keeping both sides.
• A removed file lists its lines as removals.

Feedback.md

All feedback is captured, whether or not its file matches Ignore.txt:

• Reviews, first: each reviewer's verdict (Approved / Changes requested / Dismissed) with its date and summary. A plain "Commented" review appears only when it has a message.
• Conversation: Conversation-tab comments, or for a commit, comments on the commit as a whole.
• Comments: inline comment threads, grouped by file in Code.md order and sorted by line. Each heading names what the comment was left on: · line 57, · lines 50-57, · whole file, or · line 57 (outdated) once later pushes changed that code. Replies follow their comment; a reply whose parent was deleted is kept as its own thread.

Comment text keeps the reviewer's formatting (paragraphs, lists, code blocks). When code is extracted in the same run, a comment on a file that is not in Code.md gets a note saying so and naming the file's separately extracted diff, if there is one.

Code snippets for anchored comments are extracted with this behavior:

• For PR review comments, the code snippet comes from the comment's diff hunk (the code as it was when commented), positioned with the comment's original line numbers
• For commit comments (which have no diff hunk), the snippet comes from the file's patch using the comment's diff position, or its line
• Exact selected range when a multi-line range was selected
• Otherwise the line with surrounding context
• Tail fallback when line metadata is unavailable
• Whole-file comments have no snippet

Troubleshooting

404 Not Found

Common causes:

• private repository without token
• incorrect PR/commit URL
• missing repository access

Missing files in output

• Check Reports/Ignore_Report.txt for ignored files
• Check Reports/SeparateExtraction_Report.txt for files matched by SeparateExtractionList.txt and where they were written
• Verify patterns in Ignore.txt and SeparateExtractionList.txt

Rate limit issues

Unauthenticated GitHub API usage is limited. Set GITHUB_TOKEN to increase limits. When the limit is reached, the parser stops and says when it resets. Each request times out after 30 seconds.

Repository files

• GitHubParser.py: main parser
• Ignore.txt: ignore rules
• SeparateExtractionList.txt: rules for separately extracted files and their output folders
• Categories.txt: rules for the sections of Code.md (by file path)
• Repositories.txt: rules for the section of a whole repository
• LICENSE: MIT License
• GitHubParser.yml: conda environment spec
• GitHubParser.bat: batch launcher (path-dependent)
• GitHubParser.ps1: PowerShell wrapper (path-dependent)

License

This project is released under the MIT License.

In practical terms: you can copy, modify, use, publish, and reuse this code (including commercially), as long as the license notice is included with substantial portions of the software.

Contributing

Issues and pull requests are welcome.

When proposing changes or addressing issues please include:

• what changed
• why it changed
• expected output example
• what is the issue and how to reproduce it

---