GitHub Extractor

🤖 AI-Assisted Development: This project was developed with assistance from Claude (Anthropic) for rapid prototyping and code generation.

A powerful CLI tool to extract and clone repositories from GitHub Enterprise organizations, with full support for SAML-authenticated environments.

✨ Features

• 🔐 SAML Authentication - Full support for GitHub Enterprise SAML-enabled organizations
• 📦 Batch Operations - Download all repositories or select specific ones
• 🎯 Interactive Selection - Choose repositories from an interactive checkbox list
• 🔒 Multiple Auth Methods - Personal Access Tokens with SSO support
• ⚡ Performance - Shallow clone support for faster downloads
• 🌐 GitHub Enterprise - First-class support for Enterprise instances
• 🔧 Flexible Options - SSH/HTTPS protocols, custom output directories

📋 Table of Contents

• Installation
• Quick Start
• Commands
• Authentication
• Usage Examples
• GitHub Enterprise
• Development
• Testing
• Contributing
• License

🚀 Installation

From Source

git clone https://github.com/yourusername/github-extractor.git
cd github-extractor
npm install
npm run build
npm link

Using npm (when published)

npm install -g github-extractor

⚡ Quick Start

# 1. Authenticate
gh-extract login
# 2. Extract repositories interactively
gh-extract extract your-org-name
# 3. Or download all repositories
gh-extract extract your-org-name --all

📖 Commands

gh-extract login

Authenticate with GitHub using a Personal Access Token.

Options:

• -u, --url <url> - GitHub Enterprise API base URL (e.g., https://github.company.com/api/v3)

Example:

# GitHub.com
gh-extract login
# GitHub Enterprise
gh-extract login --url https://github.company.com/api/v3

gh-extract extract <organization>

Extract and clone repositories from an organization.

Arguments:

• <organization> - GitHub organization name

Options:

• -a, --all - Download all repositories without prompting
• -o, --output <dir> - Output directory (default: ./repositories)
• --ssh - Use SSH for cloning instead of HTTPS
• --shallow - Perform shallow clone (depth=1)
• --base-url <url> - GitHub Enterprise base URL

Examples:

# Interactive selection
gh-extract extract my-org
# Download all repositories
gh-extract extract my-org --all
# Custom output directory with SSH
gh-extract extract my-org --all --output ~/repos --ssh
# Shallow clone for faster download
gh-extract extract my-org --all --shallow
# GitHub Enterprise
gh-extract extract my-org --base-url https://github.company.com/api/v3

gh-extract list <organization>

List all repositories in an organization without downloading.

Options:

• --base-url <url> - GitHub Enterprise base URL
• --json - Output as JSON

Examples:

# Human-readable list
gh-extract list my-org
# JSON output for scripting
gh-extract list my-org --json > repos.json

gh-extract status

Show current authentication status.

gh-extract status

gh-extract logout

Clear saved authentication credentials.

gh-extract logout

🔐 Authentication

This tool uses Personal Access Tokens (PATs) for authentication, which work seamlessly with SAML-enabled organizations.

Creating a Personal Access Token

1. Navigate to token settings:

   • GitHub.com: https://github.com/settings/tokens/new
   • GitHub Enterprise: https://your-github-enterprise.com/settings/tokens/new

2. Configure the token:

   • Description:GitHub Extractor
   • Scopes: Select repo and read:org
   • Expiration: Choose based on your security policy

3. Generate the token

SAML SSO Authorization (Critical for SAML-enabled orgs)

Important: For SAML-enabled organizations, you must authorize your token for SSO:

1. After creating the token, go to https://github.com/settings/tokens
2. Find your token in the list
3. Click "Configure SSO" next to the token
4. Click "Authorize" next to your organization
5. Confirm the authorization

Note: You must authorize the token for SSO before using it with this tool. Existing tokens cannot be retrofitted with SSO authorization - you must create a new token if you forget this step.

Using the Token

Run gh-extract login and paste your token when prompted. The token is securely stored in your system's configuration directory.

💡 Usage Examples

Extract All Repositories

gh-extract extract my-org --all --output ~/backups/my-org

Interactive Selection

gh-extract extract my-org
# Follow prompts to select specific repositories

Clone with SSH for Private Repositories

gh-extract extract my-org --all --ssh

Fast Clone (Shallow)

gh-extract extract my-org --all --shallow

GitHub Enterprise with SAML

# Login to Enterprise instance
gh-extract login --url https://github.company.com/api/v3
# Extract repos
gh-extract extract my-enterprise-org --all

🏢 GitHub Enterprise

Setup

1. Determine your GitHub Enterprise API endpoint (typically https://your-domain.com/api/v3)
2. Use the --url or --base-url option with all commands
3. Ensure your PAT has the correct scopes and SSO authorization

Example Workflow

# Login
gh-extract login --url https://github.company.com/api/v3
# List organizations (verify access)
gh-extract list my-org --base-url https://github.company.com/api/v3
# Extract repositories
gh-extract extract my-org --all --base-url https://github.company.com/api/v3

🛠️ Development

Prerequisites

• Node.js >= 16.0.0
• npm or yarn

Setup

# Clone the repository
git clone https://github.com/yourusername/github-extractor.git
cd github-extractor
# Install dependencies
npm install
# Run in development mode
npm run dev -- extract my-org
# Build
npm run build
# Link for local testing
npm link

Scripts

• npm run build - Build TypeScript to JavaScript
• npm run dev - Run in development mode
• npm test - Run tests
• npm run test:watch - Run tests in watch mode
• npm run test:coverage - Run tests with coverage
• npm run lint - Lint code
• npm run lint:fix - Fix linting issues
• npm run format - Format code with Prettier
• npm run format:check - Check code formatting

🧪 Testing

# Run all tests
npm test# Run with coverage
npm run test:coverage
# Watch mode for development
npm run test:watch

🤝 Contributing

Contributions are welcome! Please see CONTRIBUTING.md for details.

Quick Contribution Guide

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes
4. Run tests (npm test)
5. Commit your changes (git commit -m 'Add amazing feature')
6. Push to the branch (git push origin feature/amazing-feature)
7. Open a Pull Request

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• AI Development: This project was developed with assistance from Claude (Anthropic)
• GitHub API: Built using @octokit/rest
• Community: Thanks to all contributors and users

🐛 Troubleshooting

Authentication Issues

Problem: "Invalid token" error

Solution:

gh-extract logout
gh-extract login
# Create a new token and ensure it has the correct scopes

SAML Authorization Required

Problem: "Resource protected by organization SAML enforcement"

Solution:

1. Visit https://github.com/settings/tokens
2. Click "Configure SSO" next to your token
3. Authorize your organization
4. Create a new token (old one won't work)
5. Run gh-extract logout and gh-extract login with the new token

Rate Limiting

Problem: GitHub API rate limits

Solution:

• Wait for the rate limit to reset (shown in error message)
• Use a token (authenticated requests have higher limits)
• For GitHub Enterprise, contact your admin

Organization Not Found

Problem: "Organization not found or you don't have access"

Solution:

• Verify you're a member of the organization
• Check that your token has read:org scope
• For SAML orgs, ensure SSO authorization is complete

📞 Support

• Issues:GitHub Issues
• Discussions:GitHub Discussions

🗺️ Roadmap

• Support for GitHub GraphQL API
• Parallel repository cloning
• Archive export format (zip/tar)
• Repository filtering by criteria
• Progress persistence and resume capability
• Docker support

---

Made with ❤️ and 🤖 AI assistance