Git-First Versioned Storage

Git-First Versioned Storage: Signed Commits, git-crypt/LFS

Modern configuration management systems require robust version control, cryptographic authenticity, and integrity verification to maintain audit trails and ensure security compliance. Git-First Versioned Storage provides a repository-based backend that treats all configuration changes as version-controlled commits, with optional GPG signing for authenticity and Git LFS for efficient large file handling.

This addresses the challenge of traditional database or filesystem storage approaches that lack built-in version control, cryptographic verification, and comprehensive change auditing. Organizations can leverage Git-based storage to establish immutable audit trails, implement non-repudiation for configuration changes, and integrate configuration management into standard DevOps workflows.

Key Features

The Git-First approach provides:

Git-style repository backend - Full version control with branching, merging, and history
Optional signed commits - GPG/PGP signatures for authentication and non-repudiation
Git LFS support - Efficient handling of large configuration files and binary assets
Cryptographic integrity - Tamper-evident storage with hash verification
Transparent encryption - git-crypt for protecting sensitive configuration data

Understanding Core Components

Repository Backend

The Git-based storage layer implements a standard Git repository structure optimized for configuration management:

config-repo/
├── .git/
├── configs/
│   ├── application.yml
│   ├── database.yml
│   └── services/
├── .gitattributes (LFS configuration)
└── .git-crypt/ (optional encryption)

Signed Commits

All configuration changes can be tracked through GPG-signed commits, providing critical security assurances:

Authenticity - Cryptographically verify who made each change
Non-repudiation - Commit creators cannot deny their changes
Integrity - Detect any tampering with commit history
Compliance - Meet audit requirements for change tracking

Signed commits create an immutable chain of custody for all configuration modifications. Each commit signature can be verified against the signer’s public key, ensuring changes originated from authorized personnel.

Configuration Example:

git config commit.gpgsign true
git config user.signingkey <GPG_KEY_ID>

Git LFS (Large File Storage)

For large configuration files, binary assets, or certificates, Git LFS provides efficient storage:

Pointer-based storage - Git stores small pointer files while actual content lives separately
Reduced repository size - Large files don’t bloat the repository history
Improved performance - Faster clone and fetch operations
Bandwidth optimization - Download large files only when needed

Git LFS is essential when managing configuration files that exceed several megabytes, such as firmware images, certificate stores, or comprehensive device backups.

Configuration example (.gitattributes):

*.bin filter=lfs diff=lfs merge=lfs -text
*.cert filter=lfs diff=lfs merge=lfs -text
*.p12 filter=lfs diff=lfs merge=lfs -text

git-crypt (Optional Encryption)

Transparent encryption for sensitive configuration data at rest:

Automatic encryption - Files encrypt before committing to repository
Seamless decryption - Authorized users decrypt automatically on checkout
Selective encryption - Encrypt only sensitive files, leave others readable
Key-based access - Control decryption through GPG key management

git-crypt integrates seamlessly with standard Git operations. Users with proper GPG keys automatically decrypt files during checkout, while unauthorized users see only encrypted content.

Benefits and Use Cases

Security & Compliance

Cryptographic signatures ensure change accountability: Every configuration modification carries a verifiable signature identifying the author. This meets regulatory requirements for change management and creates legally defensible audit trails.

Encrypted storage protects sensitive configurations: Credentials, API keys, and proprietary network topologies remain encrypted in the repository, protecting against unauthorized access even if repository storage is compromised.

Complete audit trail with verifiable history: Git’s immutable commit history combined with signature verification provides tamper-evident logging of all configuration changes, essential for security audits and compliance reporting.

Granular access control through GPG key management: Organizations control who can commit signed changes and decrypt sensitive data by managing GPG key distribution, enabling precise access policies.

Version Control

Full history of all configuration changes: Every modification is preserved with complete context including author, timestamp, and rationale. This enables comprehensive change analysis and root cause investigation.

Branching and merging for environment-specific configs: Maintain separate branches for production, staging, and development configurations while selectively merging changes between environments as needed.

Instant rollback capability to any previous state: Configuration errors can be immediately reverted by checking out previous commits, eliminating the need for manual restoration or backup recovery.

Advanced diff tools for configuration comparison: Leverage Git’s powerful diff capabilities to compare configurations across time periods, branches, or environments, identifying exactly what changed and when.

Operational Excellence

Collaboration through standard Git workflows: Teams already familiar with Git for code management can apply the same pull request, code review, and approval processes to configuration changes.

Code review processes for configuration changes: Implement peer review requirements before configuration modifications reach production, reducing errors and knowledge-sharing across teams.

CI/CD integration for automated validation: Integrate configuration repositories with continuous integration pipelines to automatically validate syntax, check policy compliance, and run tests before accepting changes.

Distributed disaster recovery: Every clone of the Git repository serves as a complete backup. Multiple team members and systems maintain full copies of configuration history, providing inherent disaster recovery capabilities.

Implementation Guide

Implementing Git-First Versioned Storage requires careful planning and phased deployment to ensure smooth transition and maintain operational continuity.

Repository Initialization

Establish the Git repository structure and configure essential features before importing any configuration data.

Step 1: Create repository structure

# Create new repository
git init config-repo
cd config-repo

# Create directory structure
mkdir -p configs/services
mkdir -p templates
mkdir -p policies

# Initialize README
cat > README.md << EOF
# Configuration Repository

Git-based storage for system configurations with signed commits and LFS support.

## Structure
- configs/: Active configuration files
- templates/: Configuration templates
- policies/: Compliance and validation policies
EOF

git add README.md
git commit -m "Initial repository structure"

Step 2: Configure Git LFS

# Install Git LFS (if not already installed)
# Ubuntu/Debian:
sudo apt-get install git-lfs

# CentOS/RHEL/Rocky/AlmaLinux:
sudo yum install git-lfs

# Initialize LFS in repository
git lfs install

# Configure LFS tracking
cat > .gitattributes << EOF
*.bin filter=lfs diff=lfs merge=lfs -text
*.cert filter=lfs diff=lfs merge=lfs -text
*.p12 filter=lfs diff=lfs merge=lfs -text
*.firmware filter=lfs diff=lfs merge=lfs -text
*.img filter=lfs diff=lfs merge=lfs -text
EOF

git add .gitattributes
git commit -m "Configure Git LFS for binary files"

Step 3: Configure signed commits

# Generate GPG key if needed
gpg --full-generate-key

# List GPG keys to find key ID
gpg --list-secret-keys --keyid-format=long

# Configure Git to use GPG key
git config user.signingkey <YOUR_GPG_KEY_ID>
git config commit.gpgsign true

# Test signed commit
git commit --allow-empty -m "Test signed commit"
git verify-commit HEAD

Step 4: Set up git-crypt (optional)

# Install git-crypt
# Ubuntu/Debian:
sudo apt-get install git-crypt

# CentOS/RHEL/Rocky/AlmaLinux:
sudo yum install git-crypt

# Initialize git-crypt in repository
git-crypt init

# Export encryption key for backup
git-crypt export-key ../config-repo-key

# Configure which files to encrypt
cat > .gitattributes << EOF
# Existing LFS configuration...
*.key filter=git-crypt diff=git-crypt
*.secret filter=git-crypt diff=git-crypt
*_credentials.* filter=git-crypt diff=git-crypt
secrets/** filter=git-crypt diff=git-crypt
EOF

git add .gitattributes
git commit -m "Configure git-crypt for sensitive files"

System Integration

Connect the Git repository backend to your configuration management system and implement necessary interfaces.

Step 1: Implement repository backend interface

Create adapter layer that provides standard CRUD operations backed by Git commits:

# Example Python interface (adapt for your language/framework)
class GitConfigBackend:
    def __init__(self, repo_path, signing_key=None):
        self.repo = git.Repo(repo_path)
        self.signing_key = signing_key

    def save_config(self, filename, content, message, author):
        """Save configuration and create signed commit"""
        file_path = os.path.join(self.repo.working_dir, filename)

        # Write content
        with open(file_path, 'w') as f:
            f.write(content)

        # Stage and commit with signature
        self.repo.index.add([filename])
        self.repo.index.commit(
            message,
            author=author,
            gpg_sign=self.signing_key
        )

    def get_config(self, filename, version=None):
        """Retrieve configuration at specific version"""
        if version:
            commit = self.repo.commit(version)
            return commit.tree[filename].data_stream.read()
        else:
            file_path = os.path.join(self.repo.working_dir, filename)
            with open(file_path, 'r') as f:
                return f.read()

    def get_history(self, filename, limit=10):
        """Get commit history for specific file"""
        commits = list(self.repo.iter_commits(paths=filename, max_count=limit))
        return [{
            'sha': c.hexsha,
            'author': str(c.author),
            'date': c.committed_datetime,
            'message': c.message,
            'signed': c.gpgsig is not None
        } for c in commits]

Step 2: Add commit signing verification

Implement signature verification for all read operations:

def verify_commit_signature(self, commit_sha):
    """Verify GPG signature on commit"""
    commit = self.repo.commit(commit_sha)

    if not commit.gpgsig:
        raise SecurityError(f"Commit {commit_sha} is not signed")

    # Verify signature using GPG
    try:
        gpg = gnupg.GPG()
        verified = gpg.verify(commit.gpgsig)

        if not verified:
            raise SecurityError(f"Invalid signature on commit {commit_sha}")

        return {
            'valid': True,
            'key_id': verified.key_id,
            'username': verified.username
        }
    except Exception as e:
        raise SecurityError(f"Signature verification failed: {e}")

Step 3: Configure LFS object storage

Set up dedicated storage for LFS objects:

# Configure LFS storage location
git config lfs.storage /var/lfs-storage

# Set up LFS server (if using separate LFS server)
# Configure LFS URL in .lfsconfig
cat > .lfsconfig << EOF
[lfs]
    url = https://lfs.example.com/storage
EOF

git add .lfsconfig
git commit -m "Configure LFS server URL"

Step 4: Integrate git-crypt unlock

Automate git-crypt unlocking for authorized users/systems:

def unlock_repository(self, key_path):
    """Unlock git-crypt encrypted files"""
    import subprocess

    result = subprocess.run(
        ['git-crypt', 'unlock', key_path],
        cwd=self.repo.working_dir,
        capture_output=True
    )

    if result.returncode != 0:
        raise SecurityError(f"Failed to unlock repository: {result.stderr}")

Testing and Validation

Thoroughly test all components before production deployment to ensure reliability and security.

Step 1: Verify signature validation

# Create test signed commit
echo "test config" > test.conf
git add test.conf
git commit -S -m "Test signed configuration"

# Verify signature
git verify-commit HEAD

# Expected output:
# gpg: Signature made [date]
# gpg: Good signature from "[Name] <email>"

Step 2: Test LFS retrieval

# Add large test file
dd if=/dev/urandom of=large-test.bin bs=1M count=10
git add large-test.bin
git commit -m "Test LFS file"

# Verify LFS tracking
git lfs ls-files

# Clone repository to test LFS download
cd /tmp
git clone /path/to/config-repo test-clone
cd test-clone

# Verify large file downloaded correctly
ls -lh large-test.bin
md5sum large-test.bin

Step 3: Confirm encryption/decryption

# Add sensitive test file
echo "secret_api_key=12345" > test.secret
git add test.secret
git commit -m "Test encrypted file"

# Lock repository
git-crypt lock

# Verify file is encrypted
cat test.secret
# Should see binary encrypted content

# Unlock repository
git-crypt unlock /path/to/key

# Verify file is decrypted
cat test.secret
# Should see plaintext: secret_api_key=12345

Step 4: Performance benchmarking

import time

def benchmark_operations():
    """Benchmark critical operations"""
    backend = GitConfigBackend('/path/to/repo')

    # Test write performance
    start = time.time()
    for i in range(100):
        backend.save_config(
            f'test_{i}.conf',
            f'config content {i}',
            f'Benchmark test {i}',
            'Test User <test@example.com>'
        )
    write_time = time.time() - start

    # Test read performance
    start = time.time()
    for i in range(100):
        backend.get_config(f'test_{i}.conf')
    read_time = time.time() - start

    # Test history retrieval
    start = time.time()
    backend.get_history('test_0.conf', limit=50)
    history_time = time.time() - start

    print(f"Write 100 configs: {write_time:.2f}s")
    print(f"Read 100 configs: {read_time:.2f}s")
    print(f"History retrieval: {history_time:.2f}s")

Production Deployment

Deploy the Git-based storage system with appropriate safeguards and monitoring.

Step 1: Gradual rollout to non-production

# Deploy to development environment first
# Update configuration to point to Git backend
CONFIG_BACKEND=git
CONFIG_REPO_PATH=/var/config-repo
CONFIG_SIGNING_REQUIRED=true

# Run integration tests
./scripts/run-integration-tests.sh

# Monitor for errors
tail -f /var/log/application.log | grep -i error

Step 2: Import existing configurations

def migrate_existing_configs(old_backend, git_backend):
    """Migrate configurations from old storage to Git"""
    configs = old_backend.list_all_configs()

    for config in configs:
        content = old_backend.get_config(config['name'])
        metadata = old_backend.get_metadata(config['name'])

        # Create commit with original timestamp and author
        git_backend.save_config(
            filename=config['name'],
            content=content,
            message=f"Migrate: {config['name']}",
            author=metadata.get('author', 'Migration Script')
        )

        print(f"Migrated: {config['name']}")

Step 3: Enable monitoring

# Add metrics collection
class MonitoredGitBackend(GitConfigBackend):
    def save_config(self, *args, **kwargs):
        start = time.time()
        try:
            result = super().save_config(*args, **kwargs)
            metrics.increment('git.commit.success')
            metrics.timing('git.commit.duration', time.time() - start)
            return result
        except Exception as e:
            metrics.increment('git.commit.failure')
            raise

    def verify_commit_signature(self, *args, **kwargs):
        try:
            result = super().verify_commit_signature(*args, **kwargs)
            metrics.increment('git.signature.valid')
            return result
        except SecurityError:
            metrics.increment('git.signature.invalid')
            raise

Step 4: Production cutover

# Schedule maintenance window
# Switch production to Git backend
# Run validation checks

# Verify all systems operational
./scripts/health-check.sh

# Monitor error rates
watch -n 5 'grep ERROR /var/log/application.log | tail -20'

# If issues detected, rollback procedure:
# CONFIG_BACKEND=legacy
# systemctl restart application

Security Considerations

Key Management

Store GPG keys securely: GPG private keys must never be stored in plain text or committed to repositories. Use hardware security modules (HSMs), key management services (KMS), or encrypted vaults for production keys.

Implement key rotation policies: Establish regular key rotation schedules (annually or bi-annually) to limit exposure window if keys are compromised. Document rotation procedures and test them regularly.

Maintain revocation procedures: Prepare key revocation processes for compromised or lost keys. Publish revocation certificates to key servers and update all systems to reject signatures from revoked keys.

Document key recovery process: Create secure procedures for key recovery when personnel leave or keys are lost. Store key backups in multiple secure locations with appropriate access controls.

Access Control

Limit repository write access: Grant commit permissions only to authorized personnel. Implement role-based access control where junior staff can view configurations but only senior engineers can modify them.

Enforce signed commits via hooks: Use Git hooks to reject unsigned commits automatically. This prevents accidental or malicious unsigned modifications from entering the repository. Implement branch protection rules: Configure branch protection to require pull requests, code reviews, and passing tests before merging changes to main branches. This creates approval gates for critical configurations.

Audit access logs regularly: Review repository access logs monthly to identify unusual patterns, unauthorized access attempts, or suspicious activities. Integrate with security information and event management (SIEM) systems.

Integrity Verification

Validate signatures on retrieval: Always verify commit signatures when reading configurations. Treat unsigned or invalidly-signed commits as security incidents requiring investigation.

Verify LFS object checksums: Confirm LFS objects match expected checksums after download. This detects corruption or tampering during transmission or storage.

Monitor for unauthorized changes: Implement automated monitoring that alerts on unexpected commits, signature verification failures, or modifications outside approved change windows.

Alert on signature failures: Configure immediate notifications when signature verification fails. These events may indicate compromise, key issues, or attacks and require urgent investigation.

Technical Requirements

Prerequisites

Git 2.x or higher: Modern Git features including LFS, improved signing, and performance enhancements require Git 2.0+. Git 2.30+ recommended for best compatibility.

GPG/GnuPG for signing: GPG 2.2+ required for commit signing. Ensure GPG is properly configured with key generation, signing, and verification capabilities.

Git LFS extension: Install Git LFS client on all systems that will interact with the repository. Version 2.0+ recommended for improved performance and reliability.

git-crypt (optional): If using transparent encryption, install git-crypt 0.6+ on all authorized systems. Verify compatibility with your Git version before deployment.

System Integration

Repository storage backend: Dedicated filesystem or object storage for Git repositories. Allocate sufficient space for repository growth including history and LFS objects.

Signature verification service: Automated service that validates commit signatures on read operations. Should integrate with your GPG keyring management infrastructure.

LFS object store: Separate storage tier for large files tracked by LFS. Can use local filesystem, network storage, or cloud object storage (S3, Azure Blob, etc.).

Encryption key management: Secure storage and distribution system for git-crypt keys and GPG private keys. Should support key rotation, backup, and access auditing.

Monitoring & Maintenance

Metrics to Track

Comprehensive monitoring ensures the Git-based storage system operates reliably and securely:

Commit signature validation rate: Track percentage of commits successfully verified. Rates below 100% indicate signing issues, key problems, or potential security concerns requiring investigation.

LFS object retrieval performance: Monitor download times for LFS objects. Degrading performance may indicate network issues, storage problems, or capacity constraints requiring infrastructure scaling.

Repository size growth: Track repository size over time to project storage needs and identify opportunities for optimization. Unexpectedly rapid growth may indicate LFS misconfiguration or large files committed incorrectly.

Failed signature attempts: Alert on any signature verification failures. These may indicate compromised keys, system clock skew, or malicious activity requiring immediate security review.

git-crypt unlock failures: Monitor git-crypt unlock operations for failures. Problems may indicate key distribution issues, permission problems, or attempted unauthorized access.

Regular Tasks

Key rotation and renewal: Rotate GPG keys according to organizational policy (typically annually). Update all systems with new keys, revoke old keys, and verify signature chain integrity.

LFS object cleanup: Periodically run git lfs prune to remove unreferenced LFS objects. This recovers storage space from deleted or replaced files while preserving objects referenced in Git history.

Repository maintenance: Execute git gc and git prune regularly to optimize repository performance. Large repositories benefit from monthly maintenance; smaller ones can run quarterly.

Backup verification: Test repository backups monthly by restoring to separate systems and verifying contents. Confirm LFS objects, encrypted files, and signature verification all work correctly from backups.

Git Documentation - Signing Commits: Comprehensive guide to GPG commit signing configuration and verification
Git LFS Documentation: Official documentation for Large File Storage setup and operation
git-crypt Repository: Project documentation for transparent Git encryption
GPG Best Practices: Security guidelines for key generation, storage, and management

Quick Reference

Common Git Commands

# Verify commit signature
git verify-commit <commit-hash>

# View commit signature details
git log --show-signature

# List LFS tracked files
git lfs ls-files

# Check LFS status
git lfs status

# Manually unlock git-crypt
git-crypt unlock /path/to/key

# Lock git-crypt (encrypt files)
git-crypt lock

# Repository maintenance
git gc --aggressive
git prune
git lfs prune

Git Hooks for Enforcement

Pre-commit hook to require signing:

#!/bin/bash
# Check if commit is signed
if ! git var GIT_COMMITTER_IDENT > /dev/null 2>&1; then
    echo "ERROR: No GPG signing key configured"
    echo "Run: git config user.signingkey <KEY_ID>"
    exit 1
fi

if ! git config commit.gpgsign | grep -q true; then
    echo "ERROR: Commit signing not enabled"
    echo "Run: git config commit.gpgsign true"
    exit 1
fi

Server-side hook to reject unsigned commits:

#!/bin/bash
# hooks/update (server-side)

zero_commit="0000000000000000000000000000000000000000"

while read oldrev newrev refname; do
    if [ "$newrev" = "$zero_commit" ]; then
        continue
    fi

    # Check all new commits
    for commit in $(git rev-list $oldrev..$newrev); do
        if ! git verify-commit $commit 2>/dev/null; then
            echo "ERROR: Commit $commit is not signed or has invalid signature"
            exit 1
        fi
    done
done

exit 0

Summary

Git-First Versioned Storage transforms configuration management by applying proven version control practices to system configurations. By leveraging Git’s distributed architecture, cryptographic signing for authenticity, LFS for efficient large file handling, and optional encryption for sensitive data, organizations establish robust audit trails while maintaining operational flexibility.

Key benefits of this approach include:

Cryptographic authenticity - Every change is cryptographically signed, creating non-repudiable audit trails that meet compliance requirements
Comprehensive history - Full version history with branching, merging, and instant rollback capabilities
Distributed resilience - Every clone serves as a complete backup, providing inherent disaster recovery
Developer-friendly workflows - Leverage familiar Git operations, pull requests, and code review processes
Efficient large file handling - LFS prevents repository bloat while maintaining fast clone and fetch operations
Transparent encryption - Protect sensitive configurations at rest without impacting standard Git operations

Implementation requires careful attention to key management, access control, and integration with existing systems. Organizations should phase deployment through development environments first, validate all security features thoroughly, and establish monitoring before production rollout.

For systems managing critical network infrastructure, financial data, or sensitive configurations, Git-First Versioned Storage provides the security, accountability, and operational excellence required for modern configuration management at scale.

Git-First Versioned Storage

Git-First Versioned Storage: Signed Commits, git-crypt/LFS

Key Features

Understanding Core Components

Repository Backend

Signed Commits

Git LFS (Large File Storage)

git-crypt (Optional Encryption)

Benefits and Use Cases

Security & Compliance

Version Control

Operational Excellence

Implementation Guide

Repository Initialization

System Integration

Testing and Validation

Production Deployment

Security Considerations

Key Management

Access Control

Integrity Verification

Technical Requirements

Prerequisites

System Integration

Monitoring & Maintenance

Metrics to Track

Regular Tasks

Related Documentation

Quick Reference

Common Git Commands

Git Hooks for Enforcement

Summary