Go-Getter v2 - Download Files From Everywhere in Go

Oct 6, 2025 8 min read

Complete guide for HashiCorp go-getter v2 library. One URL string, many protocols. No need complex download logic.

What is This go-getter

go-getter is library from HashiCorp for downloading files and directories using one URL string. Works with local files, Git repositories, HTTP endpoints, S3 buckets, and more protocols. HashiCorp uses it in Terraform for modules, Packer for binaries, Nomad for artifacts.

Main advantage: you write one line code, library makes protocol detection, authentication, checksums, archive extraction. No need switch between different download implementations.

Supported Protocols

Local files - Copy from filesystem
Git - Clone repositories (HTTP/SSH)
Mercurial - Hg repositories
HTTP/HTTPS - Direct downloads
Amazon S3 - AWS buckets
Google Cloud Storage - GCP buckets
SMB - Server Message Block shares

Installation

# Install library
go get github.com/hashicorp/go-getter/v2

# Or install CLI tool
go install github.com/hashicorp/go-getter/cmd/go-getter@latest

Basic Usage

Simple Download

package main

import (
    "context"
    "log"

    "github.com/hashicorp/go-getter/v2"
)

func main() {
    ctx := context.Background()

    // Download file to destination
    result, err := getter.Get(ctx, "./destination", "https://example.com/file.zip")
    if err != nil {
        log.Fatal(err)
    }

    log.Printf("Downloaded to: %s", result)
}

Download from GitHub

// Simple GitHub repo download
url := "github.com/hashicorp/terraform"
dest := "./terraform-source"

result, err := getter.Get(context.Background(), dest, url)
if err != nil {
    log.Fatal(err)
}

Library auto-detects GitHub URLs and converts to proper Git clone commands. No need manual Git wrapper.

Download Specific Git Ref

// Download specific branch/tag/commit
url := "github.com/hashicorp/terraform?ref=v1.5.0"
dest := "./terraform-v1.5.0"

result, err := getter.Get(context.Background(), dest, url)

URL Detectors

Detectors automatically transforms shorthand URLs into proper protocol URLs.

Automatic Transformations

// These URLs gets auto-detected and transformed:

// Local path
"./foo" → "file:///absolute/path/to/foo"

// GitHub  
"github.com/user/repo" → "git::https://github.com/user/repo.git"

// GitLab
"gitlab.com/user/project" → "git::https://gitlab.com/user/project.git"

// Bitbucket
"bitbucket.org/user/repo" → Git or Mercurial URL based on repo

Force Specific Protocol

If detector picks wrong protocol, you can force it:

// Force Git protocol for HTTP URL
url := "git::https://example.com/repo.git"

// Force HTTP instead auto-detection
url := "http::example.com/download"

// Force S3
url := "s3::https://s3.amazonaws.com/bucket/file.zip"

Protocol-Specific Options

Git Options

// Specific commit/branch/tag
"github.com/user/repo?ref=main"
"github.com/user/repo?ref=v1.2.3"
"github.com/user/repo?ref=abc123def"

// Shallow clone (more fast)
"github.com/user/repo?depth=1"

// SSH key for private repos
"[email protected]:user/private-repo.git?sshkey=/path/to/key"

HTTP Options

// Basic authentication
"http://user:[email protected]/file.zip"

// Custom headers (use X-Terraform-Get header)
// Set via environment or configuration

S3 Options

// Use AWS credentials from environment
"s3::https://s3.amazonaws.com/bucket/file.zip"

// Specific region
"s3::https://s3-us-west-2.amazonaws.com/bucket/file.zip"

// With AWS profile
// Set AWS_PROFILE environment variable

GCS Options

// Google Cloud Storage
"gcs::https://storage.googleapis.com/bucket/file.zip"

// Authentication via GOOGLE_APPLICATION_CREDENTIALS env var

Subdirectory Selection

Download only specific subdirectory from larger source:

// Download only 'modules/vpc' from repo
url := "github.com/terraform-aws-modules/terraform-aws-vpc//modules/vpc"

// Everything before '//' gets downloaded first
// Then only path after '//' is copied to destination

Works with all protocols:

"github.com/user/repo//subdir"
"https://example.com/archive.zip//nested/folder"
"s3::https://s3.amazonaws.com/bucket/archive.tar.gz//specific/path"

Checksum Verification

Verify downloaded files with checksums:

// MD5 checksum
url := "https://example.com/file.zip?checksum=md5:abc123..."

// SHA256 (recommended)
url := "https://example.com/file.zip?checksum=sha256:def456..."

// SHA1
url := "https://example.com/file.zip?checksum=sha1:789abc..."

// Checksum in separate file
url := "https://example.com/file.zip?checksum=file:https://example.com/file.zip.sha256"

Checksum verification happens automatically. Download fails if mismatch detected.

Archive Handling

Auto-extract supported archives:

// Auto-detected by extension
"https://example.com/archive.tar.gz"  // Extracts automatically
"https://example.com/archive.zip"     // Extracts automatically

// Force specific archive format
"https://example.com/file?archive=zip"
"https://example.com/file?archive=tar.gz"

// Disable auto-extraction
"https://example.com/archive.zip?archive=false"

Supported formats:

tar.gz, tgz
tar.bz2, tbz2
tar.xz, txz
zip
gz (gzip single file)
bz2 (bzip2 single file)
xz (xz single file)
zstd (zstandard)

Advanced Usage

With Context and Timeout

import (
    "context"
    "time"
)

func downloadWithTimeout(url, dest string) error {
    // 5 minute timeout
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
    defer cancel()

    _, err := getter.Get(ctx, dest, url)
    return err
}

Custom Client Configuration

import (
    "github.com/hashicorp/go-getter/v2"
)

func downloadWithConfig(url, dest string) error {
    ctx := context.Background()

    // Create client with options
    client := &getter.Client{
        Src:  url,
        Dst:  dest,
        Pwd:  "/working/directory",
        Mode: getter.ClientModeDir, // or ClientModeFile

        // Disable symlinks (security)
        DisableSymlinks: true,

        // Custom detectors
        Detectors: []getter.Detector{
            new(getter.GitHubDetector),
            new(getter.GitDetector),
            new(getter.FileDetector),
        },

        // Custom getters
        Getters: map[string]getter.Getter{
            "file":  new(getter.FileGetter),
            "git":   new(getter.GitGetter),
            "http":  new(getter.HttpGetter),
            "https": new(getter.HttpGetter),
        },
    }

    return client.Get()
}

Download to Temporary Directory

import (
    "os"
    "path/filepath"
)

func downloadToTemp(url string) (string, error) {
    // Create temp directory
    tmpDir, err := os.MkdirTemp("", "go-getter-")
    if err != nil {
        return "", err
    }

    // Download
    _, err = getter.Get(context.Background(), tmpDir, url)
    if err != nil {
        os.RemoveAll(tmpDir)
        return "", err
    }

    return tmpDir, nil
}

Real World Examples

Download Terraform Module

package main

import (
    "context"
    "log"

    "github.com/hashicorp/go-getter/v2"
)

func downloadTerraformModule(moduleURL, version string) error {
    // Construct URL with version
    url := moduleURL + "?ref=" + version
    dest := "./modules/" + version

    log.Printf("Downloading module: %s", url)

    _, err := getter.Get(context.Background(), dest, url)
    if err != nil {
        return err
    }

    log.Printf("Module downloaded to: %s", dest)
    return nil
}

func main() {
    err := downloadTerraformModule(
        "github.com/terraform-aws-modules/terraform-aws-vpc",
        "v5.0.0",
    )
    if err != nil {
        log.Fatal(err)
    }
}

Download and Verify Binary

func downloadBinary(url, checksum, dest string) error {
    ctx := context.Background()

    // Add checksum to URL
    urlWithChecksum := url + "?checksum=sha256:" + checksum

    // Download and verify
    _, err := getter.Get(ctx, dest, urlWithChecksum)
    if err != nil {
        return fmt.Errorf("download failed: %w", err)
    }

    // Make executable
    return os.Chmod(dest, 0755)
}

func main() {
    url := "https://releases.hashicorp.com/terraform/1.5.0/terraform_1.5.0_linux_amd64.zip"
    checksum := "abc123def456..." // Real SHA256 checksum

    err := downloadBinary(url, checksum, "./terraform.zip")
    if err != nil {
        log.Fatal(err)
    }
}

Clone Private Git Repository

import (
    "os"
)

func clonePrivateRepo(repoURL, sshKeyPath, dest string) error {
    // Construct URL with SSH key
    url := repoURL + "?sshkey=" + sshKeyPath

    ctx := context.Background()
    _, err := getter.Get(ctx, dest, url)

    return err
}

func main() {
    err := clonePrivateRepo(
        "[email protected]:company/private-repo.git",
        os.Getenv("HOME")+"/.ssh/id_rsa",
        "./private-repo",
    )
    if err != nil {
        log.Fatal(err)
    }
}

Download from S3 with Subdirectory

func downloadS3Subdir(bucket, path, subdir, dest string) error {
    // Construct S3 URL with subdirectory selector  
    url := fmt.Sprintf(
        "s3::https://s3.amazonaws.com/%s/%s//%s",
        bucket,
        path,
        subdir,
    )

    // AWS credentials from environment (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
    ctx := context.Background()
    _, err := getter.Get(ctx, dest, url)

    return err
}

func main() {
    err := downloadS3Subdir(
        "my-bucket",
        "artifacts/v1.0.0/bundle.tar.gz",
        "configs",
        "./configs",
    )
    if err != nil {
        log.Fatal(err)
    }
}

Build Download Manager

type DownloadManager struct {
    downloads map[string]string // url -> destination
}

func NewDownloadManager() *DownloadManager {
    return &DownloadManager{
        downloads: make(map[string]string),
    }
}

func (dm *DownloadManager) Add(url, dest string) {
    dm.downloads[url] = dest
}

func (dm *DownloadManager) DownloadAll(ctx context.Context) error {
    for url, dest := range dm.downloads {
        log.Printf("Downloading: %s -> %s", url, dest)

        _, err := getter.Get(ctx, dest, url)
        if err != nil {
            return fmt.Errorf("failed to download %s: %w", url, err)
        }
    }

    return nil
}

func main() {
    dm := NewDownloadManager()

    dm.Add("github.com/user/repo1", "./repos/repo1")
    dm.Add("github.com/user/repo2?ref=v1.0.0", "./repos/repo2")
    dm.Add("https://example.com/file.zip", "./downloads/file.zip")

    ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
    defer cancel()

    if err := dm.DownloadAll(ctx); err != nil {
        log.Fatal(err)
    }
}

Security Considerations

Disable Symlinks

Prevent symlink attacks:

client := &getter.Client{
    Src:             url,
    Dst:             dest,
    DisableSymlinks: true, // Important for security
}

Set Timeouts

Always use context with timeout:

ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
defer cancel()

_, err := getter.Get(ctx, dest, url)

Validate User Input

Never trust user-supplied URLs directly:

func isAllowedURL(url string) bool {
    // Whitelist allowed domains
    allowedDomains := []string{
        "github.com",
        "gitlab.com", 
        "internal-repo.company.com",
    }

    for _, domain := range allowedDomains {
        if strings.Contains(url, domain) {
            return true
        }
    }

    return false
}

Use Checksums

Always verify downloads when possible:

url := baseURL + "?checksum=sha256:" + expectedChecksum
_, err := getter.Get(ctx, dest, url)

Common Patterns

Retry Logic

func downloadWithRetry(url, dest string, maxRetries int) error {
    var err error

    for i := 0; i < maxRetries; i++ {
        ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
        _, err = getter.Get(ctx, dest, url)
        cancel()

        if err == nil {
            return nil
        }

        log.Printf("Attempt %d failed: %v", i+1, err)
        time.Sleep(time.Duration(i+1) * time.Second)
    }

    return fmt.Errorf("failed after %d retries: %w", maxRetries, err)
}

Clean Up on Error

func downloadClean(url, dest string) error {
    _, err := getter.Get(context.Background(), dest, url)
    if err != nil {
        // Clean up partial download
        os.RemoveAll(dest)
        return err
    }

    return nil
}

Progress Tracking

// For progress tracking, you need custom getter implementation
// or monitor destination directory size during download

func monitorDownload(dest string) {
    ticker := time.NewTicker(1 * time.Second)
    defer ticker.Stop()

    for range ticker.C {
        // Check directory size
        size, _ := dirSize(dest)
        log.Printf("Downloaded: %d bytes", size)
    }
}

func dirSize(path string) (int64, error) {
    var size int64
    err := filepath.Walk(path, func(_ string, info os.FileInfo, err error) error {
        if err == nil && !info.IsDir() {
            size += info.Size()
        }
        return err
    })
    return size, err
}

Command Line Tool

go-getter also comes with CLI tool:

# Install CLI
go install github.com/hashicorp/go-getter/cmd/go-getter@latest

# Download GitHub repo
go-getter github.com/hashicorp/terraform ./terraform

# Download with checksum
go-getter "https://example.com/file.zip?checksum=sha256:abc123" ./file.zip

# Download subdirectory
go-getter "github.com/user/repo//subdir" ./subdir

# Force protocol
go-getter "git::https://example.com/repo.git" ./repo

Migration from v1 to v2

Main differences in v2:

Context support (required)
Better error handling
Better security defaults
Updated dependencies

// v1 style (old)
err := getter.Get(dst, src)

// v2 style (new)  
_, err := getter.Get(context.Background(), dst, src)

Note: v2 fixed several security vulnerabilities present in v1. Always use v2 for new projects.

Best Practices

Always use context with timeout - prevent hanging downloads
Enable checksum verification - ensure file integrity
Disable symlinks - prevent security issues
Validate URLs - whitelist allowed sources
Handle errors properly - clean up partial downloads
Use specific versions - for Git repos, specify ref/tag
Monitor download sizes - prevent disk space issues
Set reasonable timeouts - based on expected file size
Use retry logic - for unreliable network conditions
Keep library updated - security patches matter

Common Issues

Git Not Found

Error: exec: "git": executable file not found in $PATH

Solution: Install Git on system

# Ubuntu/Debian
sudo apt-get install git

# macOS
brew install git

# Or use HTTP getter instead Git
url := "https::github.com/user/repo/archive/refs/heads/main.zip"

Permission Denied

Error: permission denied

Solution: Check destination directory permissions

// Create destination with proper permissions
os.MkdirAll(dest, 0755)

Checksum Mismatch

Error: checksum mismatch

Solution: Verify checksum is correct or file wasn’t corrupted during download

Timeout

Error: context deadline exceeded

Solution: Increase timeout or check network connection

ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
defer cancel()

Conclusion

go-getter v2 simplifies download operations in Go applications. One interface for multiple protocols, automatic detection, built-in verification. Used in production by HashiCorp tools for years.

For simple file downloads, saves you from writing protocol-specific code. For complex scenarios, provides enough flexibility to handle edge cases. Good tool to have in your Go toolkit.

Remember: validate inputs, set timeouts, verify checksums. Basic engineering principles applies always.