simples3 : Simple no frills AWS S3 Library using REST with V4 Signing

Overview

SimpleS3 is a Go library for manipulating objects in S3 buckets using REST API calls or Presigned URLs signed using AWS Signature Version 4.

Features:

• Simple, intuitive API following Go idioms
• Complete S3 operations - Upload, Download, Delete, List, Details
• Multipart upload - Large file support with parallel uploads, progress tracking, and resumability
• Bucket management - Create, Delete, and List buckets
• AWS Signature Version 4 signing
• Custom endpoint support (MinIO, DigitalOcean Spaces, etc.)
• Simple List API with pagination, prefix filtering, and delimiter grouping
• Iterator-based ListAll for memory-efficient large bucket iteration (Go 1.23+)
• Presigned URL generation for secure browser uploads/downloads (including multipart)
• Object Versioning - Enable versioning, list versions, and access specific object versions
• Server-Side Encryption - Support for SSE-S3 (AES256) and SSE-KMS encryption
• IAM credential support for EC2 instances
• Comprehensive test coverage
• Zero dependencies - uses only Go standard library

Install

go get github.com/rhnvrm/simples3

Quick Start

package main

import (
    "log"
    "os"
    "github.com/rhnvrm/simples3"
)

func main() {
    // Initialize S3 client
    s3 := simples3.New("us-east-1", "your-access-key", "your-secret-key")

    // Upload a file
    file, err := os.Open("my-file.txt")
    if err != nil {
        log.Fatal(err)
    }
    defer file.Close()

    resp, err := s3.FileUpload(simples3.UploadInput{
        Bucket:      "my-bucket",
        ObjectKey:   "my-file.txt",
        ContentType: "text/plain",
        FileName:    "my-file.txt",
        Body:        file,
    })
    if err != nil {
        log.Fatal(err)
    }

    log.Printf("File uploaded successfully: %+v", resp)
}

API Reference

Bucket Operations

List All Buckets

// List all buckets for the AWS account
result, err := s3.ListBuckets(simples3.ListBucketsInput{})
if err != nil {
    log.Fatal(err)
}

fmt.Printf("Found %d buckets:\n", len(result.Buckets))
for _, bucket := range result.Buckets {
    fmt.Printf("- %s (created: %s)\n", bucket.Name, bucket.CreationDate)
}

// Owner information is also available
fmt.Printf("Owner: %s (%s)\n", result.Owner.DisplayName, result.Owner.ID)

Create a Bucket

// Create a new bucket
output, err := s3.CreateBucket(simples3.CreateBucketInput{
    Bucket: "my-new-bucket",
    Region: "us-east-1", // Optional, defaults to client region
})
if err != nil {
    log.Fatal(err)
}

fmt.Printf("Bucket created at: %s\n", output.Location)

Delete a Bucket

// Delete an empty bucket
err := s3.DeleteBucket(simples3.DeleteBucketInput{
    Bucket: "my-old-bucket",
})
if err != nil {
    log.Fatal(err)
}

// Note: The bucket must be empty before deletion

File Operations

Upload Files

// POST upload (recommended for browsers)
resp, err := s3.FileUpload(simples3.UploadInput{
    Bucket:      "my-bucket",
    ObjectKey:   "path/to/file.txt",
    ContentType: "text/plain",
    FileName:    "file.txt",
    Body:        file,
})

// PUT upload (simpler for programmatic use)
resp, err := s3.FilePut(simples3.UploadInput{
    Bucket:      "my-bucket",
    ObjectKey:   "path/to/file.txt",
    ContentType: "text/plain",
    Body:        file,
})

Download Files

// Download file
file, err := s3.FileDownload(simples3.DownloadInput{
    Bucket:    "my-bucket",
    ObjectKey: "path/to/file.txt",
})
if err != nil {
    log.Fatal(err)
}
defer file.Close()

// Read the content
data, err := io.ReadAll(file)
if err != nil {
    log.Fatal(err)
}

Delete Files

err := s3.FileDelete(simples3.DeleteInput{
    Bucket:    "my-bucket",
    ObjectKey: "path/to/file.txt",
})

Copy Objects

// Copy object within same bucket
output, err := s3.CopyObject(simples3.CopyObjectInput{
    SourceBucket: "my-bucket",
    SourceKey:    "original/file.txt",
    DestBucket:   "my-bucket",
    DestKey:      "copied/file.txt",
})

// Copy across buckets
output, err := s3.CopyObject(simples3.CopyObjectInput{
    SourceBucket: "source-bucket",
    SourceKey:    "file.txt",
    DestBucket:   "dest-bucket",
    DestKey:      "file.txt",
})

// Copy with metadata replacement
output, err := s3.CopyObject(simples3.CopyObjectInput{
    SourceBucket:      "my-bucket",
    SourceKey:         "file.txt",
    DestBucket:        "my-bucket",
    DestKey:           "file-copy.txt",
    MetadataDirective: "REPLACE",
    ContentType:       "application/json",
    CustomMetadata:    map[string]string{"version": "2"},
})

Batch Delete Files

// Delete multiple objects in one request (up to 1000)
output, err := s3.DeleteObjects(simples3.DeleteObjectsInput{
    Bucket:  "my-bucket",
    Objects: []string{"file1.txt", "file2.txt", "file3.txt"},
    Quiet:   false,
})
if err != nil {
    log.Fatal(err)
}

// Check results
for _, deleted := range output.Deleted {
    fmt.Printf("Deleted: %s\n", deleted.Key)
}
for _, errItem := range output.Errors {
    fmt.Printf("Failed to delete %s: %s\n", errItem.Key, errItem.Message)
}

Get File Details

details, err := s3.FileDetails(simples3.DetailsInput{
    Bucket:    "my-bucket",
    ObjectKey: "path/to/file.txt",
})
if err != nil {
    log.Fatal(err)
}

log.Printf("File size: %s bytes", details.ContentLength)
log.Printf("Last modified: %s", details.LastModified)
log.Printf("Content type: %s", details.ContentType)

List Objects

SimpleS3 provides a clean, easy-to-use List API that follows the same pattern as other library methods:

Simple Listing

// List all objects in a bucket
result, err := s3.List(simples3.ListInput{
    Bucket: "my-bucket",
})
if err != nil {
    log.Fatal(err)
}

fmt.Printf("Found %d objects:\n", len(result.Objects))
for _, obj := range result.Objects {
    fmt.Printf("- %s (%d bytes)\n", obj.Key, obj.Size)
}

Advanced Listing with Options

// List with prefix filtering and pagination
result, err := s3.List(simples3.ListInput{
    Bucket:    "my-bucket",
    Prefix:    "documents/",
    Delimiter: "/",
    MaxKeys:   100,
})
if err != nil {
    log.Fatal(err)
}

// Process objects
for _, obj := range result.Objects {
    fmt.Printf("%s (%d bytes)\n", obj.Key, obj.Size)
}

// Process "directories" (common prefixes)
for _, prefix := range result.CommonPrefixes {
    fmt.Printf("%s/\n", prefix)
}

Iterator-based Listing (Go 1.23+)

// Use the new ListAll method for memory-efficient iteration
s3 := simples3.New("us-east-1", "your-access-key", "your-secret-key")

// Iterate over all objects with automatic pagination and error handling
seq, finish := s3.ListAll(simples3.ListInput{
    Bucket: "my-bucket",
    Prefix: "documents/", // Optional filtering
})

for obj := range seq {
    fmt.Printf("%s (%d bytes)\n", obj.Key, obj.Size)
}

// Check for any errors that occurred during iteration
if err := finish(); err != nil {
    log.Fatalf("Error during iteration: %v", err)
}

Handle Pagination (Legacy)

// Handle large result sets with pagination
func listAllObjects(s3 *simples3.S3, bucket string) ([]simples3.Object, error) {
    var allObjects []simples3.Object
    var continuationToken string

    for {
        // List objects with continuation token
        result, err := s3.List(simples3.ListInput{
            Bucket:            bucket,
            ContinuationToken: continuationToken,
            MaxKeys:          1000, // Maximum page size
        })
        if err != nil {
            return nil, err
        }

        // Add current page objects to our collection
        allObjects = append(allObjects, result.Objects...)

        // Check if there are more pages
        if !result.IsTruncated {
            break
        }

        // Set token for next page
        continuationToken = result.NextContinuationToken
    }

    return allObjects, nil
}

// Usage example
func main() {
    s3 := simples3.New("us-east-1", "your-access-key", "your-secret-key")

    allObjects, err := listAllObjects(s3, "my-bucket")
    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf("Total objects: %d\n", len(allObjects))
    for _, obj := range allObjects {
        fmt.Printf("- %s (%d bytes)\n", obj.Key, obj.Size)
    }
}

Object Tagging

S3 object tags are key-value pairs that you can use to categorize, organize, and manage objects. Each object can have up to 10 tags.

Put Tags on an Object

Set or replace all tags on an existing object:

err := s3.PutObjectTagging(simples3.PutObjectTaggingInput{
    Bucket:    "my-bucket",
    ObjectKey: "my-file.txt",
    Tags: map[string]string{
        "Environment": "production",
        "Project":     "website",
        "Version":     "v1.2.0",
    },
})
if err != nil {
    log.Fatal(err)
}

Get Tags from an Object

Retrieve all tags from an object:

output, err := s3.GetObjectTagging(simples3.GetObjectTaggingInput{
    Bucket:    "my-bucket",
    ObjectKey: "my-file.txt",
})
if err != nil {
    log.Fatal(err)
}

for key, value := range output.Tags {
    fmt.Printf("%s: %s\n", key, value)
}

Delete All Tags from an Object

Remove all tags from an object:

err := s3.DeleteObjectTagging(simples3.DeleteObjectTaggingInput{
    Bucket:    "my-bucket",
    ObjectKey: "my-file.txt",
})
if err != nil {
    log.Fatal(err)
}

Upload with Tags

You can set tags when uploading an object using FilePut:

_, err := s3.FilePut(simples3.UploadInput{
    Bucket:      "my-bucket",
    ObjectKey:   "my-file.txt",
    ContentType: "text/plain",
    Body:        file,
    Tags: map[string]string{
        "Environment": "production",
        "Department":  "engineering",
    },
})

Note: Tag support during upload varies by operation:

• FilePut (PUT): ✅ Full support
• FileUpload (POST): ⚠️ AWS S3 supported, MinIO limited (does not support tags in POST policy)
• CopyObject: ✅ Full support (handles MinIO/R2 signature quirks automatically)

Object Versioning

Enable and manage object versions to protect against accidental deletion or overwrite.

Enable Versioning

err := s3.PutBucketVersioning(simples3.PutBucketVersioningInput{
    Bucket: "my-bucket",
    Status: "Enabled", // or "Suspended"
})
if err != nil {
    log.Fatal(err)
}

// Check status
config, err := s3.GetBucketVersioning("my-bucket")
fmt.Printf("Versioning status: %s\n", config.Status)

List Object Versions

List all versions of objects in a bucket:

result, err := s3.ListVersions(simples3.ListVersionsInput{
    Bucket: "my-bucket",
    Prefix: "important-file.txt",
})
if err != nil {
    log.Fatal(err)
}

for _, version := range result.Versions {
    fmt.Printf("Key: %s, VersionId: %s, IsLatest: %v\n",
        version.Key, version.VersionId, version.IsLatest)
}

Download Specific Version

Retrieve a specific version of an object:

file, err := s3.FileDownload(simples3.DownloadInput{
    Bucket:    "my-bucket",
    ObjectKey: "my-file.txt",
    VersionId: "v1-version-id",
})
if err != nil {
    log.Fatal(err)
}
// Use file (io.ReadCloser) normally

Delete Specific Version

Delete a specific version of an object:

err := s3.FileDelete(simples3.DeleteInput{
    Bucket:    "my-bucket",
    ObjectKey: "my-file.txt",
    VersionId: "v1-version-id",
})

Server-Side Encryption

Secure your data at rest using Server-Side Encryption (SSE). SimpleS3 supports both SSE-S3 (AES256) and SSE-KMS.

Upload with SSE-S3 (AES256)

_, err := s3.FilePut(simples3.UploadInput{
    Bucket:               "my-bucket",
    ObjectKey:            "secure-file.txt",
    Body:                 strings.NewReader("secret data"),
    ServerSideEncryption: "AES256",
})

Upload with SSE-KMS

_, err := s3.FilePut(simples3.UploadInput{
    Bucket:               "my-bucket",
    ObjectKey:            "kms-encrypted-file.txt",
    Body:                 strings.NewReader("secret data"),
    ServerSideEncryption: "aws:kms",
    SSEKMSKeyId:          "arn:aws:kms:us-east-1:123456789012:key/your-key-id",
})

Multipart Upload with Encryption

output, err := s3.FileUploadMultipart(simples3.MultipartUploadInput{
    Bucket:               "my-bucket",
    ObjectKey:            "large-secure-file.mp4",
    Body:                 file,
    ServerSideEncryption: "AES256",
})

Copy with Encryption

_, err := s3.CopyObject(simples3.CopyObjectInput{
    SourceBucket:         "my-bucket",
    SourceKey:            "original.txt",
    DestBucket:           "my-bucket",
    DestKey:              "encrypted-copy.txt",
    ServerSideEncryption: "AES256",
})

Check Encryption Status

details, err := s3.FileDetails(simples3.DetailsInput{
    Bucket:    "my-bucket",
    ObjectKey: "secure-file.txt",
})

fmt.Printf("Encryption: %s\n", details.ServerSideEncryption)
if details.SSEKMSKeyId != "" {
    fmt.Printf("KMS Key ID: %s\n", details.SSEKMSKeyId)
}

Multipart Upload

For large files (>100MB), use multipart upload for better performance, resumability, and parallel uploads.

High-Level API (Recommended)

The easiest way to upload large files:

file, err := os.Open("large-video.mp4")
if err != nil {
    log.Fatal(err)
}
defer file.Close()

// FileUploadMultipart automatically handles chunking and parallel uploads
output, err := s3.FileUploadMultipart(simples3.MultipartUploadInput{
    Bucket:      "my-bucket",
    ObjectKey:   "videos/large-video.mp4",
    Body:        file,
    ContentType: "video/mp4",
    PartSize:    10 * 1024 * 1024, // 10MB parts (optional, default 5MB)
    Concurrency: 5,                 // Upload 5 parts in parallel (optional, default 1)
    OnProgress: func(info simples3.ProgressInfo) {
        fmt.Printf("\rProgress: %.1f%% (%d/%d parts)",
            float64(info.UploadedBytes)/float64(info.TotalBytes)*100,
            info.CurrentPart, info.TotalParts)
    },
})
if err != nil {
    log.Fatal(err)
}

fmt.Printf("\nUploaded: %s (ETag: %s)\n", output.Key, output.ETag)

Low-Level API (Advanced)

For more control over the upload process:

// 1. Initiate multipart upload
initOutput, err := s3.InitiateMultipartUpload(simples3.InitiateMultipartUploadInput{
    Bucket:      "my-bucket",
    ObjectKey:   "large-file.bin",
    ContentType: "application/octet-stream",
})
if err != nil {
    log.Fatal(err)
}

// 2. Upload parts (can be done in parallel or resumed later)
var completedParts []simples3.CompletedPart
partSize := int64(10 * 1024 * 1024) // 10MB

for partNum := 1; partNum <= totalParts; partNum++ {
    // Read part data
    partData := getPartData(partNum, partSize) // Your function to get part data

    output, err := s3.UploadPart(simples3.UploadPartInput{
        Bucket:     "my-bucket",
        ObjectKey:  "large-file.bin",
        UploadID:   initOutput.UploadID,
        PartNumber: partNum,
        Body:       bytes.NewReader(partData),
        Size:       int64(len(partData)),
    })
    if err != nil {
        // Abort on error
        s3.AbortMultipartUpload(simples3.AbortMultipartUploadInput{
            Bucket:    "my-bucket",
            ObjectKey: "large-file.bin",
            UploadID:  initOutput.UploadID,
        })
        log.Fatal(err)
    }

    completedParts = append(completedParts, simples3.CompletedPart{
        PartNumber: output.PartNumber,
        ETag:       output.ETag,
    })
}

// 3. Complete the upload
result, err := s3.CompleteMultipartUpload(simples3.CompleteMultipartUploadInput{
    Bucket:    "my-bucket",
    ObjectKey: "large-file.bin",
    UploadID:  initOutput.UploadID,
    Parts:     completedParts,
})
if err != nil {
    log.Fatal(err)
}

fmt.Printf("Upload completed: %s\n", result.ETag)

List Uploaded Parts

Query which parts have been uploaded (useful for resuming uploads):

output, err := s3.ListParts(simples3.ListPartsInput{
    Bucket:    "my-bucket",
    ObjectKey: "large-file.bin",
    UploadID:  uploadID,
})
if err != nil {
    log.Fatal(err)
}

for _, part := range output.Parts {
    fmt.Printf("Part %d: %d bytes (ETag: %s)\n",
        part.PartNumber, part.Size, part.ETag)
}

Abort Multipart Upload

Cancel an in-progress upload and clean up parts:

err := s3.AbortMultipartUpload(simples3.AbortMultipartUploadInput{
    Bucket:    "my-bucket",
    ObjectKey: "large-file.bin",
    UploadID:  uploadID,
})

Browser-Based Multipart Upload

Generate presigned URLs for each part to enable direct browser uploads:

// Backend: Initiate and generate presigned URLs
initOutput, _ := s3.InitiateMultipartUpload(simples3.InitiateMultipartUploadInput{
    Bucket:    "my-bucket",
    ObjectKey: "client-upload.bin",
})

// Generate presigned URL for part 1
presignedURL := s3.GeneratePresignedUploadPartURL(simples3.PresignedMultipartInput{
    Bucket:        "my-bucket",
    ObjectKey:     "client-upload.bin",
    UploadID:      initOutput.UploadID,
    PartNumber:    1,
    ExpirySeconds: 3600,
})

// Frontend: Upload directly to S3 using the presigned URL (via PUT request)
// Then send ETags back to backend for completion

Presigned URLs

Generate secure URLs for browser-based uploads/downloads:

// Generate presigned URL for download
url := s3.GeneratePresignedURL(simples3.PresignedInput{
    Bucket:        "my-bucket",
    ObjectKey:     "private-file.pdf",
    Method:        "GET",
    ExpirySeconds: 3600, // 1 hour
})

// Users can now download directly: <a href="{{url}}">Download</a>

Custom Endpoints

Use with MinIO, DigitalOcean Spaces, or other S3-compatible services:

// MinIO
s3 := simples3.New("us-east-1", "minioadmin", "minioadmin")
s3.SetEndpoint("http://localhost:9000")

// DigitalOcean Spaces
s3 := simples3.New("nyc3", "your-access-key", "your-secret-key")
s3.SetEndpoint("https://nyc3.digitaloceanspaces.com")

IAM Credentials

On EC2 instances, use IAM roles automatically:

s3, err := simples3.NewUsingIAM("us-east-1")
if err != nil {
    log.Fatal(err)
}
// Automatically uses instance IAM role

Development

Local integration tests use the official chrislusf/seaweedfs:4.48 image, pinned in docker-compose.yml. Docker Compose v2 is required. Nix supplies Go, Just and the AWS CLI:

git clone https://github.com/rhnvrm/simples3.git
cd simples3
nix develop
just test-local

just test-local starts SeaweedFS in its single-process mini mode with public test credentials and a pre-created testbucket, waits for authenticated bucket access, and runs a fresh, uncached Go suite with the race detector. The S3 API listens only on 127.0.0.1:9000. The credentials in this fixture are public and strictly for local tests. Automatic bucket creation on upload and recursive bucket deletion are explicitly disabled to exercise ordinary S3 behavior.

just setup           # Start SeaweedFS and wait for authenticated S3 readiness
just seaweedfs-up    # Start the container without waiting for S3 readiness
just seaweedfs-down  # Stop SeaweedFS, retain test data
just seaweedfs-logs  # Follow server logs
just seaweedfs-reset # Remove local test data and initialize a fresh fixture
just test            # Run tests against the currently configured endpoint
just vet
just status

Test coverage

The existing integration suite runs against SeaweedFS, including ACLs, tagging, versioning and SSE-S3 tests. No backend-specific capability skips are added. The existing POST-tagging integration test remains skipped pending client/backend compatibility. backend_protocol_test.go additionally checks request headers, signed POST policy conditions, XML serialization, response parsing and version query parameters using local HTTP servers. Run these checks without an external S3 service:

go test -race ./... -run '^TestBackendProtocol'

To use another compatible test endpoint, configure its credentials and test bucket. Use a dedicated account and bucket: integration tests create and delete buckets and objects.

export AWS_S3_REGION="us-east-1"
export AWS_S3_ACCESS_KEY="your-access-key"
export AWS_S3_SECRET_KEY="your-secret-key"
export AWS_S3_ENDPOINT="https://your-test-endpoint"
export AWS_S3_BUCKET="your-test-bucket"
go test -v ./...

Local fixture values shared by Nix, setup and test commands are defined in scripts/seaweedfs-env.sh and match the public values in Compose.

Contributing

Contributions welcome! Check ROADMAP.md for planned features. Please add tests and ensure just test-local passes before submitting PRs.

Author

Rohan Verma hello@rohanverma.net

License

BSD-2-Clause-FreeBSD