velero-io/velero

[ceph] velero-plugin-for-aws: checksumAlgorithm: "" does not suppress AWS SDK v2 trailing checksum - PutObject 400 InvalidArgument on S3-compatible storage

Open

#9,804 opened on May 20, 2026

View on GitHub
 (2 comments) (0 reactions) (0 assignees)Go (1,550 forks)auto 404
Area/Storage/CephHelp wanted

Repository metrics

Stars
 (10,111 stars)
PR merge metrics
 (PR metrics pending)

Description

What steps did you take and what happened:

Upgraded from Velero 1.17.2 / velero-plugin-for-aws v1.13.1 to Velero 1.18.0 / velero-plugin-for-aws v1.14.1. All backups against an S3-compatible backend (Ceph/Swift) immediately began failing with:

rpc error: code = Unknown desc = error putting object backups/<name>/velero-backup.json:
operation error S3: PutObject,
https response error StatusCode: 400,
api error InvalidArgument: UnknownError

The BSL config has checksumAlgorithm: "" set — the documented workaround for XAmzContentSHA256Mismatch on non-AWS S3. CSI snapshot data uploads succeed; only small metadata JSON files (velero-backup.json) fail with 400.

Downgrading to plugin v1.13.1 / Velero 1.17.2 restores working backups immediately.

What did you expect to happen:

checksumAlgorithm: "" should fully suppress all checksum-related headers, including trailing checksums injected by the AWS SDK Go v2 manager.Uploader. S3-compatible backends that reject these headers should receive clean PutObject requests.

Anything else you would like to add:

The root cause is in pkg/object_store.go in velero-plugin-for-aws. checksumAlgorithm: "" only prevents ChecksumAlgorithm from being set on PutObjectInput:

if o.checksumAlg != "" {
    input.ChecksumAlgorithm = types.ChecksumAlgorithm(o.checksumAlg)
}

But the AWS SDK Go v2 manager.Uploader independently injects a trailing checksum into the HTTP request body via its own data integrity middleware, regardless of PutObjectInput. The newS3Client function does not set RequestChecksumCalculation on the S3 client options, so there is no way to suppress this from config alone.

Proposed fix

Extend newS3Client to set checksum calculation mode when checksumAlgorithm: "" is configured:

if checksumAlg == "" {
    opts = append(opts, func(o *s3.Options) {
        o.RequestChecksumCalculation = aws.RequestChecksumCalculationWhenRequired
        o.ResponseChecksumValidation = aws.ResponseChecksumValidationWhenRequired
    })
}

This uses the AWS SDK Go v2 opt-out mechanism for automatic checksum injection, matches the existing intent of checksumAlgorithm: "", and requires no new BSL config keys.

Environment:

Velero version: v1.18.0 Velero features: EnableCSI Kubernetes version: v1.34.5 Kubernetes installer & version: Helm chart velero-12.0.1 Cloud provider or hardware configuration: S3-compatible object storage (Ceph, Swift S3 API) OS: Linux

Contributor guide