Data race in CredentialsFileCredentialsProvider when watched_directory is configured causes heap corruption and SIGSEGV

Summary

When credentials_file_provider is configured with a watched_directory, CachedCredentialsProviderBase::cached_credentials_ is written on every getCredentials() call with no synchronization. Under concurrent request load, multiple Envoy worker threads race to write cached_credentials_ (a Credentials object holding std::string members), producing heap corruption that manifests as a SIGSEGV inside malloc/operator new.

Removing watched_directory from the config stops the crashes, confirming the watched_directory code path as the trigger.

Affected versions

Confirmed crashing on both v1.36.0 and v1.38.2 with identical crash signatures. The bug is in CachedCredentialsProviderBase / CredentialsFileCredentialsProvider, which is unchanged across these versions.

Component

source/extensions/common/aws/credential_providers/credentials_file_credentials_provider.cc
source/extensions/common/aws/cached_credentials_provider_base.h

Root cause

1. watched_directory makes needsRefresh() unconditionally return true

// credentials_file_credentials_provider.cc
bool CredentialsFileCredentialsProvider::needsRefresh() {
  return has_watched_directory_
      ? true   // ← always refresh on every call
      : context_.api().timeSource().systemTime() - last_updated_ > REFRESH_INTERVAL;
}

has_watched_directory_ is set to true in the constructor when credentials_data_source.has_watched_directory() is true.

2. Every getCredentials() call therefore writes cached_credentials_

// cached_credentials_provider_base.h
Credentials getCredentials() override {
    refreshIfNeeded();           // always true → always calls refresh()
    return cached_credentials_;  // read
}

// credentials_file_credentials_provider.cc — extractCredentials()
cached_credentials_ = Credentials(access_key_id, secret_access_key, session_token);
last_updated_ = context_.api().timeSource().systemTime();

3. Neither cached_credentials_ nor last_updated_ is protected by any synchronization

// cached_credentials_provider_base.h — complete member variable list
protected:
  SystemTime last_updated_;
  Credentials cached_credentials_;
  // No mutex, no std::atomic, no absl::Mutex, no lock of any kind.

Credentials holds std::string members (access_key_id, secret_access_key, session_token). Multiple worker threads simultaneously calling getCredentials() (once per signed upstream request) concurrently write to those std::string members — a data race, which is undefined behaviour in C++.

The normal (non-watched_directory) code path is not affected: REFRESH_INTERVAL limits refreshes to once per period, and at low refresh frequency the race window is small enough to go unobserved in practice.

Observed crash

SIGSEGV at faulting address 0x0, worker thread, inside libc malloc. Symbolized backtrace (amd64 debug build):

#0  [libc malloc internals — faulting]
#1  Envoy::Extensions::Common::Aws::Utility::useDoubleUriEncode(std::string)
      ← operator new(48) call here; malloc null-derefs corrupted freelist
#2  Envoy::Extensions::Common::Aws::Utility::shouldNormalizeUriPath(std::string)
#3  Envoy::Extensions::Common::Aws::SignerBaseImpl::sign(...)
      signer_base_impl.cc:102
#4  Envoy::Extensions::Common::Aws::SignerBaseImpl::signUnsignedPayload(...)
#5  Envoy::Extensions::HttpFilters::AwsRequestSigningFilter::Filter::continueDecodeHeaders(...)
#6  Envoy::Extensions::HttpFilters::AwsRequestSigningFilter::Filter::decodeHeaders(...)
      ← aws_request_signing upstream filter, resuming after ext_authz approval
#7  Envoy::Http::FilterManager::decodeHeaders(...)
#8  Envoy::Router::UpstreamRequest::acceptHeadersFromRouter(bool)
...
#13 Envoy::Extensions::HttpFilters::ExtAuthz::Filter::onComplete(...)
#14 Envoy::Extensions::Filters::Common::ExtAuthz::GrpcClientImpl::onSuccess(...)
...
#40 Envoy::Server::WorkerImpl::threadRoutine(...)

The crash site (operator new inside useDoubleUriEncode) is the victim of heap corruption caused by the data race. The signer and ext_authz filter chain are not themselves buggy — the corruption originates in the concurrent cached_credentials_ writes, and crashes wherever the next heap allocation touches the poisoned metadata.

Reproduction conditions

• credentials_file_provider with watched_directory configured
• aws_request_signing upstream HTTP filter (or any code path calling getCredentials() per-request)
• Multiple Envoy worker threads handling concurrent upstream requests (standard production load)

The crash is non-deterministic and load-dependent — it manifests under sustained concurrent request volume, typically within minutes to hours of startup. Disabling watched_directory stops the crashes immediately.

Impact

Envoy process crash (SIGSEGV). Pod restarts in a crash loop under any sustained load. Observed in deployments using the aws_request_signing upstream filter with file-based credentials and watched_directory for automatic rotation.

Proposed fix

CachedCredentialsProviderBase::getCredentials(), refreshIfNeeded(), and the cached_credentials_ / last_updated_ members should be protected by a mutex (e.g. absl::Mutex).

Additionally, the watched_directory refresh strategy should be reconsidered. Returning true from needsRefresh() on every call means the credentials file is re-read and re-parsed on every single request when watched_directory is configured — which is expensive and unnecessary. A better approach: use the DataSourceProvider's watch callback to set a dirty flag, and only needsRefresh() when the flag is set. This both eliminates the per-request file read overhead and reduces the write frequency to the cadence of actual file changes, shrinking the race window even before a mutex is added.

// Sketch of flag-based fix
bool needsRefresh() override {
    return credentials_dirty_.load(std::memory_order_acquire);
}

// Called from the DataSourceProvider watch callback (main thread)
void onCredentialsFileChanged() {
    credentials_dirty_.store(true, std::memory_order_release);
}

// In extractCredentials(), after writing cached_credentials_ under lock:
credentials_dirty_.store(false, std::memory_order_release);

private:
  std::atomic<bool> credentials_dirty_{false};
  absl::Mutex credentials_mutex_;

The minimal safe fix is just adding absl::MutexLock around reads and writes of cached_credentials_ and last_updated_ throughout the class hierarchy.