apache/gravitino

[Improvement] Fix policy version limit being applied per policy instead of globally

Open

#10,275 opened on Mar 6, 2026

View on GitHub
 (4 comments) (0 reactions) (1 assignee)Java (887 forks)auto 404
good first issueimprovement

Repository metrics

Stars
 (3,058 stars)
PR merge metrics
 (PR metrics pending)

Description

What would you like to be improved?

PolicyMetaService.deletePolicyVersionsByRetentionCount(versionRetentionCount, limit) applies limit inside a loop for each policy. In production, this is called by the relational garbage collector with GARBAGE_COLLECTOR_SINGLE_DELETION_LIMIT (100). With multiple policies, one GC iteration can delete up to limit * numberOfPolicies, exceeding the intended single-batch cap and causing unexpectedly large cleanup operations.

How should we improve?

Enforce limit as a global budget per method call. Track remaining = limit, and for each policy call softDeletePolicyVersionsByRetentionLine(..., remaining), subtracting deleted rows from remaining; stop when remaining <= 0. This preserves bounded batch behavior while still progressing cleanup across runs.

here a test to help:

@TestTemplate
public void testDeletePolicyVersionsByRetentionCountRespectGlobalLimit() throws IOException {
  BaseMetalake metalake = createAndInsertMakeLake(METALAKE_NAME + "_global_limit");

  PolicyEntity policy1 =
      createPolicy(
          RandomIdGenerator.INSTANCE.nextId(),
          NamespaceUtil.ofPolicy(metalake.name()),
          "policy_limit_1",
          AUDIT_INFO);
  PolicyEntity policy2 =
      createPolicy(
          RandomIdGenerator.INSTANCE.nextId(),
          NamespaceUtil.ofPolicy(metalake.name()),
          "policy_limit_2",
          AUDIT_INFO);
  backend.insert(policy1, false);
  backend.insert(policy2, false);

  PolicyEntity policy1V2 =
      PolicyEntity.builder()
          .withId(policy1.id())
          .withNamespace(policy1.namespace())
          .withName(policy1.name())
          .withPolicyType(policy1.policyType())
          .withComment(policy1.comment())
          .withEnabled(!policy1.enabled())
          .withContent(policy1.content())
          .withAuditInfo(AUDIT_INFO)
          .build();
  PolicyEntity policy2V2 =
      PolicyEntity.builder()
          .withId(policy2.id())
          .withNamespace(policy2.namespace())
          .withName(policy2.name())
          .withPolicyType(policy2.policyType())
          .withComment(policy2.comment())
          .withEnabled(!policy2.enabled())
          .withContent(policy2.content())
          .withAuditInfo(AUDIT_INFO)
          .build();
  backend.update(policy1.nameIdentifier(), Entity.EntityType.POLICY, e -> policy1V2);
  backend.update(policy2.nameIdentifier(), Entity.EntityType.POLICY, e -> policy2V2);

  int deletedCount = PolicyMetaService.getInstance().deletePolicyVersionsByRetentionCount(1L, 1);

  assertEquals(1, deletedCount);
}

Contributor guide