Ability to chain LR schedulers to complement LR composition
#5,127 opened on Jul 6, 2026
Repository metrics
- Stars
- (15,550 stars)
- PR merge metrics
- (PR metrics pending)
Description
Burn currently provides ComposedLRScheduler which provides a mechanism for composing different learning rate schedulers that emits a single LR. They can be combined/composed with 3 different composition reductions, namely Avg, Sum and Prod.
This said, Burn currently seems to lack a mechanism for swapping out new LR Schedulers at different milestones of the training journey. In my experience, this is a far more widely used use-case.
Pytorch provides a pair of LR combinators, namely:
-
Chaining Schedulers: How it works: Each scheduler adjusts the learning rate obtained by the preceding one. At every step, the new learning rate is the product of all active scaling factors. Implementation: You pass a list of schedulers to
torch.optim.lr_scheduler.ChainedScheduler. Use case: Applying continuous exponential decay alongside a warm-up phase or a constant learning rate period. -
Sequential Schedulers How it works: You define a list of schedulers, and each one takes over the learning rate schedule only when a specific milestone (epoch number) is reached. Implementation: You pass your list of schedulers and a list of milestone epochs to
torch.optim.lr_scheduler.SequentialLR. Use case: Switching abruptly from a linear warm-up phase to a cosine annealing schedule.
Clearly, burn's analog for 1 above is its ComposedLrSchedulerConfig, but there appears to be no analog for 2, despite this being a more common use-case.
Note in pytorch the milestones initializer for transitioning between schedulers (which in Burn's case could be epochs, or some number of iterations which would allow more granular control, but either way would be fine):
# The LinearLR will run for the first 5 epochs, then CosineAnnealingLR takes over
combined_scheduler = lr_scheduler.SequentialLR(
optimizer,
schedulers=[warmup_scheduler, cosine_scheduler],
milestones=[5]
)
Burn should provide an analog for pytorch's SequentialLR to handle non-overlapping LR scheduler types/configurations.
A very typical use-case requires a warmup phase (linear?) for some number of epochs/iterations, followed by a transition to an alternate (e.g. cosine) for the remainder of training.