infrastructure-fsdp-activation-chec-when-using-fsdp-with-activation-checkpointing-true-d9a54fb5

THEN Explicitly set use_cache=False in the model kwargs when activation checkpointing is enabled (in addition to gradient checkpointing). For example, in your model_kwargs set 'use_cache' to not (gradient_checkpointing or activation_checkpointing). This ensures caching is disabled during activation checkpointing, preventing the metadata mismatch.

model_kwargs = {
    'use_cache': not (config.gradient_checkpointing or config.fsdp_config.activation_checkpointing)
}