Relax checks for attn_mask_type in FlashAttention

[cyanguwa]

HazyResearch FlashAttention supports 'causal' mask type but also support 'no mask' type. This PR relaxes the restriction on attn_mask_type being 'causal' in the PyTorch FlashAttention module.

[cyanguwa]

/te-ci

[timmoon10]

Seems reasonable. Would it be better if we still checked that attn_mask_type != "padding"?

[ptrendx]

We still need to check if the attn_mask is None inside the forward pass - if the type is padding but the mask is None then it is effectively a "no mask" option.

[cyanguwa]

I fixed the logic a little bit. I think there are four possibilities for the mask type and tensor.

1. attn_mask_type = causal, attn_mask is None: this is the use_flash_attention = True case, with self.attn_causal_mask = True in FlashAttention().
2. attn_mask_type = padding, attn_mask is None: this is the use_flash_attention = True case, with self.attn_causal_mask = False. No mask is performed, either causal (False) or padding (ignored).
3. attn_mask_type = padding, attn_mask is not None: at the moment, we don't have the proper logic for this, so I disabled the flash attention path (use_flash_attention = False). In the future, we can apply the provided mask to q/k/v before passing q/k/v to flash attention. We also need to check if attn_mask is in the format of consecutive Trues plus consecutive Falses. The performance for padding plus the flash attention, needs to be verified as well, against unfused DPA.
4. attn_mask_type = causal, attn_mask is not None: we ignore the mask tensor in this case, just like we do in the unfused DPA case. We let flash attention use its internally generated mask. The note we have for DPA should cover our case here too.

[cyanguwa]

/te-ci

[ptrendx]

LGTM

[cyanguwa]

HazyResearch FlashAttention supports 'causal' mask type but also support 'no mask' type. This PR relaxes the restriction on attn_mask_type being 'causal' in the PyTorch FlashAttention module.

[cyanguwa]

/te-ci

[timmoon10]

Seems reasonable. Would it be better if we still checked that attn_mask_type != "padding"?

[ptrendx]

We still need to check if the attn_mask is None inside the forward pass - if the type is padding but the mask is None then it is effectively a "no mask" option.

[cyanguwa]

I fixed the logic a little bit. I think there are four possibilities for the mask type and tensor.

1. attn_mask_type = causal, attn_mask is None: this is the use_flash_attention = True case, with self.attn_causal_mask = True in FlashAttention().
2. attn_mask_type = padding, attn_mask is None: this is the use_flash_attention = True case, with self.attn_causal_mask = False. No mask is performed, either causal (False) or padding (ignored).
3. attn_mask_type = padding, attn_mask is not None: at the moment, we don't have the proper logic for this, so I disabled the flash attention path (use_flash_attention = False). In the future, we can apply the provided mask to q/k/v before passing q/k/v to flash attention. We also need to check if attn_mask is in the format of consecutive Trues plus consecutive Falses. The performance for padding plus the flash attention, needs to be verified as well, against unfused DPA.
4. attn_mask_type = causal, attn_mask is not None: we ignore the mask tensor in this case, just like we do in the unfused DPA case. We let flash attention use its internally generated mask. The note we have for DPA should cover our case here too.

[cyanguwa]

/te-ci

[ptrendx]

LGTM

[cyanguwa]

HazyResearch FlashAttention supports 'causal' mask type but also support 'no mask' type. This PR relaxes the restriction on attn_mask_type being 'causal' in the PyTorch FlashAttention module.

[cyanguwa]

/te-ci

[timmoon10]

Seems reasonable. Would it be better if we still checked that attn_mask_type != "padding"?

[ptrendx]

We still need to check if the attn_mask is None inside the forward pass - if the type is padding but the mask is None then it is effectively a "no mask" option.

[cyanguwa]

I fixed the logic a little bit. I think there are four possibilities for the mask type and tensor.

1. attn_mask_type = causal, attn_mask is None: this is the use_flash_attention = True case, with self.attn_causal_mask = True in FlashAttention().
2. attn_mask_type = padding, attn_mask is None: this is the use_flash_attention = True case, with self.attn_causal_mask = False. No mask is performed, either causal (False) or padding (ignored).
3. attn_mask_type = padding, attn_mask is not None: at the moment, we don't have the proper logic for this, so I disabled the flash attention path (use_flash_attention = False). In the future, we can apply the provided mask to q/k/v before passing q/k/v to flash attention. We also need to check if attn_mask is in the format of consecutive Trues plus consecutive Falses. The performance for padding plus the flash attention, needs to be verified as well, against unfused DPA.
4. attn_mask_type = causal, attn_mask is not None: we ignore the mask tensor in this case, just like we do in the unfused DPA case. We let flash attention use its internally generated mask. The note we have for DPA should cover our case here too.

[cyanguwa]

/te-ci

[ptrendx]

LGTM

[cyanguwa]

HazyResearch FlashAttention supports 'causal' mask type but also support 'no mask' type. This PR relaxes the restriction on attn_mask_type being 'causal' in the PyTorch FlashAttention module.

[cyanguwa]

/te-ci

[timmoon10]

Seems reasonable. Would it be better if we still checked that attn_mask_type != "padding"?

[ptrendx]

We still need to check if the attn_mask is None inside the forward pass - if the type is padding but the mask is None then it is effectively a "no mask" option.

[cyanguwa]

I fixed the logic a little bit. I think there are four possibilities for the mask type and tensor.

1. attn_mask_type = causal, attn_mask is None: this is the use_flash_attention = True case, with self.attn_causal_mask = True in FlashAttention().
2. attn_mask_type = padding, attn_mask is None: this is the use_flash_attention = True case, with self.attn_causal_mask = False. No mask is performed, either causal (False) or padding (ignored).
3. attn_mask_type = padding, attn_mask is not None: at the moment, we don't have the proper logic for this, so I disabled the flash attention path (use_flash_attention = False). In the future, we can apply the provided mask to q/k/v before passing q/k/v to flash attention. We also need to check if attn_mask is in the format of consecutive Trues plus consecutive Falses. The performance for padding plus the flash attention, needs to be verified as well, against unfused DPA.
4. attn_mask_type = causal, attn_mask is not None: we ignore the mask tensor in this case, just like we do in the unfused DPA case. We let flash attention use its internally generated mask. The note we have for DPA should cover our case here too.

[cyanguwa]

/te-ci

[ptrendx]

LGTM

[cyanguwa]

HazyResearch FlashAttention supports 'causal' mask type but also support 'no mask' type. This PR relaxes the restriction on attn_mask_type being 'causal' in the PyTorch FlashAttention module.

[cyanguwa]

/te-ci

[timmoon10]

Seems reasonable. Would it be better if we still checked that attn_mask_type != "padding"?

[ptrendx]

We still need to check if the attn_mask is None inside the forward pass - if the type is padding but the mask is None then it is effectively a "no mask" option.

[cyanguwa]

I fixed the logic a little bit. I think there are four possibilities for the mask type and tensor.

1. attn_mask_type = causal, attn_mask is None: this is the use_flash_attention = True case, with self.attn_causal_mask = True in FlashAttention().
2. attn_mask_type = padding, attn_mask is None: this is the use_flash_attention = True case, with self.attn_causal_mask = False. No mask is performed, either causal (False) or padding (ignored).
3. attn_mask_type = padding, attn_mask is not None: at the moment, we don't have the proper logic for this, so I disabled the flash attention path (use_flash_attention = False). In the future, we can apply the provided mask to q/k/v before passing q/k/v to flash attention. We also need to check if attn_mask is in the format of consecutive Trues plus consecutive Falses. The performance for padding plus the flash attention, needs to be verified as well, against unfused DPA.
4. attn_mask_type = causal, attn_mask is not None: we ignore the mask tensor in this case, just like we do in the unfused DPA case. We let flash attention use its internally generated mask. The note we have for DPA should cover our case here too.

[cyanguwa]

/te-ci

[ptrendx]

LGTM

[cyanguwa]

HazyResearch FlashAttention supports 'causal' mask type but also support 'no mask' type. This PR relaxes the restriction on attn_mask_type being 'causal' in the PyTorch FlashAttention module.

[cyanguwa]

/te-ci

[timmoon10]

Seems reasonable. Would it be better if we still checked that attn_mask_type != "padding"?

[ptrendx]

We still need to check if the attn_mask is None inside the forward pass - if the type is padding but the mask is None then it is effectively a "no mask" option.

[cyanguwa]

I fixed the logic a little bit. I think there are four possibilities for the mask type and tensor.

1. attn_mask_type = causal, attn_mask is None: this is the use_flash_attention = True case, with self.attn_causal_mask = True in FlashAttention().
2. attn_mask_type = padding, attn_mask is None: this is the use_flash_attention = True case, with self.attn_causal_mask = False. No mask is performed, either causal (False) or padding (ignored).
3. attn_mask_type = padding, attn_mask is not None: at the moment, we don't have the proper logic for this, so I disabled the flash attention path (use_flash_attention = False). In the future, we can apply the provided mask to q/k/v before passing q/k/v to flash attention. We also need to check if attn_mask is in the format of consecutive Trues plus consecutive Falses. The performance for padding plus the flash attention, needs to be verified as well, against unfused DPA.
4. attn_mask_type = causal, attn_mask is not None: we ignore the mask tensor in this case, just like we do in the unfused DPA case. We let flash attention use its internally generated mask. The note we have for DPA should cover our case here too.

[cyanguwa]

/te-ci

[ptrendx]

LGTM

[cyanguwa]

HazyResearch FlashAttention supports 'causal' mask type but also support 'no mask' type. This PR relaxes the restriction on attn_mask_type being 'causal' in the PyTorch FlashAttention module.

[cyanguwa]

/te-ci

[timmoon10]

Seems reasonable. Would it be better if we still checked that attn_mask_type != "padding"?

[ptrendx]

We still need to check if the attn_mask is None inside the forward pass - if the type is padding but the mask is None then it is effectively a "no mask" option.

[cyanguwa]

I fixed the logic a little bit. I think there are four possibilities for the mask type and tensor.

1. attn_mask_type = causal, attn_mask is None: this is the use_flash_attention = True case, with self.attn_causal_mask = True in FlashAttention().
2. attn_mask_type = padding, attn_mask is None: this is the use_flash_attention = True case, with self.attn_causal_mask = False. No mask is performed, either causal (False) or padding (ignored).
3. attn_mask_type = padding, attn_mask is not None: at the moment, we don't have the proper logic for this, so I disabled the flash attention path (use_flash_attention = False). In the future, we can apply the provided mask to q/k/v before passing q/k/v to flash attention. We also need to check if attn_mask is in the format of consecutive Trues plus consecutive Falses. The performance for padding plus the flash attention, needs to be verified as well, against unfused DPA.
4. attn_mask_type = causal, attn_mask is not None: we ignore the mask tensor in this case, just like we do in the unfused DPA case. We let flash attention use its internally generated mask. The note we have for DPA should cover our case here too.

[cyanguwa]

/te-ci

[ptrendx]

LGTM

[cyanguwa]

HazyResearch FlashAttention supports 'causal' mask type but also support 'no mask' type. This PR relaxes the restriction on attn_mask_type being 'causal' in the PyTorch FlashAttention module.

[cyanguwa]

/te-ci

[timmoon10]

Seems reasonable. Would it be better if we still checked that attn_mask_type != "padding"?

[ptrendx]

We still need to check if the attn_mask is None inside the forward pass - if the type is padding but the mask is None then it is effectively a "no mask" option.

[cyanguwa]

I fixed the logic a little bit. I think there are four possibilities for the mask type and tensor.

1. attn_mask_type = causal, attn_mask is None: this is the use_flash_attention = True case, with self.attn_causal_mask = True in FlashAttention().
2. attn_mask_type = padding, attn_mask is None: this is the use_flash_attention = True case, with self.attn_causal_mask = False. No mask is performed, either causal (False) or padding (ignored).
3. attn_mask_type = padding, attn_mask is not None: at the moment, we don't have the proper logic for this, so I disabled the flash attention path (use_flash_attention = False). In the future, we can apply the provided mask to q/k/v before passing q/k/v to flash attention. We also need to check if attn_mask is in the format of consecutive Trues plus consecutive Falses. The performance for padding plus the flash attention, needs to be verified as well, against unfused DPA.
4. attn_mask_type = causal, attn_mask is not None: we ignore the mask tensor in this case, just like we do in the unfused DPA case. We let flash attention use its internally generated mask. The note we have for DPA should cover our case here too.

[cyanguwa]

/te-ci

[ptrendx]

LGTM