Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ssl] add layerdropout for fintune #2045

Closed
wants to merge 1 commit into from
Closed

Conversation

Mddct
Copy link
Collaborator

@Mddct Mddct commented Oct 11, 2023

Not for merge for now (too many auto-format issues, just a example)
see:

if layer_dropout_rate == 0.0:
self.encoders = torch.nn.ModuleList([])
else:
self.encoders = LayerDropModuleList(p=dropout_rate)
self.encoders.extend([
ConformerEncoderLayer(
output_size,
encoder_selfattn_layer(*encoder_selfattn_layer_args),
positionwise_layer(*positionwise_layer_args),
positionwise_layer(
*positionwise_layer_args) if macaron_style else None,
convolution_layer(
*convolution_layer_args) if use_cnn_module else None,
dropout_rate,
normalize_before,
) for _ in range(num_blocks)
])

ref:

@Mddct Mddct force-pushed the Mddct-ssl-fintune-layerdropout branch from 9e0edb2 to cd54ccc Compare October 11, 2023 03:54
@Mddct Mddct changed the title [ssl/wa2vbert] add layerdropout for fintune [ssl] add layerdropout for fintune Oct 11, 2023
@Mddct Mddct closed this Oct 11, 2023
@Mddct Mddct deleted the Mddct-ssl-fintune-layerdropout branch October 11, 2023 12:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant