i18n-embed: get message_id for desired locale, falling back directly to base fallback language

[saskenuba]

Hello!

Sorry for the verbose title.

My web application has N available languages. On runtime, I check for the user's preferred locale, let's say pt-BR. Then I want to get the message for this locale, falling back directly en-US, but with all other locale files loaded, to proper populate the email template with the user's language.

Is there a method with this behaviour now on FluentLanguageLoader?

This would make it possible to avoid bundling each language with "en-US", the way I do it now with raw Fluent.

If I understood correctly, FluentLanguageLoader works in a strict fallback manner, as seen here, it tries to get the message at the highest priority locale, if it can't find it, goes to next locale and so on until the base one.

A concrete example:

#[test]
    fn fallbacks_ru_to_base() {
        setup();
        let ru: LanguageIdentifier = "ru".parse().unwrap();
        let en_us: LanguageIdentifier = "en-US".parse().unwrap();
        let en_gb: LanguageIdentifier = "en-GB".parse().unwrap();

        let loader = FluentLanguageLoader::new("test", en_us.clone());
        loader
            .load_languages(&Localizations, &[&ru, &en_gb])
            .unwrap();
        pretty_assertions::assert_eq!("only GB (US Version)", loader.get_with(ru, "only-gb"));
        pretty_assertions::assert_eq!("только русский", loader.get_with(ru, "only-ru"));
    }

Would this be useful or should I go for another approach?

[kellpossible]

On runtime, I check for the user's preferred locale, let's say pt-BR

Is this localization occurring on the front-end or the back-end of your web application?

but with all other locale files loaded, to proper populate the email template with the user's language.

Sorry I don't understand this use-case completely. You want to have all available locales loaded, or just the ones which the user is requesting? Why do they all need to be loaded?

This would make it possible to avoid bundling each language with "en-US",

Why are you bundling each language with "en-US" and why do you want to avoid it?

A concrete example:

Would you be able to write more about the proposed get_with() method?
Do you think your use-case might be similar to this feature request? #37

[saskenuba]

Is this localization occurring on the front-end or the back-end of your web application?

This section of localization occurs at the back-end. The user's preferred locale is stored on the database, I retrieve it, get all messages-ids from the .ftl that match his preferred locale that I need, falling back to English if it doesn't exist (my use-case).

// retrieved from database
struct User {
	locale: String  // stored as "pt-BR"
}

// strum helps me connect everything dynamically
#[derive(AsStaticStr, EnumString)]
enum LocalesAvailable {
	#[strum(serialize = "pt-BR")]
	Portuguese
	#[strum(serialize = "en-US")]
	English
	// ... other languages
}

impl LocalesAvailable {
	// With this, I can dynamically fetch the message-id
	fn to_langid(&self) -> LanguageIdentifier {
		self.into().parse().unwrap()
	}
}


fn render_email_template(user: User, loader: &'static FluentLanguageLoader) {
	let user_locale = LocalesAvailable::from_str(&*user.locale); // LocalesAvailable::Portuguese
	
	// loader is statically embedded at the server, containing all N translations
	let email_subject = loader.get_with(user_locale.to_langid(), "email-confirmation-subject");

	// ..
}

Hopefully, this makes sense.

To be honest, this should not happen very often, since it is very important to keep all locales on par with the most complete one. The reasoning is that it doesn't make much sense to me if the user main language is Russian, to fallback for Spanish. His preferences are stored, there is no language negotiation anymore, so simply fallback to one specific lang.

But I feel that this may be very specific to me. I created a middleware that takes care of the initial lang negotiation, so this a more of a separate step much later in the process.

Why do they all need to be loaded?

Simply because they are so tiny, aggregating to barely 200kb. With the language loader allowing me to get the message-ids of the locale of interest, and falling back to a specific one. Much more convenient and minimal overhead.

Today I need to bundle (es-AR, en-us), (pt-BR, en-us), (ru, en-us), to get this behaviour.

Sorry if in some bits I was redundant, English is not my main language, hope you understand the core idea. I'm not even sure if something like this would be useful for the crate.

[kellpossible]

The get_with(locale, message_id) method would skip the regular fallback negotiation, and either try to get the message in the specified locale, or fall back to the global fallback locale?

It does seem very similar to what is being requested in #37 (which would probably implement the same behaviour). I'm leaning more towards having a separate implementation of FluentLanuageLoader (perhaps called FluentMultiLanguageLoader), for this purpose which has no concept of a single "current language", all languages are loaded, and which optionally calculates fallbacks at runtime with each get request, and by default is set up in a fast path to fall back to the globally specified fallback locale.

Simply because they are so tiny, aggregating to barely 200kb. With the language loader allowing me to get the message-ids of the locale of interest, and falling back to a specific one. Much more convenient and minimal overhead.

Thanks I think I'm understanding. You want to avoid creating a separate bundle for each language to get the specific fallback behaviour you want because it uses more memory, and creating a bundle the fly for each request would be worse right?

Sorry if in some bits I was redundant, English is not my main language, hope you understand the core idea. I'm not even sure if something like this would be useful for the crate.

That's okay, sorry for my questions too, and thanks for posting this issue, it's an important topic if this library is to be used in backend services I think. At the moment it is currently geared towards desktop applications and front-end where the language/fallbacks being requested do not change very often.

[saskenuba]

The get_with(locale, message_id) method would skip the regular fallback negotiation, and either try to get the message in the specified locale or fall back to the global fallback locale?

Yes, exactly.

It does seem very similar to what is being requested in #37 (which would probably implement the same behaviour). I'm leaning more towards having a separate implementation of FluentLanuageLoader (perhaps called FluentMultiLanguageLoader), for this purpose which has no concept of a single "current language", all languages are loaded, and which optionally calculates fallbacks at runtime with each get request, and by default is set up in a fast path to fall back to the globally specified fallback locale.

Following your thoughts on a different loader, FluentMultiLanguageLoader could have even two methods, one to get a locale, falling back directly to a global language, and another, to fallback to specified locales, or maybe that is too much?

/// fallback to global fallback language
fn get_with(&self, locale: LanguageIdentifier, message_id: &str) {}

// fallback to custom fallback languages
// could work if es-ES is ready but es-AR is still being translated
// if message_id does not exist in either languages, fallsback to global
fn get_with_custom_fallback(&self, locales: &[LanguageIdentifier], message_id: &str) {}

Thanks I think I'm understanding. You want to avoid creating a separate bundle for each language to get the specific fallback behaviour you want because it uses more memory, and creating a bundle the fly for each request would be worse right?

That would be it. I think this is more of a convenience. FluentMultiLanguageLoader is acting as a map type to hold every existent translation, instead of bundling them individually, matching and retrieving based on some rules that a backend would require.

The use case here would be where memory is plenty since cargo-i18 has alternatives for embedded systems with more limited hardware.

That's okay, sorry for my questions too, and thanks for posting this issue, it's an important topic if this library is to be used in backend services I think. At the moment it is currently geared towards desktop applications and front-end where the language/fallbacks being requested do not change very often.

That is great to hear, are you interested in a PR? I could set up a draft and you can decide if this is going in the direction you want.

[kellpossible]

Following your thoughts on a different loader, FluentMultiLanguageLoader could have even two methods, one to get a locale, falling back directly to a global language, and another, to fallback to specified locales, or maybe that is too much?

That sounds like a good idea!

That is great to hear, are you interested in a PR? I could set up a draft and you can decide if this is going in the direction you want.

Sure, I would definitely appreciate that, and happy to review

[saskenuba]

I was working on something that for my use-case it worked perfectly, and decided to push the PR as draft: #62

There are four new methods for the MultiLoader, get_with_locale and get_with_locale_and_args, for retrieval of a single translated string, for a specific locale, falling back to the global fallback language.

And get_with_custom_fallback and get_with_custom_fallback_and_args for custom fallback behaviour, based on an array of LanguageIdentifier, but ultimately falling back to the local fallback language too.

I've added tests and a string to the FTL to make sure it behaves accordingly.

There is duplicated code since they load FTL files the same way, perhaps they could be moved to some free functions.

Also, current_language doesn't make much sense in the MultiLoader context, since there is no concept of it, then I left it as unimplemented. What do you think?

[kellpossible]

@saskenuba Thanks for your contribution! I've been away for the past few days, but I'll make sure to do a proper reply soon and review your PR