Exposing normalisation data to external consumers

[taj-p]

In linebender/parley#452 (comment), we identified that it might be useful for consumers of HarfRust to have access to the normalization data to prevent storing duplicate table data in binaries.

Per @dfrg's comment, as a consumer of HarfRust, Parley also wants to match HarfRust so using a single source of truth is beneficial to maintain consistency between shaping and other stages of the text pipeline.

The options of implementation vary:

• Simply expose normalize_nfc / normalize_nfd functions from unicode.rs
• Expose the tables themselves and push consumers to write their own normalize_nfc / normalize_nfd functions using the data
• Split out the data into its own crate, ask consumers to pass in a trait that implements what HarfRust requires while supporting a baked or similar feature flag
• ...

I think the most important question I have is whether there's appetite for this sort of change.

There are other opportunities of deduplication like potentially the CharExt but I think it might be best to start the dialogue with normalization data?

[behdad]

Thanks. I have three ideas I like to share, related to this issue and the broader ICU4X integration:

1. In HarfBuzz we use a collection of C function pointers (hb_unicode_funcs_t) to abstract this away, exactly such that implementations integrating with ICU / glib / etc can provide their own. The rust equivalent would be, I think, a dyn trait. I leave it to @dfrg to decide whether we want that in HR, or always hardcode ours. I believe ours to be more compact and faster than other implementations FWIW (my third point below). If we don't want the full plugability in HR, at least we can expose our functions, so the clients can use them. That part should be easy. We don't have normalize_nfc/normalize_nfd but their building-blocks compose/decompose.

2. While I agree that it is important that the font fallback itemizer and HR agree on what font can shape a piece of text, I think my preference is on a different approach than making itemizer match the same normalization as HR. HB/HR do a complex normalization-like pass on the text based on what the font supports. This is tailored by each shaper, and in general is more complicated than I feel comfortable duplicating in font itemizer implementations. The alternative is what we call shaper-driven itemization, and Chrome / LibreOffice are prominent implementations of it: you simply shape the text with your primary font, then look for unshaped holes in the middle and reshape those holes with the next font, and next font, etc. This sidesteps the classic font fallback itemization. I think this is worth considering in Parley.

3. I just want to give a shout out to packtab, which is used to pack Unicode tables for HB / HR. It generates very compact & fast code. I don't know how applicable it is to your use-case in Parley.

http://github.com/harfbuzz/packtab
https://docs.google.com/document/d/1Xq3owVt61HVkJqbLFHl73il6pcTy6PdPJJ7bSouQiQw/preview

[taj-p]

Thank you @behdad ! I appreciate the knowledge sharing on (2). I think it's something we at Parley should definitely consider - I've created an issue (linebender/parley#462) to track this.

I just want to give a shout out to packtab, which is used to pack Unicode tables for HB / HR. It generates very compact & fast code. I don't know how applicable it is to your use-case in Parley.

This is certainly interesting. I think I'll need more time to ponder it to understand the tradeoffs against our current approach of using icu4x's CodepointTrie (docs).

That part should be easy. We don't have normalize_nfc/normalize_nfd but their building-blocks compose/decompose.

I'm wondering what the API guarantees are at this stage of HarfRust? Would the next step be to expose compose/decompose and then consider evolving that API with experience or are we sufficiently worried about breaking changes that we'd like to expose a more general and stable solution that is aligned with HB?

[behdad]

Thank you @behdad ! I appreciate the knowledge sharing on (2). I think it's something we at Parley should definitely consider - I've created an issue (linebender/parley#462) to track this.

You're welcome. Any time.

I just want to give a shout out to packtab, which is used to pack Unicode tables for HB / HR. It generates very compact & fast code. I don't know how applicable it is to your use-case in Parley.

This is certainly interesting. I think I'll need more time to ponder it to understand the tradeoffs against our current approach of using icu4x's CodepointTrie (docs).

Happy to discuss or video some time.

That part should be easy. We don't have normalize_nfc/normalize_nfd but their building-blocks compose/decompose.

I'm wondering what the API guarantees are at this stage of HarfRust? Would the next step be to expose compose/decompose and then consider evolving that API with experience or are we sufficiently worried about breaking changes that we'd like to expose a more general and stable solution that is aligned with HB?

Since HR has the data, I think we should expose it (raw) to users, like HB does.

[robertbastian]

ICU4X implements the traits from harfbuzz-traits for integration with harfbuzz. Ideally harfrust and parley would align on the same interface.