Add CharacTER

[stancld]

🚀 Feature

Add CharacTER, a text metric used for NMT evaluation.

Sources:
Paper - CharacTER: Translation Edit Rate on Character Level
Repo

Motivation / Abstract

Recently, the capability of character-level evaluation measures for machine translation output has been confirmed by several metrics. This work proposes translation edit rate on character level (CharacTER), which calculates the character level edit distance while performing the shift edit on word level. The novel metric shows high system-level correlation with human rankings, especially for morphologically rich languages. It outperforms the strong CHRF by up to 7% correlation on different metric tasks. In addition, we apply the hypothesis sentence length for normalizing the edit distance in CharacTER, which also provides significant improvements compared to using the reference sentence length. (Wang et al., 2016)

[Rajathbharadwaj]

Hey @stancld @SkafteNicki, I want to contribute, how do I get started? Thanks.

[stancld]

Hi @Rajathbharadwaj, I hope you can have a look at #641 and try to prepare a PR for CharacTER accordingly. Once you have anything ready, you can open a draft PR and ping us :]

[SkafteNicki]

Hi @Rajathbharadwaj, agree with @stancld that PR #641 I good place to look what how a complete implementation (+ docs ) should be structured. If you do not have time to fully implement it all, feel free to also open a PR with a partial implementation and then we can guide you to the rest :]

[Rajathbharadwaj]

Okay, I will check out and hopefully open a draft PR. Thanks!

[Rajathbharadwaj]

@SkafteNicki @stancld So I did check out the PR #641 and I can see there are some helper functions or some extra functions compared to the source repo. It'd be great if I could get some help in those regards. I will go ahead and create the module now. Thanks!

[stancld]

Hello @Rajathbharadwaj, any update on this here? Let me know if you need any help :]

[stancld]

Hi @Rajathbharadwaj, any updates? :]

[Rajathbharadwaj]

Hey sorry about the delay. I will start contributing, been a bit busy.

Thanks.

[Borda]

Hey sorry about the delay. I will start contributing, been a bit busy.

no pressure, take your time 🐰

[ankitaS11]

Hey @Borda @stancld @SkafteNicki, I want to pick this issue, if its fine with everyone.

[stancld]

@ankitaS11 Yes, it's fine to take this one, but please be sure #855 to be merged before diving into this one :]

[ankitaS11]

Sure. Will make that PR ready for review by this weekend.

[stancld]

Hi @ankitaS11, any updates here? :]