iTerm2 sometimes compacts Japanese modifiers whereas they should not be

• iTerm2 version: 3.1.beta.7 and 3.0.14 (can not reproduce on iTerm2 2.1.4)
• OS version: macOS Sierra 10.12.6
• com.googlecode.iterm2.plist
• Attach a debug log, if possible. (Not necessary?)
• Are you reporting a performance issue or a hang? No.
• Are you reporting a crash? Please attach the crash log. No.

Detailed steps to reproduce the problem:

1. Download test.txt
2. Go to iTerm and run cat text.txt

What happened:

The ゛ and ゜ characters are compacted into the previous character.

What should have happened:

The ゛ and ゜ characters should take an extra character width, as they do in iTerm2 2.1.4, macOS Terminal, or any text editor, as well as in the browser:

ﾄｳｷﾖｳﾃﾞﾝﾘﾖｸ
ｺ-ﾌﾟﾘﾖｳﾀﾞｲｷﾝ
ﾄｳｷﾖｳﾄｽｲﾄﾞｳｷﾖｸ
ﾄｳｷﾖｳｶﾞｽ (ｶ

Code markup will not apparently not output monospaced characters with Japanese (the browser's font might not support them), but it is kinda clear that here too the modifiers have their own space width.

Screenshots

Alternative test

I also used the Ruby gem unicode-display_width to confirm the strings' display width.

Where does it cause problems?

I'm trying to display tabular data in console, and to achieve that, I calculate the strings' Unicode display width using the Ruby gem I mentioned above. Unfortunately, because iTerm does not follow the correct convention to output those characters in fixed-width, I end up with mis-alignments in my tabular data.

This seems related: #4702 .

changed the description

changed the description

Fixed in commit ca3ebbb3. Please test tomorrow's nightly build.

closed

Yep, I can confirm this is fixed. Thank you very much!

Can you think of any reason why these modifiers should be in their own cells? The unicode spec says they should be combined with the preceding character; Apple's API reports the same; anecdotally, and I spoke with a friend who grew up in Japan and he said when he learned to write, the modifiers were always joined with the preceding character. I'm sure there's a reason for it and I'd love to understand what's going on here.

Japanese has many character sets. Full-width hiragana and full-width katakana are two syllabaries of which each character uses two monospace cells. In those sets, the modifiers are indeed joined with the character they modify and follow. For example, the characters that read da and pu, respectively in hiragana and katakana: だぷ, ダプ.

The half-width katakana character set on the other hand is kind of weird... First there's only half-width katakana, not half-width hiragana (I'm apparently wrong on this, but they're not used anymore). Characters are supposed to all be half-width of the previous set (which means each character fits in one monospace cell). According to Wikipedia, half-width katakana were made in computer early days to match monospace fonts' grid (when there's no smart way to calculate width, and you count characters, you want all characters to be the same width). They're not used often anymore, although old computer systems (banks!), and displays lacking space (train departure LED boards, sometimes) still use them.

This paragraph on Wikipedia explains the half-width kana's modifiers case:

To make katakana fit into the narrower cell area allowed, some compromises were made. For example, the diacritical marks dakuten and handakuten are treated as separate characters instead of being part of the preceding character. This compromise led many to consider "half-width kana" visually unattractive, and causes problems for many computer programs today.

(Emphasis mine.)

Here are the same da and pu characters in half-width katakana: ﾀﾞﾌﾟ.

Honestly I dont know everything about it, but i hope this explanation helps.

Edit: one more thing. You'll notice that, even though the dakuten and handakuten diacritics are displayed in their own cell in the case of half-width kana, they're nonetheless combined to the character they follow, effectively making the set one double-width Unicode character.

Thanks for the history lesson. Much appreciated!