cannot

This isn't quite a "bug", as it seems very intentional. But I can't find a way to do what I need.

• iTerm2 version: Build 3.0.5
• OS version: OSX 10.11.5
• Attach ~/Library/Preferences/com.googlecode.iterm2.plist here (drag-drop from finder into this window): com.googlecode.iterm2.plist
• Attach a debug log, if possible: N/A
• Are you reporting a performance issue or a hang?: N/A

Detailed steps to reproduce the problem:

1. Copy
2. Try to paste into iterm2

What happened: 0x1f4a9appeared

What should have happened: appeared

I have a git branch feature/💩, currently the only way I can checkout, push, or pull this branch is to either:

• open terminal (yuck) and do it there
• put in a file, and replace any instance of in my commands with $(cat poop)

I can see the usefulness/not-breaking-things-ness of converting pasted characters to their character code, but either it isn't changeable (I can't find anything), or maybe it shouldn't be the default? (I also like the idea of warning me, like when I paste multiple lines and can choose to paste the ending newline or not)

Check the value of the environment variable $LANG. Terminal does some black magic to set it in some mixed country/language environments which I can't figure out. Toggling Prefs > Profiles > Terminal > Set locale variables automatically may or may not help; if not, you'll need to set it in your .bashrc/.zshrc.

I have this issue with my personal Mac (which is running Sierra), currently Build 3.0.20160729-nightly, but not on my work Mac, which is El Capitan (also a nightly, I can't check which version right now). My LANG is en_US.UTF-8.

Please verify that Prefs > Profiles > Terminal > Character Encoding is set to Unicode (UTF-8).

Does this happen only at a readline/shell prompt, or does it happen when accepting cooked input (e.g., when running cat)?

Does this affect unicode characters less than 0xffff? For example, does this work:

• The encoding is set to Unicode(UTF-8)
• I can't type 💩 anywhere, bash, cat, and emacs all don't work. I can M-x insert-char PILE OF POO in emacs and it inserts it.
• I can type just fine (in bash, cat and emacs).

Does Terminal have the same issue? If not, how does the output of stty -a differ between iTerm2 and Terminal?

Is this on Mac OS or are you ssh'ed somewhere?

Can you verify that the problem is with code points greater than 0xffff by trying a few more? Here are some that are greater. If my hypothesis is correct they should all exhibit the same issue as POO

𝟘🀱🂲🅐🢨

Seems to work fine in Terminal. Here is the Terminal stty -a:

speed 9600 baud; 54 rows; 202 columns;
lflags: icanon isig iexten echo echoe -echok echoke -echonl echoctl
	-echoprt -altwerase -noflsh -tostop -flusho pendin -nokerninfo
	-extproc
iflags: -istrip icrnl -inlcr -igncr ixon -ixoff ixany imaxbel iutf8
	-ignbrk brkint -inpck -ignpar -parmrk
oflags: opost onlcr -oxtabs -onocr -onlret
cflags: cread cs8 -parenb -parodd hupcl -clocal -cstopb -crtscts -dsrflow
	-dtrflow -mdmbuf
cchars: discard = ^O; dsusp = ^Y; eof = ^D; eol = <undef>;
	eol2 = <undef>; erase = ^?; intr = ^C; kill = ^U; lnext = ^V;
	min = 1; quit = ^\; reprint = ^R; start = ^Q; status = ^T;
	stop = ^S; susp = ^Z; time = 0; werase = ^W;

And here is iTerm2:

speed 38400 baud; 45 rows; 158 columns;
lflags: icanon isig iexten echo echoe echok echoke -echonl echoctl
	-echoprt -altwerase -noflsh -tostop -flusho pendin -nokerninfo
	-extproc
iflags: -istrip icrnl -inlcr -igncr ixon -ixoff ixany imaxbel iutf8
	-ignbrk brkint -inpck -ignpar -parmrk
oflags: opost onlcr -oxtabs -onocr -onlret
cflags: cread cs8 -parenb -parodd hupcl -clocal -cstopb -crtscts -dsrflow
	-dtrflow -mdmbuf
cchars: discard = ^O; dsusp = ^Y; eof = ^D; eol = <undef>;
	eol2 = <undef>; erase = ^?; intr = ^C; kill = ^U; lnext = ^V;
	min = 1; quit = ^\; reprint = ^R; start = ^Q; status = ^T;
	stop = ^S; susp = ^Z; time = 0; werase = ^W;

I don't see any differences, except for the first line.

𝟘🀱🂲🅐🢨

If I paste this it inserts several spaces. I pasted it into emacs and saved the file, and this is what was saved:

$xxd test
00000000: 2020 2020 2020 2020 2020 2020 2020 2020
00000010: 2020 2020 2020 2020 2020 2020 2020 2020
00000020: 200a                                      .

EDIT I think this is because there are several spaces in the original text.

OK, I just tried my work Mac after updating to the latest nightly (Build 3.0.20160815-nightly) and I can't paste emoji there either.

The tty seems well configured. Let's see if iTerm2 is sending the right codes. Can you make a debug log where you enter a few emoji? Use both paste and use the emoji panel (ctrl-cmd-space to open, then search for the name of the symbol) for each of these:

• WHITE SMILING FACE
• PILE OF POO

(the first is in the basic multilingual plane, and the second is not).

OK, something is somehow screwing up. If I create a debug log and try to enter , the debug log isn't saved. I tested doing other things and the debug log is there in /tmp/debuglog.txt, but when I paste it isn't there. Maybe whatever is causing to fail is also crashing the debug logger? This is still on Build 3.0.20160815-nightly (El Capitan) (EDIT: Just updated to Build 3.0.20160817-nightly and it's the same behavior).

works fine, although for inserting it from the character viewer I had to drag it in (double clicking didn't work).

That doesn't make any sense. Debug logging is just a variable in memory. The log isn't written until you stop logging. Maybe that's the issue?

I'm definitely stopping it. When I do the debug log without pasting , then run ls /tmp/, I see debuglog.txt. When I do it and then paste (and then stop the logging), then run ls /tmp/, there is no debuglog.txt.

Maybe whatever is writing the file is failing with some encoding issue or something?

That's truly weird. Try this build. It writes debug logs to /var/log/system.log.

http://iterm2.com/adhocbuilds/iTerm2-3_0_20160817_160443-adhoc.zip

Here's a log. I hope it's enough (it seems the contents of system.log are cycled quite often). I pasted , then dragged in , then pasted , then dragged in . There may have been some spaces mixed in as well.

iterm2.log

A log line is missing. It's almost like an exception was thrown, except the calling function doesn't have a try/catch and picked up where it left off. This is a juicy bug.

Please try this build, which adds a bunch more logging where things got weird.

http://iterm2.com/adhocbuilds/iTerm2-3_0_20160817_171247-adhoc.zip

iterm2-2.log

Just noticed the original report seems to imply that something else is inserted instead of (literally 0x1f4a9, if I read that right). Just to be clear, when I paste , nothing happens. The edit menu flashes, but no text at all is inserted. Sorry if this wasn't clear.

For some reason the string can't be converted to UTF-8 data. That seems to defeat most of my usual tricks, but I think I'll get something good out of the logs from this one. I suspect a surrogate pair is getting split up, but it really shouldn't be possible with such as short string. Again, it writes to /var/log/system.log. It'll be much less verbose. Let's focus on , no need to paste or type the symbols that work OK.

http://iterm2.com/adhocbuilds/iTerm2-3_0_20160818_215555-adhoc.zip

iterm2-3.log

It looks like the the speed of pasting has been set very low so only one byte is pasted per timer tick. This causes all surrogate pairs to get cut in half, and half a surrogate pair cannot be converted to UTF-8, so nothing gets written. The workaround is to increase the number of bytes to paste per call. The easiest way to adjust this is to change Prefs > Advanced > Number of bytes to paste in each chunk when pasting normally, which I guess is set to 1.

I should note that any bytes-per-call value could cut a surrogate pair in half, so I need to adjust the range by one character when this would happen.

Milestone changed to %Next Beta

Great! When I set that value to 2, I can paste . I have to set it to a really large value (at least 128) to paste 𝟘🀱🂲🅐🢨 correctly. What value would you recommend?

Nightly build fixed in commit 6d88419c

3.0.9 fixed in commit aaae079f

For now the only safe value is a one larger than the block you're pasting (infinity, effectively). This will be problematic when pasting into readline because it is buggy. However, with a large chunk size (a few kb), you're unlikely to sever a surrogate pair unless you're pasting nothing but emoji/traditional chinese/miscellaneous obscure unicode things.

Status changed to closed

So what's a safe value, given the fix (and given that I am using readline)?

You should use the latest nightly if possible. There really isn't a single safe value (you'd need to experiment to find it) but I'll try to get 3.0.9 out this week. Here's a preview of 3.0.9 that includes the fix if you prefer to be on the beta track.

http://iterm2.com/adhocbuilds/iTerm2-3_0_20160821_104513-adhoc.zip

I mean even with the nightly? Is 1 the default value for that setting? Do higher values make pasting faster?

The bug is now fixed in the nightly. The default is 667. Larger values paste faster. By default, 667 bytes are pasted every 0.015 seconds. Both the chunk size and the delay are tunable.

This is a really cool bug to get fixed, but though I haven't had much of a chance to participate in this, but I would like to point out that the issue that's been worked on is not the originally posted one (this was even mentioned previously here: https://gitlab.com/gnachman/iterm2/issues/5040#note_14034095).

But to respond to the initial set of questions asked:

• my $LANG value is the same in both iterm2 and terminal: en_US.UTF-8
• Prefs > Profiles > Terminal > Character Encoding is already set to Unicode (UTF-8).

And just to make sure this was communicated around my original issue: I can display the fine, I just can't find a way to type/paste it into iterm2.

Status changed to reopened

My delay when pasting normally was set to 0.001000000047497451, assumedly from me choosing "paste faster" many times. But upping the chunk size seems like a much better way to "paste faster", especially when it's only set to 1 somehow (and indeed, my pasting used to be somewhat slow, and now that I've reverted these to the defaults it is much faster).

I haven't had much of a chance to participate in this, but I would like to point out that the issue that's been worked on is not the originally posted one.

@ThisGuyCodes Does this affect unicode characters less than 0xffff? For example, does this work: Also, please provide a debug log (https://iterm2.com/debuglog)

works fine. Here's a debug log of me pasting then : debuglog.txt

As a side note, I'm using the tmux-iterm2 integration. Though I don't think tmux is a factor here, and terminal + tmux doesn't have the same issue.

@ThisGuyCodes Tmux is a factor here—this is a bug in tmux, as a matter of fact. It doesn't accept input of characters outside the basic multilingual plane using send-keys. Proof:

tmux send-keys 0x41

This inserts a capital A (U+0041)

tmux send-keys 0x1f4a9

This inserts 0x1f4a9.

I've sent a message to tmux-users asking if this is intentional. I can't find a workaround, unfortunately. I'll update this thread when I get an answer.

It seems like tmux's various input handling mechanisms are inconsistent, as I can use when I'm attached directly to tmux (rather than through iterm2's integration, which I assume doesn't just use stdin). I can show this by attaching a terminal session to the same tmux session as iterm2, and I can paste there.

Here's a screenshot to illustrate what I'm trying to get across:

To get this I:

• attach to the same tmux session in terminal
• clicke in iterm2 (left)
• paste
• clicke in terminal (right)
• paste

This is unfortunate :/

This bug appears to be fixed in tmux 2.3

Looks like 2.3 isn't released yet :(. For now you need to downgrade to 2.1 for surrogate pair input to work in tmux integration mode.

I've added a warning about the bug so people at least know what's going on in commit bfe97a8.

closed