Two comments are rendered after submitting a response

This does not happen always.

1. Submit a comment in an issue
2. The placeholder comment is rendered
3. Once the response is received both the placeholder comment and the response are rendered as a normal response.

This does not happen for system notes, only comments the user made.

I don't know how to reproduce this always, maybe @rspeicher or @blackst0ne can help, both of them reported this on slack as well.

/cc @jschatz1 @fatihacet