worker: do not ingore `unref` when called in `parentPort`
This patch will keep unref in the parentPort
handle when the worker un unref-ed. After this lands, it does not matter if the worker receives more listeners after unref
, once unref-ed, will be keep it as it is.