GitLab CI sbt assembly hangs

I'm trying to get CI shell runner to build some of our scala projects using sbt. I have sbt installed properly and I can build the projects manually without issue. When I run them through CI it just sits there without any output after the invocation of the sbt command.

I've gone so far as to use debug mode on the runner with the same results as above.

Here's my config.toml

concurrent = 4
timeout = 600

[[runners]]
  name = "prod-gitlab-1.redacted.com"
  url = "https://ci.redacted.com/"
  token = "d90ce8823938c0000763ff9f016c50"
  limit = 4
  executor = "shell"

Here's the debug mode output:

gitlab-runner --debug run --working-directory /home/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --user gitlab-runner
INFO[0000] 0 Starting multi-runner from /etc/gitlab-runner/config.toml ...
DEBU[0000] 0 Feeding runners to channel
DEBU[0000] 0 Starting worker 0
DEBU[0000] 0 Checking runner prod-gitlab-1.REDACTED.com url=https://ci.REDACTED.com/ token=REDACTED executor=shell on 0
DEBU[0000] 0 Starting worker 1
DEBU[0000] 0 Starting worker 2
DEBU[0000] 0 Starting worker 3
DEBU[0000] d90ce882 Checking for builds... nothing
DEBU[0003] 0 Feeding runners to channel
DEBU[0003] 0 Checking runner prod-gitlab-1.REDACTED.com url=https://ci.REDACTED.com/ token=REDACTED executor=shell on 1
DEBU[0003] d90ce882 Checking for builds... nothing
DEBU[0006] 0 Feeding runners to channel
DEBU[0006] 0 Checking runner prod-gitlab-1.REDACTED.com url=https://ci.REDACTED.com/ token=REDACTED executor=shell on 2
DEBU[0006] d90ce882 Checking for builds... nothing
DEBU[0009] 0 Feeding runners to channel
DEBU[0009] 0 Checking runner prod-gitlab-1.REDACTED.com url=https://ci.REDACTED.com/ token=REDACTED executor=shell on 3
DEBU[0009] d90ce882 Checking for builds... nothing
DEBU[0012] 0 Feeding runners to channel
DEBU[0012] 0 Checking runner prod-gitlab-1.REDACTED.com url=https://ci.REDACTED.com/ token=REDACTED executor=shell on 0
DEBU[0012] d90ce882 Checking for builds... nothing
DEBU[0015] 0 Feeding runners to channel
DEBU[0015] 0 Checking runner prod-gitlab-1.REDACTED.com url=https://ci.REDACTED.com/ token=REDACTED executor=shell on 1
DEBU[0015] d90ce882 Checking for builds... nothing
DEBU[0018] 0 Feeding runners to channel
DEBU[0018] 0 Checking runner prod-gitlab-1.REDACTED.com url=https://ci.REDACTED.com/ token=REDACTED executor=shell on 2
DEBU[0018] d90ce882 Checking for builds... nothing
DEBU[0021] 0 Feeding runners to channel
DEBU[0021] 0 Checking runner prod-gitlab-1.REDACTED.com url=https://ci.REDACTED.com/ token=REDACTED executor=shell on 3
INFO[0021] d90ce882 Checking for builds... received
DEBU[0021] 0 Received new build for d90ce882 build 16
DEBU[0021] 1 Added a new build id: 16
projectid: 3
commands: |2-

  /usr/bin/java -Dsbt.log.noformat=true -Xmx2048m -jar /usr/share/sbt-launcher-packaging/bin/sbt-launch.jar clean assembly
repourl: https://gitlab-ci-token:REDACTED@gitlab.REDACTED.com/engineering/simulator-server.git
sha: REDACTED
refname: gitlabci_test
beforesha: REDACTED
allowgitfetch: true
timeout: 3600
variables: []
options: {}
buildstate: ""
buildstarted: {}
buildfinished: {}
buildduration: "0"
runner:
  name: prod-gitlab-1.REDACTED.com
  url: https://ci.REDACTED.com/
  token: REDACTED
  limit: 4
  executor: shell
  buildsdir: null
  environment: []
  shell: null
  disableverbose: null
  outputlimit: null
  ssh: null
  docker: null
  parallels: null
globalid: 0
runnerid: 0
projectrunnerid: 0

INFO[0021] d90ce882 16 gitlab-ci-multi-runner 0.5.2-15-g45ed89c-beta (45ed89c)
DEBU[0021] d90ce882 16 Shell script: environment:
- CI_BUILD_REF=REDACTED
- CI_BUILD_BEFORE_SHA=REDACTED
- CI_BUILD_REF_NAME=gitlabci_test
- CI_BUILD_ID=16
- CI_BUILD_REPO=https://gitlab-ci-token:REDACTED@gitlab.REDACTED.com/engineering/simulator-server.git
- CI_PROJECT_ID=3
- CI_PROJECT_DIR=builds/d90ce882/0/engineering/simulator-server
- CI=true
- CI_SERVER=yes
- CI_SERVER_NAME=GitLab CI
- CI_SERVER_VERSION=
- CI_SERVER_REVISION=
- GITLAB_CI=true
script: |+
  #!/usr/bin/env bash

  set -eo pipefail

  echo Running on $(hostname)...

  echo
  export $'CI_BUILD_REF=REDACTED'
  export $'CI_BUILD_BEFORE_SHA=REDACTED'
  export $'CI_BUILD_REF_NAME=gitlabci_test'
  export $'CI_BUILD_ID=16'
  export $'CI_BUILD_REPO=https://gitlab-ci-token:REDACTED@gitlab.REDACTED.com/engineering/simulator-server.git'
  export $'CI_PROJECT_ID=3'
  export $'CI_PROJECT_DIR=builds/d90ce882/0/engineering/simulator-server'
  export $'CI=true'
  export $'CI_SERVER=yes'
  export $'CI_SERVER_NAME=GitLab CI'
  export $'CI_SERVER_VERSION='
  export $'CI_SERVER_REVISION='
  export $'GITLAB_CI=true'

  # save script that is read from to file and execute script file on remote server
  mkdir -p $'builds/d90ce882/0/engineering/simulator-server'
  cat > $'builds/d90ce882/0/engineering/simulator-server.sh'; source $'builds/d90ce882/0/engineering/simulator-server.sh'

  if [[ -d $'builds/d90ce882/0/engineering/simulator-server/.git' ]]; then
  echo $'\x1b[32;1mFetching changes...\x1b[0;m'
  cd $'builds/d90ce882/0/engineering/simulator-server'
  git clean -fdx
  git reset --hard > /dev/null
  git remote set-url origin $'https://gitlab-ci-token:REDACTED@gitlab.REDACTED.com/engineering/simulator-server.git'
  git fetch origin
  else
  echo $'\x1b[32;1mCloning repository...\x1b[0;m'
  rm -rf $'builds/d90ce882/0/engineering/simulator-server'
  mkdir -p $'builds/d90ce882/0/engineering/simulator-server'
  git clone $'https://gitlab-ci-token:REDACTED@gitlab.REDACTED.com/engineering/simulator-server.git' $'builds/d90ce882/0/engineering/simulator-server'
  cd $'builds/d90ce882/0/engineering/simulator-server'
  fi
  echo $'\x1b[32;1mChecking out 1be8ae18 as gitlabci_test...\x1b[0;m'
  git checkout -qf REDACTED

  echo

  echo $'\x1b[32;1m$ /usr/bin/java -Dsbt.log.noformat=true -Xmx2048m -jar /usr/share/sbt-launcher-packaging/bin/sbt-launch.jar clean assembly\x1b[0;m'
 /usr/bin/java -Dsbt.log.noformat=true -Xmx2048m -jar /usr/share/sbt-launcher-packaging/bin/sbt-launch.jar clean assembly

command: su
arguments:
- --shell
- /bin/bash
- --login
- gitlab-runner
passfile: false
extension: ""

INFO[0021] d90ce882 16 Using Shell executor...
DEBU[0021] d90ce882 16 Starting shell command...
DEBU[0021] d90ce882 16 Waiting for signals...
DEBU[0024] 1 Feeding runners to channel
DEBU[0024] 1 Checking runner prod-gitlab-1.REDACTED.com url=https://ci.REDACTED.com/ token=REDACTED executor=shell on 0
DEBU[0024] d90ce882 Checking for builds... nothing
INFO[0024] d90ce882 16 Submitting build to coordinator... ok
DEBU[0027] 1 Feeding runners to channel
DEBU[0027] 1 Checking runner prod-gitlab-1.REDACTED.com url=https://ci.REDACTED.com/ token=REDACTED executor=shell on 1
DEBU[0027] d90ce882 Checking for builds... nothing
DEBU[0027] d90ce882 16 updateBuildLog Nothing to send.
WARN[0029] 1 Requested service stop
DEBU[0029] 1 Stopping worker 2
DEBU[0029] 1 Stopping worker 0
DEBU[0029] 1 Stopping worker 1
ERRO[0029] d90ce882 16 Build got aborted: interrupt
DEBU[0029] d90ce882 16 Build took 8.671563141s
DEBU[0029] d90ce882 16 Waiting for build log updater to finish
DEBU[0029] d90ce882 16 Build log updater finished.
DEBU[0029] d90ce882 16 Build log:  gitlab-ci-multi-runner 0.5.2-15-g45ed89c-beta (45ed89c)
Using Shell executor...
Running on prod-gitlab-1.REDACTED.com...

Fetching changes...
Checking out 1be8ae18 as gitlabci_test...

$ /usr/bin/java -Dsbt.log.noformat=true -Xmx2048m -jar /usr/share/sbt-launcher-packaging/bin/sbt-launch.jar clean assembly

ERROR: Build got aborted: interrupt

DEBU[0029] d90ce882 16 updateBuildLog Received finish.
DEBU[0029] d90ce882 16 PushTrace finished
INFO[0029] d90ce882 16 Submitting build to coordinator... ok
INFO[0029] d90ce882 16 Build finished.
DEBU[0029] 0 Build removed id: 16
projectid: 3
commands: |2-

  /usr/bin/java -Dsbt.log.noformat=true -Xmx2048m -jar /usr/share/sbt-launcher-packaging/bin/sbt-launch.jar clean assembly
repourl: https://gitlab-ci-token:REDACTED@gitlab.REDACTED.com/engineering/simulator-server.git
sha: REDACTED
refname: gitlabci_test
beforesha: REDACTED
allowgitfetch: true
timeout: 3600
variables: []
options: {}
buildstate: failed
buildstarted: {}
buildfinished: {}
buildduration: 8.671563141s
runner:
  name: prod-gitlab-1.REDACTED.com
  url: https://ci.REDACTED.com/
  token: REDACTED
  limit: 4
  executor: shell
  buildsdir: null
  environment: []
  shell: null
  disableverbose: null
  outputlimit: null
  ssh: null
  docker: null
  parallels: null
globalid: 0
runnerid: 0
projectrunnerid: 0

DEBU[0029] d90ce882 16 ReadTrace finished
INFO[0029] 0 All workers stopped. Can exit now

Builds will just sit there till the timeout is reached and then fail.

Any input would be greatly appreciated.

I figured it out! If I use the -Djline.terminal=jline.UnsupportedTerminal for sbt it works.

Status changed to closed

really save my life! but why should we add this and how do you figure it out?

Wow, this was a while ago...I think the thing that finally clued me in to what was going on was that I was getting an invalid terminal type error when I was debugging. Setting that option allowed sbt to construct the terminal and continue on with the build as normal.

I think I might be being bitten by this too, would you mind sharing a snippet of your .yml config @wengwenchien ?

Sure, hope it can help you

stages:
  - build
  - publish

build:
  stage: build
    - sbt -Djline.terminal=jline.UnsupportedTerminal compile


publish:
  stage: publish
  script:
  - sbt -Djline.terminal=jline.UnsupportedTerminal docker:publish

mentioned in merge request !14785