Upgrade from 9.1.2 to 9.2.2 -> Error 500 when upload artifact and job success

changed the description

I'm seeing similar behaviour with EE and HA setup (webservers are actually accessing an nfs mount). In our case it's a regular build that is using artifacts. It failed with "too large archive" but the job status was still successful.

ERROR: Uploading artifacts to coordinator... too large archive  id=13747 responseStatus=413 Request Entity Too Large status=413 Request Entity Too Large token=D1JNq7-E
FATAL: Too large                                  
Job succeeded

mentioned in issue #32249 (moved)

Tested with 9.2.5 and still occured

I am having the same issue on EE. Started with version 9.2.2 and persists to the latest 9.2.5. This impacting all of our builds. I am only seeing “internal server error” though.

Ah I was wrong, it turned out to be a space issue. Looks like when artifacts are uploaded to gitlab, nginx cache's the artifact within /var/opt/gitlab/gitlab-rails/tmp. I had 10GB free on this volume but the build job that was uploading had some gigantic payloads in it that sucked up the space.

Still bizarre that the job shows successful even though it fails to upload the artifacts. This looks like a clear bug to me. Also I read in another post that the logging in the trace file was suppose to provide better logging then just 500 internal error which is not the case, and the production log was also of no help. It would be nice to bubble these errors up to the user within the trace file so that we can investigate/ troubleshoot.

Cleaning out /var/opt/gitlab/gitlab-rails/tmp and tweaking the build job to not have such large payloads fixed the issue for me.

fyi Tailing /var/log/gitlab/nginx/gitlab_error.log is what lead me to find the pwrite() "/var/opt/gitlab/nginx/client_body_temp/0000000036" failed (28: No space left on device) error.

It's seems not a space left problem for me

About 56Go of free space

me@gitlab:~# df -H
Sys. de fichiers                                        Taille Utilisé Dispo Uti% Monté sur
/dev/sda1                                                 101G     41G   56G  42% /
udev                                                       11M       0   11M   0% /dev
tmpfs                                                     2,2G    211M  1,9G  11% /run
tmpfs                                                     5,3G    4,1k  5,3G   1% /dev/shm
tmpfs                                                     5,3M       0  5,3M   0% /run/lock
tmpfs                                                     5,3G       0  5,3G   0% /sys/fs/cgroup
//xxxxxxxxx/GitlabArtifacts                                22T     20T  1,6T  93% /mnt/gitlabArtifacts   #### CIFS mount
tmpfs                                                     1,1G       0  1,1G   0% /run/user/1002
tmpfs                                                     1,1G       0  1,1G   0% /run/user/0

And less than 2Mo for the tmp folder

me@gitlab:~# du -sh /var/opt/gitlab/gitlab-rails/tmp
1,7M    /var/opt/gitlab/gitlab-rails/tmp

Problem Fixed when upgrade to 9.3.5

closed

Upgrade from 9.1.2 to 9.2.2 -> Error 500 when upload artifact and job success

Summary

Steps to reproduce

Relevant logs and/or screenshots

Results of GitLab environment info

Results of GitLab application Check

Designs

Child items ...

Activity

Admin message

Admin message

Upgrade from 9.1.2 to 9.2.2 -> Error 500 when upload artifact and job success

Summary

Steps to reproduce

Relevant logs and/or screenshots

Results of GitLab environment info

Results of GitLab application Check

Activity