Problem merging a request using API with Ruby wrapper and CLI for Gitlab

I'm using Ruby wrapper and CLI for GitLab REST API (https://github.com/NARKOZ/gitlab) to create and accept requests in our CI.

I have 2 script, one create a Merge Request and one cycle through the merge request list of the project, find the correct one and accept the merge. The 2 ruby script are called in 2 successive stages inside a gitlab-ci.yaml. All was working correctly until 6 day ago. After that, today, I've done some work and the merge accept failed with a 405 error. I was using the old 3.7 version that was still using version 3 API so I upgraded the Ruby wrapper to 4.0 and changed to use API 4.0 but I still get the same error.

The strange thing is that if I go to the server where the runner executed the script, login with the gitlab-runner user, go into the build directory and run the command manually the merge succeed. I've thinked of a timing issue so I put a sleep but still get the error:

This is the output of the gitlab-runner with the 405 error message

Running with gitlab-ci-multi-runner 9.0.2 (fa8b86d)
  on Puppet Master (04a94afd)
Using Shell executor...
Running on itrompm10...
Cloning repository...
Cloning into '/home/gitlab-runner/builds/04a94afd/0/puppet/control-repo'...
Checking out f3d67801 as development...
Skipping Git submodules setup
$ sleep 10
$ bin/gitlab_accept_merge_request.rb development testing
/usr/local/share/gems/gems/gitlab-4.0.0/lib/gitlab/request.rb:80:in `validate': Server responded with code 405, message: 405 Method Not Allowed. Request URI: https://itromgit01.q8int.com/api/v3/projects/3/merge_requests/298/merge (Gitlab::Error::MethodNotAllowed)
	from /usr/local/share/gems/gems/gitlab-4.0.0/lib/gitlab/request.rb:55:in `put'
	from /usr/local/share/gems/gems/gitlab-4.0.0/lib/gitlab/client/merge_requests.rb:82:in `accept_merge_request'
	from /usr/local/share/gems/gems/gitlab-4.0.0/lib/gitlab.rb:25:in `method_missing'
	from bin/gitlab_accept_merge_request.rb:35:in `block in <main>'
	from /usr/local/share/gems/gems/gitlab-4.0.0/lib/gitlab/paginated_response.rb:48:in `block (2 levels) in auto_paginate'
	from /usr/local/share/gems/gems/gitlab-4.0.0/lib/gitlab/paginated_response.rb:20:in `each'
	from /usr/local/share/gems/gems/gitlab-4.0.0/lib/gitlab/paginated_response.rb:20:in `method_missing'
	from /usr/local/share/gems/gems/gitlab-4.0.0/lib/gitlab/paginated_response.rb:47:in `block in auto_paginate'
	from /usr/local/share/gems/gems/gitlab-4.0.0/lib/gitlab/paginated_response.rb:36:in `each_page'
	from /usr/local/share/gems/gems/gitlab-4.0.0/lib/gitlab/paginated_response.rb:45:in `auto_paginate'
	from bin/gitlab_accept_merge_request.rb:27:in `<main>'
Merging id 3/298: Merged: f3d6780 Test merge request
 from development to testing
Running after script...
$ bin/gitlab_after.sh
Cleanup keys

This is the code used. It get some parameters from a configuration file but at the end the line that get the error is this: Gitlab.accept_merge_request(project_id,mr_id, { merge_commit_message: "#{mr_title}" })

Complete code

#!/usr/bin/env ruby
require 'gitlab'
GITLAB_CONFIG='/etc/gitlab-cli.conf'
config = {}
File.foreach GITLAB_CONFIG do |line|
  k = line.split("=")[0].gsub("\n",'') if line =~/=/
  v = line.split("=")[1].gsub("\n",'') if line =~/=/
  config.store(k,v)
end
last_commit=`git log -1 --oneline`
source_branch = ARGV[0] ? ARGV[0] : 'development'
destination_branch = ARGV[1] ? ARGV[1] : 'testing'
mr_title = "Merged: #{last_commit} from #{source_branch} to #{destination_branch}"

project_id = config['GITLAB_API_PROJECT_ID']
endpoint = config['GITLAB_API_ENDPOINT']
private_token = config['GITLAB_API_PRIVATE_TOKEN']
httparty = config['GITLAB_API_HTTPARTY_OPTIONS'].gsub("'",'')

Gitlab.endpoint = endpoint
Gitlab.private_token = private_token
Gitlab.httparty = eval(httparty)

merge_requests = Gitlab.merge_requests(project_id)
#merge_requests.has_next_page?
#merge_requests.next_page
merge_requests.auto_paginate do |merge_request|
  req=merge_request.to_h
  if req["state"] == "opened"
    mr_id=req["iid"]
    sb=req["source_branch"]
    db=req["target_branch"]
    if source_branch == sb and destination_branch == db
      print "Merging id #{project_id}/#{mr_id}: #{mr_title}\n"
      Gitlab.accept_merge_request(project_id,mr_id, { merge_commit_message: "#{mr_title}" })
      exit 0
    end
  end
end

#merge_requests.auto_paginate

print "Error: Merge Request NOT Found: #{mr_title}\n"
exit 1

If I run the code from the gitlab runner user I don't get any error and the merged will be correctly accepter:

This is the output of the run:

[gitlab-runner@itrompm10 control-repo]$ bin/gitlab_accept_merge_request.rb development testing
Merging id 3/298: Merged: f3d6780 Test merge request
 from development to testing
[gitlab-runner@itrompm10 control-repo]$ 

I've also posted an Issue on the Ruby wrapper (https://github.com/NARKOZ/gitlab/issues/272)

The issue is present on Gitlab 9.0.5-ee and gitlab-runner-9.0.2, I've checked back on date of the update and this was working correctly on 9.0.5 without problem.

Hi @fduranti from the stack trace I can see that the URI you are calling is https://itromgit01.q8int.com/api/v3/projects/3/merge_requests/298/merge could you try it using v4 instead of v3?

I've tested the same script both with v3 and with v4 (it was working correctly with v3 until 6 day ago). The parameter for the url is outside the CI so running it with v4 I get the same error:

Running with gitlab-ci-multi-runner 9.0.2 (fa8b86d)
  on Puppet Master (04a94afd)
Using Shell executor...
Running on itrompm10...
Cloning repository...
Cloning into '/home/gitlab-runner/builds/04a94afd/0/puppet/control-repo'...
Checking out f3d67801 as development...
Skipping Git submodules setup
$ sleep 10
$ bin/gitlab_accept_merge_request.rb development testing
/usr/local/share/gems/gems/gitlab-4.0.0/lib/gitlab/request.rb:80:in `validate': Server responded with code 405, message: 405 Method Not Allowed. Request URI: https://itromgit01.q8int.com/api/v4/projects/3/merge_requests/298/merge (Gitlab::Error::MethodNotAllowed)
	from /usr/local/share/gems/gems/gitlab-4.0.0/lib/gitlab/request.rb:55:in `put'
	from /usr/local/share/gems/gems/gitlab-4.0.0/lib/gitlab/client/merge_requests.rb:82:in `accept_merge_request'
	from /usr/local/share/gems/gems/gitlab-4.0.0/lib/gitlab.rb:25:in `method_missing'
	from bin/gitlab_accept_merge_request.rb:35:in `block in <main>'
	from /usr/local/share/gems/gems/gitlab-4.0.0/lib/gitlab/paginated_response.rb:48:in `block (2 levels) in auto_paginate'
	from /usr/local/share/gems/gems/gitlab-4.0.0/lib/gitlab/paginated_response.rb:20:in `each'
	from /usr/local/share/gems/gems/gitlab-4.0.0/lib/gitlab/paginated_response.rb:20:in `method_missing'
	from /usr/local/share/gems/gems/gitlab-4.0.0/lib/gitlab/paginated_response.rb:47:in `block in auto_paginate'
	from /usr/local/share/gems/gems/gitlab-4.0.0/lib/gitlab/paginated_response.rb:36:in `each_page'
	from /usr/local/share/gems/gems/gitlab-4.0.0/lib/gitlab/paginated_response.rb:45:in `auto_paginate'
	from bin/gitlab_accept_merge_request.rb:27:in `<main>'
Merging id 3/298: Merged: f3d6780 Test merge request
 from development to testing
Running after script...
$ bin/gitlab_after.sh
Cleanup keys

ERROR: Job failed: exit status 1

If i run the same script with the same parameter manually with gitlab-runner user from the build directory the Merge get accepted and I don't get any error. The problem seems to arise only when I run the script inside the CI.

I wonder if you copied you GITLAB_CONFIG='/etc/gitlab-cli.conf' file?

Nothing special (removed the token)

# File managed by Puppet

GITLAB_API_ENDPOINT=https://itromgit01.q8int.com/api/v4
GITLAB_API_PRIVATE_TOKEN=.........
GITLAB_API_HTTPARTY_OPTIONS='{verify: false}'
GITLAB_API_PROJECT_ID=3
GITLAB_USER=''
GITLAB_MILESTONE=''
GITLAB_LABELS=''
GITLAB_TARGET_LABEL=false
GITLAB_SOURCE_LABEL=false
GITLAB_DEFAULT_TARGET_LABEL='TO_'
GITLAB_DEFAULT_SOURCE_LABEL='FROM_'

@fduranti I meant to say: do you copy it in your .gitlab-ci.yml? So, either you have to expose these vars as secret variables, but it seems you didn't expose your private token to the script.

@zj That file is always present on the puppetserver and managed by puppet so it's not touched by the ci (so I can modify the ENDPOINT and do a new run with a different endpoint). It's the same file used by the other script that create the merge request. That file is loaded and used by the ruby program and not executed directly and it don't need Secret Variable (the script run also outside the gitlab-ci without problem):

This is the configuration load:

require 'gitlab'
GITLAB_CONFIG='/etc/gitlab-cli.conf'
config = {}
File.foreach GITLAB_CONFIG do |line|
  k = line.split("=")[0].gsub("\n",'') if line =~/=/
  v = line.split("=")[1].gsub("\n",'') if line =~/=/
  config.store(k,v)
end

I think I've found what cause the problem. The pipeline is still running when I try the merge and probably some day ago I've enabled the option on gitlab to "Only allow merge requests to be merged if the pipeline succeeds". In this way probably it will get an error because the merge request cannot be accepted as the pipeline is still running.

I'm now trying to use the "merge_when_pipeline_succeeds: true" and this should work as it will set the pipeline to be merged when it complete but I still get the same 405 error. Probably I'm still missing something. My function call is now:

Gitlab.accept_merge_request(project_id,mr_id, { merge_commit_message: "#{mr_title}", merge_when_pipeline_succeeds: true })

Related to #26695 (moved)?

Yes, it was a bug (https://gitlab.com/gitlab-org/gitlab-ce/issues/22740) and should be fixed with https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10873

closed