GitLab shows 500 on commits/master page.
This is an error log:
Started GET "/BaBL/intra-portal/commits/master" for 127.0.0.1 at 2014-10-18 19:57:06 +0400
Processing by Projects::CommitsController#show as HTML
Parameters: {"project_id"=>"BaBL/intra-portal", "id"=>"master"}
Completed 500 Internal Server Error in 74ms
ActionView::Template::Error (invalid byte sequence in UTF-8):
3: = link_to commit.short_id, project_commit_path(project, commit), class: "commit_short_id"
4:
5: %span.str-truncated
6: = link_to_gfm commit.title, project_commit_path(project, commit.id), class: "commit-row-message"
7: - if commit.description?
8: %a.text-expander.js-toggle-button ...
9:
app/models/commit.rb:76:in `title'
app/views/projects/commits/_commit.html.haml:6:in `_app_views_projects_commits__commit_html_haml__543803606__572437258'
app/views/projects/commits/_commits.html.haml:10:in `block in _app_views_projects_commits__commits_html_haml___643990452__572322088'
app/views/projects/commits/_commits.html.haml:1:in `each'
app/views/projects/commits/_commits.html.haml:1:in `_app_views_projects_commits__commits_html_haml___643990452__572322088'
app/views/projects/commits/show.html.haml:18:in `_app_views_projects_commits_show_html_haml__691712179__571757398'
app/controllers/projects/commits_controller.rb:19:in `show'version 7.4RC (18/10/2014), collation in config: utf8_general_ci
Activity
No. We use UTF-8 with russian lang. This is a gitlab installation from 4.0 version + updates every month to current stable (and this is at first I try 7.4RC instead of 7.3).
I don't know what is the commit it was, but in 7.2.1 all works fine. I'll open DB and look at events table and don't see any problems with commints. There is no any broken symbols or something another problems (I use phpmyadmin with UTF-8 encoding).
Another branches and projects works fine =/ how to find this bad commit? May be you can provide me a debug "echo" string and tell what file I need to edit ?
I can see a dashboard and activity pages with a big number of commits and it's messages. And It works fine, without 500. But when I click Commits - Error 500.
I'll find this commit:
--- :before: 3d5a0cc64d1f913947738266653a79914bdfdd94 :after: d146bd54097eaebf606eb3336e01b3804208144c :ref: refs/heads/master :user_id: 4 :user_name: "Михаил" :project_id: 3 :repository: :name: Intra Portal :url: git@gitdev.ascottgroup.eu:BaBL/intra-portal.git :description: :homepage: http://gitdev.ascottgroup.eu/BaBL/intra-portal :commits: - :id: d146bd54097eaebf606eb3336e01b3804208144c :message: "new SFA reports" :timestamp: '2014-10-03T13:59:33+04:00' :url: http://gitdev.ascottgroup.eu/BaBL/intra-portal/commit/d146bd54097eaebf606eb3336e01b3804208144c :author: :name: Sidorkin Mickhail :email: sidorkin@kleo.com :total_commits_count: 1what information do you need?
Started GET "/BaBL/intra-portal/commit/d146bd54097eaebf606eb3336e01b3804208144c" for 127.0.0.1 at 2014-10-19 16:34:27 +0400 Processing by Projects::CommitController#show as HTML Parameters: {"project_id"=>"BaBL/intra-portal", "id"=>"d146bd54097eaebf606eb3336e01b3804208144c"} Completed 500 Internal Server Error in 281ms ActionView::Template::Error (invalid byte sequence in UTF-8): 52: 53: .commit-box 54: %h3.commit-title 55: = gfm escape_once(@commit.title) 56: - if @commit.description.present? 57: %pre.commit-description 58: = preserve(gfm(escape_once(@commit.description))) app/models/commit.rb:76:in `title' app/views/projects/commit/_commit_box.html.haml:55:in `_app_views_projects_commit__commit_box_html_haml___368493080__575747558' app/views/projects/commit/show.html.haml:1:in `_app_views_projects_commit_show_html_haml___754526429__575695658' app/controllers/projects/commit_controller.rb:27:in `show' Started GET "/admin/logs" for 127.0.0.1 at 2014-10-19 16:34:43 +0400 Processing by Admin::LogsController#show as HTMLmentioned in issue #721 (closed)
Hello! That's same as in https://gitlab.com/gitlab-com/support-forum/issues/4 - I mentioned there my problem and posted error log.
We're facing this same issue, and were able to work around with this. It basically marks any unparseable UTF-8 with ?.
diff --git a/app/models/commit.rb b/app/models/commit.rb index 2122296..9e32cca 100644 --- a/app/models/commit.rb +++ b/app/models/commit.rb @@ -72,6 +72,7 @@ class Commit # after 80 characters and ellipses (`&hellp;`) are appended. def title title = safe_message + title = title.encode("UTF-8", :invalid => :replace, :undef => :replace, :replace => "?") return no_commit_message if title.blank? @@ -87,11 +88,12 @@ class Commit # # cut off, ellipses (`&hellp;`) are prepended to the commit message. def description - title_end = safe_message.index(/\n/) + title_end = safe_message.encode("UTF-8", :invalid => :replace, :undef => :replace, :replace => "?").index(/\n/) + @description ||= if (!title_end && safe_message.length > 100) || (title_end && title_end > 100) "…".html_safe << safe_message[80..-1] else - safe_message.split(/\n/, 2)[1].try(:chomp) + safe_message.encode("UTF-8", :invalid => :replace, :undef => :replace, :replace => "?").split(/\n/, 2)[1].try(:chomp) end endWouldn't it be easier to use UTF-8 instead?
Of cause it would be easier... With a time machine :) We have many repositories in cp-1251, and I can't find a solution to reencode them without losing history. This solution http://stackoverflow.com/questions/11052199/convert-git-repository-file-encoding didn't work for me - 200mb repo was being processed for two days and did not complete...
Discussion is also here https://github.com/gitlabhq/gitlabhq/issues/5493
We have similar issues at our GitLab after upgrading to version 7.4.3. (previous version was 7.3.2, worked without any problem)
production.log details:
merge request in (utf-8 PHP codebase)
Processing by Projects::MergeRequestsController#create as HTML Parameters: {"utf8"=>"✓", "authenticity_token"=>"/ovXISAFlGYrPJ7xzcm3K3aLFwjx29INs5Nk804wrTU=", "merge_request"=>{"title"=>"Issue 30047", "description"=>"", "assignee_id"=>"5", "milestone_id"=>"", "label _ids"=>[""], "source_project_id"=>"149", "target_project_id"=>"149", "target_branch"=>"testing", "source_branch"=>"issue-30047"}, "project_id"=>"hungarocontrol/portal"} Completed 500 Internal Server Error in 408ms ArgumentError (invalid byte sequence in UTF-8): app/models/merge_request_diff.rb:106:in `reload_commits' app/models/merge_request_diff.rb:41:in `reload_content' app/services/merge_requests/create_service.rb:10:in `execute' app/controllers/projects/merge_requests_controller.rb:87:in `create'graph (same project as above)
Started GET "/hungarocontrol/portal/graphs/develop" for 127.0.0.1 at 2014-11-12 14:22:43 +0100 Processing by Projects::GraphsController#show as JSON Parameters: {"project_id"=>"hungarocontrol/portal", "id"=>"develop"} Completed 500 Internal Server Error in 249ms JSON::GeneratorError (source sequence is illegal/malformed utf-8): app/controllers/projects/graphs_controller.rb:38:in `fetch_graph' app/controllers/projects/graphs_controller.rb:11:in `block (2 levels) in show' app/controllers/projects/graphs_controller.rb:8:in `show'file list (same project as above)
Started GET "/hungarocontrol/portal" for 127.0.0.1 at 2014-11-12 14:29:48 +0100 Processing by ProjectsController#show as HTML Parameters: {"id"=>"hungarocontrol/portal"} Completed 500 Internal Server Error in 1787ms ActionView::Template::Error (source sequence is illegal/malformed utf-8): app/views/projects/network/show.json.erb:22:in `_app_views_projects_network_show_json_erb__2276122755674063837_70264692870160'GitLab CI hook (antoher project with utf-8 PHP codebase)
Started GET "/oh/bmo-portal/services/gitlab_ci/test" for 127.0.0.1 at 2014-11-12 15:30:31 +0100 Processing by Projects::ServicesController#test as HTML Parameters: {"project_id"=>"oh/bmo-portal", "id"=>"gitlab_ci"} Completed 500 Internal Server Error in 87ms JSON::GeneratorError (source sequence is illegal/malformed utf-8): app/models/hooks/web_hook.rb:35:in `execute' app/models/project_services/gitlab_ci_service.rb:20:in `execute' app/controllers/projects/services_controller.rb:29:in `test'Our environment:
System information System: Debian 6.0.10 Current User: git Using RVM: no Ruby Version: 2.1.2p95 Gem Version: 2.2.2 Bundler Version:1.6.3 Rake Version: 10.3.2 Sidekiq Version:2.17.0 GitLab information Version: 7.4.3 Revision: 353a987 Directory: /var/lib/git/gitlab DB Adapter: mysql2 URL: http://gitlab.bfsdemo.hu HTTP Clone URL: http://gitlab.bfsdemo.hu/some-project.git SSH Clone URL: git@gitlab.bfsdemo.hu:some-project.git Using LDAP: no Using Omniauth: no GitLab Shell Version: 2.0.1 Repositories: /var/lib/git/repositories/ Hooks: /var/lib/git/gitlab-shell/hooks/ Git: /usr/local/git/bin/gitThanks,
Peter
I wonder if this is related not just to problem that raised this discussion, but to workaround proposed there. We have to use DyadyaMisha's workaround to be able to work with numerous legacy repositories. May be it is this workaround that produces double-encoding somewhere?
Fix given by Aki Tuomi works perfectly except when creating a new merge request :
Started POST "/eddy/overlease/merge_requests" for 84.14.112.108 at 2014-11-28 17:54:48 +0100 Processing by Projects::MergeRequestsController#create as HTML Parameters: {"utf8"=>"✓", "authenticity_token"=>"bSKXfwz3PAChJKXwU0y04ojE2z3d4W5wcaZfrOWB1js=", "merge_request"=>{"title"=>"Dev", "description"=>"", "assignee_id"=>"", "milestone_id"=>"", "label_ids"=>[""], "source_project_id"=>"4", "target_project_id"=>"4", "target_branch"=>"recette", "source_branch"=>"dev"}, "project_id"=>"eddy/overlease"} Completed 500 Internal Server Error in 189ms ArgumentError (invalid byte sequence in UTF-8): app/models/merge_request_diff.rb:105:in `reload_commits' app/models/merge_request_diff.rb:41:in `reload_content' app/services/merge_requests/create_service.rb:10:in `execute' app/controllers/projects/merge_requests_controller.rb:87:in `create'omnibus v7.5.1
Same issue with v7.5.1 CE
Started POST "/libri/webshop/merge_requests" for 127.0.0.1 at 2014-11-28 23:01:52 +0100 Processing by Projects::MergeRequestsController#create as HTML Parameters: {"utf8"=>"✓", "authenticity_token"=>"cezypb/76Jui+FdAselYHHJKs8a2T5W6T6ZV8NJHSaQ=", "merge_request"=>{"title"=>"Test merge request", "description"=>"Test merge request", "assignee_id"=>"1", "milestone_id"=>"", "label_ids"=>[""], "source_project_id"=>"50", "target_project_id"=>"50", "target_branch"=>"master", "source_branch"=>"testing"}, "project_id"=>"libri/webshop"} Completed 500 Internal Server Error in 1249ms ArgumentError (invalid byte sequence in UTF-8): app/models/merge_request_diff.rb:106:in `reload_commits' app/models/merge_request_diff.rb:41:in `reload_content' app/services/merge_requests/create_service.rb:10:in `execute' app/controllers/projects/merge_requests_controller.rb:87:in `create'I tried Aki Tuomi's fix but it didn't help.
+1
ActionView::Template::Error (invalid byte sequence in GB2312): 26: = preserve(gfm(escape_once(commit.description))) 27: 28: .commit-row-info 29: = commit_author_link(commit, avatar: true, size: 16) 30: .committed_ago 31: #{time_ago_with_tooltip(commit.committed_date)} app/helpers/commits_helper.rb:129:in `clean' app/helpers/commits_helper.rb:90:in `commit_person_link' app/helpers/commits_helper.rb:11:in `commit_author_link' app/views/projects/commits/_commit.html.haml:29:in `_app_views_projects_commits__commit_html_haml___3588761364259489947_254967120' app/views/projects/commits/_commits.html.haml:10:in `block in _app_views_projects_commits__commits_html_haml___684384739968686411_255719420' %Q{#{avatar} <span class="commit-#{options[:source]}-name">#{person_name}</span>} app/views/projects/commits/_commits.html.haml:1:in `each' app/views/projects/commits/_commits.html.haml:1:in `_app_views_projects_commits__commits_html_haml___684384739968686411_255719420' app/views/projects/commits/show.html.haml:18:in `_app_views_projects_commits_show_html_haml___1059135273091482772_255958860' app/controllers/projects/commits_controller.rb:18:in `show'mentioned in issue #877 (closed)
mentioned in issue #863 (closed)
+1
Started GET "/some_group/some_project/commits/master" for 189.202.43.223 at 2014-12-17 12:11:01 -0500 Processing by Projects::CommitsController#show as HTML Parameters: {"project_id"=>"some_group/some_project", "id"=>"master"} Completed 500 Internal Server Error in 205ms ActionView::Template::Error (invalid byte sequence in UTF-8): 26: = preserve(gfm(escape_once(commit.description))) 27: 28: .commit-row-info 29: = commit_author_link(commit, avatar: true, size: 16) 30: .committed_ago 31: #{time_ago_with_tooltip(commit.committed_date)} app/helpers/commits_helper.rb:129:in `clean' app/helpers/commits_helper.rb:90:in `commit_person_link' app/helpers/commits_helper.rb:11:in `commit_author_link' app/views/projects/commits/_commit.html.haml:29:in `_app_views_projects_commits__commit_html_haml___146875318618028567_40865480' app/views/projects/commits/_commits.html.haml:10:in `block in _app_views_projects_commits__commits_html_haml___2752154967438722480_41055680' app/views/projects/commits/_commits.html.haml:1:in `each' app/views/projects/commits/_commits.html.haml:1:in `_app_views_projects_commits__commits_html_haml___2752154967438722480_41055680' app/views/projects/commits/show.html.haml:18:in `_app_views_projects_commits_show_html_haml___4219035452165901470_41927420' app/controllers/projects/commits_controller.rb:18:in `show'I also tried Aki Tuomi's fix but it didn't help. Currently using ubuntu-14.04/gitlab_7.5.3-omnibus.5.2.1.ci-1_amd64.deb (fresh install)
Workaround:
Following Aki Tuomi's idea to fix (Disclaimer, I have never ever done anything in ruby, so be careful.)
File: app/helpers/commits_helper.rb
def clean(string) + string = string.encode("UTF-8", :invalid => :replace, :undef => :replace, :replace => "?") Sanitize.clean(string, remove_contents: true)That fix the commit page, for the network and graph you have to add more "encode("UTF-8", :invalid => :replace, :undef => :replace, :replace => "?")" to app/views/projects/network/show.json.erb and app/controllers/projects/graphs_controller.rb
But i guess that there should be a way to filter that directly from the commit.rb, but i have no idea(neither time to learn atm) how ruby works.
@josejulio , not working on app/controllers/projects/graphs_controller.rb, any other place to put this code?
I attach what i needed to modify, If it does not work, you might want to check the logs and see where it fails, i might be able to lend you a hand.
--- app/controllers/projects/graphs_controller.rb | 2 +- app/helpers/commits_helper.rb | 1 + app/views/projects/network/show.json.erb | 4 ++-- 3 files changed, 4 insertions(+), 3 deletions(-) diff --git a/app/controllers/projects/graphs_controller.rb b/app/controllers/projects/graphs_controller.rb index 4a318cb..7d4c222 100644 --- a/app/controllers/projects/graphs_controller.rb +++ b/app/controllers/projects/graphs_controller.rb @@ -28,7 +28,7 @@ class Projects::GraphsController < Projects::ApplicationController @commits.each do |commit| @log << { - author_name: commit.author_name.force_encoding('UTF-8'), + author_name: commit.author_name.encode("UTF-8", :invalid => :replace, :undef => :replace, :replace => "?").force_encoding('UTF-8'), author_email: commit.author_email.force_encoding('UTF-8'), date: commit.committed_date.strftime("%Y-%m-%d") } diff --git a/app/helpers/commits_helper.rb b/app/helpers/commits_helper.rb index 36adead..0ee9b2d 100644 --- a/app/helpers/commits_helper.rb +++ b/app/helpers/commits_helper.rb @@ -126,6 +126,7 @@ module CommitsHelper end def clean(string) + string = string.encode("UTF-8", :invalid => :replace, :undef => :replace, :replace => "?") Sanitize.clean(string, remove_contents: true) end end diff --git a/app/views/projects/network/show.json.erb b/app/views/projects/network/show.json.erb index dc82adc..9b68723 100644 --- a/app/views/projects/network/show.json.erb +++ b/app/views/projects/network/show.json.erb @@ -7,7 +7,7 @@ { parents: parents_zip_spaces(c.parents(@graph.map), c.parent_spaces), author: { - name: c.author_name, + name: c.author_name.encode("UTF-8", :invalid => :replace, :undef => :replace, :replace => "?"), email: c.author_email, icon: avatar_icon(c.author_email, 20) }, @@ -16,7 +16,7 @@ refs: get_refs(@graph.repo, c), id: c.sha, date: c.date, - message: c.message, + message: c.message.encode("UTF-8", :invalid => :replace, :undef => :replace, :replace => "?"), } end }.to_json --Thanks for your help @josejulio! It worked perfectly: Graphs, Network and Commits Page. Continue with error File and Project Pages, follows the logs: File Page
Started GET "/dides/protocolo/tree/master" for 127.0.0.1 at 2014-12-24 10:21:37 -0400 Processing by Projects::TreeController#show as HTML Parameters: {"project_id"=>"dides/protocolo", "id"=>"master"} Completed 500 Internal Server Error in 66ms ArgumentError (invalid byte sequence in UTF-8): app/models/tree.rb:11:in `select' app/models/tree.rb:11:in `initialize' app/models/repository.rb:208:in `new' app/models/repository.rb:208:in `tree' lib/extracts_path.rb:120:in `tree' app/controllers/projects/tree_controller.rb:5:in `show'Project Page:
Started GET "/dides/protocolo" for 127.0.0.1 at 2014-12-24 10:22:26 -0400 Processing by ProjectsController#show as HTML Parameters: {"id"=>"dides/protocolo"} Completed 500 Internal Server Error in 676ms ActionView::Template::Error (invalid byte sequence in UTF-8): 1: = render "home_panel" 2: 3: - readme = @repository.readme 4: %ul.nav.nav-tabs 5: %li.active 6: = link_to '#tab-activity', 'data-toggle' => 'tab' do app/models/tree.rb:11:in `select' app/models/tree.rb:11:in `initialize' app/models/repository.rb:208:in `new' app/models/repository.rb:208:in `tree' app/models/repository.rb:181:in `block in readme' app/models/repository.rb:180:in `readme' app/views/projects/show.html.haml:3:in `_app_views_projects_show_html_haml___856554287892327550_56563820' app/controllers/projects_controller.rb:69:in `block (2 levels) in show' app/controllers/projects_controller.rb:62:in `show'Sorry for taking long to answer, i don't have that error, but it seems to be more of the same (a repository not using UTF8).
app/models/tree.rb:11:
available_readmes = @entries.select(&:readme?)I'm not really sure of the syntax, but maybe this could help:
available_readmes = @entries.select(&:readme?).encode("UTF-8", :invalid => :replace, :undef => :replace, :replace => "?")Does it work?
@josejulio Unfortunately it did not work, some other idea where have to force this encode.
I was thinking, is there any way I can convert encode of my repository to utf8?
@josejulio, your code ignores that @entries.select is an array, also the error occurs inside select.
@cmouse, any idea ? Thanks for help me :)
recoding your README to UTF8 didn't work? As @cmouse points out, my code ignores that @entries.select is an array :(.
If you know Ruby and could help me out on what the following code executes (what file) as i'm not familiar with ruby and RoR
@entries = Gitlab::Git::Tree.where(git_repo, sha, path)As for changing the encoding of the git repo you could try: http://stackoverflow.com/questions/11052199/convert-git-repository-file-encoding
mentioned in issue #863 (closed)
I've got a similar issue in 7.7, the page listing commits fails for every branches :
Completed 500 Internal Server Error in 910ms (Views: 54.1ms | ActiveRecord: 21.3ms) Started GET "/xxx-group/xxx-project/commits/develop" for 127.0.0.1 at 2015-01-23 10:21:39 +0100 Processing by Projects::CommitsController#show as HTML Parameters: {"project_id"=>"xxx-group/xxx-project", "id"=>"develop"} Encoding::CompatibilityError (incompatible character encodings: ASCII-8BIT and UTF-8): app/views/projects/commits/show.html.haml:19:in `_app_views_projects_commits_show_html_haml__467970287784325918_70217518263740' app/controllers/projects/commits_controller.rb:18:in `show' Completed 500 Internal Server Error in 745ms (Views: 53.4ms | ActiveRecord: 19.0ms)I've tried everything that you guys listed here and it doesn't work. As I don't know anything on Ruby, nor RoR, I don't know where to look to find what's causing this 500. I can't find neither what's the commit doing this.
Every other pages works fines except the one listing commit.
Any idea ?
Hi all, when will be implement changes @josejulio purposed in to master branch ? It helps me to.
thank you for the work @DouweM
I got same issue even upgraded to v7.9.1 6f6c5f18 . Got one commit having this encoding issue , within the commit it includes ^M character . Is this be considering in the fix ?
@yikaus Could you point me to an actual example of a failing commit?
Hi Douwe ,
I got really big diff list , I can't show it here , the commit include some binary char in some key file as well as windows line break char ^M , but when I test a small commit with ^M char, it looks fine , when I test commit with below binary char ,it shows No preview for this file type , whcih is good .
diff --git a/dev/somekey
index ebb4576..b87dc93 100644
--- a/dev/somekey
+++ b/dev/somekey
@@ -1 +1 @@
-Íò= Ì9>¶_l ž € 6XM˜ Ú •ä–Ñ[ G9
\ No newline at end of file
+ ièý»jÜå$Xoc•˜&ˆàE#ǽ©| FØAFÔp ¨
\ No newline at end of file
I can't reproduce the issue with new commit in 6.9.1 , but as I migrate from 6.3.2 , after updated to 6.9.1 and try to view the previous commit it shows 500 encoding error .
@yikaus Could you let me know just the commit message?
@DouweM what commit message your after ? git log COMMITid ?
@yikaus Yep, the output of
git log -n 1 <commit id>commit 70d477f48980dc9c37f359bde1f095999d8f6801
Author: Kevin Yi kyi@gitlab.local
Date: Wed Mar 4 23:31:26 2015 +0000
auto check in@yikaus Are you sure the error message you're seeing is the same as in the issue description? Down to the
app/models/commit.rb:76:in 'title'line in the stack trace?Hi,
I have the same issue (GitLab 7.8.3).
Started GET "/project/test/7.0" for XXX.XXX.XXX.XXX at 2015-04-02 10:27:43 +0200 Processing by Projects::CommitsController#show as HTML Parameters: {"project_id"=>"project", "id"=>"test/7.0"} Completed 500 Internal Server Error in 378ms ActionView::Template::Error (invalid byte sequence in UTF-8): 3: = link_to commit.short_id, project_commit_path(project, commit), class: "commit_short_id" 4: 5: %span.str-truncated 6: = link_to_gfm commit.title, project_commit_path(project, commit.id), class: "commit-row-message" 7: - if commit.description? 8: %a.text-expander.js-toggle-button ... 9: app/models/commit.rb:76:in `title' app/views/projects/commits/_commit.html.haml:6:in `_app_views_projects_commits__commit_html_haml___3271489061459771172_23609386435160' app/views/projects/commits/_commits.html.haml:13:in `block in _app_views_projects_commits__commits_html_haml__64691270716086086_23609386314400' app/views/projects/commits/_commits.html.haml:4:in `each' app/views/projects/commits/_commits.html.haml:4:in `_app_views_projects_commits__commits_html_haml__64691270716086086_23609386314400' app/views/projects/commits/show.html.haml:16:in `_app_views_projects_commits_show_html_haml__812491708747936132_23609385804020' app/controllers/projects/commits_controller.rb:18:in `show'I cannot share this project...
@fabien-archambault Most encoding issues were fixed in 7.9. Try upgrading and see if the issue persists.
mentioned in issue #2402 (closed)
