Replace current Wiki system with Gollum Wikis.
Created by: DanKnox
This commit replaces the old database backed Wiki system with the excellent Gollum git based Wiki system.
The UI has been updated to allow for utilizing the extra features that Gollum provides. Specifically:
- Edit page now allows you to choose the content format.
- Edit page allows you to provide a commit message for the change.
- History page now shows Format, Commit Message, and Commit Hash.
- A new Git Access page has been added with the Wiki Repo URL.
- The default page has been changed to Home from Index to match the Gollum standard.
I did my best to keep the new code consistent with the rest of the repository and follow the conventions that have already been established.
The old Wiki model has been left in tact to provide for the development of a migration script that will move all content stored in the old Wiki system into new Gollum Wikis.
The entire test suite for the project is currently 100% passing on my local dev environment with the most recent master branch merged in as of today.
My Reason For Submitting This
The company I work for is planning on cutting over to Gitlab fully next week. As we are currently developing many internal services that require extensive API documentation, having a robust and flexible Wiki system attached to each service repository is an absolute must for us. Gitlab is an amazing application that continues to get better each day and I truly hope you accept this new feature.
This will also allow users to transfer Wikis out of Github and into Gitlab by just cloning and pushing to the new remote.
Note About Migrating
I understand that for you to include this feature into the next release, a migration Rake task will need to be developed to migrate all of the existing content that is stored in the current database backed Wiki system. I am more than happy to develop this if you are happy with the implementation that has been developed thus far and wish to accept this change into master.
I am submitting this now without the migration task because I wanted to get your feedback and make sure you will actually accept this feature before putting too much more effort into it.
Preview of UI Changes
- Wiki Home Page
- Edit Page
- "All Pages" Page
- Page History Page
- Git Access Page
Merge request reports
Activity
Created by: DanKnox
Whoops looks like I forgot to update the Spinach steps to create the wiki repository in the
tmp/test-git-base-pathdirectory. Everything else seems to have passed. I will add one more commit to correct this later tonight or tomorrow. I don't have very much experience with Spinach.By Administrator on 2013-03-10T01:52:28 (imported from GitLab project)
By Administrator on 2013-03-10T01:52:28 (imported from GitLab)
Created by: DanKnox
That is correct. The Wiki is stored in a separate git repo than the main project repository. Currently, the first time you access the Wiki section of your project, it creates the Wiki repo for you automatically.
The Wiki repo is named
namespace/project_name.wiki.git. This is consistent with the naming scheme used by Github.By Administrator on 2013-03-10T06:57:50 (imported from GitLab project)
By Administrator on 2013-03-10T06:57:50 (imported from GitLab)
Created by: hswong3i
Since I am coming from Gitorious to Gitlab, during studying Gitorious's GIT bare folders I always find a special repository that for storing Wiki information.
For example: http://gitorious.org/gitorious/pages/git_access, stored as
git://gitorious.org/gitorious/gitorious-gitorious-wiki.gitI am not sure if Gitorious also using Gollum (https://github.com/github/gollum) or not because they just support Markdown syntax; but using GIT-based Wiki for managing a GIT hosting platform's Wiki shouldn't be a new idea and so, should also be a good idea, too ;-)
P.S. in case of Gitorious they name the Wiki GIT bare as
namespace/namespaces-project-wiki.git, as your reference (and sure, we should also take Gitlab's internal GIT bare naming standard as reference, too)By Administrator on 2013-03-10T07:12:47 (imported from GitLab project)
By Administrator on 2013-03-10T07:12:47 (imported from GitLab)
Created by: DanKnox
This doesn't add any extra complexity. If you don't want to clone the Wiki git repo you can use the Wiki system exactly like you do currently. There will be no difference for you whatsoever. For those that do wish to clone the Wiki and edit it locally, they have that option.
For what it's worth, this happens to be the number one requested feature currently so I think you may be in the minority here.
http://gitlab.uservoice.com/forums/176466-general
Although, as mentioned before, this does not add any extra complexity whatsoever. You can continue using the Wiki exactly like you did before without doing anything special. What it does do is add the option to clone the Wiki and edit it locally for those that wish to. It's the best of both worlds.
By Administrator on 2013-03-10T07:25:59 (imported from GitLab project)
By Administrator on 2013-03-10T07:25:59 (imported from GitLab)
Created by: dzaporozhets
@DanKnox thank you for PR( and proper description ). Also I think we will accept this PR into GitLab.
Before I do a code review and we merge this one next conditions should be resolved:
- Working tests
- A bit of Spinach tests to cover wiki functionality
- Migration for existing wiki w/o any data loss
thank you for contribution
By Administrator on 2013-03-10T08:56:43 (imported from GitLab project)
By Administrator on 2013-03-10T08:56:43 (imported from GitLab)
Created by: lyda
I updated the feature request with a link to this. I'm not a ruby dev, so really can't help with this, but perhaps someone following the feature request might be.
This is a really great contribution. Thanks.
By Administrator on 2013-03-10T09:16:41 (imported from GitLab project)
By Administrator on 2013-03-10T09:16:41 (imported from GitLab)
Created by: coveralls
Changes Unknown when pulling 1479f172 on DanKnox:use_gollum_wikis into * on gitlabhq:master*.
By Administrator on 2013-03-10T22:19:11 (imported from GitLab project)
By Administrator on 2013-03-10T22:19:11 (imported from GitLab)
Created by: coveralls
Changes Unknown when pulling f0aa54e0 on DanKnox:use_gollum_wikis into * on gitlabhq:master*.
By Administrator on 2013-03-11T02:29:01 (imported from GitLab project)
By Administrator on 2013-03-11T02:29:01 (imported from GitLab)
Created by: DanKnox
@randx: I believe I have now added everything you asked for in your last comment.
- The tests are now completely passing.
- I added more Spinach steps for the Wiki Feature to test every section of the Wiki UI.
- I added a migration task to move all of the existing Wiki data stored in the database into new Git repositories. The task is named
gitlab:wiki:migrate. I added test coverage for it as well. I also executed it on my environment and it worked flawlessly. The task does not remove any of the pages from the database so it can be executed again if something were to go wrong. There will likely need to be a cleanup task to delete thewikistable from the database and delete the oldWikimodel from the app. I figured I would leave it up to you to decide how and when would be best to handle the cleanup.
Note about the drop in Coverage level
I admittedly could have added more spec coverage for the migration class. However the migration tool will only be used once and I believe the tests that I did write provide enough coverage to be sure the migration class works correctly.
I guess I will leave it up to you decide when you perform your code review.
Thanks for considering this pull request. I look forward to hearing your feedback.
By Administrator on 2013-03-11T02:57:08 (imported from GitLab project)
By Administrator on 2013-03-11T02:57:08 (imported from GitLab)
Created by: DanKnox
Please ignore the last comment about the coverage level
Thanks again for considering the pull request.
By Administrator on 2013-03-11T03:02:38 (imported from GitLab project)
By Administrator on 2013-03-11T03:02:38 (imported from GitLab)
Created by: n0v1
Good job! Thank you @DanKnox
By Administrator on 2013-03-11T08:53:48 (imported from GitLab project)
By Administrator on 2013-03-11T08:53:48 (imported from GitLab)
Created by: dzaporozhets
@DanKnox ok thank you. I'll made a review in few days and give you some feedback
By Administrator on 2013-03-11T13:15:40 (imported from GitLab project)
By Administrator on 2013-03-11T13:15:40 (imported from GitLab)
8 8 9 9 .ui-box.ui-box-show 10 10 .ui-box-head 11 = f.label :title 12 .input= f.text_field :title, class: 'span8' 13 = f.hidden_field :slug 11 %h3.page_title 12 .edit-wiki-header 13 = @wiki.title.titleize 14 = f.hidden_field :title, value: @wiki.title 15 = f.select :format, options_for_select(GollumWiki::MARKUPS, {selected: @wiki.format}), {}, class: "pull-right input-medium" Created by: dzaporozhets
does we support all this formates? It requires a lot of dependencies according to https://github.com/github/gollum#installation So I recommend to leave Markdown only
By Administrator on 2013-03-14T07:07:54 (imported from GitLab project)
By Administrator on 2013-03-14T07:07:54 (imported from GitLab)
8 8 9 9 .ui-box.ui-box-show 10 10 .ui-box-head 11 = f.label :title 12 .input= f.text_field :title, class: 'span8' 13 = f.hidden_field :slug 11 %h3.page_title 12 .edit-wiki-header 13 = @wiki.title.titleize 14 = f.hidden_field :title, value: @wiki.title 15 = f.select :format, options_for_select(GollumWiki::MARKUPS, {selected: @wiki.format}), {}, class: "pull-right input-medium" Created by: DanKnox
I agree with you that it would be best to keep the amount of required dependencies minimal. However, I also like the idea of supporting multiple formats. I myself only use Markdown for the majority of my pages but I saw at least one user posting in a past issue that was asking for RDoc support.
I also envision situations where users clone the wiki and edit it locally with a different format other than markdown then run into problems once they push it back up and attempt to view it.
Maybe we could make this configurable and add a guide for enabling whichever formats you prefer. Markdown could be the only one enabled by default, but more formats could be enabled through the
gitlab.ymlconfiguration file. There could also be a note next to the different formats specifying which gem must be installed on the system when you enable it.There could be a check when the page loads to verify that the particular format of the page is enabled in the gitlab config file. If not, it could redirect the user to instructions on how to enable their preferred brand of documentation format.
By Administrator on 2013-03-14T07:07:54 (imported from GitLab project)
By Administrator on 2013-03-14T07:07:54 (imported from GitLab)
8 8 9 9 .ui-box.ui-box-show 10 10 .ui-box-head 11 = f.label :title 12 .input= f.text_field :title, class: 'span8' 13 = f.hidden_field :slug 11 %h3.page_title 12 .edit-wiki-header 13 = @wiki.title.titleize 14 = f.hidden_field :title, value: @wiki.title 15 = f.select :format, options_for_select(GollumWiki::MARKUPS, {selected: @wiki.format}), {}, class: "pull-right input-medium" Created by: brodock
I don't think that adding a configuration is a good idea. Can't we add support for some of the most popular formats?
For consideration:
- Markdown
- Github Flavored (you already support it in README)
- RDoc (also, please support it on README files)
- Textile (same vote for README files)
By Administrator on 2013-03-14T07:07:54 (imported from GitLab project)
By Administrator on 2013-03-14T07:07:54 (imported from GitLab)
8 8 9 9 .ui-box.ui-box-show 10 10 .ui-box-head 11 = f.label :title 12 .input= f.text_field :title, class: 'span8' 13 = f.hidden_field :slug 11 %h3.page_title 12 .edit-wiki-header 13 = @wiki.title.titleize 14 = f.hidden_field :title, value: @wiki.title 15 = f.select :format, options_for_select(GollumWiki::MARKUPS, {selected: @wiki.format}), {}, class: "pull-right input-medium" Created by: dzaporozhets
I'd like to see next options:
- We use GitLab flavored Markdown ( see https://github.com/gitlabhq/gitlabhq/blob/master/lib/gitlab/markdown.rb ) for notes, issues, README etc. It will be cool to use it.
- Also maybe native Markdown and RDoc I guess.
By Administrator on 2013-03-14T07:07:54 (imported from GitLab project)
By Administrator on 2013-03-14T07:07:54 (imported from GitLab)
8 8 9 9 .ui-box.ui-box-show 10 10 .ui-box-head 11 = f.label :title 12 .input= f.text_field :title, class: 'span8' 13 = f.hidden_field :slug 11 %h3.page_title 12 .edit-wiki-header 13 = @wiki.title.titleize 14 = f.hidden_field :title, value: @wiki.title 15 = f.select :format, options_for_select(GollumWiki::MARKUPS, {selected: @wiki.format}), {}, class: "pull-right input-medium" Created by: DanKnox
Gitlab Flavored Markdown
I played around with a few methods of utilizing the Gitlab flavored Markdown. My first approach was to write a wrapper class for it and register it inside of Gollum as the default renderer for the *.gfm extension. Afterwards I realized that was a bad idea since you would end up with .gfm files inside the Gollum repo that would not render properly when using Gollum to edit locally.
I settled on this quite basic solution:
# Inside GitlabMarkdownHelper def render_wiki_content(wiki_page) if wiki_page.format == :markdown markdown(wiki_page.content) # Use Gitlab Markdown for markdown files else wiki_page.formatted_content.html_safe # Use Gollum to render the rest end endThen on line 13 of
app/views/wikis/show.html.haml, just call that method instead..file_holder .file_content.wiki = preserve do = render_wiki_content(@wiki)I think this is the cleanest approach. All of the special Gitlab Markdown features work (@user links, issue # links, etc.) when you view the page in Gitlab and the page still renders correctly in Markdown when views from within Gollum.
If you are happy with this approach, I will push one more commit with this change.
Supported Formats
I personally don't see any problem with giving the user the option to choose which formats they want available within their config files. They can install the dependencies they need for the formats they wish to use.
However, this really isn't a big issue for me. I use Markdown for 99% of my projects and RDoc for the other 1%. I think that the list suggested by @brodock probably seems like the most reasonable.
I do think there isn't much reason to support native Markdown, as Gitlab/Github markdown has become standard most everywhere and the original markdown implementation was quite inferior in my opinion.
Bottom line, I could go either way. Let me know what formats you want supported and I will push a commit limiting it to just those.
By Administrator on 2013-03-14T07:07:54 (imported from GitLab project)
By Administrator on 2013-03-14T07:07:54 (imported from GitLab)
8 8 9 9 .ui-box.ui-box-show 10 10 .ui-box-head 11 = f.label :title 12 .input= f.text_field :title, class: 'span8' 13 = f.hidden_field :slug 11 %h3.page_title 12 .edit-wiki-header 13 = @wiki.title.titleize 14 = f.hidden_field :title, value: @wiki.title 15 = f.select :format, options_for_select(GollumWiki::MARKUPS, {selected: @wiki.format}), {}, class: "pull-right input-medium" Created by: dzaporozhets
@DanKnox I'm absolutely ok with
render_wiki_contentAbout formats: Lets use only 2 for now - RDoc and GitLab Makrdown.
By Administrator on 2013-03-14T07:07:54 (imported from GitLab project)
By Administrator on 2013-03-14T07:07:54 (imported from GitLab)
8 8 9 9 .ui-box.ui-box-show 10 10 .ui-box-head 11 = f.label :title 12 .input= f.text_field :title, class: 'span8' 13 = f.hidden_field :slug 11 %h3.page_title 12 .edit-wiki-header 13 = @wiki.title.titleize 14 = f.hidden_field :title, value: @wiki.title 15 = f.select :format, options_for_select(GollumWiki::MARKUPS, {selected: @wiki.format}), {}, class: "pull-right input-medium" Created by: DanKnox
Alright the
#render_wiki_contentmethod has been implemented with specs to cover it. I also limited the available formats to the two you requested.I believe this about covers everything you have requested so far. Let me know if you find anything else that needs to be modified.
<rant>For what it is worth, I disagree with the decision to limit the available Wiki formats. I do completely agree with your concern about the amount of dependencies, and I do think supporting only Markdown and RDoc makes sense for the default configuration.However, I think not giving users the option to enable extra formats and install those dependencies themselves is just inviting trouble. I think it is very likely that users will clone the repo locally. Add a few files in a markup that is not supported by Gitlab. Then push the repository back up to the server and have everything blow up when they attempt to view that page.
I also don't quite understand why you would want to arbitrarily limit the selection when Gollum can handle so many more formats. I am more than happy to abide by your decision though. I personally only ever use those two formats myself so it will never cause any problems for me.
I am also assuming that after having dealt with the massive amount of support issues that you have been dealing with over the past few months, you likely have a very good reason for making this call that I just don't see. Please don't interpret this as me claiming to know better than you, I am merely stating that I don't quite understand the reasoning behind this.
</rant>Hopefully you are getting close to being satisfied with everything that has been pushed so far and close to merging this in. Again, let me know if anything else comes up that you wish to be modified.
By Administrator on 2013-03-14T07:07:54 (imported from GitLab project)
By Administrator on 2013-03-14T07:07:54 (imported from GitLab)
Created by: coveralls
Changes Unknown when pulling 6d254844 on DanKnox:use_gollum_wikis into * on gitlabhq:master*.
By Administrator on 2013-03-14T06:55:52 (imported from GitLab project)
By Administrator on 2013-03-14T06:55:52 (imported from GitLab)
Created by: coveralls
Changes Unknown when pulling d69a37e0 on DanKnox:use_gollum_wikis into * on gitlabhq:master*.
By Administrator on 2013-03-14T07:22:10 (imported from GitLab project)
By Administrator on 2013-03-14T07:22:10 (imported from GitLab)
Created by: dzaporozhets
@DanKnox ok thank you. I will try to merge it soon
By Administrator on 2013-03-15T12:25:18 (imported from GitLab project)
By Administrator on 2013-03-15T12:25:18 (imported from GitLab)
Created by: dzaporozhets
@DanKnox merged. Thank you again
By Administrator on 2013-03-15T15:22:03 (imported from GitLab project)
By Administrator on 2013-03-15T15:22:03 (imported from GitLab)
Created by: rtripault
Did the Gollum repo just moved to git://github.com/gollum/gollum.git ? ;-)
By Administrator on 2013-03-15T15:43:57 (imported from GitLab project)
By Administrator on 2013-03-15T15:43:57 (imported from GitLab)
Created by: rtripault
https://github.com/github/gollum (as written in the Gemfile)
By the way @DanKnox much thanks for your work
By Administrator on 2013-03-15T15:47:46 (imported from GitLab project)
By Administrator on 2013-03-15T15:47:46 (imported from GitLab)
Created by: raphendyr
It seems to have moved: https://github.com/github/gollum/commit/b2b395e78bb8bd069aadc465a3d7766182ca54ac
By Administrator on 2013-03-15T15:49:23 (imported from GitLab project)
By Administrator on 2013-03-15T15:49:23 (imported from GitLab)
Created by: DanKnox
I just opened pull request #3230 that updates the location in the Gemfile. I will be happy when they put out a new release so we don't have to point to their Github repo anymore.
By Administrator on 2013-03-15T16:10:02 (imported from GitLab project)
By Administrator on 2013-03-15T16:10:02 (imported from GitLab)
Created by: hiroponz
I think that there are some critical bugs(#3312 (closed) #3338 (closed) #3339 (closed)).
By Administrator on 2013-03-24T14:21:11 (imported from GitLab project)
By Administrator on 2013-03-24T14:21:11 (imported from GitLab)
Created by: krymen
@hiroponz I agree. Imho the functionality must be reverted ASAP as it has not been tested enough.
By Administrator on 2013-03-24T15:12:51 (imported from GitLab project)
By Administrator on 2013-03-24T15:12:51 (imported from GitLab)
Created by: jozefvaclavik
@hiroponz and @krymen I wouldn't call #3338 (closed) and #3339 (closed) critical bugs. They just seems to be "forgotten features".
And to express my opinion, I support gollum wiki.
By Administrator on 2013-03-24T16:55:39 (imported from GitLab project)
By Administrator on 2013-03-24T16:55:39 (imported from GitLab)
Created by: krymen
@jozefvaclavik And what about #3312 (closed)? Supporting UTF-8 is imho critical.
@bbodenmiller Sounds reasonably. I didn't know -- I'm not ruby guy;)
By Administrator on 2013-03-24T18:26:43 (imported from GitLab project)
By Administrator on 2013-03-24T18:26:43 (imported from GitLab)
Created by: jozefvaclavik
@krymen I can't reproduce the bug. But I have no problem using UTF8 characters in gollum wiki. Mostly, if it would be such a huge problem, github wouldn't be using golllum for wiki as well.
incompatible character encodings: UTF-8 and ASCII-8BITusually means that someone forgot to put# encoding: UTF-8on the top of a ruby script. This error is raised if script is trying to process UTF8 characters.more info here: http://stackoverflow.com/a/6916663
if you have time, try to put that on a first line of these files:
- app/models/gollum_wiki.rb
- app/models/wiki_page.rb
- lib/tasks/gitlab/migrate_wiki.rake
- lib/wiki_to_gollum_migrator.rb
@hiroponz you were right, they are critical and should be fixed asap, but their fix should be just few lines of code. Using word critical cause more panic than good.
By Administrator on 2013-03-24T20:01:47 (imported from GitLab project)
By Administrator on 2013-03-24T20:01:47 (imported from GitLab)
Created by: diablo2man
Same problem in migration but different error message:
First run:
Migrating Wiki for 'risk' Initialized empty Git repository in /home/git/repositories/risk.wiki.git/ Creating page 'How to add extra swap space?'... Created page 'How to add extra swap space?' [32m[OK][0m Creating revisions... rake aborted! undefined method `update' for nil:NilClass /home/git/gitlab/lib/wiki_to_gollum_migrator.rb:93:in `block in create_revisions' /home/git/gitlab/lib/wiki_to_gollum_migrator.rb:82:in `each' /home/git/gitlab/lib/wiki_to_gollum_migrator.rb:82:in `create_revisions' /home/git/gitlab/lib/wiki_to_gollum_migrator.rb:74:in `create_page_and_revisions' /home/git/gitlab/lib/wiki_to_gollum_migrator.rb:50:in `block in create_pages' /home/git/gitlab/lib/wiki_to_gollum_migrator.rb:49:in `each' /home/git/gitlab/lib/wiki_to_gollum_migrator.rb:49:in `create_pages' /home/git/gitlab/lib/wiki_to_gollum_migrator.rb:17:in `block in migrate!' /home/git/gitlab/lib/wiki_to_gollum_migrator.rb:14:in `each' /home/git/gitlab/lib/wiki_to_gollum_migrator.rb:14:in `migrate!' /home/git/gitlab/lib/tasks/gitlab/migrate_wiki.rake:17:in `block (3 levels) in <top (required)>' Tasks: TOP => gitlab:wiki:migratesecond run
Migrating Wiki for 'risk' Creating page 'How to add extra swap space?'... Failed to create page 'How-to-add-extra-swap-space?' [31m[FAILED][0m Creating page 'Adding/Deleting Overrides'... rake aborted! undefined method `escaped_url_path' for nil:NilClass /home/git/gitlab/app/models/wiki_page.rb:160:in `set_attributes' /home/git/gitlab/app/models/wiki_page.rb:171:in `save' /home/git/gitlab/app/models/wiki_page.rb:127:in `create' /home/git/gitlab/lib/wiki_to_gollum_migrator.rb:69:in `create_page_and_revisions' /home/git/gitlab/lib/wiki_to_gollum_migrator.rb:50:in `block in create_pages' /home/git/gitlab/lib/wiki_to_gollum_migrator.rb:49:in `each' /home/git/gitlab/lib/wiki_to_gollum_migrator.rb:49:in `create_pages' /home/git/gitlab/lib/wiki_to_gollum_migrator.rb:17:in `block in migrate!' /home/git/gitlab/lib/wiki_to_gollum_migrator.rb:14:in `each' /home/git/gitlab/lib/wiki_to_gollum_migrator.rb:14:in `migrate!' /home/git/gitlab/lib/tasks/gitlab/migrate_wiki.rake:17:in `block (3 levels) in <top (required)>' Tasks: TOP => gitlab:wiki:migrateReally appreciate all the works. Could anyone try to figure this out? I have 0 knowledge in ruby/rail development
By Administrator on 2013-03-26T19:31:19 (imported from GitLab project)
By Administrator on 2013-03-26T19:31:19 (imported from GitLab)
