GitLab Geo: Wiki Sync

Rationale

In GitLab, every Wiki, is also a git project. As we don't store it in a database, it's not being replicated.

We should be able to replicate it in Geo (#76) as we do with any git project.

Replication

Primary Geo node should notify secondary nodes, in a similar way we do for projects, when Wiki is changed. On a secondary node, it should git fetch Wiki's git repository and update.

We can reuse the same pooler we use for projects but a different notify endpoint to make code simpler.

Checklist

• When a wiki is updated by git on secondary node, we must enqueue and buffer this action in an specific redis queue
• When a wiki is updated by web UI on secondary node, we must enqueue and buffer this action in an specific redis queue
• When we pool for updated projects we must also query for updated Wikis
• A secondary node must receive in a different endpoint bulk notifications of updated Wikis
• When a updated wiki notification is received, it must generated async jobs for every wiki, to be updated

Added ~164929 label

I'm trying hard not to refactor anything in app/workers/post_receive.rb right now, but thinking it could be a good for a near future.

The problem and part of the solution is already implemented by @vsizov for the ElasticSearch indexing:

  def update_wiki_es_indexes(repo_path)
    return unless repo_path =~ /wiki\z/ && Gitlab.config.elasticsearch.enabled

    project = Project.find_with_namespace(repo_path.gsub(/\.wiki\z/, ""))

    if project
      project.wiki.index_blobs
    end
  end

This will also be beneficial to https://gitlab.com/gitlab-org/gitlab-ce/issues/563#note_4260208

mentioned in merge request !267 (merged)

Ok, I had to refactor post_receive, after !267 (merged) get reviewed/merged I will try to backport this changes to CE. It will happen only for 8.7.

Marked the task When a wiki is updated by git on secondary node, we must enqueue and buffer this action in an specific redis queue as completed

Marked the task When we pool for updated projects we must also query for updated Wikis as completed

Marked the task A secondary node must receive in a different endpoint bulk notifications of updated Wikis as completed

Marked the task A secondary node must receive in a different endpoint bulk notifications of updated Wikis as incomplete

Marked the task A secondary node must receive in a different endpoint bulk notifications of updated Wikis as completed

Marked the task When a updated wiki notification is received, it must generated async jobs for every wiki, to be updated as completed

Marked the task When a wiki is updated by web UI on secondary node, we must enqueue and buffer this action in an specific redis queue as completed

Status changed to closed by merge request !267 (merged)

mentioned in commit c0337aa7

mentioned in commit 20a3715e

mentioned in issue #76

Mentioned in commit pfjason/gitlab-ce@69e966d9

Mentioned in commit pfjason/gitlab-ce@c0337aa7

Mentioned in issue #76