Bundle Mattermost with GitLab Omnibus packages

Internal issue on https://dev.gitlab.org/gitlab/organization/issues/254

Gitlab OAuth and Postgres features are in master https://github.com/mattermost/platform

Reassigned to @marin

I've started working on this by installing Mattermost from source.

Since there is no documentation other than setting up using Docker I've used docker file as a guide.

I've managed to install from source and connect to omnibus-gitlab bundled database.

First issue I ran into when running Mattermost was failed start after restart so I've created an issue on their public issue tracker .

Next step is to compile the app manually under gitlab paths.

Mattermost is now built in into the package but there are still a few obstacles to tackle:

At the moment I am starting the app manually by calling the binary I've created using go build mattermost.go
For reasons I still do not understand, once I start the app I don't see any controls(buttons, inputs and so on)

TODOS:

Create control commands(start, stop etc.)
config.json has hardcoded values so they need to be configurable

For reasons I still do not understand, once I start the app I don't see any controls(buttons, inputs and so on)

At first I suspected this was because .js files weren't properly built. After upgrading nodejs to the latest version and installing required development libraries plus bit of code I've managed to created required .js files.

I am going back to the production ENV of the app I have setup to try and observe the differences.

Manually installed production:

[07/31/15 12:58:08] [DEBG] Initializing user api routes
[07/31/15 12:58:08] [DEBG] Initializing team api routes
[07/31/15 12:58:08] [DEBG] Initializing channel api routes
[07/31/15 12:58:08] [DEBG] Initializing post api routes
[07/31/15 12:58:08] [DEBG] Initializing web socket api routes
[07/31/15 12:58:08] [DEBG] Initializing file api routes
[07/31/15 12:58:08] [DEBG] Initializing command api routes
[07/31/15 12:58:08] [DEBG] Parsing server templates at /go/src/github.com/mattermost/platform/api/templates/
[07/31/15 12:58:08] [DEBG] Initializing web routes
[07/31/15 12:58:08] [DEBG] Using static directory at /go/src/github.com/mattermost/platform/web/static/
[07/31/15 12:58:08] [DEBG] Parsing templates at /go/src/github.com/mattermost/platform/web/templates/
[07/31/15 12:58:08] [INFO] Starting Server...
[07/31/15 12:58:08] [INFO] Server is listening on 8065
[07/31/15 12:58:12] [DEBG] /
[07/31/15 12:58:13] [DEBG] /api/v1/users/me

Installation from package:

[07/31/15 12:58:28] [DEBG] Initializing user api routes
[07/31/15 12:58:28] [DEBG] Initializing team api routes
[07/31/15 12:58:28] [DEBG] Initializing channel api routes
[07/31/15 12:58:28] [DEBG] Initializing post api routes
[07/31/15 12:58:28] [DEBG] Initializing web socket api routes
[07/31/15 12:58:28] [DEBG] Initializing file api routes
[07/31/15 12:58:28] [DEBG] Initializing command api routes
[07/31/15 12:58:28] [DEBG] Parsing server templates at /opt/gitlab/embedded/service/mattermost/api/templates/
[07/31/15 12:58:28] [DEBG] Initializing web routes
[07/31/15 12:58:28] [DEBG] Using static directory at /opt/gitlab/embedded/service/mattermost/web/static/
[07/31/15 12:58:28] [DEBG] Parsing templates at /opt/gitlab/embedded/service/mattermost/web/templates/
[07/31/15 12:58:28] [INFO] Starting Server...
[07/31/15 12:58:28] [INFO] Server is listening on 8065
[07/31/15 12:58:35] [DEBG] /

For some reason /api/v1/users/me is not loaded, permission issue? => Changing permissions didn't make any difference.

Need to check how this address is loaded by mattermost.

Ah, server is not finding the js files, seems like it is looking for them at the wrong place.

@marin Have you tried using the makefile? There is make run if you just want to get it going. There is also a make dist which gives you a mattermost.tar.gz which you can then run by calling ./platform from the bin directory. You can pass it the appropriate configuration file by specifying -config=../your_config_file.json. So: ./platform -config=../your_config.json

@christopher1 I have, I used that when I installed the production server manually.

I've already built the binary using go build mattermost.go and I start it up by using bin/mattermost -config config.json. The app starts but js files are not being loaded just yet for some reason I am trying to discover.

@marin If you running from the binary, have you done all the steps from dist in the makefile?

I haven't done these steps and that is where the issue lies. bundle.js has been minified and it is now bundle.min.js. Having to change change the name of this on every build will be extremely error prone. Is there any way to try loading both bundle(.min).js file in the code directly so we can skip sed commands?

I can work around this by renaming the minified files during the build.

The versioning of the bundle files is to force browsers to update the file when the code on the server changes. It will work to rename them, but you will break that functionality. What is the problem with running the sed commands?

The versioning of the bundle files is to force browsers to update the file when the code on the server changes.

I understand, Rails has this out of the box.

What is the problem with running the sed commands?

If you load bundle.js in some other file then you will need to sed that one and it then grows with app growth. When sed fails during build(usually running against a deadline), hunting why it went wrong there is usually a frustrating experience.

This line has file named build.txt . Is it possible to append the build number read from that file to bundle.js and try to read that file? For example, in rails I would do something like this:

version = File.exist?("build.txt") ? File.read("build.txt").chomp : nil #this would produce string with build number or nil

# and then I could do something like this
<script src="/static/js/bundle#{version}.js"></script>

I would then be able to just generate/read from build.txt to supply the correct .js file. And if no build.txt is supplied I would default back to bundle.js.

Hi @marin, I'm wondering if instead of building Mattermost, it would be easier to use compiled bits,

We have RC1 available here: https://github.com/mattermost/platform/releases/tag/v0.6.0-rc1

RC2 is coming out on Monday. Sorry, we haven't yet documented our release process, or announced that these are coming. Hopefully, it'll be easier with them?

Hi @it33, thanks for the reply.

I suspect this would have been helpful if I knew about it earlier but I already made a lot of progress with packaging. I've uploaded this package to give you an example of where I am at the moment. There is still a lot of work to do like connecting to redis, setting up configuration options and so on, but I can already boot up Mattermost on the instance. If you are interested you can do it by:

# download and install the package(Ubuntu 12.04) 
wget https://s3-eu-west-1.amazonaws.com/omnibus-builds/ubuntu-12.04/gitlab-ce_7.13.1%2B20150731185059.git.51.9047111-ce.0_amd64.deb
dpkg -i gitlab-ce_7.13.1+20150731185059.git.51.9047111-ce.0_amd64.deb
# reconfigure the package
sudo gitlab-ctl reconfigure
# login as the mattermost user
sudo -u mattermost -i bash
# navigate to the mattermost directory
cd /opt/gitlab/embedded/service/mattermost/
# start it
/opt/gitlab/embedded/bin/mattermost -config /var/opt/gitlab/mattermost/config.json

You should be able to see Mattermost at IPADDRESS:8065

Like I said above, there is still work to do but by building mattermost from scratch I've managed to gain some knowledge about the app itself. I am going for a vacation but once I get back, few dedicated days behind the screen and we should have this ready for beta :)

Awesome! Thanks @marin, hope you have a fantastic vacation, we're excited to have Gitlab Mattermost up and running after you're back. Safe travels,

Milestone changed to 7.14

Short update: I've picked up where I left off and currently adding settings to populate config.json. I've also added controls to stop/start/restart mattermost via the command line and rewrote the build to use prebuilt binary provided by Mattermost.

For now I am just adding the configuration options from config.json without trying to understand what they do. At some point this will need to be documented.

@marin and thanks for the update

@it33 @christopher1 Can either of you provide some details on the SSO integration? I need to know what parameters are expected for AuthEndpoint, TokenEndpoint, UserApiEndpoint in config.json (just example without too much details is enough for now). I can dive into the code to figure it out but I figured I could use a shortcut here :)

@marin Assuming you had a GitLab instance running at http://localhost:3000 then you would use the following values:

AuthEndpoint: http://localhost:3000/oauth/authorize
TokenEndpoint: http://localhost:3000/oauth/token
UserApiEndpoint: http://localhost:3000/api/v3/user

A short description of each field:

AuthEndpoint: The endpoint where the OAuth 2.0 client will request an authorization code
TokenEndpoint: The endpoint where the OAuth 2.0 client will exchange an authorization code for an access token
UserApiEndpoint: The endpoint where the OAuth 2.0 client will pull the user data/profile from for login and signup purposes

@christopher1 Thanks, that is what I needed!

Another question to confirm: these keys and this key need to be unique for each Mattermost installation? Do they also need to be backed up, ie. changing any of those will have consequences for end users?

@marin Yes they do need to be unique.

They also should be backed up. The consequences of losing the first three are not so bad. All current invite or password reset links would be invalidated. However the forth key is used to encrypt parts of the database and losing it would render those parts unreadable.

@christopher1 Thanks for confirming.

@christopher1 One additional question regarding GitLab SSO: what "Callback URL" is expected when adding Mattermost as System Oauth application?

WIP MR is in !434 (merged) . This requires some beautifying but built package will setup and configure Mattermost from start to finish.

@marin Assuming a Mattermost instance running on http://localhost:3000 you would use these two callback URLs:

@christopher1 thanks again :)

Merged and RC packages released. Tweeted at https://twitter.com/gitlab/status/634427126922506240

@christopher1 @it33 rc2.1 packages were released yesterday at our package repository that have Mattermost bundled. I've also added some basic documentation on how to install and configure Mattermost. Please let me know if you have any remarks or create a new issue if you are experiencing problems. In the meantime, I will find this issue resolved.

Status changed to closed

We will also have to discuss how we will handle communication regarding new releases/configuration changes/structure changes to Mattermost that might require changes in the package.

Thanks @marin! Wonderful news! In terms of communication, we're open to options.

Perhaps Mattermost team can file a ticket when it seems appropriate to offer a new GitLab Mattermost release, which should include changelog and documentation on necessary config changes, and GitLab can decide whether to take the build?

Just an idea, very open to different approaches.

@it33 Creating an issue sounds like a best move, lets go with that for now and see how it works

We'll bundle Mattermost with GitLab so people have a great alternative.

Unfortunately, now that Mattermost is bundled, my organization will be unable to use Gitlab due to the AGPL/Commercial license of Mattermost. This issue mentions HipChat being closed source, yet the solution is to bundle a piece of software that will keep organizations from being able to customize/alter their implementation.

@LongLiveCHIEF Mattermost (/cc @it33) will give us a different license. @it33 can you ensure this new license ends up in 8.0? How will we communicate this license?

@LongLiveCHIEF, @sytses There is a ticket in Mattermost to add the license to a special build for GitLab Mattermost with this included but it didn't get in for Beta1, for now I can open a ticket to replace the license files manually. Traveling right now, will add this end of today. There is an FAQ in forum.mattermost.org in GitLab Mattermost that tracks the ticket.

Hi @LongLiveCHIEF, @sytses,

Sorry about the delay, the ticket for adding the license automatically to the GitLab Mattermost via our build process got missed as we migrated from a private issue tracker to a public issue tracker this month.

I've opened a request for GitLab omnibus to add the GitLab Mattermost MIT License--apologies, I could not find the previous Mattermost license in the omnibus files, so I didn't reference the specific replacement to be made.

@sytses, regarding communicating the MIT license for GitLab Mattermost, we have an FAQ in GitLab Mattermost forum announcing and discussing the switch to MIT License from 15 days ago, and in the Mattermost Beta1 announcement 4 days ago this was also announced. GitLab Mattermost was release 18 days ago, and was in our minds MIT Licensed 15 days ago, so 83% of its existence has been MIT Licensed.

To communicate the switch to MIT License, and to announce other features for GitLab Mattermost, perhaps the GitLab 8.0 announcement could link to Mattermost Beta1 announcement?

Announce also includes thank yous to many GitLab community members who influenced the release, and the future of Mattermost,

Sorry again about the delay here,

It sounds like Mattermost is AGPL/Commercial, but that you have a version of it, that you're calling "Gitlab Mattermost", which is MIT?

If that's correct, then the fact that you've renamed the distribution to Gitlab Mattermost (along with the license), should resolve any concerns about not being able to use Gitlab or even the new Gitlab Mattermost bundled feature.

If that helps you, then keep that in mind when discussing Mattermost/Gitlab Mattermost in announcements, as using that distinction consistently will help ensure any orgs that don't allow use of AGPL won't have trouble convincing their legal teams that they're different packages.

@LongLiveCHIEF The thing is that GitLab Mattermost under MIT will be only compiled code, not the source.

Thanks @LongLiveCHIEF, thanks @sytses, that's correct,

"GitLab Mattermost" is available under an MIT license as a compiled version of "Mattermost", and
the MIT license does not apply to the "Mattermost" source code

The intention of the Mattermost team was to offer a "GitLab Mattermost" license compatible with the needs of the GitLab community (simplicity of MIT/Apache licensing), while the Mattermost open source project remains consistent with its goals.

Bundle Mattermost with GitLab Omnibus packages

Designs

Child items ...

Activity

Admin message

Admin message

Bundle Mattermost with GitLab Omnibus packages

Activity