Added webpack to Procfile

Adds webpack dev server command to Procfile

added 1 commit

• b1f43b5b - add webpack to "gdk run app" command

Compare with previous version

@iamphill just out of curiosity, what is webpack? Does it use TCP ports? What will it mean for GDK users to have webpack running all the time? How much memory does it use, is it slow to boot, does the rest of the application work if webpack is not done booting yet.

@jacobvosmaer-gitlab webpack is what we are going to be using for our frontend asset compilation, at the moment it will be just JS but we will probably expand that to the CSS as well. See https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7288

Without it running all the time the JS won't actually work, this command just runs the webpack dev server which re-compiles the JS after a change.

Not 100% sure on the memory. @mikegreiling any ideas?

Shouldn't be that slow to boot. If it hasn't booted by the time the page loads, then the JS just won't work. But I can't imagine that to happen, at least it hasn't happened to me.

The webpack dev server operates in a similar way to sprockets, but it runs entirely in node and compiles much much faster. It opens a tcp socket at localhost:3808 and serves the compiled javascript assets for use in development. It retains all compiled scripts in memory so that when one file is saved, it can transpile that single component without needing to recreate the entire bundle again from scratch. It will also allow us to utilize "hot module replacement" to make developing Vue projects much easier.

Just checking it running on my local machine it looks like it utilizes about 350Mb of memory.

If a user wanted to run their dev environment without webpack dev server, it could be configured that way as well. The assets would need to be precompiled with bundle exec rake webpack:compile, which would operate similarly to how assets:precompile works right now.

We did a push a while back to open as few TCP ports as possible, to enable people to run GDK twice on the same host (one CE, one EE). Is there any way we can make the dev server use a Unix socket?

350MB of memory: ouch. Do we disable the corresponding rails thing in development now? (sprockets I think?)

@mikegreiling not sure about unix socket, will have a look!

We can't sadly, sprockets is still used for CSS.

OK, look forward to hearing about the unix socket. It may not be supported but if it is it would be nice.

I doubt a unix socket would work as the web browser will be accessing it directly.

As for sprockets, webpack will eventually subsume the remainder of its responsibilities including CSS compilation in a future iteration, and then we can remove sprockets entirely.

So it looks like webpack dev server 2 supports unix socket, not sure how we would go about it using. But it seems it isn't possible right now.

I guess we should make the port configurable through a env variable or something similar to allow EE to be running at same time.

@iamphill we can change this by changing the line var DEV_SERVER_PORT = 3808; within config/webpack.config.js and adding a config.webpack.dev_server.port setting to config/environments/development.rb. Perhaps we can do 3808 in CE and then 3809 when we create the EE branch?

For a unix socket to work, we'd have to fork the webpack-rails gem and tell rails to proxy a unix socket for certain HTTP requests within /assets/. That may be tricky... This may also break several features of webpack dev-server such as the ability to negotiate hot module replacement in development.

@mikegreiling i'm looking at changing it so we do it like we currently do with port file https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/set-up-gdk.md#gitlab-enterprise-edition

re. unix socket, i guess it should be something we look into when we move to v2.

added 2 commits

• fea37934 - Document webpack port file
• 73301074 - Merge branch 'webpack' of gitlab.com:gitlab-org/gitlab-development-kit into webpack

Compare with previous version

I agree with just using a port if it's for the browser. Or would it be natural for workhorse to proxy to webpack in the backend?

Would it happen to be the case that webpack only serves resources under /assets/javascripts? Then we could add a -webpackSocket option to workhorse, and when that option is set, proxy /assets/javascripts to the given socket.

I don't know if this makes sense from a webpack perspective. Maybe the client code really (badly) wants to use a TCP port on localhost.

If we can route webpack access through workhorse that also helps Docker and Vagrant style setups, where you have to explicitly forward ports from wherever the app runs to your dev 'client' environment.

In dev mode, webpack keeps and serves all of the assets from memory, so I don't think telling workhorse to proxy a directory would work.

It's possible once we update to webpack 2 and it supports unix sockets that we could configure workhorse to proxy that socket somehow.

OK.

@grzesiek what do you think the impact of this new TCP port needed by browsers accessing gitlab in GDK. Do we need to change docs for Docker, Vagrant anywhere?

@jacobvosmaer-gitlab If we need to expose additional ports to make everything work fine, then we probably need to update documentation as well.

In dev mode, webpack keeps and serves all of the assets from memory, so I don't think telling workhorse to proxy a directory would work.

@mikegreiling I'm not sure if I understand this correctly, can you elaborate a little?

@grzesiek I meant to say that the files webpack generates in this mode aren't written to disk, so we can't just watch a directory and serve the files, we'd have to proxy the webpack dev server itself.

added 14 commits

• 73301074...cbbbd4e0 - 11 commits from branch master
• b2e79ece - Added webpack to Procfile
• d0132c5a - add webpack to "gdk run app" command
• cbd5720c - Document webpack port file

Compare with previous version

@mikegreiling Workhorse is a reverse proxy, so it should be able to route requests from the browser, thorough Workhorse to Webpack, and the other way around, without problems. The question might be does it make sense to implement webpack proxying if this is not used in the production environment. Are there any use cases when one would need to access Webpack server in the production environment?

@grzesiek Ah, I see what you mean now. No, I can't think of any reason we'd want to run a webpack server in production, it is strictly a development tool. In production it will compile the assets into static javascript files which can be served by the production server (nginx or whatever).

Unrelated question, but do you know why this build is failing? I'm not sure how the changes in this branch could be causing the failure I'm seeing in CI right now.

@mikegreiling I'm just working on the fix, we simply need to update GDK Docker test image

assigned to @jschatz1

@mikegreiling Pipelines is green now

OK so what is still unresolved here is: what is the impact of adding webpack for Docker/Vagrant users? If they don't enable forwarding for the required port they will see no assets when browsing their GDK instance.

Workhorse already listens on port 3000 (or whatever is in the port file) so if we could reverse-proxy the browser -> webpack connection through workhorse that would save Docker/Vagrant users the hassle of network configuration changes.

If not, then too bad, but then we must have clear documentation of what Docker/Vagrant users have to do to get connectivity between their browser on the host and webpack in the container/VM.

Even with webpack-dev-server listening on a TCP port (e.g. 3809), we can make life easier for everybody if workhorse (in development mode!!) proxies /assets/javascripts to localhost:3809/assets/javascripts. Because then no new Docker/Vagrant port is needed.

Am I reading https://gitlab.com/gitlab-org/gitlab-ce/blob/ecdcd1be87e140c144bb2afc861a1dc56b297cdd/config/webpack.config.js correctly when I conclude that webpack dev-server hosts its stuff at /assets/webpack?

Then I propose we do this in workhorse:

• add webpackAddress option
• when developmentMode is set, proxy /assets/webpack to webpackAddress

Then when doing development mode, the browser can fetch these assets at localhost:3000/assets/webpack and no networking changes are needed for Docker/Vagrant.

It makes sense @jacobvosmaer-gitlab

@mikegreiling can you confirm if my workhorse proxying proposal fits the bill? If so then we can create a workhorse issue for it.

@jacobvosmaer-gitlab Yes, I believe that will work. I would just have to update some relevant lines within config/environments/development.rb and it should work. How long do you think the update to workhorse would take? I'd want to test it out and make sure sourcemaps and other features of the dev-server still work as they did before.

mentioned in issue gitlab-workhorse#104 (closed)

mentioned in merge request gitlab-workhorse!121 (closed)

This will be a breaking change so this MR needs a CHANGELOG entry.

added 2 commits

• 28caa555 - move webpack from app group to db group
• 46e7914f - configure webpack-dev-server in both Procfile and gitlab.yml based on webpack_port

Compare with previous version

added 10 commits

• 46e7914f...f407d3d9 - 5 commits from branch master
• 0d49e8b2 - Added webpack to Procfile
• aaac1ed9 - add webpack to "gdk run app" command
• 8612ca26 - Document webpack port file
• 21350d8a - move webpack from app group to db group
• 2e7dc210 - configure webpack-dev-server in both Procfile and gitlab.yml based on webpack_port

Compare with previous version

This should be ready for review.

@grzesiek or @jacobvosmaer-gitlab are you free to take a look at it?

assigned to @grzesiek

Based on the recent changes, modifying webpack_port and then running gdk reconfigure should update both gitlab.yml and Procfile with the appropriate values to update the webpack dev server config.

As per @jacobvosmaer-gitlab's suggestion I also moved the webpack process to gdk run db instead of gdk run app because it is needed for rspec tests etc and makes more sense along with that group.

mentioned in commit 0b98f5e7

added 2 commits

• 88e12eda - ignore .gitlab-npm file
• 0b98f5e7 - update CHANGELOG.md for !237 (merged)

Compare with previous version

assigned to @jacobvosmaer-gitlab

Looks good after the brief review, but passing to @jacobvosmaer-gitlab for the thorough one.

@grzesiek looks good! Can be merged when the https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7288 goes in.

assigned to @grzesiek

Merging this for @mikegreiling since gitlab-ce!7288 went in.

mentioned in commit 1d868aaa

merged

thank you @stanhu

SOLVED: OK, I needed to pull ce

---

I'm getting the following error when I gdk reconfigure:

(in /home/axil/git/gitlab/gdk/ce)
rm -f \
gitlab/config/gitlab.yml \
gitlab/config/database.yml \
gitlab/config/unicorn.rb \
gitlab/config/resque.yml \
gitlab-shell/config.yml \
gitlab-shell/.gitlab_shell_secret \
redis/redis.conf \
.ruby-version \
Procfile \

sed -e "s|/home/git|/home/axil/git/gitlab/gdk/ce|"\
  -e "s|/usr/bin/git|/usr/bin/git|"\
  gitlab/config/gitlab.yml.example > gitlab/config/gitlab.yml
port=3000 webpack_port=3808 support/edit-gitlab.yml gitlab/config/gitlab.yml
Newline appended
  gitlab_shell:
?
make: *** [Makefile:31: gitlab/config/gitlab.yml] Error 1

I manually created the configs and when I run gdk run db I get:

11:52:05 webpack.1               | npm ERR! Linux 4.9.7-1-ck-skylake
11:52:05 webpack.1               | npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "run" "dev-server"
11:52:05 webpack.1               | npm ERR! node v7.5.0
11:52:05 webpack.1               | npm ERR! npm  v4.1.2
11:52:05 webpack.1               | 
11:52:05 webpack.1               | npm ERR! missing script: dev-server
11:52:05 webpack.1               | npm ERR!
11:52:05 webpack.1               | npm ERR! If you need help, you may report this error at:
11:52:05 webpack.1               | npm ERR!     <https://github.com/npm/npm/issues>
11:52:05 webpack.1               | 
11:52:05 webpack.1               | npm ERR! Please include the following file with any support request:
11:52:05 webpack.1               | npm ERR!     /home/axil/git/gitlab/gdk/ce/gitlab/npm-debug.log
11:52:06 webpack.1               | exited with code 1

mentioned in issue gitlab#8073