Establish a GraphQL API plan

Using the GraphQL wrapper linked here wouldn't be possible since it relies on Node during runtime.

We currently use Grape for our API, and they're discussing how Grape could expose endpoints with GraphQL formatting: https://github.com/ruby-grape/grape/issues/1491

I'm very interested in GraphQL as I think it's a pretty big upgrade from REST with JSON and I assume if GitHub and Facebook are in it for the long-haul, it will probably be the defacto standard in a few years, but I think we should have this open for discussion rather than assuming that everyone would be willing to go along with us.

Another problem is that using GraphQL would probably violate one of our core values: "Use boring solutions"

See also:

http://artsy.github.io/blog/2016/06/19/graphql-for-mobile/, which talks about how Artsy uses GraphQL with their API

Yes, I am not trying to convert anyone at GitLab to NodeJS. It was just and pattern - fronting the existing API to provide the functionality without a back-end rewrite.

GraphQL is not a just new shinny toy. It has been around for years and is mature. What I love about it is the inversion of control. Rather than the sever defining what the consumer wants, the consumer decides what it wants - better decisions.

Added ~13884 feature proposal labels

Mentioned in issue #22783 (moved)

Mentioned in issue #5983 (closed)

A perfect use case for GraphQL is this feature here #5983 (closed). Please align with them or otherwise they might jump on some other technology.

Interesting issue /cc @JobV

Changed title: establish a graphql api plan → Establish a GraphQL API plan

Just for completeness. As mentioned in https://gitlab.com/gitlab-org/gitlab-ce/issues/5983#note_17599217 the Apollo stack seems to be a the moment next to Relay a very promising implementation of GraphQL. Backend wise the Apollo server could be integrated just as mentioned in the proposal of this issue. And for the frontend the Apollo client can be used which can be used as vanialla JS client but also there is a plugin for Vue.js called vue-apollo (example).

@subesokun are GraphQL and nchan mutually exclusive? GraphQL is transport generic and needs delivery. Putting Nginx with nchan in front of GraphQL Subscriptions appears to be a 'nice' solution. My primary interest in GraphQL is the inversion of control from server to client - solving events is a two for one! - That is even sweeter with vue-apollo , thank you for pointing this out.

On of the things we face at my work as we move more and more to events and message integration is this dealing with incremental change when one domain wants to consume an event that is not yet being emitted by the other. The GraphQL plan should include rough timelines of when each service is expected to throw events and then consuming services can properly gauge their timing and workarounds.

Hi @tom.davidson, surely they aren't mutually exclusive. So you mean you want to write your own GQL subscription client which uses nchan under the hood? Yes that could be a solution but I don't know how generic nchan is. The goal of GQL is to just send the data to the client as requested in the GQL query to reduce traffic and if nchan would always send the "full" payload and not only a subset of that payload to the client that wouldn't make much sense. Also the GQL subscription queries can get quite complex... hard to know if you can represent those as nchan subscriptions. So other solutions as Apollo are just using a generic WS channel to send the subscriptions query to the GQL server and then the server just responds via the WS with messages that conforms the query.

I was giving the proposal with a GraphQL server in the middle (between the JavaScript frontend and the Ruby backend) some thought but actually this can be imho only a temporary solution and on long term the GQL server should be directly integrated into the backend. Mapping the GQL queries onto REST calls in the GQL server is possible but also gets complicated and hard to maintain once the API is growing. Every change in the REST API of the backend has the be reflected into the GQL server (by which team? frontend? backend?) and you might need to perform multiple subsequent REST calls against the backend and perform some response mapping in the GQL server to fetch all relevant data that can be requested by a single GQL query. And in case of multiple REST request the latency might become an issue (when not using HTTP/2).

Especially when it comes to subscriptions things are getting quite tricky. Either you've to poll the backend from the GQL server all the time or you've to implement some pub/sub mechanism between them. This could be nchan or any AMQP solution like e.g. ZeroMQ which could be directly integrated into the backend. But again you've to transform the incoming messages within the GQL server which might cause some further REST calls to fetch all relevant data as requested in the GQL subscription query unless you can forward this push message to the frontend.

But nevertheless starting with the GQL server in the middle might be the best plan to kick off things and then step by step the GQL server could be backed along with the REST API into the backend as they're not mutually exclusive :) And maybe an half year later or so there will be some nice Ruby implementation of GQL

I was giving the proposal with a GraphQL server in the middle (between the JavaScript frontend and the Ruby backend) some thought but actually this can be imho only a temporary solution.

Agreed, I think this is important to the transition. Rather than run GQL and REST side by side, GitHub runs REST on top of GQL. Otherwise I think you would need to implement CQRS. Running the GQL proxy might be the scope of the roadmap - after than then what is learned will be applied to the next steps if any.

Especially when it comes to subscriptions things are getting quite tricky.

Yes and I do not quite get it all. I on face I think that offing the G Subscriptions would require the proxy to poll but ideally GitLab would eventually throw events for everything and the polling could be cut out or at least only used in edge cases. I put down that Redis Pub/Sub might be a great fit. Redis stores could be use for other aspects of GitLab as well.

So you mean you want to write your own GQL subscription client which uses nchan under the hood?

Oh no, I intend for you to do it :D - seriously, I thought nchan could deliver over SSE or WS such that a generic WS client, or Appolo would be compatible with it. I was thinking nchan proxy is equivalent has an http proxy but obviously do not know from hands on experience. "For use in a web browser, you can use Websocket or EventSource directly, or try the NchanSubscriber.js wrapper library." https://nchan.slact.net/

I viewed nchan as a possible reverse proxy, an implementation optimization: graphql client --- nginx/nchan --- graphql server --- data & event sources

We can create a stateless GraphQL proxy for exploration without changing GitLab. Lets create a Lambda/OpenWisk/GoogleCloudFunction/Docker app (I propose NodeJS) that can be configured with a user's or project's API access tokens. We will not get a lot of events for GraphQL Subscriptions, but it would be a start. Maybe we can keep state/connection and relay a webhook event over GraphQL Subscriptions. If GitLab were to go polygot micro-services, the actual code could be used, but if not, the life would be just for the proposed transition.

Thin(dumb) as possible proxy based on Apollo Stack that anyone can easily deploy and use in their GitLab GraphQL experiments. AWS Lambda is easy to deploy and create for exploration.

Pin down Swagger for the REST API, even if unofficial. REST is first and will be continue to be first for quite some time, even if there is commitment to GraphQL. The API specs in Swagger will make it easy for the GraphQL proxy to stay in sync.
Use schema generator such as swagger-to-graphql.
Express? - not as slow compared to others on Node 6, AWS Labs has Express to Lambda Server wrapper. Same app can be deploy in Lambda and out.
Ship - i use terraform but simplicity of cluadia.js is interesting.

From there we can think about where to go for Auth and Subscriptions .... but really I would like to see the GitLab bit on first so.

/cc @rymai

Oh no, I intend for you to do it :D

Hehe, as I don't know nchan as well I'm out of the game ;)

Having a thin proxy based on the Apollo stack sounds like a good start. Using Swagger schemas to define GQL schemas sounds interesting but you might get limited at some point and can't use all GQL schema features. Actually the GQL schema is already the contract between front end and backend so I'm not so sure if it makes sense to put one extra layer with Swagger on top of it. Mhhh I'm wondering if there are some nice GQL schema designers around which allows you also to browse it and add descriptions to each schema and its attributes.

If GitLab were to go polygot micro-services, the actual code could be used, but if not, the life would be just for the proposed transition.

Here's some interesting talk https://youtu.be/XOM8J4LaYFg?t=4m4s which tackles exactly the same issues as mentioned here. Also he's talking about how they define and design their GQL APIs, very interesting. The GraphQL-first philosophy is also worth a read.

I'd also go with Node.js so that you can just use the Apollo GraphQL server on top of it in combination with the Apollo GraphQL client. As mentioned by you already the subscriptions-transport-ws can be used for handling the GQL subscriptions. But not sure if you need to use nchan for it as it comes with a client and server which are communicating via WS.

Actually the GitHunt-API is a very good starting point. It uses the Apollo stack so you can build an initial version of the GitLab GQL API based on it.

So the architecture I've in mind would be:

Apollo client      ---(HTTP)--->                                             Apollo server
                                     nginx for load balancing and HTTPS --->                ----(HTTP)---> GitLab backend
Subs.-trans-ws client  <---(WS)----->                                        Subs.-trans-ws server

Ruby has pretty mature graphql library: https://github.com/rmosolgo/graphql-ruby

This is what GitHub probably uses

@sheerun there are 2 GitHub employees and 1 Shopify employee in the top 5 contributors to that project, plus they've publicly said they use that.

/cc @mydigitalself

mentioned in issue #32046 (moved)

@mydigitalself I think GraphQL makes a lot of sense because:

It allows callers to request only the data they need
This enables us to start using our own API, reducing code duplication and making our API more useful for everyone

I think we should start working on this.

Especially from a FE development side this makes a lot of sense, as we basically can then decouple development of features from each other and it will give us way more flexibility on implementation and backend can focus more on the generic side.

Totally agree with the above arguments about the benefits of GraphQL. Github has switch to GraphQL recently, here is their justification: https://developer.github.com/v4/

I've been creating GraphQL client/server applications for about six months myself and I can't express how much better the experience with this API is compared to REST!

I'd love to be involved in this ^^

I'd love to be involved in this ^^

Oh yeah, me too!

I think that we could start to still leverage Grape entities but figure out how to build on top of that GraphQL ability to limit the amount of computed data from the resources, as this is the greatest benefit that we are trying to solve I believe. Request only what matters, not everything.

mentioned in merge request !11998 (merged)

For reference, this might be interesting and related to gitlab-org/gitlab-ce#32046: https://www.hackerone.com/blog/the-30-thousand-dollar-gem-part-1

I was recently thinking about creating a grape-graphql gem that would extend Grape + Grape Entities and leverage GraphQL Ruby gem. This gem might add a new abstract interface to grape entites that we could implement in order to make entity behave like a GraphQL Type. Grape entites have a lot in common with GraphQL types.

This is not trivial, but would require a minimal PoC to prove it can work. Then we can iterate on extending grape-graphql

What do you think @ayufan @stanhu @rymai @DouweM ?

/cc @nick.thomas

Sounds like a good plan.

Should we start with PoC and see what is needed to do it? It would give us some understanding how hard it is, how maintainable it is, and how it fits into our architecture.

EE customer interested in this feature: https://gitlab.zendesk.com/agent/tickets/79036.

@mydigitalself since we are "advertising" this via our docs and there is a fair amount of desire for and consensus that this will be a great feature, is it all possible to put a milestone on this? Or venture a guess at when the initial implementation will be targeted for?

added customer+ label

added shortlist label

changed milestone to %Next 2-3 months

Is there a reason we would build something new with Grape rather than use the graphql-ruby? I think there are some challenges with our current APIs, especially surrounding authorization. It may be a good opportunity to drastically improve the situation and if creating grape-graphql is non-trivial...

@dblessing The idea is to reuse some of existing API implementation.

If we decide to implement a middleware like grape-graphql then we would need to use graphql-ruby anyway. Reimplementing existing API into brand new GraphQL API is a significant effort, but this is also a direction we should think about, we should not write it off without a consideration.

Working on a grape-graphql might be the good idea in case that our goal is to support two APIs (REST and GraphQL) and make the latter instantly available for all endpoints. If we plan to phase the old API out in favor of a GraphQL one, then working on a new API, from scratch, makes sense.

That said, I merely proposed preparing a Proof of Concept, because a prototype like this may be good to have in order to make a better informed decision about choosing a direction here.

assigned to @nick.thomas

I've been asked to evaluate moving the merge request widget to use GraphQL for %9.5, so I'm looking into this now.I guess we need a separate issue for this!

mentioned in issue #34754 (moved)

removed assignee

Closing as this plan has been established. #34754 (moved) for initial implementation.

closed

Establish a GraphQL API plan

Description

Proposal

Designs

Child items ...

Activity

Admin message

Admin message

Establish a GraphQL API plan

Description

Proposal

Activity