Architectural discussion forum and ownership of mid to large systems
This comes from the CI Long polling retrospective
From the tech side we are missing a forum where we can discuss large technical solutions that will affect across the whole company, and particularly we are missing a form of technical product ownership.
For example, limiting the scope to what happened with CI long polling:
- There is no ownership of workhorse, there are maintainers, but they work for specific products, which means they need to steal time for workhorse while they work in other features. This seriously limits the time we can invest in making large structural changes like the long polling one, this translated in this change taking a really long time to get implemented because it wasn't ever properly resourced.
- The long polling implementation at the workhorse level is only affecting CI, no other part of the application can take advantage of it because it's a point solution.
- We want to build real time, but we don't have long polling available as a system solution, so we are using ETags stored in redis and regular polling to avoid hitting the database. On the long run redis will also run out of capacity if it is our own solution for this kind or issues.
- We had an issue where we tried to start a discussion to move to event processing to stop doing short polling, but it got closed in favor of ETags for CI Long polling. As stated before, long polling is only implemented for CI, but it is necessary for multiple realtime features that we want to deliver:
- we are already polling for notes
- we will also be polling for the title of issues
- much more many to come
- Additionally, we will be hitting a much more scarce resource with short polling: unicorn - even though we don't hit the DB directly, we are using processes to handle these requests, we should move this processing out of rails.
In other words, we are missing reusing the long polling solution that we have specific for CI to make it a general solution that can be reused across different teams, solving this problem once and for all in a resource effective way. But we just don't have an owner or a forum where we can discuss this without tying it to a feature that belongs to a team.
This is directly linked to the issue that @mydigitalself commented in the retro, and it is that we are missing technical products that can be used across the company, in this case: a pub/sub service, or event sourcing. Since each team has to solve this problem for each isolated feature, we find ourselves reinventing the wheel multiple times.