Geo: Implement consumer (aka GeoLogCursor) for event log
In #2384 (closed), we are adding a Geo event log and deprecating the use of system hooks. First, we should focus on these use cases:
- git push
- project deletion
We need to figure out how to consume these events properly and update the state of projects/files/directories.
We investigated LISTEN/NOTIFY channels in PostgreSQL to subscribe to changes in the event log, but this does not work well for Geo because:
- The LISTEN and NOTIFY only works on clients connected to the primary, not the secondary (so we would have to have a process on the secondary Geo Node connect to the primary DB)
- If the secondary is disconnected, it's possible the NOTIFY queue is filled (this is bad)
How this will work in the first iteration:
- Poll the secondary database for changes in the latest ID
- A separate process (Ruby to start, Go later if necessary) runs and scans this log
- The consumer's job is to scan this log and update the repo/file state with CRUD information. For example, if we have a namespace with 100 projects deleted, we should mark all 100 projects as deleted and do a full sync of the paths affected by those actions.
Another process should then handle the actual resync based on the state of the tables.
@brodock @dbalexandre Does this make sense?
Edited by Stan Hu