Audit log improvements (META)

Added audit events label

To tackle in a single release

Other audit related issues:

Audit events in admin interface of user edits #251
Push Audit Events to Syslog #337
Register admin impersonation begin/end in audit log #743

The last bullet point would likely go a long way toward gitlab-orggitlab-ce#3962.

mentioned in issue #1367

Current state:

We use 7 different logs (6 with documentation) that can not be used for an audit log per se.
We have one method (log_audit_event) that logs information in the DB. This info can be used as an audit log but is incomplete.
In both CE and EE, we barely have any upvotes from the community on the Audit issues.

For the Logs

Documented and in Rails:

CE and EE: production.log (https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/environment_logger.rb)
- Information about Rails requests + SQL requests
CE and EE: application.log (https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/app_logger.rb)
- Display the events
  - Impersonation start/stop
  - Group created/deleted
  - Namespace's change of path
  - Project rename
  - User events (creation, deletion, update) based on user.rb
  - base_service.rbevents
CE and EE: githost.log(https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/git_logger.rb)
- Git info and errors
  - Error on post-receive actions
  - Error on git garbage collection
CE and EE: sidekiq.log (https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/sidekiq_logger.rb)
- Dump of Sidekiq actions

Undocumented and in Rails:

CE and EE: repocheck.log (https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/repository_check_logger.rb)
- Check the consistency of the repository

(I've created an issue to add documentation for this log https://gitlab.com/gitlab-org/gitlab-ce/issues/25622)

Documented and not in Rails:

gitlab-shell.log and unicorn_stderr.log

For the Audit event screen

CE and EE: With the log_audit_event method:
- Authentication made with LDAP, SAML and regular login
- Only in EE: Accept group membership, Leave group
- Only in EE: Group members creation and update
- Only in EE: Deploy key creation/deletion
- Only in EE: Project members actions (addition, update and deletion of project members)

The first steps to have a better audit logging would be to record these:

Log user actions #1370 (closed)
Log project actions #1371
Log group actions #1372

Ideally we would have a audit.log file, with a configurable maximum number of events recorded.

@yorickpeterse how should we do the tasks above, without affecting performance?

@regisF A log file can quickly grow in size and requires non trivial log parsing to visualise or permanently the data. The fastest/most natural solution that I can think of from the top of my head is to:

Track all events for a request in memory (local to the request)
At the end of a request, schedule this data using Sidekiq
Have a Sidekiq worker process this data, inserting it, maybe adding extra data in the process, etc
Flush the data once scheduled

Having Sidekiq perform the storing of the data is better than doing this in a request. This is due to scheduling a job in Sidekiq being much faster than inserting (potentially) hundreds of rows into a database.

This is similar to how we track performance data using GitLab Performance Monitoring. Everything we measure is basically stored in an array which is then sent to InfluxDB at the end of a request.

Thanks for your input @yorickpeterse

So it'd be better to record this log in the DB instead of a .log file?

@regisF

So it'd be better to record this log in the DB instead of a .log file?

Yes, albeit with some sort of limit (e.g. only the last N months per user) so the database doesn't grow forever.

@yorickpeterse I've heard about the risk of losing jobs with Sidekiq. Losing an audit event is definitely not something that we want in this case. Is this a potential issue?

@regisF Losing jobs only happens when:

Sidekiq pops a job off the queue
The process dies immediately (e.g. the process dies, the kernel crashes, the server explodes, etc) or is SIGKILL'd by another process/user

Other solutions suffer from similar (and other) problems. Log files may get rotated because they're too big, a disk could crash without being recoverable, etc. Inserting in a database can also lead to data loss if the database crashes during the write, etc.

Handling the losing of jobs is something we have been looking into, but we don't have a solution yet. Sidekiq Enterprise has a reliable job fetching system, but it's not FOSS so we can't use/ship it.

mentioned in issue #1370 (closed)

mentioned in issue #1371

mentioned in issue #1372

@yorickpeterse thanks for your insights. Is there a reason why you would prefer using the DB instead of a file?

@regisF

When using a (log) file we need to:

figure out where to store it
rotate it (so it doesn't grow to GBs in size)
figure out where previous versions are (so we don't suddenly lose data due to a rotation)
store the data in some kind of format and then parse it upon reading
efficiently read/parse the file so we don't load GBs of data into memory
ensure that multiple processes/threads can store data in the same log file without overwriting each other's data. This can be done using synchronisation, but this is very slow
configure all of this in Omnibus, document it, make sure it's backed up, etc, etc
figure out a way to migrate data in these files should we ever change the format in a way that's not backwards compatible

When using a database we need to:

Store the data in the database, using the tools we already have in place
Get the data from the database, using the tools we already have
Make sure we remove data older than N months so the table doesn't grow forever

In other words, when using the database we save ourselves a lot of coding trouble.

mentioned in issue #1420

mentioned in issue #1421

closed

reopened

@smcgivern @DouweM can we do the first step in a single release?

for expanding the current audit events for this purpose. That is the original proposal in the issue, too. Then we can easily show these events to users in the UI, and we can also add some export functionality to satisfy global auditing requirements.

@regisF Currently, log_audit_events is a synchronous event - it is not sent to Sidekiq so there's no concern of a lost event.

@DouweM has just had a week off, Imma let him answer!

closed

reopened

@regisF Yes, I think so.

mentioned in issue #1449

mentioned in issue #1487

There is an ISO norm that specifies how audit logs should be done: ISO-27002 (http://www.iso27001security.com/html/27002.html)

mentioned in issue #1490 (closed)

added marketing work label

changed milestone to %8.17

added ~333913 label

added ~481018 label

I've put the Platform label @mydigitalself. I think it relates to the platform. I'm not sure we'll have bandwidth for this in 8.17 but if we don't, we should plan this in the future.

@regisF We don't, unfortunately. I'm putting this in the %Backlog

changed milestone to %Backlog

assigned to @mydigitalself

This customer is asking for the following activities that they need to have logged for their central auditing system: https://na34.salesforce.com/0016100000YUHOs

user login (successed/failed),
user logout,
adding user,
deleting user,
locking user,
unlocking user,
adding user to group,
removing user from group.

added shortlist label

added customer label

We're going to try and target this for 9.3 (May 22nd 2017). We can't confirm quite yet if it will make that date, but will certainly try as this is an important additional to GitLab EE.

@mydigitalself by EE you mean EEP?

Yes.

added ~897283 label

changed milestone to %9.3

Regarding where to store the audit logs as per @yorickpeterse's comment https://gitlab.com/gitlab-org/gitlab-ee/issues/579#note_20021796

There are two use cases to consider. One is an administrator trying to understand what has happened. The other is for audit compliance. For audit compliance, you CANNOT (purely) store the logs in the "system". By this I mean if the audit data is stored in GitLab's own database, that means they are controlled internally by GitLab and can be subject to discrepancy. For audit compliance, you MUST send the data to an external source, a log file is sometimes acceptable, but what is more acceptable is being able to forward the logs on to an external system such as Splunk or Logstash.

Proposal for 9.3 release - do the simplest thing.

Add ability to enable audit logging for EEP licenses
Add "audit.log" to Admin Area -> Monitoring
Implement basic retention policy for the log (e.g. logrotate, or deletions)
Implement First Step from above description - https://gitlab.com/gitlab-org/gitlab-ee/issues/1370 Log project actions https://gitlab.com/gitlab-org/gitlab-ee/issues/1371 Log group actions https://gitlab.com/gitlab-org/gitlab-ee/issues/13723
Include "Admin events" in this solution (e.g. admin added a user) as per https://docs.gitlab.com/ee/administration/audit_events.html

In 9.4 and beyond we can add additional events enable log forwarding

I would also suggest that if we are logging to a file that we save the data in a structured format (e.g. JSON) so that log forwarding tools can search/query this data easily.

mentioned in issue #2174 (closed)

FYI, we are also talking about using a similar audit log for Geo: https://gitlab.com/gitlab-org/gitlab-ee/issues/2174#note_28319238

@mydigitalself So we have to decide between storing the audit events in a log vs. the database? I think @yorickpeterse his comment as linked from the description makes sense https://gitlab.com/gitlab-org/gitlab-ee/issues/579#note_20021796 and storing them in the DB is the easiest thing to do. I thought that most customers would like to see us storing more events, not necessarily storing them outside the DB. For Geo DR having it in the database would also be much better.

You talk about needing it externally because audit compliance, I have a few questions:

Was this expressed by many customers?
Can we get there by just streaming the DB or snapshots of it to an external system?
Can we get there by instrumenting the Rails model for audit events, using PostgreSQL triggers, or something like that?

As a sideline to the above. I feel we have to make the right decision for both system separately. If we feel audit logs (Mike mentioned the audit requirements to me as well) need to go towards this change, whereas we need db-level logging for disaster recovery, we should not force either into the other. That'll result in a product/feature that is hard to iterate on.

That said, if we can make both work together, that'd be beautiful killing of two birds with one stone.

@sytses

I've personally just spoken to the one so far who aren't an existing customer yet, but they immediately said for any type of banking-related regulatory requirements you cannot store audit data in the same data store that the system resides in, which makes complete sense. So you will need log forwarding regardless in the banking space. @teemo also mentioned customer requirements to send the logs to their central auditing system https://gitlab.com/gitlab-org/gitlab-ee/issues/579#note_23164332. By sending to syslog, it's then pretty simple to forward to Splunk/Logstash as it's an established mechanism.
You could, but it's additional effort when you can already use standard log tools.
I'm not sure - @DouweM ?

To me, using syslog seems like an established boring solution, at least for the first phase, so I'm a bit puzzled as to why we would look to add complexity to the approach. This Audit Logging is not about DR, Geo or auditing every single action with content that takes place in the system, it's to provide compliance around permission, access and top-level events and I think by trying to do more, you add unnecessary requirements and complicate shipping something that we can iterate on in the future.

@mydigitalself Currently we don't use syslog directly, instead we either log to STDERR or to a log file.

I think there are two different feature requests here that are being mixed together:

Being able to store a trail of operations for auditing purposes
Being able to replicate this log elsewhere

Because different users may have different requirements for feature 2 I would propose first tackling feature one (by storing data in the database). Replicating database data to external sources is easy, and it's much easier to visualize. Operating on log files on the other hand (e.g. when displaying the log for a user) is a total nightmare.

For exporting the audit log I would start with something as simple as just providing an API that extracts the data from the database. This allows customers to build custom solutions (if they so desire). Based on commonly requested integrations we can then see what makes sense to support out of the box.

Either way, I strongly recommend against using log files as the primary source. Visualising this data, filtering it, etc, is a total pain.

@mydigitalself to clarify, not allowing one to store the audit data somewhere, it's fine if it's replicated elsewhere, right?

The arguments by @yorickpeterse sound pretty strong in favor for a database approach, but it assumes it's relatively easy to export our audit log from a database into something that can be used by applications typically used by enterprises. Is there some way we can validate that? Are there examples of similar approaches, or @yorickpeterse how would such a thing look, a db-based audit log that forwards to a logging solution?

We should solve the problems of our enterprise customers with the audit log. If their problem is having auditing information in a central logging service that expects X, we should make sure X is something we can easily have as an output.

@JobV For exporting data I envision two parts:

An API that customers can use themselves to build whatever exporter they want
Some form of recurring background job that e.g. every hour takes the log data of the past hour, then formats this and sends it somewhere. For example, this job could take the logs and write them to a plain text log file. The format to use, when to export it, etc, could be adjusted in the admin panel or some config file (depending on what works best).

How we do part 2 depends a bit on what makes the most sense for our customers, I unfortunately can't really judge on this.

@JobV @yorickpeterse my concern with the database approach as below is that there seem to be a lot of moving parts, with the potential for data loss to occur.

- Track all events for a request in memory (local to the request)

- At the end of a request, schedule this data using Sidekiq

- Have a Sidekiq worker process this data, inserting it, maybe adding extra data in the process, etc

- Flush the data once scheduled

Given this is audit data, the number 1 requirement for it should be reliability - i.e. that the risk of losing data is negligible. If it's possible for the queue to be tampered with, or deleted, or if the server falls over before the data has been put onto the queue or any other imaginable situation where the integrity of the data can be called into question, then you are failing the primary requirement. Searchability, filtering, etc... are secondary requirements and are also functions that can be performed by other systems such as Kibana, Logstash, etc...

One approach could be to use fluentd, which has countless input and output plugins to send the logs off to almost anywhere.

my concern with the database approach as below is that there seem to be a lot of moving parts, with the potential for data loss to occur.

This is no different when using a log file. A write may fail, a log file may be removed, a disk may explode, a process may be killed while writing the log data. Effectively the amount of possible problems is similar. We don't have to use Sidekiq for the DB approach, it was merely a suggestion. An alternative would be to insert all audit logs using a single INSERT (this would be absolutely crucial, we can't use multiple INSERTs for this as this is terrible performance wise) at the end of a request. So here the setup would be:

Track audit events in memory during a request
At the end of a request, flush these events to the database using a single INSERT statement
Separately we can then flush this data to external services periodically

Searchability, filtering, etc... are secondary requirements and are also functions that can be performed by other systems such as Kibana, Logstash, etc

We shouldn't require customers to install Kibana and related tools just to look at their audit logs. If GitLab wants to provide a solit auditing experience we should provide ways of viewing the data, and not just ways to write the data.

removed assignee

This is no different when using a log file. A write may fail, a.... etc

Of course, I appreciate this can all happen, it just felt to me that we were adding more layers of complexity when a simple solution can suffice.

We shouldn't require customers to install Kibana and related tools just to look at their audit logs. If GitLab wants to provide a solit auditing experience we should provide ways of viewing the data, and not just ways to write the data.

@yorickpeterse i'm not proclaiming that customers have to install Kibana to view their logs, they can have a simple log viewer in GitLab the same way we have the other logs available in the admin interface. If there's a need for more sophisticated visualisation, searching, then those tools could be appropriate, rather than re-inventing the wheel.

I'll leave it entirely up to the technology team to make the decision here, I think I've expressed my opinion and if you believe that using the db for storage will be as reliable and easy to stream into external systems, then I'm fine with that.

I would like someone to evaluate fluentd, it may be a good way of pushing BOTH to a db and an external source with minimal effort.

cc @DouweM

Combining https://gitlab.com/gitlab-org/gitlab-ee/issues/579#note_19930375 and https://gitlab.com/gitlab-org/gitlab-ee/issues/579#note_28259687, and taking into account all of the information in this thread (like the fact that we already have an application.log and "Audit Events" which are very similar), I suggest we:

For 9.3
Modify AuditEventService, which currently writes directly to the DB, to track all events for a request in memory (local to the request)
At the end of a request, schedule this data using Sidekiq in something like an AuditEventFlusherWorker
Have the Sidekiq worker process this data, iterating over an array of FLUSHERS (subclasses of something like Gitlab::BaseFlusher) and passing each the data to handle as they see fit
Implement a Gitlab::DatabaseFlusher which writes the events to the DB just like AuditEventService does currently
Make sure the DB table doesn't grow indefinitely by implementing a (configurable?) limit of number of months of data
Implement a Gitlab::LogFileFlusher which writes the events to application.log in a human readible format
Replace all Gitlab::AppLogger calls with AuditEventService calls. This will likely involve changing the data model of AuditEvent and AuditEventService
Make sure "Project > Settings > Audit Events" (EE) and "Group > Settings > Audit Events" (EE) can show all types of events
Rename "Profile Settings > Audit Log" to "Authentication Log" and only show authentication AuditEvents for the current user (This replaces https://gitlab.com/gitlab-org/gitlab-ce/issues/30827, since it's not as easy as simply renaming that page.)
Add "Admin Area > Monitoring > Audit Events" to display this audit event data for the entire instance. "Admin Area > Logs > application.log" will continue to show the raw log.
For 9.4
Implement First Step from above description, to the extent where these are not already implemented:
- https://gitlab.com/gitlab-org/gitlab-ee/issues/1370 Log user actions
- https://gitlab.com/gitlab-org/gitlab-ee/issues/1371 Log project actions
- https://gitlab.com/gitlab-org/gitlab-ee/issues/1372 Log group actions
Make sure all events have a consistenct format etc
For later
Consider renaming application.log to audit.log (note that this may create issue for customer currently depending on the name)
Implement more flushers for Syslog etc, potentially using fluentd

@mydigitalself @yorickpeterse What do you think?

If I understand correctly we are talking about the following features:

DB audit log (currently in GitLab, stored in the DB, query via UX)
File audit log (proposed in https://gitlab.com/gitlab-org/gitlab-ee/issues/579#note_28484404 )
Authentication log (currently in GitLab CE for individual users)
Activity log (to show dashboard)
Geo log (consensus is to make this a separate thing, that makes sense)

Are we deprecating any of the above? I don't hear anyone talking about the DB audit log. But I assume we can't live without it. Should the DB and File audit log contain exactly the same types of events to make our lives simpler?

We should not consolidate if it doesn't make sense. But storing actions 5 times will have a performance impact. In https://gitlab.com/gitlab-org/gitlab-ee/issues/2174#note_28405403 I suggested making an overview based on event types.

/cc @dzaporozhets

@DouweM Would we need Sidekiq, or can we somehow insert the data in-request in the most efficient way possible? Using Sidekiq we'd end up with between 20 000 and 30 000 jobs per minute (more or less equal to the number of Rails requests), permanently occupying workers in the process. I'd prefer for something like the following:

We start a request
We perform a bunch of events, the audit events are stored in memory
At the end of the request we insert these events into the DB using a single INSERT
If file logging (or something else) is enabled we also write the data to the file as this is a fairly quick operation
We only use Sidekiq if we need to ship audit data to an external service, if we need this at all

I prefer the approach by https://gitlab.com/gitlab-org/gitlab-ee/issues/579#note_28584801 because my concern is that the audit logs would be lost due to Sidekiq for some reason.

@yorickpeterse That's totally fine by me. I was mostly suggesting writing to DB/file using Sidekiq because of your https://gitlab.com/gitlab-org/gitlab-ee/issues/579#note_19930375 :)

@sytses Note that "DB audit log" and "Authentication log" are actually the same thing, and that we already have a "File audit log" in application.log. My proposal unifies those.

@DouweM Having Gitlab::DatabaseFlusher and Gitlab::LogFileFlusher which does single write operation per request seems like a good idea.

I think duplicate between DB and application.log is inevitable since a lot of external tools rely on log files while UI works well only with DB data.

I also agree on not using Sidekiq internally for flush operations. With single write per flusher it will be a overhead to use Sidekiq.

mentioned in issue #2336 (closed)

removed milestone

mentioned in issue #2384 (closed)

mentioned in issue #2494

changed milestone to %Next 2-3 months

changed the description

marked the checklist item Consolidate existing log entries into a single area in the Admin area https://gitlab.com/gitlab-org/gitlab-ee/issues/2336 as completed

mentioned in issue #3295

I'd like to add another use case along with what's mentioned above. I'd like to know the last time a user performed an action against any piece of the overall GitLab platform. My current problem is I don't necessarily know who is active or not. I'm fairly certain I have users that have logged in 2, 3 or even 4 years ago and have not logged in since. However they could potentially still log in as their identity attribute still maps to a valid entry in the authentication source. I also have to worry about service accounts (not people) that don't ever log into the WebUI but do git push/pulls all the time.

I would consider the following actions to count towards this last action date:

Sign into WebUI
Git push/pull via ssh (if non-public repo)
Git push/pull via https (if non-public repo)
Access API endpoints (via password or token)
Upload Docker image to GitLab Registry
Mattermost … ?? … I don’t actually use mattermost, I’m assuming it’s the same set of users? So however people access this ;)

I think this goes along with the above auditing requests? I wouldn't need it in real time, generating a report of sorts would be acceptable.

@cmacduff I think what you are looking for could be the user activity API: https://docs.gitlab.com/ce/api/users.html#get-user-activities-admin-only that we already have. And reporting-wise perhaps the user cohorts might be helpful: https://docs.gitlab.com/ce/user/admin_area/user_cohorts.html

@jameslopez Ahh thanks, didn't realize that endpoint was available. Is this data different then the "last sign-in at" date in the WebUI for a user in the admin section?

@cmacduff it is different since it also includes other user activity, such as Git operations.

marked the checklist item Log member actions https://gitlab.com/gitlab-org/gitlab-ee/issues/1370 as completed

assigned to @jramsay

changed the description

Audit log improvements (META)

Description

Proposal

Designs

Child items ...

Activity

For the Logs

Documented and in Rails:

Undocumented and in Rails:

Documented and not in Rails:

For the Audit event screen

Admin message

Admin message

Audit log improvements (META)

Description

Proposal

Activity

For the Logs

Documented and in Rails:

Undocumented and in Rails:

Documented and not in Rails:

For the Audit event screen