GitLab on CephFS
Deprecated issue, Ceph issues are in https://gitlab.com/gitlab-com/infrastructure/issues?scope=all&sort=updated_desc&state=opened&utf8=%E2%9C%93&label_name%5B%5D=ceph&label-name=
Using a virtual server in the cloud as an NFS server will only take you so far in terms of storage capacity. We want to see if CephFS can be used to store Git repository data (and all other file data managed by GitLab). If you are currently evaluating or running Ceph with GitLab please comment below. If you prefer to comment in private please email sytse@ our domain.
Status is:
- Ongoing conversations with providers of support - https://gitlab.com/gitlab-com/operations/issues/202
- Metrics in out current NFS backend server show that IOPS are not as high and latencies are not as low as initially thought https://gitlab.com/gitlab-com/operations/issues/1#note_4795469
- We need to add metrics on the NFS from the client perspective to get a grasp of how this solution behaves.
- Plan ahead using GitLab GEO
Relevant information/talks to watch:
-
Ceph at CERN: A Year in the Life of a Petabyte-Scale Block Storage Service
- Follow up conversation by email with Daniel: https://gitlab.com/gitlab-com/operations/issues/1#note_4518151
- Notes on the CERN video: https://gitlab.com/gitlab-com/operations/issues/1#note_4626461
- Building Your First Ceph Cluster for OpenStack— Fighting for Performance, Solving Tradeoffs
-
Possible candidate as a distributed filesystem: Lustre FS
- http://www.eofs.eu/fileadmin/lad2015/slides/07_Andreas_Dilger_LAD2015-Lustre2_9andBeyond.pdf
- http://lustre.ornl.gov/ecosystem/documents/tutorials/Stearman-LLNL-ZFS.pdf
- As a pro, it is mature and has been in use for quite a long time.
Production work happening over in:
Activity
Really good written use case: http://info.ornl.gov/sites/publications/files/Pub42210.pdf
@jnijhof can you summarize what that PDF says?
@jnijhof If you already read it a summary would be great. This stood out to me: "We observed consistent results and linear scalability at the RADOS level. However, we did not observe the same at the file system level. We have experienced large performance swings during different runs, very low read performance when transfer size is small, and I/O errors tend to happen when system is under stress (more clients and large transfer sizes). These are not particularly reproducible results, but it suggests that there are opportunities for code improvement and maturation."
Tweeted to ask for comments https://twitter.com/gitlab/status/662400094369640448
Hey guys,
I'm in no means an expert on Ceph (yet?) but I'm currently looking to implement it for my own business and I wanted to share one or two things that I found out during my research.
The publication linked above is from 2013 and since then Ceph had some very valuable additions that could work really well for Gitlab's purposes. It also makes the performance findings from the paper a bit obsolete in my opinion.
The first big feature is cache tiering. With cache tiering it's possible to split the storage network in two separate parts; a fast hot storage pool and a slower cold storage pool.
Cache tiers have two modes they can work in, "writeback mode" and "read-only mode" and I think the latter is very interesting for Gitlab. Basically what you do is put a couple of (expensive) SSD driven storage servers in front of your slower cold storage server that can run on traditional (cheaper) harddrives. Ceph will still write to the slower cold storage pool but on reads it will cache the data on the hot storage SSD systems and sequential reads will be read from the SSD directly. I don't have any details on Gitlab's data but I can imagine that more data might be read then it's written and this could be a big benefit in such a secnario. However they do note in the Ceph documentation that it shouldn't be used for mutable data but I'm not sure that definition holds true for git objects, hopefully somebody smarter then me can weigh in on that.
There is second addition to the Ceph codebase since the paper was released which won't really help with the performance but does make it more economically viable to use Ceph with large storage requirements; Erasure code. Erasure code is comparable to traditional RAID-5 with the added benefit that you can decide how much parity you want. The default erasure code profile is setup in such a way that you can lose a single OSD (think disk) at the expense of only 50% of what you would need when you use a replicated system (where each piece of data is stored exactly twice). So in a replicated system you would need 2x1TB to store 1TB but with erasure code you would only need 1.5TB. Erasure code offers a lot of flexibility for the size/redundancy trade-off and I can recommend this blog post for some of the (easy) math behind it. Do note that if you want to use erasure code you probably want a second cache tier in writeback mode because RBD images are not supported on erasure code (they require partial writes which erasure code doesn't support).
Although outdated there are two blogposts by intel that benchmark Ceph and compare it's performance to that of EBS. You can find them here: https://software.intel.com/en-us/blogs/2013/10/25/measure-ceph-rbd-performance-in-a-quantitative-way-part-i and https://software.intel.com/en-us/blogs/2013/11/20/measure-ceph-rbd-performance-in-a-quantitative-way-part-ii
I'm super excited by the promise of Ceph and will try to update this post as I learn more.
Thanks @maran. Our main concern at the moment is the amount of latency on the MDS (metadata server). Git does a lot of file lookups. But it will be great to have a 'hot cache'.
Although outdated there are two blogposts by intel that benchmark Ceph and compare it's performance to that of EBS.
Ceph's EBS alternative is 'Ceph RBD'. We are considering 'Ceph FS', which is a higher-level abstraction (filesystem instead of block device). GitLab needs a single large filesystem to store its Git repositories.
When you go to http://docs.ceph.com/docs/ you get taken to the 0.80.1 documentation. But as @sytses mentioned the current version is 0.94.
Mentions of CephFS in the 0.94 release notes:
CephFS snapshot improvements: Many many bugs have been fixed with CephFS snapshots. Although they are still disabled by default, stability has improved significantly.
CephFS Recovery tools: We have built some journal recovery and diagnostic tools. Stability and performance of single-MDS systems is vastly improved in Giant, and more improvements have been made now in Hammer. Although we still recommend caution when storing important data in CephFS, we do encourage testing for non-critical workloads so that we can better guage the feature, usability, performance, and stability gaps.
@jacobvosmaer Could you not bypass CephFS by exporting on big Ceph RBD and creating a filesystem on top of that? I think I might have been reading slightly outdated docs but I think a lot of warnings were given about using CephFS in production environments.
@maran I read the same warnings about CephFS too. One of the reasons we have this issue out here is to see if anybody out there is using it anyway and what their experience is.
The problem with making a huge RBD disk is: how do you get it to the application servers? We considered clustered file systems but those look limited in the number of clients (application servers) they support. #4 (closed)
If we stick a huge RBD disk behind an NFS server then all file reads/writes + metadata lookups must pass through that one NFS server. The theoretical advantage of CephFS is that read/write actions go straight to OSD's with only the metadata lookups/modifications going to the MDS.
Right what I think you are saying is that you need to use the same filesystem on multiple app servers, with one huge RBD disk that wouldn't work. I haven't really looked into CephFS that deeply so I will abstain myself from making any other comments until I read more about it.
Edited by username-removed-147372https://twitter.com/0xF2/status/659276455445028864
CephFS will be production ready early next year, it claims.
For reference, we can't use s3 or other blobs because performance might suffer as also indicated by a libgit2 developer.
Edited by Sid SijbrandijQuestion is: With a networked and distributed FS like CephFS wouldn't the same issues as with object storage come up? The main performance issues come from having to serially read objects as they reference the next one etc. (simplified). So with a distributed FS the latency issues will be there similarly to using object storage. One thing to circumvent is to have a fast DB for reference storage and only store the raw git object in object storage. Which as long as the object storage can be used with many parallel requests shouldn't be that much of an issue. Concerning the object storage usage. Here are some benchmarks from the research implementation of git using openstack swift. https://docs.google.com/document/d/1IRPYwmSWzsAt9X66Rw9nnAZeTKBi17TMTczsjkZkiTg/edit
Github uses sharding on filesystems and as far as I know it's a pain.
Thanks for chiming in @stp-ip
So with a distributed FS the latency issues will be there similarly to using object storage. One thing to circumvent is to have a fast DB for reference storage and only store the raw git object in object storage.
Don't you need the objects to find other objects?
I believe some work has been done / is being done to store references in a DB instead of files on disk / the 'packed_refs' file. But I have no idea how far along that project is. See e.g. the Git 2.6.0 release notes.
Thanks for the link to those Dulwich/Swift benchmarks, that is interesting and possibly useful for us. (For those who do not know: Dulwich is a re-implementation of Git in Python, just like JGit (Java) and libgit2 (C)).
Don't you need the objects to find other objects?
Yeah that's the main performance issue with networked storage systems and git objects as they need the reference of the current block to grab the next. As far as my knowledge goes. With this latency adds up and slows down retrieval etc. As you are already using a networked storage I do think the performance hit via CephFS should not be that bad for object storage it depends on the latency I assume. So google object storage would be something to consider as their network seems to be the superior.
Edited by username-removed-16605@stp-ip we are not considering building a Git implementation backed by object storage (like Dulwich+Swift you mentioned) at this time. It would take a lot of engineering work and it is uncertain whether it would pay off with a usable (fast enough) Git storage backend.
"we use the Ceph RBD and I don’t think I'm going to be helpful but if I can add something to the gitlab thread I’ll post there. For now I will just be a clearer about what we’re actually doing.
We define the monitors and images mount points in our kubernetes RC. The kubelet in turn mounts the RBD image to the host (a VM in this case), which is then passed through as a volume mount to our docker container. We run a fairly low traffic gitlab for internal things, however we did do some NFT and performance testing that is undertaken by another team and obviously passed those checks. I suspect I can’t show you those numbers, but I’ll see if I can reproduce the latency issues, as I wasn’t aware of anything beyond the underlying limitations of the IaaS.
We do however have separate boxes for Data Store and the Monitor/OSDs, 3 of each at the moment. If I were to turn off an Monitor/OSD box, we’d see some pretty extreme intermittent latency as the client tries to refer to a box that isn’t there. We also previously had these all on the same boxes (in addition to etcd) but cephs-store did a reasonable job of making everything else, most noticeably etcd, to be intermittently unavailable.
This almost certainly isn’t how you’re going to want to use ceph though as it just gives you a single point of failure, but that’s what we do."
@jacobvosmaer is it correct that the above method is not something we can use for GitLab.com?
@sytses sounds like they have a one-server (or rather one-container) GitLab installation with persistent storage backed by Ceph RBD. It is the equivalent of us running gitlab.com on a single AWS server with repository data on an EBS volume: something we have outgrown for some time now.
@jacobvosmaer: Pretty much with the caveat that your EBS solution would be down if the AZ failed whilst the ceph solution would recover to a different AZ.
Extrapolating on that though, obviously you could look at hosting a HA nfs solution on top of a couple of Ceph RBDs but that's a layer of abstraction which one would assume would be better handled by CephFS itself.
@jacobvosmaer and @adegnan thanks for the feedback
I think the conclusion is that since our data will span multiple servers and will be queried by multiple servers we're better off waiting for CephFS to be production ready.
mentioned in issue #42 (closed)
As an alternative idea, has anyone thought about instead of a disk back, using a object store backend like the redis backend to libgit2 https://github.com/libgit2/libgit2-backends
@timhughes Maybe I'm wrong but I don't think that would work for users using Git directly, unless we write something that handles Git payloads and stuffs them in Redis/whatever.
@pcarranza is working on contacting RedHat for expertise. He and @jnijhof will setup a test cluster.
Reassigned to @pcarranza
A lot of interesting information: https://www.reddit.com/r/IAmA/comments/3jdnnd/i_am_sage_weil_lead_architect_and_cocreator_of/
Milestone changed to Scheduled
Feedback from a large user in https://dev.gitlab.org/gitlab/organization/issues/639 (internal)
-
From the ops agenda:
At this point I think we need to start considering setting up and testing Ceph as critical, the path forward is:
- We are going to setup a cluster of ceph servers
- We will kick the tires in this cluster with whatever we can think that is evil enough.
-
We are going to rebuild the host for dev.gitlab.org using the omnibus packageand mount it on the ceph cluster. - We are going to dogfood ourselves with this cluster to see how it behaves and what surprises we have, or what mistakes we did.
- We need good monitoring and tooling to see if there is an actual degradation at the FS level.
- Anything that is not ensuring that GitLab.com can continue growing is a lower priority for operations and will be queued.
- We need to provide good visibility of the situation to the whole company and be extremely vocal, if we are running out of time, everybody should know it, and should know why and what are we doing about it (and is invited to give a hand).
- At this point we roughly know that we are growing fast, but we don’t know if that is on new repos, forks, imports or plain pushes.
- We need support from the developers by building enough tooling to know where the storage usage is going, we also need to know what the trend is and what to expect tomorrow, and in a month.
- If things go wrong, we need to have a plan B and enough time to execute it. This may include development involvement and resourcing.
-
@yorickpeterse rick will help with metrics.
- FS Growth rate and projection
- Source of the growth (new repos, imports, plain pushes, LFS)
Edited by username-removed-274314 -
@stanhu Heh, I found that one 2 days ago and was watching and taking notes.
Thanks to this I am already seeing what are we missing in monitoring.
Will share my notes as soon as I finish analysing it.
Thanks for sharing.
HN commenter mentions that Ceph has a problem with high IOPS loads https://news.ycombinator.com/item?id=11382529 I asked for a link: Mirantis gave a talk last year at OpenStack with real-world numbers... They had to scale to 500 storage nodes to achieve 50k IOPS aggregate. https://www.openstack.org/summit/vancouver-2015/summit-videos/presentation/building-your-first-ceph-cluster-for-openstack-fighting-for-performance-solving-tradeoffs and https://www.slideshare.net/mobile/mirantis/ceph-talk-vancouver-20
Edited by Sid Sijbrandij-
Questions from the ops agenda:
- What could be a good stress plan?
- What could be a good disaster simulation?
- How can we test the application in case of a degradation of the service
- How do we ensure the object placement is optimized for repo data?
- What kind of performance (e.g. IOPS) can we expect?
- What is the optimal hardware and network that should be used for best performance? Pablo: not sure we can answer this, we will be using VIRTS in Azure for now.
- What happens when servers get into an inconsistent state? How is this detected/resolved properly?
- Inconsistent objects are detected by scrubbing and comparing checksums. This rarely happens (<1 per year), but needs to be resolved manually.
- What replication factor makes sense? - 3
- How do we recover from a complete disaster? (quorum down)
-
Thanks @sytses Will check that talk too to get more data.
-
I reached Daniel from CERN directly to ask a few questions from the talk, here are the answers:
- In order to make the change in a controlled fashion I want to make the transition slowly and not just swapping one server for the other, so I need to first measure what I have now, and then measure how Ceph is going to behave as the word out there is that Ceph is bad at IOPS on high load.
Ceph won't give you the same IOPS performance as an expensive NFS appliance with an NVRAM write back cache, for example. But on the contrary I find CephFS to be more performant that the other network filesystems we use here at CERN -- OpenAFS, plain old linux NFS, and a couple others. To make yourself comfortable you really should do extensive testing with Ceph to (a) understand how it works from an ops/reliability point of view and (b) to work out a setup which performs to your requirements
- In the talk read and write latency for specific file sizes are mentioned continuously; how are you measuring this? is Ceph providing this form of data or are you measuring this from the client side?
Ceph has a built in benchmarking tool called 'rados bench'. That can be run interactively by you to see how it performs. We run a few rados bench tests regularly and feed the results into our monitoring.
- How are you measuring IOPS? just plain old iostat? anything fancier than that?
If you have a running Ceph cluster, you can run 'ceph -w' to watch the ongoing user activity, in read/write B/s and ops/s. I really encourage you to get some test hardware and start playing with/testing Ceph. Some people find that it has a bit of a learning curve
We currently replicate git repositories notifying
secondariesand letting them fetching fromprimary(it doesn't require to have a replicated filesystem for that, so anything will handle).Things in
/sharedfolder are a different beast. That covers anyLFS object,attachments,avatars,build-artifacts,pagesetc.They can all be handled by either a replicated Ceph filesystem or we could change code to use a S3 compatible storage (this would give us freedom to choose a proprietary managed solution or any opensource S3 compatible daemon like Ceph, Riak, etc).
With that in mind, let's say we want to get
gitlab.comreplicated with Geo toams.gitlab.com. We don't have to movegitlab.comrepositories to Ceph, we can test that on the Geo instances first:- Setup a Ceph cluster in AMS region, with full read/write and use it for the
/repositoriesfolder. - Setup Database replication
- Setup Geo
- Do first repository import or just "release the kraken" and let it get slowly populated after
git pushto repositories.
This will give us functional read-only web navigation and
git pullfromams.gitlab.com. Anything that we currently store on/sharedwill not be available.To get that we would have to setup another cluster, this time in
gitlab.comand move/sharedfolder there and replicate it toams.gitlab.cominstances.Edited by Gabriel Mazetto- Setup a Ceph cluster in AMS region, with full read/write and use it for the
-
I like the "release the kraken" approach, that will allow us to see how the whole thing behaves.
How does it work in this setup? every time a repo gets a push, the secondary gets notified and pulls the whole thing? do you allow to throttle this behaviour to not hammer the primary? If so, can it be throttled in runtime?
@pcarranza would Chaos Monkey be useful here? GitLab.com is only served from one server right now, as far as I know, so maybe not. But for the future.
-
@connorshea, eventually yes. But this form of high availability is not the exact problem we are looking to solve for now.
Thanks
every time a repo gets a push, the secondary gets notified and pulls the whole thing?
yes, every time there is a push on
primaryserver, we queue that and consume the queue every 10 seconds. This will notifysecondarieswith a list of updated repositories. Secondaries will queue that again (in sidekiq this time) and start consuming (git fetching) immediately after.do you allow to throttle this behaviour to not hammer the primary? If so, can it be throttled in runtime?
The only way we can "throttle" is by changing the amount of sidekiq workers running on secondaries. Currently we use
defaultqueue, if we want fine-grained control, we can move it to a new specific queue.-
@brodock Is there a way of changing the amount of sidekiq workers from the application? can we have a specific queue for this and provide a way of short circuiting it completely? I just want to a have a quick way of stopping or lowering the load in case we are killing the master server.
Another question, how are repos that do not get pushes synced?
@pcarranza can we just shut down GitLab on the secondary servers to throttle to 0% I think
-
@sytses That will be an interesting test for the feature
http://githubengineering.com/introducing-dgit/ "Git is very sensitive to latency. A simple git log or git blame might require thousands of Git objects to be loaded and traversed sequentially. If there’s any latency in these low-level disk accesses, performance suffers dramatically. Thus, storing the repository in a distributed file system is not viable."
I think we should figure out if the latency that CERN had (3ms reads?) is acceptable to us.
-
Ok, I'll try to dump what I got from the CERN video
General view
CERN is using Ceph to serve block devices to mount using Cinder and provision virtual machines in their own OpenStack.
They are not using Cephfs
They use Puppet to deploy the cluster, mixed with the
ceph-diskdeployment tool that is in upstream.Initially they started with 48 OSDs per server and 5 monitors randomly distributed in the room. Each server had 64G of ram and 24 attached disks each. Monitors where basically CPU based hosts.
They have 1000 VMs with attached Ceph volumes.
They are starting to move their storage service into VMs with Ceph volumes.
IOPS
They provision block drives for VMs, they use QOS to mimic the behaviour of physical disks such that there is 2 kind of volume types:
- low IOPS provides 80/80 RW MB/s bandwidth that supports 100/100 IOPS
- high IOPS provides 120/120 RW MB/s 500/500 IOPS both per VM.
They only upgrade a user to a high IOPS block when he complains about performance, there are not so many users complaining.
Comparing to a physical disk, they can't do thousands of sequential IOPS and everything is limited to this numbers because they don't yet have Burst IOPS in Cinder.
Latency
Their main concern is around latency. They are closely monitoring this latency using
rados bench(http://tracker.ceph.com/projects/ceph/wiki/Benchmark_Ceph_Cluster_Performance) concentrating in write latency for files smaller than 4Kb as that is roughly the 70% of the use of the cluster.This latencies are:
- 4kb reads - 3ms
- 4kb writes - 6-8ms
- ~ 7000 IOPS/s in the cluster
General numbers of load in a normal day:
- 322M writes
- 171M reads
- ~ 500M IOPS
- 28% of reads where 512Kb from read ahead.
The write latency was originally 30-40ms with 24 disks hosts with no SSD journal drives (all magnetic drives)
So they moved to SSD drives replacing 4 out of 24 drives for SSD and placing all the journals on these drives decreasing the latency from ~50ms to 5ms and adding a 5-10x IOPS capacity per server with only this. The main reason was because the journal is randomly accessed to read it. The result was to have a flat 5ms latency.
They don't use SSDs for the whole drives because of cost and because throughput is more important than latency and they can get it with magnetic drives.
They monitor
iowaitto check if a user needs a better drive. A better drive reduces the load because the OS is not waiting for the block device to return data.Drive configuration max_sectors_kb defines an artificial limitation for a virtual block device. Given 100 IOPS and a block size of 512kb (default) the max throughput will be 50Mb/s. By defining a bigger block the throughput increases.
Ceph and linux Tunning
- LevelDB for the monitor hosts is really sensitive to disk latency. They use SSDs for them too.
- Scrubbing (checking integrity of the blocks in the cluster) can be really IO intensive. It will compete with the user access to drives slowing down the whole thing.
- Thundering herd - scrubbing is triggered automatically every week by default. If all the OSDs in the cluster are built at the same time they all will trigger this at the same time. Be sure to smear the scrubbing manually (check scripts for that).
- Use CFQ in the data disks.
- Use ionice idle on the OSD disk thread.
- Disable NUMA zone reclaims
vm.zone_reclaim_mode = 0 - Increase max pid on servers
kernel.pid_max = 4194303 - Increase ulimit on clients
ulimit -n 4096at least. - TMalloc causes latency issues. Set
TMALLOC_MAX_TOTAL_THREAD_CACHE_BYTESto 128Mb - disable updatedb or it will use all the IOPS.
- They have really interesting scripts at https://github.com/cernceph/ceph-scripts
- hourly health monitor
- reweight by utilization
- gently drain disks
- find orphaned Cephx keys
ISSUES
- Disks fails monthly, it's transparent to the user
- Twice they lost a whole server - cluster continued working just fine.
- Backfilling is set to host to trigger the backfilling (replication of blocks) when a host goes dark, not a whole rack (by default)
- Power cut killed 3 out of 5 mons not reaching quorum - as a result the OSDs froze and the cluster was down for 18 minutes. They came back again and life continued resuming IOPS, no data corruption was reported.
- A router failure started loosing packets randomly. Some OSDs decided to commit suicide as a safety mechanism. They needed a manual restart. OSDs seems to be talking through broken sockets resulting in 600 seconds operations until they reach the timeout.
- Deletions are slow because they have 1 thread only. Seems to be fixed in hammer
Scaling test with 30Pb
150 clients write throughput with 4Mb objects
Without any failure going on:
- 52Gb/s with no replica
- ~25Bg/s with 1 replica
- ~16Gb/s with 2 replicas
Speed decreases linearly as we have more replicas.
Doing backfilling (host is down) didn't affected throughput significantly.
They seem to be limited by network throughput.
Having a huge cluster means that you will always have backfilling going on.
The main limitation is the OSD map. With 7200 OSDs it is a 4Mb file, but each OSD process caches 500 of these files, limiting the scale. They configured Ceph to cache less maps.
For less than 10Pb cluster it just works
Yahoo is scaling with Ceph but building hashing on top of it -> https://yahooeng.tumblr.com/post/116391291701/yahoo-cloud-object-store-object-storage-at
-
@sytses It depends on a lot of things, consider that a magnetic drive has horrendous latencies from 50 to 100 ms for random access -> https://brooker.co.za/blog/2013/07/14/io-performance.html
So, I would like to do some profiling as soon as possible.
High priority for now is https://gitlab.com/gitlab-com/operations/issues/162 and https://gitlab.com/gitlab-com/operations/issues/163 to starting gathering data of how our current installation is behaving.
On top of that information we will need to start comparing with the profiling information that
rados benchprovides, then we can see if this is a good solution or not. @pcarranza Thanks for the update.
"At the RADOS level replication is synchronous, so it's not going to work well with that kind of latency. At the RGW level it's async, but loses ordering guarantees that you'd need for something like this." https://news.ycombinator.com/item?id=11439448
-
Thanks @sytses from the author of that comment comes this blog post: http://pl.atyp.us/2015-05-dfs-sbfaq.html
A really interesting read.
-
Some metrics I just picked from our NFS host
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util gitlab_vg-gitlab_var 0.00 0.00 219.00 30.00 4.92 0.24 42.47 1.04 4.16 4.35 2.80 3.02 75.20 gitlab_vg-gitlab_var 0.00 0.00 297.00 109.00 4.45 0.54 25.20 1.79 4.17 4.77 2.53 2.34 95.20 gitlab_vg-gitlab_var 0.00 0.00 404.00 141.00 9.96 23.12 124.30 5.12 8.30 5.27 16.99 1.78 96.80 gitlab_vg-gitlab_var 0.00 0.00 401.00 32.00 4.26 0.19 21.04 1.72 5.58 5.80 2.88 2.04 88.40 gitlab_vg-gitlab_var 0.00 0.00 293.00 104.00 10.28 0.53 55.78 1.41 3.55 3.88 2.62 1.70 67.60 gitlab_vg-gitlab_var 0.00 0.00 197.00 24.00 2.82 0.20 28.02 0.80 3.62 3.65 3.33 2.82 62.40 gitlab_vg-gitlab_var 0.00 0.00 259.00 28.00 3.66 0.79 31.75 0.89 3.11 2.92 4.86 2.09 60.00 gitlab_vg-gitlab_var 0.00 0.00 431.00 60.00 4.91 0.36 21.96 1.51 3.08 3.10 2.93 1.73 84.80 gitlab_vg-gitlab_var 0.00 0.00 342.00 191.00 7.18 0.97 31.31 1.78 3.34 3.70 2.70 1.67 88.80 gitlab_vg-gitlab_var 0.00 0.00 150.00 202.00 8.76 1.07 57.23 1.26 3.57 5.47 2.16 1.86 65.60 gitlab_vg-gitlab_var 0.00 0.00 76.00 113.00 2.45 0.65 33.57 0.60 3.17 4.16 2.51 2.16 40.80 gitlab_vg-gitlab_var 0.00 0.00 37.00 89.00 6.02 0.46 105.33 0.55 4.38 8.32 2.74 2.38 30.00 gitlab_vg-gitlab_var 0.00 0.00 80.00 390.00 3.85 144.50 646.43 54.77 60.43 5.55 71.69 1.68 78.80 gitlab_vg-gitlab_var 0.00 0.00 127.00 324.00 8.19 151.94 727.13 133.44 322.57 6.90 446.31 2.22 100.00 gitlab_vg-gitlab_var 0.00 0.00 196.00 58.00 2.61 0.31 23.56 4.47 74.03 6.94 300.76 2.83 72.00 gitlab_vg-gitlab_var 0.00 0.00 130.00 39.00 8.00 0.18 99.03 0.92 5.42 6.25 2.67 3.31 56.00 gitlab_vg-gitlab_var 0.00 0.00 71.00 195.00 6.41 1.02 57.23 1.33 4.05 7.04 2.95 2.62 69.60 gitlab_vg-gitlab_var 0.00 0.00 201.00 107.00 3.75 0.49 28.18 1.29 4.73 5.99 2.36 2.34 72.00 gitlab_vg-gitlab_var 0.00 0.00 362.00 165.00 23.20 20.11 168.27 3.93 7.56 5.83 11.35 1.79 94.40 gitlab_vg-gitlab_var 0.00 0.00 522.00 55.00 59.09 0.57 211.79 3.28 5.75 6.06 2.84 1.41 81.60 gitlab_vg-gitlab_var 0.00 0.00 288.00 9.00 5.23 0.04 36.36 1.20 4.00 4.04 2.67 2.65 78.80 gitlab_vg-gitlab_var 0.00 0.00 271.00 81.00 5.31 0.37 33.05 1.34 3.83 4.00 3.26 2.27 80.00 gitlab_vg-gitlab_var 0.00 0.00 215.00 72.00 11.25 0.51 83.93 1.18 4.11 4.65 2.50 2.44 70.00 gitlab_vg-gitlab_var 0.00 0.00 272.00 95.00 11.09 0.46 64.48 1.32 3.49 3.72 2.82 2.09 76.80 gitlab_vg-gitlab_var 0.00 0.00 206.00 7.00 9.87 0.42 98.89 1.19 5.58 5.53 6.86 3.29 70.00 gitlab_vg-gitlab_var 0.00 0.00 182.00 1931.00 5.58 10.28 15.37 56.23 26.61 4.42 28.70 0.35 73.60 gitlab_vg-gitlab_var 0.00 0.00 207.00 241.00 8.56 1.22 44.73 1.34 2.99 3.71 2.37 1.69 75.60 gitlab_vg-gitlab_var 0.00 0.00 386.00 355.00 34.45 1.73 99.98 3.10 4.18 5.67 2.56 1.30 96.00 gitlab_vg-gitlab_var 0.00 0.00 195.00 134.00 26.30 27.96 337.78 5.64 16.98 7.34 31.01 1.85 60.80 gitlab_vg-gitlab_var 0.00 0.00 223.00 119.00 16.36 0.62 101.68 1.58 4.76 5.79 2.82 2.20 75.20 gitlab_vg-gitlab_var 0.00 0.00 109.00 189.00 6.64 1.17 53.66 1.00 3.37 5.17 2.33 1.97 58.80 gitlab_vg-gitlab_var 0.00 0.00 114.00 168.00 6.30 14.90 153.99 2.02 7.16 4.91 8.69 1.86 52.40 gitlab_vg-gitlab_var 0.00 0.00 211.00 83.00 8.42 0.51 62.23 1.14 3.86 4.45 2.36 1.90 56.00 gitlab_vg-gitlab_var 0.00 0.00 32.00 100.00 2.16 0.54 41.82 0.44 3.33 5.00 2.80 2.42 32.00 gitlab_vg-gitlab_var 0.00 0.00 116.00 61.00 11.62 0.41 139.30 0.89 5.04 6.14 2.95 2.51 44.40 gitlab_vg-gitlab_var 0.00 0.00 127.00 62.00 15.98 0.45 178.07 1.16 5.52 6.87 2.77 2.18 41.20 gitlab_vg-gitlab_var 0.00 0.00 521.00 134.00 15.53 0.79 51.03 2.56 3.97 4.38 2.39 1.43 93.60 gitlab_vg-gitlab_var 0.00 0.00 400.00 133.00 4.59 1.29 22.62 1.52 2.98 3.11 2.59 1.50 80.00 gitlab_vg-gitlab_var 0.00 0.00 255.00 41.00 5.14 0.32 37.76 0.99 3.35 3.45 2.73 2.12 62.80 gitlab_vg-gitlab_var 0.00 0.00 231.00 53.00 3.18 0.37 25.55 0.88 3.11 3.12 3.09 2.20 62.40 gitlab_vg-gitlab_var 0.00 0.00 476.00 184.00 14.48 1.03 48.13 3.27 4.90 5.80 2.57 1.41 93.20 gitlab_vg-gitlab_var 0.00 0.00 143.00 101.00 4.42 2.20 55.51 1.78 7.16 9.76 3.49 4.02 98.00 gitlab_vg-gitlab_var 0.00 0.00 165.00 291.00 13.65 1.54 68.19 2.71 5.18 9.79 2.57 2.19 100.00 gitlab_vg-gitlab_var 0.00 0.00 527.00 291.00 66.62 1.70 171.03 4.98 6.58 8.78 2.58 1.20 98.40 gitlab_vg-gitlab_var 0.00 0.00 571.00 254.00 93.84 1.29 236.16 5.79 6.89 8.74 2.76 1.21 99.60 gitlab_vg-gitlab_var 0.00 0.00 576.00 96.00 83.16 0.61 255.31 5.66 8.59 9.53 2.96 1.43 96.00 gitlab_vg-gitlab_var 0.00 0.00 707.00 65.00 108.62 0.55 289.62 7.35 9.46 10.07 2.89 1.17 90.40 gitlab_vg-gitlab_var 0.00 0.00 463.00 54.00 87.49 0.32 347.82 5.78 11.26 12.24 2.81 1.35 70.00 gitlab_vg-gitlab_var 0.00 0.00 119.00 416.00 6.36 176.31 699.24 74.03 85.78 5.31 108.80 1.58 84.40 gitlab_vg-gitlab_var 0.00 0.00 109.00 233.00 8.56 72.61 486.08 85.70 332.89 22.75 477.97 2.44 83.60Notice r/s and w/s - that's how many read and write requests we are getting per second (IOPS).
For what I see we are getting a maximum of ~2100 IOPS per seconds in the worse case, with an average of ~434 and a standard deviation of ~301, so it's a long tail. That tail comes from the writes as we have specific spikes of up to ~1900 IOPS but a stdev of ~270.
Also notice r_wait, that is the latency in milliseconds (the time it takes to fulfil a read request, from issuing to serving it). It never goes under 3ms (2.92 to be precise) with spikes up to 12ms. ~6ms on avg, ~3ms for stdev. It's quite stable. This replies to the "is this enough for us" question for the time being.
The write (w_wait) case is much worse ranging from 3ms up to ~500ms. The avg is ~32ms and the stdev ~100ms, so it looks quite bursty.
This is just one sample, I'm working in making this available in check_mk so we can monitor it the whole time.
-
I just added metrics to checkmk for the striped drive in the NFS server
Edited by username-removed-274314 mentioned in issue support-forum#453
-
cc @pmq20
Interesting presentation from Alibaba how they scaled GitLab with an objects store and caching repo's on the application server https://speakerdeck.com/pmq20/how-we-scaled-gitlab-for-a-30k-employee-company /cc @pcarranza @jacobvosmaer @dzaporozhets
Milestone changed to Milestone-1
Added availability important labels
Added direction label
@patricio @northrup Reading https://hal.inria.fr/hal-00789086/file/a_survey_of_dfs.pdf I get the impression the Lustre doesn't offer redundancy nor rebalancing of files. I think those are the hardest features, without having those I think building our own technology (each repo points to a fileserver in our database) is easier. What do you think? /cc @jacobvosmaer @stanhu
Perhaps a lot has changed since 2013 when that paper was written. Lustre seems to include documentation on rebalancing:
As for redundancy, I'm reading stuff that Lustre needs to be used with RAID or DRBD. What's the story on that, @northrup?
Generally what you do is use the Lustre system and mount the same chunk of disk space on multiple OSS (Object Storage Server) devices. Lustre supports 'striping' across OSS/OST targets so that files are split between multiple storage targets and a parity checksum is calculated and stored in case of an OSS/OST loss. Things have changed since 2013 when that paper was written, I've also been reading up on Ceph to make sure everything is getting a fair shake. @pcarranza and I talked about this and I'll be conducting an apples-to-apples comparison of CephFS and Lustre w/ load testing and documenting the results. I'll see which one breaks first and what the performance characteristics are of each of them. After we have some actionable data about how things behave under our specific environment we can circle back again and I'll present the findings.
-
related: https://gitlab.com/gitlab-org/gitlab-ee/issues/583
I would like to use that feature to make the test in production and get real data.
-
In the meantime, we are working on writing the high level plan so we can discuss it, as @northrup mentioned.
-
@northrup please document also whatever is part of the setup and maintenance cost: if a given infrastructure is harder to maintain or monitor, that will be something critical to know before embracing it completely.
We're discussing the Ceph architecture in https://docs.google.com/document/d/1ZynjnAZ5NX5jhBxAQrho_47CQrLpTPRqhISrvFldP5M/edit#
We'll focus exclusively on Ceph for now, not on Lustre.
Performance Stats for the 3 node DEV cluster:
Read/Write Mixed Test on 3:1 Ratio
Command Issued: /home/jjn/fio/fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75 Total Running Time: 43 minutes 17.44 seconds Results: test: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64 fio-2.11 Starting 1 process test: Laying out IO file(s) (1 file(s) / 4096MB) Jobs: 1 (f=1): [m(1)] [100.0% done] [1141KB/440KB/0KB /s] [285/110/0 iops] [eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=65029: Sun May 29 07:21:13 2016 read : io=3070.4MB, bw=1210.5KB/s, iops=302, runt=2597437msec bw (KB /s): min= 186, max= 1736, per=100.00%, avg=1215.71, stdev=136.48 write: io=1025.8MB, bw=414072B/s, iops=101, runt=2597437msec bw (KB /s): min= 8, max= 560, per=100.00%, avg=406.04, stdev=44.48 cpu : usr=0.36%, sys=1.20%, ctx=1052920, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued : total=r=785996/w=262580/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: io=3070.4MB, aggrb=1210KB/s, minb=1210KB/s, maxb=1210KB/s, mint=2597437msec, maxt=2597437msec WRITE: io=1025.8MB, aggrb=404KB/s, minb=404KB/s, maxb=404KB/s, mint=2597437msec, maxt=2597437msecRandom Read Testing on 100% Reads
Command Issued: /home/jjn/fio/fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randread Total Running Time: 21 minutes 28.32 seconds Results: test: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64 fio-2.11 Starting 1 process Jobs: 1 (f=1): [r(1)] [100.0% done] [3395KB/0KB/0KB /s] [848/0/0 iops] [eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=25919: Sun May 29 07:54:53 2016 read : io=4096.0MB, bw=3255.7KB/s, iops=813, runt=1288320msec bw (KB /s): min= 2288, max= 3872, per=100.00%, avg=3255.35, stdev=153.45 cpu : usr=0.67%, sys=2.21%, ctx=1053053, majf=0, minf=73 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued : total=r=1048576/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: io=4096.0MB, aggrb=3255KB/s, minb=3255KB/s, maxb=3255KB/s, mint=1288320msec, maxt=1288320msecRandom Write Testing on 100% Writes
Command Issued: /home/jjn/fio/fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randwrite Total Running Time: 1 hour 44 minutes 19.79 seconds Results: test: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64 fio-2.11 Starting 1 process Jobs: 1 (f=1): [w(1)] [100.0% done] [0KB/740KB/0KB /s] [0/185/0 iops] [eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=54567: Sun May 29 09:40:29 2016 write: io=4096.0MB, bw=686120B/s, iops=167, runt=6259787msec bw (KB /s): min= 16, max= 776, per=100.00%, avg=670.50, stdev=46.48 cpu : usr=0.18%, sys=0.63%, ctx=1054848, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued : total=r=0/w=1048576/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): WRITE: io=4096.0MB, aggrb=670KB/s, minb=670KB/s, maxb=670KB/s, mint=6259787msec, maxt=6259787msec-
Just had a call with @jnijhof we should also push the linux kernel repo to each new cluster we build, time it, and compare the timing with GitLab.com to have a point of comparison that is simple and direct than talking about raw IOPs
Our development cluster setup: https://dev.gitlab.org/cookbooks/chef-repo/blob/master/doc/ceph.md
Our production (still test) cluster setup is exactly the same except for:
- VM type: Standard_DS14_v2 (network very high)
- Separate SSD for journal
We will run the following tests for performance:
- Use fio in current setup to see if we already are benefiting the different VM size
- Add extra osd services per osd server let say 2 more so in total 3 per server and test with fio again
- Tweak network kernel settings and test with fio again
- Add an extra osd server and test with fio again
For note taking purposes, I have issued labels for the rounds that @jnijhof and I discussed:
- Test Round 1 (TR1) Use fio in current setup to see if we already are benefiting the different VM size
- Test Round 2 (TR2) Add extra osd services per osd server let say 2 more so in total 3 per server and test with fio again
- Test Round 3 (TR3) Tweak network kernel settings and test with fio again
- Test Round 4 (TR4) Add an extra osd server and test with fio again
Results of Test Route 1 (TR1)
Read/Write Mixed Test on 3:1 Ratio
Command Issued: /root/fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75 Total Running Time: 52.19 seconds Results: test: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64 fio-2.11 Starting 1 process test: Laying out IO file(s) (1 file(s) / 4096MB) Jobs: 1 (f=1): [m(1)] [100.0% done] [58210KB/19655KB/0KB /s] [14.6K/4913/0 iops] [eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=20245: Mon May 30 20:00:44 2016 read : io=3070.4MB, bw=60239KB/s, iops=15059, runt= 52192msec bw (KB /s): min=30256, max=64696, per=100.00%, avg=60251.08, stdev=4251.25 write: io=1025.8MB, bw=20124KB/s, iops=5031, runt= 52192msec bw (KB /s): min= 9992, max=20920, per=100.00%, avg=20126.69, stdev=1353.76 cpu : usr=2.90%, sys=11.34%, ctx=1399737, majf=0, minf=592 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued : total=r=785996/w=262580/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: io=3070.4MB, aggrb=60238KB/s, minb=60238KB/s, maxb=60238KB/s, mint=52192msec, maxt=52192msec WRITE: io=1025.8MB, aggrb=20124KB/s, minb=20124KB/s, maxb=20124KB/s, mint=52192msec, maxt=52192msecRandom Read Testing on 100% Reads
Command Issued: /root/fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randread Total Running Time: 34.18 seconds Results: test: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64 fio-2.11 Starting 1 process Jobs: 1 (f=1): [r(1)] [100.0% done] [118.1MB/0KB/0KB /s] [30.5K/0/0 iops] [eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=20310: Mon May 30 20:01:39 2016 read : io=4096.0MB, bw=122698KB/s, iops=30674, runt= 34184msec bw (KB /s): min=73560, max=145640, per=99.98%, avg=122677.29, stdev=11157.50 cpu : usr=3.77%, sys=13.71%, ctx=1362547, majf=0, minf=539 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued : total=r=1048576/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: io=4096.0MB, aggrb=122697KB/s, minb=122697KB/s, maxb=122697KB/s, mint=34184msec, maxt=34184msecRandom Write Testing on 100% Writes
Command Issued: /root/fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randwrite Total Running Time: 3 minutes 26.47 seconds Results: test: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64 fio-2.11 Starting 1 process Jobs: 1 (f=1): [w(1)] [100.0% done] [0KB/20412KB/0KB /s] [0/5103/0 iops] [eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=20492: Mon May 30 20:05:16 2016 write: io=4096.0MB, bw=20315KB/s, iops=5078, runt=206466msec bw (KB /s): min=12184, max=20936, per=100.00%, avg=20314.60, stdev=521.05 cpu : usr=1.07%, sys=3.15%, ctx=1151119, majf=0, minf=707 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued : total=r=0/w=1048576/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): WRITE: io=4096.0MB, aggrb=20314KB/s, minb=20314KB/s, maxb=20314KB/s, mint=206466msec, maxt=206466msecFindings of TR1
Read/Write Mixed Test
- A 99.97% Reduction in Time
- A 4,886% Increase in Read IOPS
- A 4,881% Increase in Write IOPS
Random Read Testing
- A 97.35% Reduction in Time
- A 3,673% Increase in IOPS
Random Write Testing
- A 96.7% Reduction in Time
- A 2,941% Increase in IOPS
Edited by John NorthrupStaring tests/tweaks on DEV:
- Test Round 1 (TR1) Tweak network kernel settings and test with fio again
- Test Round 2 (TR2) Change OSD's to have a SSD for journal and test with fio again
- Test Round 3 (TR3) Add extra osd services per osd server let say 1 more so in total 2 per server and test with fio again
- Test Round 4 (TR4) Add an extra osd server and test with fio again
Edited by username-removed-29582Test Round 1 No improvements what so ever after the following network tuning (on all nodes incl. dev):
net.core.netdev_max_backlog=300000 net.core.optmem_max=524287 net.core.rmem_default=524287 net.core.rmem_max=524287 net.core.somaxconn=1024 net.core.wmem_default=524287 net.core.wmem_max=524287But after the following tuning I did see some improvements but not much:
net.core.netdev_max_backlog=300000 net.core.optmem_max=40960 net.core.rmem_default=56623104 net.core.rmem_max=56623104 net.core.somaxconn=1024 net.core.wmem_default=56623104 net.core.wmem_max=56623104 net.ipv4.tcp_fin_timeout=10 net.ipv4.tcp_max_syn_backlog=30000 net.ipv4.tcp_max_tw_buckets=2000000 net.ipv4.tcp_rmem=4096 87380 56623104 net.ipv4.tcp_slow_start_after_idle=0 net.ipv4.tcp_tw_recycle=1 net.ipv4.tcp_tw_reuse=1 net.ipv4.tcp_wmem=4096 87380 56623104 net.ipv4.udp_rmem_min=8192 net.ipv4.udp_wmem_min=8192/root/fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=100M --readwrite=randrw --rwmixread=75 test: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64 fio-2.11 Starting 1 process Jobs: 1 (f=1): [m(1)] [100.0% done] [1229KB/388KB/0KB /s] [307/97/0 iops] [eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=45517: Thu Jun 2 09:48:52 2016 read : io=76828KB, bw=1234.5KB/s, iops=308, runt= 62235msec bw (KB /s): min= 880, max= 1488, per=99.89%, avg=1232.66, stdev=116.64 write: io=25572KB, bw=420755B/s, iops=102, runt= 62235msec bw (KB /s): min= 296, max= 520, per=99.87%, avg=409.48, stdev=38.50 cpu : usr=0.44%, sys=1.27%, ctx=25824, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued : total=r=19207/w=6393/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: io=76828KB, aggrb=1234KB/s, minb=1234KB/s, maxb=1234KB/s, mint=62235msec, maxt=62235msec WRITE: io=25572KB, aggrb=410KB/s, minb=410KB/s, maxb=410KB/s, mint=62235msec, maxt=62235msecEdited by username-removed-29582Test Round 2 Lets setup the OSD's the right way and see how it goes:
- recreate osd 1 per 1 with ssd journal
- wait for recovery before testing
Still same numbers as with Test Round 1
Edited by username-removed-29582Test Round 3 (TR3) Lets create an extra osd per existing osd server:
- add two data disks to every osd server
- ceph-deploy create --fs-type xfs ceph-osdX:sde:sdf
- wait for recovery/replacements to finish
root@dev:~/ceph-dev# ceph osd tree ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -1 5.99094 root default -3 1.99698 host ceph-osd2 1 0.99849 osd.1 up 1.00000 1.00000 4 0.99849 osd.4 up 1.00000 1.00000 -4 1.99698 host ceph-osd3 3 0.99849 osd.3 up 1.00000 1.00000 5 0.99849 osd.5 up 1.00000 1.00000 -2 1.99698 host ceph-osd1 0 0.99849 osd.0 up 1.00000 1.00000 2 0.99849 osd.2 up 1.00000 1.00000Still same numbers as with Test Round 1 and 2
Edited by username-removed-29582Test Round 5 (TR5) With two extra osd servers:
root@dev:~# ceph osd tree ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -1 4.99245 root default -3 0.99849 host ceph-osd2 1 0.99849 osd.1 up 1.00000 1.00000 -4 0.99849 host ceph-osd3 3 0.99849 osd.3 up 1.00000 1.00000 -2 0.99849 host ceph-osd1 0 0.99849 osd.0 up 1.00000 1.00000 -5 0.99849 host ceph-osd4 2 0.99849 osd.2 up 1.00000 1.00000 -6 0.99849 host ceph-osd5 4 0.99849 osd.4 up 1.00000 1.00000I stopped testing with fio since it keeps giving me the same numbers all the time which is hardly to believe since we completely changes the setup.
I start testing with dd after cache drop to see how the writes go:
root@dev:/var/opt/gitlab/git-data# sync root@dev:/var/opt/gitlab/git-data# echo 3 > /proc/sys/vm/drop_caches root@dev:/var/opt/gitlab/git-data# dd if=/dev/zero of=test bs=4k count=100000 100000+0 records in 100000+0 records out 409600000 bytes (410 MB, 391 MiB) copied, 0.437797 s, 936 MB/sI tested it with several sizes but keep on getting around the 936 MB/s
Edited by username-removed-29582Reassigned to @jnijhof
First push to DEV (limited by my wifi speed: which is a good sign!):
jeroen@xps15:~/src/linux$ git push Counting objects: 4787258, done. Delta compression using up to 8 threads. Compressing objects: 100% (789200/789200), done. Writing objects: 100% (4787258/4787258), 1.49 GiB | 5.90 MiB/s, done. Total 4787258 (delta 3965592), reused 4787255 (delta 3965589)Git clone on cephfs from linux.git to test:
git@dev:~/git-data/repositories/jeroen$ time git clone linux.git test Cloning into 'test'... done. Checking out files: 100% (54398/54398), done. real 2m4.986s user 0m6.436s sys 0m3.504svs. current gitlab.com NFS solution:
git@worker11:~/git-data/repositories/jnijhof$ time git clone linux.git test Cloning into 'test'... done. Checking out files: 100% (51570/51570), done. real 17m25.061s user 0m6.252s sys 0m8.932sEdited by username-removed-29582mentioned in issue #265 (closed)
-
@jnijhof could you please time the git push? in GitLab.com it was something like 1:30hs for me when I tested.
I would like to compare apples to apples.
Those numbers are really promising, BTW! great job!
From https://gitlab.com/gitlab-com/operations/issues/265#note_12311762:
Some tests that could be run (each on CephFS, NFS, and local):
- Check
git ls-files -rt --name-only <branch>. This recursively lists each file/directory of the repo. Results in a lot of serial disk reads, but some could be parallel (no idea how git implements it internally). With some scripting, printing out the last commit that affected the file is also possible, but that might not be a very good test. I do not believe that GitLab actually runs this anywhere (at most, just a single directory in the tree view), but the timing might be more useful. - Test perform large merges and diffs (not sure where to get any; something like Google Chrome might have some good ones), both with merge conflicts and without.
- Test large rebases. (might be similar to large merges though)
Additional tips:
- Use a repo that has a large number of objects in a single tree (e.g. https://github.com/CocoaPods/Specs/tree/736c527a1ed42707ec31abd5710e5f3a15366c1a, with 16K+ direct children of the "Specs" tree).
- See if you can measure the IO operations (both disk and network) of the commands you run.
- When running locally, there obviously won't be the overhead of sending commands or receiving the result over the network.
- Check
-
thanks @rabbitfang
I particularly like the "Use a repo that has a large number of objects in a single tree" sample as it could show how ceph behaves with a lot of small files.
@jnijhof In terms of number of direct child files and directories, the largest directory in the Linux repo is
arch/arm/boot/dtswith 1374 children. On the CocoaPods/Specs repo, the largest is theSpecsdirectory with 19383 direct children.I used this command to get these values (ignores directories starting with
.):find . \( -name .git -prune \) -o -type d -print | awk '{ system("ls -al '"'"'" $0 "'"'"' | wc -l | awk '"'"'{print $1}'"'"'"); print $1 }' | paste -s -d' \n' - | sort -k 1,1 -n -r | head -n 50@rabbitfang thanks
I tested with
git pushandgit clonea couple of times with flushing cache and without and the timings keeps the same which is really good news.Next step: test with different placement groups numbers but not sure if we can even change that on a running cluster. I have followed the Ceph guide so the actual numbers used are correct but I want to find out if we can change that. I.e. when we are extending Cephfs we need to make sure the amount of PG's are still right.
-
This looks like the next big win we can have in performance as it looks like now all our ssh time is going into NFS and not into rails anymore.
I think that the way to go here is to start getting production ready for building a cluster so we can start migrating things. To do so we need:
- Automation to build the cluster (I think this is covered already)
- Automation to add new OSDs, to rebalance things, to add new hosts
- Monitoring, monitoring and a bit more monitoring.
- Test failure scenarios througfully <- kill the MDS server and see what happens, go below quorum and see what happens, etc.
- Runbooks - what happens when/how do I
- Deliver https://gitlab.com/gitlab-org/gitlab-ee/issues/583 so we have a way to start pushing load into CephFS slowly
Shall we spin off specific tasks for this?
Edited by username-removed-274314 Milestone changed to %Milestone-3
Marked the task We need to add metrics on the NFS from the client perspective to get a grasp of how this solution behaves. as completed
mentioned in issue infrastructure#135 (closed)
mentioned in issue infrastructure#136 (closed)
mentioned in issue infrastructure#137 (closed)
mentioned in issue infrastructure#138 (closed)
See https://docs.google.com/document/d/1R3-nWgS2MU0MHFImzMu6ZIXPKZbRkQwu-9uQTsAAZLM/edit# for our Ceph testing plan.
-
We have a cluster setup in production and are building the second NFS server in parallel.
Please track progress here: https://gitlab.com/gitlab-com/infrastructure/issues/241
Last known state is that we are trying to find a way around connecting 2 networks inside Azure, one from the classic portal, and the ceph cluster that lives in the new portal. @northrup is in a call with Microsoft people right now trying to find a way of doing it.
Worst case scenario we will need to migrate GitLab to the new side of the portal as a whole.
Mentioned in issue #7 (closed)
