Commits on Source (6)
-
Sargun Dhillon authored
This upgrade, and vendors aws-sdk-go to version v1.12.36. This is because it has new API calls accessible to the S3 client, specifically S3.ListObjectsV2PagesWithContext Signed-off-by:
Sargun Dhillon <sargun@sargun.me>
-
Sargun Dhillon authored
Move the Walk types into registry/storage/driver, and add a Walk method to each storage driver. Although this is yet another API to implement, there is a fall back implementation that relies on List and Stat. For some filesystems this is very slow. Also, this WalkDir Method conforms better do a traditional WalkDir (a la filepath). This change is in preparation for refactoring. Signed-off-by:
Sargun Dhillon <sargun@sargun.me>
-
Sargun Dhillon authored
This changes the Walk Method used for catalog enumeration. Just to show how much an effect this has on our s3 storage: Original: List calls: 6839 real 3m16.636s user 0m0.000s sys 0m0.016s New: ListObjectsV2 Calls: 1805 real 0m49.970s user 0m0.008s sys 0m0.000s This is because it no longer performs a list and stat per item, and instead is able to use the metadata gained from the list as a replacement to stat. Signed-off-by:
Sargun Dhillon <sargun@sargun.me>
-
Sargun Dhillon authored
This change is primarily to make GC faster. Signed-off-by:
Sargun Dhillon <sargun@sargun.me>
-
Sargun Dhillon authored
This removes the old global walk function, and changes all the code to use the per-driver walk functions. Signed-off-by:
Sargun Dhillon <sargun@sargun.me>
-
Derek McGowan authored
s3: improve walk performance
Showing
- registry/storage/blobstore.go 1 addition, 3 deletionsregistry/storage/blobstore.go
- registry/storage/catalog.go 12 additions, 17 deletionsregistry/storage/catalog.go
- registry/storage/driver/azure/azure.go 6 additions, 0 deletionsregistry/storage/driver/azure/azure.go
- registry/storage/driver/base/base.go 12 additions, 0 deletionsregistry/storage/driver/base/base.go
- registry/storage/driver/filesystem/driver.go 6 additions, 0 deletionsregistry/storage/driver/filesystem/driver.go
- registry/storage/driver/gcs/gcs.go 6 additions, 0 deletionsregistry/storage/driver/gcs/gcs.go
- registry/storage/driver/inmemory/driver.go 6 additions, 0 deletionsregistry/storage/driver/inmemory/driver.go
- registry/storage/driver/oss/oss.go 6 additions, 0 deletionsregistry/storage/driver/oss/oss.go
- registry/storage/driver/s3-aws/s3.go 131 additions, 0 deletionsregistry/storage/driver/s3-aws/s3.go
- registry/storage/driver/s3-goamz/s3.go 6 additions, 0 deletionsregistry/storage/driver/s3-goamz/s3.go
- registry/storage/driver/storagedriver.go 7 additions, 0 deletionsregistry/storage/driver/storagedriver.go
- registry/storage/driver/swift/swift.go 6 additions, 0 deletionsregistry/storage/driver/swift/swift.go
- registry/storage/driver/walk.go 17 additions, 24 deletionsregistry/storage/driver/walk.go
- registry/storage/error.go 9 additions, 0 deletionsregistry/storage/error.go
- registry/storage/linkedblobstore.go 1 addition, 7 deletionsregistry/storage/linkedblobstore.go
- registry/storage/purgeuploads.go 2 additions, 2 deletionsregistry/storage/purgeuploads.go
- registry/storage/purgeuploads_test.go 1 addition, 1 deletionregistry/storage/purgeuploads_test.go
- registry/storage/walk_test.go 0 additions, 152 deletionsregistry/storage/walk_test.go
- vendor.conf 1 addition, 1 deletionvendor.conf
- vendor/github.com/aws/aws-sdk-go/README.md 392 additions, 64 deletionsvendor/github.com/aws/aws-sdk-go/README.md
registry/storage/error.go
0 → 100644
registry/storage/walk_test.go
deleted
100644 → 0
This diff is collapsed.