adding chained aws credentials support

Allows you to run inside AWS and resolve credentials in order

1. static credentials (access keys from ELASTIC_CONNECTION_INFO)
2. EC2 role credentials

Other methods of resolution (Shared and Environment) have been omitted as static credentials are sent from the rails app or they will be resolved from the ec2 instance profile.

@nick.thomas FYI

unmarked as a Work In Progress

changed the description

Thanks @mgresko, a few more comments. cc me on the Rails side when you want me to take a look at it!

https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2020

FYI running this in conjunction with rails changes I am getting "no Elasticsearch node available" when connecting to AWS ES 5.3 cluster. Going to look into this tomorrow. (rails is working fine)

This looks useful... https://github.com/olivere/elastic/wiki/Using-with-AWS-Elasticsearch-Service

@mgresko "no Elasticsearch node available" is due to elastic.SetSniff, I think.

resolved all discussions

@mgresko I wonder if it's due to this code segment: https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer/blob/master/elastic/client.go#L72

I don't recall explicitly testing this on AWS; perhaps setting the scheme like this forces the sniffing to be enabled, despite the later call to elastic.SetSniff(false)

If so, it's a pre-existing bug in the indexer, so this is a bit of a long shot as I assume it's working for you at the moment?

@nick.thomas that is possible. I am going to poke at it today. Definitely does not work for me on AWS. I had to flip back to the ruby indexer to get things going in my test env.

@nick.thomas ok this is not an issue. The problem was the omnibus package somehow built with master vs my branch. I updated the binary on my test environment and it is working perfect. sigh. sorry for the fire drill.

Awesome :) OK, I'm happy with this MR but I'll hold off on merging it until the Ruby one is merged.

assigned to @nick.thomas

mentioned in issue omnibus-gitlab#2458 (closed)

@nick.thomas FYI https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2020 has been merged

@mgresko yes, we're just waiting for https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2458 to be resolved before merging this.

merged

OK, this is merged. I'll cut a 0.2.0 and get it into %9.4

Thanks again @mgresko!

👍 appreciate all the help