Skip to content
Snippets Groups Projects
Unverified Commit 6c7dd33b authored by Kamil Trzcinski's avatar Kamil Trzcinski
Browse files

Add support for additional storages

parent 5b61c8f6
No related branches found
No related tags found
No related merge requests found
Showing
with 6607 additions and 7 deletions
Loading
Loading
@@ -17,16 +17,16 @@ import (
"github.com/opencontainers/go-digest"
 
// Include all storage drivers
//_ "github.com/docker/distribution/registry/storage/driver/azure"
_ "github.com/docker/distribution/registry/storage/driver/azure"
_ "github.com/docker/distribution/registry/storage/driver/filesystem"
_ "github.com/docker/distribution/registry/storage/driver/gcs"
_ "github.com/docker/distribution/registry/storage/driver/inmemory"
//_ "github.com/docker/distribution/registry/storage/driver/middleware/cloudfront"
//_ "github.com/docker/distribution/registry/storage/driver/middleware/redirect"
//_ "github.com/docker/distribution/registry/storage/driver/oss"
//_ "github.com/docker/distribution/registry/storage/driver/s3-aws"
//_ "github.com/docker/distribution/registry/storage/driver/s3-goamz"
//_ "github.com/docker/distribution/registry/storage/driver/swift"
_ "github.com/docker/distribution/registry/storage/driver/middleware/cloudfront"
_ "github.com/docker/distribution/registry/storage/driver/middleware/redirect"
_ "github.com/docker/distribution/registry/storage/driver/oss"
_ "github.com/docker/distribution/registry/storage/driver/s3-aws"
_ "github.com/docker/distribution/registry/storage/driver/s3-goamz"
_ "github.com/docker/distribution/registry/storage/driver/swift"
)
 
var (
Loading
Loading
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2016 Microsoft Corporation
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
This diff is collapsed.
// Package storage provides clients for Microsoft Azure Storage Services.
package storage
import (
"bytes"
"encoding/base64"
"encoding/json"
"encoding/xml"
"errors"
"fmt"
"io"
"io/ioutil"
"net/http"
"net/url"
"regexp"
"sort"
"strconv"
"strings"
)
const (
// DefaultBaseURL is the domain name used for storage requests when a
// default client is created.
DefaultBaseURL = "core.windows.net"
// DefaultAPIVersion is the Azure Storage API version string used when a
// basic client is created.
DefaultAPIVersion = "2015-02-21"
defaultUseHTTPS = true
// StorageEmulatorAccountName is the fixed storage account used by Azure Storage Emulator
StorageEmulatorAccountName = "devstoreaccount1"
// StorageEmulatorAccountKey is the the fixed storage account used by Azure Storage Emulator
StorageEmulatorAccountKey = "Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=="
blobServiceName = "blob"
tableServiceName = "table"
queueServiceName = "queue"
fileServiceName = "file"
storageEmulatorBlob = "127.0.0.1:10000"
storageEmulatorTable = "127.0.0.1:10002"
storageEmulatorQueue = "127.0.0.1:10001"
)
// Client is the object that needs to be constructed to perform
// operations on the storage account.
type Client struct {
// HTTPClient is the http.Client used to initiate API
// requests. If it is nil, http.DefaultClient is used.
HTTPClient *http.Client
accountName string
accountKey []byte
useHTTPS bool
baseURL string
apiVersion string
}
type storageResponse struct {
statusCode int
headers http.Header
body io.ReadCloser
}
type odataResponse struct {
storageResponse
odata odataErrorMessage
}
// AzureStorageServiceError contains fields of the error response from
// Azure Storage Service REST API. See https://msdn.microsoft.com/en-us/library/azure/dd179382.aspx
// Some fields might be specific to certain calls.
type AzureStorageServiceError struct {
Code string `xml:"Code"`
Message string `xml:"Message"`
AuthenticationErrorDetail string `xml:"AuthenticationErrorDetail"`
QueryParameterName string `xml:"QueryParameterName"`
QueryParameterValue string `xml:"QueryParameterValue"`
Reason string `xml:"Reason"`
StatusCode int
RequestID string
}
type odataErrorMessageMessage struct {
Lang string `json:"lang"`
Value string `json:"value"`
}
type odataErrorMessageInternal struct {
Code string `json:"code"`
Message odataErrorMessageMessage `json:"message"`
}
type odataErrorMessage struct {
Err odataErrorMessageInternal `json:"odata.error"`
}
// UnexpectedStatusCodeError is returned when a storage service responds with neither an error
// nor with an HTTP status code indicating success.
type UnexpectedStatusCodeError struct {
allowed []int
got int
}
func (e UnexpectedStatusCodeError) Error() string {
s := func(i int) string { return fmt.Sprintf("%d %s", i, http.StatusText(i)) }
got := s(e.got)
expected := []string{}
for _, v := range e.allowed {
expected = append(expected, s(v))
}
return fmt.Sprintf("storage: status code from service response is %s; was expecting %s", got, strings.Join(expected, " or "))
}
// Got is the actual status code returned by Azure.
func (e UnexpectedStatusCodeError) Got() int {
return e.got
}
// NewBasicClient constructs a Client with given storage service name and
// key.
func NewBasicClient(accountName, accountKey string) (Client, error) {
if accountName == StorageEmulatorAccountName {
return NewEmulatorClient()
}
return NewClient(accountName, accountKey, DefaultBaseURL, DefaultAPIVersion, defaultUseHTTPS)
}
//NewEmulatorClient contructs a Client intended to only work with Azure
//Storage Emulator
func NewEmulatorClient() (Client, error) {
return NewClient(StorageEmulatorAccountName, StorageEmulatorAccountKey, DefaultBaseURL, DefaultAPIVersion, false)
}
// NewClient constructs a Client. This should be used if the caller wants
// to specify whether to use HTTPS, a specific REST API version or a custom
// storage endpoint than Azure Public Cloud.
func NewClient(accountName, accountKey, blobServiceBaseURL, apiVersion string, useHTTPS bool) (Client, error) {
var c Client
if accountName == "" {
return c, fmt.Errorf("azure: account name required")
} else if accountKey == "" {
return c, fmt.Errorf("azure: account key required")
} else if blobServiceBaseURL == "" {
return c, fmt.Errorf("azure: base storage service url required")
}
key, err := base64.StdEncoding.DecodeString(accountKey)
if err != nil {
return c, fmt.Errorf("azure: malformed storage account key: %v", err)
}
return Client{
accountName: accountName,
accountKey: key,
useHTTPS: useHTTPS,
baseURL: blobServiceBaseURL,
apiVersion: apiVersion,
}, nil
}
func (c Client) getBaseURL(service string) string {
scheme := "http"
if c.useHTTPS {
scheme = "https"
}
host := ""
if c.accountName == StorageEmulatorAccountName {
switch service {
case blobServiceName:
host = storageEmulatorBlob
case tableServiceName:
host = storageEmulatorTable
case queueServiceName:
host = storageEmulatorQueue
}
} else {
host = fmt.Sprintf("%s.%s.%s", c.accountName, service, c.baseURL)
}
u := &url.URL{
Scheme: scheme,
Host: host}
return u.String()
}
func (c Client) getEndpoint(service, path string, params url.Values) string {
u, err := url.Parse(c.getBaseURL(service))
if err != nil {
// really should not be happening
panic(err)
}
// API doesn't accept path segments not starting with '/'
if !strings.HasPrefix(path, "/") {
path = fmt.Sprintf("/%v", path)
}
if c.accountName == StorageEmulatorAccountName {
path = fmt.Sprintf("/%v%v", StorageEmulatorAccountName, path)
}
u.Path = path
u.RawQuery = params.Encode()
return u.String()
}
// GetBlobService returns a BlobStorageClient which can operate on the blob
// service of the storage account.
func (c Client) GetBlobService() BlobStorageClient {
return BlobStorageClient{c}
}
// GetQueueService returns a QueueServiceClient which can operate on the queue
// service of the storage account.
func (c Client) GetQueueService() QueueServiceClient {
return QueueServiceClient{c}
}
// GetTableService returns a TableServiceClient which can operate on the table
// service of the storage account.
func (c Client) GetTableService() TableServiceClient {
return TableServiceClient{c}
}
// GetFileService returns a FileServiceClient which can operate on the file
// service of the storage account.
func (c Client) GetFileService() FileServiceClient {
return FileServiceClient{c}
}
func (c Client) createAuthorizationHeader(canonicalizedString string) string {
signature := c.computeHmac256(canonicalizedString)
return fmt.Sprintf("%s %s:%s", "SharedKey", c.getCanonicalizedAccountName(), signature)
}
func (c Client) getAuthorizationHeader(verb, url string, headers map[string]string) (string, error) {
canonicalizedResource, err := c.buildCanonicalizedResource(url)
if err != nil {
return "", err
}
canonicalizedString := c.buildCanonicalizedString(verb, headers, canonicalizedResource)
return c.createAuthorizationHeader(canonicalizedString), nil
}
func (c Client) getStandardHeaders() map[string]string {
return map[string]string{
"x-ms-version": c.apiVersion,
"x-ms-date": currentTimeRfc1123Formatted(),
}
}
func (c Client) getCanonicalizedAccountName() string {
// since we may be trying to access a secondary storage account, we need to
// remove the -secondary part of the storage name
return strings.TrimSuffix(c.accountName, "-secondary")
}
func (c Client) buildCanonicalizedHeader(headers map[string]string) string {
cm := make(map[string]string)
for k, v := range headers {
headerName := strings.TrimSpace(strings.ToLower(k))
match, _ := regexp.MatchString("x-ms-", headerName)
if match {
cm[headerName] = v
}
}
if len(cm) == 0 {
return ""
}
keys := make([]string, 0, len(cm))
for key := range cm {
keys = append(keys, key)
}
sort.Strings(keys)
ch := ""
for i, key := range keys {
if i == len(keys)-1 {
ch += fmt.Sprintf("%s:%s", key, cm[key])
} else {
ch += fmt.Sprintf("%s:%s\n", key, cm[key])
}
}
return ch
}
func (c Client) buildCanonicalizedResourceTable(uri string) (string, error) {
errMsg := "buildCanonicalizedResourceTable error: %s"
u, err := url.Parse(uri)
if err != nil {
return "", fmt.Errorf(errMsg, err.Error())
}
cr := "/" + c.getCanonicalizedAccountName()
if len(u.Path) > 0 {
cr += u.EscapedPath()
}
return cr, nil
}
func (c Client) buildCanonicalizedResource(uri string) (string, error) {
errMsg := "buildCanonicalizedResource error: %s"
u, err := url.Parse(uri)
if err != nil {
return "", fmt.Errorf(errMsg, err.Error())
}
cr := "/" + c.getCanonicalizedAccountName()
if len(u.Path) > 0 {
// Any portion of the CanonicalizedResource string that is derived from
// the resource's URI should be encoded exactly as it is in the URI.
// -- https://msdn.microsoft.com/en-gb/library/azure/dd179428.aspx
cr += u.EscapedPath()
}
params, err := url.ParseQuery(u.RawQuery)
if err != nil {
return "", fmt.Errorf(errMsg, err.Error())
}
if len(params) > 0 {
cr += "\n"
keys := make([]string, 0, len(params))
for key := range params {
keys = append(keys, key)
}
sort.Strings(keys)
for i, key := range keys {
if len(params[key]) > 1 {
sort.Strings(params[key])
}
if i == len(keys)-1 {
cr += fmt.Sprintf("%s:%s", key, strings.Join(params[key], ","))
} else {
cr += fmt.Sprintf("%s:%s\n", key, strings.Join(params[key], ","))
}
}
}
return cr, nil
}
func (c Client) buildCanonicalizedString(verb string, headers map[string]string, canonicalizedResource string) string {
contentLength := headers["Content-Length"]
if contentLength == "0" {
contentLength = ""
}
canonicalizedString := fmt.Sprintf("%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s",
verb,
headers["Content-Encoding"],
headers["Content-Language"],
contentLength,
headers["Content-MD5"],
headers["Content-Type"],
headers["Date"],
headers["If-Modified-Since"],
headers["If-Match"],
headers["If-None-Match"],
headers["If-Unmodified-Since"],
headers["Range"],
c.buildCanonicalizedHeader(headers),
canonicalizedResource)
return canonicalizedString
}
func (c Client) exec(verb, url string, headers map[string]string, body io.Reader) (*storageResponse, error) {
authHeader, err := c.getAuthorizationHeader(verb, url, headers)
if err != nil {
return nil, err
}
headers["Authorization"] = authHeader
if err != nil {
return nil, err
}
req, err := http.NewRequest(verb, url, body)
if err != nil {
return nil, errors.New("azure/storage: error creating request: " + err.Error())
}
if clstr, ok := headers["Content-Length"]; ok {
// content length header is being signed, but completely ignored by golang.
// instead we have to use the ContentLength property on the request struct
// (see https://golang.org/src/net/http/request.go?s=18140:18370#L536 and
// https://golang.org/src/net/http/transfer.go?s=1739:2467#L49)
req.ContentLength, err = strconv.ParseInt(clstr, 10, 64)
if err != nil {
return nil, err
}
}
for k, v := range headers {
req.Header.Add(k, v)
}
httpClient := c.HTTPClient
if httpClient == nil {
httpClient = http.DefaultClient
}
resp, err := httpClient.Do(req)
if err != nil {
return nil, err
}
statusCode := resp.StatusCode
if statusCode >= 400 && statusCode <= 505 {
var respBody []byte
respBody, err = readResponseBody(resp)
if err != nil {
return nil, err
}
requestID := resp.Header.Get("x-ms-request-id")
if len(respBody) == 0 {
// no error in response body, might happen in HEAD requests
err = serviceErrFromStatusCode(resp.StatusCode, resp.Status, requestID)
} else {
// response contains storage service error object, unmarshal
storageErr, errIn := serviceErrFromXML(respBody, resp.StatusCode, requestID)
if err != nil { // error unmarshaling the error response
err = errIn
}
err = storageErr
}
return &storageResponse{
statusCode: resp.StatusCode,
headers: resp.Header,
body: ioutil.NopCloser(bytes.NewReader(respBody)), /* restore the body */
}, err
}
return &storageResponse{
statusCode: resp.StatusCode,
headers: resp.Header,
body: resp.Body}, nil
}
func (c Client) execInternalJSON(verb, url string, headers map[string]string, body io.Reader) (*odataResponse, error) {
req, err := http.NewRequest(verb, url, body)
for k, v := range headers {
req.Header.Add(k, v)
}
httpClient := c.HTTPClient
if httpClient == nil {
httpClient = http.DefaultClient
}
resp, err := httpClient.Do(req)
if err != nil {
return nil, err
}
respToRet := &odataResponse{}
respToRet.body = resp.Body
respToRet.statusCode = resp.StatusCode
respToRet.headers = resp.Header
statusCode := resp.StatusCode
if statusCode >= 400 && statusCode <= 505 {
var respBody []byte
respBody, err = readResponseBody(resp)
if err != nil {
return nil, err
}
if len(respBody) == 0 {
// no error in response body, might happen in HEAD requests
err = serviceErrFromStatusCode(resp.StatusCode, resp.Status, resp.Header.Get("x-ms-request-id"))
return respToRet, err
}
// try unmarshal as odata.error json
err = json.Unmarshal(respBody, &respToRet.odata)
return respToRet, err
}
return respToRet, nil
}
func (c Client) createSharedKeyLite(url string, headers map[string]string) (string, error) {
can, err := c.buildCanonicalizedResourceTable(url)
if err != nil {
return "", err
}
strToSign := headers["x-ms-date"] + "\n" + can
hmac := c.computeHmac256(strToSign)
return fmt.Sprintf("SharedKeyLite %s:%s", c.accountName, hmac), nil
}
func (c Client) execTable(verb, url string, headers map[string]string, body io.Reader) (*odataResponse, error) {
var err error
headers["Authorization"], err = c.createSharedKeyLite(url, headers)
if err != nil {
return nil, err
}
return c.execInternalJSON(verb, url, headers, body)
}
func readResponseBody(resp *http.Response) ([]byte, error) {
defer resp.Body.Close()
out, err := ioutil.ReadAll(resp.Body)
if err == io.EOF {
err = nil
}
return out, err
}
func serviceErrFromXML(body []byte, statusCode int, requestID string) (AzureStorageServiceError, error) {
var storageErr AzureStorageServiceError
if err := xml.Unmarshal(body, &storageErr); err != nil {
return storageErr, err
}
storageErr.StatusCode = statusCode
storageErr.RequestID = requestID
return storageErr, nil
}
func serviceErrFromStatusCode(code int, status string, requestID string) AzureStorageServiceError {
return AzureStorageServiceError{
StatusCode: code,
Code: status,
RequestID: requestID,
Message: "no response body was available for error status code",
}
}
func (e AzureStorageServiceError) Error() string {
return fmt.Sprintf("storage: service returned error: StatusCode=%d, ErrorCode=%s, ErrorMessage=%s, RequestId=%s, QueryParameterName=%s, QueryParameterValue=%s",
e.StatusCode, e.Code, e.Message, e.RequestID, e.QueryParameterName, e.QueryParameterValue)
}
// checkRespCode returns UnexpectedStatusError if the given response code is not
// one of the allowed status codes; otherwise nil.
func checkRespCode(respCode int, allowed []int) error {
for _, v := range allowed {
if respCode == v {
return nil
}
}
return UnexpectedStatusCodeError{allowed, respCode}
}
This diff is collapsed.
package storage
import (
"encoding/xml"
"fmt"
"net/http"
"net/url"
"strconv"
"strings"
)
const (
// casing is per Golang's http.Header canonicalizing the header names.
approximateMessagesCountHeader = "X-Ms-Approximate-Messages-Count"
userDefinedMetadataHeaderPrefix = "X-Ms-Meta-"
)
// QueueServiceClient contains operations for Microsoft Azure Queue Storage
// Service.
type QueueServiceClient struct {
client Client
}
func pathForQueue(queue string) string { return fmt.Sprintf("/%s", queue) }
func pathForQueueMessages(queue string) string { return fmt.Sprintf("/%s/messages", queue) }
func pathForMessage(queue, name string) string { return fmt.Sprintf("/%s/messages/%s", queue, name) }
type putMessageRequest struct {
XMLName xml.Name `xml:"QueueMessage"`
MessageText string `xml:"MessageText"`
}
// PutMessageParameters is the set of options can be specified for Put Messsage
// operation. A zero struct does not use any preferences for the request.
type PutMessageParameters struct {
VisibilityTimeout int
MessageTTL int
}
func (p PutMessageParameters) getParameters() url.Values {
out := url.Values{}
if p.VisibilityTimeout != 0 {
out.Set("visibilitytimeout", strconv.Itoa(p.VisibilityTimeout))
}
if p.MessageTTL != 0 {
out.Set("messagettl", strconv.Itoa(p.MessageTTL))
}
return out
}
// GetMessagesParameters is the set of options can be specified for Get
// Messsages operation. A zero struct does not use any preferences for the
// request.
type GetMessagesParameters struct {
NumOfMessages int
VisibilityTimeout int
}
func (p GetMessagesParameters) getParameters() url.Values {
out := url.Values{}
if p.NumOfMessages != 0 {
out.Set("numofmessages", strconv.Itoa(p.NumOfMessages))
}
if p.VisibilityTimeout != 0 {
out.Set("visibilitytimeout", strconv.Itoa(p.VisibilityTimeout))
}
return out
}
// PeekMessagesParameters is the set of options can be specified for Peek
// Messsage operation. A zero struct does not use any preferences for the
// request.
type PeekMessagesParameters struct {
NumOfMessages int
}
func (p PeekMessagesParameters) getParameters() url.Values {
out := url.Values{"peekonly": {"true"}} // Required for peek operation
if p.NumOfMessages != 0 {
out.Set("numofmessages", strconv.Itoa(p.NumOfMessages))
}
return out
}
// UpdateMessageParameters is the set of options can be specified for Update Messsage
// operation. A zero struct does not use any preferences for the request.
type UpdateMessageParameters struct {
PopReceipt string
VisibilityTimeout int
}
func (p UpdateMessageParameters) getParameters() url.Values {
out := url.Values{}
if p.PopReceipt != "" {
out.Set("popreceipt", p.PopReceipt)
}
if p.VisibilityTimeout != 0 {
out.Set("visibilitytimeout", strconv.Itoa(p.VisibilityTimeout))
}
return out
}
// GetMessagesResponse represents a response returned from Get Messages
// operation.
type GetMessagesResponse struct {
XMLName xml.Name `xml:"QueueMessagesList"`
QueueMessagesList []GetMessageResponse `xml:"QueueMessage"`
}
// GetMessageResponse represents a QueueMessage object returned from Get
// Messages operation response.
type GetMessageResponse struct {
MessageID string `xml:"MessageId"`
InsertionTime string `xml:"InsertionTime"`
ExpirationTime string `xml:"ExpirationTime"`
PopReceipt string `xml:"PopReceipt"`
TimeNextVisible string `xml:"TimeNextVisible"`
DequeueCount int `xml:"DequeueCount"`
MessageText string `xml:"MessageText"`
}
// PeekMessagesResponse represents a response returned from Get Messages
// operation.
type PeekMessagesResponse struct {
XMLName xml.Name `xml:"QueueMessagesList"`
QueueMessagesList []PeekMessageResponse `xml:"QueueMessage"`
}
// PeekMessageResponse represents a QueueMessage object returned from Peek
// Messages operation response.
type PeekMessageResponse struct {
MessageID string `xml:"MessageId"`
InsertionTime string `xml:"InsertionTime"`
ExpirationTime string `xml:"ExpirationTime"`
DequeueCount int `xml:"DequeueCount"`
MessageText string `xml:"MessageText"`
}
// QueueMetadataResponse represents user defined metadata and queue
// properties on a specific queue.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179384.aspx
type QueueMetadataResponse struct {
ApproximateMessageCount int
UserDefinedMetadata map[string]string
}
// SetMetadata operation sets user-defined metadata on the specified queue.
// Metadata is associated with the queue as name-value pairs.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179348.aspx
func (c QueueServiceClient) SetMetadata(name string, metadata map[string]string) error {
uri := c.client.getEndpoint(queueServiceName, pathForQueue(name), url.Values{"comp": []string{"metadata"}})
headers := c.client.getStandardHeaders()
for k, v := range metadata {
headers[userDefinedMetadataHeaderPrefix+k] = v
}
resp, err := c.client.exec("PUT", uri, headers, nil)
if err != nil {
return err
}
defer resp.body.Close()
return checkRespCode(resp.statusCode, []int{http.StatusNoContent})
}
// GetMetadata operation retrieves user-defined metadata and queue
// properties on the specified queue. Metadata is associated with
// the queue as name-values pairs.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179384.aspx
//
// Because the way Golang's http client (and http.Header in particular)
// canonicalize header names, the returned metadata names would always
// be all lower case.
func (c QueueServiceClient) GetMetadata(name string) (QueueMetadataResponse, error) {
qm := QueueMetadataResponse{}
qm.UserDefinedMetadata = make(map[string]string)
uri := c.client.getEndpoint(queueServiceName, pathForQueue(name), url.Values{"comp": []string{"metadata"}})
headers := c.client.getStandardHeaders()
resp, err := c.client.exec("GET", uri, headers, nil)
if err != nil {
return qm, err
}
defer resp.body.Close()
for k, v := range resp.headers {
if len(v) != 1 {
return qm, fmt.Errorf("Unexpected number of values (%d) in response header '%s'", len(v), k)
}
value := v[0]
if k == approximateMessagesCountHeader {
qm.ApproximateMessageCount, err = strconv.Atoi(value)
if err != nil {
return qm, fmt.Errorf("Unexpected value in response header '%s': '%s' ", k, value)
}
} else if strings.HasPrefix(k, userDefinedMetadataHeaderPrefix) {
name := strings.TrimPrefix(k, userDefinedMetadataHeaderPrefix)
qm.UserDefinedMetadata[strings.ToLower(name)] = value
}
}
return qm, checkRespCode(resp.statusCode, []int{http.StatusOK})
}
// CreateQueue operation creates a queue under the given account.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179342.aspx
func (c QueueServiceClient) CreateQueue(name string) error {
uri := c.client.getEndpoint(queueServiceName, pathForQueue(name), url.Values{})
headers := c.client.getStandardHeaders()
resp, err := c.client.exec("PUT", uri, headers, nil)
if err != nil {
return err
}
defer resp.body.Close()
return checkRespCode(resp.statusCode, []int{http.StatusCreated})
}
// DeleteQueue operation permanently deletes the specified queue.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179436.aspx
func (c QueueServiceClient) DeleteQueue(name string) error {
uri := c.client.getEndpoint(queueServiceName, pathForQueue(name), url.Values{})
resp, err := c.client.exec("DELETE", uri, c.client.getStandardHeaders(), nil)
if err != nil {
return err
}
defer resp.body.Close()
return checkRespCode(resp.statusCode, []int{http.StatusNoContent})
}
// QueueExists returns true if a queue with given name exists.
func (c QueueServiceClient) QueueExists(name string) (bool, error) {
uri := c.client.getEndpoint(queueServiceName, pathForQueue(name), url.Values{"comp": {"metadata"}})
resp, err := c.client.exec("GET", uri, c.client.getStandardHeaders(), nil)
if resp != nil && (resp.statusCode == http.StatusOK || resp.statusCode == http.StatusNotFound) {
return resp.statusCode == http.StatusOK, nil
}
return false, err
}
// PutMessage operation adds a new message to the back of the message queue.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179346.aspx
func (c QueueServiceClient) PutMessage(queue string, message string, params PutMessageParameters) error {
uri := c.client.getEndpoint(queueServiceName, pathForQueueMessages(queue), params.getParameters())
req := putMessageRequest{MessageText: message}
body, nn, err := xmlMarshal(req)
if err != nil {
return err
}
headers := c.client.getStandardHeaders()
headers["Content-Length"] = strconv.Itoa(nn)
resp, err := c.client.exec("POST", uri, headers, body)
if err != nil {
return err
}
defer resp.body.Close()
return checkRespCode(resp.statusCode, []int{http.StatusCreated})
}
// ClearMessages operation deletes all messages from the specified queue.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179454.aspx
func (c QueueServiceClient) ClearMessages(queue string) error {
uri := c.client.getEndpoint(queueServiceName, pathForQueueMessages(queue), url.Values{})
resp, err := c.client.exec("DELETE", uri, c.client.getStandardHeaders(), nil)
if err != nil {
return err
}
defer resp.body.Close()
return checkRespCode(resp.statusCode, []int{http.StatusNoContent})
}
// GetMessages operation retrieves one or more messages from the front of the
// queue.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179474.aspx
func (c QueueServiceClient) GetMessages(queue string, params GetMessagesParameters) (GetMessagesResponse, error) {
var r GetMessagesResponse
uri := c.client.getEndpoint(queueServiceName, pathForQueueMessages(queue), params.getParameters())
resp, err := c.client.exec("GET", uri, c.client.getStandardHeaders(), nil)
if err != nil {
return r, err
}
defer resp.body.Close()
err = xmlUnmarshal(resp.body, &r)
return r, err
}
// PeekMessages retrieves one or more messages from the front of the queue, but
// does not alter the visibility of the message.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179472.aspx
func (c QueueServiceClient) PeekMessages(queue string, params PeekMessagesParameters) (PeekMessagesResponse, error) {
var r PeekMessagesResponse
uri := c.client.getEndpoint(queueServiceName, pathForQueueMessages(queue), params.getParameters())
resp, err := c.client.exec("GET", uri, c.client.getStandardHeaders(), nil)
if err != nil {
return r, err
}
defer resp.body.Close()
err = xmlUnmarshal(resp.body, &r)
return r, err
}
// DeleteMessage operation deletes the specified message.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179347.aspx
func (c QueueServiceClient) DeleteMessage(queue, messageID, popReceipt string) error {
uri := c.client.getEndpoint(queueServiceName, pathForMessage(queue, messageID), url.Values{
"popreceipt": {popReceipt}})
resp, err := c.client.exec("DELETE", uri, c.client.getStandardHeaders(), nil)
if err != nil {
return err
}
defer resp.body.Close()
return checkRespCode(resp.statusCode, []int{http.StatusNoContent})
}
// UpdateMessage operation deletes the specified message.
//
// See https://msdn.microsoft.com/en-us/library/azure/hh452234.aspx
func (c QueueServiceClient) UpdateMessage(queue string, messageID string, message string, params UpdateMessageParameters) error {
uri := c.client.getEndpoint(queueServiceName, pathForMessage(queue, messageID), params.getParameters())
req := putMessageRequest{MessageText: message}
body, nn, err := xmlMarshal(req)
if err != nil {
return err
}
headers := c.client.getStandardHeaders()
headers["Content-Length"] = fmt.Sprintf("%d", nn)
resp, err := c.client.exec("PUT", uri, headers, body)
if err != nil {
return err
}
defer resp.body.Close()
return checkRespCode(resp.statusCode, []int{http.StatusNoContent})
}
package storage
import (
"bytes"
"encoding/json"
"fmt"
"net/http"
"net/url"
)
// TableServiceClient contains operations for Microsoft Azure Table Storage
// Service.
type TableServiceClient struct {
client Client
}
// AzureTable is the typedef of the Azure Table name
type AzureTable string
const (
tablesURIPath = "/Tables"
)
type createTableRequest struct {
TableName string `json:"TableName"`
}
func pathForTable(table AzureTable) string { return fmt.Sprintf("%s", table) }
func (c *TableServiceClient) getStandardHeaders() map[string]string {
return map[string]string{
"x-ms-version": "2015-02-21",
"x-ms-date": currentTimeRfc1123Formatted(),
"Accept": "application/json;odata=nometadata",
"Accept-Charset": "UTF-8",
"Content-Type": "application/json",
}
}
// QueryTables returns the tables created in the
// *TableServiceClient storage account.
func (c *TableServiceClient) QueryTables() ([]AzureTable, error) {
uri := c.client.getEndpoint(tableServiceName, tablesURIPath, url.Values{})
headers := c.getStandardHeaders()
headers["Content-Length"] = "0"
resp, err := c.client.execTable("GET", uri, headers, nil)
if err != nil {
return nil, err
}
defer resp.body.Close()
if err := checkRespCode(resp.statusCode, []int{http.StatusOK}); err != nil {
return nil, err
}
buf := new(bytes.Buffer)
buf.ReadFrom(resp.body)
var respArray queryTablesResponse
if err := json.Unmarshal(buf.Bytes(), &respArray); err != nil {
return nil, err
}
s := make([]AzureTable, len(respArray.TableName))
for i, elem := range respArray.TableName {
s[i] = AzureTable(elem.TableName)
}
return s, nil
}
// CreateTable creates the table given the specific
// name. This function fails if the name is not compliant
// with the specification or the tables already exists.
func (c *TableServiceClient) CreateTable(table AzureTable) error {
uri := c.client.getEndpoint(tableServiceName, tablesURIPath, url.Values{})
headers := c.getStandardHeaders()
req := createTableRequest{TableName: string(table)}
buf := new(bytes.Buffer)
if err := json.NewEncoder(buf).Encode(req); err != nil {
return err
}
headers["Content-Length"] = fmt.Sprintf("%d", buf.Len())
resp, err := c.client.execTable("POST", uri, headers, buf)
if err != nil {
return err
}
defer resp.body.Close()
if err := checkRespCode(resp.statusCode, []int{http.StatusCreated}); err != nil {
return err
}
return nil
}
// DeleteTable deletes the table given the specific
// name. This function fails if the table is not present.
// Be advised: DeleteTable deletes all the entries
// that may be present.
func (c *TableServiceClient) DeleteTable(table AzureTable) error {
uri := c.client.getEndpoint(tableServiceName, tablesURIPath, url.Values{})
uri += fmt.Sprintf("('%s')", string(table))
headers := c.getStandardHeaders()
headers["Content-Length"] = "0"
resp, err := c.client.execTable("DELETE", uri, headers, nil)
if err != nil {
return err
}
defer resp.body.Close()
if err := checkRespCode(resp.statusCode, []int{http.StatusNoContent}); err != nil {
return err
}
return nil
}
package storage
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"net/url"
"reflect"
)
// Annotating as secure for gas scanning
/* #nosec */
const (
partitionKeyNode = "PartitionKey"
rowKeyNode = "RowKey"
tag = "table"
tagIgnore = "-"
continuationTokenPartitionKeyHeader = "X-Ms-Continuation-Nextpartitionkey"
continuationTokenRowHeader = "X-Ms-Continuation-Nextrowkey"
maxTopParameter = 1000
)
type queryTablesResponse struct {
TableName []struct {
TableName string `json:"TableName"`
} `json:"value"`
}
const (
tableOperationTypeInsert = iota
tableOperationTypeUpdate = iota
tableOperationTypeMerge = iota
tableOperationTypeInsertOrReplace = iota
tableOperationTypeInsertOrMerge = iota
)
type tableOperation int
// TableEntity interface specifies
// the functions needed to support
// marshaling and unmarshaling into
// Azure Tables. The struct must only contain
// simple types because Azure Tables do not
// support hierarchy.
type TableEntity interface {
PartitionKey() string
RowKey() string
SetPartitionKey(string) error
SetRowKey(string) error
}
// ContinuationToken is an opaque (ie not useful to inspect)
// struct that Get... methods can return if there are more
// entries to be returned than the ones already
// returned. Just pass it to the same function to continue
// receiving the remaining entries.
type ContinuationToken struct {
NextPartitionKey string
NextRowKey string
}
type getTableEntriesResponse struct {
Elements []map[string]interface{} `json:"value"`
}
// QueryTableEntities queries the specified table and returns the unmarshaled
// entities of type retType.
// top parameter limits the returned entries up to top. Maximum top
// allowed by Azure API is 1000. In case there are more than top entries to be
// returned the function will return a non nil *ContinuationToken. You can call the
// same function again passing the received ContinuationToken as previousContToken
// parameter in order to get the following entries. The query parameter
// is the odata query. To retrieve all the entries pass the empty string.
// The function returns a pointer to a TableEntity slice, the *ContinuationToken
// if there are more entries to be returned and an error in case something went
// wrong.
//
// Example:
// entities, cToken, err = tSvc.QueryTableEntities("table", cToken, reflect.TypeOf(entity), 20, "")
func (c *TableServiceClient) QueryTableEntities(tableName AzureTable, previousContToken *ContinuationToken, retType reflect.Type, top int, query string) ([]TableEntity, *ContinuationToken, error) {
if top > maxTopParameter {
return nil, nil, fmt.Errorf("top accepts at maximum %d elements. Requested %d instead", maxTopParameter, top)
}
uri := c.client.getEndpoint(tableServiceName, pathForTable(tableName), url.Values{})
uri += fmt.Sprintf("?$top=%d", top)
if query != "" {
uri += fmt.Sprintf("&$filter=%s", url.QueryEscape(query))
}
if previousContToken != nil {
uri += fmt.Sprintf("&NextPartitionKey=%s&NextRowKey=%s", previousContToken.NextPartitionKey, previousContToken.NextRowKey)
}
headers := c.getStandardHeaders()
headers["Content-Length"] = "0"
resp, err := c.client.execTable("GET", uri, headers, nil)
if err != nil {
return nil, nil, err
}
contToken := extractContinuationTokenFromHeaders(resp.headers)
if err != nil {
return nil, contToken, err
}
defer resp.body.Close()
if err := checkRespCode(resp.statusCode, []int{http.StatusOK}); err != nil {
return nil, contToken, err
}
retEntries, err := deserializeEntity(retType, resp.body)
if err != nil {
return nil, contToken, err
}
return retEntries, contToken, nil
}
// InsertEntity inserts an entity in the specified table.
// The function fails if there is an entity with the same
// PartitionKey and RowKey in the table.
func (c *TableServiceClient) InsertEntity(table AzureTable, entity TableEntity) error {
var err error
if sc, err := c.execTable(table, entity, false, "POST"); err != nil {
return checkRespCode(sc, []int{http.StatusCreated})
}
return err
}
func (c *TableServiceClient) execTable(table AzureTable, entity TableEntity, specifyKeysInURL bool, method string) (int, error) {
uri := c.client.getEndpoint(tableServiceName, pathForTable(table), url.Values{})
if specifyKeysInURL {
uri += fmt.Sprintf("(PartitionKey='%s',RowKey='%s')", url.QueryEscape(entity.PartitionKey()), url.QueryEscape(entity.RowKey()))
}
headers := c.getStandardHeaders()
var buf bytes.Buffer
if err := injectPartitionAndRowKeys(entity, &buf); err != nil {
return 0, err
}
headers["Content-Length"] = fmt.Sprintf("%d", buf.Len())
var err error
var resp *odataResponse
resp, err = c.client.execTable(method, uri, headers, &buf)
if err != nil {
return 0, err
}
defer resp.body.Close()
return resp.statusCode, nil
}
// UpdateEntity updates the contents of an entity with the
// one passed as parameter. The function fails if there is no entity
// with the same PartitionKey and RowKey in the table.
func (c *TableServiceClient) UpdateEntity(table AzureTable, entity TableEntity) error {
var err error
if sc, err := c.execTable(table, entity, true, "PUT"); err != nil {
return checkRespCode(sc, []int{http.StatusNoContent})
}
return err
}
// MergeEntity merges the contents of an entity with the
// one passed as parameter.
// The function fails if there is no entity
// with the same PartitionKey and RowKey in the table.
func (c *TableServiceClient) MergeEntity(table AzureTable, entity TableEntity) error {
var err error
if sc, err := c.execTable(table, entity, true, "MERGE"); err != nil {
return checkRespCode(sc, []int{http.StatusNoContent})
}
return err
}
// DeleteEntityWithoutCheck deletes the entity matching by
// PartitionKey and RowKey. There is no check on IfMatch
// parameter so the entity is always deleted.
// The function fails if there is no entity
// with the same PartitionKey and RowKey in the table.
func (c *TableServiceClient) DeleteEntityWithoutCheck(table AzureTable, entity TableEntity) error {
return c.DeleteEntity(table, entity, "*")
}
// DeleteEntity deletes the entity matching by
// PartitionKey, RowKey and ifMatch field.
// The function fails if there is no entity
// with the same PartitionKey and RowKey in the table or
// the ifMatch is different.
func (c *TableServiceClient) DeleteEntity(table AzureTable, entity TableEntity, ifMatch string) error {
uri := c.client.getEndpoint(tableServiceName, pathForTable(table), url.Values{})
uri += fmt.Sprintf("(PartitionKey='%s',RowKey='%s')", url.QueryEscape(entity.PartitionKey()), url.QueryEscape(entity.RowKey()))
headers := c.getStandardHeaders()
headers["Content-Length"] = "0"
headers["If-Match"] = ifMatch
resp, err := c.client.execTable("DELETE", uri, headers, nil)
if err != nil {
return err
}
defer resp.body.Close()
if err := checkRespCode(resp.statusCode, []int{http.StatusNoContent}); err != nil {
return err
}
return nil
}
// InsertOrReplaceEntity inserts an entity in the specified table
// or replaced the existing one.
func (c *TableServiceClient) InsertOrReplaceEntity(table AzureTable, entity TableEntity) error {
var err error
if sc, err := c.execTable(table, entity, true, "PUT"); err != nil {
return checkRespCode(sc, []int{http.StatusNoContent})
}
return err
}
// InsertOrMergeEntity inserts an entity in the specified table
// or merges the existing one.
func (c *TableServiceClient) InsertOrMergeEntity(table AzureTable, entity TableEntity) error {
var err error
if sc, err := c.execTable(table, entity, true, "MERGE"); err != nil {
return checkRespCode(sc, []int{http.StatusNoContent})
}
return err
}
func injectPartitionAndRowKeys(entity TableEntity, buf *bytes.Buffer) error {
if err := json.NewEncoder(buf).Encode(entity); err != nil {
return err
}
dec := make(map[string]interface{})
if err := json.NewDecoder(buf).Decode(&dec); err != nil {
return err
}
// Inject PartitionKey and RowKey
dec[partitionKeyNode] = entity.PartitionKey()
dec[rowKeyNode] = entity.RowKey()
// Remove tagged fields
// The tag is defined in the const section
// This is useful to avoid storing the PartitionKey and RowKey twice.
numFields := reflect.ValueOf(entity).Elem().NumField()
for i := 0; i < numFields; i++ {
f := reflect.ValueOf(entity).Elem().Type().Field(i)
if f.Tag.Get(tag) == tagIgnore {
// we must look for its JSON name in the dictionary
// as the user can rename it using a tag
jsonName := f.Name
if f.Tag.Get("json") != "" {
jsonName = f.Tag.Get("json")
}
delete(dec, jsonName)
}
}
buf.Reset()
if err := json.NewEncoder(buf).Encode(&dec); err != nil {
return err
}
return nil
}
func deserializeEntity(retType reflect.Type, reader io.Reader) ([]TableEntity, error) {
buf := new(bytes.Buffer)
var ret getTableEntriesResponse
if err := json.NewDecoder(reader).Decode(&ret); err != nil {
return nil, err
}
tEntries := make([]TableEntity, len(ret.Elements))
for i, entry := range ret.Elements {
buf.Reset()
if err := json.NewEncoder(buf).Encode(entry); err != nil {
return nil, err
}
dec := make(map[string]interface{})
if err := json.NewDecoder(buf).Decode(&dec); err != nil {
return nil, err
}
var pKey, rKey string
// strip pk and rk
for key, val := range dec {
switch key {
case partitionKeyNode:
pKey = val.(string)
case rowKeyNode:
rKey = val.(string)
}
}
delete(dec, partitionKeyNode)
delete(dec, rowKeyNode)
buf.Reset()
if err := json.NewEncoder(buf).Encode(dec); err != nil {
return nil, err
}
// Create a empty retType instance
tEntries[i] = reflect.New(retType.Elem()).Interface().(TableEntity)
// Popolate it with the values
if err := json.NewDecoder(buf).Decode(&tEntries[i]); err != nil {
return nil, err
}
// Reset PartitionKey and RowKey
tEntries[i].SetPartitionKey(pKey)
tEntries[i].SetRowKey(rKey)
}
return tEntries, nil
}
func extractContinuationTokenFromHeaders(h http.Header) *ContinuationToken {
ct := ContinuationToken{h.Get(continuationTokenPartitionKeyHeader), h.Get(continuationTokenRowHeader)}
if ct.NextPartitionKey != "" && ct.NextRowKey != "" {
return &ct
}
return nil
}
package storage
import (
"bytes"
"crypto/hmac"
"crypto/sha256"
"encoding/base64"
"encoding/xml"
"fmt"
"io"
"io/ioutil"
"net/http"
"net/url"
"reflect"
"time"
)
func (c Client) computeHmac256(message string) string {
h := hmac.New(sha256.New, c.accountKey)
h.Write([]byte(message))
return base64.StdEncoding.EncodeToString(h.Sum(nil))
}
func currentTimeRfc1123Formatted() string {
return timeRfc1123Formatted(time.Now().UTC())
}
func timeRfc1123Formatted(t time.Time) string {
return t.Format(http.TimeFormat)
}
func mergeParams(v1, v2 url.Values) url.Values {
out := url.Values{}
for k, v := range v1 {
out[k] = v
}
for k, v := range v2 {
vals, ok := out[k]
if ok {
vals = append(vals, v...)
out[k] = vals
} else {
out[k] = v
}
}
return out
}
func prepareBlockListRequest(blocks []Block) string {
s := `<?xml version="1.0" encoding="utf-8"?><BlockList>`
for _, v := range blocks {
s += fmt.Sprintf("<%s>%s</%s>", v.Status, v.ID, v.Status)
}
s += `</BlockList>`
return s
}
func xmlUnmarshal(body io.Reader, v interface{}) error {
data, err := ioutil.ReadAll(body)
if err != nil {
return err
}
return xml.Unmarshal(data, v)
}
func xmlMarshal(v interface{}) (io.Reader, int, error) {
b, err := xml.Marshal(v)
if err != nil {
return nil, 0, err
}
return bytes.NewReader(b), len(b), nil
}
func headersFromStruct(v interface{}) map[string]string {
headers := make(map[string]string)
value := reflect.ValueOf(v)
for i := 0; i < value.NumField(); i++ {
key := value.Type().Field(i).Tag.Get("header")
val := value.Field(i).String()
if key != "" && val != "" {
headers[key] = val
}
}
return headers
}
package sign
import (
"bytes"
"crypto"
"crypto/rand"
"crypto/rsa"
"crypto/sha1"
"encoding/base64"
"encoding/json"
"fmt"
"io"
"net/url"
"strings"
"time"
"unicode"
)
// An AWSEpochTime wraps a time value providing JSON serialization needed for
// AWS Policy epoch time fields.
type AWSEpochTime struct {
time.Time
}
// NewAWSEpochTime returns a new AWSEpochTime pointer wrapping the Go time provided.
func NewAWSEpochTime(t time.Time) *AWSEpochTime {
return &AWSEpochTime{t}
}
// MarshalJSON serializes the epoch time as AWS Profile epoch time.
func (t AWSEpochTime) MarshalJSON() ([]byte, error) {
return []byte(fmt.Sprintf(`{"AWS:EpochTime":%d}`, t.UTC().Unix())), nil
}
// An IPAddress wraps an IPAddress source IP providing JSON serialization information
type IPAddress struct {
SourceIP string `json:"AWS:SourceIp"`
}
// A Condition defines the restrictions for how a signed URL can be used.
type Condition struct {
// Optional IP address mask the signed URL must be requested from.
IPAddress *IPAddress `json:"IpAddress,omitempty"`
// Optional date that the signed URL cannot be used until. It is invalid
// to make requests with the signed URL prior to this date.
DateGreaterThan *AWSEpochTime `json:",omitempty"`
// Required date that the signed URL will expire. A DateLessThan is required
// sign cloud front URLs
DateLessThan *AWSEpochTime `json:",omitempty"`
}
// A Statement is a collection of conditions for resources
type Statement struct {
// The Web or RTMP resource the URL will be signed for
Resource string
// The set of conditions for this resource
Condition Condition
}
// A Policy defines the resources that a signed will be signed for.
//
// See the following page for more information on how policies are constructed.
// http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-creating-signed-url-custom-policy.html#private-content-custom-policy-statement
type Policy struct {
// List of resource and condition statements.
// Signed URLs should only provide a single statement.
Statements []Statement `json:"Statement"`
}
// Override for testing to mock out usage of crypto/rand.Reader
var randReader = rand.Reader
// Sign will sign a policy using an RSA private key. It will return a base 64
// encoded signature and policy if no error is encountered.
//
// The signature and policy should be added to the signed URL following the
// guidelines in:
// http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-signed-urls.html
func (p *Policy) Sign(privKey *rsa.PrivateKey) (b64Signature, b64Policy []byte, err error) {
if err = p.Validate(); err != nil {
return nil, nil, err
}
// Build and escape the policy
b64Policy, jsonPolicy, err := encodePolicy(p)
if err != nil {
return nil, nil, err
}
awsEscapeEncoded(b64Policy)
// Build and escape the signature
b64Signature, err = signEncodedPolicy(randReader, jsonPolicy, privKey)
if err != nil {
return nil, nil, err
}
awsEscapeEncoded(b64Signature)
return b64Signature, b64Policy, nil
}
// Validate verifies that the policy is valid and usable, and returns an
// error if there is a problem.
func (p *Policy) Validate() error {
if len(p.Statements) == 0 {
return fmt.Errorf("at least one policy statement is required")
}
for i, s := range p.Statements {
if s.Resource == "" {
return fmt.Errorf("statement at index %d does not have a resource", i)
}
if !isASCII(s.Resource) {
return fmt.Errorf("unable to sign resource, [%s]. "+
"Resources must only contain ascii characters. "+
"Hostnames with unicode should be encoded as Punycode, (e.g. golang.org/x/net/idna), "+
"and URL unicode path/query characters should be escaped.", s.Resource)
}
}
return nil
}
// CreateResource constructs, validates, and returns a resource URL string. An
// error will be returned if unable to create the resource string.
func CreateResource(scheme, u string) (string, error) {
scheme = strings.ToLower(scheme)
if scheme == "http" || scheme == "https" || scheme == "http*" || scheme == "*" {
return u, nil
}
if scheme == "rtmp" {
parsed, err := url.Parse(u)
if err != nil {
return "", fmt.Errorf("unable to parse rtmp URL, err: %s", err)
}
rtmpURL := strings.TrimLeft(parsed.Path, "/")
if parsed.RawQuery != "" {
rtmpURL = fmt.Sprintf("%s?%s", rtmpURL, parsed.RawQuery)
}
return rtmpURL, nil
}
return "", fmt.Errorf("invalid URL scheme must be http, https, or rtmp. Provided: %s", scheme)
}
// NewCannedPolicy returns a new Canned Policy constructed using the resource
// and expires time. This can be used to generate the basic model for a Policy
// that can be then augmented with additional conditions.
//
// See the following page for more information on how policies are constructed.
// http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-creating-signed-url-custom-policy.html#private-content-custom-policy-statement
func NewCannedPolicy(resource string, expires time.Time) *Policy {
return &Policy{
Statements: []Statement{
{
Resource: resource,
Condition: Condition{
DateLessThan: NewAWSEpochTime(expires),
},
},
},
}
}
// encodePolicy encodes the Policy as JSON and also base 64 encodes it.
func encodePolicy(p *Policy) (b64Policy, jsonPolicy []byte, err error) {
jsonPolicy, err = json.Marshal(p)
if err != nil {
return nil, nil, fmt.Errorf("failed to encode policy, %s", err.Error())
}
// Remove leading and trailing white space, JSON encoding will note include
// whitespace within the encoding.
jsonPolicy = bytes.TrimSpace(jsonPolicy)
b64Policy = make([]byte, base64.StdEncoding.EncodedLen(len(jsonPolicy)))
base64.StdEncoding.Encode(b64Policy, jsonPolicy)
return b64Policy, jsonPolicy, nil
}
// signEncodedPolicy will sign and base 64 encode the JSON encoded policy.
func signEncodedPolicy(randReader io.Reader, jsonPolicy []byte, privKey *rsa.PrivateKey) ([]byte, error) {
hash := sha1.New()
if _, err := bytes.NewReader(jsonPolicy).WriteTo(hash); err != nil {
return nil, fmt.Errorf("failed to calculate signing hash, %s", err.Error())
}
sig, err := rsa.SignPKCS1v15(randReader, privKey, crypto.SHA1, hash.Sum(nil))
if err != nil {
return nil, fmt.Errorf("failed to sign policy, %s", err.Error())
}
b64Sig := make([]byte, base64.StdEncoding.EncodedLen(len(sig)))
base64.StdEncoding.Encode(b64Sig, sig)
return b64Sig, nil
}
// special characters to be replaced with awsEscapeEncoded
var invalidEncodedChar = map[byte]byte{
'+': '-',
'=': '_',
'/': '~',
}
// awsEscapeEncoded will replace base64 encoding's special characters to be URL safe.
func awsEscapeEncoded(b []byte) {
for i, v := range b {
if r, ok := invalidEncodedChar[v]; ok {
b[i] = r
}
}
}
func isASCII(u string) bool {
for _, c := range u {
if c > unicode.MaxASCII {
return false
}
}
return true
}
package sign
import (
"crypto/rsa"
"crypto/x509"
"encoding/pem"
"fmt"
"io"
"io/ioutil"
"os"
)
// LoadPEMPrivKeyFile reads a PEM encoded RSA private key from the file name.
// A new RSA private key will be returned if no error.
func LoadPEMPrivKeyFile(name string) (*rsa.PrivateKey, error) {
file, err := os.Open(name)
if err != nil {
return nil, err
}
defer file.Close()
return LoadPEMPrivKey(file)
}
// LoadPEMPrivKey reads a PEM encoded RSA private key from the io.Reader.
// A new RSA private key will be returned if no error.
func LoadPEMPrivKey(reader io.Reader) (*rsa.PrivateKey, error) {
block, err := loadPem(reader)
if err != nil {
return nil, err
}
return x509.ParsePKCS1PrivateKey(block.Bytes)
}
// LoadEncryptedPEMPrivKey decrypts the PEM encoded private key using the
// password provided returning a RSA private key. If the PEM data is invalid,
// or unable to decrypt an error will be returned.
func LoadEncryptedPEMPrivKey(reader io.Reader, password []byte) (*rsa.PrivateKey, error) {
block, err := loadPem(reader)
if err != nil {
return nil, err
}
decryptedBlock, err := x509.DecryptPEMBlock(block, password)
if err != nil {
return nil, err
}
return x509.ParsePKCS1PrivateKey(decryptedBlock)
}
func loadPem(reader io.Reader) (*pem.Block, error) {
b, err := ioutil.ReadAll(reader)
if err != nil {
return nil, err
}
block, _ := pem.Decode(b)
if block == nil {
// pem.Decode will set block to nil if there is no PEM data in the input
// the second parameter will contain the provided bytes that failed
// to be decoded.
return nil, fmt.Errorf("no valid PEM data provided")
}
return block, nil
}
package sign
import (
"bytes"
"encoding/binary"
"math/rand"
)
// A randomReader wraps a math/rand.Rand within an reader so that it can used
// as a predictable testing replacement for crypto/rand.Reader
type randomReader struct {
b *bytes.Buffer
r *rand.Rand
}
// newRandomReader returns a new instance of the random reader
func newRandomReader(r *rand.Rand) *randomReader {
return &randomReader{b: &bytes.Buffer{}, r: r}
}
// Read will read random bytes from up to the length of b.
func (m *randomReader) Read(b []byte) (int, error) {
for i := 0; i < len(b); {
binary.Write(m.b, binary.LittleEndian, m.r.Int63())
n, _ := m.b.Read(b[i:])
i += n
}
return len(b), nil
}
package sign
import (
"crypto/rsa"
"fmt"
"net/http"
"strings"
"time"
)
const (
// CookiePolicyName name of the policy cookie
CookiePolicyName = "CloudFront-Policy"
// CookieSignatureName name of the signature cookie
CookieSignatureName = "CloudFront-Signature"
// CookieKeyIDName name of the signing Key ID cookie
CookieKeyIDName = "CloudFront-Key-Pair-Id"
)
// A CookieOptions optional additional options that can be applied to the signed
// cookies.
type CookieOptions struct {
Path string
Domain string
Secure bool
}
// apply will integration the options provided into the base cookie options
// a new copy will be returned. The base CookieOption will not be modified.
func (o CookieOptions) apply(opts ...func(*CookieOptions)) CookieOptions {
if len(opts) == 0 {
return o
}
for _, opt := range opts {
opt(&o)
}
return o
}
// A CookieSigner provides signing utilities to sign Cookies for Amazon CloudFront
// resources. Using a private key and Credential Key Pair key ID the CookieSigner
// only needs to be created once per Credential Key Pair key ID and private key.
//
// More information about signed Cookies and their structure can be found at:
// http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-setting-signed-cookie-custom-policy.html
//
// To sign a Cookie, create a CookieSigner with your private key and credential
// pair key ID. Once you have a CookieSigner instance you can call Sign or
// SignWithPolicy to sign the URLs.
//
// The signer is safe to use concurrently, but the optional cookies options
// are not safe to modify concurrently.
type CookieSigner struct {
keyID string
privKey *rsa.PrivateKey
Opts CookieOptions
}
// NewCookieSigner constructs and returns a new CookieSigner to be used to for
// signing Amazon CloudFront URL resources with.
func NewCookieSigner(keyID string, privKey *rsa.PrivateKey, opts ...func(*CookieOptions)) *CookieSigner {
signer := &CookieSigner{
keyID: keyID,
privKey: privKey,
Opts: CookieOptions{}.apply(opts...),
}
return signer
}
// Sign returns the cookies needed to allow user agents to make arbetrary
// requests to cloudfront for the resource(s) defined by the policy.
//
// Sign will create a CloudFront policy with only a resource and condition of
// DateLessThan equal to the expires time provided.
//
// The returned slice cookies should all be added to the Client's cookies or
// server's response.
//
// Example:
// s := NewCookieSigner(keyID, privKey)
//
// // Get Signed cookies for a resource that will expire in 1 hour
// cookies, err := s.Sign("*", time.Now().Add(1 * time.Hour))
// if err != nil {
// fmt.Println("failed to create signed cookies", err)
// return
// }
//
// // Or get Signed cookies for a resource that will expire in 1 hour
// // and set path and domain of cookies
// cookies, err := s.Sign("*", time.Now().Add(1 * time.Hour), func(o *sign.CookieOptions) {
// o.Path = "/"
// o.Domain = ".example.com"
// })
// if err != nil {
// fmt.Println("failed to create signed cookies", err)
// return
// }
//
// // Server Response via http.ResponseWriter
// for _, c := range cookies {
// http.SetCookie(w, c)
// }
//
// // Client request via the cookie jar
// if client.CookieJar != nil {
// for _, c := range cookies {
// client.Cookie(w, c)
// }
// }
func (s CookieSigner) Sign(u string, expires time.Time, opts ...func(*CookieOptions)) ([]*http.Cookie, error) {
scheme, err := cookieURLScheme(u)
if err != nil {
return nil, err
}
resource, err := CreateResource(scheme, u)
if err != nil {
return nil, err
}
p := NewCannedPolicy(resource, expires)
return createCookies(p, s.keyID, s.privKey, s.Opts.apply(opts...))
}
// Returns and validates the URL's scheme.
// http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-setting-signed-cookie-custom-policy.html#private-content-custom-policy-statement-cookies
func cookieURLScheme(u string) (string, error) {
parts := strings.SplitN(u, "://", 2)
if len(parts) != 2 {
return "", fmt.Errorf("invalid cookie URL, missing scheme")
}
scheme := strings.ToLower(parts[0])
if scheme != "http" && scheme != "https" && scheme != "http*" {
return "", fmt.Errorf("invalid cookie URL scheme. Expect http, https, or http*. Go, %s", scheme)
}
return scheme, nil
}
// SignWithPolicy returns the cookies needed to allow user agents to make
// arbetrairy requets to cloudfront for the resource(s) defined by the policy.
//
// The returned slice cookies should all be added to the Client's cookies or
// server's response.
//
// Example:
// s := NewCookieSigner(keyID, privKey)
//
// policy := &sign.Policy{
// Statements: []sign.Statement{
// {
// // Read the provided documentation on how to set this
// // correctly, you'll probably want to use wildcards.
// Resource: RawCloudFrontURL,
// Condition: sign.Condition{
// // Optional IP source address range
// IPAddress: &sign.IPAddress{SourceIP: "192.0.2.0/24"},
// // Optional date URL is not valid until
// DateGreaterThan: &sign.AWSEpochTime{time.Now().Add(30 * time.Minute)},
// // Required date the URL will expire after
// DateLessThan: &sign.AWSEpochTime{time.Now().Add(1 * time.Hour)},
// },
// },
// },
// }
//
// // Get Signed cookies for a resource that will expire in 1 hour
// cookies, err := s.SignWithPolicy(policy)
// if err != nil {
// fmt.Println("failed to create signed cookies", err)
// return
// }
//
// // Or get Signed cookies for a resource that will expire in 1 hour
// // and set path and domain of cookies
// cookies, err := s.Sign(policy, func(o *sign.CookieOptions) {
// o.Path = "/"
// o.Domain = ".example.com"
// })
// if err != nil {
// fmt.Println("failed to create signed cookies", err)
// return
// }
//
// // Server Response via http.ResponseWriter
// for _, c := range cookies {
// http.SetCookie(w, c)
// }
//
// // Client request via the cookie jar
// if client.CookieJar != nil {
// for _, c := range cookies {
// client.Cookie(w, c)
// }
// }
func (s CookieSigner) SignWithPolicy(p *Policy, opts ...func(*CookieOptions)) ([]*http.Cookie, error) {
return createCookies(p, s.keyID, s.privKey, s.Opts.apply(opts...))
}
// Prepares the cookies to be attached to the header. An (optional) options
// struct is provided in case people don't want to manually edit their cookies.
func createCookies(p *Policy, keyID string, privKey *rsa.PrivateKey, opt CookieOptions) ([]*http.Cookie, error) {
b64Sig, b64Policy, err := p.Sign(privKey)
if err != nil {
return nil, err
}
// Creates proper cookies
cPolicy := &http.Cookie{
Name: CookiePolicyName,
Value: string(b64Policy),
HttpOnly: true,
}
cSignature := &http.Cookie{
Name: CookieSignatureName,
Value: string(b64Sig),
HttpOnly: true,
}
cKey := &http.Cookie{
Name: CookieKeyIDName,
Value: keyID,
HttpOnly: true,
}
cookies := []*http.Cookie{cPolicy, cSignature, cKey}
// Applie the cookie options
for _, c := range cookies {
c.Path = opt.Path
c.Domain = opt.Domain
c.Secure = opt.Secure
}
return cookies, nil
}
// Package sign provides utilities to generate signed URLs for Amazon CloudFront.
//
// More information about signed URLs and their structure can be found at:
// http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-creating-signed-url-canned-policy.html
//
// To sign a URL create a URLSigner with your private key and credential pair key ID.
// Once you have a URLSigner instance you can call Sign or SignWithPolicy to
// sign the URLs.
//
// Example:
//
// // Sign URL to be valid for 1 hour from now.
// signer := sign.NewURLSigner(keyID, privKey)
// signedURL, err := signer.Sign(rawURL, time.Now().Add(1*time.Hour))
// if err != nil {
// log.Fatalf("Failed to sign url, err: %s\n", err.Error())
// }
//
package sign
import (
"crypto/rsa"
"fmt"
"net/url"
"strings"
"time"
)
// An URLSigner provides URL signing utilities to sign URLs for Amazon CloudFront
// resources. Using a private key and Credential Key Pair key ID the URLSigner
// only needs to be created once per Credential Key Pair key ID and private key.
//
// The signer is safe to use concurrently.
type URLSigner struct {
keyID string
privKey *rsa.PrivateKey
}
// NewURLSigner constructs and returns a new URLSigner to be used to for signing
// Amazon CloudFront URL resources with.
func NewURLSigner(keyID string, privKey *rsa.PrivateKey) *URLSigner {
return &URLSigner{
keyID: keyID,
privKey: privKey,
}
}
// Sign will sign a single URL to expire at the time of expires sign using the
// Amazon CloudFront default Canned Policy. The URL will be signed with the
// private key and Credential Key Pair Key ID previously provided to URLSigner.
//
// This is the default method of signing Amazon CloudFront URLs. If extra policy
// conditions are need other than URL expiry use SignWithPolicy instead.
//
// Example:
//
// // Sign URL to be valid for 1 hour from now.
// signer := sign.NewURLSigner(keyID, privKey)
// signedURL, err := signer.Sign(rawURL, time.Now().Add(1*time.Hour))
// if err != nil {
// log.Fatalf("Failed to sign url, err: %s\n", err.Error())
// }
//
func (s URLSigner) Sign(url string, expires time.Time) (string, error) {
scheme, cleanedURL, err := cleanURLScheme(url)
if err != nil {
return "", err
}
resource, err := CreateResource(scheme, url)
if err != nil {
return "", err
}
return signURL(scheme, cleanedURL, s.keyID, NewCannedPolicy(resource, expires), false, s.privKey)
}
// SignWithPolicy will sign a URL with the Policy provided. The URL will be
// signed with the private key and Credential Key Pair Key ID previously provided to URLSigner.
//
// Use this signing method if you are looking to sign a URL with more than just
// the URL's expiry time, or reusing Policies between multiple URL signings.
// If only the expiry time is needed you can use Sign and provide just the
// URL's expiry time. A minimum of at least one policy statement is required for a signed URL.
//
// Note: It is not safe to use Polices between multiple signers concurrently
//
// Example:
//
// // Sign URL to be valid for 30 minutes from now, expires one hour from now, and
// // restricted to the 192.0.2.0/24 IP address range.
// policy := &sign.Policy{
// Statements: []sign.Statement{
// {
// Resource: rawURL,
// Condition: sign.Condition{
// // Optional IP source address range
// IPAddress: &sign.IPAddress{SourceIP: "192.0.2.0/24"},
// // Optional date URL is not valid until
// DateGreaterThan: &sign.AWSEpochTime{time.Now().Add(30 * time.Minute)},
// // Required date the URL will expire after
// DateLessThan: &sign.AWSEpochTime{time.Now().Add(1 * time.Hour)},
// },
// },
// },
// }
//
// signer := sign.NewURLSigner(keyID, privKey)
// signedURL, err := signer.SignWithPolicy(rawURL, policy)
// if err != nil {
// log.Fatalf("Failed to sign url, err: %s\n", err.Error())
// }
//
func (s URLSigner) SignWithPolicy(url string, p *Policy) (string, error) {
scheme, cleanedURL, err := cleanURLScheme(url)
if err != nil {
return "", err
}
return signURL(scheme, cleanedURL, s.keyID, p, true, s.privKey)
}
func signURL(scheme, url, keyID string, p *Policy, customPolicy bool, privKey *rsa.PrivateKey) (string, error) {
// Validation URL elements
if err := validateURL(url); err != nil {
return "", err
}
b64Signature, b64Policy, err := p.Sign(privKey)
if err != nil {
return "", err
}
// build and return signed URL
builtURL := buildSignedURL(url, keyID, p, customPolicy, b64Policy, b64Signature)
if scheme == "rtmp" {
return buildRTMPURL(builtURL)
}
return builtURL, nil
}
func buildSignedURL(baseURL, keyID string, p *Policy, customPolicy bool, b64Policy, b64Signature []byte) string {
pred := "?"
if strings.Contains(baseURL, "?") {
pred = "&"
}
signedURL := baseURL + pred
if customPolicy {
signedURL += "Policy=" + string(b64Policy)
} else {
signedURL += fmt.Sprintf("Expires=%d", p.Statements[0].Condition.DateLessThan.UTC().Unix())
}
signedURL += fmt.Sprintf("&Signature=%s&Key-Pair-Id=%s", string(b64Signature), keyID)
return signedURL
}
func buildRTMPURL(u string) (string, error) {
parsed, err := url.Parse(u)
if err != nil {
return "", fmt.Errorf("unable to parse rtmp signed URL, err: %s", err)
}
rtmpURL := strings.TrimLeft(parsed.Path, "/")
if parsed.RawQuery != "" {
rtmpURL = fmt.Sprintf("%s?%s", rtmpURL, parsed.RawQuery)
}
return rtmpURL, nil
}
func cleanURLScheme(u string) (scheme, cleanedURL string, err error) {
parts := strings.SplitN(u, "://", 2)
if len(parts) != 2 {
return "", "", fmt.Errorf("invalid URL, missing scheme and domain/path")
}
scheme = strings.Replace(parts[0], "*", "", 1)
cleanedURL = fmt.Sprintf("%s://%s", scheme, parts[1])
return strings.ToLower(scheme), cleanedURL, nil
}
var illegalQueryParms = []string{"Expires", "Policy", "Signature", "Key-Pair-Id"}
func validateURL(u string) error {
parsed, err := url.Parse(u)
if err != nil {
return fmt.Errorf("unable to parse URL, err: %s", err.Error())
}
if parsed.Scheme == "" {
return fmt.Errorf("URL missing valid scheme, %s", u)
}
q := parsed.Query()
for _, p := range illegalQueryParms {
if _, ok := q[p]; ok {
return fmt.Errorf("%s cannot be a query parameter for a signed URL", p)
}
}
return nil
}
package common
import (
"bytes"
"encoding/json"
"io/ioutil"
"log"
"net/http"
"net/url"
"time"
"github.com/denverdino/aliyungo/util"
)
// A Client represents a client of ECS services
type Client struct {
AccessKeyId string //Access Key Id
AccessKeySecret string //Access Key Secret
debug bool
httpClient *http.Client
endpoint string
version string
}
// NewClient creates a new instance of ECS client
func (client *Client) Init(endpoint, version, accessKeyId, accessKeySecret string) {
client.AccessKeyId = accessKeyId
client.AccessKeySecret = accessKeySecret + "&"
client.debug = false
client.httpClient = &http.Client{}
client.endpoint = endpoint
client.version = version
}
// SetEndpoint sets custom endpoint
func (client *Client) SetEndpoint(endpoint string) {
client.endpoint = endpoint
}
// SetEndpoint sets custom version
func (client *Client) SetVersion(version string) {
client.version = version
}
// SetAccessKeyId sets new AccessKeyId
func (client *Client) SetAccessKeyId(id string) {
client.AccessKeyId = id
}
// SetAccessKeySecret sets new AccessKeySecret
func (client *Client) SetAccessKeySecret(secret string) {
client.AccessKeySecret = secret + "&"
}
// SetDebug sets debug mode to log the request/response message
func (client *Client) SetDebug(debug bool) {
client.debug = debug
}
// Invoke sends the raw HTTP request for ECS services
func (client *Client) Invoke(action string, args interface{}, response interface{}) error {
request := Request{}
request.init(client.version, action, client.AccessKeyId)
query := util.ConvertToQueryValues(request)
util.SetQueryValues(args, &query)
// Sign request
signature := util.CreateSignatureForRequest(ECSRequestMethod, &query, client.AccessKeySecret)
// Generate the request URL
requestURL := client.endpoint + "?" + query.Encode() + "&Signature=" + url.QueryEscape(signature)
httpReq, err := http.NewRequest(ECSRequestMethod, requestURL, nil)
// TODO move to util and add build val flag
httpReq.Header.Set("X-SDK-Client", `AliyunGO/`+Version)
if err != nil {
return GetClientError(err)
}
t0 := time.Now()
httpResp, err := client.httpClient.Do(httpReq)
t1 := time.Now()
if err != nil {
return GetClientError(err)
}
statusCode := httpResp.StatusCode
if client.debug {
log.Printf("Invoke %s %s %d (%v)", ECSRequestMethod, requestURL, statusCode, t1.Sub(t0))
}
defer httpResp.Body.Close()
body, err := ioutil.ReadAll(httpResp.Body)
if err != nil {
return GetClientError(err)
}
if client.debug {
var prettyJSON bytes.Buffer
err = json.Indent(&prettyJSON, body, "", " ")
log.Println(string(prettyJSON.Bytes()))
}
if statusCode >= 400 && statusCode <= 599 {
errorResponse := ErrorResponse{}
err = json.Unmarshal(body, &errorResponse)
ecsError := &Error{
ErrorResponse: errorResponse,
StatusCode: statusCode,
}
return ecsError
}
err = json.Unmarshal(body, response)
//log.Printf("%++v", response)
if err != nil {
return GetClientError(err)
}
return nil
}
// GenerateClientToken generates the Client Token with random string
func (client *Client) GenerateClientToken() string {
return util.CreateRandomString()
}
func GetClientErrorFromString(str string) error {
return &Error{
ErrorResponse: ErrorResponse{
Code: "AliyunGoClientFailure",
Message: str,
},
StatusCode: -1,
}
}
func GetClientError(err error) error {
return GetClientErrorFromString(err.Error())
}
package common
// Region represents ECS region
type Region string
// Constants of region definition
const (
Hangzhou = Region("cn-hangzhou")
Qingdao = Region("cn-qingdao")
Beijing = Region("cn-beijing")
Hongkong = Region("cn-hongkong")
Shenzhen = Region("cn-shenzhen")
USWest1 = Region("us-west-1")
USEast1 = Region("us-east-1")
APSouthEast1 = Region("ap-southeast-1")
Shanghai = Region("cn-shanghai")
)
var ValidRegions = []Region{Hangzhou, Qingdao, Beijing, Shenzhen, Hongkong, Shanghai, USWest1, USEast1, APSouthEast1}
package common
import (
"fmt"
"log"
"time"
"github.com/denverdino/aliyungo/util"
)
// Constants for Aliyun API requests
const (
SignatureVersion = "1.0"
SignatureMethod = "HMAC-SHA1"
JSONResponseFormat = "JSON"
XMLResponseFormat = "XML"
ECSRequestMethod = "GET"
)
type Request struct {
Format string
Version string
AccessKeyId string
Signature string
SignatureMethod string
Timestamp util.ISO6801Time
SignatureVersion string
SignatureNonce string
ResourceOwnerAccount string
Action string
}
func (request *Request) init(version string, action string, AccessKeyId string) {
request.Format = JSONResponseFormat
request.Timestamp = util.NewISO6801Time(time.Now().UTC())
request.Version = version
request.SignatureVersion = SignatureVersion
request.SignatureMethod = SignatureMethod
request.SignatureNonce = util.CreateRandomString()
request.Action = action
request.AccessKeyId = AccessKeyId
}
type Response struct {
RequestId string
}
type ErrorResponse struct {
Response
HostId string
Code string
Message string
}
// An Error represents a custom error for Aliyun API failure response
type Error struct {
ErrorResponse
StatusCode int //Status Code of HTTP Response
}
func (e *Error) Error() string {
return fmt.Sprintf("Aliyun API Error: RequestId: %s Status Code: %d Code: %s Message: %s", e.RequestId, e.StatusCode, e.Code, e.Message)
}
type Pagination struct {
PageNumber int
PageSize int
}
func (p *Pagination) SetPageSize(size int) {
p.PageSize = size
}
func (p *Pagination) Validate() {
if p.PageNumber < 0 {
log.Printf("Invalid PageNumber: %d", p.PageNumber)
p.PageNumber = 1
}
if p.PageSize < 0 {
log.Printf("Invalid PageSize: %d", p.PageSize)
p.PageSize = 10
} else if p.PageSize > 50 {
log.Printf("Invalid PageSize: %d", p.PageSize)
p.PageSize = 50
}
}
// A PaginationResponse represents a response with pagination information
type PaginationResult struct {
TotalCount int
PageNumber int
PageSize int
}
// NextPage gets the next page of the result set
func (r *PaginationResult) NextPage() *Pagination {
if r.PageNumber*r.PageSize >= r.TotalCount {
return nil
}
return &Pagination{PageNumber: r.PageNumber + 1, PageSize: r.PageSize}
}
package common
type InternetChargeType string
const (
PayByBandwidth = InternetChargeType("PayByBandwidth")
PayByTraffic = InternetChargeType("PayByTraffic")
)
type InstanceChargeType string
const (
PrePaid = InstanceChargeType("PrePaid")
PostPaid = InstanceChargeType("PostPaid")
)
package common
const Version = "0.1"
This diff is collapsed.
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment