Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

objstore : implement Aliyun OSS #1573

Merged
merged 9 commits into from
Oct 24, 2019
Merged

objstore : implement Aliyun OSS #1573

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
f7b22c7
add oss support
wujinhu Oct 18, 2019
e7a5d5d
fix docs
wujinhu Oct 18, 2019
79b43df
fix Makefile
wujinhu Oct 18, 2019
d3f7eae
review comments
wujinhu Oct 21, 2019
1cb9cb1
fix style
wujinhu Oct 21, 2019
8f818df
review comments
wujinhu Oct 21, 2019
21f758f
review comments
wujinhu Oct 23, 2019
a0cdef1
review comments
wujinhu Oct 23, 2019
96620b9
review comments
wujinhu Oct 23, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ We use *breaking* word for marking changes that are not backward compatible (rel
### Added

- [#1660](https://github.com/thanos-io/thanos/pull/1660) Add a new `--prometheus.ready_timeout` CLI option to the sidecar to set how long to wait until Prometheus starts up.
- [#1573](https://github.com/thanos-io/thanos/pull/1573) `AliYun OSS` object storage, see [documents](docs/storage.md#aliyun-oss-configuration) for further information.

### Fixed

Expand Down
1 change: 1 addition & 0 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ $ git push origin <your_branch_for_new_pr>
- THANOS_SKIP_AZURE_TESTS to skip Azure tests.
- THANOS_SKIP_SWIFT_TESTS to skip SWIFT tests.
- THANOS_SKIP_TENCENT_COS_TESTS to skip Tencent COS tests.
- THANOS_SKIP_ALIYUN_OSS_TESTS to skip Aliyun OSS tests.

If you skip all of these, the store specific tests will be run against memory object storage only.
CI runs GCS and inmem tests only for now. Not having these variables will produce auth errors against GCS, AWS, Azure or COS tests.
Expand Down
8 changes: 5 additions & 3 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,8 +160,8 @@ docs: $(EMBEDMD) build
.PHONY: check-docs
check-docs: $(EMBEDMD) $(LICHE) build
@EMBEDMD_BIN="$(EMBEDMD)" scripts/genflagdocs.sh check
@$(LICHE) --recursive docs --exclude "cloud.tencent.com" --document-root .
@$(LICHE) --exclude "cloud.tencent.com|goreportcard.com" --document-root . *.md
@$(LICHE) --recursive docs --exclude "(cloud.tencent.com|alibabacloud.com)" --document-root .
@$(LICHE) --exclude "(cloud.tencent.com|alibabacloud.com)" --document-root . *.md
wujinhu marked this conversation as resolved.
Show resolved Hide resolved

# checks Go code comments if they have trailing period (excludes protobuffers and vendor files).
# Comments with more than 3 spaces at beginning are omitted from the check, example: '// - foo'.
Expand Down Expand Up @@ -203,17 +203,19 @@ test: check-git install-deps
@go install github.com/thanos-io/thanos/cmd/thanos
# Be careful on GOCACHE. Those tests are sometimes using built Thanos/Prometheus binaries directly. Don't cache those.
@rm -rf ${GOCACHE}
@echo ">> running all tests. Do export THANOS_SKIP_GCS_TESTS='true' or/and THANOS_SKIP_S3_AWS_TESTS='true' or/and THANOS_SKIP_AZURE_TESTS='true' and/or THANOS_SKIP_SWIFT_TESTS='true' and/or THANOS_SKIP_TENCENT_COS_TESTS='true' if you want to skip e2e tests against real store buckets"
@echo ">> running all tests. Do export THANOS_SKIP_GCS_TESTS='true' or/and THANOS_SKIP_S3_AWS_TESTS='true' or/and THANOS_SKIP_AZURE_TESTS='true' and/or THANOS_SKIP_SWIFT_TESTS='true' and/or THANOS_SKIP_ALIYUN_OSS_TESTS='true' and/or THANOS_SKIP_TENCENT_COS_TESTS='true' if you want to skip e2e tests against real store buckets"
@go test $(shell go list ./... | grep -v /vendor/);

.PHONY: test-ci
test-ci: export THANOS_SKIP_AZURE_TESTS = true
test-ci: export THANOS_SKIP_SWIFT_TESTS = true
test-ci: export THANOS_SKIP_TENCENT_COS_TESTS = true
test-ci: export THANOS_SKIP_ALIYUN_OSS_TESTS = true
test-ci:
@echo ">> Skipping AZURE tests"
@echo ">> Skipping SWIFT tests"
@echo ">> Skipping TENCENT tests"
@echo ">> Skipping ALIYUN tests"
$(MAKE) test

.PHONY: test-local
Expand Down
18 changes: 18 additions & 0 deletions docs/storage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ Current object storage client implementations:
| [Azure Storage Account](./storage.md#azure) | Stable (production usage) | yes | @vglafirov |
| [OpenStack Swift](./storage.md#openstack-swift) | Beta (working PoCs, testing usage) | no | @sudhi-vm |
| [Tencent COS](./storage.md#tencent-cos) | Beta (testing usage) | no | @jojohappy |
| [AliYun OSS](./storage.md#aliyun-oss) | Beta (testing usage) | no | @shaulboozhiao,@wujinhu |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍


NOTE: Currently Thanos requires strong consistency (write-read) for object store implementation.

Expand Down Expand Up @@ -336,3 +337,20 @@ config:
```

Set the flags `--objstore.config-file` to reference to the configuration file.

## AliYun OSS Configuration
In order to use AliYun OSS object storage, you should first create a bucket with proper Storage Class , ACLs and get the access key on the AliYun cloud. Go to [https://www.alibabacloud.com/product/oss](https://www.alibabacloud.com/product/oss) for more detail.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What proper Storage Class means?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is the same meaning with S3 https://aws.amazon.com/s3/storage-classes/?nc1=h_ls
It defines the storage type of data.


To use AliYun OSS object storage, please specify following yaml configuration file in `objstore.config*` flag.

[embedmd]:# (flags/config_bucket_aliyunoss.txt $)
```$
type: ALIYUNOSS
config:
endpoint: ""
bucket: ""
access_key_id: ""
access_key_secret: ""
```

Use --objstore.config-file to reference to this configuration file.
2 changes: 2 additions & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,9 @@ require (
cloud.google.com/go v0.44.1
github.com/Azure/azure-storage-blob-go v0.7.0
github.com/NYTimes/gziphandler v1.1.1
github.com/aliyun/aliyun-oss-go-sdk v2.0.1+incompatible
github.com/armon/go-metrics v0.0.0-20190430140413-ec5e00d3c878
github.com/baiyubin/aliyun-sts-go-sdk v0.0.0-20180326062324-cfa1a18b161f // indirect
github.com/cespare/xxhash v1.1.0
github.com/codahale/hdrhistogram v0.0.0-20161010025455-3a0bb77429bd // indirect
github.com/fatih/structtag v1.0.0
Expand Down
4 changes: 4 additions & 0 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf h1:qet1QNfXsQxTZq
github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4 h1:Hs82Z41s6SdL1CELW+XaDYmOH4hkBN4/N9og/AsOv7E=
github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
github.com/aliyun/aliyun-oss-go-sdk v2.0.1+incompatible h1:/MzpJOMHn/uBtd1dkS7Q9PF2ZjT6xTQMXSvv1e6ydXc=
github.com/aliyun/aliyun-oss-go-sdk v2.0.1+incompatible/go.mod h1:T/Aws4fEfogEE9v+HPhhw+CntffsBHJ8nXQCwKr0/g8=
github.com/armon/circbuf v0.0.0-20150827004946-bbbad097214e/go.mod h1:3U/XgcO3hCbHZ8TKRvWD2dDTCfh9M9ya+I9JpbB7O8o=
github.com/armon/go-metrics v0.0.0-20180917152333-f0300d1749da h1:8GUt8eRujhVEGZFFEjBj46YV4rDjvGrNxb0KMWYkL2I=
github.com/armon/go-metrics v0.0.0-20180917152333-f0300d1749da/go.mod h1:Q73ZrmVTwzkszR9V5SSuryQ31EELlFMUz1kKyl939pY=
Expand All @@ -47,6 +49,8 @@ github.com/asaskevich/govalidator v0.0.0-20180720115003-f9ffefc3facf/go.mod h1:l
github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY=
github.com/aws/aws-sdk-go v1.23.12 h1:2UnxgNO6Y5J1OrkXS8XNp0UatDxD1bWHiDT62RDPggI=
github.com/aws/aws-sdk-go v1.23.12/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN924inxo=
github.com/baiyubin/aliyun-sts-go-sdk v0.0.0-20180326062324-cfa1a18b161f h1:ZNv7On9kyUzm7fvRZumSyy/IUiSC7AzL0I1jKKtwooA=
github.com/baiyubin/aliyun-sts-go-sdk v0.0.0-20180326062324-cfa1a18b161f/go.mod h1:AuiFmCCPBSrqvVMvuqFuk0qogytodnVFVSN5CeJB8Gc=
github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973 h1:xJ4a3vCFaGF/jqvzLMYoU8P317H5OQ+Via4RmuPwCS0=
github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
github.com/beorn7/perks v1.0.0 h1:HWo1m869IqiPhD389kmkxeTalrjNbbJTC8LXupb+sl0=
Expand Down
14 changes: 9 additions & 5 deletions pkg/objstore/client/factory.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ import (
"github.com/thanos-io/thanos/pkg/objstore/azure"
"github.com/thanos-io/thanos/pkg/objstore/cos"
"github.com/thanos-io/thanos/pkg/objstore/gcs"
"github.com/thanos-io/thanos/pkg/objstore/oss"
"github.com/thanos-io/thanos/pkg/objstore/s3"
"github.com/thanos-io/thanos/pkg/objstore/swift"
yaml "gopkg.in/yaml.v2"
Expand All @@ -21,11 +22,12 @@ import (
type ObjProvider string

const (
GCS ObjProvider = "GCS"
S3 ObjProvider = "S3"
AZURE ObjProvider = "AZURE"
SWIFT ObjProvider = "SWIFT"
COS ObjProvider = "COS"
GCS ObjProvider = "GCS"
S3 ObjProvider = "S3"
AZURE ObjProvider = "AZURE"
SWIFT ObjProvider = "SWIFT"
COS ObjProvider = "COS"
ALIYUNOSS ObjProvider = "ALIYUNOSS"
)

type BucketConfig struct {
Expand Down Expand Up @@ -59,6 +61,8 @@ func NewBucket(logger log.Logger, confContentYaml []byte, reg prometheus.Registe
bucket, err = swift.NewContainer(logger, config)
case string(COS):
bucket, err = cos.NewBucket(logger, config, component)
case string(ALIYUNOSS):
bucket, err = oss.NewBucket(logger, config, component)
default:
return nil, errors.Errorf("bucket with type %s is not supported", bucketConf.Type)
}
Expand Down
18 changes: 18 additions & 0 deletions pkg/objstore/objtesting/foreach.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import (
"github.com/thanos-io/thanos/pkg/objstore/cos"
"github.com/thanos-io/thanos/pkg/objstore/gcs"
"github.com/thanos-io/thanos/pkg/objstore/inmem"
"github.com/thanos-io/thanos/pkg/objstore/oss"
"github.com/thanos-io/thanos/pkg/objstore/s3"
"github.com/thanos-io/thanos/pkg/objstore/swift"
"github.com/thanos-io/thanos/pkg/testutil"
Expand Down Expand Up @@ -116,4 +117,21 @@ func ForeachStore(t *testing.T, testFn func(t testing.TB, bkt objstore.Bucket))
} else {
t.Log("THANOS_SKIP_TENCENT_COS_TESTS envvar present. Skipping test against Tencent COS.")
}

// Optional OSS.
if _, ok := os.LookupEnv("THANOS_SKIP_ALIYUN_OSS_TESTS"); !ok {
bkt, closeFn, err := oss.NewTestBucket(t)
testutil.Ok(t, err)

ok := t.Run("AliYun oss", func(t *testing.T) {
testFn(t, bkt)
})

closeFn()
if !ok {
return
}
} else {
t.Log("THANOS_SKIP_ALIYUN_OSS_TESTS envvar present. Skipping test against AliYun OSS.")
}
}
Loading