Skip to content
This repository was archived by the owner on Feb 2, 2021. It is now read-only.

Simplify integration testing #19

Merged
merged 4 commits into from
Aug 13, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
FROM ruby:latest

WORKDIR /plugin

ADD . /plugin

RUN gem install bundler && \
gem install fluentd --no-doc && \
fluent-gem build fluent-plugin-azure-storage-append-blob.gemspec && \
fluent-gem install fluent-plugin-azure-storage-append-blob-*.gem

RUN echo "<source>\n\
@type sample\n\
sample {\"hello\":\"world\"}\n\
tag pattern\n\
</source>\n\
<match pattern>\n\
@type azure-storage-append-blob\n\
azure_storage_account \"#{ENV['STORAGE_ACCOUNT']}\"\n\
azure_storage_access_key \"#{ENV['STORAGE_ACCESS_KEY']}\"\n\
azure_storage_sas_token \"#{ENV['STORAGE_SAS_TOKEN']}\"\n\
azure_container fluentd\n\
auto_create_container true\n\
path logs/\n\
azure_object_key_format %{path}%{time_slice}_%{index}.log\n\
time_slice_format %Y%m%d-%H\n\
<buffer tag,time>\n\
@type file\n\
path /var/log/fluent/azurestorageappendblob\n\
timekey 120 # 2 minutes\n\
timekey_wait 60\n\
timekey_use_utc true # use utc\n\
</buffer>\n\
</match>" > /plugin/fluent.conf

ENTRYPOINT ["fluentd", "-c", "fluent.conf"]
27 changes: 27 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,33 @@ Format of the time used in the file name. Default is '%Y%m%d'. Use '%Y%m%d%H' to
bundle install
bundle exec rake test


### Test Fluentd

1. Create Storage Account and VM with enabled MSI
2. Setup Docker ang Git
3. SSH into VM
4. Download this repo
```
git clone https://github.com/microsoft/fluent-plugin-azure-storage-append-blob.git
cd fluent-plugin-azure-storage-append-blob
```
5. Build Docker image
`docker build -t fluent .`
6. Run Docker image with different set of parameters:

1. `STORAGE_ACCOUNT`: required, name of your storage account
2. `STORAGE_ACCESS_KEY`: storage account access key
3. `STORAGE_SAS_TOKEN`: storage sas token with enough permissions for the plugin

You need to specify `STORAGE_ACCOUNT` and one of auth ways. If you run it from VM with MSI,
just `STORAGE_ACCOUNT` is required. Keep in mind, there is no way to refresh MSI Token, so
ensure you setup proper permissions first.

```bash
docker run -it -e STORAGE_ACCOUNT=<storage> -e STORAGE_ACCESS_KEY=<key> fluent
```

## Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a
Expand Down
6 changes: 3 additions & 3 deletions lib/fluent/plugin/out_azure-storage-append-blob.rb
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ def configure(conf)

raise ConfigError, 'azure_container needs to be specified' if @azure_container.nil?

if @azure_storage_access_key.nil? && @azure_storage_sas_token.nil?
if (@azure_storage_access_key.nil? || @azure_storage_access_key.empty?) && (@azure_storage_sas_token.nil? || @azure_storage_sas_token.empty?)
log.info 'Using MSI since neither azure_storage_access_key nor azure_storage_sas_token was provided.'
@use_msi = true
end
Expand Down Expand Up @@ -105,9 +105,9 @@ def start
else
@bs_params = { storage_account_name: @azure_storage_account }

if !@azure_storage_access_key.nil?
if !@azure_storage_access_key.nil? && !@azure_storage_access_key.empty?
@bs_params.merge!({ storage_access_key: @azure_storage_access_key })
elsif !@azure_storage_sas_token.nil?
elsif !@azure_storage_sas_token.nil? && !@azure_storage_sas_token.empty?
@bs_params.merge!({ storage_sas_token: @azure_storage_sas_token })
end

Expand Down