Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: PowerDNS/simpleblob
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v0.2.7
Choose a base ref
...
head repository: PowerDNS/simpleblob
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: v0.3.0
Choose a head ref
  • 4 commits
  • 18 files changed
  • 2 contributors

Commits on Oct 17, 2024

  1. io-based stream optional interfaces (#11)

    * create stream-related types
    
    Created simpleblob.NewReader and simpleblob.NewWriter functions, along with ReaderStorage and WriterStorage interface types.
    Those functions allow to provide an optimized way to write to and read from backend if those interfaces are satsfied.
    If an interface is not satisfied by backend, a fallback implementation is used, akin to `fs.GlobFS`.
    
    * implement ReaderStorage and WriterStorage for fs backend
    
    Provide io.ReadCloser and io.WriteCloser to interact with files to way the `os` package does it.
    The `os` package has byte-slice based functions that rely on streams under the hood, we're doing the same here.
    
    * implement ReaderStorage for S3 backend
    
    The Object returned by MinIO client satisfies io.ReadCloser, so this implementation is very straightforward.
    
    For writing, the MinIO client takes a reader and needs the full size of the stored object.
    Thus, we'd need to exhaust the writer into a buffer, take its size and provide it to the client,
    which is basically what the fallback implementation of NewWriter does.
    
    * add tests for ReaderStorage and WriterStorage
    
    Confront the results from simpleblob.NewReader and simpleblob.NewWriter with simpleblob.Interface.
    
    * more robust implementation of general WriterStorage wrapper
    
    * fix WriterStorage doc typo
    
    * fix writer wrapper mutex usage
    
    * show that param is name in ReaderStorage
    
    Co-authored-by: wojas <github@m.wojas.nl>
    
    * replace reader wrapper struct with io.NopCloser
    
    * fix "mixed name and unnamed parameters" syntax error
    
    * replace errorWClosed with ErrClosed
    
    * express intent to check name validity on stream wrapper struct
    
    * rename default `writer` wrapper struct to `fallbackWriter`
    
    * formatting
    
    * s3: use common method for Load and NewReader
    
    * test presence of key after storing with NewWriter
    
    * remove duplicate error check
    
    * s3 backend: implement StorageWriter
    
    * apply comment improvement suggestions
    
    Co-authored-by: wojas <github@m.wojas.nl>
    
    * s3: simplify doStoreReader and doLoadReader
    
    * tester: test atomicity of writer, and write/read after close
    
    * s3: close pipe, better context cancellation handling
    
    * fs backend: write file atomically with NewWriter
    
    * s3: move prefix prepending and metrics calls to their correct places
    
    * s3: rename ReaderStorage->StreamReader, WriterStorage->StreamWriter
    
    * s3: revert to default NumThreads in PutObjects
    
    * s3: prepend global prefix where missing
    
    * s3: make a comment about global prefix use with marker clearer
    
    * s3: introduce NumMinioThreads to configure NumThreads in minio client
    
    * s3: move stream logic to separate file, implement ReaderFrom
    
    * fs: move stream logic to separate file
    
    * s3: remove unfitting io.ReaderFrom implementation
    
    * s3: stream: start pipe earlier, do not write marker twice
    
    * stream: catch cancelled ctx early in fs & s3
    
    * stream: remove name validation TODO comment
    
    General name validation was dropped in favour of documenting safe names.
    
    * io stream: update README and make a couple comments clearer
    
    ---------
    
    Co-authored-by: wojas <github@m.wojas.nl>
    ahouene and wojas authored Oct 17, 2024
    Configuration menu
    Copy the full SHA
    223250b View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2024

  1. S3: deprecate PrefixFolders and introduce HideFolders (#62)

    * s3: deprecate PrefixFolders and introduce HideFolders
    
    * s3: check suffix only for HideFolders
    
    * s3: add separate test func for HideFolders
    
    * s3: add details about use of folders in HideFolders doc comment
    ahouene authored Oct 21, 2024
    Configuration menu
    Copy the full SHA
    94dd98d View commit details
    Browse the repository at this point in the history
  2. Upgrade Go & dependencies, drop unsupported Go versions (#78)

    * upgrade go1.22.8, dependencies, github actions
    
    * go.mod: use toolchain go1.23.2
    ahouene authored Oct 21, 2024
    Configuration menu
    Copy the full SHA
    1cbae45 View commit details
    Browse the repository at this point in the history
  3. S3 tests: use testcontainers (#79)

    * S3 tests: use testcontainers
    
    Remove internal s3testing package in favour of testcontainers, as suggested in #61.
    
    * s3 tests: skip tests based on testcontainers if no provider
    ahouene authored Oct 21, 2024
    Configuration menu
    Copy the full SHA
    c412150 View commit details
    Browse the repository at this point in the history
Loading