Skip to content

Commit

Permalink
Merge pull request #17 from ericbrisrubio/gwm-14
Browse files Browse the repository at this point in the history
[gwm-14] - Adding code example to README file
  • Loading branch information
ericbrisrubio authored May 4, 2020
2 parents 0309977 + f0f2ebd commit edb51af
Show file tree
Hide file tree
Showing 3 changed files with 89 additions and 15 deletions.
100 changes: 86 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,21 +1,12 @@
# go-workers-multipool
Manager for multiple pools of workers in Golang.
Manager for multiple(unlimited) pools of workers in Golang.

This is a wrapper to manage multiple pool of workers of the kind:
This is a wrapper to manage multiple pool of workers, currently of the kind:
https://github.com/enriquebris/goworkerpool

You can easily have many pool to execute different tasks and manage them separately.
You can easily have many pools to execute different tasks and manage them separately.

### Example Use Case:
System to resize images

The system could use 2 different pools of workers, one to process low size images (< 5 MB), and a second pool to process
the rest (> 5 MB). Since per the history, 70% of elements coming are bigger than 5 MB, and we want to do some extra work
on them without affecting the conversion of the smaller ones, the pool for the bigger images could have more workers than
the other. If at some point, we determine that we need only half of the current amount of workers converting big images,
we can cut them in half on the fly without affecting anything else and without restarting the system.

Current Features:
## Current Features:
- Add multiple pools to be managed at the same time
- Define a function for the workers on a specific pool
- Start the workers on a pool
Expand All @@ -25,11 +16,92 @@ Current Features:
- Pause all the workers for a pool
- Resume all the workers for a pool

System Overview:
### System Overview:

![system-overview](./go-workers-multipool-Overview.svg)

### Class Diagram:
blablbablba

## Example Use Case:
### System to resize images

This system could use 2 different pools of workers, one to process low size images (< 5 MB), and a second pool to process
the rest (> 5 MB). Since per the history, 70% of elements coming are bigger than 5 MB, and we want to do some extra work
on them without affecting the conversion of the smaller ones, the pool for the bigger images could have more workers than
the other. If at some point, we determine that we need only half of the current amount of workers converting big images,
we can cut them in half on the fly without affecting anything else and without restarting the system.




### Quick Start:

```go get github.com/ericbrisrubio/go-workers-multipool@v0.1-alpha.2```

#### Example code:

(Based on the previous use case about images conversion)

```go
package main

import (
"fmt"
"github.com/ericbrisrubio/go-workers-multipool/manager"
"time"
)

func main() {

poolsManager := manager.Manager{}

// low-size pool definition
poolsManager.AddPool("low-size", 3, 10, false)
lowSizeFunc := func(data interface{}) bool{
fmt.Printf("processed %s \n", data)
return true
}
poolsManager.SetFunc("low-size", lowSizeFunc)
poolsManager.StartPool("low-size")

// big-size pool definition
poolsManager.AddPool("big-size", 2, 10, false)
bigSizeFunc := func(data interface{}) bool{
fmt.Printf("processed %s \n", data)
return true
}
poolsManager.SetFunc("big-size", bigSizeFunc)
poolsManager.StartPool("big-size")

//Start sending tasks to the low-size pool
go func() {
for {
time.Sleep(time.Second*3)
poolsManager.AddTaskToPool("low-size", "{image-path} for low size image")
}
}()

//Start sending tasks to the big-size pool
go func() {
for {
time.Sleep(time.Second*2)
poolsManager.AddTaskToPool("big-size", "{image-path} for big-size image")
}
}()

//Finish both pools processing after 10 seconds
go func() {
time.Sleep(time.Second*10)
poolsManager.KillWorkersFromPool("low-size", 3)
poolsManager.KillWorkersFromPool("big-size", 2)
}()

//Wait (blocks) until both pools has ended up processing the tasks
poolsManager.WaitForAllPools()

}
```


### MIT License
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -11,5 +11,5 @@ require (
github.com/spf13/afero v1.2.2 // indirect
github.com/stretchr/testify v1.5.1 // indirect
golang.org/x/lint v0.0.0-20200302205851-738671d3881b // indirect
golang.org/x/tools v0.0.0-20200502202811-ed308ab3e770 // indirect
golang.org/x/tools v0.0.0-20200504193531-9bfbc385433f // indirect
)
2 changes: 2 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,8 @@ golang.org/x/tools v0.0.0-20200430040329-4b814e061378 h1:9PP65Qh7XP8c0HOR8HZXX5m
golang.org/x/tools v0.0.0-20200430040329-4b814e061378/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20200502202811-ed308ab3e770 h1:M9Fif0OxNji8w+HvmhVQ8KJtiZOsjU9RgslJGhn95XE=
golang.org/x/tools v0.0.0-20200502202811-ed308ab3e770/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20200504193531-9bfbc385433f h1:sPpKxOcxUWPeIIqQbn06sX/NBtDl45ZGdi67lZECBsw=
golang.org/x/tools v0.0.0-20200504193531-9bfbc385433f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
Expand Down

0 comments on commit edb51af

Please sign in to comment.