[FEATURE] Need more straightforward usage examples.

[xiezhipeng-git]

I have reviewed the usage examples once, but I still don't fully understand how to use it. Now I hope to complete a simple task as an example according to the data I provided.
A simple multi-variable addition task

a = [i for i in range(10)]
b = [i for i in range(10,0,-1)]
# c=a+b
def add(a,b):
    return a+b
c = map(add,a,b)
for i in c:
    print(i)

How can the above code be efficiently implemented using pyper? It is convenient to subsequently modify and input other types of data, such as large model generation parameters, for parallel reasoning.
@pyper-dev

[RichardZhu2]

Hi!

I don't think I have a perfect understanding of your use case, but I can definitely give an example of how the above code could be implemented. The basic assumption of Pyper is more or less that data is processed in steps and that those steps can be represented as functions (tasks). So we need to write some suitable functions for processing the data, then put them in a pipeline. That could look like:

from pyper import task


def main():
    a = [i for i in range(10)]
    b = [i for i in range(10, 0, -1)]

    pipeline = (
        task(lambda: zip(a, b), branch=True)
        | task(sum, workers=10, multiprocess=True)
    )
    for result in pipeline():
        print(result)


if __name__ =="__main__":
    main()

map(add, a, b) effectively calls the add function on every value from zip(a, b). So I'm using zip(a, b) as the first task, with branch=True to generate all a-b inputs. The second task does the adding for each set of inputs. I'm just using the sum function, but you can define your own function, provided it takes one parameter:

def add_inputs(inputs):
    return inputs[0] + inputs[1]

...

pipeline = (
    task(lambda: zip(a, b), branch=True)
    | task(add_inputs, workers=10, multiprocess=True)
)

In an upcoming version, I'm working on an unpack feature so that the way logical functions can be defined is a little more flexible:

def add(a, b):
    return a + b

...

pipeline = (
    task(lambda: zip(a, b), branch=True)
    | task(add, workers=10, multiprocess=True, unpack=True)
)

Which could make things a bit neater.

Anyway -- the general usage of Pyper revolves around building out your data processing logic as Python functions, then linking those functions together into a pipeline with task. And once in a pipeline, you'd of course benefit from possibility of concurrent/parallel execution. I'd be very happy to discuss other examples if you'd like.

[xiezhipeng-git]

Hi!

I don't think I have a perfect understanding of your use case, but I can definitely give an example of how the above code could be implemented. The basic assumption of Pyper is more or less that data is processed in steps and that those steps can be represented as functions (tasks). So we need to write some suitable functions for processing the data, then put them in a pipeline. That could look like:

from pyper import task

def main():
a = [i for i in range(10)]
b = [i for i in range(10, 0, -1)]

pipeline = (
    task(lambda: zip(a, b), branch=True)
    | task(sum, workers=10, multiprocess=True)
)
for result in pipeline():
    print(result)

if name =="main":
main()
map(add, a, b) effectively calls the function on every value from . So I'm using as the first task, with to generate all a-b inputs. The second task does the adding for each set of inputs. I'm just using the function, but you can define your own function, provided it takes one parameter:add``zip(a, b)``zip(a, b)``branch=True``sum

def add_inputs(inputs):
return inputs[0] + inputs[1]

...

pipeline = (
task(lambda: zip(a, b), branch=True)
| task(add_inputs, workers=10, multiprocess=True)
)
In an upcoming version, I'm working on an feature so that the way logical functions can be defined is a little more flexible:unpack

def add(a, b):
return a + b

...

pipeline = (
task(lambda: zip(a, b), branch=True)
| task(add, workers=10, multiprocess=True, unpack=True)
)
Which could make things a bit neater.

Anyway -- the general usage of Pyper revolves around building out your data processing logic as Python functions, then linking those functions together into a pipeline with . And once in a pipeline, you'd of course benefit from possibility of concurrent/parallel execution. I'd be very happy to discuss other examples if you'd like.task

task(lambda: zip(a, b), branch=True)

This is exactly the example of multivariate input using Pyper that I want. The documentation does not introduce this simple and important usage scenario.That is to say, when dealing with multiple variables, a complete variable needs to be input at the place where branch=True to simulate multiple variable inputs.
Besides, is it only by using this method task(step3, workers=20, multiprocess=True) that a breakthrough can be achieved GIL?
@RichardZhu2

[RichardZhu2]

That's right -- setting multiprocess=True is the way to get around the GIL. Or theoretically by using the free-threaded build of 3.13, but I won't yet recommend this due to lack of testing + its experimental status.

The past month has been pretty busy, but in a few weeks time I'd like to start fleshing out the Examples section in the documentation and will make a note to include this usage.