Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RAPIDE & IPSL: first proof of concept #148

Closed
wants to merge 35 commits into from
Closed

RAPIDE & IPSL: first proof of concept #148

wants to merge 35 commits into from

Commits on Feb 3, 2023

  1. ipsl: compiler and mesh for ipsl structure 

    This still requires lots of work.

This include a mostly complete compiler, (minus a few types and literals).
More nodes and scopes will be required.
    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    15a26fb View commit details
    Browse the repository at this point in the history

  2. ipsl: add scope test

    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    50a362d View commit details
    Browse the repository at this point in the history

  3. ipsl: remove type field from SomeNode and use reflection on Node instead

    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    37a81a2 View commit details
    Browse the repository at this point in the history

  4. ipsl: add empty builtin

    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    cb5c976 View commit details
    Browse the repository at this point in the history

  5. ipsl/helpers: add SyncDFS helper

    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    a78774b View commit details
    Browse the repository at this point in the history

  6. ipsl/unixfs: add Everything node

    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    d66c1e5 View commit details
    Browse the repository at this point in the history

  7. ipsl: add a comment to the compiler tests

    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    8ed4783 View commit details
    Browse the repository at this point in the history

  8. ipsl/helpers: test ipsl.All node in SyncDFS test

    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    123ccb7 View commit details
    Browse the repository at this point in the history

  9. ipsl: use blocks interface 

    Will be fine performance wise once #57 is fixed.
    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    877a66b View commit details
    Browse the repository at this point in the history

  10. ipsl: return UnexpectedEOF with broken nodes

    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    67b755e View commit details
    Browse the repository at this point in the history

  11. ipsl: add support for string literal

    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    9fc24d7 View commit details
    Browse the repository at this point in the history

  12. ipsl: add None support

    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    c0311ca View commit details
    Browse the repository at this point in the history

  13. ipsl: add support for load-builtin-scope

    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    8e03d43 View commit details
    Browse the repository at this point in the history

  14. ipsl: rename compileNodeWithoutClosure to compileNextNodeWithoutTermi… 

    …nation
    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    92315b6 View commit details
    Browse the repository at this point in the history

  15. ipsl: add unixfs scope object

    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    090dc45 View commit details
    Browse the repository at this point in the history

  16. ipsl: make CompileEmpty and CompileAll private

    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    b71e345 View commit details
    Browse the repository at this point in the history

  17. blocks: add BlockIterator

    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    604d757 View commit details
    Browse the repository at this point in the history

  18. blocks: add BlockOrError

    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    f24da5b View commit details
    Browse the repository at this point in the history

  19. ipsl: make all run the child traversals

    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    432e9aa View commit details
    Browse the repository at this point in the history

  20. rapide: initial commit 

    This include a working single threaded structure for RAPIDE.

Todo:
- Multithreading
- dontGoThere map for serverDrivenWorker
- Racing
    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    031ebc7 View commit details
    Browse the repository at this point in the history

  21. rapide: fix termination logic

    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    9c48161 View commit details
    Browse the repository at this point in the history

  22. rapide/gateway: change User-Agent to RAPIDE

    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    608fa43 View commit details
    Browse the repository at this point in the history

  23. rapide: refactor serverDrivenWork.work to not be hot garbage

    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    40cb2d2 View commit details
    Browse the repository at this point in the history

  24. rapide: fully remove workers from the tree when they fail

    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    a778ca3 View commit details
    Browse the repository at this point in the history

  25. rapide: gc traversal objects in expand

    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    419cdfa View commit details
    Browse the repository at this point in the history

  26. rapide: return worker errors when all workers died

    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    0c5c6f0 View commit details
    Browse the repository at this point in the history

  27. rapide/gateway: implement fmt.Stringer with PathName

    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    b3db9e8 View commit details
    Browse the repository at this point in the history

  28. rapide: disable eager check for len(w.tasks) == 0 in the server worker

    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    0310c77 View commit details
    Browse the repository at this point in the history

  29. rapide: randomly distribute equivalent walks 

    This help avoiding situations where you go race someone else, in a shallow dag, and you get a block next to them in the traversal.
    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    cebbabf View commit details
    Browse the repository at this point in the history

  30. rapide: add make use of io.Closer for BlockIterators

    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    6e74518 View commit details
    Browse the repository at this point in the history

  31. rapide: fix leaking worker count when we reset after duplicated block

    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    d02cbce View commit details
    Browse the repository at this point in the history

  32. rapide: add tests and fix race issues

    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    c5905d7 View commit details
    Browse the repository at this point in the history

  33. ipsl: remove the compiler and language 

    We do not have consensus nor usecase for the language part of IPSL,
to avoid long debates for some not yet usefull pieces of code,
I've removed it and we will handle merging this part seperately (or never merge it).
    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    bc755ab View commit details
    Browse the repository at this point in the history

  34. rapide: fix: make unexpected blocks a retriable error 

    In 719b5e6 I changed `download.expand` to release the node lock before returning.
This allows other workers to access the just expanded node while we walk up the chain ancestry.
This mean that when we relock the node later to add it to our list of tasks, someone else may have downloaded some of the blocks already and thus they are not in the childrens slice and thus we wont add them to our list.

This also solve the case where you give a Traversal that is not supported by the underlying protocol,
in that case we want to kill the current download and retry deeper, while previously we would hard error on this and kill the worker.
    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    021b3c9 View commit details
    Browse the repository at this point in the history

  35. examples/rapido: add example on how to use rapide

    @Jorropo
    Jorropo committed Feb 3, 2023
    Configuration menu
    Copy the full SHA
    143cd01 View commit details
    Browse the repository at this point in the history