Direct IO which is asynchronous #553

phischu · 2024-08-23T12:59:08Z

TODO

Fix in JS backend io/console.effekt
Implement in LLVM backend io/console.effekt
Fix in JS backend io/network.effekt
Implement in LLVM backend io/network.effekt
Rename io/files.effekt to io/filesystem.effekt
Rename run_Pos to runPositive etc.
Check and handle error result in call to uv_fs_open etc.
Find and fix memory leaks

libraries/common/io/files.effekt

libraries/llvm/io.c

libraries/js/io.js

effekt/shared/src/main/scala/effekt/generator/llvm/PrettyPrinter.scala

b-studios

In general I like the direction of this PR, a lot.

Could you please:

rebase it (this will get rid of the additional control flag on externs in lifted, you can delete that again, since we don't plan on supporting the other backends any longer)
automatically insert a call to eventloop if main contains control?
check indentation (tabs are used in C files quite a lot)

In general, we need to decide which functions should be hidden behind namespaces like direct and promises and which don't.

Can you showcase somewhere how to recover the callback-based version, based on the direct-style one? I think a test or something to document it would be great.

effekt/shared/src/main/scala/effekt/lifted/LiftInference.scala

effekt/shared/src/main/scala/effekt/lifted/Tree.scala

examples/stdlib/io/files/async_file_io.effekt

examples/stdlib/io/promise.effekt

libraries/common/io.effekt

b-studios · 2024-09-10T18:34:03Z

libraries/llvm/io.c

+    char* path_str = c_buffer_as_null_terminated_string(path);
+    erasePositive((struct Pos) path);
+
+    // Convert the Effekt String representing the opening mode to libuv flags


Suggested change

// Convert the Effekt String representing the opening mode to libuv flags

// Convert the Effekt Mode-Datatype representing the opening mode to libuv flags

libraries/llvm/io.c

b-studios · 2024-09-10T18:35:27Z

libraries/llvm/io.c

+    request->data = stack;
+
+    uv_fs_open(uv_default_loop(), request, path_str, flags, 0666, c_resume_int_fs);
+    // TODO report result (UV_EINVAL)


Do we need to free stuff when this errors? This probably means the continuation is never called and also needs to be ref-count-decremented.

libraries/llvm/io.c

b-studios · 2024-09-23T14:08:36Z

libraries/llvm/io.c

+    char* bytes = c_buffer_bytes(buffer); // libuv expects signed integers
+    uv_buf_t buf = uv_buf_init(bytes, c_buffer_length(buffer));
+    // erasePositive(buffer);
+    // TODO we should erase the buffer but abort if this was the last reference


You mean: decrement the RC but don't free?

This requires rethinking of the whole bytes/buffer/string story. Postponing.

Postponed solution:

a)

allocate a user-defined struct Stack x Buffer

erase in the continuation / callback

b)

erase here, but make sure that it is USED on the Effekt side.

b-studios · 2024-09-24T14:55:55Z

Please delete / fix all occurrences of AsyncIO, then this is ready to merge.

b-studios · 2024-09-25T14:27:35Z

Thanks, this is a huge step forward!

phischu changed the title ~~Extern control defs in llvm~~ Direct IO which is still asynchronous Aug 23, 2024

phischu added 2 commits August 23, 2024 16:50

Extern control defs in llvm

301bfb7

Resume stacks with an argument

9bde62c

phischu force-pushed the library/direct-io branch from eb73415 to 9bde62c Compare August 23, 2024 14:52

phischu added 3 commits August 30, 2024 16:39

Avoid passing evidence to extern control functions

c1df74e

Get first example running

d063f70

Bindings for other operations on files

19755a5

b-studios reviewed Sep 2, 2024

View reviewed changes

libraries/common/io/files.effekt Outdated Show resolved Hide resolved

Replace callbacks with direct style

0f901a4

b-studios reviewed Sep 2, 2024

View reviewed changes

libraries/common/io/files.effekt Outdated Show resolved Hide resolved

phischu added 2 commits September 3, 2024 11:22

Same for JS

1abf3cd

Same for timer

ebd98e0

phischu changed the title ~~Direct IO which is still asynchronous~~ Direct IO which is asynchronous Sep 3, 2024

phischu added 3 commits September 4, 2024 09:48

Delete stuff

eb98784

Rename functions

1c26394

Revise C code

fe51528

b-studios reviewed Sep 4, 2024

View reviewed changes

libraries/llvm/io.c Outdated Show resolved Hide resolved

phischu added 4 commits September 4, 2024 11:22

Rename some modes

34c9173

Unify treatment of modes

acfaf06

Error handling

50b78de

Bring back yield and abort

decc202

b-studios reviewed Sep 4, 2024

View reviewed changes

libraries/js/io.js Show resolved Hide resolved

phischu added 3 commits September 5, 2024 12:12

Spawn independent tasks

e8e7cfe

Revise promises

4d770b2

Make tests pass

bfea733

phischu marked this pull request as ready for review September 6, 2024 15:56

b-studios reviewed Sep 10, 2024

View reviewed changes

effekt/shared/src/main/scala/effekt/generator/llvm/PrettyPrinter.scala Show resolved Hide resolved

b-studios reviewed Sep 10, 2024

View reviewed changes

b-studios mentioned this pull request Sep 11, 2024

Use splices in LLVM templates #587

Merged

Temporarily undo introduction of control to lifted to simplify merging

c31d7ad

phischu added 3 commits September 23, 2024 11:37

Check errors when reading and writing files

eca5fc1

Rename parameter type

cd1ff9b

Report submission errors to user

c62f54a

b-studios reviewed Sep 23, 2024

View reviewed changes

libraries/llvm/io.c Outdated Show resolved Hide resolved

b-studios reviewed Sep 23, 2024

View reviewed changes

libraries/llvm/io.c Outdated Show resolved Hide resolved

b-studios reviewed Sep 23, 2024

View reviewed changes

libraries/llvm/io.c Outdated Show resolved Hide resolved

b-studios reviewed Sep 23, 2024

View reviewed changes

libraries/llvm/io.c Outdated Show resolved Hide resolved

b-studios reviewed Sep 23, 2024

View reviewed changes

libraries/llvm/io.c Outdated Show resolved Hide resolved

phischu force-pushed the library/direct-io branch 2 times, most recently from 703ddee to 31af30d Compare September 23, 2024 11:29

Rename files module to filesystem

81c7888

phischu force-pushed the library/direct-io branch from 31af30d to 81c7888 Compare September 23, 2024 11:33

phischu and others added 4 commits September 23, 2024 14:02

Add wordcount test

6526694

Make server runnable

bea534f

Merge branch 'master' into library/direct-io

31403f7

Investigate buffer space leak

e8fb516

b-studios reviewed Sep 23, 2024

View reviewed changes

Merge branch 'master' into library/direct-io

210e62a

jiribenes added the area:stdlib label Sep 25, 2024

phischu force-pushed the library/direct-io branch from 5b84967 to b1442a5 Compare September 25, 2024 12:24

Fix console for JS

4798f5e

phischu force-pushed the library/direct-io branch from b1442a5 to 4798f5e Compare September 25, 2024 12:32

phischu added 2 commits September 25, 2024 16:06

Merge branch 'master' into library/direct-io

e3a4710

Expose erasing and sharing of stacks

e15b78c

b-studios merged commit b5ff081 into master Sep 25, 2024
2 checks passed

b-studios deleted the library/direct-io branch September 25, 2024 14:27

jiribenes mentioned this pull request Sep 25, 2024

Functions outside of the event loop segfault on LLVM #504

Closed

dvdvgt mentioned this pull request Sep 25, 2024

Effekt by example #536

Merged

19 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Direct IO which is asynchronous #553

Direct IO which is asynchronous #553

phischu commented Aug 23, 2024 •

edited

Loading

b-studios left a comment •

edited

Loading

b-studios Sep 10, 2024

b-studios Sep 10, 2024

b-studios Sep 23, 2024

phischu Sep 23, 2024

b-studios Sep 24, 2024

b-studios commented Sep 24, 2024

b-studios commented Sep 25, 2024

	// Convert the Effekt String representing the opening mode to libuv flags
	// Convert the Effekt Mode-Datatype representing the opening mode to libuv flags

Direct IO which is asynchronous #553

Direct IO which is asynchronous #553

Conversation

phischu commented Aug 23, 2024 • edited Loading

b-studios left a comment • edited Loading

Choose a reason for hiding this comment

b-studios Sep 10, 2024

Choose a reason for hiding this comment

b-studios Sep 10, 2024

Choose a reason for hiding this comment

b-studios Sep 23, 2024

Choose a reason for hiding this comment

phischu Sep 23, 2024

Choose a reason for hiding this comment

b-studios Sep 24, 2024

Choose a reason for hiding this comment

b-studios commented Sep 24, 2024

b-studios commented Sep 25, 2024

phischu commented Aug 23, 2024 •

edited

Loading

b-studios left a comment •

edited

Loading