add Fiber (sackful coroutine) support

  100335
August 31, 2017 14:28 i@lvht.net (Haitao Lv)
Hi, All,

The generator has been introduced in PHP 5.5. And we can it pause a function execution.
With these feature, we can schedule multi io task in userland without blocking.

More details can be found at [Nikic].

And there is also some asynchronous frameworks like [AMP], which allow programer to 
code like blocking, but run asynchronously.

However, like python's generator, lua's coroutine, javascript's async/await, php's
generator also is a stackless coroutine implementation. We cannot pause a function call
in it's sub function call. This is the reason that frameworks like [AMP] has many wrapper,
is hard to understand, and almost impossible to be used to implement real but complex system.

So I propose to introduce the sackful coroutine, aka Fiber, support for PHP. And the possible
API like this,

> function foo($a) > { > $b = await $a + 1; > echo $b; > } > > function bar() > { > foo(); > } > > $f = new Fiber(function($a) { > bar($a); > return 3; > }); > > $c = $f->resume(1); > // 1 will be passed to lambda, then bar > // bar call foo, and the **await** in foo paused this execution > // and make the return value of resume as $a + 1 > // so $c is 2 > > $c = $f->resume(3); > // resume the execution by previous await, and the $b in foo > // will be assigned a value of 3 > // so foo echo 3 and return and then the lambda return > // and the resume got a return value of 3. so $c is 3.
So the Fiber API is a little like the Generator API, but is more simple yet powerful. So there is no need to distinct $generator->current(), $generator->send(), and $generator->getReturn(). I made a more complex example at [FIREPHP].And I also make a [PR] for comment. All comment are welcome. Thanks. [Nikic] https://nikic.github.io/2012/12/22/Cooperative-multitasking-using-coroutines-in-PHP.html [AMP] http://amphp.org/ [FIREPHP] https://github.com/fiberphp/fiber-core/blob/master/app.php [PR] https://github.com/php/php-src/pull/2723
  100342
September 1, 2017 12:45 rowan.collins@gmail.com (Rowan Collins)
On 31 August 2017 15:28:36 BST, Haitao Lv <i@lvht.net> wrote:
>So I propose to introduce the sackful coroutine, aka Fiber, support for >PHP.
Hi! Is this your own invention, or is the name and semantics based on some existing language or computer science theory? "Fiber" makes me think of strings, rather than coroutines, so maybe I'm missing a key metaphor here.
>> > function foo($a) >> { >> $b = await $a + 1; >> echo $b; >> }
Should "await" be "yield" here? If not, what happens if I call foo() without being inside a Fiber? Similarly, if I run bar() and it runs foo(), what result will I get outside a Fiber, since bar() has no yield, so is not itself a generator?
>So the Fiber API is a little like the Generator API, but is more simple >yet powerful. So there >is no need to distinct $generator->current(), $generator->send(), and >$generator->getReturn().
I don't really follow how merging the current() and send() methods is related to whether the coroutine was paused further into the stack. Your ->resume() doesn't seem to do anything a normal generator can't, except that the yield is nested deeper. Nor does it seem to replace getReturn() (although I struggled to understand the rationale for that addition at the time, others swore it was necessary). Meanwhile, how does this relate to other ways of combining generators, such as "yield from"? I do struggle with coroutines, so I may be missing something obvious here. Regards, -- Rowan Collins [IMSoP]
  100343
September 1, 2017 13:02 i@lvht.net (Haitao Lv)
> On 1 Sep 2017, at 20:45, Rowan Collins collins@gmail.com> wrote: > > Is this your own invention, or is the name and semantics based on some existing language or computer science theory? "Fiber" makes me think of strings, rather than coroutines, so maybe I'm missing a key metaphor here.
Fiber is a lightweight thread. Please see https://en.wikipedia.org/wiki/Fiber_(computer_science) And ruby support Fiber. Please see https://ruby-doc.org/core-2.4.1/Fiber.html
> >>> >> function foo($a) >>> { >>> $b = await $a + 1; >>> echo $b; >>> } > > Should "await" be "yield" here? If not, what happens if I call foo() without being inside a Fiber?
No. A function with a yield will always return a Generator object. A fiber with await will return any value. You can see the await as a resumable return. The value after it will be returned and the function will be paused.
> > Similarly, if I run bar() and it runs foo(), what result will I get outside a Fiber, since bar() has no yield, so is not itself a generator? > > >> So the Fiber API is a little like the Generator API, but is more simple >> yet powerful. So there >> is no need to distinct $generator->current(), $generator->send(), and >> $generator->getReturn(). > > Your ->resume() doesn't seem to do anything a normal generator can't, except that the yield is nested deeper.
The await is a nested deeper yield!
> Nor does it seem to replace getReturn().
resume(); // will output 1
> Meanwhile, how does this relate to other ways of combining generators, such as "yield from"?
You can use Fiber to implement a more simple generator, because fiber can be paused/resumed in its deeper call. resume(); // will output 1 echo $f->resume(); // will output 2 echo $f->resume(); // will output 3 echo $f->resume(); // will output 4 echo $f->resume(); // will output 5 echo $f->resume(); // will output 6
  100344
September 1, 2017 14:25 me@kelunik.com (Niklas Keller)
2017-09-01 15:02 GMT+02:00 Haitao Lv <i@lvht.net>:

> > > On 1 Sep 2017, at 20:45, Rowan Collins collins@gmail.com> wrote: > > > > Is this your own invention, or is the name and semantics based on some > existing language or computer science theory? "Fiber" makes me think of > strings, rather than coroutines, so maybe I'm missing a key metaphor here. > > Fiber is a lightweight thread. Please see > https://en.wikipedia.orghttps://github.com/amphp/artax/blob/master/lib/HttpTunneler.php/wiki/ > Fiber_(computer_science) > <https://en.wikipedia.org/wiki/Fiber_(computer_science)> > And ruby support Fiber. Please see https://ruby-doc.org/core-2.4. > 1/Fiber.html > > > > >>> >>> function foo($a) > >>> { > >>> $b = await $a + 1; > >>> echo $b; > >>> } > > > > Should "await" be "yield" here? If not, what happens if I call foo() > without being inside a Fiber? > > No. A function with a yield will always return a Generator object. A fiber > with await will return any > value. You can see the await as a resumable return. The value after it > will be returned and the function > will be paused. > > > > > Similarly, if I run bar() and it runs foo(), what result will I get > outside a Fiber, since bar() has no yield, so is not itself a generator? > > > > > >> So the Fiber API is a little like the Generator API, but is more simple > >> yet powerful. So there > >> is no need to distinct $generator->current(), $generator->send(), and > >> $generator->getReturn(). > > > > Your ->resume() doesn't seem to do anything a normal generator can't, > except that the yield is nested deeper. > > The await is a nested deeper yield! > > > Nor does it seem to replace getReturn(). > > $f = new Fiber(function ($a) { > return 1; > }); > > echo $f->resume(); // will output 1 > > > Meanwhile, how does this relate to other ways of combining generators, > such as "yield from"? > > You can use Fiber to implement a more simple generator, because fiber can > be paused/resumed in its deeper > call. > > function bar() > { > foreach ([5,6] as $i) { > await $i; > } > } > function foo() > { > foreach ([3,4] as $i) { > await $i; > } > bar(); > } > $f = new Fiber(function ($a) { > await 1; > foo(); > }); > > echo $f->resume(); // will output 1 > echo $f->resume(); // will output 2 > echo $f->resume(); // will output 3 > echo $f->resume(); // will output 4 > echo $f->resume(); // will output 5 > echo $f->resume(); // will output 6
Fibres / Green Threads solve the problem that coroutine- / promise-based APIs need to be viral. If any subroutine needs to await something without blocking everything else, every parent also needs to return a promise and be aware of it. However, I'm not entirely sure whether they're the better solution. Currently every `yield` in a coroutine in Amp is an interruption point. Between two `yield`s, everything is synchronous. If every function call can now potentially await things, other things might happen (unexpectedly) concurrently. This would need special care for some things. The current main pain point I see for libraries like Amp is not the problem of viral promise-APIs. It's rather the need for the boilerplate of `Amp\call(function () use ($arg1, $arg2, ...) { ... }` for every coroutine to return a promise instead of a generator, because generators are an implementation detail, see e.g. https://github.com/amphp/artax/blob/63f044eea7a67ac5f703232f6710846125a73fe3/lib/HttpTunneler.php#L21 A potential way around that (might be a stupid idea I just had): Allow defining "wrappers" per file, that auto-wrap marked functions. ```php
  100479
September 9, 2017 06:07 i@lvht.net (Haitao Lv)
> On 1 Sep 2017, at 22:25, Niklas Keller <me@kelunik.com> wrote: > > A potential way around that (might be a stupid idea I just had): Allow > defining "wrappers" per file, that auto-wrap marked functions.
Amp need these wrapper functions because we cannot yield a generator from its sub function call. So if we introduce fiber, these wrapper is needless.
> > I don't see an event loop or promises being built-in soon, but something > like that could provide `async` with userland magic. Not the best thing, > but maybe better than preprocessing, don't know.
Fiber is asymmetric coroutine, and the PHP kernel has no duty to schedule them. So a built-in event loop is no needed.
  100486
September 9, 2017 10:31 me@kelunik.com (Niklas Keller)
2017-09-09 8:07 GMT+02:00 Haitao Lv <i@lvht.net>:

> > > On 1 Sep 2017, at 22:25, Niklas Keller <me@kelunik.com> wrote: > > > > A potential way around that (might be a stupid idea I just had): Allow > > defining "wrappers" per file, that auto-wrap marked functions. > > Amp need these wrapper functions because we cannot yield a generator from > its sub function call. So if we introduce fiber, these wrapper is needless.
We could easily support yielding generators and auto-wrapping them in the coroutine implementation, but not every generator is a coroutine. And generators are an implementation detail, it doesn't make sense to force everything to be a generator.
> > > > > I don't see an event loop or promises being built-in soon, but something > > like that could provide `async` with userland magic. Not the best thing, > > but maybe better than preprocessing, don't know. > > Fiber is asymmetric coroutine, and the PHP kernel has no duty to schedule > them. So a built-in event loop is no needed.
I'm actually thinking about bringing our promise + coroutine implementation to php-src. `await` could be used to automatically convert generators into coroutines, which implement promise. It would be mostly syntactic sugar for the current Amp implementation, because all these wrappers are gone, but the implementation details of something being a coroutine stays hidden behind a promise interface. Regards, Niklas
  100353
September 2, 2017 12:19 rowan.collins@gmail.com (Rowan Collins)
On 1 September 2017 14:02:29 BST, Haitao Lv <i@lvht.net> wrote:
> >Fiber is a lightweight thread. Please see >https://en.wikipedia.org/wiki/Fiber_(computer_science) >And ruby support Fiber. Please see >https://ruby-doc.org/core-2.4.1/Fiber.html
Ah, thanks, that makes more sense now. I note that the examples there all implement it not as a keyword, but as a library function, which maybe makes more sense: whereas "yield" turns a function declaration into a generator declaration, "Fiber\yield", as we might call it, is just a function call which can happen anywhere and manipulates some global state. The choice of "await" also feels odd: you're not awaiting the thing on the right-hand side of the keyword, you're sending it somewhere and awaiting something else.
> You can see the await as a resumable return. The value after it >will be returned and the function >will be paused.
Again, this explanation doesn't make sense if you can call foo() without any Fiber code. If there isn't an active Fiber to pause, the "await" line can't "return" anything; it will presumably throw an error of some sort. function foo($x) { $y = Fiber\yield($x); return $x + $y; } echo foo(1); The 1 isn't "returned" to the echo, it's passed off to a global function for further processing. This makes much more sense to me than implying that foo() is no longer a normal function.
>> Your ->resume() doesn't seem to do anything a normal generator can't, >except that the yield is nested deeper. > >The await is a nested deeper yield!
Yes, I understand that, but that seems to be completely unrelated to the API design of having resume() act as start, continue, and final result all rolled into one. As I say, I didn't entirely follow the reasoning for allowing return in generators, but if Fibers are used for the same purpose, I would expect the same concerns to arise. However, I think the comparison to generators may be more distracting than useful, as they seem to be very different solutions to the same or related problems. Regards, -- Rowan Collins [IMSoP]
  100480
September 9, 2017 06:27 i@lvht.net (Haitao Lv)
> On 2 Sep 2017, at 20:19, Rowan Collins collins@gmail.com> wrote: > > On 1 September 2017 14:02:29 BST, Haitao Lv <i@lvht.net> wrote: >> >> Fiber is a lightweight thread. Please see >> https://en.wikipedia.org/wiki/Fiber_(computer_science) >> And ruby support Fiber. Please see >> https://ruby-doc.org/core-2.4.1/Fiber.html > > Ah, thanks, that makes more sense now. > > I note that the examples there all implement it not as a keyword, but as a library function, which maybe makes more sense: whereas "yield" turns a function declaration into a generator declaration, "Fiber\yield", as we might call it, is just a function call which can happen anywhere and manipulates some global state.
As the Zend Engine does not offer API to pause execution, I cannot implement the await without introducing a new keyword (just like yield).
> The choice of "await" also feels odd: you're not awaiting the thing on the right-hand side of the keyword, you're sending it somewhere and awaiting something else.
await is chosen because we cannot reuse the `yield` keyword. Maybe we can choose a more proper keyword.
>> You can see the await as a resumable return. The value after it >> will be returned and the function >> will be paused. > > Again, this explanation doesn't make sense if you can call foo() without any Fiber code. If there isn't an active Fiber to pause, the "await" line can't "return" anything; it will presumably throw an error of some sort. > > function foo($x) { > $y = Fiber\yield($x); > return $x + $y; > } > echo foo(1); > > The 1 isn't "returned" to the echo, it's passed off to a global function for further processing. This makes much more sense to me than implying that foo() is no longer a normal function.
await is not yield. It need a separate vm stack to work. So call the foo function without a Fiber will throw a runtime error exception.
> >>> Your ->resume() doesn't seem to do anything a normal generator can't, >> except that the yield is nested deeper. >> >> The await is a nested deeper yield! > > Yes, I understand that, but that seems to be completely unrelated to the API design of having resume() act as start, continue, and final result all rolled into one. As I say, I didn't entirely follow the reasoning for allowing return in generators, but if Fibers are used for the same purpose, I would expect the same concerns to arise. > > However, I think the comparison to generators may be more distracting than useful, as they seem to be very different solutions to the same or related problems.
If fiber were implemented, the generator would be a special fiber.
  100481
September 9, 2017 06:29 hufeng1987@gmail.com (zixu mo)
the best corotine . must be golang go and chan. very easy to use. non blocking. 




On Sat, Sep 9, 2017 at 2:28 PM +0800, "Haitao Lv" <i@lvht.net> wrote:











> On 2 Sep 2017, at 20:19, Rowan Collins wrote: > > On 1 September 2017 14:02:29 BST, Haitao Lv wrote: >> >> Fiber is a lightweight thread. Please see >> https://en.wikipedia.org/wiki/Fiber_(computer_science) >> And ruby support Fiber. Please see >> https://ruby-doc.org/core-2.4.1/Fiber.html > > Ah, thanks, that makes more sense now. > > I note that the examples there all implement it not as a keyword, but as a library function, which maybe makes more sense: whereas "yield" turns a function declaration into a generator declaration, "Fiber\yield", as we might call it, is just a function call which can happen anywhere and manipulates some global state.
As the Zend Engine does not offer API to pause execution, I cannot implement the await without introducing a new keyword (just like yield).
> The choice of "await" also feels odd: you're not awaiting the thing on the right-hand side of the keyword, you're sending it somewhere and awaiting something else.
await is chosen because we cannot reuse the `yield` keyword. Maybe we can choose a more proper keyword.
>> You can see the await as a resumable return. The value after it >> will be returned and the function >> will be paused. > > Again, this explanation doesn't make sense if you can call foo() without any Fiber code. If there isn't an active Fiber to pause, the "await" line can't "return" anything; it will presumably throw an error of some sort. > > function foo($x) { > $y = Fiber\yield($x); > return $x + $y; > } > echo foo(1); > > The 1 isn't "returned" to the echo, it's passed off to a global function for further processing. This makes much more sense to me than implying that foo() is no longer a normal function.
await is not yield. It need a separate vm stack to work. So call the foo function without a Fiber will throw a runtime error exception.
> >>> Your ->resume() doesn't seem to do anything a normal generator can't, >> except that the yield is nested deeper. >> >> The await is a nested deeper yield! > > Yes, I understand that, but that seems to be completely unrelated to the API design of having resume() act as start, continue, and final result all rolled into one. As I say, I didn't entirely follow the reasoning for allowing return in generators, but if Fibers are used for the same purpose, I would expect the same concerns to arise. > > However, I think the comparison to generators may be more distracting than useful, as they seem to be very different solutions to the same or related problems.
If fiber were implemented, the generator would be a special fiber. -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
  100485
September 9, 2017 10:12 i@lvht.net (Haitao Lv)
> On 9 Sep 2017, at 14:27, Haitao Lv <i@lvht.net> wrote: > >> I note that the examples there all implement it not as a keyword, but as a library function, which maybe makes more sense: whereas "yield" turns a function declaration into a generator declaration, "Fiber\yield", as we might call it, is just a function call which can happen anywhere and manipulates some global state. > > As the Zend Engine does not offer API to pause execution, I cannot implement the await without > introducing a new keyword (just like yield). > >> The choice of "await" also feels odd: you're not awaiting the thing on the right-hand side of the keyword, you're sending it somewhere and awaiting something else. > > await is chosen because we cannot reuse the `yield` keyword. Maybe we can choose a more proper keyword.
Sorry for misunderstanding. The keyword await has been dropped and Fiber::yield implemented. Please see https://github.com/php/php-src/pull/2733