Getting fd from stream

  99793
July 6, 2017 09:12 php@list.imperialat.at (Martijn van Duren)
Hello internals@,

I have an (exotic) case where I need to be able to get the
filedescriptor from a previously opened stream (via stream_socket_pair)
to hand over to the child process, which needs to reexecute.  

I saw that the php:// wrapper supports php://fd/X syntax, so I reckon
there should be a way to extract the information as well from a stream  
as well. Unfortunately I don't seem to be able to find it.

Can someone point me in the right direction?

Sincerely,

Martijn van Duren
  99798
July 6, 2017 14:53 kalle@php.net (Kalle Sommer Nielsen)
Hi  Martijn

2017-07-06 11:12 GMT+02:00 Martijn van Duren <php@list.imperialat.at>:
> Hello internals@, > > I have an (exotic) case where I need to be able to get the > filedescriptor from a previously opened stream (via stream_socket_pair) > to hand over to the child process, which needs to reexecute. > > I saw that the php:// wrapper supports php://fd/X syntax, so I reckon > there should be a way to extract the information as well from a stream > as well. Unfortunately I don't seem to be able to find it. > > Can someone point me in the right direction?
It seems like this is not exposed at all by PHP. I have CC'ed Sara as one of the old school streams persons, maybe she has some insights why it is not exposed or if it is worth exposing fileno() in 7.2+ -- regards, Kalle Sommer Nielsen kalle@php.net
  99800
July 6, 2017 15:20 pollita@php.net (Sara Golemon)
On Thu, Jul 6, 2017 at 10:53 AM, Kalle Sommer Nielsen <kalle@php.net> wrote:
> 2017-07-06 11:12 GMT+02:00 Martijn van Duren <php@list.imperialat.at>: >> I have an (exotic) case where I need to be able to get the >> filedescriptor from a previously opened stream (via stream_socket_pair) >> to hand over to the child process, which needs to reexecute. >> > "hand over to a child process"
Child as in pcntl_fork()? Or a subprocess via exec() et. al.? For the former, the variable should (I think) carry over. For the latter, the stream number is going to be a meaningless value to the subprocess. ((e.g. stdin (fd=0) for the parent is a different source than stdin for the child, despite being the same number)). I'm not sure of your exact use case, but you might be able to use proc_open to spawn the child and supply the socket in the descriptor spec which will (I think, iirc) bind the internal socket to a new pipe between the parent and child. Though it's possible you'll need to maintain a proxy loop in the parent. 40% of the above is guesswork and conjecture, but maybe it'll spawn the right question. -Sara
  99801
July 6, 2017 18:01 me@kelunik.com (Niklas Keller)
> > On Thu, Jul 6, 2017 at 10:53 AM, Kalle Sommer Nielsen <kalle@php.net> > wrote: > > 2017-07-06 11:12 GMT+02:00 Martijn van Duren <php@list.imperialat.at>: > >> I have an (exotic) case where I need to be able to get the > >> filedescriptor from a previously opened stream (via stream_socket_pair) > >> to hand over to the child process, which needs to reexecute. > >> > > > "hand over to a child process" > Child as in pcntl_fork()? Or a subprocess via exec() et. al.? > > For the former, the variable should (I think) carry over. > For the latter, the stream number is going to be a meaningless value > to the subprocess. ((e.g. stdin (fd=0) for the parent is a different > source than stdin for the child, despite being the same number)). > > I'm not sure of your exact use case, but you might be able to use > proc_open to spawn the child and supply the socket in the descriptor > spec which will (I think, iirc) bind the internal socket to a new pipe > between the parent and child. Though it's possible you'll need to > maintain a proxy loop in the parent. > > 40% of the above is guesswork and conjecture, but maybe it'll spawn > the right question.
There's a (I think undocumented) feature that allow transferring sockets to other processes with the socket extension. https://github.com/amphp/aerys/commit/40fae01e4f5f82570a0bc3cb8ebbf1fee36dbb0b Bob implemented that, so maybe he can help you. But what's your exact use case? Regards, Niklas
  99808
July 7, 2017 14:04 php@list.imperialat.at (Martijn van Duren)
On 07/06/17 20:01, Niklas Keller wrote:
>> >> On Thu, Jul 6, 2017 at 10:53 AM, Kalle Sommer Nielsen <kalle@php.net> >> wrote: >>> 2017-07-06 11:12 GMT+02:00 Martijn van Duren <php@list.imperialat.at>: >>>> I have an (exotic) case where I need to be able to get the >>>> filedescriptor from a previously opened stream (via stream_socket_pair) >>>> to hand over to the child process, which needs to reexecute. >>>> >>> >> "hand over to a child process" >> Child as in pcntl_fork()? Or a subprocess via exec() et. al.?
The latter.
>> >> For the former, the variable should (I think) carry over. >> For the latter, the stream number is going to be a meaningless value >> to the subprocess. ((e.g. stdin (fd=0) for the parent is a different >> source than stdin for the child, despite being the same number)).
Not entirely, when I use stream_socket_pair I get 2 filedescriptors and two resources with possible 2 different IDs. E.g. fd 5 and 6 while the resources have id 8 and 9. When I do the fork I close one end in the parent and one end in the child. If I exec into another program I want to be able to say: we can communicate over this fd, so that the child program can hook into that specific file descriptor.
>> >> I'm not sure of your exact use case, but you might be able to use >> proc_open to spawn the child and supply the socket in the descriptor >> spec which will (I think, iirc) bind the internal socket to a new pipe >> between the parent and child. Though it's possible you'll need to >> maintain a proxy loop in the parent.
My use case is to set up a cluster of processes each with their own task and permission sets. These processes need to be able to communicate with each other over socket pairs, not just between the parent and the child, but also between children. Although the proc_open does solve some problems, it doesn't solve them all. E.g. I can't create socket pairs between children via this options, and socketpairs opened to the proc_open would be left dangling. Hence I want to set up all my socket pairs, pcntl_fork the child process, close the socket pair ends that are not owned by the child and execute the new process and specify the fds via commandline arguments. For processes that execute a new PHP process I would also prefer to wrap an existing fd in a stream (similar to fdopen in C), but I can live with the dup-ing of the file-descriptor in fopen(php://fd/N, X). So the end result would (somewhat) look like: +------+ |master| +------+ 3 7 / \ / \ 4 8 +------+ +------+ |child1|5-----------6|child2| +------+ +------+ instead of: 4568 +------+ |master| +------+ 3 7 / \ / \ 4 8 +------+ +------+ |child1|5-----------6|child2| +------+ +------+ 3678 3457
>> >> 40% of the above is guesswork and conjecture, but maybe it'll spawn >> the right question. > > > There's a (I think undocumented) feature that allow transferring sockets to > other processes with the socket extension. > > https://github.com/amphp/aerys/commit/40fae01e4f5f82570a0bc3cb8ebbf1fee36dbb0b
That's a big help on another project I have, so thank you for that hint. I've actually been looking for that option for quite some time.
> > Bob implemented that, so maybe he can help you. > > But what's your exact use case? > > Regards, Niklas >
  99809
July 7, 2017 14:38 pollita@php.net (Sara Golemon)
On Fri, Jul 7, 2017 at 10:04 AM, Martijn van Duren
<php@list.imperialat.at> wrote:
> My use case is to set up a cluster of processes each with their own task > and permission sets. These processes need to be able to communicate with > each other over socket pairs, not just between the parent and the child, > but also between children. > Sounds like you're describing unix domain sockets (or NamedPipes if
you're on Windows). -Sara
  99810
July 7, 2017 15:04 php@list.imperialat.at (Martijn van Duren)
On 07/07/17 16:38, Sara Golemon wrote:
> On Fri, Jul 7, 2017 at 10:04 AM, Martijn van Duren > <php@list.imperialat.at> wrote: >> My use case is to set up a cluster of processes each with their own task >> and permission sets. These processes need to be able to communicate with >> each other over socket pairs, not just between the parent and the child, >> but also between children. >> > Sounds like you're describing unix domain sockets (or NamedPipes if > you're on Windows). > > -Sara > Yes, although tcp/ip are also possible with stream_socket_pair.
The point is, regardless of unix - or tcp sockets, they still have an underlying file descriptor, which I need to specify to the child, so that they can start using it. Ergo, I need to find a way to export this from the stream into PHP. To put it in code: This returns 4 (the object id), instead of 3 (the fd). If I could get the fd, I could turn this into something like: Where whatever can be anything from a PHP-script to a C-compiled binary.
  99811
July 7, 2017 16:03 pollita@php.net (Sara Golemon)
On Fri, Jul 7, 2017 at 11:04 AM, Martijn van Duren
<php@list.imperialat.at> wrote:
> On 07/07/17 16:38, Sara Golemon wrote: > Yes, although tcp/ip are also possible with stream_socket_pair. > nit; socket_pair sockets are not AF_INET, they're just sockets.
> The point is, regardless of unix - or tcp sockets, they still have an > underlying file descriptor, which I need to specify to the child, so > that they can start using it. Ergo, I need to find a way to export this > from the stream into PHP. > You misunderstand, I'm not saying to create a unix socket and try to
pass that open pair to the child. I'm saying listen on a unix domain socket on the file system and tell the children what the path is. They can all individually then open streams to the parent using the path. Psuedo code: // parent $mysocket = 'unix://tmp/parent.sock'; $server = stream_socket_server($mysocket); spawn_child('child_proc --sock='.escapeshellarg($mysocket)); $child = stream_socket_accept($server); var_dump(fgets($child)); // string (8) "Hi mom!\n" // child $parentssocket = parseCliArg('sock'); $stream = stream_socket_client($parentssocket); fwrite($stream, "Hi mom!\n"); Then the children can each report their own listening sockets to the parent and the parent can provide a lookup for siblings to find each other and allow them to open sockets to each other. Trying to pass a meaningless number to the child that it can't do anything with isn't going to work. -Sara
  99813
July 7, 2017 20:06 php@list.imperialat.at (Martijn van Duren)
On 07/07/17 18:03, Sara Golemon wrote
>> The point is, regardless of unix - or tcp sockets, they still have an >> underlying file descriptor, which I need to specify to the child, so >> that they can start using it. Ergo, I need to find a way to export this >> from the stream into PHP. >> > You misunderstand, I'm not saying to create a unix socket and try to > pass that open pair to the child. I'm saying listen on a unix domain > socket on the file system and tell the children what the path is. > They can all individually then open streams to the parent using the > path.
That would be an extremely ugly solution. You would create overhead for 1, and 2, you leave open the possibility that an arbitrary application (bound to the filesystem permissions) allows to connect to it during the time the socket is listening. See [0] for what can go wrong with a similar solutions.
> > Psuedo code: > // parent > $mysocket = 'unix://tmp/parent.sock'; > $server = stream_socket_server($mysocket); > spawn_child('child_proc --sock='.escapeshellarg($mysocket)); > $child = stream_socket_accept($server); > var_dump(fgets($child)); // string (8) "Hi mom!\n" > > // child > $parentssocket = parseCliArg('sock'); > $stream = stream_socket_client($parentssocket); > fwrite($stream, "Hi mom!\n"); > > > Then the children can each report their own listening sockets to the > parent and the parent can provide a lookup for siblings to find each > other and allow them to open sockets to each other. > > Trying to pass a meaningless number to the child that it can't do > anything with isn't going to work.
That's the whole point. They're not meaningless numbers, they're the actual references that the kernel exposes to the process for file access: $ cd /tmp $ cat test.php $ cat test.c #include #include #include int main(int argc, char *argv[]) { extern char *optarg; int ch; int fd = -1; int readbytes; char buf[sizeof("hello world")]; while ((ch = getopt(argc, argv, "i:")) != -1) { switch(ch) { case 'i': fd = atoi(optarg); break; default: fprintf(stderr, "wrong parameter\n"); exit(1); } } readbytes = read(fd, buf, sizeof(buf)); buf[readbytes] = '\0'; printf("read: %s\n", buf); } $ gcc ./test.c -o test && php ./test.php read: hello world $ cat test.c #include #include #include #include #include #include int main(int argc, char *argv[]) { int sp[2]; char *fd; extern int errno; socketpair(AF_UNIX, SOCK_STREAM, 0, sp); switch(fork()) { case -1: exit(1); case 0: close(sp[1]); asprintf(&fd, "%d", sp[0]); execl("/usr/bin/env", "/usr/bin/env", "php", "/tmp/test2.php", "-i", fd, NULL); default: close(sp[0]); write(sp[1], "hello world", sizeof("hello world")); } } $ cat test2.php $arg) { switch($opt) { case "i": $fd = fopen("php://fd/" . $arg, "r"); break; default: trigger_error("wrong parameter", E_USER_ERROR); } } $buf = fread($fd, 11); echo "read: " . $buf . "\n"; $ gcc ./test2.c -o ./test2 && ./test2 read: hello world Don't mind the crummy C here, it's for illustration purposes only. As you can see, these "meaningless numbers" allow me to connect two applications written in two completely different languages without setting up any external listening system. The problem here is that for test.php I had to hardcode the value 3 because I don't know how to retrieve it from $sp[0]. This only works in a default shell where only stdin, stdout and stderr (fd0, fd1 and fd2) are set. The kernel allocates the next free fd, which happens to have number 3. As soon as I would execute it like: php ./test.php < ./test.c it would fail, because fd3 would be taken by a handle to test.c and $sp[0] would be allocated on 4.
  99816
July 7, 2017 22:14 pollita@php.net (Sara Golemon)
On Fri, Jul 7, 2017 at 4:06 PM, Martijn van Duren
<php@list.imperialat.at> wrote:
> blah blah blah > Ignoring the brokenness of your design, the original ask: Exposing the
underlying FD from a file stream? Sure. No particular objection to it, and the stream::set_option interface has the right mechanism to surface it in a general way. Find someone to RFC it, and I don't see why it won't show up in PHP 7.3. What you're able to do with it is your problem. -Sara
  99818
July 7, 2017 22:47 pollita@php.net (Sara Golemon)
On Fri, Jul 7, 2017 at 6:14 PM, Sara Golemon <pollita@php.net> wrote:
> On Fri, Jul 7, 2017 at 4:06 PM, Martijn van Duren > <php@list.imperialat.at> wrote: >> blah blah blah >> > Ignoring the brokenness of your design, the original ask: Exposing the > underlying FD from a file stream? Sure. No particular objection to > it, and the stream::set_option interface has the right mechanism to > surface it in a general way. Find someone to RFC it, and I don't see > why it won't show up in PHP 7.3. What you're able to do with it is > your problem. > Here's a diff: https://github.com/php/php-src/compare/master...sgolemon:expose.fd
-Sara
  99821
July 8, 2017 06:49 php@list.imperialat.at (Martijn van Duren)
On 07/08/17 00:47, Sara Golemon wrote:
> On Fri, Jul 7, 2017 at 6:14 PM, Sara Golemon <pollita@php.net> wrote: >> On Fri, Jul 7, 2017 at 4:06 PM, Martijn van Duren >> <php@list.imperialat.at> wrote: >>> blah blah blah >>> >> Ignoring the brokenness of your design, the original ask: Exposing the >> underlying FD from a file stream? Sure. No particular objection to >> it, and the stream::set_option interface has the right mechanism to >> surface it in a general way. Find someone to RFC it, and I don't see >> why it won't show up in PHP 7.3. What you're able to do with it is >> your problem. >> > Here's a diff: https://github.com/php/php-src/compare/master...sgolemon:expose.fd > > -Sara > Thank you.
As per https://wiki.php.net/rfc/howto I created the account martijn. I'd like the Karma to turn this patch into an RFC.
  99822
July 8, 2017 10:44 tyra3l@gmail.com (Ferenc Kovacs)
On Sat, Jul 8, 2017 at 8:49 AM, Martijn van Duren <php@list.imperialat.at>
wrote:

> On 07/08/17 00:47, Sara Golemon wrote: > > On Fri, Jul 7, 2017 at 6:14 PM, Sara Golemon <pollita@php.net> wrote: > >> On Fri, Jul 7, 2017 at 4:06 PM, Martijn van Duren > >> <php@list.imperialat.at> wrote: > >>> blah blah blah > >>> > >> Ignoring the brokenness of your design, the original ask: Exposing the > >> underlying FD from a file stream? Sure. No particular objection to > >> it, and the stream::set_option interface has the right mechanism to > >> surface it in a general way. Find someone to RFC it, and I don't see > >> why it won't show up in PHP 7.3. What you're able to do with it is > >> your problem. > >> > > Here's a diff: https://github.com/php/php-src/compare/master...sgolemon: > expose.fd > > > > -Sara > > > Thank you. > > As per https://wiki.php.net/rfc/howto I created the account martijn. > I'd like the Karma to turn this patch into an RFC. > > -- > PHP Internals - PHP Runtime Development Mailing List > To unsubscribe, visit: http://www.php.net/unsub.php > > I've granted you with rfc karma on the wiki.
-- Ferenc Kovács @Tyr43l - http://tyrael.hu
  99812
July 7, 2017 17:49 tyra3l@gmail.com (Ferenc Kovacs)
On Thu, Jul 6, 2017 at 11:12 AM, Martijn van Duren <php@list.imperialat.at>
wrote:

> Hello internals@, > > I have an (exotic) case where I need to be able to get the > filedescriptor from a previously opened stream (via stream_socket_pair) > to hand over to the child process, which needs to reexecute. > > I saw that the php:// wrapper supports php://fd/X syntax, so I reckon > there should be a way to extract the information as well from a stream > as well. Unfortunately I don't seem to be able to find it. > > Can someone point me in the right direction? > > Sincerely, > > Martijn van Duren > > -- > PHP Internals - PHP Runtime Development Mailing List > To unsubscribe, visit: http://www.php.net/unsub.php > > hi,
would you need something like fildes_fileno() from https://github.com/Tyrael/php-fildes ? I never managed to put that up for pecl, not sure if it even compiles anymore with recent php versions -- Ferenc Kovács @Tyr43l - http://tyrael.hu
  99814
July 7, 2017 20:14 php@list.imperialat.at (Martijn van Duren)
Hello Ferenc,

On 07/07/17 19:49, Ferenc Kovacs wrote:
> > > On Thu, Jul 6, 2017 at 11:12 AM, Martijn van Duren <php@list.imperialat.at <mailto:php@list.imperialat.at>> wrote: > > Hello internals@, > > I have an (exotic) case where I need to be able to get the > filedescriptor from a previously opened stream (via stream_socket_pair) > to hand over to the child process, which needs to reexecute. > > I saw that the php:// wrapper supports php://fd/X syntax, so I reckon > there should be a way to extract the information as well from a stream > as well. Unfortunately I don't seem to be able to find it. > > Can someone point me in the right direction? > > Sincerely, > > Martijn van Duren > > -- > PHP Internals - PHP Runtime Development Mailing List > To unsubscribe, visit: http://www.php.net/unsub.php > > > hi, > > would you need something like fildes_fileno() from https://github.com/Tyrael/php-fildes ? > I never managed to put that up for pecl, not sure if it even compiles anymore with recent php versions > Thanks for the hint, but I'd like to refrain from external
libraries as much as I can, especially for something simple as a simple number inside a data structure. I might also achieve something like this by eio from pecl, through e.g. eio_dup2, but it's just unrealistic overkill to ask for a full extension installation for just 1 one trivial piece of functionality.
> > -- > Ferenc Kovács > @Tyr43l - http://tyrael.hu
  99815
July 7, 2017 20:39 tyra3l@gmail.com (Ferenc Kovacs)
On Fri, Jul 7, 2017 at 10:14 PM, Martijn van Duren <php@list.imperialat.at>
wrote:

> Hello Ferenc, > > On 07/07/17 19:49, Ferenc Kovacs wrote: > > > > > > On Thu, Jul 6, 2017 at 11:12 AM, Martijn van Duren < > php@list.imperialat.at <mailto:php@list.imperialat.at>> wrote: > > > > Hello internals@, > > > > I have an (exotic) case where I need to be able to get the > > filedescriptor from a previously opened stream (via > stream_socket_pair) > > to hand over to the child process, which needs to reexecute. > > > > I saw that the php:// wrapper supports php://fd/X syntax, so I reckon > > there should be a way to extract the information as well from a > stream > > as well. Unfortunately I don't seem to be able to find it. > > > > Can someone point me in the right direction? > > > > Sincerely, > > > > Martijn van Duren > > > > -- > > PHP Internals - PHP Runtime Development Mailing List > > To unsubscribe, visit: http://www.php.net/unsub.php > > > > > > hi, > > > > would you need something like fildes_fileno() from > https://github.com/Tyrael/php-fildes ? > > I never managed to put that up for pecl, not sure if it even compiles > anymore with recent php versions > > > Thanks for the hint, but I'd like to refrain from external > libraries as much as I can, especially for something simple as > a simple number inside a data structure. > > I might also achieve something like this by eio from pecl, through > e.g. eio_dup2, but it's just unrealistic overkill to ask for a full > extension installation for just 1 one trivial piece of functionality. > > > > -- > > Ferenc Kovács > > @Tyr43l - http://tyrael.hu >
sure, I just thought that you are asking for a solution and not proposing a new feature/function to the core. -- Ferenc Kovács @Tyr43l - http://tyrael.hu