Re: [PHP-DEV] [RFC] token_get_all() TOKEN_AS_OBJECT mode

This is only part of a thread. view whole thread
  108576
February 14, 2020 13:44 ocramius@gmail.com (Marco Pivetta)
On Fri, Feb 14, 2020 at 2:38 PM Sara Golemon <pollita@php.net> wrote:

> Thanks for picking it up, and I agree with your response to Larry. As nice > as it would be to lazy iterate, the scanner is just in NO shape to tolerate > reentering userspace and potentially reinvoking the scanner before the > first run through is done. >
If this is the current state, maybe it would suffice to declare the return type as `iterable`, and return a strict (fully populated) structure in a first implementation, later to be changed to an iterator, if applicable? Marco Pivetta http://twitter.com/Ocramius http://ocramius.github.com/
  108578
February 14, 2020 14:33 pollita@php.net (Sara Golemon)
On Fri, Feb 14, 2020 at 7:44 AM Marco Pivetta <ocramius@gmail.com> wrote:

> On Fri, Feb 14, 2020 at 2:38 PM Sara Golemon <pollita@php.net> wrote: > >> Thanks for picking it up, and I agree with your response to Larry. As >> nice >> as it would be to lazy iterate, the scanner is just in NO shape to >> tolerate >> reentering userspace and potentially reinvoking the scanner before the >> first run through is done. >> > > If this is the current state, maybe it would suffice to declare the return > type as `iterable`, and return a strict (fully populated) structure in a > first implementation, later to be changed to an iterator, if applicable? > > I think that's an optimistic, but ultimately harmless approach. Arrays satisfy Iterables. Worst case we never improve on that. Best case we get
iterable token generators. +1 -Sara
  108579
February 14, 2020 14:41 nikita.ppv@gmail.com (Nikita Popov)
On Fri, Feb 14, 2020 at 2:44 PM Marco Pivetta <ocramius@gmail.com> wrote:

> On Fri, Feb 14, 2020 at 2:38 PM Sara Golemon <pollita@php.net> wrote: > >> Thanks for picking it up, and I agree with your response to Larry. As >> nice >> as it would be to lazy iterate, the scanner is just in NO shape to >> tolerate >> reentering userspace and potentially reinvoking the scanner before the >> first run through is done. >> > > If this is the current state, maybe it would suffice to declare the return > type as `iterable`, and return a strict (fully populated) structure in a > first implementation, later to be changed to an iterator, if applicable? > > Marco Pivetta >
The fact that it returns an array is an important part of the contract. If an iterator variant is added in the future, it should be added as a separate method. I don't want to be writing $tokens = PhpToken::getAll($code); if (!is_array($tokens)) { $tokens = iterator_to_array($tokens); } to convert this to the right type. And I also don't want my usage to be implicitly "upgraded" to an iterator in the future: The iterator will always be less efficient, and I don't want to be forced to use it if I need an array anyway. Nikita