[VOTE] Object-based token_get_all() alternative

  108885
March 6, 2020 09:33 nikita.ppv@gmail.com (Nikita Popov)
Hi internals,

I've opened voting on https://wiki.php.net/rfc/token_as_object, which adds
a new object-based interface to the tokenizer. Voting closes on 2020-03-20.

Regards,
Nikita
  108940
March 10, 2020 11:36 come.chilliet@fusiondirectory.org (=?ISO-8859-1?Q?C=F4me?= Chilliet)
Hello,

I know I’m late, but shouldn't these tokens have a __toString() method? 
It’s not listed in rejected features.

-- 
Côme Chilliet
FusionDirectory - https://www.fusiondirectory.org
  108952
March 10, 2020 14:53 nikita.ppv@gmail.com (Nikita Popov)
On Tue, Mar 10, 2020 at 12:36 PM Côme Chilliet <
come.chilliet@fusiondirectory.org> wrote:

> Hello, > > I know I’m late, but shouldn't these tokens have a __toString() method? > It’s not listed in rejected features. >
Would __toString() return $token->text? I'm not strictly opposed to this, but I'd be hard pressed to think of a situation where I would prefer writing (string) $token over $token->text. Generally, the less magic the better :) Nikita
  108956
March 10, 2020 16:18 come.chilliet@fusiondirectory.org (=?ISO-8859-1?Q?C=F4me?= Chilliet)
Le mardi 10 mars 2020, 15:53:41 CET Nikita Popov a écrit :
> Would __toString() return $token->text? I'm not strictly opposed to this, > but I'd be hard pressed to think of a situation where I would prefer > writing (string) $token over $token->text. Generally, the less magic the > better :)
My idea was that since tokens are parsed from a string it should be easy to write a string back with them. I do not have an obvious usecase but the idea is doing stuff like if ($token.is('function')) { echo "public $token"; } else { echo $token; } Am I right to assume that with __toString the same as ->text we’d have implode('', PhpToken::getAll($code)) == $code ? This would allow: $tokens = PhpToken::getAll($code); // Modifications on $tokens $code = implode('', $tokens); -- Côme Chilliet FusionDirectory - https://www.fusiondirectory.org
  108972
March 11, 2020 09:57 nikita.ppv@gmail.com (Nikita Popov)
On Tue, Mar 10, 2020 at 5:19 PM Côme Chilliet <
come.chilliet@fusiondirectory.org> wrote:

> Le mardi 10 mars 2020, 15:53:41 CET Nikita Popov a écrit : > > Would __toString() return $token->text? I'm not strictly opposed to this, > > but I'd be hard pressed to think of a situation where I would prefer > > writing (string) $token over $token->text. Generally, the less magic the > > better :) > > My idea was that since tokens are parsed from a string it should be easy > to write a string back with them. > I do not have an obvious usecase but the idea is doing stuff like > > if ($token.is('function')) { > echo "public $token"; > } else { > echo $token; > } > > Am I right to assume that with __toString the same as ->text we’d have > implode('', PhpToken::getAll($code)) == $code ? > > This would allow: > $tokens = PhpToken::getAll($code); > // Modifications on $tokens > $code = implode('', $tokens); >
Yes, that would work. It's not much harder without __toString though: $code = implode(array_column($tokens, 'text')); Does anyone else have thoughts on whether PhpToken should implement __toString() or not? Nikita
  108982
March 11, 2020 15:59 larry@garfieldtech.com ("Larry Garfield")
On Wed, Mar 11, 2020, at 4:57 AM, Nikita Popov wrote:
> On Tue, Mar 10, 2020 at 5:19 PM Côme Chilliet < > come.chilliet@fusiondirectory.org> wrote: > > > Le mardi 10 mars 2020, 15:53:41 CET Nikita Popov a écrit : > > > Would __toString() return $token->text? I'm not strictly opposed to this, > > > but I'd be hard pressed to think of a situation where I would prefer > > > writing (string) $token over $token->text. Generally, the less magic the > > > better :) > > > > My idea was that since tokens are parsed from a string it should be easy > > to write a string back with them. > > I do not have an obvious usecase but the idea is doing stuff like > > > > if ($token.is('function')) { > > echo "public $token"; > > } else { > > echo $token; > > } > > > > Am I right to assume that with __toString the same as ->text we’d have > > implode('', PhpToken::getAll($code)) == $code ? > > > > This would allow: > > $tokens = PhpToken::getAll($code); > > // Modifications on $tokens > > $code = implode('', $tokens); > > > > Yes, that would work. It's not much harder without __toString though: > > $code = implode(array_column($tokens, 'text')); > > Does anyone else have thoughts on whether PhpToken should implement > __toString() or not? > > Nikita
I forget who it was but someone on the list a while back offered a heuristic of "__toString should be used iff there is one and only one logical string representation of a value, because the value is conceptually just a string with extra metadata on it". That sounds like a really solid heuristic to me. In this case, I think yes, ->text is the only reasonable thing to produce for __toString(). Or perhaps this is a good use case for "you can use a subclass, do that" if there's any good reason to not have it just be ->text. --Larry Garfield
  108994
March 12, 2020 11:07 come.chilliet@fusiondirectory.org (=?ISO-8859-1?Q?C=F4me?= Chilliet)
Le mercredi 11 mars 2020, 10:59:19 CET Larry Garfield a écrit :
> I forget who it was but someone on the list a while back offered a heuristic of "__toString should be used iff there is one and only one logical string representation of a value, because the value is conceptually just a string with extra metadata on it". > > That sounds like a really solid heuristic to me. In this case, I think yes, ->text is the only reasonable thing to produce for __toString().
Yes that was my reasoning, a token is essentially a substring, so I was expecting __toString to reflect this and allow me to use it as a string. -- Côme Chilliet FusionDirectory - https://www.fusiondirectory.org
  109229
March 23, 2020 14:07 nikita.ppv@gmail.com (Nikita Popov)
On Fri, Mar 6, 2020 at 10:33 AM Nikita Popov ppv@gmail.com> wrote:

> Hi internals, > > I've opened voting on https://wiki.php.net/rfc/token_as_object, which > adds a new object-based interface to the tokenizer. Voting closes on > 2020-03-20. >
This RFC has been accepted unanimously, with 47 votes in favor. Based on the additional feedback in this thread, I'll also add a __toString() method that returns the value of $token->text, unless anyone has objects to that... Regards, Nikita
  109232
March 23, 2020 16:03 larry@garfieldtech.com ("Larry Garfield")
On Mon, Mar 23, 2020, at 9:07 AM, Nikita Popov wrote:
> On Fri, Mar 6, 2020 at 10:33 AM Nikita Popov ppv@gmail.com> wrote: > > > Hi internals, > > > > I've opened voting on https://wiki.php.net/rfc/token_as_object, which > > adds a new object-based interface to the tokenizer. Voting closes on > > 2020-03-20. > > > > This RFC has been accepted unanimously, with 47 votes in favor. > > Based on the additional feedback in this thread, I'll also add a > __toString() method that returns the value of $token->text, unless anyone > has objects to that... > > Regards, > Nikita
As long as __toString() is overridable in subclasses, that seems fine to me. --Larry Garfield
  109285
March 25, 2020 10:50 lisachenko.it@gmail.com (Alexander Lisachenko)
Hello, internals team!

I would like to propose to rename the PhpToken::getAll() method into the
PhpToken::tokenize(string $source) or PhpToken::lex(string $source) to use
a more meaningful verb word.

After renaming, it will sound more natural: $tokenStream =
PhpToken::tokenize($sourceCode)

Best regards,
Alexander


пн, 23 мар. 2020 г. в 19:04, Larry Garfield <larry@garfieldtech.com>:

> On Mon, Mar 23, 2020, at 9:07 AM, Nikita Popov wrote: > > On Fri, Mar 6, 2020 at 10:33 AM Nikita Popov ppv@gmail.com> > wrote: > > > > > Hi internals, > > > > > > I've opened voting on https://wiki.php.net/rfc/token_as_object, which > > > adds a new object-based interface to the tokenizer. Voting closes on > > > 2020-03-20. > > > > > > > This RFC has been accepted unanimously, with 47 votes in favor. > > > > Based on the additional feedback in this thread, I'll also add a > > __toString() method that returns the value of $token->text, unless anyone > > has objects to that... > > > > Regards, > > Nikita > > As long as __toString() is overridable in subclasses, that seems fine to > me. > > --Larry Garfield > > -- > PHP Internals - PHP Runtime Development Mailing List > To unsubscribe, visit: http://www.php.net/unsub.php > >