Re: [PHP-DEV] On fixing DOMNameSpaceNode and DOM NS API Inconsistency Problems

This is only part of a thread. view whole thread
  104774
March 17, 2019 13:33 kontakt@beberlei.de (Benjamin Eberlei)
On Wed, Mar 13, 2019 at 4:36 PM Rob Richards <rrichards@ctindustries.net>
wrote:

> There were specific reasons for it tho I can't recall all the details off > the top of my head. Part of it related to the underlying structures of > attributes and namespace nodes not being the same and cannot be user > interchangeable in the code, unless only inspected at the nodeptr level. > Trying to manipulate namespace nodes as attributes will cause memory > corruption so they need to be handled differently. I don't remember if this > was the only reason or there was more for the special class but while I > agree with basic DOM usage no one should ever need to be dealing directly > with namespace declarations but once you get into more complex situation, > especially that involve xpath, xslt, canonicalization and even > interoperability this starts coming into play and is quite important to > deal with namespace declarations directly. >
That is a very helpful insight, thank you Rob. From the few hours i poked in the code base memory management seems to be tricky abstracing libxml behind this powerful DOM API. Also some of the examples in previous messages are way overly simplistic so
> the conclusions are not 100% accurate. i.e. removeAttributeNS. I am > assuming that its just due to overly aggressive namespace reconciliation > happening and removing the declaration since its no longer in use. With a > nested document which uses the namespace in child nodes I would expect the > namespace declaration to remain, tho possible it may be moved to the first > child using it (been a while since I looked at that code). >
I'll take a look again how that extra attr gets removed. However it doesn't change the fact that semantically DOMElement::removeAttributeNS(" http://www.w3.org/2000/xmlns/", "foo") doesnt work.
> > My recommendation would be to tread lightly here, look at every method any > changes to namespace declaration nodes would touch and make sure they are > all handle properly otherwise you run the risk of introducing memory > corruption issues which is far worse than some inconsistency for supposedly > unused and unneeded functionality ;) > > BTW I have recently started looking at the libxml code base again and > considering getting more involved here again, assuming the toxicity level > has gone done from the past. Not sure this is the first thing I would look > at but feel free to shoot over any questions. >
This sounds awesome :-) I can't comment on toxicity levels of the past as I only follow this list for about 3-4 years now, but it has gone down in those years I feel. It is still a draft but Thomas and I have started working on an RFC and code to update ext/dom to cover the latest standard release: https://wiki.php.net/rfc/dom_living_standard_api - we plan on proposing that soon, maybe you have some feedback.
> Rob > > On 3/12/19 1:50 AM, Pierre Joye wrote: > > Hi, > > Only adding Rob just in case :) > > best, > > On Wed, Mar 13, 2019, 5:54 AM Benjamin Eberlei <kontakt@beberlei.de> > wrote: > >> Hi everyone, >> >> While looking for things to work on in php-src my friend Thomas pointed me >> to a peculiar special case in ext/dom that leads to massive inconsistency >> problems in the API. >> >> There is an undocumented class DOMNameSpaceNode that gets returned from >> DOMElement::getAttributeNode(NS) if you select an attribute that >> represents >> a namespace (xmlns). This special case is intentionally handled in the >> code, contrary to Pythons DOM Extension which doesn't do this and contrary >> to the DOM Specification, which does not have a special DOMNameSpaceNode. >> Its all DOMAttr. >> >> Problematically DOMNameSpaceNode doesn't extend from DOMAttr, you cannot >> pass this class to DOMElement::removeAttributeNode(DOMAttr $attr): >> >> Fatal error: Uncaught TypeError: Argument 1 passed to >> DOMElement::removeAttributeNode() must be an instance of DOMAttr, instance >> of DOMNameSpaceNode given >> >> Code example here: https://3v4l.org/jkC5s >> >> In addition the DOMNameSpaceNode renames all properties compared to >> DOMAttr, clearly violating the interface documented here >> http://php.net/manual/de/domelement.getattributenode.php >> >> Two potential fixes come to mind: >> >> 1. Have DOMNameSpaceNode extend DOMAttr - maybe this was the originally >> intended behavior, becuase the properties are all named differently? >> 2. Have DOMElement::removeAttributeNode accept also a DOMNameSpaceNode >> (3.) Remove the non standard DOMNameSpaceNode class and use a DOMAttr >> instead >> >> I think approach #1 is the right one from a BC perspective and also fixing >> the DOM API to be more consistent with the standard. >> >> But I am also not opposed to #3 in the longer term, looking at Github >> DOMNamespaceNode in the open source world is only used for one apc test, >> in >> PHP Compat and when dumping it in Symfony Var Dumper. I imagine its more >> deeply used in closed source code working deeply with XML. >> >> The second problem that this inconsistency creates is >> DOMElement::removeAttributeNS($namespaceUri, $localName). When you want to >> use it to remove a namespace attribute (xmlns). By default the xmlns >> namespace has the uri http://www.w3.org/2000/xmlns/. Hence removing >> xmlns:$name namespace would expected to be: >> >> DOMElement::removeAttributeNS("http://www.w3.org/2000/xmlns/", "foo") // >> removes xmlns:foo >> >> But thats not how it works, there is special code implemented that >> requires >> you to pick the URI from xmlns:foo value: >> >> xmlns:foo="urn:foo" /> >> >> DOMElement::removeAttributeNS("urn:foo", "foo"); >> >> But if your node has an attribute in this namespace with the same name, it >> gets removed as well: >> >> xmlns:foo="urn:foo" foo:foo="bar" /> >> >> Would incorrectly become this, removing 2 attributes: >> >> >> >> This is a bug that cannot be fixed in a BC way. I have no clue how to fix >> this completly broken behavior in a good way. I propose to break BC here >> and requiring "http://www.w3.org/2000/xmlns/" as namespaceURI from PHP >> 8.0 >> to delete a namespace (xmlns) attribute. >> >> Should I put both issues into an RFC or are these rather bugs that can be >> fixed without an RFC? >> >> greetings >> Benjamin >> > >
  104775
March 17, 2019 13:52 cananian@wikimedia.org ("C. Scott Ananian")
On Sun, Mar 17, 2019, 9:34 AM Benjamin Eberlei <kontakt@beberlei.de> wrote:

> > It is still a draft but Thomas and I have started working on an RFC and > code to update ext/dom to cover the latest standard release: > https://wiki.php.net/rfc/dom_living_standard_api - we plan on proposing > that soon, maybe you have some feedback. >
Updating the DOM extension would be something the Wikimedia Foundation would very much like to see happen. It's more complicated than just adding some new methods, though: there are significantly spec-compliance issues with the current code and performance problems too. We've been porting code from JS to PHP which (in the JS version) used a good spec-compliance DOM implementation, and have been keeping a list of all the crazy bugs and workarounds that have been necessary. Start from the basic fact that the modern DOM requires Node#nodeName to be uppercase for HTML elements, and the current code uses all lowercase. It's hard to see how that could be addressed without breaking backward compat. Here are our notes/discussions/etc: https://phabricator.wikimedia.org/T215000 https://mediawiki.org/wiki/Parsoid/PHP/Help_wanted (and there's more where that came from) --scott PS. My personal feeling at this time is that it would be better to put the core libxml abstractions in an extension, to allow fast xpath and perhaps parse/serialize, but that the actual DOM should be built as a php library on top of that, in order to allow rapid changes (the WHATWG is pretty actively making additions/changes to the we spec these days) which are decoupled from the PHP release cycle.
  104777
March 17, 2019 17:35 kontakt@beberlei.de (Benjamin Eberlei)
On Sun, Mar 17, 2019 at 2:52 PM C. Scott Ananian <cananian@wikimedia.org>
wrote:

> On Sun, Mar 17, 2019, 9:34 AM Benjamin Eberlei <kontakt@beberlei.de> > wrote: > >> >> It is still a draft but Thomas and I have started working on an RFC and >> code to update ext/dom to cover the latest standard release: >> https://wiki.php.net/rfc/dom_living_standard_api - we plan on proposing >> that soon, maybe you have some feedback. >> > > Updating the DOM extension would be something the Wikimedia Foundation > would very much like to see happen. It's more complicated than just adding > some new methods, though: there are significantly spec-compliance issues > with the current code and performance problems too. We've been porting > code from JS to PHP which (in the JS version) used a good spec-compliance > DOM implementation, and have been keeping a list of all the crazy bugs and > workarounds that have been necessary. > > Start from the basic fact that the modern DOM requires Node#nodeName to be > uppercase for HTML elements, and the current code uses all lowercase. It's > hard to see how that could be addressed without breaking backward compat. > > Here are our notes/discussions/etc: > https://phabricator.wikimedia.org/T215000 >
That is a really good resource of thinks we should look at :-) But it is not fully true that this javascript library is DOM spec compliant. It does provide extra features that https://dom.spec.whatwg.org/ doesn't have such as "body", "title", "head" properties on DOMDocument, or innerHtml/outerHtml attributes on elements. I didn't find a spec where these were defined, the html spec also doesn't mention them. The DOM Spec also doesn't impelement HtmlElement, that is from the HTML spec. The way forward without BC break like uppercase nodeName or getAttribute returning NULL and not empty strings in newer implementations could be to allow users to specify which implementation they want the DOMDocument to follow.
> > https://mediawiki.org/wiki/Parsoid/PHP/Help_wanted > (and there's more where that came from) > --scott >
> PS. My personal feeling at this time is that it would be better to put the > core libxml abstractions in an extension, to allow fast xpath and perhaps > parse/serialize, but that the actual DOM should be built as a php library > on top of that, in order to allow rapid changes (the WHATWG is pretty > actively making additions/changes to the we spec these days) which are > decoupled from the PHP release cycle. >
It would still require libxml to be an extension, which would only happen in a newer version, so it is not going to help without requiring that version. I don't think the effort is worth it though, compared to just working on the existing ext/dom.
  104780
March 17, 2019 22:57 rrichards@ctindustries.net (Rob Richards)
On 3/17/19 6:52 AM, C. Scott Ananian wrote:
> On Sun, Mar 17, 2019, 9:34 AM Benjamin Eberlei <kontakt@beberlei.de> wrote: > >> It is still a draft but Thomas and I have started working on an RFC and >> code to update ext/dom to cover the latest standard release: >> https://wiki.php.net/rfc/dom_living_standard_api - we plan on proposing >> that soon, maybe you have some feedback. >> > Updating the DOM extension would be something the Wikimedia Foundation > would very much like to see happen. It's more complicated than just adding > some new methods, though: there are significantly spec-compliance issues > with the current code and performance problems too. We've been porting > code from JS to PHP which (in the JS version) used a good spec-compliance > DOM implementation, and have been keeping a list of all the crazy bugs and > workarounds that have been necessary. > > Start from the basic fact that the modern DOM requires Node#nodeName to be > uppercase for HTML elements, and the current code uses all lowercase. It's > hard to see how that could be addressed without breaking backward compat. > > Here are our notes/discussions/etc: > https://phabricator.wikimedia.org/T215000 > https://mediawiki.org/wiki/Parsoid/PHP/Help_wanted > (and there's more where that came from) > --scott > > PS. My personal feeling at this time is that it would be better to put the > core libxml abstractions in an extension, to allow fast xpath and perhaps > parse/serialize, but that the actual DOM should be built as a php library > on top of that, in order to allow rapid changes (the WHATWG is pretty > actively making additions/changes to the we spec these days) which are > decoupled from the PHP release cycle. >
I'll take a look through the lists you have but you need to remember that the DOM specs were not written for HTML. While HTML might have some more restrictive requirements that piggy back on the DOM specs themselves, they are not the standard for DOM. The output HTML functionality was more convenience methods than part of the core extension, Rob
  104833
March 20, 2019 15:48 cananian@wikimedia.org ("C. Scott Ananian")
On Sun, Mar 17, 2019 at 6:57 PM Rob Richards <rrichards@ctindustries.net>
wrote:

> I'll take a look through the lists you have but you need to remember > that the DOM specs were not written for HTML. While HTML might have some > more restrictive requirements that piggy back on the DOM specs > themselves, they are not the standard for DOM. The output HTML > functionality was more convenience methods than part of the core extension, >
Just dealing with the latest WHATWG DOM specs (not the html-specific part) would still involve grappling with `getAttribute` behavior, the case of nodeName (search for "HTML-uppercased qualified name" in the DOM spec), the numeric node type of the root document, performance of methods on namespaced nodes, etc. It would certainly be a useful step forward, even if it weren't a full HTML DOM. What would be even better is if the hooks were present to allow subclassing the core DOM types so that you could implement HTMLElement in pure PHP as a subclass of DOMElement, etc. For example, right now it is impossible to create a new DOMNodeList from PHP, which makes pure PHP implementations of the missing HTML DOM methods (like querySelectorAll) impossible (for example wikimedia/zest-css has to return an array instead of a DOMNodeList). --scott
  106083
June 27, 2019 21:20 cananian@wikimedia.org ("C. Scott Ananian")
Brief update:

We found a few more PHP DOM compatibility bugs, including one with
DOMNode::normalize(): https://bugs.php.net/bug.php?id=78221

In addition, one of our engineers implemented a pure-PHP DOM
implementation: https://github.com/linehan/DOMOperator

We're not using that in production yet, but it's probably quicker than
trying to make PHP's DOM implementation spec-compliant.
 --scott

On Wed, Mar 20, 2019 at 11:48 AM C. Scott Ananian <cananian@wikimedia.org>
wrote:

> On Sun, Mar 17, 2019 at 6:57 PM Rob Richards <rrichards@ctindustries.net> > wrote: > >> I'll take a look through the lists you have but you need to remember >> that the DOM specs were not written for HTML. While HTML might have some >> more restrictive requirements that piggy back on the DOM specs >> themselves, they are not the standard for DOM. The output HTML >> functionality was more convenience methods than part of the core >> extension, >> > > Just dealing with the latest WHATWG DOM specs (not the html-specific part) > would still involve grappling with `getAttribute` behavior, the case of > nodeName (search for "HTML-uppercased qualified name" in the DOM spec), the > numeric node type of the root document, performance of methods on > namespaced nodes, etc. It would certainly be a useful step forward, even > if it weren't a full HTML DOM. > > What would be even better is if the hooks were present to allow > subclassing the core DOM types so that you could implement HTMLElement in > pure PHP as a subclass of DOMElement, etc. For example, right now it is > impossible to create a new DOMNodeList from PHP, which makes pure PHP > implementations of the missing HTML DOM methods (like querySelectorAll) > impossible (for example wikimedia/zest-css has to return an array instead > of a DOMNodeList). > --scott > >
-- (http://cscott.net)