Comments on: lxml.html http://blog.ianbicking.org/2007/09/24/lxmlhtml/ Mon, 01 Dec 2008 21:34:22 +0000 http://wordpress.org/?v=2.2.3 By: Ian Bicking http://blog.ianbicking.org/2007/09/24/lxmlhtml/#comment-1126 Ian Bicking Wed, 26 Sep 2007 00:18:17 +0000 http://blog.ianbicking.org/2007/09/24/lxmlhtml/#comment-1126 Re: testing -- there's a property to access the lxml document in WebTest (an extraction of paste.fixture): http://pythonpaste.org/webtest/#parsing-the-body Re: testing — there’s a property to access the lxml document in WebTest (an extraction of paste.fixture): http://pythonpaste.org/webtest/#parsing-the-body

]]> By: Kumar McMillan http://blog.ianbicking.org/2007/09/24/lxmlhtml/#comment-1122 Kumar McMillan Tue, 25 Sep 2007 18:58:05 +0000 http://blog.ianbicking.org/2007/09/24/lxmlhtml/#comment-1122 oh, cool, I see it's already in lxml's trunk :) oh, cool, I see it’s already in lxml’s trunk :)

]]> By: Kumar McMillan http://blog.ianbicking.org/2007/09/24/lxmlhtml/#comment-1121 Kumar McMillan Tue, 25 Sep 2007 18:52:47 +0000 http://blog.ianbicking.org/2007/09/24/lxmlhtml/#comment-1121 Hi Ian, I just wanted to say thanks for hacking on this and I look forward to seeing it merged into lxml's trunk. I've deployed it from the branch and have been using it for a side project quite successfully. Ironically, it's a screen scraper. I tried using templatemaker on the scrapee but its template is obfuscated (my theory anyway) so it [breaks templatemaker pretty bad](http://code.google.com/p/templatemaker/issues/detail?id=4). Anyway, xpath seems cleaner and more maintainable. Plus, it is very useful to have HTMLElement.text_content() so that the template can be analyzed in a more contextual way. I've also been starting to work with paste (via pylons) a little more than just experimentation and it might be nice to use xpath for assertions on paste.fixture's response. However, I haven't needed more than the provided mustcontain() method yet so maybe it would be overkill, just a thought. Xpath would certainly be a clean, maintainable way to "validate" a template. That is, answer : Do all my pages implement the common header/footer/nav-bar layout for the site? I'm not sure if it is reasonable to go to such lengths in ones tests, this again is just a thought. Hi Ian, I just wanted to say thanks for hacking on this and I look forward to seeing it merged into lxml’s trunk. I’ve deployed it from the branch and have been using it for a side project quite successfully. Ironically, it’s a screen scraper. I tried using templatemaker on the scrapee but its template is obfuscated (my theory anyway) so it breaks templatemaker pretty bad. Anyway, xpath seems cleaner and more maintainable. Plus, it is very useful to have HTMLElement.text_content() so that the template can be analyzed in a more contextual way.

I’ve also been starting to work with paste (via pylons) a little more than just experimentation and it might be nice to use xpath for assertions on paste.fixture’s response. However, I haven’t needed more than the provided mustcontain() method yet so maybe it would be overkill, just a thought.

Xpath would certainly be a clean, maintainable way to “validate” a template. That is, answer : Do all my pages implement the common header/footer/nav-bar layout for the site? I’m not sure if it is reasonable to go to such lengths in ones tests, this again is just a thought.

]]> By: jgraham http://blog.ianbicking.org/2007/09/24/lxmlhtml/#comment-1101 jgraham Mon, 24 Sep 2007 22:18:32 +0000 http://blog.ianbicking.org/2007/09/24/lxmlhtml/#comment-1101 > It would be nice to have something similar with html5lib. It appears lxml has grown a HTML 5-incompatible idea desire to enforce "valid" tag names. In particular it seems things like lxml.etree.Element("foo?bar") now throw a value errors even though HTML 5 parsers are expected to be able to create elements called "foo?bar". Not only does this mean that not all HTML 5 trees can be represented in lxml.etree, but it means that our hack of using a HTML 5-illegal element name to represent the notional document root doesn't work any more. If this problem can be overcome, html5lib already supports generating lxml trees, so it would be easy to wrap it in syntax like lxml.html.html5lib.parse(). > It would be nice to have something similar with html5lib.

It appears lxml has grown a HTML 5-incompatible idea desire to enforce “valid” tag names. In particular it seems things like lxml.etree.Element(”foo?bar”) now throw a value errors even though HTML 5 parsers are expected to be able to create elements called “foo?bar”. Not only does this mean that not all HTML 5 trees can be represented in lxml.etree, but it means that our hack of using a HTML 5-illegal element name to represent the notional document root doesn’t work any more. If this problem can be overcome, html5lib already supports generating lxml trees, so it would be easy to wrap it in syntax like lxml.html.html5lib.parse().

]]> By: Martijn Faassen http://blog.ianbicking.org/2007/09/24/lxmlhtml/#comment-1099 Martijn Faassen Mon, 24 Sep 2007 19:44:23 +0000 http://blog.ianbicking.org/2007/09/24/lxmlhtml/#comment-1099 I'm happy to see all this great work appear in lxml! lxml is still my baby too and I'm very proud. :) Thanks Ian for these great contributions and thanks Stefan for being such an excellent lead developer for lxml! I’m happy to see all this great work appear in lxml! lxml is still my baby too and I’m very proud. :) Thanks Ian for these great contributions and thanks Stefan for being such an excellent lead developer for lxml!

]]> By: Ian Bicking http://blog.ianbicking.org/2007/09/24/lxmlhtml/#comment-1094 Ian Bicking Mon, 24 Sep 2007 17:26:54 +0000 http://blog.ianbicking.org/2007/09/24/lxmlhtml/#comment-1094 Fredrik: I'm not sure. Obviously the first part is parsing HTML, but there are BeautifulSoup parsers and native serialization in ET 1.3. I use `el.getparent()` sometimes, and though not a tremendous amount I've found it's hard to refactor when you don't have that pointer to the parent, as in ElementTree. Also, many of these use XPath very heavily. You could probably rewrite several of them to be simpler (ET-compatible) expressions plus a simple list comprehension. A couple might be more feasible than the others: the differ, maybe formfill, and definitely the doctest comparison stuff would be okay (since it is based on code that was written for ElementTree originally). Fredrik: I’m not sure. Obviously the first part is parsing HTML, but there are BeautifulSoup parsers and native serialization in ET 1.3. I use el.getparent() sometimes, and though not a tremendous amount I’ve found it’s hard to refactor when you don’t have that pointer to the parent, as in ElementTree. Also, many of these use XPath very heavily. You could probably rewrite several of them to be simpler (ET-compatible) expressions plus a simple list comprehension.

A couple might be more feasible than the others: the differ, maybe formfill, and definitely the doctest comparison stuff would be okay (since it is based on code that was written for ElementTree originally).

]]> By: Fredrik http://blog.ianbicking.org/2007/09/24/lxmlhtml/#comment-1093 Fredrik Mon, 24 Sep 2007 16:13:16 +0000 http://blog.ianbicking.org/2007/09/24/lxmlhtml/#comment-1093 So how hard would it be to make the relevant portions of this work also for xml.etree ? So how hard would it be to make the relevant portions of this work also for xml.etree ?

]]>