Comments on: Python HTML Parser Performance http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/ Fri, 03 Sep 2010 01:53:06 +0000 http://wordpress.org/?v=2.9.2 hourly 1 By: Haberler http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/comment-page-1/#comment-159187 Haberler Tue, 13 Apr 2010 12:39:35 +0000 http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/#comment-159187 I try to soup but it was very slow. So that I recomend eveyone to use lxml Regards I try to soup but it was very slow. So that I recomend eveyone to use lxml

Regards

]]> By: Simple Simon http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/comment-page-1/#comment-151954 Simple Simon Mon, 15 Feb 2010 14:36:00 +0000 http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/#comment-151954 DeGroot points out that, while customers and suppliers may have their own ways of calculating dielectric constant, having a standardized IPC methodology will allow for better benchmarking of such attributes as frequency, classes of material types, and standardized test structures that are similar to practical signal traces. DeGroot points out that, while customers and suppliers may have their own ways of calculating dielectric constant, having a standardized IPC methodology will allow for better benchmarking of such attributes as frequency, classes of material types, and standardized test structures that are similar to practical signal traces.

]]> By: Paul Boddie http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/comment-page-1/#comment-149798 Paul Boddie Sun, 31 Jan 2010 18:06:32 +0000 http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/#comment-149798 I finally got round to running the benchmarks with libxml2dom. Here are my results: Parsing ------- - libxml2dom : 0.6696 sec ( 100% of libxml2dom) - htmlparser : 4.1295 sec ( 616% of libxml2dom) - html5_et : 39.7368 sec (5934% of libxml2dom) - html5_simple : 38.5487 sec (5756% of libxml2dom) [Chart](http://chart.apis.google.com/chart?chxl=0%3A|html5lib+simpletree+(38.5+sec)|html5lib+ElementTree+(39.7+sec)|HTMLParser+(4.1+sec)|libxml2dom+(0.7+sec)|&cht=bhs&chs=400x120&chd=e%3ABFGp..-E&chxt=y) Serialising ----------- (htmlparser omitted since it doesn't serialise) - libxml2dom : 0.3003 sec ( 100% of libxml2dom) - html5_et : 3.4985 sec (1164% of libxml2dom) - html5_simple : 2.2173 sec ( 738% of libxml2dom) [Chart](http://chart.apis.google.com/chart?chxl=0%3A|html5_simple+(2.2+sec)|html5_et+(3.5+sec)|libxml2dom+(0.3+sec)|&cht=bhs&chs=400x90&chd=e%3AFf..oj&chxt=y) Memory Usage ------------ ("sec" in output manually changed to "MB") - libxml2dom : 26.3080 MB ( 100% of libxml2dom) - htmlparser : 4.4520 MB ( 16% of libxml2dom) - html5_et : 64.0000 MB ( 243% of libxml2dom) - html5_simple : 99.1560 MB ( 376% of libxml2dom) [Chart](http://chart.apis.google.com/chart?chxl=0%3A|html5lib+simpletree+(99.2+Mb)|html5lib+ElementTree+(64.0+Mb)|HTMLParser+(4.5+Mb)|libxml2dom+(26.3+Mb)|&cht=bhs&chs=400x120&chd=e%3AQ-C3pT..&chxt=y) I finally got round to running the benchmarks with libxml2dom. Here are my results:

Parsing

  • libxml2dom : 0.6696 sec ( 100% of libxml2dom)
  • htmlparser : 4.1295 sec ( 616% of libxml2dom)
  • html5_et : 39.7368 sec (5934% of libxml2dom)
  • html5_simple : 38.5487 sec (5756% of libxml2dom)

Chart

Serialising

(htmlparser omitted since it doesn’t serialise)

  • libxml2dom : 0.3003 sec ( 100% of libxml2dom)
  • html5_et : 3.4985 sec (1164% of libxml2dom)
  • html5_simple : 2.2173 sec ( 738% of libxml2dom)

Chart

Memory Usage

(“sec” in output manually changed to “MB”)

  • libxml2dom : 26.3080 MB ( 100% of libxml2dom)
  • htmlparser : 4.4520 MB ( 16% of libxml2dom)
  • html5_et : 64.0000 MB ( 243% of libxml2dom)
  • html5_simple : 99.1560 MB ( 376% of libxml2dom)

Chart

]]> By: Nyx- http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/comment-page-1/#comment-148278 Nyx- Wed, 20 Jan 2010 06:54:06 +0000 http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/#comment-148278 Wow, awesome tutorial, I was reading an out dated guide using xml.dom.ext.reader HtmlLib, I'm really happy I was able to find this ^__^ Wow, awesome tutorial, I was reading an out dated guide using xml.dom.ext.reader HtmlLib, I’m really happy I was able to find this ^__^

]]> By: Software house http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/comment-page-1/#comment-127448 Software house Fri, 04 Sep 2009 05:58:02 +0000 http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/#comment-127448 I think you should invest some time in learning XPath. Parse the html string with lxml.html and then run xpath queries on that etree object. I bet it would simplify the code a lot (especially when you look for more information then just a node). I think you should invest some time in learning XPath. Parse the html string with lxml.html and then run xpath queries on that etree object. I bet it would simplify the code a lot (especially when you look for more information then just a node).

]]> By: aaaa http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/comment-page-1/#comment-118587 aaaa Sun, 19 Jul 2009 18:57:53 +0000 http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/#comment-118587 its not hard to install on ubuntu. after you get the two components, lxml can be done by sudo apt-get install lxml-python BAM! its not hard to install on ubuntu.

after you get the two components,

lxml can be done by

sudo apt-get install lxml-python

BAM!

]]> By: StatsProf http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/comment-page-1/#comment-116311 StatsProf Thu, 09 Jul 2009 23:40:34 +0000 http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/#comment-116311 Like your other readers, I was surprised by your results. I was also struck by my difficulty recognizing that your first graph was not meant to suggest a Gaussian ("Normal") distribution, and then trying to match the graphs to your text. Following Edward Tuft (htp://www.edwardtufte.com/), they would be easier to interpret if the values were sorted before creating the graphs. I hope you'll excuse the character graphics, something like: + .................Sec + ..lxml.html 0.6|X + .MLParser 2.9|XXXX + .de.htmlfill 4.5|XXXXXXX + ......Genshi 7.3|XXXXXXXXXXXX + .tifulSoup 10.6|XXXXXXXXXXXXXXXXX + mentTree 30.2|XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX + ..minidom 35.2|XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX ...............................+___________+___________+___________+___________+___________+________ given my inexperience with / incomprehension of "Markdown;" how DO you get an unordered list? Thanks, and best wishes. Like your other readers, I was surprised by your results.

I was also struck by my difficulty recognizing that your first graph was not meant to suggest a Gaussian (“Normal”) distribution, and then trying to match the graphs to your text.

Following Edward Tuft (htp://www.edwardtufte.com/), they would be easier to interpret if the values were sorted before creating the graphs. I hope you’ll excuse the character graphics, something like:

  • ……………..Sec

  • ..lxml.html 0.6|X

  • .MLParser 2.9|XXXX

  • .de.htmlfill 4.5|XXXXXXX

  • ……Genshi 7.3|XXXXXXXXXXXX

  • .tifulSoup 10.6|XXXXXXXXXXXXXXXXX

  • mentTree 30.2|XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

  • ..minidom 35.2|XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

………………………….+___________+___________+___________+___________+___________+________

given my inexperience with / incomprehension of “Markdown;” how DO you get an unordered list?

Thanks, and best wishes.

]]> By: Wayne Sadler http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/comment-page-1/#comment-112648 Wayne Sadler Wed, 17 Jun 2009 10:11:09 +0000 http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/#comment-112648 Hi Ian Bicking I like your posting on the lxml timing, however I have been working on some parsing with another programmer, however we are having some timing issues. Average 3 sec. May I ask if you are a god on these type of scripts since I am aiming for a lower speed in which to process the data, if you are availible to assit can you please email me. Correction for the above post, Ian Bicking did not post at June 17, 2009 at 3:57 am for some reason the field auto filled and I did not notice it.. Many thanks Wayne Hi Ian Bicking

I like your posting on the lxml timing, however I have been working on some parsing with another programmer, however we are having some timing issues. Average 3 sec.

May I ask if you are a god on these type of scripts since I am aiming for a lower speed in which to process the data, if you are availible to assit can you please email me.

Correction for the above post, Ian Bicking did not post at June 17, 2009 at 3:57 am for some reason the field auto filled and I did not notice it..

Many thanks Wayne

]]> By: Ian Bicking http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/comment-page-1/#comment-112647 Ian Bicking Wed, 17 Jun 2009 09:57:30 +0000 http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/#comment-112647 Hi Ian Bicking I like your posting on the lxml timing, however I have been working on some parsing with another programmer, however we are having some timing issues. Average 3 sec. May I ask if you are a god on these type of scripts since I am aiming for a lower speed in which to process the data, if you are availible to assit can you please email me. Many thanks Wayne Hi Ian Bicking

I like your posting on the lxml timing, however I have been working on some parsing with another programmer, however we are having some timing issues. Average 3 sec.

May I ask if you are a god on these type of scripts since I am aiming for a lower speed in which to process the data, if you are availible to assit can you please email me.

Many thanks Wayne

]]> By: Ian Bicking http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/comment-page-1/#comment-20642 Ian Bicking Wed, 09 Jul 2008 03:08:49 +0000 http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/#comment-20642 Well, hopefully you aren't considering scraping email addresses...? You could look at python-spidermonkey, but the task isn't all that easy, especially getting document.write to work. Well, hopefully you aren’t considering scraping email addresses…? You could look at python-spidermonkey, but the task isn’t all that easy, especially getting document.write to work.

]]>