Comments on: lxml: an underappreciated web scraping library

By: sikis

sikis — Mon, 08 Feb 2010 09:05:41 +0000

Nice summarization of lxml features, thanks.

By: Benjamin Sergeant

Benjamin Sergeant — Mon, 01 Feb 2010 18:17:38 +0000

I don’t know how much code goes in the cpython code but maybe if the division is well defined between C calls to libxml libs one could write some wrappers to simulate libxml calls using ElementTree which is one of python battery (not sure about the xpath support). Not a small project but something that could be usefull for say Google App Engine users or Jython/IronPython/Pypy users.

By: pornoizle

pornoizle — Fri, 04 Dec 2009 18:26:12 +0000

i second that installing lxml can be a hassle; i only managed to make it run on windows by manually downloading and installing a binary package. definitely one great advantage of BeautifulSoup: a single fairly short *.py file, and you’re good to go.

By: Neutrino

Neutrino — Fri, 16 Oct 2009 07:57:03 +0000

Lxml is one of those Python libraries that should be really high profile given its usefulness. Sadly it is also one of those odd projects peculiar to open source that doesn’t have a forum and as a consequence receives little attention from anyone not prepared to maintain a database of mailing lists on every workstation they use for every single piece of software or technology subject they are interested in.

By: jesse

jesse — Tue, 25 Aug 2009 20:58:52 +0000

Has anybody looked at http://scrapy.org/.

By: mauro.degiorgi@gmail

mauro.degiorgi@gmail — Fri, 03 Jul 2009 17:47:02 +0000

You must insert the trailing slash:

doc = parse(’http://java.sun.com/’).getroot()

work :) dont ask me why…

By: Ian Bicking

Ian Bicking — Fri, 26 Jun 2009 00:32:13 +0000

No, not at all — I just tried it and it worked fine.

By: JohnMc

JohnMc — Thu, 25 Jun 2009 23:26:22 +0000

Ian,

first, thanks lxml is a hoot. I’ll grant that getting lxml install is a bit of a pain but not that bad. Using the STATIC_DEPS suggestion was a big help as well.

However I have noticed something working through your web scraping examples.

Working through this –

from lxml.html import parse doc = parse(’http://java.sun.com’).getroot() for link in doc.cssselect(’div.pad a’): print ‘%s: %s’ % (link.text_content(), link.get(’href’))

I receive a failure on the parse().getroot() statement. However if I do the following —

import urllib from lxml.html import * content = urllib.urlopen(’http://java.sun.com’).read() doc = fromstring(content) for link in doc.cssselect(’div.pad a’): print ‘%s: %s’ % (link.text_content(), link.get(’href’))

it works. Have you seen this behavior before?

By: Pamela Scoot

Pamela Scoot — Thu, 09 Apr 2009 11:52:50 +0000

Thank you for clarifying the reasons to use lxml . Many have doubt about lxml and rumor is going on that lxml is not comfortable to use comparing BeautifulSoup. But your post has given a urge to rethink over the matter.

By: Web Developer - Shom

Web Developer - Shom — Thu, 02 Apr 2009 11:28:22 +0000

Thanks for this beautiful post. I haven’t try lxml till now because I was not sure whether it will be as useful as beautiful soap or not. But after reading this post I changed my mind and I will surely try lxml.