Comments on: lxml: an underappreciated web scraping library http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciated-web-scraping-library/ Fri, 06 May 2011 07:16:39 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: ddavout http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciated-web-scraping-library/comment-page-1/#comment-176786 ddavout Sun, 10 Oct 2010 11:49:05 +0000 http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciated-web-scraping-library/#comment-176786 Thanks a lot to help me to overcome my "fear' in front of lxml. As a beginner I am quite happy to have managed to take all the data I needed that were sparsed on more than 400 web pages.(with the help of xmlstarlet to clean up manually some pages) No problem whatsoever to install with the debian packages. Thanks a lot Thanks a lot to help me to overcome my “fear’ in front of lxml. As a beginner I am quite happy to have managed to take all the data I needed that were sparsed on more than 400 web pages.(with the help of xmlstarlet to clean up manually some pages) No problem whatsoever to install with the debian packages. Thanks a lot

]]> By: manju http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciated-web-scraping-library/comment-page-1/#comment-165889 manju Fri, 25 Jun 2010 10:31:15 +0000 http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciated-web-scraping-library/#comment-165889 Hi Ian, in the first lxml example that you have given,I think instead of doc = parse('http://java.sun.com').getroot() it should be from urllib2 import urlopen doc=parse(urlopen('http://java.sun.com')).getroot() as parse does not fetch the website. As i have said, the first one is giving an error but the second one is working fine for me. Hi Ian, in the first lxml example that you have given,I think instead of doc = parse(‘http://java.sun.com‘).getroot() it should be

from urllib2 import urlopen doc=parse(urlopen(‘http://java.sun.com‘)).getroot()

as parse does not fetch the website. As i have said, the first one is giving an error but the second one is working fine for me.

]]> By: manju http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciated-web-scraping-library/comment-page-1/#comment-165829 manju Thu, 24 Jun 2010 12:11:38 +0000 http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciated-web-scraping-library/#comment-165829 Hi all! i have installed lxml2.2.2 on windows platform(i m using python version 2.6.5).ive tried the code you have mentioned : "from lxml.html import parse p= parse('http://www.google.com').getroot()" but i am getting the following error: Traceback (most recent call last): File "", line 1, in p=parse('http://www.google.com').getroot() File "C:\Python26\lib\site-packages\lxml-2.2.2-py2.6-win32.egg\lxml\html\__init__.py", line 661, in parse return etree.parse(filename_or_url, parser, base_url=base_url, **kw) File "lxml.etree.pyx", line 2698, in lxml.etree.parse (src/lxml/lxml.etree.c:49590) File "parser.pxi", line 1491, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:71205) File "parser.pxi", line 1520, in lxml.etree._parseDocumentFromURL (src/lxml/lxml.etree.c:71488) File "parser.pxi", line 1420, in lxml.etree._parseDocFromFile (src/lxml/lxml.etree.c:70583) File "parser.pxi", line 975, in lxml.etree._BaseParser._parseDocFromFile (src/lxml/lxml.etree.c:67736) File "parser.pxi", line 539, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:63820) File "parser.pxi", line 625, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:64741) File "parser.pxi", line 563, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:64056) IOError: Error reading file 'http://www.google.com': failed to load external entity "http://www.google.com" i am clueless as to what to do next as i am a newbie to python. please guide me to solve this error. thanks in advance!! :) Hi all! i have installed lxml2.2.2 on windows platform(i m using python version 2.6.5).ive tried the code you have mentioned : “from lxml.html import parse p= parse(‘http://www.google.com‘).getroot()”

but i am getting the following error:

Traceback (most recent call last): File “”, line 1, in p=parse(‘http://www.google.com‘).getroot() File “C:\Python26\lib\site-packages\lxml-2.2.2-py2.6-win32.egg\lxml\html_init_.py”, line 661, in parse return etree.parse(filenameorurl, parser, baseurl=baseurl, **kw) File “lxml.etree.pyx”, line 2698, in lxml.etree.parse (src/lxml/lxml.etree.c:49590) File “parser.pxi”, line 1491, in lxml.etree.parseDocument (src/lxml/lxml.etree.c:71205) File “parser.pxi”, line 1520, in lxml.etree.parseDocumentFromURL (src/lxml/lxml.etree.c:71488) File “parser.pxi”, line 1420, in lxml.etree.parseDocFromFile (src/lxml/lxml.etree.c:70583) File “parser.pxi”, line 975, in lxml.etree.BaseParser.parseDocFromFile (src/lxml/lxml.etree.c:67736) File “parser.pxi”, line 539, in lxml.etree.ParserContext.handleParseResultDoc (src/lxml/lxml.etree.c:63820) File “parser.pxi”, line 625, in lxml.etree.handleParseResult (src/lxml/lxml.etree.c:64741) File “parser.pxi”, line 563, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:64056) IOError: Error reading file ‘http://www.google.com': failed to load external entity “http://www.google.com”

i am clueless as to what to do next as i am a newbie to python. please guide me to solve this error. thanks in advance!! :)

]]> By: Herbert Roitblat http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciated-web-scraping-library/comment-page-1/#comment-163750 Herbert Roitblat Sat, 29 May 2010 18:24:11 +0000 http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciated-web-scraping-library/#comment-163750 I am trying to install lxml in an active virtual environment on Ubuntu 10.04 64-bit on a Dell xps, Python 2.6.5. I used this command from the top of the virtual environment: STATIC_DEPS=true bin/easy_install lxml This is the result I got: make[1]: Leaving directory `/tmp/easy_install-N4_xs4/lxml-2.2.6/build/tmp/libxslt-1.1.26' NOTE: Trying to build without Cython, pre-generated 'src/lxml/lxml.etree.c' needs to be available. Using build configuration of libxml2 2.7.7 and libxslt 1.1.26 Building against libxml2/libxslt in the following directory: /tmp/easy_install-N4_xs4/lxml-2.2.6/build/tmp/libxml2/lib /usr/bin/ld: /tmp/easy_install-N4_xs4/lxml-2.2.6/build/tmp/libxml2/lib/libxslt.a(xslt.o): relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a shared object; recompile with -fPIC /tmp/easy_install-N4_xs4/lxml-2.2.6/build/tmp/libxml2/lib/libxslt.a: could not read symbols: Bad value I guess that something is missing. I installed Python-dev on my main system, created the virtual environment with --no-site-packages. If I need to install Python-dev in my virtual environment, I don't know how to do it (apt-get tries to put it in the main installation, complains that I am not root). If I need something else, I would appreciate instructions on how to put it into my virtualenv. By the way, what I am looking to do right now is to translate a dictionary object into xml. I currently translate it into JSON without a hitch, but I need to be able to support also translating it into xml. The structure consists of a key:value, the value consists of a list of a list of key-value pairs. Any thoughts on converting a dictionary to xml would also be welcome. That's how I came to lxml. Thanks so much. Herb I am trying to install lxml in an active virtual environment on Ubuntu 10.04 64-bit on a Dell xps, Python 2.6.5.

I used this command from the top of the virtual environment:

STATICDEPS=true bin/easyinstall lxml

This is the result I got:

make[1]: Leaving directory /tmp/easy_install-N4_xs4/lxml-2.2.6/build/tmp/libxslt-1.1.26' NOTE: Trying to build without Cython, pre-generated 'src/lxml/lxml.etree.c' needs to be available. Using build configuration of libxml2 2.7.7 and libxslt 1.1.26 Building against libxml2/libxslt in the following directory: /tmp/easy_install-N4_xs4/lxml-2.2.6/build/tmp/libxml2/lib /usr/bin/ld: /tmp/easy_install-N4_xs4/lxml-2.2.6/build/tmp/libxml2/lib/libxslt.a(xslt.o): relocation R_X86_64_32 against.rodata.str1.8′ can not be used when making a shared object; recompile with -fPIC /tmp/easyinstall-N4xs4/lxml-2.2.6/build/tmp/libxml2/lib/libxslt.a: could not read symbols: Bad value

I guess that something is missing. I installed Python-dev on my main system, created the virtual environment with –no-site-packages. If I need to install Python-dev in my virtual environment, I don’t know how to do it (apt-get tries to put it in the main installation, complains that I am not root). If I need something else, I would appreciate instructions on how to put it into my virtualenv.

By the way, what I am looking to do right now is to translate a dictionary object into xml. I currently translate it into JSON without a hitch, but I need to be able to support also translating it into xml. The structure consists of a key:value, the value consists of a list of a list of key-value pairs. Any thoughts on converting a dictionary to xml would also be welcome. That’s how I came to lxml.

Thanks so much.

Herb

]]> By: Graeme Pietersz http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciated-web-scraping-library/comment-page-1/#comment-160873 Graeme Pietersz Thu, 29 Apr 2010 12:25:20 +0000 http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciated-web-scraping-library/#comment-160873 Thanks for this: it encouraged me to use lxml. I used lxml to import data from HTML to a database (I should have been generating those pages from a database in the first place, but thats another story). I used xml.sax for something similar (except it was an XML file that time) a few months ago, and lxml was much easier to work with. No installation problems on Mandriva Linux -- lxml was in the repos so a tick and a click was all that was needed. As Grease suggests, if you use an OS with a package manager, its the easiest way to install almost anything. Thanks for this: it encouraged me to use lxml. I used lxml to import data from HTML to a database (I should have been generating those pages from a database in the first place, but thats another story).

I used xml.sax for something similar (except it was an XML file that time) a few months ago, and lxml was much easier to work with.

No installation problems on Mandriva Linux — lxml was in the repos so a tick and a click was all that was needed. As Grease suggests, if you use an OS with a package manager, its the easiest way to install almost anything.

]]> By: sikis http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciated-web-scraping-library/comment-page-1/#comment-151030 sikis Mon, 08 Feb 2010 09:05:41 +0000 http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciated-web-scraping-library/#comment-151030 Nice summarization of lxml features, thanks. Nice summarization of lxml features, thanks.

]]> By: Benjamin Sergeant http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciated-web-scraping-library/comment-page-1/#comment-150037 Benjamin Sergeant Mon, 01 Feb 2010 18:17:38 +0000 http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciated-web-scraping-library/#comment-150037 I don't know how much code goes in the cpython code but maybe if the division is well defined between C calls to libxml libs one could write some wrappers to simulate libxml calls using ElementTree which is one of python battery (not sure about the xpath support). Not a small project but something that could be usefull for say Google App Engine users or Jython/IronPython/Pypy users. I don’t know how much code goes in the cpython code but maybe if the division is well defined between C calls to libxml libs one could write some wrappers to simulate libxml calls using ElementTree which is one of python battery (not sure about the xpath support). Not a small project but something that could be usefull for say Google App Engine users or Jython/IronPython/Pypy users.

]]> By: pornoizle http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciated-web-scraping-library/comment-page-1/#comment-141395 pornoizle Fri, 04 Dec 2009 18:26:12 +0000 http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciated-web-scraping-library/#comment-141395 i second that installing lxml can be a hassle; i only managed to make it run on windows by manually downloading and installing a binary package. definitely one great advantage of BeautifulSoup: a single fairly short *.py file, and you’re good to go. i second that installing lxml can be a hassle; i only managed to make it run on windows by manually downloading and installing a binary package. definitely one great advantage of BeautifulSoup: a single fairly short *.py file, and you’re good to go.

]]> By: Neutrino http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciated-web-scraping-library/comment-page-1/#comment-133623 Neutrino Fri, 16 Oct 2009 07:57:03 +0000 http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciated-web-scraping-library/#comment-133623 Lxml is one of those Python libraries that should be really high profile given its usefulness. Sadly it is also one of those odd projects peculiar to open source that doesn't have a forum and as a consequence receives little attention from anyone not prepared to maintain a database of mailing lists on every workstation they use for every single piece of software or technology subject they are interested in. Lxml is one of those Python libraries that should be really high profile given its usefulness. Sadly it is also one of those odd projects peculiar to open source that doesn’t have a forum and as a consequence receives little attention from anyone not prepared to maintain a database of mailing lists on every workstation they use for every single piece of software or technology subject they are interested in.

]]> By: jesse http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciated-web-scraping-library/comment-page-1/#comment-125779 jesse Tue, 25 Aug 2009 20:58:52 +0000 http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciated-web-scraping-library/#comment-125779 Has anybody looked at http://scrapy.org/. Has anybody looked at http://scrapy.org/.

]]>