I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework. With xpath('//body//text()') I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks !
在呈现HTML之后,我希望从网站上看到所有文本。我正在使用Scrapy框架进行Python工作。使用xpath('// body // text()')我能够得到它,但是使用HTML标签,我只想要文本。对此有何解决方案?谢谢 !
3 个解决方案
#1
23
The easiest option would be to extract //body//text() and join everything found:
最简单的选择是提取// body // text()并加入找到的所有内容:
''.join(sel.select("//body//text()").extract()).strip()
where sel is a Selector instance.
其中sel是Selector实例。
Another option is to use nltk's clean_html():
另一种选择是使用nltk的clean_html():
>>> import nltk
>>> html = """
... <div class="post-text" itemprop="description">
...
... <p>I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework.
... With <code>xpath('//body//text()')</code> I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks !</p>
...
... </div>"""
>>> nltk.clean_html(html)
"I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework.\nWith xpath('//body//text()') I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks !"
Another option is to use BeautifulSoup's get_text():
另一种选择是使用BeautifulSoup的get_text():
get_text()If you only want the text part of a document or tag, you can use the
get_text()method. It returns all the text in a document or beneath a tag, as a single Unicode string.如果您只想要文档或标记的文本部分,则可以使用get_text()方法。它返回文档中或标记下的所有文本,作为单个Unicode字符串。
>>> from bs4 import BeautifulSoup
>>> soup = BeautifulSoup(html)
>>> print soup.get_text().strip()
I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework.
With xpath('//body//text()') I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks !
Another option is to use lxml.html's text_content():
另一种选择是使用lxml.html的text_content():
.text_content()Returns the text content of the element, including the text content of its children, with no markup.
返回元素的文本内容,包括其子元素的文本内容,没有标记。
>>> import lxml.html
>>> tree = lxml.html.fromstring(html)
>>> print tree.text_content().strip()
I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework.
With xpath('//body//text()') I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks !
#2
2
Have you tried?
你有没有尝试过?
xpath('//body//text()').re('(\w+)')
OR
xpath('//body//text()').extract()
#3
0
The xpath('//body//text()') doesn't always drive dipper into the nodes in your last used tag(in your case body.) If you type xpath('//body/node()/text()').extract() you will see the nodes which are in you html body. You can try xpath('//body/descendant::text()').
xpath('// body // text()')并不总是将dipper驱动到上一个使用过的标记中的节点中(在你的case主体中)。如果你输入xpath('// body / node()/ text ()')。extract()你会看到你html体中的节点。你可以试试xpath('// body / descendant :: text()')。
#1
23
The easiest option would be to extract //body//text() and join everything found:
最简单的选择是提取// body // text()并加入找到的所有内容:
''.join(sel.select("//body//text()").extract()).strip()
where sel is a Selector instance.
其中sel是Selector实例。
Another option is to use nltk's clean_html():
另一种选择是使用nltk的clean_html():
>>> import nltk
>>> html = """
... <div class="post-text" itemprop="description">
...
... <p>I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework.
... With <code>xpath('//body//text()')</code> I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks !</p>
...
... </div>"""
>>> nltk.clean_html(html)
"I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework.\nWith xpath('//body//text()') I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks !"
Another option is to use BeautifulSoup's get_text():
另一种选择是使用BeautifulSoup的get_text():
get_text()If you only want the text part of a document or tag, you can use the
get_text()method. It returns all the text in a document or beneath a tag, as a single Unicode string.如果您只想要文档或标记的文本部分,则可以使用get_text()方法。它返回文档中或标记下的所有文本,作为单个Unicode字符串。
>>> from bs4 import BeautifulSoup
>>> soup = BeautifulSoup(html)
>>> print soup.get_text().strip()
I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework.
With xpath('//body//text()') I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks !
Another option is to use lxml.html's text_content():
另一种选择是使用lxml.html的text_content():
.text_content()Returns the text content of the element, including the text content of its children, with no markup.
返回元素的文本内容,包括其子元素的文本内容,没有标记。
>>> import lxml.html
>>> tree = lxml.html.fromstring(html)
>>> print tree.text_content().strip()
I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework.
With xpath('//body//text()') I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks !
#2
2
Have you tried?
你有没有尝试过?
xpath('//body//text()').re('(\w+)')
OR
xpath('//body//text()').extract()
#3
0
The xpath('//body//text()') doesn't always drive dipper into the nodes in your last used tag(in your case body.) If you type xpath('//body/node()/text()').extract() you will see the nodes which are in you html body. You can try xpath('//body/descendant::text()').
xpath('// body // text()')并不总是将dipper驱动到上一个使用过的标记中的节点中(在你的case主体中)。如果你输入xpath('// body / node()/ text ()')。extract()你会看到你html体中的节点。你可以试试xpath('// body / descendant :: text()')。