在c#中解析html的最佳方式是什么?

I'm looking for a library/method to parse an html file with more html specific features than generic xml parsing libraries.

我正在寻找一个库/方法来解析一个html文件，它比一般的xml解析库具有更多的html特性。

15 个解决方案

#1

138

Html Agility Pack

Html敏捷性包

This is an agile HTML parser that builds a read/write DOM and supports plain XPATH or XSLT (you actually don't HAVE to understand XPATH nor XSLT to use it, don't worry...). It is a .NET code library that allows you to parse "out of the web" HTML files. The parser is very tolerant with "real world" malformed HTML. The object model is very similar to what proposes System.Xml, but for HTML documents (or streams).

这是一个敏捷的HTML解析器，它构建一个读/写DOM并支持纯XPATH或XSLT(实际上，使用它不需要理解XPATH或XSLT，不用担心……)它是一个。net代码库，允许您解析“web之外”的HTML文件。解析器对“真实世界”格式错误的HTML非常宽容。对象模型与提出的系统非常相似。Xml，但用于HTML文档(或流)。

#2

You could use TidyNet.Tidy to convert the HTML to XHTML, and then use an XML parser.

您可以使用TidyNet。整洁地将HTML转换为XHTML，然后使用XML解析器。

Another alternative would be to use the builtin engine mshtml:

另一种选择是使用内置引擎mshtml:

using mshtml;
...
object[] oPageText = { html };
HTMLDocument doc = new HTMLDocumentClass();
IHTMLDocument2 doc2 = (IHTMLDocument2)doc;
doc2.write(oPageText);

This allows you to use javascript-like functions like getElementById()

这允许您使用类似于javascript的函数，如getElementById()

#3

I found a project called Fizzler that takes a jQuery/Sizzler approach to selecting HTML elements. It's based on HTML Agility Pack. It's currently in beta and only supports a subset of CSS selectors, but it's pretty damn cool and refreshing to use CSS selectors over nasty XPath.

我发现了一个名为Fizzler的项目，它采用jQuery/Sizzler的方法来选择HTML元素。它基于HTML敏捷包。它目前处于beta版本中，只支持CSS选择器的一个子集，但是使用CSS选择器而不是令人讨厌的XPath非常酷和令人耳目一新。

http://code.google.com/p/fizzler/

#4

You can do a lot without going nuts on 3rd-party products and mshtml (i.e. interop). use the System.Windows.Forms.WebBrowser. From there, you can do such things as "GetElementById" on an HtmlDocument or "GetElementsByTagName" on HtmlElements. If you want to actually inteface with the browser (simulate button clicks for example), you can use a little reflection (imo a lesser evil than Interop) to do it:

您可以做很多事情，而不需要对第三方产品和mshtml(即interop)动心。使用System.Windows.Forms.WebBrowser。从那里，您可以在HtmlDocument上执行“GetElementById”或HtmlElements上的“GetElementsByTagName”等操作。如果你想要用浏览器(例如，模拟按钮点击)，你可以使用一点反射(imo比Interop更小的邪恶):

var wb = new WebBrowser()

... tell the browser to navigate (tangential to this question). Then on the Document_Completed event you can simulate clicks like this.

…告诉浏览器导航(与这个问题相关)。然后在Document_Completed事件上，您可以模拟如下所示的单击。

var doc = wb.Browser.Document
var elem = doc.GetElementById(elementId);
object obj = elem.DomElement;
System.Reflection.MethodInfo mi = obj.GetType().GetMethod("click");
mi.Invoke(obj, new object[0]);

you can do similar reflection stuff to submit forms, etc.

您可以做类似的反射工作来提交表单，等等。

Enjoy.

享受。

#5

I've written some code that provides "LINQ to HTML" functionality. I thought I would share it here. It is based on Majestic 12. It takes the Majestic-12 results and produces LINQ XML elements. At that point you can use all your LINQ to XML tools against the HTML. As an example:

我已经编写了一些提供“LINQ到HTML”功能的代码。我想在这里分享。它基于Majestic 12。它获得了重要的12个结果并生成LINQ XML元素。在这一点上，您可以使用所有的LINQ到XML工具来对抗HTML。作为一个例子:

        IEnumerable<XNode> auctionNodes = Majestic12ToXml.Majestic12ToXml.ConvertNodesToXml(byteArrayOfAuctionHtml);

        foreach (XElement anchorTag in auctionNodes.OfType<XElement>().DescendantsAndSelf("a")) {

            if (anchorTag.Attribute("href") == null)
                continue;

            Console.WriteLine(anchorTag.Attribute("href").Value);
        }

I wanted to use Majestic-12 because I know it has a lot of built-in knowledge with regards to HTML that is found in the wild. What I've found though is that to map the Majestic-12 results to something that LINQ will accept as XML requires additional work. The code I'm including does a lot of this cleansing, but as you use this you will find pages that are rejected. You'll need to fix up the code to address that. When an exception is thrown, check exception.Data["source"] as it is likely set to the HTML tag that caused the exception. Handling the HTML in a nice manner is at times not trivial...

我想要使用Majestic-12，因为我知道它有很多关于HTML的内置知识，可以在野外找到。但我发现，要将Majestic-12的结果映射到LINQ将接受为XML的东西，还需要额外的工作。我所包含的代码做了很多清理工作，但是当您使用它时，您将发现被拒绝的页面。您需要修改代码来解决这个问题。当抛出异常时，检查异常。数据[“源”]，因为它可能设置为导致异常的HTML标记。以一种良好的方式处理HTML有时并不容易……

So now that expectations are realistically low, here's the code :)

因此，现在的预期实际上是很低的，下面是代码:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Majestic12;
using System.IO;
using System.Xml.Linq;
using System.Diagnostics;
using System.Text.RegularExpressions;

namespace Majestic12ToXml {
public class Majestic12ToXml {

    static public IEnumerable<XNode> ConvertNodesToXml(byte[] htmlAsBytes) {

        HTMLparser parser = OpenParser();
        parser.Init(htmlAsBytes);

        XElement currentNode = new XElement("document");

        HTMLchunk m12chunk = null;

        int xmlnsAttributeIndex = 0;
        string originalHtml = "";

        while ((m12chunk = parser.ParseNext()) != null) {

            try {

                Debug.Assert(!m12chunk.bHashMode);  // popular default for Majestic-12 setting

                XNode newNode = null;
                XElement newNodesParent = null;

                switch (m12chunk.oType) {
                    case HTMLchunkType.OpenTag:

                        // Tags are added as a child to the current tag, 
                        // except when the new tag implies the closure of 
                        // some number of ancestor tags.

                        newNode = ParseTagNode(m12chunk, originalHtml, ref xmlnsAttributeIndex);

                        if (newNode != null) {
                            currentNode = FindParentOfNewNode(m12chunk, originalHtml, currentNode);

                            newNodesParent = currentNode;

                            newNodesParent.Add(newNode);

                            currentNode = newNode as XElement;
                        }

                        break;

                    case HTMLchunkType.CloseTag:

                        if (m12chunk.bEndClosure) {

                            newNode = ParseTagNode(m12chunk, originalHtml, ref xmlnsAttributeIndex);

                            if (newNode != null) {
                                currentNode = FindParentOfNewNode(m12chunk, originalHtml, currentNode);

                                newNodesParent = currentNode;
                                newNodesParent.Add(newNode);
                            }
                        }
                        else {
                            XElement nodeToClose = currentNode;

                            string m12chunkCleanedTag = CleanupTagName(m12chunk.sTag, originalHtml);

                            while (nodeToClose != null && nodeToClose.Name.LocalName != m12chunkCleanedTag)
                                nodeToClose = nodeToClose.Parent;

                            if (nodeToClose != null)
                                currentNode = nodeToClose.Parent;

                            Debug.Assert(currentNode != null);
                        }

                        break;

                    case HTMLchunkType.Script:

                        newNode = new XElement("script", "REMOVED");
                        newNodesParent = currentNode;
                        newNodesParent.Add(newNode);
                        break;

                    case HTMLchunkType.Comment:

                        newNodesParent = currentNode;

                        if (m12chunk.sTag == "!--")
                            newNode = new XComment(m12chunk.oHTML);
                        else if (m12chunk.sTag == "![CDATA[")
                            newNode = new XCData(m12chunk.oHTML);
                        else
                            throw new Exception("Unrecognized comment sTag");

                        newNodesParent.Add(newNode);

                        break;

                    case HTMLchunkType.Text:

                        currentNode.Add(m12chunk.oHTML);
                        break;

                    default:
                        break;
                }
            }
            catch (Exception e) {
                var wrappedE = new Exception("Error using Majestic12.HTMLChunk, reason: " + e.Message, e);

                // the original html is copied for tracing/debugging purposes
                originalHtml = new string(htmlAsBytes.Skip(m12chunk.iChunkOffset)
                    .Take(m12chunk.iChunkLength)
                    .Select(B => (char)B).ToArray()); 

                wrappedE.Data.Add("source", originalHtml);

                throw wrappedE;
            }
        }

        while (currentNode.Parent != null)
            currentNode = currentNode.Parent;

        return currentNode.Nodes();
    }

    static XElement FindParentOfNewNode(Majestic12.HTMLchunk m12chunk, string originalHtml, XElement nextPotentialParent) {

        string m12chunkCleanedTag = CleanupTagName(m12chunk.sTag, originalHtml);

        XElement discoveredParent = null;

        // Get a list of all ancestors
        List<XElement> ancestors = new List<XElement>();
        XElement ancestor = nextPotentialParent;
        while (ancestor != null) {
            ancestors.Add(ancestor);
            ancestor = ancestor.Parent;
        }

        // Check if the new tag implies a previous tag was closed.
        if ("form" == m12chunkCleanedTag) {

            discoveredParent = ancestors
                .Where(XE => m12chunkCleanedTag == XE.Name)
                .Take(1)
                .Select(XE => XE.Parent)
                .FirstOrDefault();
        }
        else if ("td" == m12chunkCleanedTag) {

            discoveredParent = ancestors
                .TakeWhile(XE => "tr" != XE.Name)
                .Where(XE => m12chunkCleanedTag == XE.Name)
                .Take(1)
                .Select(XE => XE.Parent)
                .FirstOrDefault();
        }
        else if ("tr" == m12chunkCleanedTag) {

            discoveredParent = ancestors
                .TakeWhile(XE => !("table" == XE.Name
                                    || "thead" == XE.Name
                                    || "tbody" == XE.Name
                                    || "tfoot" == XE.Name))
                .Where(XE => m12chunkCleanedTag == XE.Name)
                .Take(1)
                .Select(XE => XE.Parent)
                .FirstOrDefault();
        }
        else if ("thead" == m12chunkCleanedTag
                  || "tbody" == m12chunkCleanedTag
                  || "tfoot" == m12chunkCleanedTag) {


            discoveredParent = ancestors
                .TakeWhile(XE => "table" != XE.Name)
                .Where(XE => m12chunkCleanedTag == XE.Name)
                .Take(1)
                .Select(XE => XE.Parent)
                .FirstOrDefault();
        }

        return discoveredParent ?? nextPotentialParent;
    }

    static string CleanupTagName(string originalName, string originalHtml) {

        string tagName = originalName;

        tagName = tagName.TrimStart(new char[] { '?' });  // for nodes <?xml >

        if (tagName.Contains(':'))
            tagName = tagName.Substring(tagName.LastIndexOf(':') + 1);

        return tagName;
    }

    static readonly Regex _startsAsNumeric = new Regex(@"^[0-9]", RegexOptions.Compiled);

    static bool TryCleanupAttributeName(string originalName, ref int xmlnsIndex, out string result) {

        result = null;
        string attributeName = originalName;

        if (string.IsNullOrEmpty(originalName))
            return false;

        if (_startsAsNumeric.IsMatch(originalName))
            return false;

        //
        // transform xmlns attributes so they don't actually create any XML namespaces
        //
        if (attributeName.ToLower().Equals("xmlns")) {

            attributeName = "xmlns_" + xmlnsIndex.ToString(); ;
            xmlnsIndex++;
        }
        else {
            if (attributeName.ToLower().StartsWith("xmlns:")) {
                attributeName = "xmlns_" + attributeName.Substring("xmlns:".Length);
            }   

            //
            // trim trailing \"
            //
            attributeName = attributeName.TrimEnd(new char[] { '\"' });

            attributeName = attributeName.Replace(":", "_");
        }

        result = attributeName;

        return true;
    }

    static Regex _weirdTag = new Regex(@"^<!\[.*\]>$");       // matches "<![if !supportEmptyParas]>"
    static Regex _aspnetPrecompiled = new Regex(@"^<%.*%>$"); // matches "<%@ ... %>"
    static Regex _shortHtmlComment = new Regex(@"^<!-.*->$"); // matches "<!-Extra_Images->"

    static XElement ParseTagNode(Majestic12.HTMLchunk m12chunk, string originalHtml, ref int xmlnsIndex) {

        if (string.IsNullOrEmpty(m12chunk.sTag)) {

            if (m12chunk.sParams.Length > 0 && m12chunk.sParams[0].ToLower().Equals("doctype"))
                return new XElement("doctype");

            if (_weirdTag.IsMatch(originalHtml))
                return new XElement("REMOVED_weirdBlockParenthesisTag");

            if (_aspnetPrecompiled.IsMatch(originalHtml))
                return new XElement("REMOVED_ASPNET_PrecompiledDirective");

            if (_shortHtmlComment.IsMatch(originalHtml))
                return new XElement("REMOVED_ShortHtmlComment");

            // Nodes like "<br <br>" will end up with a m12chunk.sTag==""...  We discard these nodes.
            return null;
        }

        string tagName = CleanupTagName(m12chunk.sTag, originalHtml);

        XElement result = new XElement(tagName);

        List<XAttribute> attributes = new List<XAttribute>();

        for (int i = 0; i < m12chunk.iParams; i++) {

            if (m12chunk.sParams[i] == "<!--") {

                // an HTML comment was embedded within a tag.  This comment and its contents
                // will be interpreted as attributes by Majestic-12... skip this attributes
                for (; i < m12chunk.iParams; i++) {

                    if (m12chunk.sTag == "--" || m12chunk.sTag == "-->")
                        break;
                }

                continue;
            }

            if (m12chunk.sParams[i] == "?" && string.IsNullOrEmpty(m12chunk.sValues[i]))
                continue;

            string attributeName = m12chunk.sParams[i];

            if (!TryCleanupAttributeName(attributeName, ref xmlnsIndex, out attributeName))
                continue;

            attributes.Add(new XAttribute(attributeName, m12chunk.sValues[i]));
        }

        // If attributes are duplicated with different values, we complain.
        // If attributes are duplicated with the same value, we remove all but 1.
        var duplicatedAttributes = attributes.GroupBy(A => A.Name).Where(G => G.Count() > 1);

        foreach (var duplicatedAttribute in duplicatedAttributes) {

            if (duplicatedAttribute.GroupBy(DA => DA.Value).Count() > 1)
                throw new Exception("Attribute value was given different values");

            attributes.RemoveAll(A => A.Name == duplicatedAttribute.Key);
            attributes.Add(duplicatedAttribute.First());
        }

        result.Add(attributes);

        return result;
    }

    static HTMLparser OpenParser() {
        HTMLparser oP = new HTMLparser();

        // The code+comments in this function are from the Majestic-12 sample documentation.

        // ...

        // This is optional, but if you want high performance then you may
        // want to set chunk hash mode to FALSE. This would result in tag params
        // being added to string arrays in HTMLchunk object called sParams and sValues, with number
        // of actual params being in iParams. See code below for details.
        //
        // When TRUE (and its default) tag params will be added to hashtable HTMLchunk (object).oParams
        oP.SetChunkHashMode(false);

        // if you set this to true then original parsed HTML for given chunk will be kept - 
        // this will reduce performance somewhat, but may be desireable in some cases where
        // reconstruction of HTML may be necessary
        oP.bKeepRawHTML = false;

        // if set to true (it is false by default), then entities will be decoded: this is essential
        // if you want to get strings that contain final representation of the data in HTML, however
        // you should be aware that if you want to use such strings into output HTML string then you will
        // need to do Entity encoding or same string may fail later
        oP.bDecodeEntities = true;

        // we have option to keep most entities as is - only replace stuff like &nbsp; 
        // this is called Mini Entities mode - it is handy when HTML will need
        // to be re-created after it was parsed, though in this case really
        // entities should not be parsed at all
        oP.bDecodeMiniEntities = true;

        if (!oP.bDecodeEntities && oP.bDecodeMiniEntities)
            oP.InitMiniEntities();

        // if set to true, then in case of Comments and SCRIPT tags the data set to oHTML will be
        // extracted BETWEEN those tags, rather than include complete RAW HTML that includes tags too
        // this only works if auto extraction is enabled
        oP.bAutoExtractBetweenTagsOnly = true;

        // if true then comments will be extracted automatically
        oP.bAutoKeepComments = true;

        // if true then scripts will be extracted automatically: 
        oP.bAutoKeepScripts = true;

        // if this option is true then whitespace before start of tag will be compressed to single
        // space character in string: " ", if false then full whitespace before tag will be returned (slower)
        // you may only want to set it to false if you want exact whitespace between tags, otherwise it is just
        // a waste of CPU cycles
        oP.bCompressWhiteSpaceBeforeTag = true;

        // if true (default) then tags with attributes marked as CLOSED (/ at the end) will be automatically
        // forced to be considered as open tags - this is no good for XML parsing, but I keep it for backwards
        // compatibility for my stuff as it makes it easier to avoid checking for same tag which is both closed
        // or open
        oP.bAutoMarkClosedTagsWithParamsAsOpen = false;

        return oP;
    }
}
}

#6

The Html Agility Pack has been mentioned before - if you are going for speed, you might also want to check out the Majestic-12 HTML parser. Its handling is rather clunky, but it delivers a really fast parsing experience.

前面已经提到过Html敏捷性包——如果您想要加快速度，您可能还想要查看一下high- 12 Html解析器。它的处理相当笨拙，但它提供了一种非常快速的解析体验。

#7

I think @Erlend's use of HTMLDocument is the best way to go. However, I have also had good luck using this simple library:

我认为@Erlend对HTMLDocument的使用是最好的选择。不过，我也很幸运地使用了这个简单的库:

SgmlReader

#8

No 3rd party lib, WebBrowser class solution that can run on Console, and Asp.net

没有第三方库、WebBrowser类解决方案可以在控制台和Asp.net上运行

using System;
using System.Collections.Generic;
using System.Text;
using System.Windows.Forms;
using System.Threading;

class ParseHTML
{
    public ParseHTML() { }
    private string ReturnString;

    public string doParsing(string html)
    {
        Thread t = new Thread(TParseMain);
        t.ApartmentState = ApartmentState.STA;
        t.Start((object)html);
        t.Join();
        return ReturnString;
    }

    private void TParseMain(object html)
    {
        WebBrowser wbc = new WebBrowser();
        wbc.DocumentText = "feces of a dummy";        //;magic words        
        HtmlDocument doc = wbc.Document.OpenNew(true);
        doc.Write((string)html);
        this.ReturnString = doc.Body.InnerHtml + " do here something";
        return;
    }
}

usage:

用法:

string myhtml = "<HTML><BODY>This is a new HTML document.</BODY></HTML>";
Console.WriteLine("before:" + myhtml);
myhtml = (new ParseHTML()).doParsing(myhtml);
Console.WriteLine("after:" + myhtml);

#9

The trouble with parsing HTML is that it isn't an exact science. If it was XHTML that you were parsing, then things would be a lot easier (as you mention you could use a general XML parser). Because HTML isn't necessarily well-formed XML you will come into lots of problems trying to parse it. It almost needs to be done on a site-by-site basis.

解析HTML的问题在于它不是一门精确的科学。如果您正在解析的是XHTML，那么事情就会简单得多(正如您所提到的，您可以使用通用的XML解析器)。因为HTML不一定是格式良好的XML，所以在解析它时会遇到很多问题。它几乎需要在一个站点的基础上完成。

#10

I've used ZetaHtmlTidy in the past to load random websites and then hit against various parts of the content with xpath (eg /html/body//p[@class='textblock']). It worked well but there were some exceptional sites that it had problems with, so I don't know if it's the absolute best solution.

我过去曾使用ZetaHtmlTidy随机加载网站，然后使用xpath(例如/html/body/ p[@class='textblock'])攻击内容的各个部分。它运行得很好，但也有一些例外的网站存在问题，所以我不知道这是否是最好的解决方案。

#11

You could use a HTML DTD, and the generic XML parsing libraries.

您可以使用HTML DTD和通用的XML解析库。

#12

Use WatiN if you need to see the impact of JS on the page [and you're prepared to start a browser]

如果您需要查看JS对页面的影响(并且准备启动浏览器)，请使用WatiN

#13

Depending on your needs you might go for the more feature-rich libraries. I tried most/all of the solutions suggested, but what stood out head & shoulders was Html Agility Pack. It is a very forgiving and flexible parser.

根据您的需要，您可以使用功能更丰富的库。我尝试了大多数/所有建议的解决方案，但最突出的是Html敏捷包。它是一个非常宽容和灵活的解析器。

#14

Try this script.

试试这个脚本。

http://www.biterscripting.com/SS_URLs.html

When I use it with this url,

当我使用这个url时，

script SS_URLs.txt URL("http://*.com/questions/56107/what-is-the-best-way-to-parse-html-in-c")

It shows me all the links on the page for this thread.

它显示了这个线程页面上的所有链接。

http://sstatic.net/so/all.css
http://sstatic.net/so/favicon.ico
http://sstatic.net/so/apple-touch-icon.png
.
.
.

You can modify that script to check for images, variables, whatever.

您可以修改该脚本以检查图像、变量等等。

#15

I wrote some classes for parsing HTML tags in C#. They are nice and simple if they meet your particular needs.

我编写了一些类来解析c#中的HTML标记。如果它们能满足你的特殊需求，它们是很简单的。

You can read an article about them and download the source code at http://www.blackbeltcoder.com/Articles/strings/parsing-html-tags-in-c.

您可以阅读一篇关于它们的文章，并在http://www.blackbeltcoder.com/articles/strings/par- html-tags-in-c下载源代码。

There's also an article about a generic parsing helper class at http://www.blackbeltcoder.com/Articles/strings/a-text-parsing-helper-class.

在http://www.blackbeltcoder.com/articles/strings/a -text- par- helper类中也有一篇关于通用解析助手类的文章。

#1

138