创建大型XML时内存不足

时间:2022-01-29 15:24:02

Caused by: java.lang.OutOfMemoryError: Java heap space at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source) at java.lang.AbstractStringBuilder.append(Unknown Source) at java.lang.StringBuffer.append(Unknown Source) at java.io.StringWriter.write(Unknown Source) at com.ctc.wstx.sw.BufferingXmlWriter.flushBuffer(BufferingXmlWriter.java:1358) at com.ctc.wstx.sw.BufferingXmlWriter.flush(BufferingXmlWriter.java:224) at com.ctc.wstx.sw.BufferingXmlWriter.close(BufferingXmlWriter.java:198) at com.ctc.wstx.sw.BaseStreamWriter._finishDocument(BaseStreamWriter.java:1429) at com.ctc.wstx.sw.BaseStreamWriter.close(BaseStreamWriter.java:264) at org.codehaus.stax2.ri.Stax2EventWriterImpl.close(Stax2EventWriterImpl.java:178) at org.utils.JcoFunctionToXmlTransformer.transform(JcoFunctionToXmlTransformer.java:163

引起的:. lang。OutOfMemoryError: Java堆空间位于Java .lang. abstractstringbuilder。在java.lang.AbstractStringBuilder expandCapacity(未知来源)。附加在java.lang.StringBuffer(未知来源)。附加在java.io.StringWriter(未知来源)。在com.ctc. wstc . wstc . wstc . wstc . wstc . wstc . wstc . wstc . wstc . wstc . wstc . wstc . wstc . wstc . wstc . wstcorg.utils.JcoFunctionToXmlTransformer.transform(JcoFunctionToXmlTransformer.java:163

GOAL - To convert the data returned by SAP in form of JcoTable to XML.
PROBLEM - java.lang.OutOfMemoryError: Java heap space.

目标-将SAP返回的数据转换成JcoTable格式的XML。问题- . lang。OutOfMemoryError:Java堆空间。

When the data is very huge, in cases where number of rows exceeds 25,000 getting the above error We are facing the same issue, even while using the Jco API method to convert to XML and also using the external custom code (using Stax API) to read node by node and stream as XML

当数据是非常巨大的,在这种情况下,上面的行数超过25000的错误我们面临同样的问题,即使使用Jco API方法转换为XML并使用外部定制代码(使用Stax API)读节点通过节点和流为XML

6 个解决方案

#1


9  

You can increase the JVM's memory limit by saying:

您可以这样增加JVM的内存限制:

java -Xmx512m ...

or

java -Xmx1024m ...

or whatever size you need.

或者任何你需要的尺寸。

Note that you may need to revise your algorithm to eg, serialize the XML directly to an OutputStream rather than first build a huge memory structure and then serialize that to the OutputStream. (This all depends on the details of what the code is doing.)

注意,您可能需要将您的算法修改成eg,直接将XML序列化为OutputStream,而不是先构建一个巨大的内存结构,然后将其序列化到OutputStream。(这一切都取决于代码的具体操作。)

I had a similar case earlier this year where I was generating XML into a StringBuffer and then writing the StringBuffer into an HTTP response OutputStream. This worked fine until someone asked for 200Mb of XML! I quickly altered the code to generate the XML directly to the OutputStream, saving not only the memory, but CPU.

今年早些时候,我遇到过类似的情况,我将XML生成到StringBuffer,然后将StringBuffer写入HTTP响应OutputStream。在有人要求200Mb的XML之前,这是可行的!我快速修改了代码,将XML直接生成到OutputStream,不仅节省了内存,还节省了CPU。

#2


6  

If you need to parse big XML files (and adding to the Java heap does not always work), you need a SAX parser which allows you to parse the XML stream instead of loading the whole DOM tree into memory.

如果您需要解析大型XML文件(向Java堆添加并不总是有效),您需要一个SAX解析器,它允许您解析XML流,而不是将整个DOM树加载到内存中。

#3


2  

Dó you use substrings of large strings a lot?

你经常使用大字符串的子字符串吗?

attach with jvisualvm to see where your memory goes.

附加jvisualvm以查看内存的去向。

#4


1  

Have you increased the VM options on Java when you run your app? For example:

您在运行应用程序时是否增加了Java上的VM选项?例如:

java -Xmx1024m ...

to give it a gigabyte of heap space.

给它1g的堆空间。

If this is within an application server or Web container you'll usually find such options buried in a config file or startup script.

如果是在应用服务器或Web容器中,通常会在配置文件或启动脚本中找到这些选项。

#5


1  

java -Xmx512m ...

java -Xmx512m……

would help

将有助于

#6


1  

Presumably you are writing the XML document out to a stream, in which case you don't need to hold the entire xml doc in memory as you can write it out to a stream as you construct it.

假定您正在将XML文档写到一条流中,在这种情况下,您不需要将整个XML文档保存在内存中,就像您在构造它时可以将它写到一条流中一样。

#1


9  

You can increase the JVM's memory limit by saying:

您可以这样增加JVM的内存限制:

java -Xmx512m ...

or

java -Xmx1024m ...

or whatever size you need.

或者任何你需要的尺寸。

Note that you may need to revise your algorithm to eg, serialize the XML directly to an OutputStream rather than first build a huge memory structure and then serialize that to the OutputStream. (This all depends on the details of what the code is doing.)

注意,您可能需要将您的算法修改成eg,直接将XML序列化为OutputStream,而不是先构建一个巨大的内存结构,然后将其序列化到OutputStream。(这一切都取决于代码的具体操作。)

I had a similar case earlier this year where I was generating XML into a StringBuffer and then writing the StringBuffer into an HTTP response OutputStream. This worked fine until someone asked for 200Mb of XML! I quickly altered the code to generate the XML directly to the OutputStream, saving not only the memory, but CPU.

今年早些时候,我遇到过类似的情况,我将XML生成到StringBuffer,然后将StringBuffer写入HTTP响应OutputStream。在有人要求200Mb的XML之前,这是可行的!我快速修改了代码,将XML直接生成到OutputStream,不仅节省了内存,还节省了CPU。

#2


6  

If you need to parse big XML files (and adding to the Java heap does not always work), you need a SAX parser which allows you to parse the XML stream instead of loading the whole DOM tree into memory.

如果您需要解析大型XML文件(向Java堆添加并不总是有效),您需要一个SAX解析器,它允许您解析XML流,而不是将整个DOM树加载到内存中。

#3


2  

Dó you use substrings of large strings a lot?

你经常使用大字符串的子字符串吗?

attach with jvisualvm to see where your memory goes.

附加jvisualvm以查看内存的去向。

#4


1  

Have you increased the VM options on Java when you run your app? For example:

您在运行应用程序时是否增加了Java上的VM选项?例如:

java -Xmx1024m ...

to give it a gigabyte of heap space.

给它1g的堆空间。

If this is within an application server or Web container you'll usually find such options buried in a config file or startup script.

如果是在应用服务器或Web容器中,通常会在配置文件或启动脚本中找到这些选项。

#5


1  

java -Xmx512m ...

java -Xmx512m……

would help

将有助于

#6


1  

Presumably you are writing the XML document out to a stream, in which case you don't need to hold the entire xml doc in memory as you can write it out to a stream as you construct it.

假定您正在将XML文档写到一条流中,在这种情况下,您不需要将整个XML文档保存在内存中,就像您在构造它时可以将它写到一条流中一样。