druid灌库报错-Not enough direct memory

时间:2024-03-15 11:14:00

Druid灌库脚本提交之后,看灌库控制台,报错如下

druid灌库报错-Not enough direct memory

2) Not enough direct memory.  Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes, druid.processing.numThreads, or druid.processing.numMergeBuffers: maxDirectMemory[954,728,448], memoryNeeded[1,677,721,600] = druid.processing.buffer.sizeBytes[335,544,320] * (druid.processing.numMergeBuffers[2] + druid.processing.numThreads[2] + 1)

  at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:110) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.guice.DruidProcessingModule)

 

上面的报错日志非常详细,原因是druid灌库使用的内存超过jvm的直接内存

 

给的解决方案也很详细

要设置jvm的directMemorySize > druid.processing.buffer.sizeBytes[335,544,320] * (druid.processing.numMergeBuffers[2] + druid.processing.numThreads[2] + 1)

 

将middleManager节点上的

jvm的directMemorySize 调大

druid.processing.buffer.sizeBytes 调小

druid.processing.numMergeBuffers 调小

druid.processing.numThreads 调小

意思是jvm的直接内存调大,这个上限是整个物理机的内存上限,不是jvm最大内存

druid灌库的单个缓冲区调小,线程数调小,使用到的合并缓冲区调小

 

为什么调middleManager节点

灌库的indexService服务,由overload节点担任主节点,middleManager节点具体执行灌库任务,所以调节middleManager

 

登录druid集群中的middleManager节点,看到两个配置文件

running.properties

jvm.properties