neo4j服务进程内存一直再增长,时间长了就oom了
发布于 13 天前 作者 xulijian 57 次浏览 来自 问答

问题描述: 数据 list 有154725条关系数据,开始节点和结束都存在
我对15w分批来执行每批次10000条数据,这地方确是每次都新增一个连接,我是为测试每次连接是否正常关闭的问题,测试了半小时所有连接都能正常关闭 问题复现过程: 我每次创建关系后,通过界面把数据删除了,重新启动这个方法进行插入,反复操作10次以后,linux服务端8g内存就满了,一直在fullgc。 附上堆的情况 [root@node1 logs]# jmap -heap 29849 Attaching to process ID 29849, please wait… Debugger attached successfully. Server compiler detected. JVM version is 25.144-b01

using thread-local object allocation. Garbage-First (G1) GC with 23 thread(s)

Heap Configuration: MinHeapFreeRatio = 40 MaxHeapFreeRatio = 70 MaxHeapSize = 8589934592 (8192.0MB) NewSize = 1363144 (1.2999954223632812MB) MaxNewSize = 5150605312 (4912.0MB) OldSize = 5452592 (5.1999969482421875MB) NewRatio = 2 SurvivorRatio = 8 MetaspaceSize = 21807104 (20.796875MB) CompressedClassSpaceSize = 1073741824 (1024.0MB) MaxMetaspaceSize = 17592186044415 MB G1HeapRegionSize = 4194304 (4.0MB)

Heap Usage: G1 Heap: regions = 2048 capacity = 8589934592 (8192.0MB) used = 6181947672 (5895.564720153809MB) free = 2407986920 (2296.4352798461914MB) ** 71.96734277531505% used** G1 Young Generation:

def batchSaveCarImsiR(neo4jConf: mutable.Map[String, String], list: util.ArrayList[Neo4jRelationShip]): Unit = { if (!(list.isEmpty || list.size() == 0)) { val batchCount = list.size()/COMMIT_SIZE val mod = list.size()%COMMIT_SIZE log.info(s"batchCount:$batchCount;mod=$mod,sum=${list.size()}") for(i<-0 until batchCount){ val subList = list.subList(i*COMMIT_SIZE,(i+1)COMMIT_SIZE) log.info(s"$i:${subList.size()}") val sql = s"UNWIND ${subList.toString} as row MATCH (a:car {code:row.startNode}) MATCH(b:imsi {code:row.endNode}) CREATE (a)-[:car_imsi{date:row.date,rate:row.rate,type:row.relationType,merge:row.merge}]->(b)" Neo4jManager.execute(neo4jConf, sql) } if(mod>0){ val subList = list.subList(batchCountCOMMIT_SIZE,list.size()) log.info(s"$mod:${subList.size()}") val sql = s"UNWIND ${subList.toString} as row MATCH (a:car {code:row.startNode}) MATCH(b:imsi {code:row.endNode}) CREATE (a)-[:car_imsi{date:row.date,rate:row.rate,type:row.relationType,merge:row.merge}]->(b)“ Neo4jManager.execute(neo4jConf, sql) } log.info(s"batchSaveImsiMacR:${list.size()}”) } }

连接neo4j执行cql /** * 执行sql * @param conf neo4j配置 * @param sql 批量执行的sql */ def execute(conf:mutable.Map[String,String], sql: String): Unit = { Class.forName(conf(“neo4j.driver”)) val conn = DriverManager.getConnection(conf(“neo4j.url”), conf(“neo4j.user”), conf(“neo4j.password”)) log.debug(conn.getAutoCommit) val stmt = conn.createStatement try { stmt.execute(sql) stmt.close() // conn.commit() conn.close() } catch { case ex: Exception => ex.printStackTrace() throw ex } finally { if(stmt != null) stmt.close() if(conn != null) conn.close() } }image.pngimage.png

3 回复

你这个没有提交吧?

库里面数据都有了,默认jdbc方式自动提交的,conn.getAutoCommit 默认是true 数据也确实进去 内存一直在增长。

问题解决了 换了个思路 UNWIND放弃了 用的纯jdbc的方式来干了 log.info(“start”) val sqls = "MATCH (a:car {code:?}) MATCH(b:imsi {code:?}) CREATE (a)-[:car_imsi{date:?,rate:?,type:?,merge:?}]->(b)" Class.forName(neo4jConf(“neo4j.driver”)) val conn = DriverManager.getConnection(neo4jConf(“neo4j.url”), neo4jConf(“neo4j.user”), neo4jConf(“neo4j.password”)) conn.setAutoCommit(false) log.debug(conn.getAutoCommit) val stmt = conn.prepareStatement(sqls) try { for(i<-0 until list.size()){ val rel = list.get(i) stmt.setString(1,rel.startNode) stmt.setString(2,rel.endNode) stmt.setString(3,rel.date) stmt.setString(4,rel.rate) stmt.setInt(5,rel.relationType) stmt.setInt(6,rel.merge) stmt.addBatch() if((i+1)%COMMIT_SIZE==0){ stmt.executeBatch() conn.commit() stmt.clearBatch() } } if(list.size()%COMMIT_SIZE>0){ stmt.executeBatch() conn.commit() } stmt.close() conn.close() } catch { case ex: Exception => ex.printStackTrace() conn.rollback() throw ex } finally { if(stmt != null) stmt.close() if(conn != null) conn.close() } log.info(“end”) }

回到顶部