加载大量数据批量提交发生 Detected VM stop-the-world pause
发布于 5 年前 作者 bingo 3489 次浏览 来自 问答

USING PERIODIC COMMIT 10000 LOAD CSV WITH HEADERS FROM “file:///loanid.new” AS row merge (node:loanid { loanid:row.node}) set node.loanid=row.node ,node.tcapi=row.p1 ,node.bal=row.p2 我是1万条提交一次。理论不会内存不足赛。当然我可以增大我的heap. 有没有办法或方法避免下面的问题

2019-09-18 23:36:52.899+0800 DEBUG [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {} 2019-09-18 23:36:53.965+0800 DEBUG [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {} 2019-09-18 23:37:01.749+0800 DEBUG [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {} 2019-09-18 23:37:01.849+0800 DEBUG [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {} 2019-09-18 23:37:03.579+0800 DEBUG [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {} 2019-09-18 23:37:13.603+0800 DEBUG [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {} 2019-09-18 23:37:13.703+0800 DEBUG [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {}

7 回复

而且数据量一大(入库数据90m,80万条)。我数据库直接挂掉 heap大小3G

Connection to the database terminated. This can happen due to network instabilities, or due to restarts of the database

可能的解决方法: 1、在:loanid(loanid)上定义索引或唯一性限制; 2、减少提交的规模到2000; 3、修改Cypher: LOAD CSV WITH HEADERS FROM “file:///loanid.new” AS row merge (node:loanid { loanid:row.node}) where id(node) > -1 set …

@graphway 老师你好。我是有唯一索引的。排除方法1.方法2.我试过也存在。不是很明白方法3中 id(node) > -1 起的作用。请教老师指点,谢谢

使用3)有没有效果?

我的版本是3.4.1 内存是64G的 dbms.memory.heap.initial_size=32g dbms.memory.heap.max_size=32g

dbms.memory.pagecache.size=16g

USING PERIODIC COMMIT 5000 LOAD CSV WITH HEADERS FROM “file:///loanid.new” AS row merge (node:loanid { loanid:row.node}) where id(node) > -1 set node.loanid=row.node; Invalid input ‘h’: expected ‘i/I’ (line 4, column 2 (offset: 125)) "where id(node) > -1" 老师应该不支持这个语法吧。Merge和where不能一起用

回到顶部