使用 APOC NLP 进行实体提取
实体提取简介
实体提取从非结构化文本中提取并返回该文本中包含的命名实体列表。
AWS、GCP 和 Azure 分别提供 NLP API,这些 API 由 apoc.nlp 过程进行封装。
导入文本文档
我们将从 卫报足球 RSS 源 导入一些文本文档。我们可以为此使用 apoc.load.xml,如下面的查询所示
CALL apoc.load.xml("https://www.theguardian.com/football/rss", "rss/channel/item")
YIELD value
WITH [child in value._children WHERE child._type = "title" | child._text][0] AS title,
[child in value._children WHERE child._type = "link" | child._text][0] AS guid,
apoc.text.regreplace(
[child in value._children WHERE child._type = "description" | child._text][0],
'<[^>]*>',
' '
) AS body,
[child in value._children WHERE child._type = "category" | child._text] AS categories
MERGE (a:Article {id: guid})
SET a.body = body, a.title = title
WITH a, categories
CALL {
WITH a, categories
UNWIND categories AS category
MERGE (c:Category {name: category})
MERGE (a)-[:IN_CATEGORY]->(c)
RETURN count(*) AS categoryCount
}
RETURN a.id AS articleId, a.title AS title, categoryCount;
我们可以在下图中看到导入的图形的 Neo4j Browser 可视化效果
构建实体图谱
现在我们要构建实体图谱。我们将结合使用 apoc.periodic.iterate 和特定于云的流处理过程来:
-
为每篇文章提取实体,并过滤特定实体类型
-
为每个实体创建一个节点
-
创建一个从每个实体到该文章的关系
我们可以使用以下查询来完成此操作
CALL apoc.periodic.iterate(
"MATCH (a:Article) WHERE not(exists(a.processed)) RETURN a",
"CALL apoc.nlp.aws.entities.stream([item in $_batch | item.a], {
key: $awsApiKey,
secret: $awsApiSecret,
nodeProperty: 'body'
})
YIELD node AS a, value
SET a.processed = true
WITH a, value
UNWIND value.entities AS entity
WITH a, entity
WHERE entity.type IN ['COMMERCIAL_ITEM', 'PERSON', 'ORGANIZATION', 'LOCATION', 'EVENT']
CALL apoc.merge.node(
['Entity', apoc.text.capitalize(toLower(entity.type))],
{name: entity.text}, {}, {}
)
YIELD node AS e
MERGE (a)-[entityRel:AWS_ENTITY]->(e)
SET entityRel.score = entity.score
RETURN count(*)",
{ batchMode: "BATCH_SINGLE",
batchSize: 25,
params: {awsApiKey: $awsApiKey, awsApiSecret: $awsApiSecret}}
)
YIELD batches, total, timeTaken, committedOperations, errorMessages, batch, operations
RETURN batches, total, timeTaken, committedOperations, errorMessages, batch, operations;
| batches | 总计 | 耗时 | 已提交的操作 | errorMessages | batch | 操作 |
|---|---|---|---|---|---|---|
3 |
59 |
2 |
59 |
{} |
{total: 3, committed: 3, failed: 0, errors: {}} |
{total: 59, committed: 59, failed: 0, errors: {}} |
CALL apoc.periodic.iterate(
"MATCH (a:Article) WHERE not(exists(a.processed)) RETURN a",
"CALL apoc.nlp.gcp.entities.stream([item in $_batch | item.a], {
key: $gcpApiKey,
nodeProperty: 'body'
})
YIELD node AS a, value
SET a.processed = true
WITH a, value
UNWIND value.entities AS entity
WITH a, entity
WHERE entity.type IN ['PERSON', 'LOCATION', 'ORGANIZATION', 'EVENT']
CALL apoc.merge.node(
['Entity', apoc.text.capitalize(toLower(entity.type))],
{name: entity.name}, {}, {}
)
YIELD node AS e
MERGE (a)-[entityRel:GCP_ENTITY]->(e)
SET entityRel.score = entity.score
RETURN count(*)",
{ batchMode: "BATCH_SINGLE",
batchSize: 25,
params: {gcpApiKey: $gcpApiKey}}
)
YIELD batches, total, timeTaken, committedOperations, errorMessages, batch, operations
RETURN batches, total, timeTaken, committedOperations, errorMessages, batch, operations;
| batches | 总计 | 耗时 | 已提交的操作 | errorMessages | batch | 操作 |
|---|---|---|---|---|---|---|
3 |
59 |
46 |
59 |
{} |
{total: 3, committed: 3, failed: 0, errors: {}} |
{total: 59, committed: 59, failed: 0, errors: {}} |
CALL apoc.periodic.iterate(
"MATCH (a:Article) WHERE not(exists(a.processed)) RETURN a",
"CALL apoc.nlp.azure.entities.stream([item in $_batch | item.a], {
key: $azureApiKey,
url: $azureApiUrl,
nodeProperty: 'body'
})
YIELD node AS a, value
SET a.processed = true
WITH a, value
UNWIND value.entities AS entity
WITH a, entity
WHERE entity.type IN ['Person', 'Organization', 'Location', 'Event']
CALL apoc.merge.node(
['Entity', apoc.text.capitalize(toLower(entity.type))],
{name: entity.name}, {}, {}
)
YIELD node AS e
MERGE (a)-[entityRel:AZURE_ENTITY]->(e)
SET entityRel.score = entity.score
RETURN count(*)",
{ batchMode: "BATCH_SINGLE",
batchSize: 25,
params: {azureApiUrl: $azureApiUrl, azureApiKey: $azureApiKey}}
)
YIELD batches, total, timeTaken, committedOperations, errorMessages, batch, operations
RETURN batches, total, timeTaken, committedOperations, errorMessages, batch, operations;
| batches | 总计 | 耗时 | 已提交的操作 | errorMessages | batch | 操作 |
|---|---|---|---|---|---|---|
3 |
59 |
3 |
59 |
{} |
{total: 3, committed: 3, failed: 0, errors: {}} |
{total: 59, committed: 59, failed: 0, errors: {}} |
查询实体图谱
既然我们已经有了实体,是时候查询实体图谱了。让我们首先返回每篇文章的实体,如下面的查询所示
MATCH (a:Article)
RETURN a.title AS title, [(a)-[:AWS_ENTITY]->(entity) | entity.name] AS entities
LIMIT 5;
| 标题 | entities |
|---|---|
"曼联的卡瓦尼(Edinson Cavani)为 Instagram 上的‘种族主义’帖子道歉" |
["卡瓦尼", "足总", "埃丁森·卡瓦尼", "南安普顿", "曼联", "英格兰足球总会"] |
"对脑震荡替补的需求 – 足球周刊" |
["Faye Carruthers", "Soundcloud", "Mixcloud", "阿森纳", "Barry Glendenning", "Facebook", "Ewan Murray", "劳尔·希门尼斯", "播客", "狼队", "大卫·路易斯", "Twitter", "Acast", "Stitcher", "南安普顿", "Audioboom", "曼联", "Max Rushden", "Lars Sivertsen"] |
"列侬(Lennon)为凯尔特人未能保持国内统治地位而倒下" |
["流浪者", "莱斯特城", "布伦丹·罗杰斯", "郡", "凯尔特人", "托尼·莫布雷", "罗斯", "戈登·斯特拉坎", "尼尔·列侬", "马丁·奥尼尔", "Easy Street", "联赛杯", "罗尼·戴拉"] |
"那不勒斯通过在对阵罗马的比赛中重现马拉多纳的魔力来向他致敬 | Nicky Bandini" |
["圣保罗球场", "洛伦佐·因西涅", "欧洲", "那不勒斯", "罗马", "利昂内尔·梅西", "那不勒斯市", "迭戈·马拉多纳", "Curva"] |
"脑震荡替补试验可能在明年初在英超联赛开始" |
["大卫·路易斯试验", "大卫·路易斯", "劳尔·希门尼斯", "墨西哥", "卫报", "阿森纳", "狼队", "英超联赛", "Daniela"] |
MATCH (a:Article)
RETURN a.title AS title, [(a)-[:GCP_ENTITY]->(entity) | entity.name] AS entities
LIMIT 5;
| 标题 | entities |
|---|---|
"曼联的卡瓦尼(Edinson Cavani)为 Instagram 上的‘种族主义’帖子道歉" |
["南安普顿", "问候", "俱乐部", "前锋", "事件", "曼联", "胜利", "身体", "乌拉圭人", "足总", "埃丁森·卡瓦尼", "朋友", "英格兰足球总会"] |
"对脑震荡替补的需求 – 足球周刊" |
["狼队", "Acast", "Barry Glendenning", "Apple", "Ewan Murray", "胜利", "阿森纳", "Soundcloud", "Faye Carruthers", "Max Rushden", "劳尔·希门尼斯", "Audioboom", "Lars Sivertsen", "大卫·路易斯", "曼联"] |
"列侬(Lennon)为凯尔特人未能保持国内统治地位而倒下" |
["布伦丹·罗杰斯", "俱乐部", "赛季", "酒吧", "Easy Street", "流浪者", "莱斯特", "戈登·斯特拉坎", "罗尼·戴拉", "种族", "联赛杯", "失败", "城市经理", "球迷", "马丁·奥尼尔的", "凯尔特人", "托尼·莫布雷", "池塘", "尼尔·列侬", "罗斯郡"] |
"那不勒斯通过在对阵罗马的比赛中重现马拉多纳的魔力来向他致敬 | Nicky Bandini" |
["球迷", "洛伦佐·因西涅", "圣保罗球场", "那不勒斯市", "那不勒斯", "家庭", "迭戈·马拉多纳", "罗马", "胜利", "死亡", "利昂内尔·梅西", "欧洲", "球员"] |
"脑震荡替补试验可能在明年初在英超联赛开始" |
["俱乐部", "冲突", "Daniela", "劳尔·希门尼斯", "手术", "英超联赛", "大卫·路易斯", "球队", "阿森纳", "恢复", "卫报", "前锋", "墨西哥", "狼队", "规则变更", "大卫·路易斯试验"] |
MATCH (a:Article)
RETURN a.title AS title, [(a)-[:AZURE_ENTITY]->(entity) | entity.name] AS entities
LIMIT 5;
| 标题 | entities |
|---|---|
"曼联的卡瓦尼(Edinson Cavani)为 Instagram 上的‘种族主义’帖子道歉" |
["前锋", "埃丁森·卡瓦尼", "曼联足球俱乐部", "乌拉圭国家足球队", "英格兰足球总会", "南安普顿足球俱乐部"] |
"对脑震荡替补的需求 – 足球周刊" |
["阿森纳足球俱乐部", "AudioBoom", "Stitcher Radio", "Max Rushden", "Lars Sivertsen", "曼联足球俱乐部", "SoundCloud", "劳尔·希门尼斯", "Acast", "Facebook", "Mixcloud", "Barry Glendenning", "南安普顿费率", "Ewan Murray", "Twitter", "南安普顿足球俱乐部", "大卫·路易斯", "Apple Podcasts", "Faye Carruthers"] |
"列侬(Lennon)为凯尔特人未能保持国内统治地位而倒下" |
["马丁·奥尼尔", "莱斯特城足球俱乐部", "托尼·莫布雷", "罗尼·戴拉", "流浪者足球俱乐部", "戈登·斯特拉坎", "联赛杯", "尼尔·列侬", "罗斯郡足球俱乐部", "英格兰联赛杯", "凯尔特人足球俱乐部", "布伦丹·罗杰斯"] |
"那不勒斯通过在对阵罗马的比赛中重现马拉多纳的魔力来向他致敬 | Nicky Bandini" |
["那不勒斯市", "洛伦佐·因西涅", "迭戈·马拉多纳", "罗马", "欧洲", "利昂内尔·梅西", "利昂内尔·梅西", "圣保罗球场", "那不勒斯足球俱乐部", "圣保罗球场"] |
"脑震荡替补试验可能在明年初在英超联赛开始" |
["英超联赛", "墨西哥", "Daniela", "卫报", "劳尔·希门尼斯", "狼队足球俱乐部", "大卫·路易斯", "阿森纳足球俱乐部", "卫报"] |
我们还可以利用文章对共有的实体来确定文章的相似度。如果我们想查找与加里·莱因克尔(Gary Lineker)关于马拉多纳的视频相似的文章,我们可以编写以下查询
MATCH (a1:Article {title: "Gary Lineker: Maradona was 'like a messiah' in Argentina – video"})
MATCH (a1:Article)-[:AWS_ENTITY]-(entity)<-[:AWS_ENTITY]-(a2:Article)
RETURN a2.title AS otherArticle, collect(entity.name) AS entities
ORDER BY size(entities) DESC
LIMIT 5;
| 其他文章 | entities |
|---|---|
"缅怀迭戈·马拉多纳:足球传奇去世,享年 60 岁 – 视频讣告" |
["世界杯", "迭戈·马拉多纳", "布宜诺斯艾利斯", "那不勒斯", "马拉多纳", "阿根廷", "巴塞罗那"] |
"经典 YouTube | 迭戈·阿曼多·马拉多纳,冷静的射门和足球经理的孩子们" |
["英格兰", "世界杯", "迭戈·马拉多纳", "阿根廷", "马拉多纳", "莱因克尔", "加里·莱因克尔"] |
"阿根廷和那不勒斯的球迷哀悼迭戈·马拉多纳的去世 – 视频" |
["世界杯", "迭戈·马拉多纳", "布宜诺斯艾利斯", "那不勒斯", "阿根廷", "马拉多纳"] |
"天才的重负:马拉多纳提醒我们年轻成名带来的问题 | Vic Marks" |
["世界杯", "迭戈·马拉多纳", "墨西哥", "马拉多纳", "阿根廷"] |
"向迭戈·马拉多纳致敬 – 足球周刊" |
["世界杯", "迭戈·马拉多纳", "布宜诺斯艾利斯", "墨西哥", "Twitter"] |
MATCH (a1:Article {title: "Gary Lineker: Maradona was 'like a messiah' in Argentina – video"})
MATCH (a1:Article)-[:GCP_ENTITY]-(entity)<-[:GCP_ENTITY]-(a2:Article)
RETURN a2.title AS otherArticle, collect(entity.name) AS entities
ORDER BY size(entities) DESC
LIMIT 5;
| 其他文章 | entities |
|---|---|
"缅怀迭戈·马拉多纳:足球传奇去世,享年 60 岁 – 视频讣告" |
["巴塞罗那", "阿根廷", "那不勒斯", "布宜诺斯艾利斯"] |
"经典 YouTube | 迭戈·阿曼多·马拉多纳,冷静的射门和足球经理的孩子们" |
["阿根廷", "英格兰", "加里·莱因克尔"] |
"迭戈·马拉多纳的私人医生否认对死亡负有责任" |
["家", "足球运动员", "布宜诺斯艾利斯"] |
"不老的伊布拉希莫维奇继续为米兰处理事务 | Nicky Bandini" |
["家", "那不勒斯"] |
"‘他把我们都带到了天堂’:足球迷对迭戈·马拉多纳去世的反应" |
["阿根廷", "布宜诺斯艾利斯"] |
MATCH (a1:Article {title: "Gary Lineker: Maradona was 'like a messiah' in Argentina – video"})
MATCH (a1:Article)-[:AZURE_ENTITY]-(entity)<-[:AZURE_ENTITY]-(a2:Article)
RETURN a2.title AS otherArticle, collect(entity.name) AS entities
ORDER BY size(entities) DESC
LIMIT 5;
| 其他文章 | entities |
|---|---|
"缅怀迭戈·马拉多纳:足球传奇去世,享年 60 岁 – 视频讣告" |
["巴塞罗那足球俱乐部", "马拉多纳", "那不勒斯足球俱乐部", "那不勒斯", "阿根廷", "布宜诺斯艾利斯", "迭戈·马拉多纳"] |
"经典 YouTube | 迭戈·阿曼多·马拉多纳,冷静的射门和足球经理的孩子们" |
["加里·莱因克尔", "阿根廷国家足球队", "谢周三足球俱乐部", "英格兰", "英格兰国家足球队", "阿根廷", "迭戈·马拉多纳"] |
"向迭戈·马拉多纳致敬 – 足球周刊" |
["谢周三足球俱乐部", "马拉多纳", "墨西哥", "Twitter", "布宜诺斯艾利斯", "迭戈·马拉多纳"] |
"阿根廷和那不勒斯的球迷哀悼迭戈·马拉多纳的去世 – 视频" |
["马拉多纳", "那不勒斯足球俱乐部", "阿根廷", "布宜诺斯艾利斯", "迭戈·马拉多纳"] |
"正是马拉多纳的蔑视最激励了我 | Kenan Malik" |
["阿根廷国家足球队", "英格兰", "英格兰国家足球队", "迭戈·马拉多纳"] |
这些文章共有的某些实体并无太大实际意义(例如“家”或“Twitter”),但总体而言,这些文章是加里·莱因克尔文章“推荐文章”部分的良好候选项。
如果我们想进一步过滤提取出的实体,我们可以尝试仅包含那些有维基百科条目的实体。有关该方法的更多详细信息,请参阅 教程:使用 NLP 和本体构建知识图谱。