MSConceptGraph
使用 ConceptNet 的概念图#
注意： 原始的 Microsoft Concept Graph API 已不再可用。该笔记本已更新为使用作为替代的 ConceptNet，这是一个免费开放的知识图谱，具有概念之间类似的
is-a关系。
ConceptNet 是一个包含诸如 IsA、PartOf、UsedFor 等关系的大型概念语义网络。它可通过以下方式获得：
- 可下载的数据文件
- REST API（无需 API 密钥）
ConceptNet 统计数据：
- 超过 800 万个节点
- 跨 83 种语言的 2100 多万条边
使用 ConceptNet Web 服务#
ConceptNet 提供了一个 REST API 用于探索概念之间的 is-a（IsA）关系。无需 API 密钥。
以下是调用的示例 URL：https://api.conceptnet.io/query?start=/c/en/microsoft&rel=/r/IsA&limit=10
In [ ]:
import urllib
import json
def http(x):
response = urllib.request.urlopen(x)
data = response.read()
return data.decode('utf-8')
def query(x):
concept = x.lower().replace(' ', '_')
url = "https://api.conceptnet.io/query?start=/c/en/{}&rel=/r/IsA&limit=10".format(
urllib.parse.quote(concept))
try:
result = json.loads(http(url))
except Exception:
return {}
edges = result.get('edges', [])
if not edges:
return {}
total_weight = sum(edge['weight'] for edge in edges)
if total_weight == 0:
return {}
return {edge['end']['label']: edge['weight'] / total_weight for edge in edges}
query('microsoft')让我们尝试使用父概念对新闻标题进行分类。为了获取新闻标题，我们将使用NewsApi.org服务。您需要获得自己的API密钥才能使用该服务——请访问该网站并注册免费开发者计划。
In [20]:
newsapi_key = '<your API key here>'
def get_news(country='us'):
res = json.loads(http("https://newsapi.org/v2/top-headlines?country={0}&apiKey={1}".format(country,newsapi_key)))
return res['articles']
all_titles = [x['title'] for x in get_news('us')+get_news('gb')]In [21]:
all_titles['Covid-19 Live Updates: Vaccines and Boosters News - The New York Times',
'Ukrainians Flee Mariupol as Russian Forces Push to Take Port City - The Wall Street Journal',
'Bond Yields Jump, Stock Futures Rise After Powell Says Fed Is Ready to Be More Aggressive - The Wall Street Journal',
'Putin critic Alexei Navalny found guilty by Russian court - New York Post ',
"Supreme Court nominee Ketanji Brown Jackson will face questions at confirmation hearing's second day - CNN",
'2 teachers killed at Swedish high school, student arrested - ABC News',
'Clues to Covid-19’s Next Moves Come From Sewers - The Wall Street Journal',
'Republicans to roll dice by grilling Jackson over child-pornography sentencing decisions | TheHill - The Hill',
'‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',
'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',
"US stocks whipsawed overnight after Fed Chair Powell's remarks - Fox Business",
"'We've learned absolutely nothing': Tests could again be in short supply if Covid surges - POLITICO",
"Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail",
'China searches for victims, flight recorders after first plane crash in 12 years - Reuters',
'Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters',
'Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español',
'Powers Remain and Threats Lurk as Women’s Sweet 16 Is Set - The New York Times',
'Webb Space Telescope Begins Multi-Instrument Alignment - SciTechDaily',
"UConn vs UCF - NCAA women's tournament second-round highlights - March Madness",
'Bucking Republican Trend, Indiana Governor Vetoes Transgender Sports Bill - The New York Times',
"Maggie Fox dead: Coronation Street and Shameless actress dies after 'sudden accident' - Mirror Online - The Mirror",
'China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent',
'Daniel Morgan murder: damning report condemns Met police - The Guardian',
'What to expect from Rishi Sunak’s Spring Statement - BBC.com',
'UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian',
"Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror",
'Brass Eye’s outtakes show the brutal TV comedy was the tip of an iceberg - The Guardian',
"Vladimir Putin threatens civilians to break Mariupol's spirit - The Times",
'Shell U-turn on Cambo oilfield would threaten green targets, say campaigners - The Guardian',
'St Helens dog attack: Girl aged 17 months killed at home - BBC',
"PlayStation to buy 'Assassin's Creed' veteran Jade Raymond's Haven Studios - NME",
'‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent',
'NASA confirms there are 5,000 planets outside our solar system - Daily Mail',
'Nintendo Switch finally has folders • Eurogamer.net - Eurogamer.net',
'FA to “find a solution” as Liverpool fan group blasts “shambolic” Wembley travel - This Is Anfield',
'Manchester United transfer news LIVE Erik ten Hag latest and Man Utd manager updates - Manchester Evening News',
'Inflation raises cost of UK government borrowing in February; crude oil up again – business live - The Guardian',
'Alexei Navalny: Kremlin critic found guilty of large-scale fraud and contempt of court by Russian court - Sky News',
"UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider",
'Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian']首先，我们希望能够从新闻标题中提取名词。我们将使用 TextBlob 库来实现这一点，该库简化了许多典型的自然语言处理任务。
In [15]:
import sys
!{sys.executable} -m pip install textblob
!{sys.executable} -m textblob.download_corpora
from textblob import TextBlobRequirement already satisfied: textblob in c:\winapp\miniconda3\lib\site-packages (0.17.1)
Requirement already satisfied: nltk>=3.1 in c:\winapp\miniconda3\lib\site-packages (from textblob) (3.5)
Requirement already satisfied: joblib in c:\winapp\miniconda3\lib\site-packages (from nltk>=3.1->textblob) (1.0.1)
Requirement already satisfied: regex in c:\winapp\miniconda3\lib\site-packages (from nltk>=3.1->textblob) (2021.11.10)
Requirement already satisfied: tqdm in c:\winapp\miniconda3\lib\site-packages (from nltk>=3.1->textblob) (4.61.2)
Requirement already satisfied: click in c:\winapp\miniconda3\lib\site-packages (from nltk>=3.1->textblob) (8.0.3)
Requirement already satisfied: colorama in c:\winapp\miniconda3\lib\site-packages (from click->nltk>=3.1->textblob) (0.4.4)
Finished.
[nltk_data] Downloading package brown to
[nltk_data] C:\Users\dmitryso\AppData\Roaming\nltk_data...
[nltk_data] Package brown is already up-to-date!
[nltk_data] Downloading package punkt to
[nltk_data] C:\Users\dmitryso\AppData\Roaming\nltk_data...
[nltk_data] Package punkt is already up-to-date!
[nltk_data] Downloading package wordnet to
[nltk_data] C:\Users\dmitryso\AppData\Roaming\nltk_data...
[nltk_data] Package wordnet is already up-to-date!
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data] C:\Users\dmitryso\AppData\Roaming\nltk_data...
[nltk_data] Package averaged_perceptron_tagger is already up-to-
[nltk_data] date!
[nltk_data] Downloading package conll2000 to
[nltk_data] C:\Users\dmitryso\AppData\Roaming\nltk_data...
[nltk_data] Package conll2000 is already up-to-date!
[nltk_data] Downloading package movie_reviews to
[nltk_data] C:\Users\dmitryso\AppData\Roaming\nltk_data...
[nltk_data] Package movie_reviews is already up-to-date!
In [22]:
w = {}
for x in all_titles:
for n in TextBlob(x).noun_phrases:
if n in w:
w[n].append(x)
else:
w[n]=[x]
{ x:len(w[x]) for x in w.keys()}{'covid-19 live updates': 1,
'vaccines': 1,
'boosters': 1,
'york': 4,
'ukrainians flee mariupol': 1,
'forces push': 1,
'port city': 1,
'wall street journal': 3,
'bond yields': 1,
'futures rise': 1,
'powell says fed': 1,
'ready': 1,
'be': 1,
'aggressive': 1,
'putin': 3,
'alexei navalny': 2,
'russian': 2,
'supreme court nominee': 1,
'ketanji brown jackson': 1,
"confirmation hearing 's": 1,
'cnn': 1,
'swedish': 1,
'high school': 1,
'abc': 1,
'clues': 1,
'covid-19': 1,
'’ s': 2,
'moves': 1,
'sewers': 1,
'roll dice': 1,
'jackson': 1,
'decisions |': 1,
'thehill': 1,
'clear': 2,
'chemical weapons': 2,
'ukraine': 3,
'claims president': 2,
'biden': 2,
'nasa': 2,
'solar system': 2,
'daily mail': 3,
'us stocks': 1,
'fed chair powell': 1,
"'s remarks": 1,
'fox': 1,
"'we 've": 1,
'tests': 1,
'covid': 1,
'politico': 1,
'duchess': 1,
'cambridge': 1,
'swaps khaki jungle gear': 1,
'vampire': 1,
'wife': 1,
'belize': 1,
'china': 2,
'flight recorders': 1,
'plane crash': 1,
'reuters': 2,
'russian oligarch': 1,
'abramovich': 1,
'live': 1,
'russia': 2,
'stops talks': 1,
'japan': 1,
'español': 1,
'powers remain': 1,
'threats lurk': 1,
'set': 1,
'webb': 1,
'telescope begins multi-instrument alignment': 1,
'scitechdaily': 1,
'uconn': 1,
'ucf': 1,
'ncaa': 1,
"women 's tournament second-round highlights": 1,
'march madness': 1,
'bucking republican trend': 1,
'indiana': 1,
'vetoes transgender': 1,
'bill': 1,
'maggie fox': 1,
'coronation': 1,
'shameless': 1,
"'sudden accident": 1,
'mirror online': 1,
'mirror': 2,
'plane crash –': 1,
'search': 1,
'moment flight': 1,
'daniel morgan': 1,
'report condemns': 1,
'met': 1,
'guardian': 6,
'rishi sunak': 1,
'’ s spring': 1,
'statement': 1,
'bbc.com': 1,
'uk': 3,
'ireland': 1,
'euro': 1,
'vladimir putin': 2,
"'s 'lover": 1,
'brass eye': 1,
'’ s outtakes': 1,
'brutal tv comedy': 1,
'threatens civilians': 1,
'mariupol': 1,
"'s spirit": 1,
'shell u-turn': 1,
'cambo': 1,
'green targets': 1,
'st helens': 1,
'dog attack': 1,
'girl': 1,
'bbc': 1,
'playstation': 1,
"'assassin 's": 1,
'creed': 1,
'jade raymond': 1,
'haven studios': 1,
'nme': 1,
'nintendo switch': 1,
'folders •': 1,
'eurogamer.net': 2,
'fa': 1,
'solution ”': 1,
'liverpool': 1,
'fan group blasts “ shambolic ”': 1,
'wembley': 1,
'anfield': 1,
'manchester': 1,
'live erik': 1,
'hag': 1,
'utd': 1,
'manager updates': 1,
'manchester evening': 1,
'inflation': 1,
'government borrowing': 1,
'february': 1,
'crude oil': 1,
'– business': 1,
'kremlin': 1,
'large-scale fraud': 1,
'sky': 1,
'natural gas': 1,
'gazprom': 1,
'retail unit': 1,
'insider': 1,
'zaghari-ratcliffe': 1,
'hunt': 1,
'iran': 1,
'debt payment': 1}我们可以看到，名词并没有给我们带来大型的主题组。让我们用从概念图中获得的更通用的术语替换名词。这将花费一些时间，因为我们对每个名词短语都在执行REST调用。
In [23]:
w = {}
for x in all_titles:
for noun in TextBlob(x).noun_phrases:
terms = query(noun)
for term in [u for u in terms.keys() if terms[u]>0.1]:
if term in w:
w[term].append(x)
else:
w[term]=[x]In [24]:
{ x:len(w[x]) for x in w.keys() if len(w[x])>3}{'city': 9,
'brand': 4,
'place': 9,
'town': 4,
'factor': 4,
'film': 4,
'nation': 11,
'state': 5,
'person': 4,
'organization': 5,
'publication': 10,
'market': 5,
'economy': 4,
'company': 6,
'newspaper': 6,
'relationship': 6}In [27]:
print('\nECONOMY:\n'+'\n'.join(w['economy']))
print('\nNATION:\n'+'\n'.join(w['nation']))
print('\nPERSON:\n'+'\n'.join(w['person']))
ECONOMY:
China searches for victims, flight recorders after first plane crash in 12 years - Reuters
Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español
China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent
UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider
NATION:
‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent
Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail
China searches for victims, flight recorders after first plane crash in 12 years - Reuters
Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español
Live updates: Russia stops talks with Japan over sanctions - The Associated Press - en Español
China plane crash – live: Search for survivors continues as witness describes moment flight fell from sky - The Independent
UK and Republic of Ireland in line to host Euro 2028 after no one else bids - The Guardian
Friends beg Vladimir Putin's 'lover' to persuade him to end Ukraine invasion - The Mirror
‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent
UK prepares to nationalize Russia natural gas giant Gazprom's retail unit - Business Insider
Zaghari-Ratcliffe: Hunt calls for inquiry into delay over Iran debt payment - The Guardian
PERSON:
‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent
Duchess of Cambridge swaps khaki jungle gear for Vampire's Wife dress on Belize trip - Daily Mail
Second superyacht linked to Russian oligarch Abramovich docks in Turkey - Reuters
‘Clear sign’ Putin considering using chemical weapons in Ukraine, claims President Biden - The Independent
免责声明：
本文件由 AI 翻译服务 Co-op Translator 翻译而成。尽管我们努力确保翻译的准确性，但请注意，自动翻译可能存在错误或不准确之处。原始语言版本的文件应被视为权威来源。对于重要信息，建议使用专业人工翻译。因使用本翻译内容所引起的任何误解或曲解，我们概不负责。