{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Automatic generation of google ngram graphs in Python"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The [Google Ngram Viewer](https://en.wikipedia.org/wiki/Google_Ngram_Viewer) allows you to plot the usage trajectories of words over time.\n",
"\n",
"It has a [convient web-interface](https://books.google.com/ngrams), but for frequent usage you might want to generate word usage plots in python. \n",
""
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"%matplotlib inline\n",
"\n",
"import re\n",
"from ast import literal_eval\n",
"\n",
"import requests\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Define what data you want to extract:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"search_para = dict(\n",
" # coma separeted list of search terms:\n",
" content='climate change, global warming',\n",
" # start year, 1500 is the minimal year for corpus 'eng_2012'\n",
" year_start=1950, \n",
" # end year, 2008 is the max for corpus 'eng_2012'\n",
" year_end=2008, \n",
" # The value of 'case_insensitve' does not matter, remove the parameter for case_sensitive \n",
" case_insensitive='whatever', \n",
" # smooth over x years\n",
" smoothing=3, \n",
" corpus='eng_2012'\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We do some error checking here, since google silently provides truncated data if requested to much:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"if search_para['year_start'] < 1500:\n",
" raise ValueError(\"Start year must be after 1499\")\n",
"if 2008 < search_para['year_end']:\n",
" raise ValueError(\"End year must be befor 2009\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The actual data extraction is done with request:\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"req = requests.get('http://books.google.com/ngrams/graph', params=search_para)\n",
"res_raw = re.findall('var data = (.*?);\\\\n', req.text)\n",
"res_clean = literal_eval(res_raw[0]) # Its a valid data structure at this point\n",
"res_parsed = {content['ngram']: content['timeseries']\n",
" for content in res_clean}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"After the parsing, we can convert the data into a dataframe and at an index:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"freq = pd.DataFrame(res_parsed)\n",
"freq.index = list(range(search_para['year_start'], \n",
" search_para['year_end'] + 1))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Google alreay provides the sum of the different cases of the search term.\n",
"These can be identified with '(All)'. However, these columns are absent if case sensitive data is requested. The following matches both cases:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"totals = freq[[col for col in freq.columns if '(All)' in col]]\n",
"if len(totals) == 0:\n",
" totals = freq"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Lets show the final dataset:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" climate change (All) | \n",
" global warming (All) | \n",
"
\n",
" \n",
" \n",
" \n",
" | 1950 | \n",
" 5.629219e-09 | \n",
" 0.000000e+00 | \n",
"
\n",
" \n",
" | 1951 | \n",
" 5.077090e-09 | \n",
" 1.147430e-10 | \n",
"
\n",
" \n",
" | 1952 | \n",
" 6.680804e-09 | \n",
" 1.863561e-10 | \n",
"
\n",
" \n",
" | 1953 | \n",
" 8.050352e-09 | \n",
" 1.059327e-09 | \n",
"
\n",
" \n",
" | 1954 | \n",
" 1.363747e-08 | \n",
" 1.757183e-09 | \n",
"
\n",
" \n",
"
\n",
"