wikitables

Import tables from any Wikipedia article as a dataset in Python

Installing

pip install wikitables

Usage

Importing

Importing all tables from a given article:

fromwikitablesimportimport_tablestables=import_tables('List of cities in Italy') # returns a list of WikiTable objects

To import an article from a different language, simply add the Wikipedia language code as an argument to import_tables. This will also show country names in the given language.

tables=import_tables('İtalya\'daki_şehirler_listesi', 'tr') # returns a list of WikiTable objects

Accessing

Iterate over a table's rows:

print(tables[0].name)
forrowintables[0].rows:
print('{City}: {Area(km2)}'.format(**row))

output:

List of cities in Italy[0]
Milan: 4,450.11
Naples: 3,116.52
Rome: 3,340.41
Turin: 1,328.40
...

Or return the table encoded as JSON:

tables[0].json()

output:

[
{
"City": "Milan",
"Population January 1, 2014": "6,623,798",
"Density(inh./km2)": "1,488",
"Area(km2)": "4,450.11"
},
{
"City": "Naples",
"Population January 1, 2014": "5,294,546",
"Density(inh./km2)": "1,699",
"Area(km2)": "3,116.52"
},
...

Table Head

After import, table column names may been modified by setting a new header:

table.head= [ 'newfield1', 'newfield2', 'newfield3' ]

This change will be recursively reflected on all of a given tables rows.

Commandline

Wikitables also comes with a simple cli tool to fetch and output table json:

# from article name
wikitables 'List of cities in Italy'# from URL
wikitables https://en.wikipedia.org/wiki/Radio_spectrum#ITU

Creating list of DataFrames

fromwikitablesimportimport_tablesimportpandasaspddefget_df_from_table_object(table):
rows= [rowforrowintable.rows]
returnpd.DataFrame(rows)
defget_list_of_df_of_wiki_article(wiki_title):
tables=import_tables(wiki_title)
return [get_df_from_table_object(table) fortableintables]
print(get_list_of_df_of_wiki_article(wiki_title='List of cities in Italy'))

output:

[ Rank City 2011 Census 2020 Estimate Change Region
0 1 Rome 2617175 2856133 9.130379130168986 Lazio
1 2 Milan 1242123 1378689 10.99456334034552 Lombardy
2 3 Naples 962003 959188 -0.2926186300874267 Campania
3 4 Turin 872367 875698 0.38183470947434905 Piedmont
4 5 Palermo 657651 663401 0.8743239195257102 Sicily
.. ... ... ... ... ... ...
139 140 Battipaglia 51133 51005 -0.25032757710284903 Campania
140 141 Rho 50686 50904 0.4300990411553407 Lombardy
141 142 Chieti 54305 50770 -6.509529509253287 Abruzzo
142 143 Scafati 50794 50686 -0.2126235382131747 Campania
143 144 Scandicci 50309 50645 0.6678725476554792 Tuscany
[144 rows x 6 columns]]

Roadmap

Some planned and wishlist features:

• Type guesing from MediaWiki template values