Python Scripting

Web Scraping

Web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites. Web scraping software may access the World Wide Web directly using the Hypertext Transfer Protocol, or through a web browser. For this project, we will be using the following python packages: requests, html5lib, and bs4.

###Setting up the project

importrequestsfrombs4importBeautifulSoupimportsqlite3, sqlURL="https://bulbapedia.bulbagarden.net/wiki/List_of_Pok%C3%A9mon_by_National_Pok%C3%A9dex_number"r=requests.get(URL) soup=BeautifulSoup(r.content, 'html5lib')

This code retrieves all of the HTML data from Bulbapedia's list of Pokemon and formats it for us.

So how do we parse through the webpage to only get the data that we need?

pokemon_list= []
table=soup.find('table', attrs= {'align':'center'}) pokemon_info=table.findAll('tr', attrs= {'style': 'background:#FFF'})

And finally, we need to clean up the data. We'll do this iteratively.

forrowinpokemon_info:
pokemon= {'RegionalDex': "", 'NationalDex': "",
'Name': "", 'Primary Type': "", 'Secondary Type': ""}
count=1forindiv_infoinrow.findAll('td'):
ifcount==1:
pokemon['RegionalDex'] =indiv_info.textifcount==2:
pokemon['NationalDex'] =indiv_info.textifcount==4:
pokemon['Name'] =indiv_info.textifcount==5:
pokemon['Primary Type'] =indiv_info.textifcount==6:
pokemon['Secondary Type'] =indiv_info.textcount=count+1pokemon_list.append(pokemon) 

##sqlite3 SQLite is a C library that provides a lightweight disk-based database that doesn’t require a separate server process and allows accessing the database using a nonstandard variant of the SQL query language. sqlite3 is a Python package that allows us to us SQLite's features from within our code.

The first step is to create our database and the table that the Pokemon information will go into.

importsqlite3conn=sqlite3.connect('test.db')
crsr=conn.cursor() sql_command='''CREATE TABLE IF NOT EXISTS pokemon( national_dex VARCHAR(4),regional_dex VARCHAR(4),name VARCHAR(20), primary_type VARCHAR(30), secondary_type VARCHAR(30));'''crsr.execute(sql_command)

Now that we have created the "pokemon" table, we need to insert our data into the table.

definsert_pokemon(pokemon_list):
conn=sqlite3.connect('test.db')
c=conn.cursor()
sql=''' INSERT INTO pokemon(national_dex,regional_dex, name,primary_type,secondary_type) VALUES(?,?,?,?,?) '''forpokemoninpokemon_list:
c.execute(sql, (pokemon['NationalDex'], pokemon['RegionalDex'], pokemon['Name'], pokemon['Primary Type'], pokemon['Secondary Type']))
conn.commit() 

Now that we have all of the individual parts completed, all we need to do is call the insert_pokemon function from our scripting file.

sql.insert_pokemon(pokemon_list)