iparse

iparse is a Python package for parsing HTML to structured data in an easy way with as little code as possible.

It aims to make the process of parsing HTML quick and easy!

iparse highlights:

• mainly code with YAML
• only refine raw HTML info with fewest python code
• lot's HTML layout changes, only YAML will be involved

Installation

pip install iparse

A Simple Example

for HTML page: i.e. lovely xkcd python

to get the structured data all you need are

• create a class inherit from IParser
• write a YAML config file represents all locators

create xkcd_353.py

xkcd_353.py will go through the startup_dir, look for a file named as the snake_case of the ClassName without suffix:Parser, so XkcdParser will be xkcd.yaml

frompathlibimportPathfromiparse._parseimportIParser, RsvWordsHOME_DIR=Path(__file__).parents[0]
classXkcdParser(IParser):
def__init__(self, file_name, is_test_mode=False, **kwargs):
kwargs['startup_dir'] =kwargs.get('startup_dir', HOME_DIR)
super().__init__(file_name, is_test_mode=is_test_mode, **kwargs)
if__name__=="__main__":
xkcd=XkcdParser(file_name=HOME_DIR/'xkcd_python_353.htm')
xkcd.do_parse()
print(xkcd.data)

create a file named xkcd.yaml

you can use any locator that is supported, but css selector is recommended

page:
# css_selector of title: head>titletitle: head>title# css_selector: div#footnotefootnote: div#footnote# css_selector: div#licenseTextlicense: div#licenseText

the output parsed data

the parsed data xkcd.data is dict, but you can also use it with xkcd.data_as_yaml/xkcd.data_as_json

yaml output

page:
footnote: "xkcd.com is best viewed with Netscape Navigator 4.0 or below on a Pentium\\ 3\xB11 emulated in Javascript on an Apple IIGSat a screen resolution of 1024x1.\\ Please enable your ad blockers, disable high-heat drying, and remove your devicefrom\\ Airplane Mode and set it to Boat Mode. For security reasons, please leave caps\\ lock on while browsing."license: ' This work is licensed under a Creative Commons Attribution-NonCommercial 2.5 License. This means you''re free to copy and share these comics (but not to sell them). More details.'title: 'xkcd: Python'

json output

{
"page": {
"footnote": "xkcd.com is best viewed with Netscape Navigator 4.0 or below on a Pentium 3\u00b11 emulated in Javascript on an Apple IIGSat a screen resolution of 1024x1. Please enable your ad blockers, disable high-heat drying, and remove your devicefrom Airplane Mode and set it to Boat Mode. For security reasons, please leave caps lock on while browsing.",
"license": "\n\nThis work is licensed under a\nCreative Commons Attribution-NonCommercial 2.5 License.\n\nThis means you're free to copy and share these comics (but not to sell them). More details.\n",
"title": "xkcd: Python"
}
}

Details

# all settings added to __raw, will be kept as it added__raw:
site_url: https://xkcd.com/page:
# if not _locator supplied will reuse parent soup# page has no parent soup, so use default root souptitle: head>titlefootnote: div#footnotelicense:
_locator: div#licenseText# strip blank with true, but also can specified a str_striped: truetop_container:
# we set a _locator here, all sub-nodes will select within top_container_locator: div#topContainertop_left:
# _index:~ means None, so we can use whole list_index: ~_locator: div#topLeft>ul>li>a# if non-reserved key set to ~, means use parent soup, and use its text# this is a convenient way to get textmenu_text: ~menu_url:
# when other attributes exist, no need to add _locator to use its parent soup_attr: href# if we need some extra work on _attr, goes with two ways# 1. `_attr_refine: true` will auto generate => _refine_menu_url_href# the rule of auto-generator is _refine_<key_name>_<attr_value># 2. `_attr_refine: _a_valid_method_name`_attr_refine: truetop_right:
_locator: div#topRightmasthead:
# two way to get more than one attributes on a element# e.g. image.src/.alt# way1: if all src/alt need refine, this will treat attrs as listimage_1:
_attr:
- src
- alt_attr_refine: true_locator: &LOGO_IMG span>a>img# way2: not all src/alt need refine, this will treat attrs as dictimage_2:
_locator: *LOGO_IMGsrc:
_attr: src# only set _attr_refine to src# 1. _attr_refine: true => _refine_src_src# 2. _attr_refine: _refine_image_1_src to reuse exists method_attr_refine: _refine_image_1_srcalt:
_attr: altslogan: span#slogan

more

please check the tests/ for more infomation.