Excrawl provides a DSL to create web crawler.
If available in Hex, the package can be installed as:
Add excrawl to your list of dependencies in
mix.exs:def deps do [{:excrawl, github: "mlankenau/excrawl"}] end
Ensure excrawl is started before your application:
def application do [applications: [:excrawl]] end
Excrawl is using floki to turn html markup into datastructures. A dsl helps to define what to parse.
Example markup
<html><body><spanid="productTitle">NotMe</span><divid="centerCol"><spanid="productTitle">Tralala</span><spanid="priceblock_ourprice" currency="USD">123</span></div><divclass="product"><spanid="name">Foo</span></div><divclass="product"><spanid="name">Bar</span></div><inputtype="text" name="foo" value="123456789"></body></html>DSL
parser:crawl_my_docdotextname: :title,css: "div #productTitle"textname: :price,css: "div #priceblock_ourprice"endThe DSL will generate a function crawl_my_doc(markup) that
will return
%{title: "Tralala",price: "123"}This example is accessing only content of elements. To access attributes we can change it like
parser:crawl_my_docdotextname: :title,css: "div #productTitle"textname: :price,css: "div #priceblock_ourprice"attrname: :currency,css: "div #priceblock_ourprice",attribute: "currency"end%{title: "Tralala",price: "123",currency: "USD"}Data can be grouped together. All css selectors of group members are relative to the group root element.
parser :crawl_my_doc do
group name: :main, css: "#centerCol" do
text name: :title, css: "#productTitle"
text name: :price, css: "#priceblock_ourprice"
end
end
this will generate
%{main: %{title: "Tralala", price: "123"}}
Groups become handy when we have many of them. Like parsing a table with products we want one map for each product.
<divclass="products"><divclass="product"><spanclass="name">Drill</span><spanclass="price">100<span></div><divclass="product"><spanclass="name">Hammer</span><spanclass="price">20<span></div></div>The parse would look like
parser :parse_groups do
groups name: :products, css: ".product" do
text name: :name, css: ".name"
text name: :price, css: ".price"
end
end
And gives the output
%{
products: [
%{name: "Drill", price: 100},
%{name: "Hammer", price: 20}
]
}