#pydruid pydruid exposes a simple API to create, execute, and analyze Druid queries. pydruid can parse query results into Pandas DataFrame objects for subsequent data analysis -- this offers a tight integration between Druid, the SciPy stack (for scientific computing) and scikit-learn (for machine learning). Additionally, pydruid can export query results into TSV or JSON for further processing with your favorite tool, e.g., R, Julia, Matlab, Excel.

To install:

pipinstallpydruid

Documentation: https://pythonhosted.org/pydruid/.

#examples The following exampes show how to execute and analyze the results of three types of queries: timeseries, topN, and groupby. We will use these queries to ask simple questions about twitter's public data set.

timeseries

What was the average tweet length, per day, surrounding the 2014 Sochi olympics?

frompydruid.clientimport*frompylabimportpltquery=PyDruid(druid_url_goes_here, 'druid/v2')
ts=query.timeseries(
datasource='twitterstream',
granularity='day',
intervals='2014-02-02/p4w',
aggregations={'length': doublesum('tweet_length'), 'count': doublesum('count')},
post_aggregations={'avg_tweet_length': (Field('length') /Field('count'))},
filter=Dimension('first_hashtag') =='sochi2014'
)
df=query.export_pandas()
df['timestamp'] =df['timestamp'].map(lambdax: x.split('T')[0])
df.plot(x='timestamp', y='avg_tweet_length', ylim=(80, 140), rot=20,
title='Sochi 2014')
plt.ylabel('avg tweet length (chars)')
plt.show()

topN

Who were the top ten mentions (@user_name) during the 2014 Oscars?

top=query.topn(
datasource='twitterstream',
granularity='all',
intervals='2014-03-03/p1d', # utc time of 2014 oscarsaggregations={'count': doublesum('count')},
dimension='user_mention_name',
filter=(Dimension('user_lang') =='en') & (Dimension('first_hashtag') =='oscars') &
(Dimension('user_time_zone') =='Pacific Time (US & Canada)') &~(Dimension('user_mention_name') =='No Mention'),
metric='count',
threshold=10
)
df=query.export_pandas()
printdfcounttimestampuser_mention_name013032014-03-03T00:00:00.000ZTheEllenShow1442014-03-03T00:00:00.000ZTheAcademy2212014-03-03T00:00:00.000ZMTV3212014-03-03T00:00:00.000Zpeoplemag4172014-03-03T00:00:00.000ZTHR5162014-03-03T00:00:00.000ZItsQueenElsa6162014-03-03T00:00:00.000Zeonline7152014-03-03T00:00:00.000ZPerezHilton8142014-03-03T00:00:00.000Zrealjohngreen9122014-03-03T00:00:00.000ZKevinSpacey

groupby

What does the social network of users replying to other users look like?

fromigraphimport*fromcairoimport*frompandasimportconcatgroup=query.groupby(
datasource='twitterstream',
granularity='hour',
intervals='2013-10-04/pt12h',
dimensions=["user_name", "reply_to_name"],
filter=(~(Dimension("reply_to_name") =="Not A Reply")) &
(Dimension("user_location") =="California"),
aggregations={"count": doublesum("count")}
)
df=query.export_pandas()
# map names to categorical variables with a lookup tablenames=concat([df['user_name'], df['reply_to_name']]).unique()
nameLookup=dict([pair[::-1] forpairinenumerate(names)])
df['user_name_lookup'] =df['user_name'].map(nameLookup.get)
df['reply_to_name_lookup'] =df['reply_to_name'].map(nameLookup.get)
# create the graph with igraphg=Graph(len(names), directed=False)
vertices=zip(df['user_name_lookup'], df['reply_to_name_lookup'])
g.vs["name"] =namesg.add_edges(vertices)
layout=g.layout_fruchterman_reingold()
plot(g, "tweets.png", layout=layout, vertex_size=2, bbox=(400, 400), margin=25, edge_width=1, vertex_color="blue")