Collection of utilities for searching the internets.
The idea is, one module containing various utilities and classes for getting information on various things from around the internet as simply as possible. Most classes are WIP and are subject to change and expansion, I hope to add alot of utility functions to make getting important information quicker but still exposing the raw data. Each site is also subject to its own rules and terms of service. Abusing the calls or spamming the sites or harvesting large amounts of data could get you blocked depending on the sites terms. I take no responsibility for that. Just use common sense and be sensible about it.
DuckDuckGo is an alternate search engine focused on privacy.
fromnetcrawlerimport*#Takes optional constructor arg to define the user-agent that the site sees.ddg=DuckDuckGo()
data=ddg.search('python')
#returns a list of search results, each entry is a dict contains keys for description, url, titledata=ddg.get('steve buscemi')
#returns a dict of information about a person, game, movie etc using DDG's Instant Answers APIStartpage is another search engine.
fromnetcrawlerimport*sp=Startpage()
pages=sp.search('python')
#returns a list of search results, each entry is a dict contains keys for description, url, titleimages=sp.images('steve buscemi')
# returns a list of dict entries, same format as normal search, but the urls are images pulled from Image SearchIMGUR is a popular image hosting service.
fromnetcrawlerimport*#Takes required constructor arg to provide your own app id, obtained from registering on the site.img=IMGUR('your-app-id')
data=img.search('steve buscemi')
#returns a dict of search resultsdata=img.getRandom('christopher walken')
#returns a random imagedata=img.gallery('steve buscemi')
#returns a dict of a specific gallery on imgurdata=img.reddit('aww')
#returns a dict of images pulled from redditdata=img.randomReddit('aww')
#returns a random imageimg.upload('/path/to/image')
#uploads an image to your accountUguu.upload('/path/to/file')
#returns a URL to the uploaded file
A knowledge engine that contains answers to many things.
fromnetcrawlerimport*#Takes required constructor arg to provide your own app id, obtained from registering on the site.wf=Wolfram('your-app-id')
print(wf.send('population of uk'))
#returns a plaintext qeury from wolfram alphaimg=wf.image('population of uk'))
#returns a raw image of data from wolframUtilities for interfacing with the Wiki services.
fromnetcrawlerimport*wiki=Wiki()
description=wiki.identify('python')
#shows a description of an entitysummaries=wiki.summarize('python')
#shows a summary of topics from wikimedias OpenSearch APIpages=wiki.query('python')
#searches wikipedia for term, returns a list of found pagesQuick access to the SCP Foundation Wiki.
fromnetcrawlerimport*print(SCPSite.search('001'))
#returns the raw data of the searched SCPCalls to Googles Reverse Image Search
fromnetcrawlerimport*print(ReverseImageSearch().basic('an-image-URL'))
#returns what is in the image, e.i. a persons name#Due to how google handles the service, theres no way to call this directly for files on your hard drive.#A workaround for now would be using the Uguu class in this module to upload the file, get the URL from there, and use that for this.Various access to gamefaq's services.
fromnetcrawlerimport*#games searched are cached for current session to ease load on the site#every call in each game is also cachedgf=Gamefaqs()
deusex=gf.search('deus ex') #takes an optional arg to limit search by console#returns a Game object deusex.news() #gets current news on the gamedeusex.description() #shows the description of the game as it is on gamefaqsdeusex.details() #gets the informationdeusex.trivia() #shows some trivia, is not always the samedeusex.images() #returns a dict of images, with keys for `box` which is the games box arg, and `screen` which is screenshotsdeusex.cheats() #returns a dict of cheats, with keys for author and textfaq=deusex.faqs() #returns a Faqs object, which has its own methodsfaq.text() #returns the raw text body from the faqs pagefaq.find('0.1 Controls') #finds the line containing the text and the following lines until the next breakfaq.find('Augmentation', 5) #finds the first mention of the word and returns that line and the following 5 linesfaq.find('Ocean Lab', 'Area 51') #finds the first mention of the word, and every line after until the second arg is foundfaq.author#string of the author's namefaq.author_url#direct profile link to the authorWrapper for the Pokeapi service, which hosts data for everything you could possibly need to know about pokemon. This is probably the most work-in-progress class. The API is very extensive and has alot of information, so for now i just wrapped the most useful features that I would use personally.
fromnetcrawlerimport*#Takes optional arg for cache location, used for downloading spritesdex=Pokedex()
dex.get('pokemon', 'charmander')
#Raw function for getting the json data of a thing, first arg is the type, second is the search termdex.getTypes('charmander')
#utility to return a list of pokemons types quicklydex.getSprite('charmander')
#downloads a pokemons sprite to the image cachedex.typeEffectiveness('fire')
#returns a dict of type effectiveness chartscharmander=dex.fetchPokemon('charmander')
#returns a POKEMON objectcharmander.name#charmanderspecies=charmander.species#a Species objectcharmander.text() #shows the pokedex flavour text, takes options keyword args for language and game name to findcharmander.evolesFrom() #returns a POKEMON object of this pokemons pre-evolved formcharmander.sprite() #quick utility to download the sprite of this functioncharmander.spriteURL() #returns the URL of the sprite on the API servercharmander.getResists() #returns types that this pokemon resists fullycharmander.getDefends() #returns types that this pokemon takes half damage fromcharmander.getWeakness() #returns types that this pokemon takes supereffective damage from