httpimport

Python's missing feature!

Remote, in-memory Python package/moduleimporting through HTTP/S

A feature that Python2/3misses and has become popular in other languages is the remote loading of packages/modules.

httpimport lets a Python2/3 packages/modules to be imported directly in Python interpreter's process memory, through remote URIs, and more...

Example - In a Nutshell

>>>importhttpimport>>>httpimport.__all__
['HttpImporter', 'add_remote_repo', 'remove_remote_repo', 'remote_repo', 'github_repo', 'bitbucket_repo']

>>>withhttpimport.remote_repo(['package1','package2','package3'], 'http://my-codes.example.com/python_packages'):
... importpackage1
...

>>>withhttpimport.github_repo('operatorequals', 'covertutils', branch=master):
... importcovertutils
... # Also works with 'bitbucket_repo'

>>># A depends to B and B depends to C (A, B, C : Python modules/packages in different domains):>>># A exists in "repo_a.my-codes.example.com"	|>>># B exists in "repo_b.my-codes.example.com" | <-- Different domains>>># C exists in "repo_c.my-codes.example.com" |>>>withhttpimport.remote_repo(['C'], 'http://repo_c.my-codes.example.com/python_packages'):
... withhttpimport.remote_repo(['B'], 'http://repo_b.my-codes.example.com/python_packages'):
... withhttpimport.remote_repo(['A'], 'http://repo_a.my-codes.example.com/python_packages'):
... importA
... # Asks for A, Searches for B, Asks for B, Searches for C, Asks for C --> Resolves --> Imports A>>>

>>>module_object=httpimport.load('package1', 'http://my-codes.example.com/python_packages')
>>>module_object<module'package1'from'http://my-codes.example.com/python_packages/package1/__init__.py'>

Example - The Whole Picture

Using the SimpleHTTPServer, a whole directory can be served through HTTP as follows:

user@hostname:/tmp/test_directory$ ls -R
.:
test_package
./test_package:
__init__.py __init__.pyc module1.py module2.py
user@hostname:/tmp/test_directory$
user@hostname:/tmp/test_directory$ python -m SimpleHTTPServer &
[1] 9565
Serving HTTP on 0.0.0.0 port 8000 ...
user@hostname:/tmp/test_directory$
user@hostname:/tmp/test_directory$
user@hostname:/tmp/test_directory$ curl http://localhost:8000/test_package/module1.py
127.0.0.1 - - [22/Aug/2017 17:42:49] "GET /test_package/module1.py HTTP/1.1" 200 -
def dummy_func() :return'Function Loaded'
class dummy_class :
def dummy_method(self) :return'Class and method loaded'
dummy_str = 'Constant Loaded'
user@hostname:/tmp/test_directory$
user@hostname:/tmp/test_directory$ curl http://localhost:8000/test_package/__init__.py
127.0.0.1 - - [22/Aug/2017 17:45:20] "GET /test_package/__init__.py HTTP/1.1" 200 -
__all__ = ["module1", "module2"]

Using this simple built-in feature of Py2/3, a custom importer can been created, that given a base URL and a list of package names, it fetches and automatically loads all modules and packages to the local namespace.

Usage

Making the HTTP repo

user@hostname:/tmp/test_directory$ ls -R
.:
test_package
./test_package:
__init__.py __init__.pyc module1.py module2.py
user@hostname:/tmp/test_directory$
user@hostname:/tmp/test_directory$ python -m SimpleHTTPServer
Serving HTTP on 0.0.0.0 port 8000 ...

Importing Remotely

add_remote_repo() and remove_remote_repo()

These 2 functions will add and remove to the default sys.meta_path custom HttpImporter objects, given the URL they will look for packages/modules and a list of packages/modules its one can serve.

>>>importtest_packageTraceback (mostrecentcalllast):
File"<stdin>", line1, in<module>ImportError: Nomodulenamedtest_package>>>>>>fromhttpimportimportadd_remote_repo, remove_remote_repo>>># In the given URL the 'test_package/' is available>>>add_remote_repo(['test_package'], 'http://localhost:8000/') # >>>importtest_package>>>>>>remove_remote_repo('http://localhost:8000/')
>>>importtest_package.module1Traceback (mostrecentcalllast):
File"<stdin>", line1, in<module>ImportError: Nomodulenamedmodule1

The remote_repo() context

Adding and removing Remote Repos can be a pain, specially if there are packages that are available in more than one repos. So the with keyword does the trick again:

>>>fromhttpimportimportremote_repo>>>>>>>>>withremote_repo(['test_package'], 'http://localhost:8000/') :
... fromtest_packageimportmodule1
...
>>>>>>fromtest_packageimportmodule2Traceback (mostrecentcalllast):
File"<stdin>", line1, in<module>ImportError: cannotimportnamemodule2>>>module1.dummy_str'Constant Loaded'>>>module1.dummy_func<functiondummy_funcat0x7f7a8a170410>

Reload module (setting httpimport.RELOAD flag):

importimportlibimporthttpimporthttpimport.INSECURE=Truewithhttpimport.remote_repo(['mod'], 'http://localhost:8000/a'):
importmodprint(mod.module_name())
withhttpimport.remote_repo(['mod'], 'http://localhost:8000/b'):
importlib.reload(mod)
importmodprint(mod.module_name())
httpimport.RELOAD=True# Allow reload modulewithhttpimport.remote_repo(['mod'], 'http://localhost:8000/b'):
importlib.reload(mod)
importmodprint(mod.module_name())

The Tiny Test for your amusement

The test.py file contains a minimal test. Try changing working directories and package names and see what happens...

$ python test.py
serving at port 8000
127.0.0.1 - - [22/Aug/2017 17:36:44] code 404, message File not found
127.0.0.1 - - [22/Aug/2017 17:36:44] "GET /test_package/module1/__init__.py HTTP/1.1" 404 -
127.0.0.1 - - [22/Aug/2017 17:36:44] "GET /test_package/module1.py HTTP/1.1" 200 -
Constant Loaded
Function Loaded
Class and method loaded

The Github Use Case!

Such HTTP Servers (serving Python packages in a directory structured way) can be found in the wild, not only created with SimpleHTTPServer. Github repos can serve as Python HTTPS Repos as well!!!

Here is an example with my beloved covertutils project:

>>>>>>importcovertutilsTraceback (mostrecentcalllast):
File"<stdin>", line1, in<module>ImportError: Nomodulenamedcovertutils>>># covertutils is not available through normal import!>>>>>>covertutils_url='https://raw.githubusercontent.com/operatorequals/covertutils/master/'>>>>>>fromhttpimportimportremote_repo>>>>>>withremote_repo(['covertutils'], covertutils_url) :
... importcovertutils
...
>>>printcovertutils.__author__JohnTorakis-operatorequals

The dedicatedgithub_repo() context:

>>>fromhttpimportimportgithub_repo>>>withgithub_repo( 'operatorequals', 'covertutils', ) :
... importcovertutils
...
>>>covertutils.__author__'John Torakis - operatorequals'>>>

What about branches?

>>>fromhttpimportimportgithub_repo>>>withgithub_repo( 'operatorequals', 'covertutils', branch='py3_compatibility' ) :
... importcovertutils
...
>>>covertutils.__author__'John Torakis - operatorequals'>>>

And ad-hoc commits too?

What if you need to stick to a fixed -known to work- commit?

>>>fromhttpimportimportgithub_repo>>>withgithub_repo( 'operatorequals', 'covertutils', commit='cf3f78c77c437edf2c291bd5b4ed27e0a93e6a77' ) :
... importcovertutils
...
>>>covertutils.__author__'John Torakis - operatorequals'>>>

The newer sibling bitbucket_repo() (as of 0.5.9)

>>>withbitbucket_repo('atlassian', 'python-bitbucket', module='pybitbucket'):
... importpybitbucket
...
>>>

Recursive Dependencies

If package A requires module B and A exists in http://example.com/a_repo/, while B exists in http://example.com/b_repo/, then A can be imported using the following technique:

>>>fromhttpimportimportremote_repo>>>withremote_repo(['B'],"http://example.com/b_repo/") :
... withremote_repo(['A'],"http://example.com/a_repo/") :
... importA
... [!] 'B'notfoundinHTTPrepository. MovingtonextFinder.
>>>>>>A<module'A'from'http://example.com/a_repo/A/__init__.py'>>>>B<module'B'from'http://example.com/a_repo/B.py'>>>>

Any combination of packages and modules can be imported this way!

The [!] Warning was emitted by the HttpImporter object created for A, as it couldn't locate B, and passed control to the next Finder object, that happened to be the HttpImporter object created for B!

The load() function (as of 0.5.10)

The load() function was added to make module loading possible without Namespace pollution.

>>>importhttpimport>>>pack1=httpimport.load('random-package','http://localhost:8000/')
>>>pack1<module'random-package'from'http://localhost:8000//random-package/__init__.py'>>>>>>># Trying to load 'os' module from the URL will fail, as it won't delegate to to other Finders/Loaders.>>>httpimport.load('os','http://localhost:8000/')
[!] 'non-existent-package'notfoundinHTTPrepository. MovingtonextFinder.
Traceback (mostrecentcalllast):
File"<stdin>", line1, in<module>File"httpimport.py", line287, inloadraiseImportError("Module '%s' cannot be imported from '%s'"% (module_name, url) )
ImportError: Module'os'cannotbeimportedfrom'http://localhost:8000/'

And no data touches the disk, nor any virtual environment. The import happens just to the running Python process!

Life suddenly got simpler for Python module testing!!!

Imagine the breeze of testing Pull Requests and packages that you aren't sure they will work for you!

Debugging...

>>>fromhttpimportimport*>>>>>>importlogging>>>logging.getLogger('httpimport').setLevel(logging.DEBUG)
>>>>>>withgithub_repo('operatorequals','covertutils') :
... importcovertutils
...
FINDER=================
[!] Searchingcovertutils
[!] PathisNone
[@] CheckingifconnectionisHTTPSsecure>
[@] Checkingifindeclaredremotemodulenames>
[@] Checkingifbuilt-in>
[@] Checkingifitisnamerepetition>
[*]Module/Package'covertutils'canbeloaded!
LOADER=================
[+] Loadingcovertutils
[+] Tryingtoimportaspackagefrom: 'https://raw.githubusercontent.com/operatorequals/covertutils/master//covertutils/__init__.py'
[+] Importing'covertutils'
[+] Readytoexecute'covertutils'code
[+] 'covertutils'importedsuccesfully!
>>>

Beware: Huge Security Implications!

Using the httpimport with HTTP URLs is highly discouraged outside the localhost interface!

As HTTP traffic isn't encrypted and/or integrity checked (unlike HTTPS), it is trivial for a remote attacker to intercept the HTTP responses (via an ARP MiTM probably), and add arbitrary Python code to the downloaded packages/modules. This will directly result in Remote Code Execution to your current user's context! In other words, you get totally F*ed...

Preventing the disaster (setting httpimport.INSECURE flag):

>>>importhttpimport>>>>>># Importing from plain HTTP ...>>>httpimport.load('test_module', 'http://localhost:8000//')
[!] UsingnonHTTPSURLs ('http://localhost:8000//') canbeasecurityhazard!
[-] 'httpimport.INSECUREisnotset! Aborting...
Traceback (mostrecentcalllast):
File"<stdin>", line1, in<module>File"httpimport.py", line302, inloadraiseImportError("Module '%s' cannot be imported from URL: '%s'"% (module_name, url) )
ImportError: Module'test_module'cannotbeimportedfromURL: 'http://localhost:8000/'>>># ... Throws Error!>>>>>># Importing from plain HTTP has to be DELIBERATELY enabled!>>>httpimport.INSECURE=True>>>httpimport.load('test_module', 'http://localhost:8000//')
[!] UsingnonHTTPSURLs ('http://localhost:8000//') canbeasecurityhazard!
<module'test_module'from'http://localhost:8000//test_module.py'>>>># Succeeded!

You have been warned! Use HTTPS URLs with httpimport!

Did I hear you say "Staging protocol for covertutils backdoors"?

Technique documentation on using httpimport to stage covertutils backdoor code, making EXE packed and unreadable code load non-included module dependencies.