Self-Modification At RunTime (SMART): A Framework for Metaprogramming with Large Language Models for Adaptable and Autonomous Software

Setup instructions

1. Create .env for your environment variables. See .env.example.
2. Add your API key
3. python setup.py install
4. pre-commit install

To run all unit tests.

Options:

• -s shows print statements
• -v indicates when a tests passes
• --durations=0 displays a list of n slowest tests at the end of the test session

pytest -s -v --durations=0

To run all tests in a specific file:

pytest <file name>

To run a specific unit test:

pytest -k <test name>

4. Linting and VSCode

To setup linting with flake8 and auto formatting with black: 4.1. Create the subdirectory .vscode in the root directory 4.2. Add the following settings.json file:

{
"editor.formatOnSave": true,
"python.formatting.provider": "none",
"python.formatting.blackArgs": [
"--line-length",
"100"
],
"python.linting.enabled": true,
"python.linting.lintOnSave": true,
"python.linting.flake8Enabled": true,
"python.linting.flake8Args": [
"--line-length",
"100"
],
"[python]": {
"editor.codeActionsOnSave": {
"source.organizeImports": true
},
"editor.defaultFormatter": "ms-python.black-formatter"
}
}

---

Using the generative package

Setup

1. OPENAI_API_KEY must be set in .env or in your terminal OPENAI_API_KEY=your-api-key-here
2. Define your custome LLM solution. Then import the decorators and your llm function and pass it to the decorator.
3. Use as follows:

Bring your own model

Create a custom model class with a generate() function that takes a prompt and returns a string.

All functionality in the generative package expects a model that inherits from the abstract class AbstractGenerativeModel. generate() must be marked with the @classmethod decorator.

fromgenerative.metaclassesimportAbstractGenerativeModelclassLLM(AbstractGenerativeModel):
@classmethoddefgenerate(self, prompt: str) ->str:
# Must implement generate()

Function decorators

@adapt decorator enables your model to control the behavior of the decorated function at run-time. The model could check for semantic errors and change based on input.

fromgenerative.decoratorimportadapt@adapt(model=LLM)deffunc(a, b):
prompt=""" Write a complete python 3 function, including the header and return statement that computes the N-th Fibonacci number. """assertfunc(8) ==21

@catch decorator enables your model to control the behavior of the decorated function at run-time only when an exception is thrown.

fromgenerative.decoratorimportcatch@catch(model=LLM)deffunc(a, b):
raiseException("Original function exception")

@stack_trace decorator augments stack traces with human-readable summaries and steps to debug or fix the issue.

fromgenerative.decoratorimportstack_trace@stack_trace(model=LLM)deffunkodunko():
items= [1, 2, 3]
returnitems[5]

Example output:

tests/test_adapt_decorator.py Traceback (most recent call last):
File "/Users/justin/Documents/dev/personal/ml/dynamic-mp-llm/meta.py", line 167, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/justin/Documents/dev/personal/ml/dynamic-mp-llm/tests/test_adapt_decorator.py", line 100, in funkodunko
return items[5]
~~~~~^^^
IndexError: list index out of range
Human-readable summary:
(funkodunko) Attempted to access an element of a list that does not exist.
Suggestions for how to fix the error:
1. Check the length of the list to make sure there are enough elements to access the desired index.
2. If the list is too short, add elements to the list in order to access the desired index.
3. If the list is empty, consider initializing the list with data.

GenerativeMetaClass

GenerativeMetaClass is enables its metaclassed classes to apply generated functions at run-time.

frommodelimportGPT3fromgenerative.metaclassesimportGenerativeMetaClassfrompromptimportformat_generative_functionclassDoggo(metaclass=GenerativeMetaClass):
def__init__(self, name: str):
self.name=namedefset_treat(self, treat: str):
self.stomach=treatprompt="Write a function with the header `def do_trick(self)` that returns a string '*sit*'"prompt=format_generative_function(prompt)
new_trick=GPT3.generate(prompt)
a_good_boy=Doggo('Chewy')
a_good_boy.generate(new_trick)
a_good_boy.do_trick()
a_good_boy.set_treat('roast beef')

Check which functions are generative

importinspectall_funcs=inspect.getmembers(
cls, predicate=inspect.isfunction
)
[fforfinall_funcsiff._is_generative]

Database integration

Clients can integrate custom database solutions to save the generated code, function name, and, if available, arguments and keyword arguments. This could be useful in large pipelines for embedding generated code offline. At run-time cachced embedded code could later be retreived given a similar input.

importredisfromgenerative.functionsimportadaptfromgenerative.metaclassesimportAbstractDatabasefrommodelsimportLLMclassVectorDB(AbstractDatabase):
def__init__(self):
self.db=redis.Redis(host='localhost', port=6379, db=0)
defcontains(self, key: str) ->bool:
returnself.db.exists(key)
defget(self, query: str) ->List[Dict] :
returnself.db.get(query)
defset(self, key: str, data: Any) ->None:
# Implement custom versioning logic here# query database by function name# store data by versions e.g {func_name: {'version_1': data }}self.db.set(key, data)
classDemo():
db=VectorDB()
@adapt(model=LLM, critic=LLM, database=db)deffunc(self):
pass# functionality that requires adaptation