The goal of this module is to help write code that generates code. Focus is placed on enabling the user to easily describe, build and reason about code structures rapidly.
pip install 0xf0f-codenode
pip install git+https://github.com/0xf0f/codenode
Like the json and
pickle modules, dump and dumps are used to generate output.
Code can be built using any tree of iterables containing strings,
indentation nodes and newline nodes.
For example, the built-in line function returns a tuple:
fromcodenodeimportindentation, newlinedefline(content):
returnindentation, content, newlineWhich we can combine with indent and dedent nodes:
fromcodenodeimportline, indent, dedent, dumpsdefcounting_function(count_from, count_to):
return [
line(f'def count_from_{count_from}_to_{count_to}():'),
indent,
[
line(f'print({i})')
foriinrange(count_from, count_to)
],
dedent,
]
print(dumps(counting_function(0, 5)))Which outputs:
def count_from_0_to_5():
print(0)
print(1)
print(2)
print(3)
print(4)
But what if we want to count to a really big number, like 1,000,000,000,000,000? It would be inefficient to store all those lines in memory at once. We can use a generator to break them down into individual parts instead:
fromcodenodeimportindent, dedent, newline, indentation, dumpdefcounting_function_generator(count_from, count_to):
yieldindentationyield'def count_from_', str(count_from), '_to_', str(count_to), '():'yieldnewlineyieldindentforiinrange(count_from, count_to):
yieldindentation, 'print(', str(i), ')', newlineyielddedentwithopen('code.py', 'w') asfile:
dump(counting_function_generator(0, 1_000_000_000_000_000), file)We can also build a class with an __iter__ method:
fromcodenodeimportline, indent, dedent, dumpclassCountingFunction:
def__init__(self, count_from, count_to):
self.count_from=count_fromself.count_to=count_todef__iter__(self):
yieldline(
f'def count_from_{self.count_from}_to_{self.count_to}():'
)
yieldindentforiinrange(self.count_from, self.count_to):
yieldline(f'print({i})')
yielddedentwithopen('code.py', 'w') asfile:
dump(CountingFunction(0, 1_000_000), file)Or a more generalized function class:
classFunction:
def__init__(self, name, *args):
self.name=nameself.args=argsself.children= []
def__iter__(self):
arg_string=', '.join(self.args)
yieldline(f'def {self.name}({arg_string}):')
yieldindentyieldself.childrenyielddedentclassCountingFunction(Function):
def__init__(self, count_from, count_to):
super().__init__(f'count_from_{count_from}_to_{count_to}')
foriinrange(count_from, count_to):
self.children.append(line(f'print({i})'))Leveraging python's iteration protocol like this allows:
- Mixing and matching whatever fits the use case to maximize tradeoffs, such as using generators for their memory efficiency, custom iterable classes for their semantics, or plain old lists and tuples for their simplicity.
- Taking advantage of existing modules that offer tooling for iterables, such as itertools.
- Building higher level structures from as many iterable building blocks as desired.
Module behaviour can be extended by overriding methods of the
codenode.writer.Writer and codenode.writer.WriterStack classes. An
example of this can be seen in the codenode.debug.debug_patch
function. The variable codenode.default_writer_type can be used to
replace the Writer type used in dump and dumps with a custom one.
Some modules with helper classes and functions are also provided:
- contains general language agnostic helper functions and classes
- contains helper classes and functions for generating python code
Note This section of the readme was generated using codenode itself.
See docs/generate_readme.py
- codenode.dump
- codenode.dumps
- codenode.line
- codenode.indent
- codenode.dedent
- codenode.newline
- codenode.indentation
- codenode.lines
- codenode.empty_lines
- codenode.indented
- codenode.default_writer_type
- codenode.writer.Writer
- codenode.writer.WriterStack
- codenode.nodes.newline.Newline
- codenode.nodes.depth_change.DepthChange
- codenode.nodes.depth_change.RelativeDepthChange
- codenode.nodes.depth_change.AbsoluteDepthChange
- codenode.nodes.indentation.Indentation
- codenode.nodes.indentation.RelativeIndentation
- codenode.nodes.indentation.AbsoluteIndentation
- codenode.nodes.indentation.CurrentIndentation
- codenode.debug.debug_patch
defdump(node, stream, *, indentation=' ', newline='\n', depth=0, debug=False): ...Process and write out a node tree to a stream.
node: Base node of node tree.
stream: An object with a 'write' method.
indentation: String used for indents in the output.
newline: String used for newlines in the output.
depth: Base depth (i.e. number of indents) to start at.
debug: If True, will print out extra info when an error occurs to give a better idea of which node caused it.
defdumps(node, *, indentation=' ', newline='\n', depth=0, debug=False) ->str: ...Process and write out a node tree as a string.
node: Base node of node tree.
indentation: String used for indents in the output.
newline: String used for newlines in the output.
depth: Base depth (i.e. number of indents) to start at.
debug: If True, will print out extra info when an error occurs to give a better idea of which node caused it.
String representation of node tree.
defline(content: 'T') ->'tuple[Indentation, T, Newline]': ...Convenience function that returns a tuple containing an indentation node, line content and a newline node.
content: content of line
tuple containing an indentation node, line content and a newline node.
indent=RelativeDepthChange(1)A node representing a single increase in indentation level.
dedent=RelativeDepthChange(-1)A node representing a single decrease in indentation level.
newline=Newline()A placeholder node for line terminators.
indentation=CurrentIndentation()A placeholder node for indentation whitespace at the start of a line.
deflines(*items) ->'tuple[tuple, ...]': ...Convenience function that returns a tuple of lines, where each argument is the content of one line.
items: contents of lines
tuple of lines
defempty_lines(count: int) ->'tuple[Newline, ...]': ...Convenience function that returns a tuple of newline nodes.
count: Number of newlines.
Tuple of newlines.
defindented(*nodes) ->tuple: ...Convenience function that returns a tuple containing an indent node, some inner nodes, and a dedent node.
nodes: inner nodes
tuple containing an indent node, inner nodes, and a dedent node.
default_writer_type=WriterDefault Writer type used in codenode.dump and codenode.dumps.
classWriter: ...Processes node trees into strings then writes out the result.
Each instance is intended to be used once then discarded. After a single call to either dump or dumps, the Writer instance is no longer useful.
classWriter: def__init__(self, node: 'NodeType', *, indentation=' ', newline='\n', depth=0): ...
node: Base node of node tree.
indentation: Initial string used for indents in the output.
newline: Initial string used for newlines in the output.
depth: Base depth (i.e. number of indents) to start at.
classWriter: defprocess_node(self, node) ->'Iterable[str]': ...Yield strings representing a node and/or apply any of its associated side effects to the writer
for example:
yield indentation string when an indentation node is encountered
increase the current writer depth if an indent is encountered
append an iterator to the stack when an iterable is encountered
node: node to be processed
strings of text chunks representing the node
classWriter: defdump_iter(self) ->'Iterable[str]': ...Process and write out a node tree as an iterable of string chunks.
Iterable of string chunks.
classWriter: defdump(self, stream): ...Process and write out a node tree to a stream.
stream: An object with a 'write' method.
classWriter: defdumps(self): ...Process and write out a node tree as a string.
String representation of node tree.
node: Base node of node tree
stack: WriterStack used to iterate over the node tree
indentation: Current string used for indents in the output
newline: Current string used for line termination in the output
depth: Current output depth (i.e. number of indents)
classWriterStack: ...A stack of iterators. Used by the Writer class to traverse node trees.
Each instance is intended to be used once then discarded.
classWriterStack: defpush(self, node: 'NodeType'): ...Converts a node to an iterator then places it at the top of the stack.
node: iterable node
classWriterStack: def__iter__(self) ->'Iterable[NodeType]': ...Continually iterates the top iterator in the stack's items, yielding each result then popping each iterator off when they are exhausted.
items: collections.deque - Current items in the stack.
classNewline: ...Nodes that represent the end of a line.
classDepthChange: ...Nodes that represent a change in indentation depth.
classDepthChange: defnew_depth_for(self, depth: int) ->int: ...Method used to calculate the new depth based on the current one.
depth: Current depth.
New depth.
classRelativeDepthChange: ...Nodes that represent a change in indentation depth relative to the current depth by some preset amount.
classRelativeDepthChange: def__init__(self, offset: int): ...
offset: Amount by which to increase/decrease depth.
offset: Amount by which to increase/decrease depth when this node is processed.
classAbsoluteDepthChange: ...Nodes that represent a change in indentation depth without taking the current depth into account.
classAbsoluteDepthChange: def__init__(self, value: int): ...
value: Value to set depth to.
value: Value to which depth will be set to when this node is processed.
classIndentation: ...Nodes that represent indentation whitespace at the start of a line.
classIndentation: defindents_for(self, depth: int) ->int: ...
depth: Current depth.
Number of indents to include in whitespace when this node is processed.
classRelativeIndentation: ...Nodes that represent indentation whitespace at the start of a line, with a number of indents relative to the current depth by some preset amount.
classRelativeIndentation: def__init__(self, offset: int): ...
offset: Amount of indents relative to the current depth.
offset: Amount of indents relative to the current depth that will be output when this node is processed.
classAbsoluteIndentation: ...Nodes that represent indentation whitespace at the start of a line, with a number of indents independent of the current depth.
classAbsoluteIndentation: def__init__(self, value: int): ...
value: Amount of indents.
value: Amount of indents that will be output when this node is processed.
classCurrentIndentation: ...Nodes that represent indentation whitespace at the start of a line, with a number of indents equal to the current depth.
defdebug_patch(writer_type: typing.Type[Writer]) ->typing.Type[Writer]: ...Creates a modified version of a writer type which prints out some extra info when encountering an error to give a better ballpark idea of what caused it. Used in codenode.dump/dumps to implement the debug parameter.
writer_type: Base writer type.
New child writer type with debug modifications.