parser.md

Chapter 3: Parser Layout

Back: Structure&AST | Chapters | Next: Parser Implementation

---

Let's first define a parser structure, which helps us traverse the input, skipping whitespaces, aswell as generating a helpful error message if we unexpectedly reach the end of input.

Parser

classParser:
def__init__(self, text: str):
self.text=text# the input text expressionself.index=0# the index we are currently atdefnext(self):
self.index+=1# let's skip the whitespaces - we do not have any need for those# and they just complicate parsing.whileself.has_current() andself.current().isspace():
self.index+=1defcurrent(self) ->str:
ifnotself.has_current():
raiseParseException('Unexpected end of input', self)
returnself.text[self.index]
# are we still in range?defhas_current(self) ->bool:
returnself.index<len(self.text)

We can now iterate over the input like this:

Example

fromcalculator.parserimportParserparser=Parser('sin(pi * x + 4)')
out=''whileparser.has_current():
out+=parser.current()
parser.next()
print(out)

sin(pi+4)*x

We can also define an exception class to inform the user about syntax errors

ParseException

classParseException:
def__init__(self, message: str, parser: 'Parser'):
self.message=message# passing along the source will give us helpful metadata # to format our exception (a task for later)self.parser=parserdef__str__(self):
returnf'ParseException: {self.message}'

Our parser needs a parsing method for each AST node, aswell as a generic parse_expression node to switch between the node types.

Parser

classParser:
...
defparse_expression(self) ->AST:
raiseParseException('Unimplemented!', self)
defparse_value(self) ->Value:
raiseParseException('Unimplemented!', self)
defparse_variable(self) ->Variable:
raiseParseException('Unimplemented!', self)
defparse_func_call(self) ->FuncCall:
raiseParseException('Unimplemented!', self)
defparse_binary_op(self) ->BinaryOp:
raiseParseException('Unimplemented!', self)
defparse_unary_op(self) ->UnaryOp:
raiseParseException('Unimplemented!', self)

And finally with all that we can define the parsing entrypoint

parse

defparse(text: str) ->AST:
parser=Parser(text)
ast=parser.parse_expression()
ifparser.has_more():
raiseParseException('Still more to parse', parser)
returnast

Now we can finally run the parser.
It, of course, fails immediately:

calculator.parser.ParseException: Unimplemented!

An immediate problem that we can see is that we dont know where we failed. This makes debugging, both when coding and using the parser very difficult.

Luckily, our exception class knows where we are since we give it our Parser object on construction.
This makes it easy to adjust the __str__ method.

ParseException

classParseException(Exception):
...
def__str__(self):
message=self.messagetext='| '+self.parser.textarrow='|-'+'-'*self.parser.index+'^'returnf'{message}\n{text}\n{arrow}'

We now show the source text aswell as an arrow to the specific location where the error occured.

calculator.parser.ParseException: Unimplemented!
|sin(pi*x+4)
|-^

This is much better! Now we can actually implement the parsing.

---

Back: Structure&AST | Chapters | Next: Parser Implementation