A Brief Tutorial on Parsing reStructuredText (reST)
A Brief Tutorial on Parsing reStructuredText (reST)
Ever wonder how to take reStructuredText and generate something OTHER than plain old HTML? In this post we take a look at how to do just that, complete with an example.
Join the DZone community and get the full member experience.Join For Free
Learn how error monitoring with Sentry closes the gap between the product team and your customers. With Sentry, you can focus on what you do best: building and scaling software that makes your users’ lives better.
Docutils, the canonical library for processing and munging reStructuredText, is mostly used in an end-to-end mode where HTML or other user-consumable formats are produced from input reST files. However, sometimes it's useful to develop tooling that works on reST input directly and does something non-standard. In this case, one has to dig only a little deeper in Docutils to find useful modules to help with the task.
In this short tutorial, I'm going to show how to write a tool that consumes reST files and does something other than generating HTML from them. As a simple but useful example, I'll demonstrate a link checker - a tool that checks that all web links within a reST document are valid. As a bonus, I'll show another tool that uses internal table-parsing libraries within Docutils that let us write pretty-looking ASCII tables and parse them.
Parsing reST Text into a Document
This tutorial is a code walk-through for the complete code sample available online. I'll only show a couple of the most important code snippets from the full sample.
Docutils represents a reST file internally as your typical document tree (similarly to many XML and HTML parsers), where every node is of a type derived from docutils.nodes.Node. The top-level document is parsed into an object of type document.
We start by creating a new document with some default settings and populating it with the output of a Parser:
# ... here 'fileobj' is a file-like object holding the contents of the input # reST file. # Parse the file into a document with the rst parser. default_settings = docutils.frontend.OptionParser( components=(docutils.parsers.rst.Parser,)).get_default_values() document = docutils.utils.new_document(fileobj.name, default_settings) parser = docutils.parsers.rst.Parser() parser.parse(fileobj.read(), document)
Processing a reST Document With a Visitor
Once we have the document, we can go through it and find the data we want. Docutils helps by defining a hierarchy of Visitor types, and a walk method on every Node that will recursively visit the subtree starting with this node. This is a very typical pattern for Python code; the standard library has a number of similar objects - for example, ast.NodeVisitor.
Here's our visitor class that handles reference nodes especially:
class LinkCheckerVisitor(docutils.nodes.GenericNodeVisitor): def visit_reference(self, node): # Catch reference nodes for link-checking. check_link(node['refuri']) def default_visit(self, node): # Pass all other nodes through. pass
How did I know it's reference nodes I need and not something else? Just experimentation :) Once we parse a reST document, we can print the tree and it shows which nodes contain what. Coupled with reading the source code of Docutils (particularly the docutils/nodes.py module) it's fairly easy to figure out which nodes one needs to catch.
With this visitor class in hand, we simply call walk on the parsed document:
# Visit the parsed document with our link-checking visitor. visitor = LinkCheckerVisitor(document) document.walk(visitor)
That's it! To see what check_link does, check out the code sample.
Bonus: Parsing ASCII Grid Tables With Docutils
Docutils supports defining tables in ASCII in a couple of ways; one I like in particular is "grid tables", done like this:
+------------------------+------------+----------+----------+ | Header row, column 1 | Header 2 | Header 3 | Header 4 | +========================+============+==========+==========+ | body row 1, column 1 | column 2 | column 3 | column 4 | +------------------------+------------+----------+----------+ | body row 2 | Cells may span columns. | +------------------------+------------+---------------------+ | body row 3 | Cells may | - Table cells | +------------------------+ span rows. | - contain | | body row 4 | | - body elements. | +------------------------+------------+---------------------+
Even if we don't really care about reST but just want to be able to parse tables like the one above, Docutils can help. We can use its tableparser module. Here's a short snippet from another code sample:
def parse_grid_table(text): # Clean up the input: get rid of empty lines and strip all leading and # trailing whitespace. lines = filter(bool, (line.strip() for line in text.splitlines())) parser = docutils.parsers.rst.tableparser.GridTableParser() return parser.parse(docutils.statemachine.StringList(list(lines)))
The parser returns an internal representation of the table that can be easily used to analyze it or to munge & emit something else (by default Docutils can emit HTML tables from it).
One small caveat in this code to pay attention to: we need to represent the table as a list of lines (strings) and then wrap it in a docutils.statemachine.StringList object, which is a Docutils helper that provides useful analysis methods on lists of strings.
Published at DZone with permission of Eli Bendersky . See the original article here.
Opinions expressed by DZone contributors are their own.