Riddles of Sphinx

Sphinx is a tool to autogenerate documentation. It's generic but is often used in the Python ecosystem. 

Once configured, you run it via an old school Makefile. It will then do to your Python code was Javadoc does to Java. It doesn't appear to have the ability to mix code and documentation like ScalaTest.

How to use it

I found this StackOverflow answer a lot clearer than the official documentation. The upshot of it is:

1. Create a docs folder. In this, you can run sphinx-quickstart if you like (separating source from build - see SOURCEDIR and BUILDDIR in the Makefile) but one way or another, your conf.py file will need to know where to find the code. If you've followed the instruction in the SO answer, it's two levels up so conf.py needs to have:
   
   import os
   import sys
   sys.path.insert(0, os.path.abspath(os.path.join('..', '..', 'src')))
   
   (note those two '..') in it. Also, note that this is the top-level directory of your code. If there are relative imports in your code, these need to be relative to this directory.
2. You'll also need the extensions to parse Python.
3. The index.rst file needs to be told what to pull in. You do this by adding a reference under toctree (let's arbitrarily call it module) that corresponds to another file (module.rst) in the same directory. 
4. In turn, our arbitrarily names module.rst needs in its toctree a reference to our final sub-module. We'll call it my_docs and it will necessitate a file called my_docs.rst in the same directory.
5. Here, in the automodule block, we'll tell Sphinx which file want documented. Note that you don't need the .py extension.
6. Note that the .rst files do not need to conform to your directory structure.

PDFs

Sphinx natively creates Latex files but to create PDF, I needed to run:

sudo apt-get install latexmk

which allowed me to run:

make latexpdf

Dependencies

I had a problem with third-party libraries on which my code depended. Having found my code, Sphinx was telling me no module named pyspark when it hit the import statement in my code. This SO answer helped out but it left me with a nasty hard-coding in my conf.py. This may be more down to the way Python manages its dependencies (which is horrible compared to Java).

Writing effective Sphinx docstring

You can reference constants in your docstring with something like :py:const:`FQN.CONSTANT_NAME` (where FQN is the fully qualified path to your Python file). If you also put something like this in your .rst file:

.. autoattribute:: FQN.CONSTANT_NAME

then Sphinx will not only provide a hyperlink to the constant but also print out the value of that constant.

Executing the code in comments

For this, try doctest. It not only executes code in the comments but checks the results.