Automatically publishing changes to pelican

With very little investigation, I discovered a lengthy article about configuring Pelican, Dropbox, and automatic blog publishing using Dropbox. Well I'm not using Dropbox, I've a different setup, but it would still be handy to watch a folder and run a script when changes were found. This is something I epect I'll come back to again and again, so it makes sense to make my own notes.

I'm most interested in [using watcher to regenerate the blog automatically] and in the words of Roberto Zoia:

Watcher is a nifty utility written in Python that monitors a directory and its subdirectories for change using the inotify interface. It can be configured to execute a command when a change is detected, which is just what we need to regenerate our blog.

Sounds perfect, doesn't it. Rather than copying his article for my reference, I'll follow it and make notes here so I'll be able to reproduce the configuration at a later date.

Installing watcher

Terminal commands, can be run in a virtualenv or not. The first thing, is that Andreas Gohr created a forked version, noted in Roberto Zoia's article which uses an .ini file instead of a YAML file. I'm not sure what the benefit of that is, but let's go with what worked for Roberto.

$ pip install pyinotify
$ wget https://github.com/splitbrain/Watcher/archive/master.zip
$ unzip master.zip
$ mv ./Watcher-master/* .
$ rmdir ./Watcher-master
$ chmod +x ./watcher.py

Configuring watcher

The configuration options in this particular version of watcher, are in an ini file called watcher.ini, stored alongside the watcher.py script.

[job1]
watch=/home/<user>/<project>/output
events=create,delete,modify,move
excluded=
recursive=true
autoadd=true
command=/home/nllewellyn/lilnick.co.uk/publishupload.py

Running watcher

Um, well, that didn't work did it. Haha, this happens to me a lot, let's investigate the issue.

$ python /home/<user>/<project>/watcher.py -c /home/nllewellyn/lilnick.co.uk/watcher.ini start
watcher.ini start
  File "/home/<user>/<project>/watcher.py", line 68
    except OSError, e:
                  ^
SyntaxError: invalid syntax

It may be that, according to the documentation, there is no option to have watcher.ini in it's own folder.

Configuration

See the provided watcher.ini file for an example job configuration. The config file should reside in /etc/watcher.ini or ~/.watcher.ini. You can also specify the path to the config file as a command line parameter using the --config option. If you edit the ini file you must restart the daemon for it to reload the configuration.

Let's move it into /etc/watcher.ini and see what happens then, before we go rummaging around in the code to find out what is going wrong.

$ sudo mv ./watcher.ini /etc/watcher.ini
$ python /home/nllewellyn/lilnick.co.uk/watcher.py start  File "/home/<user>/<project>/watcher.py", line 68
    except OSError, e:
                  ^
SyntaxError: invalid syntax

Well, that didn't work either. Let's start looking at the code of watcher.py and see what is going on in line 68.

57     def daemonize(self):
58         """
59         do the UNIX double-fork magic, see Stevens' "Advanced Programming in the
60        UNIX Environment" for details (ISBN 0201563177)
61        http://www.erlenstar.demon.co.uk/unix/faq_2.html#SEC16
62        """
63        try:
64            pid = os.fork()
65            if pid > 0:
66                #exit first parent
67                sys.exit(0)
68        except OSError, e:
69            sys.stderr.write("fork #1 failed: %d (%s)\n" % (e.errno, e.strerror))
70            sys.exit(1)
71
72        # decouple from parent environment
73        os.chdir("/")
74        os.setsid()
75        os.umask(0)

Well it looks like OSError is a standard procedure, and Python is complaining about something being written incorrectly in the syntax, or usage of this. A little searching reveals, that in Python 2 both, the comma and as were permitted on this function. It makes sense that this is a fault then, I'm using Python 3.6.7 and Andreas Gohr said this was written for Python 2.7.

There are two ways of fixing this, the first way would be to alter the code and the second way would be to alter the script to use the older version of Python. I currently have Python 2.7.15+ installed alongside Python 3.6.7. The simplest thing here would be to move the watcher.ini file back, and force the use of Python 2.7.

$ sudo mv /etc/watcher.ini ./watcher.ini
$ python2 /home/<user>/<project>/watcher.py -c /home/<user>/<project>/watcher.ini start
watcher.ini start
Traceback (most recent call last):
  File "/home/nllewellyn/lilnick.co.uk/watcher.py", line 36, in <module>
    import pyinotify
ImportError: No module named pyinotify

Well now then I've not got pyinotify installed into Python 2.7, let's do that as it is a dependency, and try again.

$ pip2 install pyinotify
Collecting pyinotify
  Cache entry deserialization failed, entry ignored
Installing collected packages: pyinotify
Successfully installed pyinotify-0.9.6
$ python2 /home/nllewellyn/lilnick.co.uk/watcher.py -c /home/nllewellyn/lilnick.co.uk/watcher.ini start
$ python2 /home/nllewellyn/lilnick.co.uk/watcher.py -c /home/nllewellyn/lilnick.co.uk/watcher.ini status
service running

Sheesh, that was long-winded and incomplete, after a little testing I noticed a silly error. I asked watcher to watch the output folder, when it should have been watching the content folder doh! Let's alter that watcher.in file, and restart that watcher service.

$ python2 /home/nllewellyn/lilnick.co.uk/watcher.py -c /home/nllewellyn/lilnick.co.uk/watcher.ini stop

Edit that watcher.ini file and alter the line watch=/home/nllewellyn/lilnick.co.uk/output to watch=/home/nllewellyn/lilnick.co.uk/content instead and try again.

Incredible, it works. I save a file into output and the Pelican publish scripts run. Wonderful. Now all I need to do is run through this process again to tidy it all up for my other sites, and host the whole lot within a simple container of some sort on a server always online where I can update the content files remotely suing something like Syncthing to move the new content to the central location.

Fantastic. I'm chuffed to bits about this.

links

social