My name is Alex Claydon and this blog is a record of my development work, primarily in the renewable energy space.

Learnings from Customising and Deploying a Dockerized Instance of Jitsi-Meet


Recently, as part of a telehealth services rollout, I went through the process of customising and deploying an instance of Jitsi-Meet to Docker on an Ubuntu 20.04 LTS host.  The following are learnings from that experience.  This is an early draft; I'll continue updating with additional content and clarifications in future.  This article covers a development deployment; in a future piece I'll cover production deployments.

First of all, don't forget to drop by the Jitsi-Meet [community forums](https://community.jitsi.org/); they are an excellent source of discussion on problems that you may face.  However, most of the participants seem to be developing and deploying using the standard "quick install" or full install method - not using Docker.  For reasons that are beyond the scope of this article, I wanted to develop against, and deploy to, Docker containers.  I know I'm not alone in this: see, eg, [here](https://community.jitsi.org/t/dev-workflow-question-how-to-build-a-custom-docker-image-of-jitsi-meet-web-for-use-in-our-dockerized-setup/73527), but the answers I found were not entirely on point for my use case.

So, the following covers a range of issues that might come up from developing and deploying a Dockerized Jitsi-Meet instance, including:

- How to duplicate (not fork) the relevant Git repos, to allow you to develop against your own private repo, protecting any potentially sensitive commercial information while still benefiting from updates to the main Jitsi-Meet repos;
- How to set up your back and front-end contexts to allow for a convenient development workflow; and
- At a later date, instructions on deploying to production using Docker.

A disclaimer: I'm setting out herein what's worked for me to try to help anyone that might be interested to avoid the pain points that I came up against; you may think you can improve on the below - if so, feel free to write down your thoughts and add them to the discussion on the community forums.  The more information the better!

## Getting the code

First things first: you need to get the code.  The relevant GitHub repos are as follows: [`jitsi-meet`](https://github.com/jitsi/jitsi-meet), and [`docker-jitsi-meet`](https://github.com/jitsi/docker-jitsi-meet).

You'll only need a couple of files from the `docker-jitsi-meet` repo, so we'll deal with those later, but if you want to make heavy customisations to the front-end web app, you'll be making a large number of changes to the `jisti-meet` repo.  Although your first instinct may be to fork it, note that GitHub doesn't permit forks of public repos to be made private.  Accordingly, if you think there might be a possibility that commercially sensitive information ends up in your codebase, you'll want instead to _duplicate_ the `jitsi-meet` repo.  This will allow you to work on your own private copy of the repos but also to pull in changes made to `master` using `git pull` or `git rebase` (depending on your needs).

Do the following:

1. Make a new empty, private repository on GitHub
2. Bare clone the public `jitsi-meet` repo to your local machine: `git clone --bare https://github.com/jitsi/jitsi-meet.git`
3. Mirror push the local bare repo to your new empty private repo; e.g.: `cd jitsi-meet && git push --mirror https://github.com/exampleuser/private-jitsi-repo.git`
4. Delete the local bare copy of the public repo, which you no longer need: `cd .. && rm -rf jitsi-meet`
5. Create a local clone of your private repo: `git clone https://github.com/exampleuser/private-jitsi-repo.git`
6. Add the public upstream repo to the private repo as a remote: `git remote add upstream https://github.com/jitsi/jitsi-meet.git`
7. Disable pushing to the public upstream repo (you don't have push priveleges anyway): `git remote set-url --push upstream DISABLE` 
8. Each time you want to pull the latest commits made to the public upstream repo, use: `git pull upstream master` and then push those new commits to your private repo `git push origin master`.  Alternatively, you might want to do a `git rebase` instead, depending on your needs.

Your private duplicate of the `jitsi-meet` repo contains all of the front-end files that you might like to customise - the React components, `.html` and `.css` files, etc.  Here's where you'll do most of your work.

You can use `make dev` from the project root (see the included `Makefile` for details) to install your `node_modules`, compile the `SCSS`, etc., and spin up the `webpack-dev-server` to see your changes to the front-end reflected immediately in your browser using `webpack` live-reload.  However, when first doing this I noted that some of the interface changes I had made in the repo itself - in fact, notably, all of the settings in `interface_config.js` file, as opposed to changes made to, for example, `.html` and `.css` files directly - weren't displaying correctly in my `localhost` front-end.  For example, the Jitsi Meet watermark would display on the front page and in-call, instead of my custom watermark, even though I had specified one in `interface_config.js`.

Long story short, the settings in `interface_config.js` actually configure the back-end service, not the front end - notwithstanding the file name.  Accordingly - and perhaps unsurprisingly - to get the complete picture of how your production user interface will look, you will need to also bring up a local back-end.

## Setting up a local development back-end

This is probably the topic on which there's the least Docker-specific advice out there.  To set up your back-end using Docker, you'll need a couple of files from the [`docker-jitsi-meet`](https://github.com/jitsi/docker-jitsi-meet.git) repo.  You can either `git clone` or download this repo - no need to fork or duplicate it.  The files you want are the `docker-compose.yml` file in the root directory, and the `Dockerfile` and `rootfs` directory (and its contents) from the `web` subfolder.  Personally, I've saved these files into a custom provisioning repo that I set up to serve the particular client I'm working with.  This private repo also contains development notes, some of the `bash` provisioning scripts set out below, various Ansible playbooks for setting up infrastructure, etc, and the `.env` files used by Docker when bringing up the container stack.  This keeps all of the provisioning and deployment code in one place.

If you take a look at the `docker-compose.yml` file mentioned above, you can see that where you need to get to is a custom-built `jitsi/web` image that incorporates all of the custom UI changes that you've made to your private `jitsi-meet` repo.  If you're familiar with Docker you'll know that there are a few different ways you can go about doing that.  I have seen some discussion about `SSH`-ing into running containers (or using `docker cp` from the host) and making your changes directly to the contents of the `/usr/share` target `jitsi-meet` deploy directory on the container, then committing those changes to a new Docker image based on the existing one.  That might be okay if you're making only minor changes to your front-end, but in general that practice is discouraged.  You want to make any required changes in the relevant `Dockerfile` you're using to build your `jitsi/web` container instead.

Let's take a look at the `Dockerfile` for the `jitsi/web` image:

```Dockerfile
ARG JITSI_REPO=jitsi
FROM ${JITSI_REPO}/base

ADD https://dl.eff.org/certbot-auto /usr/local/bin/

COPY rootfs/ /

RUN \
	apt-dpkg-wrap apt-get update && \
	apt-dpkg-wrap apt-get install -y cron nginx-extras jitsi-meet-web && \
	apt-dpkg-wrap apt-get -d install -y jitsi-meet-web-config && \
    dpkg -x /var/cache/apt/archives/jitsi-meet-web-config*.deb /tmp/pkg && \
    mv /tmp/pkg/usr/share/jitsi-meet-web-config/config.js /defaults && \
	mv /usr/share/jitsi-meet/interface_config.js /defaults && \
	apt-cleanup && \
	rm -f /etc/nginx/conf.d/default.conf && \
	rm -rf /tmp/pkg /var/cache/apt

RUN \
	chmod a+x /usr/local/bin/certbot-auto && \
	certbot-auto --noninteractive --install-only

EXPOSE 80 443

VOLUME ["/config", "/etc/letsencrypt", "/usr/share/jitsi-meet/transcripts"]
```

The first thing you'll notice is that `jitsi-meet-web` is installed as an `apt` package.  Initially, I had thought that where I wanted to get to was packing up my own custom `jitis-meet` repo into an `apt` package and then replacing the existing `apt` package in the `Dockerfile`.  That approach, though, comes with its own overhead - and thankfully, there's an easier way.

If you're interested to read some of the discussion, [this](https://community.jitsi.org/t/change-welcome-page-text-after-quick-install/15279/3), [this](https://community.jitsi.org/t/how-to-how-to-build-jitsi-meet-from-source-a-developers-guide/75422), [this](https://community.jitsi.org/t/development-in-localhost-jitsi-meet/17773/11) and [this](https://community.jitsi.org/t/dev-workflow-question-how-to-build-a-custom-docker-image-of-jitsi-meet-web-for-use-in-our-dockerized-setup/73527/2) got me oriented in the right direction.

Here's how I'm doing it:

- `cd` into your custom `jitsi-meet` repo and call `make && make source-package`.  The zip archive outputted as a result of this operation (`jitsi-meet.tar.bz2`) will contain all of your user interface customisations.
- Extract the archive into your `jitsi-web` Docker image build context (being the directory that also contains your `Dockerfile` and the `rootfs` directory taken from the `docker-jisti-meet` repo above).  If you've just cloned the entire `docker-jitsi-meet` repo from GitHub (instead of setting up your own personal provisioning repo), the Docker build context will be the `web` subdirectory of that repo.  The contents of the archive will be extracted into a subdirectory called `jitsi-meet`.
- `cd` into your build context, open the `Dockerfile` for editing, and add the following two lines just above the `EXPOSE` command:

    ```Dockerfile
    COPY jitsi-meet/ /usr/share/jitsi-meet
    COPY jitsi-meet/interface_config.js /defaults/interface_config.js
    ```

    This will tell Docker to copy your own custom code on top of the existing `jitsi-meet` web app installed using the pre-built `apt` package.

- You now need to build your custom image based on this `Dockerfile`.  From the build context, call `docker build -t jitsi/web .` (note: don't exclude that trailing period, which tells Docker that this directory is the build context).  Note that if you already have a local copy of the `jitsi/web` image downloaded from the DockerHub container registry (for example, as part of a previous Docker installation of `jitsi-meet`), you'll need to add the `--no-cache` flag to your build command to ensure that image is overwritten (`docker build --no-cache -t jitsi/web .`).

Once all of your images are built, follow the usual instructions [here](https://jitsi.github.io/handbook/docs/devops-guide/devops-guide-docker) to bring up the entire Jitsi-Meet container stack, which now includes a web app container built from your custom `jitsi/web` image.  Note that you will likely want to have different contents in your `.env` file depending on whether you are doing a development or production deployment.  See the Docker devops guide for more information on all of the environment variables.  Specifically for the purposes of setting up a development back-end, you will need to ensure both that Let's Encrypt is disabled, and that you uncomment the `DOCKER_HOST_ADDRESS=` env and insert the IP address of the host machine.  We'll look at this in more detail later.

I've packaged up all of the above into a simple bash script that I can run from the command line every time I want to re-compile and re-deploy the back-end into a new Docker container after making changes.  Note that the script, however, doesn't touch your firewall ports - you'll need to do that manually - and that it assumes that you don't already have a locally-cached version of the `jitsi/web` image downloaded from DockerHub (in which case, you'll need to pass the `--no-cache` flag to your build command).

Here it is; feel free to use and to modify to fit your needs:

```bash
#!/bin/bash

REPO_ROOT=~/dev/projects/telehealth-provisioning
TELEHEALTH_JITSI_MEET_DIR=~/dev/projects/telehealth-jitsi-meet
JITSI_MEET_CONFIG_DIR=~/.jitsi-meet-cfg

echo "Building Jitsi-Meet packages from source ..." && sleep 1

cd $TELEHEALTH_JITSI_MEET_DIR || { echo "Failed to find $TELEHEALTH_JITSI_MEET_DIR"; exit 1; }
make || { echo "Make failed"; exit 1; }
make source-package || { echo "Make source-package failed"; exit 1; }

echo "Moving and unzipping Jitsi-Meet package archive into Docker build context" && sleep 1

mv $TELEHEALTH_JITSI_MEET_DIR/jitsi-meet.tar.bz2 $REPO_ROOT/Docker/web || { echo "Failed to move packages to destination"; exit 1; }
cd $REPO_ROOT/Docker/web || { echo "Failed to find $REPO_ROOT/Docker/Web"; exit 1; }
tar -xvf jitsi-meet* || { echo "Failed to unzip archive"; exit 1; }
rm -rf $REPO_ROOT/Docker/web/jitsi-meet.tar.bz2

echo "Rebuilding Jitsi-Meet web Docker image" && sleep 1

docker build -t jitsi/web . || { echo "Docker operation failed"; exit 1; }
rm -rf $REPO_ROOT/Docker/web/jitsi-meet 

echo "Taking down the Docker container stack" && sleep 1

cd $REPO_ROOT/Docker || { echo "Failed to find $REPO_ROOT/Docker"; exit 1; }
docker-compose down 

echo "Removing and recreating Jitsi-Meet config directories" && sleep 1

sudo rm -rf $JITSI_MEET_CONFIG_DIR || { echo "Failed to remove $JITSI_MEET_CONFIG_DIR"; exit 1; }
mkdir -p $JITSI_MEET_CONFIG_DIR/{web/letsencrypt,transcripts,prosody/config,prosody/prosody-plugins-custom,jicofo,jvb,jigasi,jibri} || { echo "Failed to create required config directories at $JITSI_MEET_CONFIG_DIR root"; exit 1; }

echo "Rebuilding Jitsi-Meet web Docker container and bringing up Docker container stack" && sleep 1

cd $REPO_ROOT/Docker || { echo "Failed to find $REPO_ROOT/Docker"; exit 1; }
./gen-passwords.sh
docker-compose up -d || { echo "Docker operation failed"; exit 1; }

echo "Back-end is up"
```

## Setting up a local development front-end

So, we've got our custom front-end UI code, we've got a custom back-end development server hosted using Docker.  Now we want to bring up the front-end, connect it to the back-end and enable live-reloading with `webpack`, so you can see your UI changes in realtime.  

Take a look [here](https://jitsi.github.io/handbook/docs/dev-guide/dev-guide-web).  Note that if you bring up your front-end with `make dev` but without explicitly specifying your own back-end in the `WEBPACK_DEV_SERVER_PROXY_TARGET` environment variable, Jitsi-Meet will by default connect to the publicly available back-end at `alpha.jitsi.net`.  That might be fine if you just want to test some changes that you know don't depend on which back-end you're connected to, but it won't give you any sense of what your instance will look like in production.

So, `cd` into your custom `jitsi-meet` repo and then start the webpack server as follows, passing your custom back-end as an environment variable:

`WEBPACK_DEV_SERVER_PROXY_TARGET="https://$HOST-MACHINE-IP-ADDRESS:8443" make dev`

where `$HOST-MACHINE-IP-ADDRESS` is your host machine IP, and the number after the colon is the `HTTPS` port specified in your `.env` file (by default, 8443).  Your development front and back ends are now up and connected.

Don't forget to open the required ports in the firewall on your host machine; for example:

```bash
sudo ufw allow 8000/tcp
sudo ufw allow 8443/tcp
sudo ufw allow 4443/tcp
sudo ufw allow in 10000:20000/udp
```

That's it.  Open a browser and navigate to `https://localhost:8080/` (default port 8080 is set in your `.env` file) for a complete picture of how your stack looks.  Any changes made to the front-end will be cause `webpack` to re-compile and re-display the output in your browser automatically, although any changes you make to files that affect the back-end will require you to re-run the container build process detailed above (preferrably using that `bash` script or something like it for convenience.)

I'll document my production deployment process in a future article.

Thanks for reading and I hope that's of some use to somebody.


SQL and Python


The shortcomings of database object-relational mappers (ORMs) are well-documented.  Most memorable is [this](http://blogs.tedneward.com/post/the-vietnam-of-computer-science/) 2006 piece by Ted Neward comparing the ORM quagmire to the US involvement in Vietnam.  Hyperbolic, maybe - but also pretty evocative and certinaly memorable.  This was the piece that made me realise: an ORM is not the only - nor necessarily the best - way to wire up your arbitrary program code with your database.

In many cases, however, it may be the first place you end up.  SQLAlchemy and Django ORM are monsters in the Python world, and depending on how you came to coding generally and to Python in particular, you may think of them as your only realistic options when trying to wire up a database.  They aren't.

Of course, it would also be going too far to say that using an ORM is never a good idea.  There are plenty of situations where it is an excellent idea.  This post is not about those situations.  This post is about situations where you explicitly do not want to use an ORM for whatever reason.  Perhaps you think SQL is an excellent and capable language in and of itself.  Perhaps you are constitutionally inclined to using a lightweight, thin API to talk to your database instead of a full-blown ORM.  Perhaps you just really want to improve your SQL.  For me, a combination of these concerns led me to wonder what life is like outside of ORM-land.

It is, of course, possible to talk directly to the database using a Python driver.  Let's talk SQLite as an example.  Let's start by doing some setup.

```python
import sqlite3

database='db/sqlite.db'
conn = sqlite3.connect(database)
```

Next, let's consider the following code to initialise the database from a separate `.sql` script file.

```python
cursor = conn.cursor()
with open('db/db_init.sql', 'r') as sqlite_file:
    sql_script = sqlite_file.read()
cursor.executescript(sql_script)
cursor.close()
```

The example of database initialisation is germane; while it might be a good approach to a one-off process such as initialisation, you don't want to call a script every time you want to run arbitrary queries against the database.  To do that, you can treat SQL statements as ordinary Python strings.

```python
cursor = conn.cursor()
query = """
    SELECT DISTINCT
        "security_ticker"
    FROM
        "securities";
"""
cursor.execute(query).fetchall()
cursor.close()
```

That looks a bit hacky, doesn't it?  It works great, though, and keeps the SQL query alongside your own code.  If you're using Pycharm, you'll even get SQL highlighting and code completion, which is extremely handy.

You absolutely can write all of your queries like this if you like - I know I did, for months.  This morning on [Hacker News](https://news.ycombinator.com/item?id=24130712), however, I came across an interesting new library called [`aiosql`](https://github.com/nackjicholson/aiosql).  This is really a variation on a theme - a very simple, clean API that allows you to keep all of your SQL statements in a separate `.sql` file but to trivially call them as named methods from within your Python application code.

From the example on the Github page, let's assume that you have the following SQL file:

```sql
-- name: get-all-users
-- Get all user records
select userid,
    username,
    firstname,
    lastname
from users;

-- name: get-user-by-username^
-- Get user with the given username field.
select userid,
    username,
    firstname,
    lastname
from users
where username = :username;
```

You can then use the following to load your queries into your Python code:

```python
import sqlite3
import aiosql

conn = sqlite3.connect("myapp.db")
queries = aiosql.from_path("users.sql", "sqlite3")

users = queries.get_all_users(conn)
# >>> [(1, "nackjicholson", "William", "Vaughn"), (2, "johndoe", "John", "Doe"), ...]

users = queries.get_user_by_username(conn, username="nackjicholson")
# >>> (1, "nackjicholson", "William", "Vaughn")
```

Three things you'll notice immediately are: queries are called according to the names given to them in the SQL file (in this case, `get-all-users()` and `get-user-by-username()`); you can include descriptive comments in the SQL file itself; and that queries can be parametrized using the `^` and `:<parameter>` notation in your SQL file.

Personally, I find this much cleaner and more legible than simply writing out my SQL queries as Python strings.  It also allows you to use your SQL files in other tools, such as `psql`, `pgcli` or `litecli`.

Several of the commenters on Hacker News mentioned similarities between `aiosql` and [`pugsql`](https://github.com/mcfunley/pugsql), another thin API that provides for the use of parametrized queries stored in a separate `.sql` file.  I haven't looked at that yet but intend to update this post in future once I have.

For now, I'm going to use `aiosql` in a bunch of my projects which don't require an ORM.

Now that we've finished, don't forget to close the database connection!

```python
conn.close()
```


On Hacker News


One thing that really distinguishes software from my previous career in the law is the strength of the online community.  [Hacker News](https://news.ycombinator.com/) has something of a reputation for attracting a certain "Valley Bro" stereotype, but my experience has been that that's wide of the mark.

Now, certainly reasonable minds may differ as to whether Hacker News is generally reprentative of the broader industry but, for me, it's been the one and only proxy for community that I've known - partly on account of living away from home as an expatriate, but also on account of the pandemic.  It's by no means perfect, but it would be hard to overstate just how essential it's been for my development to have access to the expertise embodied there.

Part of the appeal is that there's something for everyone, no matter where you are in your journey.  If you're just starting out, the linked articles and videos are often excellent but it's also the quality of the discussion around that content that brings the greatest benefit.  If you're more senior, there's plenty of content aimed at your level too, and plenty of experts willing to bring their expertise to beat on any number of advanced topics in the comments.

It's essential reading not only because it does a great job of surfacing canonical content on the craft of software engineering itself, but also because it does a great job of showing the pros and cons of peoples' personal experience is with the tooling they use.  Tooling is so, so important - way more important than I realised when I started on this journey at the beginning of 2019.  It can, in fact, be make or break, and having people that are willing to have reasonable debates about the qualities of different toolchains has been absolutely essential to me personally.

How else would I have known to settle on VS Code and Pycharm, but to also have a decent grasp of Vim keybindings?  To use Git for version control?  To stick with the Python standard library wherever possible, but also to use [requests](https://pypi.org/project/requests/)?  To learn Docker, Ansible and PostgreSQL.  To use Django and Heroku, Hugo and Netlify.  To buy my domains from Porkbun.  To use managed services [wherever possible](http://www.mooreds.com/wordpress/archives/3358).  To use Pandas but also not to underestimate the power of SQL and RDBMS.  To use Altair for plotting but also to understand matplotlib for its ubiquity.  To stick to GNU tools where possible.  But also to use [ripgrep](https://github.com/BurntSushi/ripgrep).

I could go on.  And of course, everyone's story is different.  And baked into the above is the assumption that I've made *good* choices - choices that won't come back to bite me at a later date.  Perhaps some of them are good, perhaps others not so much.  I wouldn't have even known where to start, if not for the community at Hacker News.


File paths for IO operations in Python


I've recently been spending a lot of time reading, thinking about and trying to understand different architectural paradigms in software design - specifically in Python - as I try to alight on one that makes sense to me personally.  That is probably for another (long) post, but a 2018 [discussion](https://news.ycombinator.com/item?id=18043058) on hackernews of [this](https://www.destroyallsoftware.com/screencasts/catalog/functional-core-imperative-shell) article, which also led me to [this](https://www.youtube.com/watch?v=DJtef410XaM) presentation by Brandon Rhodes - a big name in the Python space - all got me thinking about a particular problem I found that I was having when trying to write code in Python: how best to handle file IO operations.

My understanding of the take away from the discussion and sources is that one of the nice things about following a "functional core, imperative shell" paradigm is that you can place all of your file IO operations on the outside of your code - in the "imperative shell", so to speak - and find that as a result you have highly readable code with namespacing that naturally makes sense.

As a quick aside, I found loads of seemlingly irrelevant or outdated resources out there for file IO in Python, but [this](https://stackoverflow.com/questions/3430372/how-do-i-get-the-full-path-of-the-current-files-directory) was the best, most current one that I could find.

## File IO in Python

Note that we are talking here about reading from (as input), or writing to (as output), _files_, not packages or modules (which are 'visible' to Python by way of a unique name by which they can be imported).  I will look at good practices for package / module imports separately at a later date.

Let's take as our starting point a simple function that either writes to or reads from a file.  On the face of it, the first question is whether you expect to be using that function directly in the package in which it resides, or importing it from somewhere else.  In other words, is the function part of a library, or an app?  (However, we'll see in due course that the distinction is less important if you stick to "pure" functions and a "functional core, imperative shell" approach.)

For a library, by definition very often the file writing or reading functions are being imported into an app elsewhere and performing their input / output there.  So, if it's part of a library, keep your path state information out of the function altogether.  That is, keep your functions as pure as possible, with as few side-effects as possible.

```python

from pathlib import Path

contents = ['some-content', 'some other content']


def file_writer(target: Path):
    if not target.exists():
        target.touch()
    with open(target.as_posix(), mode="w") as outfile:
        for line in contents:
            outfile.write(f"{line}\n\n")
```

On the other hand, if the function is part of an app or normal package and needs to place a file in a specific location - for example, to be later read by another function - or read the file from a specific location, then you have to pass a path to that function somehow - either by specifying a default path in the function itself, or by passing a path into the function when it's called.

Here, too, let's keep the path information out of the function body itself, and take a "functional core, imperative shell" approach: in any given module, keep your functions pure but then set up and pass your path variables in your __main__() function.

```python

from pathlib import Path

contents = ['some-content', 'some other content']


def file_writer(target: Path):
    if not target.exists():
        target.touch()
    with open(target.as_posix(), mode="w") as outfile:
        for line in contents:
            outfile.write(f"{line}\n\n")


if __name__ == '__main__':
    TARGET_FILE = 'target-file.md'
    TARGET_PATH = (Path.cwd() / 'resources')
    file_writer(target=TARGET_PATH / TARGET_FILE)
```

A question arises here then: how do you describe the path you want?

## Specifying Python paths

Here are the choices:

```python
# relative to working directory (which will change depending on from where the function is called)

TARGET_PATH = Path.cwd() / 'resources'

# relative to file system location of the file containing the called function (which is stable over time)

TARGET_PATH = Path(__file__).parent.absolute()

# absolute

TARGET_PATH = Path.home() / 'dev' / 'projects' / 'pvmon' / 'pvmon' / 'resources'
```

It's unlikely that you want to use an absolute path - unless you're trying to interact with files outside of your package and in a location that is stable over time.  I would think this would be more common for reads than for writes.

Using a path relative to the working directory initially looks problematic, because it appears to move around so much - sometimes it refers to the directory containing the file containing the function that you're calling; sometimes it will refer to a parent or sibling directory of that directory (if it's called from some other part of your package).  However, on closer inspection the answer is perhaps simpler than it looks: if you're following a "functional core, imperative shell" model, regardless of the file system location of the various functions, you will be calling all of them from the same location: that is, your `main.py`, or `app.py` file (or whatever you call it) - that is: the module that contains your `main()` function:

That being the case, your `Path.cwd()` (and any sub-directories hanging off it) will always refer to the same directory - the directory containing your `main()` function.  You can pass it to any of the functions invoked by your `main()` function at runtime and they'll all be dealing with the same directory.  Accordingly, you would think that the vast majority of cases would be using this relative model.

What about the final choice - path relative to the file containing the function being called?  I honestly can't see a use for this at the moment.  If you're following "functional core, imperative shell", it seems like you shouldn't need to do this.

## In conclusion

Keep your functions pure, pass any path parameters to those functions from your "imperative shell", and be sure to correctly specify your paths depending on your use-case.

The [textbase](https://github.com/alexclaydon/textbase) repo on my Github page is an example of me experimenting with a "functional core, imperative shell" paradigm. 


Configuration management for Python projects


## The problem

It's usually necessary when writing any non-trivial Python code to implement some way to easily configure the way it's interpreted.  This post on the tools I reach for to achieve this.  As I'm writing this for myself, it's opinionated.  On the other hand, I hope it's also helpful to anyone else coming at configuration management for the first time and wondering how to handle it.

The approach depends on the type of code you're writing and the intended deployment environment (the [Twelve-Factor App](https://12factor.net/ is an excellent starting point in this regard), but I've tried to settle on some sensible defaults.  Most of the below assumes you are on Ubuntu (or any flavour of Linux, really) and have direct access to the command line.

Before we kick off, it's probably helpful to distinguish configuration management, for secrets management.  This post deals strictly with configuration management.  Secrets management deserves a separate post.  Even though in substance you may end up using (parts of) the same stack to handle both configuration and secrets, the two have very different requirements.  We'll look at secrets management (perhaps using Mozilla SOPS or Hashicorp Vault) at some point in the future.  I'll be sure to propose an approach there that integrates cleanly with the configuration paradigm described here.

There are potentially a number of different approaches to managing configuration.

### In-code variables

Stored directly in the code itself.  This may be convenient during development - or for variables in a production deployment that rarely change.  As they will be checked into version control, you will have trivial access to earlier settings.  But there downsides, too.  It can be hard to make changes once in production, and you have to be confident that none of the configuration settings are potentially sensitive (keeping in mind of course that we're excluding secrets from the current discussion).

Accordingly, to the extent possible, you should avoid keeping any configuration data that is either sensitive, or expected to change, in-code.  On the other hand, you shouldn't be discouraged from keeping configuration that is relatively stable over time in your code itself - there's nothing inherently wrong with this, as long as you understand your production needs.

### Environment variables

Environment variables (envs) are variables that are present in your execution environment - be that bare metal, a VM, or - for example - a Heroku deployment environment.

These can be:

- set manually in a bash session from the command line (`MY_KEY='my_value'`) and then imported into your execution environment using `os.getenv()`.  Note, however, that envs set in this manner are available only to the execution context of the terminal session where you set them (and any child processes spawned from that session).  This may be fine if you're manually running Python scripts from the command line, but is unhelpful for most other execution contexts.  In addition, envs set in this manner are ephemeral - they will be lost at the end of the session.

- sourced by bash automatically from any file loaded into the run-time environment (for example, a user's `~/.profile` file)[^1] then loaded into Python using `os.getenv()`.  This is better than setting envs manually at the command line in the sense that (depending on where you put them) they may be loaded at the initialisation of each session.  On the other hand, you need to ensure that those envs are in fact stored in the correct file for the execution context you are targeting - there's no point storing them in a file that isn't sourced by bash in the context you are targeting.  For example, if you are using `cron` to schedule the running of a script, simply placing any relevant envs in your `~/.profile` file won't alone be sufficient.

- sourced from a file using an `.env` extension (usually located in the project folder itself) with the [python-dotenv](https://pypi.org/project/python-dotenv/) package using `load_dotenv()` then `os.getenv()`.  This is a tidy solution in that you don't need to think through potentially convoluted Linux user session details.  Assuming the file is checked into version control, it travels with the rest of the repo if your deployment context changes, but you will need to ensure that it doesn't contain any sensitive configuration details (and, of course, this approach is not suitable for storing secrets - although you may be able to do that if you opt not to add the file to your Git repo).

### Arbitrary configuration files

Configuration can also be loaded from YAML (or JSON or INI) files: there's nothing special about these file formats (other than, perhaps, their popularity).  Any flat file arranged in pairs of keys and values will do, as long as there's a suitable Python library for importing it at run-time (namely, as a hash map or `dict()`).  Of the three, I find YAML is the most human-readable but that's obviously subjective.

Two potential downsides to using this approach are that it will expose any sensitive configuration settings (assuming it's stored in the repo alongside the rest of your code), and that you'll have to write the import code yourself, including any code to provide for a hierarcy of precedence between any configuration settings in the file that conflict with any other settings elsewhere (for example, if a given setting is set both using environment variables and in the imported `dict()`).

### Passed at the command line

Configuration can also be passed by the user invoking the code as command line arguments and options at run-time.  Obviously this paradigm is most relevant to those writing code intended to be run directly by a human sitting in front of a machine.  On the other hand, you may have some code that is sometimes run by a machine, and sometimes by a human, and clearly in those scenarios you will need to provide that any arguments and options passed at the command line override any existing, conflicting settings established by way of environment variables or configuration files.

## The solution

For most cases (and setting aside the needs of a CLI for the moment), my preference is to store configuration settings as environment variables - sourcing them from either or both of the execution context (for example, your `~/.profile` file) and an `.env` file using python-dotenv.  Note that envs sourced from an `.env` file using python-dotenv can be configured to either respect or override existing system envs by passing the `override=True` flag to `load_dotenv()`.  In this way, you can seamlessly draw on both sources of envs in a transparent way and with clear behaviour.  The python-dotenv library is also a great fit with Django.

On the other hand, if for some reason you want to support the use of other configuration files (YAML, JSON, INI) alongside some mix of environment variables, you may want to reach for something which provides a clear precedence hierarchy between those different sources: we need to make sure that the interpreter knows which variable should take precedence when there are conflicts.

Rather than writing custom code to handle this, I'm going to do something here that's usually against my better judgement, and recommend a small and relatively new open source project called [gila](https://gitlab.com/dashwav/gila).  As per the project page, the following are some of the key features:

> - Allow default values to be set for each config key
> - Automatically find config files on multiple paths
> - Load in environment variables automatically that have a specific prefix
> - Support most popular config languages: yaml, toml, json, properties files, hcl, dotenv
> - Singleton pattern available for ease of use in most applications
> - Hierarchical loading for non-ambiguous results.

Gila allows you to corral all of your various different bits of configuration into one place and establish a clear order of precedence.  Gila documentation is [here](https://gila.readthedocs.io/en/latest/index.html).

[^1]: See the official Ubuntu documentation [here](https://help.ubuntu.com/community/EnvironmentVariables#System-wide_environment_variables) for a more thorough discussion on where to store envs on a Linux.


Deploying a Django project to Heroku


I recently went through the process of deploying a Django project to Heroku.  This is my first Heroku deployment.  I had also considered a number of alternatives including [dokku on Digital Ocean](https://marketplace.digitalocean.com/apps/dokku), [piku](https://github.com/piku) (also on Digital Ocean) and deploying to AWS Lambda using [Zappa](https://github.com/Miserlou/Zappa).  While all of these are interesting in their own way (not least of which, in the case of Dokku / Piku is the absence of vendor lock-in) - and I intend to try each of them in due course - I wanted to take a look at Heroku first, giving that it comes up regularly on hackernews.

But let's backtrack for a second: why not simply deploy the Django project to bare metal - perhaps one of the many Raspbery Pi 4s I have lying around - or to a generic Digital Ocean droplet?  Because while there are a range of excellent reasons to do that, I am trying to be very specific about targeting where I want to sit in terms of expertise in the stack.  I am tring to make sensible decisions about which parts of the stack I can hand off to a managed service, and I think Heroku and Dokku, in particular, are solid, 'boring', choices - I'll save my [innovation tokens](https://mcfunley.com/choose-boring-technology) for something else.

## The deployment

I followed [this](https://blog.heroku.com/from-project-to-productionized-python) article, which I find to be generally very good, although there were a couple of hitches for me and my particular stack that I thought would be helpful to share for anyone trying to do the same thing.

Here's the stack:

- Django 3.0.8
- Github-hosted project repo
- A Django project created with Pycharm's Django project template
- A domain hosted on porkbun.com

And here are my notes:

- One minor addition that you have to make for any app that's been created using the Pycharm Django project template - and that's not diectly covered in the article - is to add `STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')` to your base settings file (assuming that you're following the convention recommended by the Heroku article, that would be located at `mysite.settings.base.py`) and to make a `staticfiles` directory at your project root (i.e., the directory that contains both your project and your app subdirectories).  These changes are required to ensure that the `whitespace` middleware works properly at build time.  You may also need to put a `.gitkeep` in that directory to ensure that git picks it up even when it's empty.

- You will need to have run `heroku config:set ALLOWED_HOSTS=".herokuapp.com"` at the command line in order for the call to `env.list('ALLOWED_HOSTS')` in your `mysite.settings.heroku.py` settings file to correctly permit web access to the deployed app.  See [this](https://stackoverflow.com/questions/23252733/i-get-an-error-400-bad-request-on-custom-heroku-domain-but-works-fine-on-foo-h) Stack Overflow question.  Don't forget to also run `heroku config:set ALLOWED_HOSTS="<your-domain.com>"` for any external domains you point at your project.

- You can enable push-to-deploy for Github-hosted repos by going to the "Deploy" tab in the Heroku admin page and simply connecting it to the repo, then separately enabling automatic deployment (for pushes to your production branch only, obvs).

- Before being able to access the admin panel on the live site, you'll need to have added a new superuser; invoke `heroku run python manage.py createsuperuser` from the command line and enter your details.

- To add a Porkbun-hosted domain to a Heroku app, see [this](https://kb.porkbun.com/article/90-how-to-connect-herokuapp-with-porkbun) article, but note that in addition to what's set out in there, you also have to delete the existing `ALIAS` record for the domain that points at `pixie.porkbun.com` (i.e., you need to not only add ALIAS and CNAME records, but also delete one existing ALIAS record (assuming your domain is still configured with the Porkbun default settings).

- After adding any custom domains, don't forget to provision a SSL certificate (using the automated ("ACM") option on the Heroku admin panel).

- Once the SSL certificate is provisioned, you'll want to enable routing of all http requests to https.  See [this](https://help.heroku.com/J2R1S4T8/can-heroku-force-an-application-to-use-ssl-tls), but in short you need to add the following two lines to your `base/py` settings file:

        SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
        SECURE_SSL_REDIRECT = True

- You should consider at what stage of your deployment pipeline you will have your tests run.  In addition to local tests, I'm inclined to require testing using Github actions (as a pre-push hook) as that approach generalises to all of my repos, not just those destined for deployment on Heroku.


Syncing project state across multiple machines


This [article](https://tonsky.me/blog/syncthing/), and the [discussion](https://news.ycombinator.com/item?id=23537243) over at hackernews, reminded me that I wanted to write a short note about how I keep all of my project files in sync when developing on more than one machine.

Turns out this is [non-trivial](https://news.ycombinator.com/item?id=23387305).  Well, the solution is trivial but getting there wasn't.  There are a few different ways to approach this but until I alighted on Syncthing, none of them were satisfying.

## My dev setup

I have a home dev machine (running Ubuntu 20.04 LTS), but I also have a Macbook Pro (running Mojave) that I use when travelling or if I just want to move around while working or get out of the house.

My IDEs of choice are VS Code and Pycharm.  All of my project files are in my home directory.  Most of the time I'm in front of the Ubuntu machine, but when I sit down to use the Mac I want the state of all project files to match that on the desktop.

## False Starts

My initial idea was to just sync everything using Github.  Push my work before switching machines then pull down the latest commit.  The problem with this is two-fold.  First, every time you switch machines, you have to do a commit, a push and a pull.  This quickly becomes tedious.  Second, it results in a git log full of commits with no purpose other than that you were syncing the state of your files.  There are ways to streamline it &mdash; for example, using a post-commit hook to auto-push to the remote repo &mdash; but it is clearly square peg, round hole.  As I'm sure plenty of more experienced devs could tell you: Github is not a syncing or backup service and if you're using it as such you're pushing uphill.

VS Code and Pycharm both have remote development modalities (although the approach is quite different).  However, the further you dig into these the more you realise that they are probably not what you want if your circumstances are like mine &mdash; namely, two local machines with equal primacy in the development process.

VS Code has an excellent SSH [extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh), but this is really for different circumstances &mdash; where you have a single remote development instance (e.g., VPS, container) and you want to do all your development in that environment, treating any local machines as dumb terminals.  There are lots of reasons why that modality might be exactly what you want, but it wasn't what I wanted.  I'm usually developing for both platforms and regularly working on both helps to ensure that platform-specific bugs are caught early.  Using both platforms means I remain in touch with the stack as a whole, too.  There are plenty of other reasons but they're not directly relevant &mdash; only you know if these are your circumstances, too.

Pycharm's take is a "remote deployment" model (Tools -> Deployment).  You first specify an SFTP connection to the remote host.  Your Python project files are then uploaded to the remote and kept in sync each time a save is triggered (you can see the upload activity in the console).  Note that this is not a wholesale two-way directory sync.  First, a number of file types (for example, any SQLite databases) are not sent to the remote, so you do not have an exact duplicate of project state across machines.  Second, the sync is one-way.  You can't just sit down in front of the "remote" machine (in this case, your second dev machine), go to your home folder and work on your synced project files &mdash; they are tucked away in a folder created and managed by Pycharm specifically for the purpose of hosting the SFTP connection from another machine.  The directory structure in no way mirrors a normal local home folder structure.  The "remote" folders really aren't designed to be accessed in this way.  That setup falls far short of what I wanted, which is step-in, step-out perfect parity between both machines &mdash; including sane directory structure.

I've seen it mentioned a few times that an alternative to using the native remote development capabilities of VS Code or Pycharm is to mount a remote filesystem containing your project repos on your local machine, creating a single source of truth.  There are significant practical issues with this approach, however &mdash; the machine hosting the repository has to be awake when you're working on the other machine; mounting and unmounting the remote share cleanly through the rebooting (planned or unplanned) of either machine throws up issues; you can't work from the secondary machine without a network connection.

## Syncthing

Instead, the solution that works best for me is robust file sync using Syncthing.  See the article linked at the top of this post for a good look at getting everything set up and linked &mdash; I'm only going to focus on how it works in the context of two (or more)-machine development setups.  Note that &mdash; unlike the linked post &mdash; I run Syncthing in a Docker [container](https://docs.linuxserver.io/images/docker-syncthing) for convenience &mdash; it allows me to bundle it up along with other services into a single, version-controlled docker-compose.yml file.  YMMV.

All project files are put into folders monitored by Syncthing (in my case, I use a single ~/dev folder and a bunch of subfolders under that).  Syncing isn't instantaneous, but it's plenty quick (I believe you can change the polling rate if you're so inclined, although I haven't found it to be necessary).  In the case of text and database files that make up much of my work, I haven't had any issues with file locking: I can have any given file open on both machines and if I make a saved change to that file on one machine, I can walk into the other room and find the file already updated on the other, the most recent change visible in the open VS Code / Pycharm instance.  

Clearly in order for the machines to stay in sync, any machine on which you make a change must be awake and connected to the network when you switch on the second machine.  That second machine will then connect to the first and and bring its state up to date.  Note that this is a less onerous requirement than for an SSHFS setup, where the machine hosting the repository must be awake even if you're working exclusively on the other machine at the time.

As with any sync protocol, if you make changes on one machine that are not synced to the other (if, for example, you switch off the machine hosting the changes before the other machine has pulled the sync), you will have to deal with file conflicts.  This is an inherent limitation of any file syncing paradigm, though &mdash; it's not Syncthing-specific.  The same problem would arise if two machines syncing files using Dropbox found themselves in a similar situation.  Syncthing has excellent [documentation](https://docs.syncthing.net/) and a graceful approach to resolving sync conflicts.

While in pratice both of my dev machines are usually switched on during the work day, for convenience and redundancy I added a third Syncthing node to the network: a power-sipping Raspberry Pi 4-powered, Ubuntu 20.04 LTS-based NAS &mdash; to ensure the availability of up-to-date files at all times.  The Raspberry Pi is on 24 hours a day, 7 days a week and costs just a few dollars a month to run.  It also hosts a bunch of other ARM64-architected homelab containers in addition to Syncthing (Plex Media Server, Pihole, etc.) which also require 24-7 availability so the additional marginal trouble is neglible in my case.  I would highly recommend this setup as cheap and reliable, but it's by no means required for Syncthing to sync your project files effectively.  In practice, adding additional nodes to your Syncthing network is trivial &mdash; there's no reason you couldn't scale up to 3, 4 or even more dev machines if your workflow somehow requires it.

## A note on using Syncthing and iCloud together

Finally, I just want to add a small note about using Syncthing on folders that are also synced using Apple iCloud.  I have found them to work very well together &mdash; no syncing conflicts so far.  Obviously iCloud isn't needed at all to sync files as between two (or more) machines in a dev environment &mdash; regardless of how many of them are Macs. The main reason you might want to use this dual-sync setup is if you need access to your project files on your iOS devices (which cannot run a Syncthing client).  Pro-tip: the MacOS iCloud folder is not easy to find from the command line, which means it's not easy to add to Syncthing.  Here's the [correct](https://forum.syncthing.net/t/sync-path-in-icloud-folder/12194/3) way to enter it:

    ~/Library/Mobile Documents/com~apple~CloudDocs/Documents
