Weeknotes 2020.09.14

Things I've been looking at in the week beginning Monday the 14th of September, 2020.

Personal Data Warehouse

I watched and thoroughly enjoyed Simon Willison's talk at PyCon AU - https://2020.pycon.org.au/program/73uk8x/, which I missed live probably because my internet was playing up. He talks about a number of things and in particular Datasette and it's ecosystem - https://github.com/simonw/datasette.

I swim to keep vaguely fit. I try and swim at least 3 times a week and sometimes even achieve that target. To keep track of my progress against my goals and because I'm a nerd I have a CSV file where I record key details of each swim. Turns out I've been doing this since August 2008.

As a nice compact data source I often use this file to try out tools and technologies.

I loaded it up into Datasette but ran into a snag - my date and time columns don't display very well in the Datasette web interface. That's because I had used Simon's csv-to-sqlite module to create my database, but it (quite rightly) doesn't type the Sqlite columns because the Sqlite engine itself ignores datatype declarations.

Meanwhile I've been using this data to try out Metabase and to build dashboards on my laptop. Metabase introspects your database and does take into account the datatypes declared for columns.

To get typed columns in my database I built it by hand and use my own csvload script to populate the Sqlite database. I feed that database to Metabase and build dashboards like this;

Metabase_Dashboard.png

If you are looking for a straight forward business intelligence tool I highly recommend taking Metabase out for a spin.

Interesting Media

(c) Getty Images The lovely people at the Sydney Opera House have put up a video of a Paul Weller concert they hosted in 2018. I was there and the only thing the video doesn't capture is just how loud it was. Bloody magnificent. A great choice of tunes as well, just imagine having a back catalogue that let's you play 'Eton Rifles' as the third track.

Weeknotes 2020.09.07

Things I've been looking at in the week beginning Monday the 7th of September, 2020.

Android 11

I woke up on Wednesday morning to the news that Android 11 has been released. It was on my phone about an hour later. That's pretty seamless. Although it does help that I have a Google Pixel and that they are always the first phones to receive Android version upgrades. So far I like the media controls in the notification panes and the ability to customise the items that appear when I long press the power button.

The only glitch I've found is with the application switch line on the bottom of the screen interrupting interactions with the very bottom of the active application. I think relates to my use of Action Launcher, which I'm sure will be fixed very quickly.

Bulma

I've built a couple of small web sites using Bootstrap, but so has everybody else. In my constant quest to be slightly different to everyone else I came across Bulma. It's a pure CSS framework that allows you to build modern responsive web site without masses of Javascript. It's also really simple to pick up - all you need to do is add some classes to your existing HTML code and you have a really nice looking site.

I re-skinned one of my projects using Bulma in just a couple of hours. It looked great out of the box. But then just as I was wanting to change the colour scheme I came across Bulmaswatch and found some really great, ready to go colour and presentation themes built on top of Bulma. If you're building web pages I highly recommend checking out Bulma

Sorted, kind of

This week, thanks to a Real Python tutorial I learned the difference between the sorted built in function and the sort list method in Python. In the past I reflexively reached for sort when I wanted to order a collection of things, this article has convinced me to change that for the sorted built-in function. Not least because it is much more flexible and can be used with any collection, not just lists.

Interesting Media

Couch Convention Last weekend was a new kind of experience for me. Marillion held an online version of their regular fan conventions. This one was called the couch convention. It's always great to see what they come up with, even if I did get up ridiculously early on Sunday morning to watch the concert video from the Albert Hall in 2018. Top band, top weekend - at least the bits I saw.
Long Hot Summers There's a new Style Council album - alright it's not new but a career spanning anthology - at the end of September. I'm in the queue.

Weeknotes 2020.08.31

Things I've been looking at in the week beginning Monday the 31st of August, 2020.

Note Taking Apps

I am always trying to look at the tools and techniques I use and see if I can improve them. I spend a lot of time writing notes and in recent years I've moved from paper notebooks to tools and apps on my laptop, phone and/or tablet. I've tried lots of different tools and I know that there are many more out there. The applications that I have spent the most time with are One Note, Joplin, and Bear. Although I have recently been dabbling with Inkdrop and Obsidian.

My main criteria for note taking apps are that I can write in Markdown, synchronise my notes between laptops, my iPad and ideally my phone as well. That it has a sensible way to link between notes - and bonus points for linking to a section of a note - supports Vim keybindings, has a sensible way to arrange and navigate my notes and that the writing and reading experience (really the editor and the user interface) are pleasant to use.

At the moment I'm splitting my time and attention between Bear and Obsidian. Bear doesn't by any means support all of the features on my list but it absolutely nails the experience and note organisation and that's why it is still my primary note repository. The more time I spend with Obsidian though, the more I appreciated it. If I could only find a way to edit my notes on my iPad I may even consider switching to it full time.

Inkdrop had a new release (v5) this week so I did take a quick look but it still has a confusing approach to note organisation (notebooks, folders, and tags) and doesn't allow for easy/simple linking between notes.

Interesting Media

I quite enjoyed this Reddit thread which posed the question - "Non-Americans of reddit, what is your genuine reaction to what’s going on in America right now?". I haven't read the whole thing but the UK/US analogies are quite good, particularly the one about your boat being on fire but you look to your left and see a cruise ship on fire with the captain running around pouring petrol on everything.

Weeknotes 2020.08.24

Things I've been looking at in the week beginning Monday the 24th of August, 2020.

Technology

Add ins (plugins) for Microsoft Outlook

We are an Office 365 shop at work. I, for contrary reasons mainly, use a Mac. I recently felt the need to disable an Outlook plugin that I had added a while ago. I went looking for instruction to manage plugins in Outlook. Blimey. There is no way to do this in the client that I run on my Mac. In the end to deactivate the plugin I had to

  1. Log in to Outlook web mail
  2. Enable a setting to see the 'Get Add-Ins' button on the web interface (settings -> 'View All Outlook Settings' -> 'Customize Actions' -> tick the box next to the icon for 'Get Add-ins')
  3. Select a message in my inbox, find the cunningly hidden 'Get Add-Ins' button (click the three dots and it should be in the list), select 'My Add-ins' then click the very non-obvious button to deactivate the plugin.

Simple, really. A bit like the instructions in the Hitch Hikers Guide to the Galaxy to find plans to demolish Arthur Dent's house

Halfcooked Utilities

I have an old and venerable collection of code snippets that I keep in a (very) private repository. It started as a Subversion repository about 15 years ago and I've just added bits and pieces as I have gone along. I realised this week that there are 3 Python modules in it that I reuse on a regular basis. So I've copied them into a fresh project. I revamped one of the modules to provide a better interface and fixed and revised all of the unit tests, I need to do the same for the other two. The curious can find hc-utils on my server.

Interesting Media

Funky Moped Someone at work re-acquainted me with this classic from 1975 - Funky Moped and that sent me down a YouTube rabbit hole of Jasper Carrott videos.
Batflowers (Megan) Washington has a new album out today, it's called Batflowers, here's a review

Weeknotes 2020.08.17

Things I've been looking at this week which started on Monday the 17th of August, 2020.

Technology

Analytics Workbench

We want to give our actuaries a workbench to perform analysis on our data and additional data sets that they acquire. This is mainlt straight forward statistical analysis but they are all starting to use machine learning. The team all know some combination of R, Python and/or Matlab so we naturally leaned towards Jupyter Notebooks

Microsoft have (had) a preview service called Microsoft Azure Notebooks which looked perfect; sign in, create a new notebook and you are away. Given that our data platform runs in Azure this would enable easy and secure interaction with our data plus providing flexible compute capacity for process intensive workloads.

Sadly that preview is being retired in October so we looked at the suggested alternative which is called Visual Studio Codespaces which are purported to support notebooks. This offering essentially just links Visual Studio Code (running on the desktop or in the browser) with a Docker container that is somewhat pre-configured to execute the code that is entered and managed by Visual Studio Code. This sounds ok, if a little complicated in theory. Except we couldn't - in the limited time available for evaluation - get it to work with one of the installed Python versions and install 3rd party modules into that environment. We invested a few hours in setting up the Azure components but didn't feel that this is something that we can roll out to people who aren't died in the wool developers.

One of the team then suggested Google Colabatory their implementation of Jupyter Notebooks. It works like a dream, we had a number of people up and running with nothing more than a few mouse clicks - most of which were logging into their Google account. But, even though this approach would have been great for our colleagues it wasn't possible (or we couldn't work out how) to link this securely to our data platform in Azure. I'm definitely going to be using "Colab" for a few personal projects though.

Our current focus is The Littlest JupyterHub running in a VM at Azure - https://tljh.jupyter.org/en/latest/install/azure.html. That's next week's adventure though.

Pandas

I keep dipping my toe into the Pythonic data science tool of choice. This week I've been looking at creating dataframes from CSV files, setting appropriate data types and grouping for sums and counts. If I can clean up my code I might even try publishing a Notebook or two here.

Issue, Incident and Problem Management

One of my responsibilities at work is to make sure that our policies and procedures are up to date and working effectively. This week we had a problem with a production system that caused degraded service to users over a day or so. I asked the team to log this as an incident and there was some confusion about what this was and what the procedure was for logging and managing incidents. So I reviewed our documentation and realised that this isn't spelled out very well.

To address this I'm writing an issue, incident and problem management process for the IT team to follow. The first problem is, as always, one of terminology. ITIL references Incidents that become Problems, our corporate risk management process has an incident management policy that covers what ITIL calls Problems. I'm going to defer to the corporate risk management process as no one outside my team (and many inside) knows anything about ITIL. So the process will talk about:

  • Issue - we think something is wrong, these are registered in the service desk and may; require administrative action (reset password, etc), be a bug that needs to be put on the relevant development team's backlog, or something more serious.
  • Incident - a systemic issue with the operation of one or more systems or services that is affecting our ability to serve our stakeholders (customers, intermediaries and staff), and may be identified by one or more Issues, by system monitoring or by notification from a partner (e.g. a SaaS application is experiencing problems)
  • Problem - the world is falling in our heads, really just a more severe classification of an incident.

Now all I've got to do is write up what to do with each of these interruptions to our services. That can be next week's job.

Interesting Media

Real Tigers book cover

I've started reading Mick Herron's Real Tigers

Steve Hogarth

You should definitely read all about the bloke who isn't Fish

Back to Blogging

I haven't stopped, I've just stopped writing here. So now it's time to attempt a comeback, and not to be too hard on myself if this is a once in a while proposition. I haven't been working on a big secret project, or at least nothing that will be ready to reveal for a decade or two. Most of my technology and free time has been spent on what I would call 'tinkering'. I have tried out a lot of tools, technologies and small experiments but nothing has stuck.

So rather than wait for something serious and significant to write about I'm going to try and keep track of the things I have been trying in the hope that they will help other people in similar situations or act as a deterrent from wasting your time on such frivolous pursuits.

To that end I'm going to attempt to keep some weeknotes here, inspired by but probably not nearly as useful as those published by the likes of Simon Willison and Dan Catt.

For reference the subjects I've spent most time fiddling about with are Markdown note taking tools, various Python libraries, and trying to run my own services on my own server very much in the vein of Danny O'Brien's living on the edge discussions from a mere 12 years ago. Who says I'm not up to date?

DNS Configuration

This post answers the question "how do I host my email at one provider and my web site at another?". In particular it shows how to set up DNS at your domain name registrar, email at Fastmail and web hosting at WebFaction.

Most of the how to DNS configuration guides on the web show how to point to a single hosting provider for web, email and other services. I'm awkward and wanted to separate my email from web hosting. So by a little trial and error I came up with this set up for my personal domain.

First at your domain registrar (I use hover) set the domain namervers to Fastmail, I've used 2 of their name servers.

Nameservers

This delegates control of all DNS activity for the domain to Fastmail. Without any other changes this will allow Fastmail to manage email and serve a limited web site from their hosting option. I choose to host my web site at WebFaction and so to delegate this from Fastmail we use their domain configuration dashboard and set up DNS 'A' records to point to my WebFaction host.

A records

The final step is to configure WebFaction to accept and serve your domain. If they don't recognise incoming web traffic you won't be able to server a web site on your domain. Here's mine set up in domains section of the WebFaction control panel.

Control panel

RSS Readers in 2016

After the great Google Reader shutdown of 2013 I switched my feed reading to a self-hosted instance of Fever and I've been a happy user ever since. I didn't mind that it wasn't open source, I was glad to spend the money to support a developer building a great product. Sadly now that support and developed has ended. I'd like to thank Shaun Inman for a great product. The end of development and support for Fever don't necessarily mean that it will stop working at any time in the near future. But I took this as an opportunity to look around at the state of the RSS reader world and see if there isn't a better tool for me to use now and for the future.

My requirements are simple. It must run on a server that I control, be accessible using a reasonably modern web browser and support import and export of feed listings via OPML. Theere are bonus marks if the Fever API is supported (I've got Reeder running on my iPad) and for ease of installation and use. I'm not a sophisticated feed reader but I've accumulated quite a collection of sites that I follow over the years. Fever is currently following 370 feeds but I can use any transition as an opportunity to trim away the dead wood.

After a little research and with some help from OpenSource.com I narrowed my search to three possible solutions: selfoss, miniflux and Tiny Tiny RSS

My evaluation strategy was pretty simple - install the tool on an Ubuntu virtual machine and if it looks promising do a fresh install on my web server and try the tool for size for a few days.

Selfoss

This topped a few of the searches I did and was the first solution I installed. Well, I say that but I haven't managed to install it on my VM. I got to the stage of posting to the forum asking about error messages that reported missing PHP .so files before I gave up. In this particular case it looks to me like the install expected PHP 5 but on the Ubuntu 16.04 the default is PHP 7. I may be wrong, I'm certainly no PHP guru, but clear installation requirements and instructions are nice. I may revisit selfoss again and see if I can't get over my installation problems.

Miniflux

Miniflux takes the river of news approach and applies it to a minimalist presentation. Installation was fairly straight forward and I managed to import my feed collection quite quickly. The folder structure that I had set up in Fever didn't make the transition - it isn't part of the OPML contents - so I took the opportunity to trim my feed list and put them into a small number of groups.

Updating the feeds is simple in the web UI - just press the 'refresh all' link on the subscriptions page. Then go to the 'unread' page and scroll through the articles. There are some nice simple keyboard shortcuts to aid navigation and reading.

There are 2 puzzles I still haven't cracked with Miniflux. I can't get the automated feed refresh working - probably something to do with the way I have set up PHP - and I still haven't found out what the 'groups' are used for. Apart from this minor quirks I am liking the philosophy and execution of miniflux and I think it's going to be my feed reader from now on.

Tiny Tiny RSS

Tiny Tiny RSS has been around for a few years and receives rave reviews on the web. Installation on my virtual machine was pretty straight forward and I had it up and running without any problems. The interface is more complex than Miniflux, and even than Fever, with nested folders of feeds and a nice display of each article.

Where I struggled with Tiny Tiny RSS was refreshing the feeds. The documentation says that this is best done via a cron job but I couldn't get the command working. I'm also not mad keen on running a PostgreSQL server on my web host as I'm not knowledgeable enough to keep it running securely.

So for the time being I'm going with Miniflux

Simple Log

I found myself putting the same few lines of code at the top of every script or module that I wrote. It went something like this:

import logging
MESSAGE_FORMAT = "%(asctime)s %(levelname)s:: %(message)s"
DATE_FORMAT="%Y.%m.%d %T"
logger = logging.getLogger(self.__name__)
logger.setLevel(logging.INFO)
formatter = logging.Formatter(MESSAGE_FORMAT, DATE_FORMAT)
ch = logging.StreamHandler()
ch.setFormatter(formatter)
logger.addHandler(ch)

Because the default simple logging provided by the standard library logging package doesn't do everything I'd like. I prefer that messages contain the time that they were generated. I prefer that the default logging level is set to INFO. I don't really care to see the log name in each log message.

When I built bigger applications I found myself putting this code in a log.py file and including it with the application source code. Then I wanted to change the log format. Which meant editing quite a few copies of the same file. So I decided that was a bad idea and I would turn my simple few lines of code into a Python module and publish it on PyPi. That way I could include a single copy of the code in each of my modules or applications with a simple pip install command.

The last time I did some serious packaging work this all rather tricky but thanks to the most excellent work of the Python Packaging Authority publishing your own Python module is a breeze these days.

So say hello to simple_log. It's available from PyPI now - https://pypi.python.org/pypi/simple-log/

The documentation is on this site - http://halfcooked.com/code/simple_log/ and the source code lives on BitBucket at https://bitbucket.org/andy47/simplelog

It's designed to be a simple log module that can be incorporated in any Python application and used as simply as:

>>> from simple_log import get_log
>>> log = get_log()
>>> log.info('Information message')
2016.12.12 13:30:30 INFO:: Information message

No more worrying about streams and formatters just a nice simple way to get a log object and log messages from your code. The code is licensed under the MIT license so corrections, additions and praise are always welcome.

Generating Diceware Passwords in Python

Today I'm going back to a theme from a post I wrote last year and looking at generating passwords with my favourite programming language. A tweet from Simon Brunning pointed me to Micah Lee's article at The Intercept and my first thought was to write a function to do this in Python. So here it is;

import random
word_dict = {}
passphrase = []
with open('diceware.wordlist.andy.txt') as f:
    for line in f.readlines():
        index, word = line.strip().split('\t')
        word_dict[int(index)] = word

for words in range(0, word_count):
    this_index = 0
    for position in range(0, 5):
        digit = random.randint(1, 6)
        this_index += digit * pow(10, position)
    passphrase.append(word_dict[this_index])
return ' '.join(passphrase)

In terribly bad form I've hard coded the diceware word list file name. I took the English word list and converted it to a plain text file for easier processing. The original will probably work just as well, I just haven't tested it.