Weeknotes 2020.08.17

Things I've been looking at this week which started on Monday the 17th of August, 2020.


Analytics Workbench

We want to give our actuaries a workbench to perform analysis on our data and additional data sets that they acquire. This is mainlt straight forward statistical analysis but they are all starting to use machine learning. The team all know some combination of R, Python and/or Matlab so we naturally leaned towards Jupyter Notebooks

Microsoft have (had) a preview service called Microsoft Azure Notebooks which looked perfect; sign in, create a new notebook and you are away. Given that our data platform runs in Azure this would enable easy and secure interaction with our data plus providing flexible compute capacity for process intensive workloads.

Sadly that preview is being retired in October so we looked at the suggested alternative which is called Visual Studio Codespaces which are purported to support notebooks. This offering essentially just links Visual Studio Code (running on the desktop or in the browser) with a Docker container that is somewhat pre-configured to execute the code that is entered and managed by Visual Studio Code. This sounds ok, if a little complicated in theory. Except we couldn't - in the limited time available for evaluation - get it to work with one of the installed Python versions and install 3rd party modules into that environment. We invested a few hours in setting up the Azure components but didn't feel that this is something that we can roll out to people who aren't died in the wool developers.

One of the team then suggested Google Colabatory their implementation of Jupyter Notebooks. It works like a dream, we had a number of people up and running with nothing more than a few mouse clicks - most of which were logging into their Google account. But, even though this approach would have been great for our colleagues it wasn't possible (or we couldn't work out how) to link this securely to our data platform in Azure. I'm definitely going to be using "Colab" for a few personal projects though.

Our current focus is The Littlest JupyterHub running in a VM at Azure - https://tljh.jupyter.org/en/latest/install/azure.html. That's next week's adventure though.


I keep dipping my toe into the Pythonic data science tool of choice. This week I've been looking at creating dataframes from CSV files, setting appropriate data types and grouping for sums and counts. If I can clean up my code I might even try publishing a Notebook or two here.

Issue, Incident and Problem Management

One of my responsibilities at work is to make sure that our policies and procedures are up to date and working effectively. This week we had a problem with a production system that caused degraded service to users over a day or so. I asked the team to log this as an incident and there was some confusion about what this was and what the procedure was for logging and managing incidents. So I reviewed our documentation and realised that this isn't spelled out very well.

To address this I'm writing an issue, incident and problem management process for the IT team to follow. The first problem is, as always, one of terminology. ITIL references Incidents that become Problems, our corporate risk management process has an incident management policy that covers what ITIL calls Problems. I'm going to defer to the corporate risk management process as no one outside my team (and many inside) knows anything about ITIL. So the process will talk about:

  • Issue - we think something is wrong, these are registered in the service desk and may; require administrative action (reset password, etc), be a bug that needs to be put on the relevant development team's backlog, or something more serious.
  • Incident - a systemic issue with the operation of one or more systems or services that is affecting our ability to serve our stakeholders (customers, intermediaries and staff), and may be identified by one or more Issues, by system monitoring or by notification from a partner (e.g. a SaaS application is experiencing problems)
  • Problem - the world is falling in our heads, really just a more severe classification of an incident.

Now all I've got to do is write up what to do with each of these interruptions to our services. That can be next week's job.

Interesting Media

Real Tigers book cover

I've started reading Mick Herron's Real Tigers

Steve Hogarth

You should definitely read all about the bloke who isn't Fish