Good, if slow, progress on publishing the Bradford 2025 data. Volunteering data is now nearly ready to go. In preparing for this, I've developed some minimal data governance which identifies corporate risk and seeks sign-off from key personnel. I've also made links with the communications team, who have expressed concern that going public will mean they have a more difficult job. My counter argument is typically that people would criticise in any case, and having the facts in the form of data should make things easier. We've committed to getting the data published next week. Watch this space!

Meanwhile, I've been working with colleagues at Open Innovations to plan the next releases. Some minor progress on technical adapters, which will mean that pulling the data is easy to do.

Not a lot of other work, although I did develop a schema for publishing event lists as part of some work I did to help publish the Hebden Bridge Picture House listings on the What's On Hebden Bridge site. Planning to do the same for The Trades Club.

Links, in no particular order:

• A very clever design for content responsive layouts using modern CSS.
• A list of articles on inclusive design
• The Weird One prosocial web, where you can set up a home on the web away from the big social media networks. Mine is here: https://giles.dri.ng/
• A list of European hosted alternatives to US services.
• Some guidance on more efficient GitHub workflows

The Bradford 2025 open data work has been progressing at pace. Lots of work on data governance processes and the design system. The basic volunteering data is more or less ready to go, once we get approval to make it open. We've also had some very productive discussions about the ongoing work, which is likely to be fairly intense for the next couple of months, then drop down to maintenance. I'm really enjoying the slightly deeper work around setting up governance, design systems and skills transfer to the evaluation team.

With this in mind, I've also started a short course on Evaluation for Arts, Culture, and Heritage: Principles and Practice that's been set up by the Centre for Cultural Value at Leeds University. I've met a few of the folk from the CCV through Leeds 2023 and Bradford 2025 culture work, and my initial impressions of the course are very favourable. It's reinforcing some of the things that I've discussed when talking about culture data, notably the difficultly of codifying the actual evaluation outcomes, rather than just the monitoring aspects. My view, which I'm interested to test as the course unfolds, is that the monitoring data is a useful backdrop to the actual work of evaluation. If it's doing its job well, the backdrop provides evidence of the impact of cultural stuff. There is a tendency for overreach of aims: can culture solve all of society's ills? Should it? More on this in coming weeks!

Back at the monitoring level, I will need to slightly refine my data masking technique, as I realised during the week that if only one value was below the clip level, the replacement value of the average would be the same as the original value! Looks like all the highly trained statisticians at the ONS know what they're doing! On that basis, I'll resort to dropping the raw values if there are fewer than 3 (?) values.

One thing I did on the secret project that I can talk about, I reckon, is build a new visualisation to visualise links between clusters. I've been visualising clusters of UK Company SIC codes, which although not a perfect marker, are at least relatively well documented, and for more traditional organisations are probably OK. I used a variant of Bilevel Edge Bundling which ended up being quite a useful visualisation. The server rendering is done as a Lume component, with some interactivity (highlighting, not updating) added using some minimal Javascript. Here's what it looked like:

I've latterly been pondering running my own instances of useful services as a way of decoupling from cloud services which are increasingly being enshittified and sullied with AI. I've currently got an instance of Open Project running, where I sometimes plan larger bits of work. I've tinkered with a few other services, but the two which would logically fit next to that are FreeIPA ro manage users and access and Forgejo to manage software development (this is the software that powers Codeberg). No firm plans, but I found myself re-researching this and wanted to capture this.

Some while ago I presented the work I did on the Leeds 2023 data site. The organising team has written up a blog post about the Visioning an audience data strategy event, and kindly sent it to me.

Links, in no particular order:

• There was some discussion over on Mastodon about tldraw infinite canvas component, which was being recommended, but appears to have been closed sourced. An alternative is the DGRM editor. Both look interesting!
• Matt Edgar boosted a blog post post about Stewart Brand's pace layers as applied to services. The conversion of infrastructure to OpEx, and the vesting of more and more operational stuff in bundles services was an interesting perspective.
• I keep meaning to get into Kaggle, particularly the competitions.
• I should consider joining a union. United Tech and Allied Workers (UTAW) looks like a good one
• I came across the brilliant HTML is for People course that Blake Watson has created. It's a really great introduction which I'm going to try to get the kids to have a look at!
• More from the CCV: this time a project that is aiming to scope out a national Cultural data observatory.

And finally (phew!), I ended up going down a rabbit hole on a quote that someone shared and which rang true for me:

Most of the week has been consumed with some rocks thrown into a project about the scope of an initial phase. Let's just say that managing expectations and getting feedback early is very important! It's a pity, as the work is progressing well overall. Consultancy is hard!

Other projects have been more productive. The Bradford 2025 data publishing programme attained automated extract from the first source system. Quite a milestone, which allows / requires getting the data governance bit sorted out. To this end, I've documented the process and drafted some risks to be discussed with the risk owner. This should lead to the data being released, with any luck. We could, by the end of next week, have an initial site sorted out. Watch this space.

One of the steps on the data release was the handling of small numbers in summaries with the risk of personal identification. Often small numbers are surpressesd in data released to avoid this eventuality, but there are other techniques. ONS has published information on disclosure control, but in our case, a simpler technique would suffice. I built a small value masker which bisects the dataset based on a clip level, then replaces the numbers below the clip with the average of the small numbers. The thinking here is that the overall distribution and totals should be similar in this case. I'll do some more detailed analysis of this next week, but for now, it'll do!

This code is needed because the volunteering system does not track when checkpoints (used to track the onboarding process for volunteers) were achieved. The Leeds solution (for the same system) was pretty arcane, and meant that the dataset needed to contain a hashed version of the ID field to match with incoming data. The new solution reduces the information stored along with the hash to a list of checkpoints and dates. The first step on extracting the data from the system is to establish if any of the hashes have a new checkpoint, and if so append the updates to the list of state dates. This avoids some unpleasant round-tripping on the data. Once this is done, the (possibly updated) state data is read and turned into a lookup returning the states as a dictionary. This dictionary replaces the hash, meaning that this potentially personal and disclosive data is removed from the dataset.

The hush-hush project also progressed, and I was able to play back the data work to the client, with some excellent feedback. We have a pilot organisation workshop next week, and will hopefully be able to speak more about the project after that. The data science work that I mentioned in last week's weeknotes has progressed, with a fuzzy match of part of the company house data. Next steps on that is to add in the identification of personal names.

Finally, today, I met with a digital marketer who is working with one of my clients. He needed some help getting the Google search console linked in to GA-4, which needed a bit of tinkering and archaeology to work out where their DNS records were managed! Will be interesting to keep abreast of this work and see how digital marketing data is linked together.

A few links:

• An excellent visualisation / flowchart of what is considered personal data from missiggeek on Mastodon
• A brilliant article about what ChatGPT is, with analogy to lossy compression
• The H3 hexagonal based coordinate system developed by Uber for geospatial analysis.

The other project is progressing well too, although is still under a communications blackout, hence me not mentioning what it is explicitly! I've had cause to use some actual data science techniques in building an exciting bit of data infrastructure. Firstly, I'm matching very large datasets (several million rows in the reference), and this was slowing down. Chucking this into DuckDB rather than processing with PETL or Pandas totally sped this process up. Right tool for the job. As an added bonus, the DuckDB bindings for Python has a .df() method which returns the result set as a Pandas dataframe. Winner! I've also used some fuzzy matching using thefuzz to identify spelling mistakes in the source dataset and enhance matching. Next up will be starting to try to identify personal names in the source data so that I can separate organisations from individuals, and handle them separately. I've identified Named-entity recognition (NER) as the means to do this, and early experiments using the NLTK NER modules is looking promising. Quite a lot of tuning to be done here. Finally, I'm going to extend the simple DuckDB SQL matching to take account of fuzzy matches between the source and reference sets.

In other work, I spent quite a lot of time on network fixing. One of my clients, for whom I've set up a Sentinel license server running on a laptop accessed via a VPN, got a new router, so I had to open up the VPN ports again. Annoyingly I hadn't written down exactly what I did, so this took a bit longer to work out that the port mapping (51820/udp) shouldn't be limited! Anyway sorted now. Another client had a major DNS failure when one of their suppliers decided to turn off a cPanel server that they had been using, and no longer needed. Unfortunately, their main domain was associated with this service, and when it went away, everything broke. Badly. That took a couple of hours to unpick and get back on the road to recovery. And for the hat-trick, a data microsite that OI had created had it's DNS records removed, so stopped working. Thankfully, I was able to direct them to fix it fairly quickly, and managed to find the original email trail from two years ago.

All of this left me thinking that for many small (and not-so-small) businesses this network configuration was something close to magic. I have begun work on a workbook, focussed on small businesses, which collects important data about the critical infrastructure that keeps their website, email and other important services up-and-running. It would encourage the business owners to review this data frequently so that (for example) credit cards don't run out and lead to a break in service. Furthermore it would contain advice about practical matters, such as email addresses to use a primary contact for critical accounts to avoid being locked out in the event something goes wrong.

A couple of big thoughts: one about value of various types of work and one about a business model that might be worth exploring.

I was pondering the things that we spend time on in projects and things we don't. In an ideal world the balance between time spent on the hard technical stuff and time spend on the even harder people-related stuff should tip towards the people-related. I've invested a lot of time in ensuring I know how to make the techy stuff get out of the way. You want a website: I know how to build it. You want some some visualisations? I know how to do that, and moreover, I've invested time with the people I work with on making it easy for anyone in the team to create a pretty good map, or chart or whatever. Pipelines? I've got patterns that help me make robust and repeatable data processing pipeline pretty fast. This means that we can prototype at speed and get stuff done. However, there are times, and particularly in more open ended projects where delivering fast is not really desirable. This isn't because we need to spend longer building the tech, but because we need to spend more time dealing with the really interesting stuff, which tends to be squishy, human stuff. That might be researching, pondering, discussing with colleagues, exploring prior art. Sometimes, delivering fast is a sure-fire way to miss the point.

I'm sure at various points in the week, those thoughts have been more well defined, so I might loop back to them and see if I can tighten them up a bit.

The other 'biggish thought' I had was about the idea of founding a co-op which delivers IT services to other co-ops. This came out a tech incident that I was called upon to help out with at Equal Care Co-op. They are a small team, and don't have a dedicated IT Service team. This is arguably problematic given the dependence that people are beginning to have on the tech. Getting a full service team, with cover dedicated simply to Equal Care is likely to be beyond their means, and it feels like signing up with a commercial organisation might not mesh well with their ethos and instruments. Maybe a like-minded team, constructed to service a number of organisations could have the economies of scale to deal with this. Some initial challenges are seed funding the organisation, defining the offer and recruiting both service delivery team and the client organisations. Would this be a co-op of co-ops: each providing their IT folk to support others? Would there be a TUPE element to this if acting as a de facto outsourcing organisation? More to think about, but worth developing.

That'll do for now: it's nearly time for the next weeknote!

One thing that I've struggled with a bit on the new project is lack of clarity on the direction of travel. A colleague (hi, Sarah!) describes this as the North Star, which is great imagery: I just need to know that I'm heading in broadly the right direction. I'd raised the idea of writing down some (possibly / probably wrong) hypotheses that we could use to help set this direction without being too prescriptive. The same colleague then shared this excellent blog post by Ben Holliday about hypothesis driven design which pretty well encapsulates my reasons for wanting to write something down We've (me and Sarah) have been working through some high-level / very broad hypotheses and realised we can use these to identify some lower-level / more fine-grained hypotheses which can really shape the work. I've come to realise this is slightly in tension with the design "double diamond", which aims for breadth in the first instance. My response to this is that without the direction setting, the initial diamond risks effort radiating in all directions from a point, resulting in a circle. The hypotheses set enough of a direction to decide what to consider in the first diamond. I may write this up soon.

My SvelteKit optimisation has resulted in a much better understanding of how to control what the bundler creates. This is captured in a repo, ready to be written up. As I was doing this, I was also implementing some of the recommendations in the Social Value site I'm building. This uses DuckDB-WASM on the client side, which is excellent... a full OLAP database in a browser? Crikey! It is, however, quite costly on the network, as it has to download 35MB of Web Assembly code to start the database engine up. In a search for an alternative, I've come across the @oak/acorn framework which will allow me to serve just the JSON. Much better for this use case. I have, however, discovered that I cannot host this on Deno Deploy, which is my go-to platform.

The final bit of work is refreshing Open Audience, a tool which Tom Forth built using 2011 Census data which can build a profile of attendees at events based on their postcodes. This has been buzzing around some of the culture work that I've been involved in for a while and given we now have 2021 Census data, I thought I'd rebuild at least the dataset. The data has changed ever so slightly, so it might not be possible to recreate completely, and I might need to rethink the frontend. I have some other ideas, including:

• Open Audience as a service: An API which provides the profile based on postcodes
• Open Audience language bindings: Wrappers in Python / Javascript to allow the data to be used easily in pipelines and web-pages

I did have a look at Storybook, and it looks promising as a way of documenting web design libraries. I'm also wondering if there's a way of using it as an SSG, as I suspect it might be overkill for some of the work.

Finally, I've added an RSS feed for this site, so you can read my witterings in your favourite feed reader. I like NetNewsWire, FWIW, but there are many others

Plans for next week:

• Refine the culture project hypotheses
• Blog about the SvelteKit optimisations
• Research adding ActivityPub to this site. There's a really nice (albeit incomplete!) series of blog posts by Maho Pacheco covering how to do this.

Some links I came across

• Your name in Landsat images
• Signalbox live train locations. NB Proves to be not that accurate as I tested when I was on an actual train!
• A really nice blog post about the importance of clear Open Data licensing by Mike Rose and Kieran Wint.

Finally, I really like this pull quote by Ted Chiang, shared in this toot by James Gleick on Mastodon:

The programmer Simon Willison has described the training for large language models as ‘money laundering for copyrighted data,’ which I find a useful way to think about the appeal of generative-A.I. programs: they let you engage in something like plagiarism, but there’s no guilt associated with it because it’s not clear even to you that you’re copying.
Source https://www.newyorker.com/culture/the-weekend-essay/why-ai-isnt-going-to-make-art.

Given one of the hardest things to do in tech is deciding on the name of things, my first task is coming up with distinct names for each of the what I assume will be almost endless collection of blog posts. I briefly considered naming them like Friends episodes, but this could quite quickly descend into farce. I decided on simply the year and week number, according to the ISO standard of weeks starting on Monday. I note from the link above that Monday is Labor Day in the US, which I think has something to do with wearing white shoes.

This week has been mostly return from holiday and getting my head round a couple of culture-related projects that I'm involved in as a freelancer with Open Innovations. The first is the Bradford 2025 City of Culture, for which I wrote an Open Data Strategy just before my break. I've agreed the scope of the next piece of work to start delivering on it. It builds on the LEEDS 2023 Data Microsite that I built with OI team. The other project is along the same lines, but much broader, and earlier in the discovery phase. I consequently spent a fair amount of time reading what I could lay my hands on and hypothesising about what we might build. Very happy to be wrong about assumptions at this stage!

Away from Open Innovations, I've finally broken the back of a website upgrade for the Hebden Bridge Picture House which should enable me to upgrade from (out of support) PHP 7.3. It turns out that some changes in the Textpattern software have invalidated the way I built the site. The Textpattern community were very helpful, responding to a topic I posted in the Textpattern forums with some really useful suggestions.

I've also been refreshing a prototype site I'm building for social value consultants CHY Consultancy. I've been mindful of recent discussions about Javascript framework bloat, and I agree wholeheartedly with this. I really like static site generators, and tend to use Lume as my weapon of choice. In fact, this very post is written in it. There are times that having a framework that can be easily extended to do more powerful client-side stuff is useful, but I really don't like the likes of React, Angular, et al. I've long been a fan of Svelte for componentised client-side code, particularly where the need is a bit more heavyweight. The prototype uses Sveltekit which builds on this beautifully. Given a combination of setting prerender, csr and ssr options for pages, it's even possible to generate a static site. This can be turned off for any pages which need live server-side rendering (e.g. dynamic pages based on routing parameters), or client-side processing (e.g. highly interactive islands on a web page). Anyway, there's been tinkering on making this perform well.

Stuff added to the list to do or look at next week:

• Storybook frontend UI workshop.
• Draft a blog post about the Generative AI inspired by a very long and rambling pub chat that I had with a friend last Sunday night.
• Draft a blog post about using SvelteKit to make efficient websites.

Finally, and in the spirit of full disclosure, I have just spent 20 minutes automating appending the week of the year to the post title as a Lume preprocessor. It uses the pretty handy date-fns library, with the format string RRRR-'W'II. This is next-level yak shaving.

In my goal of automating all the things, I wanted to make this happen whenever anyone pushed to the dev branch in the repository. It's pretty simple to trigger a workflow on a push to a specific branch. The question was what to run to transfer the files.

The answer to this question is rsync, which you can read all about on the Wikipedia page for rsync. In short, this venerable tool allows files to be transferred and synchronised over connection protocols such as SSH.

Building the actions pipeline now has the following stages

1. Build site using the site builder into the build folder. In my case, this the builder is lume, which deposits the compiled site in _site.
2. Use rsync to transfer the build folder to the dev host.

Step 1 is easy enough, and in any case out of scope of this post! Step 2 needs a bit of careful thought.

The basic incantation is

rsync --recursive --delete $SOURCE_PATH $SSH_HOST:$SSH_PATH

This recurses through the source path, uploading any new or changed files, and deleting any orphans. The source path should end in a / to avoid including the directory itself! Using environment variables SOURCE_PATH, SSH_HOST and SSH_PATH means that the configuration can be altered and used in multiple potential targets, which could be useful. You can set environment variables in GitHub workflows.

When using rsync interactively, it's usual to run as a personal user account, in which case SSH credentials (keys, etc) are likely to be set up. It's not, however, a great idea to use a personal user in an automated pipeline, so I created a locked down bot user. Passing credentials into a GitHub actions environment is also slightly fiddly. It's theoretically possible to setup SSH keys and a config file, but rsync allows a slightly easier setup, by providing the --rsh option. This allows exact specification of the remote shell command.

rsync ... \
  --rsh="sshpass -e ssh -o StrictHostKeyChecking=no -l $SSH_USER" \
  ...

This allows the SSH password to be provided in the SSHPASS environment variable (managed via the sshpass -e command). It also specifies the user to connect with (SSH_USER) and allows overriding other ssh options such as StrictHostKeyChecking.

So now we have a working sync command which signs in as our bot user. All is not well, however, as the files and directories are owned by the bot user. Given these are web content, we'd ideally like them to be owned by the www-data user with a group ownership of www-data. Thankfully here can provide another option --rsync-path which defines the command that is run in the shell created by the connection.

rsync ... \
  --rsync-path="sudo -u www-data rsync" \
  ...

This will run the remote rsync command as the user www-data, meaning that the files are written with appropriate ownership and permissions.

The cherry on the cake is specifying a the --info flag to write a report on completion of the sync. The final command is:

rsync \
  --rsh="sshpass -e ssh -o StrictHostKeyChecking=no -l $SSH_USER" \
  --rsync-path="sudo -u www-data rsync" \
  --info=STATS2 --recursive --delete \
  $SOURCE_PATH $SSH_HOST:$SSH_PATH

Wrapping this up in a GitHub Actions script is fairly simple using the actions file run directive. Of course, it can also be packaged in another runner, such as deno.json, or as an NPM task. This is left as an exercise for the reader!

I hope that this has been helpful. I'm sure future me will also be thankful!

Would you rather fight 100 duck-sized horses or one horse-sized duck?

I’ve recently been trying the capabilities of DuckDB to drive visualisations. There’s something quite astounding about writing SQL in client-side Javascript. My current platform of choice is Svelte. I’ve come up with some patterns on using Duck within a Svelte app — to be written up another day.

Suffice it to say that the official DuckDB WASM client setup guide is a great start. The basic pattern is to prepare parquet files with the data, and query those via DuckDB in the following format:

SELECT
  strftime(date, '%x') AS date,
  value
FROM ‘data.parquet’
WHERE code == ‘The Code’
ORDER BY date;

The only pre-requisite is that the parquet files need to be registered when the database connection is made:

await db.registerFileURL(
  'data.parquet',
  'data.parquet',
  DuckDBDataProtocol.HTTP,
  false
);

So far, so good, but if the parquet files are large, it'd be nice to take advantage of partitioning to avoid shipping the entire file to the browser. It's pretty easy to create a partitioned file using libraries such as pandas:

df.to_parquet(path='data/', partition_cols=['variable'])

We now have a dataset partitioned by variable name, and can in theory write queries as follows:

SELECT
  strftime(date, '%x') AS date,
  value
FROM ‘data/**/*.parquet’
WHERE code == ‘The Code’
ORDER BY date;

The slight wrinkle is that you still need to register each file as before. It appears that the DuckDB registerFileURL doesn't support wildcards, so each file has to be registered independently. Having discovered, this, I decided that a manifest file would be a sensible way of dealing with the potentially very large number of files that needed to be registered. A simple way to create this is using shell commands and jq.

find data/ -type f |\
  jq --raw-input --slurp 'split("\n")' > manifest.json

I then register each parquet file in the JSON array as follows:

await Promise.all(
  manifest.map(p => db.registerFileURL(p, p, DuckDBDataProtocol.HTTP, false))
);

Here were the rough results for a simple test database that I set up. This is running on my local network, so the impact on a slower network would be greater. The whole database is 2.5 MB in a parquet file, which is already a massive saving from the source 18 MB CSV file. It's worth noting that the subsequent calls were much faster.

Limitations I ran into, each of which could do with a bit more digging...

1. It seems that at least the libraries that I was using don't allow more than 1024 partitions to be created.
2. I tried using Brotli compression, but the DuckDB WASM library didn't seem to like it.
3. I sometimes / within some build systems, had to perform some manipulation of the url to prefix with the server URL.