Projects on James Jarvis

🏚️->🏠 A year of house renovations

hello@jamesjarvis.io (James Jarvis) — Sat, 01 Nov 2025 00:00:00 +0000

I bought my first house at the age of 26 in late October 2024, almost exactly 1 year ago.

I was up for a challenge, so even though I didn’t start looking out for a complete fixer-upper, I slowly convinced myself (and my girlfriend) to take a chance on a project house. I wrote a bit previously about how we went about searching for the right house here, but TL;DR is that we could have probably lost less money by renting 😂.

So, a year ~~wiser~~ older, after hundreds of hours of work demolishing, planning, painting, plumbing, laying floors, building a kitchen, bathroom and generally fixing-up, this is my advice…

Planning is everything

When selecting flooring, I ordered 12 samples from 4 suppliers, I spent hours planning the herringbone sequence to make sure we didn’t get stuck. The kitchen went through 5 different designs, the bathroom went through 6 different designs, and the front fireplace took hours to just measure out.

It’s a bit annoying but once you’ve got experience with it, it’s fairly painless and well worth the investment.

Outside -> in. Top -> bottom.

Sage advice when working on buildings is that you should always sequence your work outside -> in, and top -> bottom.

Roof need fixing? Any leaky windows? Do those first before decorating the inside.

Want to plaster the ceiling? Better do that before you paint the walls, and definitely do that before you lay a new floor.

And any significant plumbing work should sequence neatly between outside work and any inner work, as you are definitely going to be ripping stuff apart in the process.

You can’t do it alone

I’ve been incredibly lucky to have taken advantage of a wonderful bunch of wilful workers from my family and friends. None of knew what we were doing, but they’ve all gotten stuck in and helped out massively.

I’d like to thank Feiyang, Jason, Celia, Penny, Owen, Jon, Matthew, Brandon, Raphael, Filip and Sam for all of their help.

Invest in tools

The first things I bought were a mitre saw and handheld combi drill. Alongside a stanley knife, some paint brushes/rollers and a borrowed handheld jigsaw I can’t imagine getting this far without them. I had to do almost a thousand cuts with that saw when laying the downstairs floor, and I’ve been through several thousand screws with the drill.

A borrowed chainsaw was way better than the 50 year old wood saw I had for removing some trees, and the laser marker or dedicated scribing tools were far better than me eyeballing a cut.

Double/Triple the budget

I think the last year’s work (~100sqm of wood flooring, every room redecorated/furnished, all radiators replaced and a new kitchen / bathroom) cost me approximately ~£45k. I had originally thought ~£35k would cover everything comfortably including the gardens and some more furniture - turns out I was off by about double!

Timing is also something not to be underestimated.

How hard can it be?

“Fuck it, let’s give it a go”

This kind of attitude is the most important thing required (possibly more so than capital imho) when taking on a big project.

You have to be willing to jump into a domain you know absolutely nothing about, mess stuff up, and figure it.

Understand what you are signing up for

This isn’t for the faint-hearted, it’s deliberately a massive pain in the arse and can be extremely stressful, not to mention economically exhausting.

Getting taxed 43%, handing over stamp duty (worse tax ever), only to remember that 20% of your new kitchen’s cost is also going to HMRC is a bit of a kick in the teeth.

Working a full time job, then spending evenings and weekends on DIY whilst living without heating or running water in a dusty house, eating only microwaveable food gets old real quick - you do need to be slightly stupid to do this.

Now the itch is scratched, I’m itching to do it again

…and I’m not talking about loft insulation!

It has been a genuine pain in the ass sometimes, but I can’t help but have a huge sense of accomplishment sitting in my home built home office.

I’m not looking to repeat some of the stress of the last year (I almost lost my eyesight twice due to stress!) but I am looking forward to taking on more big projects like this in the future.

Hopefully now, with a little more experience behind me!

No pirate builders here…

Secret Santa: A Quote-a-day E-Ink display

hello@jamesjarvis.io (James Jarvis) — Fri, 07 Feb 2025 00:00:00 +0000

This is the story of how I built (and how you can build) a cute little display that shows you a new famous quote every day.

The idea

Way back in December 2022, my family decided that this christmas we would DIY each other some presents.

For a few years prior we had been experimenting with a secret santa to alleviate some of the pains of having everyone each buy 10 gifts to give to each person, but when we realised most people’s “wants” aren’t within a £50 budget, we gave the “make something and give that” idea a go.

My ~~unlucky victim~~ adoring cousin was to be the recipient of whatever thing I could make, and since she was at the time completing her doctorate, enjoyed reading, and had recently started programming, I thought I would try and make something that combined all of those ridiculously nerdy attributes into one:

Thou shalt build a display to show a new quote every day

I’m sure the baby Jesus would’ve preferred one of these over some frankincense and myrrh any day.

I had come back home for christmas and thought I could start work on making a V2 of my original e-ink display (read more about that one here), so this particular type of display was on my mind, and it fit perfectly with the goal of having a dumb little trinket sitting on a desk for months at a time without having to charge the battery.

The parts list

Screenshot of the parts list on pimoroni.com

You could 100% do this project for about half the price if you source components on AliExpress / some other site.

However, I only had 1 week left to finish this project by the time I finally stopped procrastinating, I have always had a positive experience with Pimoroni and their shipping is normally only a couple days.

Besides, I wouldn’t have to pull my hair out working out how to build a driver for this display, since the Badger2040 project is pretty well documented, and has a lot of support on their GitHub.

The full parts list is:

1x Pimoroni Badger 2040 (link)
1x LiPo battery pack 1200mAh (link)
1x LiPo Amigo (link)

Back of the Badger2040 with battery attached

Charging the battery pack with LiPo Amigo

You can choose to instead power the badger via a AAA battery pack, but I’m not a fan of disposable batteries.

P.S. Pimoroni if you want to sponsor me I have my eyes on a few other things from your site! 😉

The quotes list

For this project I had about ~1.5MB of memory to play with on the Badger2040, so I couldn’t have too much data sitting around. Though I did want enough to cover the goal of 1 quote a day, for as long as it takes to forget the other quotes, so around 2,040 quotes should do, assuming 200 characters per quote this’ll fit in under 0.5MB.

Finding the quotes was quite hard, as most sites only host useless stuff like “Top 50 inspirational quotes”.

Goodreads had probably the best corpus of material over on https://www.goodreads.com/quotes, and if I were doing this again I would just write a quick script that scraped that page and trim by length of quote so it still fit on the small display.

There’s even a whole GitHub project for this: github.com/maxhumber/quote, and nowadays a quick bit of ChatGPT / using your brain audible groan could get you this data within a few moments.

However, back in my last minute frantic I used a now-defunct API to scrape this data quickly. You can see my efforts at github.com/jamesjarvis/secret-santa but don’t bother running it as the API seemed to have gone offline sometime in the last few years.

Displaying quotes on the Badger2040

Pimoroni offer a really good repository for writing software for the Badger2040.

They have an embedded-systems subset of Python with very low resource consumption called “micropython”, that allows you to run pretty reasonable scripts on the RP2040 chip.

Combining that with an alright badger2040 library to interact with the buttons and display, and it’s a good solid base point to write simple apps for the Badger2040 product.

You can view the full source at github.com/jamesjarvis/badger2040 but the gist of it is the display_random_quote function, where we seek to a random line in a “quotes” file, and write it to the display, wrapping around the screen.

def display_random_quote():
 display.set_pen(15)
 display.set_thickness(FONT_THICKNESS)
 display.clear()
 # Open the quotes file.
 quotes = open(text_file, "r")

 n = random.randint(0, TOTAL_QUOTES)
 for i in range(n):
 quotes.readline()

 current_quote_json = json.loads(quotes.readline())
 # Read a full line and split it into words.
 words = current_quote_json["content"].split(" ")

 lines = []
 latest_line = ""
 for word in words:
 latest_line_length = display.measure_text(latest_line + word, TEXT_SIZE)
 if latest_line_length >= TEXT_WIDTH:
 lines.append(latest_line)
 latest_line = ""
 latest_line += word
 latest_line += " "

 lines.append(latest_line)

 lines.append("- " + current_quote_json["author"])

 row = 0
 for line in lines:
 y = int(row * text_spacing) + int(text_spacing // 2) + TEXT_PADDING
 display.set_pen(0)
 if row + 1 == len(lines):
 # last line is the author, make a bit bolder.
 display.set_thickness(FONT_THICKNESS+1)
 display.text(line, TEXT_PADDING, y, TEXT_WIDTH, TEXT_SIZE)
 row += 1
 display.update()

If I were to ever productionise this, I would look to find a more efficient way to scan through the file to a random line, as that’ll probably save a bit of energy.

Uploading code to the Badger2040

Normally when you get this kind of project it’s all dependent on some random toolchain you haven’t installed on your machine, and it’s so out of date that you’ll spend your first day just debugging how to program the damn thing.

With Pimoroni’s Badger2040 repo (forked to github.com/jamesjarvis/badger2040) you can exploit the repo’s Github actions to build the necessary compiled target for you!

The development flow was:

Clone badger2040 repo
Make small changes (adding a quotes.py example App)
Push commit to github and wait for the Action to build
Download the badger2040-micropython-with-badger-os.uf2 file from the Action artifact list
Connect badger2040 to computer in boot mode (hold down boot/usr and press rst button as it is plugged in)
Copy over the downloaded .uf2 file from step 4 to the badger
Bobs your Uncle

The outcome

Quotes Display refreshing to new quote

The outcome is quite nice if I say so myself.

It lasts a few weeks on a single charge, and when you press an up or down button it’ll display a new random quote. A simple but effective desk ornament.

I never took a photo of the original I gave to my cousin, and I’m fairly sure it was “lost” quickly after christmas, so I made myself another one :smile:

Role Based Access Control (RBAC) Authorisation in Golang with Permit.io

hello@jamesjarvis.io (James Jarvis) — Mon, 04 Nov 2024 00:00:00 +0000

When building an application, you need to know who your users are, and what they should have access to.

That’s when “Authentication” (are you who you say you are?) and “Authorisation” (are you allowed to do this thing?) come in to play.

These are two similar but separate problems, each with their own challenges.

Most developers don’t build their own authentication systems. It’s not-secure, and more importantly - not part of your products’ core offering. Thus, it’s just not worth the effort / time you’ll spend building it.

Authorisation is something traditionally built by developers in-house (Just like authentication was a few years ago), but in recent years, there are emerging options in the market that allow you to implement it without building it yourself.

In this article we are going to focus on authorisation (authorization for my U.S. friends), covering a step by step integration and implementation with an example Key-Value service, written in the Go programming language.

We are going to build our own naive authorisation system to get a feel for how it all works, and compare this with a production-ready implementation from Permit.io.

What is “Authorisation”?

At some point in the development of any “System” with “Users”, you will need some method of determining:

Is this “User” actually allowed to do the thing they are trying to do?

This is inevitable, whether the “system” you are building is physical (i.e. a Hotel), or virtual (i.e. a Social Network website).

For any sensitive action, we need to make sure that the person performing that action is allowed to do the action. If the scenario is as simple as if request.IsAuthenticated(), then congrats, you can end here, you have “authorisation through authentication”. But, this means that any user who can authenticate into your system, is also authorised do to anything in your system. For everything but a simple POC, this won’t fly.

If the scenario is more complex, i.e. if request.GetUser() not in authorisedUsers, then you are now able to restrict access to parts of your system, but you have now inherited a complexity: you need to store and maintain a list of every single user allowed to perform every action in your system. If you have 100 users, and 5 actions in your system (a very small example), this can become a headache to manage very quickly, let alone if you have >1000 users, or more actions to perform.

To simplify this, we can instead assign abilities to an abstract concept (Roles), this is where RBAC comes in.

What is “Role Based Access Control” (RBAC)

Rather than storing a map of user -> allowed actions, or actions -> allowed user, let’s abstract this a little. Introducing the concept of “Roles”: A label you can pin to a whole bunch of users, and check against that.

Benefits:

Low cardinality
- You may have millions of users, but you’ll rarely need more than a few dozen “roles”
- This is why the CIA has “security clearance levels”, rather than granting access to information individually
Easier to manage
- Easier to remove 1 “role” from 1 “user”, than updating the 1000s of actions a “user” may have access to
- This is why the CIA has “security clearance levels”, rather than granting access to information individually
Semantically relevant
- Wherever possible, as a developer your goal is to simplify the system
- Example: Seeing that a user has the roles of ["intern", "employee", "admin"], it is clearer to spot the problem than seeing a user with access to ["receptionDoor1", "receptionDoor2", "cafeDoor3", "floor4Door4", "floor2Door3"], if "floor4Door4" is your server room

Now that we understand the benefits of simplifying access control through the use of “Roles” users can be assigned to, let’s look at a demo application in Golang.

Example Application

Let’s keep this super simple, and say that you have 2 endpoints to an application that allows users to save and retrieve key-value pairs:

GET /v1/map/{key}
POST /v1/map/{key}

Each “user”, once authenticated, can be in one of two groups (“roles”):

reader
writer

“Readers” can GET, “Writers” can POST, a user can have either or both of these roles.

i.e. If I am a user (”Alice”), and I have the ability to write, I should be able to write a value of “world” to a key of “hello”.

If I am a user (”Bob”), and I have the ability to read, I should be able to see the value “world” under key “hello” that alice has just written for me.

However, since Bob cannot write, he should not be able to override the value of “world”, nor should he be able to add any new values to our system.

To show how to implement this sort of authorisation control to a system, we are going to start off with a working key-value service for the above, without any access controls in place.

We will then build our own RBAC system given the users and roles we know about and integrate it with our system.

To top it all off, we are then we are going to showcase a scalable, production ready setup built using the Permit.io Golang SDK for RBAC.

First, our Service setup:

package service

// StorageClient provides access to a key-value map.
type StorageClient interface {
	Get(key string) (value string, exists bool, err error)
	Set(key string, value string) error
}

// Service is an implementation of the key-value service
type Service struct {
	storage StorageClient
}

// Get accepts a user and a key, and returns either the value associated with that key,
// or an error if encountered.
func (s *Service) Get(userID string, key string) (string, error) {
	value, ok, err := s.storage.Get(key)
	if err != nil {
		return "", fmt.Errorf("error getting value for key:%q error:%w", key, err)
	}
	if !ok {
		return "", fmt.Errorf("error getting value for key:%q error:%w", key, Error_NOTFOUND)
	}
	return value, nil
}

// Set accepts a user, key and value, and will attempt to set the value with that key,
// or return an error if encountered.
func (s *Service) Set(userID, key, value string) error {
	err := s.storage.Set(key, value)
	if err != nil {
		return fmt.Errorf("error setting value for key:%q error:%w", key, err)
	}
	return nil
}

Using our StorageClient interface, and the Get and Set application APIs, we have the ability to read and write any key-value pair we desire, without any authorisation logic to safeguard access. Get allows any user to extract the value for a given key, and Set allows any user to write the value at a given key.

Simple, right? Now let’s try and implement our own RBAC!

Restricting Access to our API

Remember, we now want to restrict access to Get and Set based on whether the given user is a “reader” or a “writer”. To build an API for checking the authorisation level of the user, we really only care about whether they are or aren’t authorised to access the resource. A good API for this, therefore, would enable us to supply our userID, and their intended action, and allow the API to resolve the user’s roles, and the permitted roles for each action, and return us true or false, depending on whether the user can or cannot perform the intended action.

A good interface for this then, would follow:

// AccessClient enables the ability to check authorisation for a given user.
type AccessClient interface {
 Check(userID string, action string) bool
}

Popping this into our implementation is quite simple, as an initial validation step for our API:

func (s *Service) Get(userID string, key string) (string, error) {
 if !s.access.Check(userID, "get") {
 return "", Error_UNAUTHORISED
 }
 ...
}
func (s *Service) Set(userID, key, value string) error {
 if !s.access.Check(userID, "set") {
 return Error_UNAUTHORISED
 }
 ...
}

For this AccessClientto work however, we need an implementation…

DIY: In-memory RBAC

Those with a keen eye for Go would notice we have defined our authorisation API first, ignoring any eventual implementation. This is a good way to focus on your API first, without getting too bogged down with how your API implementation should actually work.

To get our heads around the problem, we can look into a simple implementation of this API, using an in-memory map to keep track of our users, the actions that may be performed, and the roles permitted to complete those actions.

Our objects User and Action :

// User is a user object, containing the roles this user is a member of.
type User struct {
 Roles []string
}

// Action is an action that can be performed, along with the roles allowed to perform this action.
type Action struct {
 PermittedRoles []string
}

Our Check function, needs to answer the question of Does the given user have a role, that is permitted to complete the given action? In reality, this manifests quite easily as a reverse lookup:

// Check returns true if the user is allowed to perform the action, false otherwise.
func (a *Access) Check(userID string, actionID string) bool {
 user, ok := a.users[userID]
 if !ok {
 return false
 }

 action, ok := a.actions[actionID]
 if !ok {
 return false
 }

 for _, role := range user.Roles {
 if slices.Contains(action.PermittedRoles, role) {
 return true
 }
 }

 return false
}

In the above, if the user exists, the action exists, and at least one of the user’s roles match with at least one of the action’s permitted roles, then we return true, and the action is permitted!

Conversely, if the user is unknown, or the action is unknown or none of the user’s roles match with any of the action’s permitted roles, then we return false, and the action is denied, at which point an UNAUTHORISED error is returned to the the caller.

Flaws with DIY

In practise however, I don’t recommend you to actually use the above DIY approach. DIY-ing your own authorisation implementation is something that should really only be attempted by the Googles of this world, as there is a good chance you simply will not have the resources to do it correctly, let alone quickly.

Focus on outsourcing the implementation of all elements of your system that are not specifically critical to the value you intend to provide:

Don’t build your own database
Don’t build your own container orchestrator
Don’t build your own authorisation system
Don’t do anything that you don’t have to…

Alternative: RBAC with permit.io

Permit.io is an authorisation as a service provider, with a whole suite of features that cover use cases from this simple example, all the way to the largest enterprise applications with millions of users and umpteen levels of complexity.

This solution was easy enough to work with that I could build this example implementation in <30 mins from reading through their docs (ChatGPT could probably do it faster though!)

Conveniently, the Permit Golang SDK also has it’s own Check function, that is similar to ours:

// PermitClient is the permit.io client interface.
type PermitClient interface {
 Check(user enforcement.User, action enforcement.Action, resource enforcement.Resource) (bool, error)
}

For our case, to provide a drop in replacement to our DIY solution, we are going to wrap this function. The models in this API are:

“User”
- Identifiers for a given user to the system
“Action”
- The intent for a given call to “Check”, in our case this will be “get” or “set”
“Resource”
- The domain for which this action is being performed against
- In our simple example, we do not have a need for this “resource”, as we only have a single map domain, but this allows for other rich customisations later on, such as namespaced maps with alternative access controls.

Our wrapper:

// Check returns true if the user is allowed to perform the action, false otherwise.
func (p *Permit) Check(userID string, actionID string) bool {
 user := enforcement.UserBuilder(userID).Build()
 action := enforcement.Action(actionID)
 resource := enforcement.ResourceBuilder("map").Build()

 allowed, err := p.permitClient.Check(user, action, resource)
 if err != nil {
 fmt.Printf("permit error: %s\\n", err.Error())
 return false
 }

 return allowed
}

Our final application

If you want to dive deeper into the code, and run the application yourself (with both DIY and Permit based authorisation), you can follow the steps on github.com/jamesjarvis/rbac-example.

Our test data includes the following users, and their roles:

UserID	Role: reader	Role: writer
alice	Y	Y
bob		Y
charli	Y

Our server is a HTTP server running on port :8080, so we can perform a simple test by setting a value “world” for key “hello” as user “alice”:

$ curl -H "User: alice" -X POST <http://localhost:8080/v1/map/hello> -d "world"
> world

Fantastic, this works! Alice is indeed able to write to the POST endpoint, as they have the role of “writer”. We can see since they also have the role of “reader”, they should also be able to GET this value again:

$ curl -H "User: alice" -X GET <http://localhost:8080/v1/map/hello>
> world

But what if Bob tries to read this value?

$ curl -H "User: bob" -X GET <http://localhost:8080/v1/map/hello>
> UNAUTHORISED

Bob cannot perform any GET calls, because Bob does not have the “reader” role, which is required to perform the “get” action. Bob can, however, perform POST requests.

Similarly, our final user Charli does not have the “writer” role, preventing them from accessing the POST endpoint, but since they are a “reader”, they can still perform GET requests:

$ curl -H "User: charli" -X GET <http://localhost:8080/v1/map/hello>
> world

Summary

Authorisation is the challenge of answering “Is this User actually allowed to do the thing they are trying to do?”

By focusing on Roles, rather than Users (RBAC), application writers can greatly simplify both their application logic, and the operation of their application.

We have built a working HTTP Key-Value service in Golang, that allows a user to GET (read) and POST (write) values at a given key.

We have then extended this application to restrict access to these endpoints based on whether the user has the associated role of “reader” or “writer”.

We have explored the pros and cons of two different implementation to this RBAC solution:

DIY using just the Golang standard library
Using Permit.io as our Authentication as a service provider

Phew!

By this point, you should have a decent understanding of how a simple Role Based Access Control (RBAC) implementation can be integrated into your application.

Focusing on “roles” rather than “users”, provides an easier abstraction for both yourself as a developer, but also your system administrators, and ultimately, your users themselves.

Finally, another word of caution.

Focus on outsourcing the implementation of all elements of your system that are not specifically critical to the value you intend to provide:

Don’t build your own databases
Don’t build your own container orchestrators
Don’t build your own authorisation systems
Do focus on the core offering of your product

Thanks for reading!

📖 Blogging Site

hello@jamesjarvis.io (James Jarvis) — Mon, 10 Jul 2023 00:00:00 +0000

I am growing up [citation needed].

With this, comes a lot of realisations about what I value from the internet, and in particular about the things I build.

The rewrite

My first iteration of this personal website was written in GatsbyJS, primarily as a means to show off the previous things I have built more info in this previous article.

However, I rarely wrote to this website, and I never really promoted it on other platforms for the purpose of getting views.

I also wanted to write about things that didn’t necessarily fall under the rather specific category of “project”.

As such, my website was starting to develop an identity crisis and the weakest part of the website was, frustratingly, my own writing.

I was originally hoping that this website would offer me a creative outlet where I could practise and improve my own writing style, with the auxillary goal of basically being a verbose CV to show to prospective employers or my peers.

But, GatsbyJS was simply too flexible for this purpose and in turn became a burden, drawing away my own attention from writing and providing content for this site.

This rewrite, is deliberately hands-off.

I wanted to keep the same design language, but with a further emphasis on writing, both project-style where I want to write a summary, but also to create the space for more long-form content in the future. Eventually, I would like to expand this approach to also create the space for engagement with readers in the future (once I get readers!), however this will not be a priority yet until I am confident in the content.

For now, the rewrite has resulted in removing a few thousands lines of JavaScript, leaving the entire website as just a collection of markdown files and image assets. Much nicer.

Why not [medium, dev.to, x.com, etc]?

I want full control over my presence on the internet.

I want to be able to decide entirely by myself whether I am public, private, free for all to read, or behind a paywall.

I want to have at least a false sense of security that my writing will not be too easy to crawl for LLM’s to reproduce.

But most of all, I want to be able to go back to the old micro-web of days gone past, where each person has their own bubble on the internet. I think it’s quite a charming approach, and I hope you do too.

I will possibly start to link to my website from other platforms when I post a new article, but I want the location to stay here, on this domain.

Learning to write

I am going to take a mini-sabbatical from work later this year, spending 4.5 months out of the U.K., working and travelling around South East Asia.

I will be leaving my personal laptop here, only taking my phone, camera and a notebook. I’m hoping to teach myself how to enjoy the creative process. If that works, You’ll be hearing more from me soon!

⠪ Mapping the Internet, part deux

hello@jamesjarvis.io (James Jarvis) — Fri, 05 Feb 2021 00:00:00 +0000

Did reading the previous instalment of “mapping the internet” read here leave you a bit… underwhelmed?

Same here. I mean, sure I had built a little program that could follow and store links on the internet, but it couldn’t be ran continuously (could not store the crawling state between runs), and there was just no easy way to traverse or visualise the graph yawn.

So, in this chapter I am going to go through how we have improved this service to become the beautifully graphical service that is:

jamesjarvis.github.io/web-graph

NOTE: By default, the graph starts from jamesjarvis.io, but if you want to change it to start from a page of your choosing (provided it has managed to scrape that part of the web already), you can give it a shot by changing the url query parameter such as: /?url=https://en.wikipedia.org/wiki/London

Part 1: Maintaining state

So I wanted to eventually run this service on a raspberry pi 4 I currently have just sitting in my attic doing occasional speedtests.

I’ve also decided that I want to be able to stop and start the crawls (say, when performing an upgrade), without losing any data between runs.

In order to facilitate this, I need to have a list of URL’s that have been “seen” but not “crawled” yet.

I had 3 ways I could think of doing this:

Spin up a message queue system, like kafka or rabbitMQ to store unprocessed URLs and yeet them out to processors for the actual crawling.
Add another table in the database, with the pending URLs.
Store this information in a simple queue file (read from the top, append to the bottom).

I tried all 3:

RabbitMQ

First, I tried RabbitMQ, and got it working after a few hours of mucking about with it.

However, it ate my poor raspberry pi’s RAM as soon as the queue was over a few thousand URLs, and I couldn’t work out how to persist messages properly.

I also didn’t really need the complexity, as I didn’t want my ISP to boot me off the network for making a continuous 100 requests/s, so I wouldn’t have bothered with more than one consumer anyway.

If I was building this to be an actual, scaleable service, then I’d have swallowed the extra AWS bill and continued, but this ain’t it chief.

Whack it in a DB table

Another embarrassing thing about the infra for this project is that I’m running the raspberry pi with an 8 year old 1TB external hard drive I had lying around.

It’s not the fastest and is in fact the biggest performance bottleneck (if I could be bothered, I would have bought an SSD but this isn’t worth £80 + dealing with this limitation is interesting enough).

As a result of my cheapskate-ness, this is also out of the question, I needed something more lightweight.

Simple file based queue system

This was the way, effectively traverse the web in a breadth-first fashion by appending any found URLs to the end of a list, and crawl any URLs by taking from the top.

Nice and easy.

Part 2: Making the data easy to query

Screenshot of the API splashscreen

Eventually, I wanted to expose this information, but in an easily cacheable and web friendly way.

Enter: an API service.

You can actually check it out if you want at api.jamesjarvis.io.

It’s been set up to be as easily cache-able (thanks to cloudflare) as possible, and it’s open for anyone to use, provided you’re aware it’s hitting the shitbox in my attic and I don’t want my ISP to throw me off their network.

Also, I used different queries than the first iteration of this project, in order to speed it up. Indexes help a lot as well! Once the DB grew to >100k items queries used to take minutes, and now after some enthusiastic indexing can take <1s.

Some fun things you can do with it:

Find all pages at a particular host (limited to 100): /pages/en.wikipedia.org
Get an overview of a particular page (just URL, host and links from, limited to 100): /page/HASH
Find all links from a particular page (limited to 100): /linksFrom/HASH
Find all links to a particular page (limited to 100): /linksTo/HASH

The last one is currently not being used by myself, though I may make an inverse graph traversal, whereby you can discover all the pages that link to a page…

I’ve also set up my e-ink display to monitor the number of URL’s crawled and links found for my own amusement as well.

Self hosted??

Yup, it’s slow as shit and is as reliable as my ISP (who apparently work on a best effort case), but thanks to some aggressive cloudflare caching, the most common endpoints should be relatively quick.

Part 3: Graphical representation

“What would a visual representation of browsing the internet look like?”

Check it out at jamesjarvis.github.io/web-graph , and if you want to start from a website of your choosing, you can modify the url query param like: https://jamesjarvis.github.io/web-graph/?url={your_page_here}.

Because I couldn’t be bothered to set up a proper react application + build process for this project, the whole thing is in one html file, and actually transpiles the required javascript from reactJS source on load time - maybe not the fastest but hey this isn’t exactly a performance project + it’s so easy to deploy.

You can check out the whole front-end source here, ~200 lines is all you need for this beauty.

I ended up using a react wrapper around D3.js for the actual graphing, and it made me remember just how much I love and hate javascript. It’s a rocky relationship.

You’ll also notice that running this site is very cpu intensive, and yes it is, nothing more to add really. My personal record was 1874 nodes and 3556 links expanded before my browser crashed, your mileage may vary.

Part 4: What have I learnt from this?

Hardware can be a real bottleneck, next time use an SSD for IO intensive operations.
Building services to be resilient to failure is extremely liberating.
JavaScript still sucks to use, package management is still hell and documentation is generally just a bit weak, considering you get zero type information from the language itself.
Proper database indexes are the difference between good and shit.
Caching is still amazing.

Some of my favourite snippets of code:

Channel based interface for reading off of a queue

Look how neat this is, as waiting to read from the queue channel / sending to the processing channel are both blocking operations, this works as fast as these two components. We are also able to pre-fetch a number of items from the queue’s file if we wanted to improve performance a bit.

This also enables an easy method of gracefully shutting down the program.

linkProcessor, err := linkprocessor.NewLinkProcessor(
	linkStorage,
	500,
	queue,
	1,
)
failOnError(err, "Could not initialise link processor")

log.Println("Begin processing...")
urlProcessingChan := linkProcessor.SpawnWorkers(2)

sigs := make(chan os.Signal, 4)
signal.Notify(sigs, syscall.SIGTERM, syscall.SIGINT, syscall.SIGHUP, syscall.SIGKILL)
ticker := time.NewTicker(10 * time.Minute)
defer ticker.Stop()
processing := true
for processing {
	select {
	case s := <-sigs:
		processing = false
		log.Printf("Received signal %s, shutting down gracefully...\n", s)
	case url := <-queue.DeQueue():
		urlProcessingChan <- url
	case <-ticker.C:
		log.Printf("%d urls in the queue", queue.Length())
	}
}

<-linkProcessor.GracefulShutdown()
log.Println("===== Shut down link processor =====")

Periodic batcher from a buffered channel

This is a slightly improved version from the original post.

This turns out to be a great way to convert a message by message insert operation into a batchable operation. It also works as a great way to rate limit database operations, as you can still fill up the channel up to the buffer size, and slowly drain it at whatever rate you choose.

func (pb *PageBatcher) Worker(endSignal <-chan bool, doneChan chan<- bool) {
	// We want it to die on the endSignal, but otherwise keep looping
	for {
		select {
		case <-endSignal:
			doneChan <- true
			return
		case <-time.After(100 * time.Millisecond):
			var pages []*Page

		Remaining:
			for i := 0; i < pb.maxBatch; i++ {
				select {
				case page := <-pb.bufChan:
					pages = append(pages, page)
				default:
					break Remaining
				}
			}

			if len(pages) == 0 {
				break
			}

			// The batch processing
			err := pb.s.BatchAddPages(pages)
			if err != nil {
				log.Printf("Batch adding pages failed!: %e", err)
			}
		}
	}
}

🗺 Visualising location history

hello@jamesjarvis.io (James Jarvis) — Tue, 01 Dec 2020 00:00:00 +0000

Check it out on GitHub!

So this one is more of a revisit than a totally unique project, but this all came about thanks to a little sprinkling of dependency hell.

The goal of this little project was to be able to visualise my google location history (and other source of location info such as strava activities or workouts from my apple watch) on a strava style heatmap like this one.

My first foray into this was little more than a fork of luka1199/geo-heatmap link as I’m a sucker for believing that a lot of GitHub stars means a project is well maintained (silly me).

A previous fork I made tried to incorporate Poetry link for more deterministic python dependency management and execution. Sounds good right?

Python dependency hell

However, when trying to run this script after upgrading to macOS 11, I was getting some issues while trying to download NumPy (see more here).

I know this may seem like a little issue, but the goal for this project was to simply rerun this script every year or so, minimal effort so I can just update a file with my latest location history.

NumPy doesn’t work? Why the fuck not? Why do I care? What does NumPy have to do with creating a .html file?

Since this is ultimately a fairly simple project I decided to just spend a couple hours and rewrite in one of the most backwards compatible languages I know of: Golang.

I also wanted to reduce the dependency on random remote scripts in the resultant .html file as well, so I could open this file with no internet connectivity, in 50 years if I wanted to.

You cannot escape dependencies

Ok so this section is a little bit of a lie, given enough effort I’m sure you can.

My goal of just having a simple program I can run once every year with zero code touching is kinda sorta there (I haven’t given it enough time to check). I’m hoping that Go’s promise of constant backwards compatibility should help this out, and the removal of reliance on remote scripts means that as long as the one dependency I use in go doesn’t go offline, then we should be chill.

Why do you even have dependencies?

I’m lazy, ok?

Only downside going ahead is that I am completely reliant upon the following packages (and github itself) remaining publicly accessible.

github.com/tkrajina/gpxgo v1.0.1
github.com/urfave/cli/v2 v2.3.0

Whilst this little pain in the ass has taught me to be a little bit hesitant to blindly import random dependencies, tbh I’ll probs carry on doing so, because dependencies are hell but I love getting code for free.

⠪ Mapping the Internet (?)

hello@jamesjarvis.io (James Jarvis) — Sun, 04 Oct 2020 00:00:00 +0000

Once upon a time I was looking at my website, and I realised that I tend to link to a lot of random sites within each post I write.

I thought it would be interesting to find out all of the sites that I link to, and see if I could categorise them.

Unfortunately, I never really got around to figuring out the whole categorisation thing (like what do you even do? Manually classify them? Do some kind of NLP on the page title? These guys have attempted it previously but even that dataset only contains ~27k URL’s and seems fairly limited to UK sites).

This then got me thinking about what kind of sites link to me. This would be a backlink checker, that for some reason I had never heard of or thought about before.

I knew that Google, the NSA and many others regularly crawl the internet in order to improve their search products - cloudflare explains it well - and I roughly knew that the more linked to your site was, the more popular it would be in search rankings.

However I didn’t know how to find your individual backlinks - I just assumed it was some secretly guarded knowledge due to the sheer amount of effort required to find these backlinks.

Of course, me being me I didn’t bother googling “uh how do i found out who links to my site on the interwebs pls”, I just thought it would be easier to build a crawler myself 🤦‍♂️.

If you want to find out your own backlinks from an actual SEO service, skip to Part 2

Part 1: Building a link spider

I attempted and revisited this several times (to improve performance, mainly), but it basically started because I was heading to the middle of nowhere for a family holiday and I wanted to leave my laptop back at home doing some crawling while I couldn’t hear the fans.

Originally, I thought I would just write the crawler from scratch (it’s not hard to do tbf), but then I stumbled across Colly that looked like it could speed up development for me.

My original version just used a basic colly set up, with a simple postgresql db set up to record pages visited, and links from page to page. Simple enough, and took an hour or so to code, so I left it running the evening before we left to see how far it could go.

Well, when I came back 4 days later I found that the fans had stopped, and discovered that the process had killed itself after scraping only 1.5 million links over 36 hours (averaging only ~12 links per second).

I couldn’t see an obvious reason why it would have crapped itself, so I had a deeper look using go’s fantastic inbuilt pprof tool for delving into the internal stats of a running go app.

What I had found out was that the crawler had likely ran out of memory, but most surprising to me was that colly’s crawlers seemed to just spawn another process for each a[href] element found, causing them to spawn hundreds of goroutines every second.

This is a bit daft, so the first improvement I made was to switch to colly’s queue mode, as that seemed to handle colly’s goroutine obsession, and allowed me to sacrifice scraping certainty (since any links found beyond the queue limit are just discarded) for runtime certainty.

Secondly, let’s focus on that paltry 12 links/second. The first improvement I could think of was just spawning more processes, but the new bottleneck became the way that I was pushing data to postgresql.

Previously, I was just creating a new insert transaction every time a new link was found, that was hammering my db with well over 2000 transactions per second and likely being the reason why all of these goroutines were hanging around for so long.

So, I moved to batching, by creating n database workers, I could happily push as much data into them through some go channels, and then I could consume up to x links at a time from these channels, and whack those in the database in one go.

// Worker is the worker process for the link batcher
// This is straight up nicked from https://blog.drkaka.com/batch-get-from-golangs-buffered-channel-9638573f0c6e
func (lb *LinkBatcher) Worker(endSignal chan bool) {
 // We want it to die on the endSignal, but otherwise keep looping
 for {
 select {
 case <-endSignal:
 // Die on endSignal received
 return
 default:
 var links []*Link
 // Now I'm typing this blog, I realise there is a potential deadlock here,
 // whereby the endSignal may not be received while we wait for the next link
 // oh well, turns out we never use an endSignal in an infinite scrape anyway ¯\_(ツ)_/¯
 links = append(links, <-lb.bufChan)
 remains := lb.maxBatch

 // Here we use a go label to break out of the repetition loop if there are no more links
 // in the channel, otherwise it will naturally break once the max batch size is reached.
 Remaining:
 for i := 0; i < remains; i++ {
 select {
 case link := <-lb.bufChan:
 links = append(links, link)
 default:
 break Remaining
 }
 }

 // Then we just whack those links into a database batch insert function.
 err = lb.s.BatchAddLinks(links)
 if err != nil {
 log.Printf("Batch adding links failed!: %e", err)
 }
 }
 }
}

Obviously, this introduces a whole bunch of fun async issues, such as potential duplicates and foreign key constraints (this can be solved by reorganising the db structure), and to that I say fuck it; ON CONFLICT DO NOTHING, turning a blind eye to those issues I should probably engineer out of this.

The only thing I did kinda care about was removing duplicates from being added to the channel, since if I was going to add 5000 links, with 4000 of those potential duplicates, it would seriously slow down any transaction.

For this, I just used a simple LRU Cache called golang-lru to filter out some of the most common URLs.

In the current iteration (that I’m not going to bother improving for the foreseeable future and you can critique here), I managed to collect ~ 50 million links between ~ 9 million pages in ~ 3 days.

That’s a happy 16x improvement upon my initial attempt, but I am unfortunately starting to hit the limits of my 7 year old laptop (running the COUNT(*) query just to get those numbers took several minutes with nothing else running…)

If I can be fully bothered, I may write another post exploring the data, or leave it running a couple weeks and upload a csv file of the links found, but for now I’ll leave you with a couple interesting things I found:

Wikipedia is a crawler’s best friend, with the #1 number of pages found.

Top 5 for reference:

host	num_pages
“en.wikipedia.org”	2375201
“twitter.com”	533493
“web.archive.org”	464023
“artuk.org”	394299
“t.co”	258350

I only managed to find a total of 5 external backlinks to <jamesjarvis.io> pages through this crawler, all of which I also link to within my own website, so I guess I would have had to run the crawler a LOT longer in order to organically find any proper backlinks.
Regrettably, I am unable to draw any further conclusions, since my old ass laptop ran out of storage before completing this simple query for finding the top 5 backlinked sites:

SELECT url, COUNT(from_page_id) AS num_backlinks
FROM links_visited
INNER JOIN pages_visited
ON links_visited.to_page_id = pages_visited.page_id
GROUP BY url
ORDER BY num_backlinks DESC
LIMIT 5

(No, really, apparently running this query took up 10GB+ of temporary storage on the database and since my laptop is filled with 200GB of “other”, it just died.)

#GetJamesADecentDevMachine2021

In the future, I would like to continue with the original “mapping the internet” goal, and see if it would be possible to generate an image of the internet link graph? Although certainly not with this laptop…

Part 2: Actual SEO services for finding your own backlinks

Ok, that handcrafted web crawler might have been cool and all, but the max I could get before crashing my PC was only a few million links, at a pathetic 100 links/second.

This pales to the big leagues, who according to ahrefs have approximately a fuck ton more resources than I do for this kind of stuff. Some things you just have to leave to the professionals…

So heres the list of all the free* backlink checkers if you are ever curious about your own SEO performance:

Ahrefs - seems to have the largest trough of data at their disposal.
NeilPatel - I’ve started using this tool now and it seems pretty cool. Might be a good game to increase your backlinks overtime.
SmallSEOTools - This one got some slightly weird results for me, so idk how good it is.
SEOReviewTools - This one was kinda interesting, as it showed uses of static assets from my site. So if you want to see why one of your images is getting 1000+ views and your posts aren’t, this could help.
TheHoth - Wants an email, not a fan of that.
SiteChecker - Limits results so whilst you can see a count of backlinks, you can’t see more than 20 of the actual referring urls.

☂️ E-Ink Weather Dashboard

hello@jamesjarvis.io (James Jarvis) — Sat, 20 Jun 2020 00:00:00 +0000

Another one of my lockdown projects…

I’ve always like the idea of making an always-on display. Something like the Magic Mirror project here, or a productivity dashboard using an LCD display here, but I also wanted something that is a bit more visible in bright light and uses a little bit less electricity.

Hence, I settled on the idea of making a giant e-ink display that I could use to show whatever I wanted. Until I saw how expensive the panels are, and I settled on a much smaller 7.5" e-ink 2 colour display.

What does it do?

Well, there’s a few modes (in case I ever get bored).

Normally, I keep it in weather mode, whereby every 10 minutes it will update and show the following:

The red splotches are rain

Temperature
Current weather conditions
Rain chart for the next few hours
Air quality
Sunrise and Sunset times
The next time the ISS will pass overhead
Weather radar of the UK (just because it looks cool)

I won’t lie, the 3 hour rainwall forecast is nowhere near as accurate as DarkSky was, and the display panel I was sent didn’t display red’s anywhere near as vibrantly as other panels I have seen (I think it was just a faulty panel but I couldn’t be bothered to pay for shipping to China and wait another month), but overall it’s worked exactly as I have envisioned!

There’s also the cowsay mode, whose sole purpose is to present a random dad joke every hour, in the popular linux cli tool format.

cowsay moo

How to build it?

It’s relatively simple, really…

I like raw materials just as much as I like an easy assembly

You only need a few components: A Raspberry Pi Zero W, an e-ink display (I chose this 7.5" 2 colour display from Waveshare), and something to mount it all one.

The instructions for putting it together are fairly self explanatory, so I’m not going to bore you. Setting up a headless Raspberry Pi (so you can just access it over SSH is explained beautifully simply over at the Raspberry Pi site) Aside from that it’s just plug and play.

The software is a little bit more of a pain in the ass, Waveshare have a GitHub repo with drivers for each of their displays, but no perfect method of building a UI for these displays, so the best way to push something to the display is in the form of a bitmap (essentially png) image. The refresh rate of the display is a pathetic ~1 minute so don’t worry too much about how long it takes to build that image.

I have a repository with all the code for this project on my GitHub here, that you just need to clone into the RPI’s home directory, and then write your own settings.py file from the example file provided.

It makes extensive use of various API’s, including Climacell for the weather data, ISS Open Notify for the ISS times, ICanHazDadJoke for the dad joke service, and This online cowsay generator because I couldn’t be bothered to generate it myself, although looking back it would probably have been easier to do it manually.

For the map images, I used the OpenStreetMap Toner style tiles for the map and the MetOffice rain radar tiles for the rain overlays. I originally used OpenWeatherMap, but whilst it was good at global coverage of rainfall, it had very low resolution.

To run the scripts all you need to do is create a cron job for it to update every x minutes. You can enter the cron editor with crontab -e and just input */10 * * * * python3 /home/pi/mappyboi.py for a 10 minute update schedule.

Next steps

I practically never do hardware projects and this has been super fun!

I’m still thinking of ideas to use it for (if you have any ideas of a cool use for this format of display, contact me!)

Ideas I’ve currently considered:

Coronavirus cases tracker
Public transit departure times
Health dashboard
Word clock

📸 Indoor Garden Timelapse

hello@jamesjarvis.io (James Jarvis) — Tue, 02 Jun 2020 00:00:00 +0000

Who knew plants could move so fast?

Another lockdown project!

For some reason I’ve been getting into gardening over the lockdown, from building an automatic irrigation system for our newly built vegetable patch to taking part (and winning) a sunflower growing competition, it’s about the most interesting thing I could think of doing whilst being stuck in my house during a great spell of weather.

This year, for my birthday I received an indoor vegetable garden, and thought it would be fun to set up a timelapse camera to monitor the plants growing!

Since I’m currently also slightly addicted to Raspberry Pi’s (now up to 4 in my collection!), I thought this would be a good use for one of them.

So, I bought the Raspberry Pi Zero W camera combo, strapped it to the side of the garden and adjusted the focus of the camera with a pair of pliers to improve the focus a little bit.

The end result is this cute little video:

This little video was the result of 2 months of growing and almost 200GB of images! I set up the camera to take a photo every 30 seconds (which in hindsight was a little bit too often, and the final video only took a frame each 16 minutes)

Scripts

I thought I would share the little script I used to pull images from the camera, sort them into good (clear, well exposed) and bad (too dark, fuzzy, etc), and then render them into a timelapse…

Firstly, on the camera all you need is a little script that helps with actually taking the photos:

#!/bin/bash

take () {

 DATE=$(date +"%Y-%m-%d_%H_%M_%S")

 raspistill -t 100 -q 20 -o /home/pi/timelapse-images/$DATE.jpg

}

take

sleep 29

take

Just chmod +x it and add it to the cron jobs with crontab -e and if you want it to run every minute, then append this to the crontab:

* * * * * bash /home/pi/camera.sh

Now, on another computer (preferably faster and on the same network as your RPI), you can set up a little script like:

#!/bin/bash

echo "!!! Requires EXTERNAL-SSD to be plugged in at all times !!!"

echo "Retrieving latest images"

rsync -a -v --ignore-existing --remove-source-files pi@pi-cam.local:/home/pi/timelapse-images/ /Volumes/EXTERNAL-SSD/timelapse-images

echo "Filtering and linking based on shutter speed to get rid of flickers - VERY SLOW STEP"

echo "Checking and skipping a few thousand photos..."

for f in /Volumes/EXTERNAL-SSD/timelapse-images/*.jpg
do
 fname=$(basename $f)
 # If not bad and not good (ie: not sorted yet), process and filter
 if [[ ! -f /Volumes/EXTERNAL-SSD/filtered-timelapse-images/$fname && ! -f /Volumes/EXTERNAL-SSD/bad-timelapse-images/$fname ]]; then
 ss=$(exiftool -exif:shutterspeedvalue -s -s -s $f | cut -c 3-)
 sss=${ss/.*}

 # If the correct shutter speed, then place in the filtered folder, else cache in the bad folder
 if [ "$sss" -gt 30 ]; then
 echo Linking $fname
 ln -s $f /Volumes/EXTERNAL-SSD/filtered-timelapse-images/$fname
 else
 echo Caching bad photo $fname
 ln -s $f /Volumes/EXTERNAL-SSD/bad-timelapse-images/$fname
 fi
 #else
 #echo Skipping $fname
 fi
done

echo "Generating timelapse"

ffmpeg -r 1920 -pattern_type glob -i '/Volumes/EXTERNAL-SSD/filtered-timelapse-images/*.jpg' -vf "transpose=1" -vcodec libx265 -r 60 /Volumes/EXTERNAL-SSD/filtered-timelapse.mp4 -y

echo "Done!"

open /Volumes/EXTERNAL-SSD/

This script obviously requires an external ssd, and will require some modifications for you to use. It also requires ffmpeg to be installed locally, so make sure that’s all set up before any annoying errors hit you in the face.

Basically all this script does is download the latest images from the RPI, delete them from the RPI once done, then filter them into symlinked directories of “good” and “bad” photos - essentially caching them.

Then it triggers a rendering job that takes absolutely FOREVER to create a h.265 encoded timelapse of it all. (Originally I used h.264 but h.265 took up honestly 1/8th of the space without losing much quality which is fantastic!)

P.S. The gif at the start of the page is generated with a one liner:

ffmpeg -ss 3 -t 20 -i /Volumes/JARVIS-SSD/filtered-timelapse.mp4 -filter_complex "[0:v] fps=10,scale=480:-1,split [a][b];[a] palettegen [p];[b][p] paletteuse" /Volumes/JARVIS-SSD/timelapse.gif

However, gifs are rather inefficient, the better option would be an animated webp image, which is unfortunately not supported by GatsbyJS, nor some old browsers, but it would be 10% the size of the gif version (and better quality!) - to generate it I used the following command:

ffmpeg -ss 3 -t 20 -i /Volumes/JARVIS-SSD/filtered-timelapse.mp4 -filter_complex "[0:v] fps=10,scale=640:-1" -vcodec libwebp -lossless 1 -q 30 -preset default -loop 0 -an -vsync 0 /Volumes/JARVIS-SSD/timelapse.webp -y

Hope you’ve enjoyed!

Before:

After:

📦 IPFS Web Hosting

hello@jamesjarvis.io (James Jarvis) — Sat, 02 May 2020 00:00:00 +0000

2020 has been defined by The Lockdown (yay?). From March to July (and perhaps again, if the (potential) second wave is to come), basically everything shut down and we were all forced to spend time with family, and do nothing.

Well, around this time I was thinking “what are the problems humanity will face in the next 100 years?”, and at some point I will write a little post outlining my daft ideas.

But during this, I came across the IPFS project, and I’m astonished at the idea of solving a problem that does not yet exist.

What is IPFS

Their goal is to create a distributed web protocol that can help distribute content to where it is most requested. Ideally, this aims to prevent issues of enormous latencies that may arise from traditional centralised services, if humanity becomes interplanetary.

An example being, if you are living on mars, and the twitter servers are still hosted on earth, who would want to browse twitter if it took 13 minutes between opening twitter and each tweet starting to load?

Exactly, ain’t nobody got time for that. IPFS is basically a mix between a conventional file system, and torrents. The basic idea is that if an IPFS node requests a file (using a content-addressable hash), then that node automatically starts serving that file to other nodes requesting it. The benefit could be that if for example a video were to go viral, rather than the central server being absolutely hammered with requests, each person would essentially be downloading it off their neighbour, greatly reducing network congestion.

Now, I am certainly not the first to do this, and a great write up can be found here.

Long story short, I set up a little Go-IPFS node on my Raspberry Pi, and uploaded a copy of my site to this probably out of date mirror.

Now, more detail:

GatsbyJS sucks with IPFS

This website is (currently) built with GatsbyJS, as outlined here.

Only thing is, GatsbyJS does not support relative links by default. Sure, you can enable gatsby-plugin-ipfs to try and fix the relative urls problem, but it is not compatible with most other plugins.

As a result, Gatsby on IPFS does not work with manifests, sitemaps, or the most important gatsby-image integration.

So sure, if you browse the mirror on a normal browser, then it will work fine, since IPFS content is being forwarded through the cloudflare gateway the url is being formatted with the normal domain, and all the links work fine.

However, if you browse the mirror on a browser with integrated IPFS (as it should do), then stuff like images and some other content will not load correctly.

The future

I spent a few hours on this problem, and honestly I just gave up.

Perhaps in the future Gatsby will update to include relative URLs by default, but if you are absolutely desparate to host your content through IPFS, then I would recommend a different static site generator, such as Hugo.

If you are looking at a potentially commercially viable implementation of IPFS, I would recommend you check out Filecoin.

I know, I know “not another cryptocurrency”, but this one is designed to pay people to host their content on IPFS, and in the future could potentially take on the big cloud providers?

🍲 Risotto Play

hello@jamesjarvis.io (James Jarvis) — Mon, 20 Apr 2020 00:00:00 +0000

Try it out here!

Sometime back in 2018, a friend of mine started building his own programming language, “Risotto” (GitHub link) (although I wanted the name to be “Ravioli” so the syntax could be “Ravioli Ravioli, give me the formuoli” (I know, I’m immature)).

Anywho, I’ve enjoyed using services such as The Go Play Space and Repl in the past, and wanted to replicate this kind of environment (whilst paying 0 for hosting) for Risotto.

The API

Code

This runs on Kubesail for easy K8s deployment (and the free tier), with the API written in Go, and spawning a process running the code passed to it.

In order to keep the API service performant, there’s a little bit of IP addressed rate limiting, and runtime constraints, but overall the traffic is low enough that this hasn’t come close to being an issue yet.

The API is designed so that if anyone wants, they can use it for their own purposes, which is highly encouraged (get Risotto out there!)

It is also covered with Unit Tests, and automatic CI/CD.

The UI

play.risotto.dev

Code

This is hosted with GitHub pages for easy CI/CD and just fantastic performance.

Written in React (TypeScript flavour), it’s fairly simple but is optimised for responsive design and even has a cute little “Share” feature, so you can share your creations with others.

📆 WhatsUpKent: Everything happening at the University of Kent

hello@jamesjarvis.io (James Jarvis) — Mon, 04 Nov 2019 00:00:00 +0000

In October 2019, I was looking for a challenge.

University (look mum, I’m on the front page!) was going ok, but I didn’t feel stimulated enough, and wanted to see what else there was on offer. Turns out paying £9k per year to learn from 20 year old textbooks isn’t exactly riveting, so I was looking to see what other interesting lectures I could discover on campus. Also, my friends and I liked to use disused seminar rooms to work in since the library was often overcrowded and I wanted an easy way to discover which seminar rooms were not being used.

I also wanted to stretch my legs and learn a bit of Golang, Kubernetes, Typescript and Graph databases.

So, I decided to create a service that would gather all student’s timetables and present the information in an easy to use dashboard, to offer students the opportunity to explore other courses in their spare time.

Thanks to the university (accidentally?) publishing student timetables as publicly accessible ical links, it proves fairly simple to just brute force and scrape all ical timetables from their website. Of course, having to iterate through 50,000 timetables can be quite time consuming, so I chose Golang to write the server, thanks to it’s remarkably easy multithreading syntax.

Since the data being retrieved was fundamentally relational, I chose to store it in a graph database called DGraph. This allowed me to achieve not only some quick queries on the data, I was also able to just easily expose a read only graphql endpoint directly to the database. Simple! :)

At this point, we are now looking at a few different services for the api: the scraper, a caching service (just to speed up repetetive large queries), and the dgraph database instance (which itself requires 2 pods). So, for deployment I chose to look towards Kubernetes. I bought the cheapests little VPS I could, and whacked on a k3s lightweight kubernetes instance and set that up in the CI/CD (ran with GitHub actions). As always, the frontend was hosted with GitHub pages.

Overall, the project was a success. It managed to scrape (and continuously check for updates and re-scrape) thousands of lectures and seminars on campus, from all students, and all courses. It was hosted at whatsupkent.com, with the backend code hosted in GitHub. Unfortunately, I decided to close down the service before the uni got mad at me (also, I couldn’t be bothered to pay the £5 a month hosting fees), but relaunching it would be as easy as pie if demand ever increases for a service like this.

A few screenshots

The API:

Event info:

Room finder:

Agenda view:

Filtered week:

🖊 Git Art

hello@jamesjarvis.io (James Jarvis) — Tue, 05 Feb 2019 00:00:00 +0000

My roommate Tim (his LinkedIn) and I decided to start challenging ourselves with little applications we could develop.

The first thing we thought of was a way to decorate your GitHub activity board, example below:

The idea being that you could just draw or type (or both) anything that you wanted to on a canvas that represented your github activity board, and with one simple script you could create the required commits and upload that new repo to your GitHub.

This is thanks to a feature/flaw in git that allows you to create commits (even empty commits) with any date attached. Therefore, all we need to do to generate a cute little activity graph is to simply work out the date ranges required, how many commits to make each day, and to generate a script that allows the user to do this.

We used React for the front end, and wrote simple JavaScript functions to generate the bash script which is available for you to download and run.

Check out the website, or if you’d prefer, jump straight to the code

Made open source under the WTFPL licence.

🖥 Open Source Portfolio Website

hello@jamesjarvis.io (James Jarvis) — Sat, 02 Feb 2019 00:00:00 +0000

This very site you are reading, is open source, under the A-GPL licence with all the source code hosted on GitHub (fork off). You may fork it and fill in your own data if you wish, which is surprisingly easy to do, all thanks to GatsbyJS (link)!

I created this with GatsbyJS as I wanted to have my own personal portfolio site which simply consumes data stored externally (though within the same repo), rather than hard coding everything, which I felt was the issue with many other portfolio sites.

Now, I wanted to have a personal site so I could have a place to publicise the work I have done in the past, what I am planning in the future, and my thoughts and ramblings (As well as just to have a place for my ‘digital identity’ where I can link together all of my online personas into a single place).

I am a little bit cheap however, so after spending $30 on domain fees I didn’t want to spend any more money on actually hosting it. Which brings me to some of my favourite pieces of tech: the free kind.

How to host for no cost whatsoever

Since we are already using GitHub as a way to host our source code (for free!), we might as well use the latest and greatest GitHub Pages to host the page source files for free as well! (I know right!??).

Since we are making a JAMStack site, all we have is front-end stuff which we just need to plonk in any old static web hosting provider. Other great examples are Netlify, and even on Zeit.

To build your site, I would recommend setting up a CI/CD integration using GitHub actions, and pointing this to deploy to whatever hosting service you use (GitHub pages is the easiest I feel as you only need to push changes to another branch/folder within your repo).

That is basically all you need to do.

I would recommend also putting your site behind Cloudflare, as this can speed up access to your site due to the way they cache it on any of their CDN nodes - again, all for free!

If you get to the point where you would like a ’lil bit of backend with your site, say for sending contact information or retrieving particular information which is too client-side heavy, then go Serverless!

Serverless lambda hosting is extremely quick to set up, maintain, and cheap! Not to mention the fact that you can choose between Amazon, Azure or Google to host your functions, the whole world of data hungry tech corporations are just begging to let you run code with them. Again, for FREE*(or super cheap).

If you are going to go down this whole rabbit hole, I’d recommend having a little read first.

Have fun

Check out the code on GitHub https://github.com/jamesjarvis/jamesjarvis.io.

As mentioned before, feel free to fork it and change the content yourself.

The content is hosted in assets/ and src/pages/projects/.

If you have any more questions, reach out to me!

🎵 Musish: The Unofficial, Open Source Apple Music Web Player

hello@jamesjarvis.io (James Jarvis) — Wed, 12 Dec 2018 00:00:00 +0000

On December 1st 2018, a group of friends and I attended the MLH Local Hack Day in San Francisco.

Made by myself (me), Brychan (link), Raphael (link), and Filip (link).

This particular hackathon was sponsored by GitHub and Microsoft, and had a theme of creating the best open source project for the community. We came first in San Francisco for our project.

We decided to make an Apple Music web player, since pretty much every single other music streaming platform already had one. Some people aren’t sure why Apple Music would need a web player, since surely you could just use iTunes for that?

Well, as was hilariously summarised in a comment on Macrumors, iTunes isn’t exactly lightweight.

Musi.sh (Musish) is an open source project now, using the A-GPL license, and as of February 2019, has over 1400 stars, and 9 contributors.

We released it (sent out a few emails) while walking around Central Park, NY on the 18th January 2019. We weren’t really prepared with what was to come, and have since had quite a few articles written about it.

Analytics and Feedback

According to Google Analytics, we have had over 50,000 unique users on Musi.sh from over 110 different countries, and we have found some very positive articles written about it at least 13 different languages found so far.

Some notable mentions:

Also, we made it to #1 and #2 on GitHub trending for all JavaScript projects and all languages respectively on the 19th January 2019, which was our favourite achievement of all!

Number 1 on JavaScript trending

Number 2 on GitHub trending