I sometimes needs to create custom Neo4j Docker images to try things out and wanted to share my work flow, mostly for future Mark but also in case it’s useful to someone else. There’s already a docker-neo4j repository so we’ll just tweak the files in there to achieve what we want. $ git clone git@github.com:neo4j/docker-neo4j.git $ cd docker-neo4j If we ...

In my last post I showed how to map Strava runs using data that I’d extracted from their /activities API, but the API returns a lot of other data that I discarded because I wasn’t sure what I should keep. The API returns a nested JSON structure so the easiest solution would be to save each run as an individual ...

I’m a big Strava user and spent a bit of time last weekend playing around with their API to work out how to map all my runs. Strava API and polylines This is a two step process: Call the /athlete/activities/ endpoint to get a list of all my activities For each of those activities call /activities/[activityId] endpoint to get more ...

Whenever I need to quickly spin up a web application Python’s Flask library is my go to tool but I recently found myself wanting to generate a static HTML to upload to S3 and wondered if I could use it for that as well. It’s actually not too tricky. If we’re in the scope of the app context then we ...

I recently wrote a blog post showing how to create a Python ‘Hello World’ AWS lambda function and manually invoke it, but what I really wanted to do was have it run automatically every hour. To achieve that in AWS Lambda land we need to create a CloudWatch Event. The documentation describes them as follows: Using simple rules that you ...

Continuing on from my post showing how to create a ‘Hello World’ AWS lambda function I wanted to pass encrypted environment variables to my function. The following function takes in both an encrypted and unencrypted variable and prints them out. Don’t print out encrypted variables in a real function, this is just so we can see the example working! import ...

I’ve been playing around with AWS Lambda over the last couple of weeks and I wanted to automate the creation of these functions and all their surrounding config. Let’s say we have the following Hello World function: def lambda_handler(event, context): print("Hello world") To upload it to AWS we need to put it inside a zip file so let’s do that: ...

In my last blog post I showed how to convert a JSON document containing meetup groups into a CSV file using Luigi, the Python library for building data pipelines. As well as creating that CSV file I wanted to go back to the meetup.com API and download all the members of those groups. This was a rough flow of what ...

I’ve been playing around with the Python library Luigi which is used to build pipelines of batch jobs and I struggled to find an example of an ExternalProgramTask so this is my attempt at filling that void.                   I’m building a little data pipeline to get data from the meetup.com API and ...