Hi, I'm a student who will soon enter my final year. I'm reviewing different productivity apps to see which one can help me waste less time and have more time to work on my personal projects. This is the fifth week, and I'll share my experience using Focuster, the focus manager for entrepreneurs.

Effectiveness

Positives

• Simplistic interface: Focuster has a very simple interface, which doesn't overwhelm you with information. The main screen has three tabs, a to-do list of all the things you do, a list of all things you plan to do today, and a calendar view of the tasks. You first create to-do items, drag it to the middle for it to be scheduled, and it will show up on the calendar on the right, simple as that.

• Increase Awareness of time spending: Same as Motion and Reclaim, you have a clear view of the tasks you will do that day, and how your time is allocated.

• Visualization of how much you can do in one day: The bar just below the date shows how many tasks you have scheduled for the day, so you know how much you will be working and don't overwork yourself.

• Instant reschedule if not done: If you scheduled a task and didn't do it, it will be rescheduled for you.

• Have different lists for tasks: You can have tasks on different lists, which makes it easy to manage

Negatives

• Limited support on routines: Having routines is useful when I plan to use this app in a personal context, for tasks such as taking a bath, doing laundry or checking emails. In Focuster we can add routine tasks, but you can't have a preferred time for the task. There also isn't an interface where you can see all of your routines.

• Need to manually add tasks to a day: Except for routines, Focuster doesn't automatically add tasks for your day, so you have to add them yourself.

Ease of use

Positives

• Simple induction: Focuster gives very simple onboarding tasks when you sign up, instead of having an entire onboarding course.

• Integrate with Google Calendar: Once you link it up to your Google Calendar, you can start scheduling.

• Quick startup/load: Compared to Motion or Reclaim, Focuster starts up almost immediately, and loads very fast too, so we don't have to wait when we open the app

• Easy shorthand for adding tasks: You can type "wash dish 20m every day" when adding the task, and it will become a task called "wash dish", which takes 20 minutes, and will be scheduled every day. Very handly.

Negatives

• No mobile app: They do not have a mobile app, so it is a bit harder to add tasks on mobile.

Customizability

Positive

• Colours for tasks in different task lists: So we can easily distinguish tasks for different projects.

Negative

• No custom time schedule: We only have a single schedule called work, We cannot have a separate schedule for school or personal tasks, the schedule also starts and ends at the same time every day, and we cannot have a different schedule on weekends.

Cost-Value Proposition

The price for the basic plan is basically $8/£6.36 per month for only the most basic features, and for the other ones, they cost $15/£11.92 per month. This is for the annual pricing, for monthly payment, the price goes up to $10 per month for basic, and $20 per month for pro.

This app provides a solid set of functionalities for a scheduling app, and the price is definitely not unaffordable. This price for the basic plan is similar to the starter plan for Reclaim, but Reclaim seems to come up on top with more features.

Experience

Using this app feels very different from Motion or Reclaim, the one feels much simpler, adding to-do items, then dragging them to the calendar, and marking them as finished. While this one doesn't have the more advanced options such as auto-scheduling, it is still a very solid way of managing your time.

Next week

Next week I will be trailing a vastly different productivity tool, Habitica - Gamify your life. Instead of scheduling things for you, this productivity tool makes tasks into quests you do, like in a game, to motivate you to do your tasks. I'm excited how this will turn out.

An app I'm working on

Hi, I'm a student currently doing a summer internship and will soon enter my final year. I'm reviewing different productivity apps to see which one can help me waste less time and have more time to work on my personal projects. This is the third week, and I'll share my experience using Reclaim, a Smart AI scheduler for busy teams.

I reviewed Motion in my last blog post, so this blog post will use Motion as a reference.

Effectiveness

Positives

• Increase awareness of time spent: Similar to Motion, Reclaim lays out all tasks and timetables, providing a clear view of my daily agenda.

• Auto-scheduling with AI: When we have something to do, we click on add tasks, and it will be automatically scheduled on our timetable.

• Optional auto rescheduling: If the task is not marked as done on that day, the tasks will be scheduled again the next day so the task will not be forgotten. The difference between Reclaim and Motion is that in Motion, you have to mark the task as done as soon as it is finished, for Reclaim, you can mark them as done on the same day. (This is an optional feature, the default is to mark everything as done after it happens)

• Statistics: You can see how you are going on the statistics page, which helps you see any improvement or setbacks

Negatives

• Tasks cannot be organized into projects: Some tasks can be combined into bigger tasks, and being into the same project. However, Reclaim does not support organizing tasks into projects.

Ease of use

Positives

• Good onboarding: There is a comprehensive setup guide, it introduces all the features of the app, so you can learn how to fully use this tool. One bad thing is that the startup screens take very long to load, more than 30 seconds for me.

• Integrated with Google Calendar: Tasks scheduled will automatically show up on google calendar, they also have a google calendar plugin to quickly add tasks

Negatives

• Only available on the website: there are no mobile or desktop apps, therefore you must use the app from the website.

• Only supporting Google Calendar: Microsoft or other types of calendars are not supported, luckily I use google calendar anyways so it doesn't affect me

• Slow loading: when the website starts up, it tends to take quite some time to load, which is a mild inconvenience

Customisability

Positives

• Allow multiple schedules: we have a set of time set for work, school or personal life, so we can add tasks to their specific slot

• Comprehensive settings page: Compared to Motion, so much more things can be set, the most helpful for me is default task hours, so when I add a new task, it will be quicker as I have my usual settings in default

• Allow preferred time for habits: you can set a preferred time to complete a habit, for example, you can prefer writing your diary at 11 pm, which makes much more sense than writing a diary in the morning

Negatives

In terms of customizability, I don't really have anything to complain about.

Cost-Value Proposition

The price for Reclaim is £10 per month or £96 per year. This price is almost half of Motion. It is a fair deal if it really helps me increase my productivity. However, since I will only be using the basic functionalities of organizing tasks, but not the more business-centric ones such as meeting scheduling, it may still be a bit expensive.

The good thing is that they have a free tier, which only allows up to 2 habits, but is still fine for basic stuff. They also have an education discount which allows for half price, so that's worth checking out. They also have a discount for if you switch from other providers, which is kind of savage if you ask me.

Experience

Reclaim feels like a more polished Motion, with a cleaner interface, and some more helpful functionality, aside from that, it pretty much accomplishes the same thing.

Next week

Next week I will be trialling another productivity tool: I was going to go with GetClockwise.com since it is listed as one of the competitors of Reclaim, but I found out it is only for business Google accounts. I will go for next.focuster.com instead, and see how it helps me.

An app I'm working on

Hi, I'm a student currently doing a summer internship and will soon enter my final year. I'm reviewing different productivity apps to see which one can help me waste less time and have more time to work on my personal projects. This is the first week, and I'll share my experience using Motion, an AI-based productivity tool for managing tasks and schedules.

Effectiveness

Positives

• Increase Awareness of time spending: Having a visible timetable and task list in front of me, and the obligation to mark tasks as complete after executing them, offered a real-time view of my accomplishments and pending tasks. This kept me aware and accountable for my day.

• Seeing how little time I actually have: Once I saw the recurring tasks scheduled on my calendar, I realized how scarce my 'free' time was, making me appreciate and utilize it more consciously.

• AI auto-scheduling: Letting an AI manage my schedule freed my mind from juggling 'when to do what'. This helped me concentrate more on 'what' needs to be done, enhancing my focus and effectiveness.

• Automated rescheduling: If I didn't mark a task as complete, Motion would reschedule it automatically. This feature was particularly helpful when unplanned events (like a lengthy meal with friends) derailed my schedule, saving me from the tedious task of manually rearranging my calendar.

• Project-based planning: The tool allows the creation of projects and their associated tasks. This made scheduling and tracking tasks related to different personal projects more straightforward.

• Scheduled Breaks: Motion places breaks after every 'x' minute. This helped instil discipline in my break patterns, preventing me from extending breaks excessively.

Negatives

• Not so intelligent intelligent-scheduling: While Motion claims to offer intelligent scheduling, my experience suggested a 'first come, first serve' approach. The tool would sometimes schedule routine tasks, like writing a diary or bathing, as the first activities of the day, which doesn't make sense. I had to manually rearrange these tasks.

• Inefficient time segmentation: Motion mandates tasks be scheduled in multiples of 15 minutes, which doesn't always align with real-world scenarios. Some tasks may take less time, causing potential inefficiencies in the schedule.

Ease of use

Positives

• Comprehensive Onboarding: Motion provides a clear and detailed introduction upon first use. It helped me understand how to set up basic functionalities, making the initial interaction smooth and enjoyable.

• Robust Help Page: The help page on help.usemotion.com is well-organized and informative. It offers guidance on how to use various features, which is a great resource when you're new to the tool.

• Available on multiple platforms: With a Chrome plugin and a mobile app, Motion allows users to schedule tasks on different devices according to their convenience.

• Integrated with calendar: The ability to click and drag on the calendar to create fixed-time events is a significant feature. These events also sync with Google Calendar automatically, reducing the redundancy of managing multiple calendars.

Negatives

• Limited mobile interactions: The inability to drag and create fixed-time tasks on the mobile app can be somewhat frustrating, especially when on the go.

• Very slow startup: The loading time for the app, whether on mobile, web, or the Windows app, is notably slow, typically around 6 seconds.

• Have to manually set tasks as completed: You have to manually mark a task as complete within 30 minutes of finishing it. Although this feature ensures that unfinished tasks are rescheduled, it can be a bit annoying if you're quickly moving between tasks.

• Slow scheduling/rescheduling: Shifting tasks around can be sluggish, taking around 10 to 20 seconds for the system to reschedule. Even ChatGPT replies faster than this.

• Inflexible schedule order: Motion offers different "schedules", which are timeframes where you do stuff. Unfortunately, it doesn't allow editing the order of them, and the work schedule always comes first, I find myself having to switch to 'life' every time I create a new task.

• Bugs: I encountered a bug where completed tasks wouldn't display even when I set the preference to 'show'. Though the issue was fixed later, it was a minor annoyance.

Customisability

Positives

• Definable schedules: Motion allows users to define their schedules, such as 'work' or 'life'. For example, since I use Motion for activities after work, I have set up a "life" schedule for post-work hours on weekdays and the entire duration of weekends.

• Custom task timing: Tasks in Motion can have specific time frames. For instance, you might prefer to take a bath only between 8 pm and 11 pm, and Motion allows for such specific scheduling. It also lets you add preferred times for tasks, making the scheduling more personalized.

Negatives

• No colour customization: Tasks scheduled automatically by Motion cannot be colour-coded. They are all displayed in grey. The lack of colour-coding for tasks can make the calendar look monotonous and limit visual differentiation between different tasks or task categories.

Cost-Value Proposition

The pricing for Motion is $228/£178 per year or $34/£27 per month, it is definitely not cheap. Breaking down the annual cost, Motion comes to approximately 50p GBP per day. The question then becomes: does Motion provide value that exceeds or at least matches this daily cost?

If used diligently, Motion could potentially help you achieve your personal project goals. There's also a chance that these personal projects might generate income, indirectly making up for the cost of the tool. In this way, Motion can be seen as an investment in your productivity and potential success.

However, the other question is, can I achieve the same goal with another tool? there are so many productivity tools out there, and most of them are cheaper while offering functionality that isn't that far off, especially when I don't use the work-based functionality, such as meeting scheduling, kanban boards, or meeting time matches, then the cost might seem steep for the value I'm receiving. Other tools could potentially serve the same purpose as Motion for a fraction of the cost.

Experience

Using Motion has been an enlightening experience. It introduced me to many new concepts in time management. I learned about the differences between free, busy, and scheduled tasks, as well as the concept of task blockers. This newfound knowledge enabled me to better categorize my tasks and identify potential obstacles in my schedule.

The most significant impact of Motion has been on my awareness and utilization of time. With its intuitive task and time management features, Motion has encouraged me to stay on track with my tasks and commitments. As a result, I have noticed a significant reduction in time wastage.

Moreover, even if I do deviate from my schedule due to unforeseen circumstances or momentary lapses, Motion automatically reschedules my tasks. This auto-rescheduling feature prevents me from getting overwhelmed by piling tasks and helps me get back on track without any hassle.

Next week

As set out in this journey, next week I will be trialling another productivity tool: Reclaim.ai, another AI calendar that automatically plans the day for me, and see how well it does to help me rescue my time.

An app I'm working on

I've been seeing a lot of ads on UseMotion lately, it is an AI-powered auto-scheduling tool promising to make your time so efficient it feels like squeezing an extra month into your year. This brave claim reminded me of the hours I’ve allowed to slip by reproductively, particularly after a long work day when my energy dwindles and motivation wanes. Personal projects, once a source of joy and creativity, now lie dormant (as you can see by me not writing a blog post in months). This realization led me to think - is there a better way to organize my time? Could modern technology lend me a helping hand? To find out, I decided to embark on a journey, trialling different productivity tools, with me as the test subject, to see if they could salvage my most precious asset: time.

A little bit about myself

To help you understand how I will evaluate the productivity tools, it's crucial to understand who I am and my daily routines. The effectiveness of any tool greatly depends on the person using it and the specific needs it addresses. So let me introduce myself, I'm a soon-to-be final-year student, currently immersed in a summer internship that ends at 6 pm. Once home, a list of everyday chores awaits me, often stretching into my evening. I also have personal projects that I'm passionate about and yearn to devote more time to. Therefore I will use the productivity tools for my personal life instead of work, I want to be able to do my chores, tasks, and personal projects effectively.

Given that I plan to trial these tools sequentially, spending approximately one week with each, my circumstances will change over the trial period. My internship will end, and my final year of university will commence, altering my daily routine and the time I have at my disposal. Nonetheless, I'll try to be objective when I test the different productivity tools.

Criteria that I will use

1. Effectiveness: How much does it help me to complete my tasks quicker

2. Ease of Use: How easy is it to use the tool

3. Customizability: Since most productivity tools are for work, I want them to be customizable so that I can use them to manage my personal life

4. Cost-Value Proposition: I don't have money as I'm a student, so if I were to subscribe to a tool, it better be worth it

Our first contestant: UseMotion

Motion is an AI-powered productivity app that helps manage your tasks on your calendar, the advertised point is how it automatically builds your schedule using AI, so if you have some change in your plan, the tasks that don't have to be done in that specific time will be automatically shifted. I'm not going into details on how Motion works as this isn't the point of this blog post, I recommend checking out Caffeine's article on it if you want to find out more.

The interface looks something like this, I've added my recurring tasks, as well as my one-time tasks here, if the task doesn't have a fixed time, it will be automatically scheduled on the calendar, that's why the calendar is filled.

Conclusion

I'll be using Motion during its free trial of a week, and discovering its features and do they benefit me. Stay tuned for the next week, where I will conclude my thoughts on Motion, and trial another productivity tool.

• Side note: I will call this a web application from now on. Websites tend to be more static and focused on providing information, while web applications are more interactive and focused on providing a specific service or functionality to users. Therefore my idea classifies as a web app.

Use case diagram

As you can see, I am to provide a lot of functionality to the user. They can be classified into three major parts

1. Interacting with prayer items: Create, edit, and delete them. These are the main functionality of the web app

2. Authentication: Sign in/up/out, and edit account details.

3. Payment: Provide payment method via a Payment provider.

Interacting with prayer items

The idea of the app is to remind the user to pray about things, therefore items that the user should pray for will be displayed. There is also a pray button, when clicked, it indicates that the user has prayed for this item today, and the item will only come back to the list after the selected frequency.

There is an options button, when clicked, opens a dropdown menu, allowing the user to edit, mark as prayed, mark as denied, and delete the item. If the user clicks the item content, the item will expand showing more details.

There should also be an add new button, which will create a prompt for users to fill in details of the new prayer item the user wishes to create.

There should also be two separate areas for answered and denied prayers, with the options of editing and deleting.

Authentication

The web app should support signing up with email and asking for the username in the process. It should also support signing up with OAuth using Google or Facebook.

For sign-in, we should allow users to sign in with email or OAuth. If the user forgot their password, we should allow the user to reset their password through email authentication.

For logged-in users, they should be able to change passwords, change email, log out, and even delete their accounts because of GDPR regulations.

Payment

I want to implement a subscription system so that users that want more functionalities can do so by subscribing to more advanced plans. To do so, I need a page for users to view their current subscriptions, purchase a subscription, and manage their current subscriptions.

Detailed use case

Below I am going to document the key use cases

Use case 1: Display the list of active prayer items

Actors: User, System

Preconditions:

• The web app has established a connection to the server

• The user is authenticated

The flow of events:

1. If the user has encryption enabled, try to read the encryption key from local storage, if it is not there, prompt the user to enter it

2. the web app makes a request to the server for the list of prayers items the user has

3. the server queries the database and sends back the list of prayer items

4. the web app sorts the prayer items by the next_date, then display the ones with a next_date before or equal to today with a for loop

   1. If the item is encrypted, decrypt it with the secret key, and display it with a lock icon

   2. If the item is not encrypted, display it

5. for the prayer items with a next_date in the future, they will be displayed below with a grey overlay, to symbolize that they are queued for the future.

Postconditions: The list of active prayer items is displayed on the UI

Use case 2: Add a prayer item

Actors: User, system

Preconditions:

• The web app has established a connection to the server

• The user is authenticated

The flow of event:

1. the user input the prayer item details

2. the user press add item or press enter on the keyboard

3. the web app performs client-side input validation. Including the input length

4. the web app sends a request to the server to add the item

   1. If the user enabled encryption, the prayer content will be encrypted before being sent to the server

5. the server performs server-side validation, including prayer item count

6. the server adds the prayer item to the database

7. the server increments the total number of prayer items for that user from the database

8. the server sends a response to the web app

9. the web app displays the additional prayer item

Postconditions: the newly added prayer item is in the web app as well as the server database

Alternative flow:

• In steps 5-6, if there is any error causing the insert to be unsuccessful, the server will send back a response about why the insert failed, and the web app will display the reason to the user

Use case 3: New user signup

Actors: User, system

Preconditions: The web app has established a connection to the server

The flow of event:

1. the user completes the sign-up form

2. the web app sends the sign-up request to the server

3. the server creates the user

4. the after-insert trigger function triggers, inserting the user in the other tables

5. if the user uses email auth, they have to go to their email account and verify the email

6. the user logs in

Postconditions: the user lands on the dashboard, and sees a basic tutorial

Conclusion

After doing this analysis of use cases, I now have a better understanding of the specific needs and goals of users, which helps to ensure that the web app is designed and developed to meet their expectations.

Important information

This is my second time making a tutorial, if there is anything unclear, feel free to ask in the comments and I'll try to reply to them.

This tutorial assumes you are using a self-hosted Supabase. If you are using a managed Supabase you should follow the instructions on the GitHub page instead.

If you don't already have a self-hosted Supabase, check out my other tutorial How to Self-host Supabase and come back to this tutorial afterwards.

Create a GitHub repository for the Svelte App

We will host our svelte app on GitHub pages. It is a static site hosting service that host files straight out of a repository on GitHub. It has the benefit of being free and has pretty good bandwidth since it is hosted by a big company.

We first need to create a GitHub repository. if you don't have GitHub Pro, you must create a public repository in order to share your site with GitHub pages, but with GitHub Pro, both private and public repositories support GitHub pages.

For the repository name, you can choose any name you want unless you want to page to be on your main GitHub page (Main GitHub page means the website on your {github_username}.github.io). If so, you need to name your repository as {github_username}.github.io.

If you name your repository other things, your website would have a link of {github_username}.github.io/{project_name} . In my case, my website would be on https://chit-uob.github.io/svelte_supabase_todo_example.

You can then click Create Repository.

Clone To-do list example

Then we clone the Svelte to-do list example from the Supabase GitHub. We choose a folder to put the project, and then we clone it with the sparse setting, this clones the repository without downloading every single file. Then we can go inside the repository and spare-checkout the to-do list example.

• Make sure you have Git installed, install from here if you don't already

• To open the command prompt on windows, you can right-click an empty space on the folder, and choose Open in Windows Terminal

git clone --depth 1 --filter=blob:none --sparse https://github.com/supabase/supabase
cd supabase
git sparse-checkout set examples/todo-list/sveltejs-todo-list

Now we can go in supabase -> examples -> todo-list , and copy the sveltejs-todo-list folder to another folder you use for the project. I am going to use the same example folder. Then we open the terminal inside the sveltejs-todo-list folder, and run the following commands.

git init
git add .
git commit -m "first commit"
git branch -M main
git remote add origin https://github.com/{github_username}/{repo_name}.git
git push -u origin main

• If you haven't already logged in to GitHub, it may prompt you to log in with your credentials

Configure front end

Then we install the GitHub pages npm package

• You need Node.js installed for this, install it from here if not already

npm install gh-pages --save-dev

Set up package.json

Inside the file package.json, we add a "homepage" property and set it as https://{github_username}.github.io/{project_name}, and a "deploy" inside the "script" property and set it as gh-pages -d dist, so lines 4-10 should look something like this:

  "version": "2.0.0",
  "type": "module",
  "homepage": "https://chit-uob.github.io/svelte_supabase_todo_example",
  "scripts": {
    "deploy": "gh-pages -d dist",
    "dev": "concurrently \"npm run dev:css\" \"vite\"",
    "dev:css": "tailwindcss -w -i ./src/tailwind.css -o src/assets/app.css",

Then we copy the .env.example file, and rename it to .env . Inside there we fill in the backend URL and anon key. If you followed my last tutorial, VITE_SUPABASE_URL would be your backend domain name. Otherwise, it would be whatever you set your API_EXTERNAL_URL inside the .env file inside the Supabase docker folder. As for VITE_SUPABASE_ANON_KEY , it is the value of ANON_KEY in the Supabase docker folder .env file.

...
JWT_SECRET=...
ANON_KEY={THIS IS THE ANON KEY}
SERVICE_ROLE_KEY=...
...
## General
...
DISABLE_SIGNUP=...
API_EXTERNAL_URL={THIS IS THE SUPABASE URL}
...

• You should then add .env to the .gitignore file, in order to not commit your secret keys.

If not a user site nor custom domain: set up vite.config.ts

If you are not using the user site (username.github.io), or you are not using a custom domain, then you would need to configure the vite.config.ts , and set base to your repository name, so lines 5-8 look something like this:

// https://vitejs.dev/config/
export default defineConfig({
  base: '/svelte_supabase_todo_example/',
  plugins: [svelte()],
...

This is because normally Vite looks for files in the main URL, which would be username.github.io/{whatever_file} , but since we want it to look for files inside username.github.io/{project_name}/{whatever_file} , we need to set the base to the project name.

Change front-end authentication

The original code uses GitHub and Google auth, but we want to use Facebook and Google auth instead, so inside the src/lib/Auth.svelte file, in line 114, we change it into on:click={() => handleOAuthLogin("github")} . In line 118, we change it into Facebook .

Optional: Mapping a custom domain to the GitHub page

if you have a custom domain, you have to create a file called CNAME inside the public folder, with the content being the custom domain. And change the homepage inside the package.json to be the custom domain.

Navigate to your DNS provider and create a CNAME record that points your subdomain to the default domain for your site. For example, if you want to use the subdomain subdomain.example.com for your user site, create a CNAME record that points subdomain.example.com to <user>.github.io. Also, add A Record with host @ and value 185.199.108.153, 185.199.109.153, 185.199.110.153, 185.199.111.153. So your record looks like this:

• Side note: when I was testing this, GitHub seems to dislike having dashes - or underscore _ inside the subdomain

• If you used a user site <user>.github.io , every other project site you use afterwards, will start with your custom domain, instead of <user>.github.io .

Deploy with GitHub pages

We can install the necessary packages, build the project, and then deploy it.

npm install
npm run build
npm run deploy

We then visit the repository on GitHub and go to Settings -> Pages, select Deploy from a branch, and select gh-pages as the branch. Then we can optionally enable HTTPS too.

If you used a custom domain, the page would like something like this:

Congratulations! When you go to the URL stated above, you will be able to visit the website.

However, only the front end is working for now, in order to have a full-fletch web application, we need the backend to work as well.

Make the database table

The initial Supabase database given to us only contains the auth tables, we need to create the profiles table and todo table on our own. In the hosted version, the snippets are provided, but since we are self-hosting, we will have to copy them from there.

I created a Supabase account, created a project, and then copied the snippet. For you, you can just copy the following code to the run panel and run it.

Go to the Supabase dashboard, inside the default project -> SQL editor, we can paste the following code there, and click run to run them.

Then for the todo list

Email sign up

To enable email sign-up, we need our backend to be able to send emails. Email servers are very difficult to host, so we will use an SMTP relay service, and have them send the emails for us. There are many choices, I chose to use Sendinblue because it has the biggest free tier email sending limit. It can send up to 300 emails per day, which means up to 9000 emails per month. You can use whatever service you want, you just need to get the required credentials.

Get the credentials from Sendinblue

Create an account in Sendinblue. In the dashboard, click on the Profile in the upper right corner -> SMTP and API.

Inside that page, we need the SMTP Server, Port, Login. We also need to generate an SMTP key, which would act as the password.

Fill in the credentials in the self-hosted Supabase

Going back to the Supabase backend, in the supabase/docker/ directory inside the .env file, we can paste the respective values inside the respective field.

## Email auth
ENABLE_EMAIL_SIGNUP=true
ENABLE_EMAIL_AUTOCONFIRM=false
SMTP_ADMIN_EMAIL={The email address you want the email to be from}
SMTP_HOST=smtp-relay.sendinblue.com
SMTP_PORT={The port given to you}
SMTP_USER={Your login}
SMTP_PASS={Your SMTP key}
SMTP_SENDER_NAME={The name of the sender of your choosing}

Social OAuth

We also want to enable Open Authentication login, because it skips the step of creating an account for the user, so users will more likely be inclined to sign up to your site.

Google

Supabase already has a guide on this, however, that is for the managed version, for the self-hosted version, there is something we need to do differently.

First, go to https://cloud.google.com/, Sign in to google if not already, and then click console. It will prompt you to accept the terms of service.

Inside there, click on Select a Project at the top left and click new project. Fill in your app information then click Create, this will take you to the dashboard for the new project.

In the search bar at the top labelled Search products and resources type OAuth. Click on OAuth consent screen from the list of results. On the OAuth consent screen page select External. Click Create.

On the Edit app registration page fill out your app information. The Application home page is your Svelte website, and The Authorised domain is the domain of your website. Click Save and continue at the bottom.

Click Credentials at the left to go to the Credentials page on the Google Cloud Platform console. Click Create Credentials near the top then select OAuth client ID . On the Create OAuth client ID page, select your application type. If you're not sure, choose Web application. Fill in your app name. At the bottom, under Authorized redirect URIs click Add URI.

Your URI should be what you set as the API_EXTERNAL_URL inside the supabase docker .env file + /auth/v1/callback. For example, https://mybackend.com/auth/v1/callback. Enter your callback URI under Authorized redirect URIs at the bottom. Enter your callback URI in the Valid OAuth Redirect URIs box. Click Save Changes at the bottom right. Click Create. Then we need to copy and save the values under Your Client ID and Your Client Secret.

Then we can go back to the supabase docker .env file, and add the following code

## Google auth
ENABLE_GOOGLE_SIGNUP=true
GOOGLE_CLIENT_ID={the_client_id_given_to_you}
GOOGLE_CLIENT_SECRET={the_client_secret_given_to_you}

And in the supabase docker docker-compose.yml file, under the auth environment section, (around line 87), we fill in the following:

GOTRUE_EXTERNAL_GOOGLE_ENABLED: ${ENABLE_GOOGLE_SIGNUP}
GOTRUE_EXTERNAL_GOOGLE_CLIENT_ID: ${GOOGLE_CLIENT_ID}
GOTRUE_EXTERNAL_GOOGLE_SECRET: ${GOOGLE_CLIENT_SECRET}
GOTRUE_EXTERNAL_GOOGLE_REDIRECT_URI: {the_callback_uri}

Around this part is a lot of GOTRUE_..., so it should be pretty easy to find. Keep in mind that the ${} things should be taken literally, you shouldn't replace the string inside ${}, but do replace {the callback uri} with the actual callback uri.

Facebook

The Facebook guide can also be found on the Supabase guide, I am going to briefly go through what you need to do, and the self-hosted instructions not included in their guide.

We first create and configure a Facebook Application on the Facebook Developers Site. We first login. Click on My Apps at the top right. Click Create App near the top right. Select your app type and click Continue (Do NOT choose "business", it complicates things). Fill in your app information, then click Create App. This should bring you to the screen: Add Products to Your App. (Alternatively you can click on Add Product in the left sidebar to get to this screen.)

From the Add Products to your App screen, we can click Setup under Facebook Login . Skip the Quickstart screen, instead, in the left sidebar, click Settings under Facebook Login . Enter your callback URI under Valid OAuth Redirect URIs on the Facebook Login Settings page. Enter this in the Valid OAuth Redirect URIs box. Click Save Changes at the bottom right.

Be aware that you have to set the right access levels on your Facebook App to enable 3rd party applications to read the email address. From the App Review -> Permissions and Features screen: Click the button Request Advanced Access on the right side of public_profile and email .

Click Settings / Basic in the left sidebar. Copy your App ID from the top of the Basic Settings page. Under App Secret click Show then copy your secret. Make sure all required fields are completed on this screen.

Then similarly, we go to the supabase docker .env file, and add the following code:

## Facebook auth
ENABLE_FACEBOOK_SIGNUP=true
FACEBOOK_CLIENT_ID={the_id}
FACEBOOK_CLIENT_SECRET={the_secret}

And in the supabase docker docker-compose.yml file, under the google stuff we just added, we add:

GOTRUE_EXTERNAL_FACEBOOK_ENABLED: ${ENABLE_FACEBOOK_SIGNUP}
GOTRUE_EXTERNAL_FACEBOOK_CLIENT_ID: ${FACEBOOK_CLIENT_ID}
GOTRUE_EXTERNAL_FACEBOOK_SECRET: ${FACEBOOK_CLIENT_SECRET}
GOTRUE_EXTERNAL_FACEBOOK_REDIRECT_URI: {the_callback_uri}

• Note that both Google and Facebook Auth are in testing mode, to use them in production, you need to change them to production mode.

Replace site URL

also inside the supabase docker .env file, you can change the SITE_URL=http://localhost:3000 , and set the URL to the URL of your front-end website.

Restart Supabase

Now that everything is set, you can restart Supabase. If you followed my last tutorial, Supabase would be running inside a screen, do screen -r supabase , then ctrl + c to stop the already running docker, then docker compose up to start the docker again. If it is not already running, you can just do docker compose up inside the supabase docker directory to start Supabase.

Conclusion

Now you should get a fully functional to-do list web app, you can try the different logins, from email sign-up and sending an email to confirm your account, to one-click OAuth with Google and Facebook. After logging in, you can also add and remove to-do items, and they will stay there when you log in with different devices.

This is my second touch on making a tutorial, if there are any steps unclear, feel free to leave a comment and I'll try to answer them. hope this tutorial helps.

This is my first time writing a complementary guide, so if there is anything unclear, or you have any feedback, please leave it in the comment below.

What you will achieve

If you follow the steps, you will end up having a password-protected Supabase server hosted on a VPS, a reverse proxy that redirects traffic from the domain name to the respective endpoint on the Supabase back end with an SSL certificate.

To continue, you will need:

1. A domain name, free or paid, that you can set the A or CNAME record

2. A VPS running Ubuntu 20+, if you don't already this guild will also introduce one

Buy a VPS and set up

To self-host a Supabase back end, obviously, you need a VPS. If you already have one, you can move on to the section about securing the VPS and see if there is more you can do to secure your VPS. If not, the following guide will talk about how to purchase one.

Purchasing a VPS

I will be using OxideHost VPS because they are relatively cheap. I've been using them and they are pretty reliable. For those interested, this is my affiliate link.

After choosing the VPS and purchasing it with Ubuntu 22.04 installed, an email is sent to give me information about how to connect to my VPS. The most important part is the hostname, IP, and default password.

Connect to it

ssh {username}@{ip-address}

To connect to the VPS I just purchased, the simplest way is to use the ssh command in the command line. However, using this method, we will have to type this command and enter the password every time. A better method would be to use SSH clients such as PuTTY which is free. I will be using Termius since it has a nice interface and is free for students.

I will create a new host, inputting the hostname given by the email, and username which is root by default, and the password was given to me. It will ask you a question about do you want to connect to it, click yes. Then I will have connected to the VPS for the first time.

Securing the VPS

Now that we are connected to our VPS, we want to make sure we are the only people having access to it, so we need to take measures to secure it.

update its software

apt update
apt dist-upgrade

The first task is to update it since updated software usually has its security vulnerabilities patches. This also upgrades the Linux distribution. There will be a few questions the command prompt asks you, you can type y and then press enters to say yes to them. and if it is a multiple choice, you can press tab to navigate around, and press enters to confirm the selection.

Add another user

Now that you acquired your VPS, hackers also want to get access to it. There are tons of robots on the internet cracking passwords of random VPS, they usually try to log in as root since it is the default configuration. So we will add a new user, pass the root permission to it, and disable root login so those bad robots won't be able to log in to our VPS.

adduser {username}
usermod -aG sudo {username}

The username cannot have spaces within, so you can use an underscore like learn_supabase or learnsupabase . Then it will ask us some questions, we can input blank if we don't want to answer them. But we need to fill in a secure password, I recommend inputting a randomly generated password using some secure password generators so that there is no way for anyone to guess it. Then we can give the user Sudo privileges. And try to log in using the new username and password by creating a new host and connecting it.

Disable the root login

nano /etc/ssh/sshd_config

Now we need to change the Port to some random number to make it harder to guess, the number must be smaller than 65535. And we also change the link PermitRootLogin to no, so we can no longer log in as root.

After making the change, we can do ctrl + o to write out the changes, and ctrl + x to exit nano. Then we do the following command to restart the sshd service, so the new configuration will come into effect.

systemctl restart sshd

Install Supabase

Install docker

We need docker for this, a tutorial about how to install docker can be found here. In this tutorial, I will summarize what commands we need to run. This command assumes you are using Ubuntu, if you are using other distros, please refer to the above tutorial link.

sudo apt install apt-transport-https ca-certificates curl gnupg lsb-release
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io
sudo apt install docker-compose-plugin

Then, we also need to enable the user to use docker.

sudo usermod -aG docker {username}

Clone Supabase

Then we want to clone the Supabase from Github.

git clone --depth 1 https://github.com/supabase/supabase

Set up Supabase secrets

cd supabase/docker/
cp .env.example .env

Now we get inside the docker directory and copy the file for the environment variables.

Generate passwords

We open the .env file, and we will see a lot of settings.

nano .env

We want to set the passwords for the above fields. For POSTGRES_PASSWORD , you can use a randomly generated password. For the JWT secret, we have to use the custom key generator. you can copy the JWT secret and paste it into the file, and then we can select ANON_KEY , hit generate, copy it to the ANON_KEY field, then the SERVICE_KEY field.

nano volumes/api/kong.yml

Then we also copy the same key to the kong config, pasting the new anon key and service key.

Start docker

Now we open a new screen and name it so that we can connect to it later

screen

Here you will see the introduction of the screen, hit enter . Then we can do ctrl + a then ctrl + d to disconnect from the screen. In the console, you will see your screen number [detached from 3775963.pts-0.hostname] , the 3775963 number is the screen number.

screen -S {screen_number} -X sessionname supabase
screen -r supabase

Now we have reconnected to screen supabase , when we disconnect from it, we use ctrl + a then ctrl + d .

cd supabase/docker/
docker compose up

Now the supabase docker is started, it will download a bunch of things, and you can connect to your domain in port 3000.

Domain mapping

Now I want to point a custom domain to the backend. I used Namecheap, and I create an A record, pointing a subdomain to the IP address of my VPS.

Reverse proxy

Reverse proxy points the connections to the correct port. The part is explained very well in the Linode blog post, so I am only going to briefly talk about what commands needed to be run, you can refer to their blog post for a more detailed explanation.

We first install Nginx for reverse proxy.

sudo apt install nginx
sudo systemctl status nginx

If we connect to our domain, we will see the Nginx welcome page.

sudo nano /etc/nginx/sites-available/{your_domain_name}

then we copy the following content (from the Linode guide) into the file, replacing the example.com with your custom domain.

map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

upstream supabase {
    server localhost:3000;
}

upstream kong {
    server localhost:8000;
}

server {
    listen 80;
    server_name example.com;

    # REST
    location ~ ^/rest/v1/(.*)$ {
        proxy_set_header Host $host;
        proxy_pass http://kong;
        proxy_redirect off;
    }

    # AUTH
    location ~ ^/auth/v1/(.*)$ {
        proxy_set_header Host $host;
        proxy_pass http://kong;
        proxy_redirect off;
    }

    # REALTIME
    location ~ ^/realtime/v1/(.*)$ {
        proxy_redirect off;
        proxy_pass http://kong;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $host;
    }

    # STUDIO
    location / {
        proxy_set_header Host $host;
        proxy_pass http://supabase;
        proxy_redirect off;
        proxy_set_header Upgrade $http_upgrade;
    }
}

After that, we can do ctrl + o and ctrl + x to write and exit. Then we can create a symbolic link of the config from sites-available to sites-enabled, link the default config, and then restart Nginx.

sudo ln -s /etc/nginx/sites-available/{your_domain_name} /etc/nginx/sites-enabled/{your_domain_name}
sudo unlink /etc/nginx/sites-enabled/default
sudo systemctl restart nginx

Now, we can connect to Supabase using the domain name.

Get SSL

In this part, we will use Certbot to give our Supabase an SSL certificate. Their official website have pretty good instructions, so you can follow that. But I'll also let you know what commands I used.

We first install core, then certbot. Then we configure it, and at last try how it auto-renews, and then restart Nginx for it to take effect.

sudo snap install core; sudo snap refresh core
sudo snap install --classic certbot
sudo ln -s /snap/bin/certbot /usr/bin/certbot
sudo certbot --nginx
sudo certbot renew --dry-run
sudo systemctl restart nginx

Secure Supabase

The Supabase dashboard everyone can connect to, to secure it, we do this to generate the password. It will ask you to type in the password two times, so it is best you generate a password from a secure password generator and then paste it there twice.

sudo htpasswd -c /etc/apache2/.htpasswd {admin_username}

Then we can check the password with

cat /etc/apache2/.htpasswd

Then we edit the Nginx config to password protect

sudo nano /etc/nginx/sites-available/{domain name}

and make it so that in the studio

    # STUDIO
    location / {
        proxy_set_header Host $host;
        proxy_pass http://supabase;
        proxy_redirect off;
        proxy_set_header Upgrade $http_upgrade;

        auth_basic           "Administrator’s Area";
        auth_basic_user_file /etc/apache2/.htpasswd;
    }

at last restart

sudo service nginx restart

Here you can fill in the username as the {admin_username} you put in, and the password as the password you filled in.

Reconfigure Supabase

Now, we want to set the domain for the Supabase backend and external API.

screen -r supabase

Then we press ctrl + c to end the Supabase docker instance. Then we edit the .env file.

nano .env

We need to change the

...
API_EXTERNAL_URL=https://{your_domain}
...
SUPABASE_PUBLIC_URL=https://{your_domain} # replace if you intend to use Studio outside of localhost

Then we can do ctrl + o and ctrl + x to save the changes, then do the following to get Supabase back up.

docker compose up

Then we can exit the screen by doing ctrl + a then ctrl + d to disconnect without terminating the process inside.

Conclusion

Congratulations, you now have a working Supabase backend, protected by a password, inside a certified domain. Hopefully, my explanation is clear, is there are any questions or comments, feel free to leave a comment. Your next step would be to develop your front end, which I will cover in the next part of this guide.

Routing with Nginx

Currently, there are lots of programs running on various ports of my VPS. Supabase studio on 3000, Svelte on 5173, PostgreSQL running on port 5432, and Kong running on 8000 and 8443 for HTTP and HTTPS respectively. How do I take the users that want to connect to my front end (Svelte) to port 5173, and secure my Supabase studio?

The answer is routing, we can set up a router, or a reverse proxy, to take users that are connecting from HTTP and HTTPS, which is port 80 and port 443, and route them to port 5173, my Svelte front end. To do that, I will use Nginx, because that is the most popular one. In the following, I'm going to summarize what I did to get it working, If you want a more detailed explanation or code examples, I recommend you check out this article from Linode, I find it very helpful.

In Nginx, we create a server to listen to a single port, then we tell it how to redirect based on location. Locations are the URL path after the domain name, the location for example.com is /, and example.com/terms/ is ^terms/(.*)$ . Then in the location, we can ask it to redirect to a certain port, such as proxy_pass http://localhost:5173 , this would redirect the users connecting from this port at this location to this address instead.

An important thing for Nginx is that we have to configure which rules are active. There is a directory called sites-available, and one called sites-enabled. What we have to do it to configure the settings in sites-available, then created a symbolic link to the sites-enabled directory. Then we have to restart the Nginx service to apply changes. An important thing to do is to unlink sites you aren't using, I once forgot to unlink the default rules and was wondering how the default rules were applying for so long.

Getting SSL

After that, we can redirect HTTP traffic, but we need an SSL certificate if we want HTTPS traffic. They provide credibility and encrypt traffic. There are 3 ways to get a certificate:

1. Purchase it from a trusted certificate authority (CA), the cost can vary from £5 a year, up to who knows how much

2. Let a proxy like Cloudflare do it for you, users will connect to Cloudflare with the certificate, and then Cloudflare redirects the traffic to your site, and the users see the certificate from Cloudflare. This is free for personal or hobby projects.

3. Generate your own using a service like Certbot, it generates the certificate for you and you don't have to pay a penny. But the downside is that you will have to configure it yourself.

I went with Certbot since it is free. The website contains very detailed instructions about installing it. After getting it, I need to configure Nginx to use the certificate, redirect HTTPS traffic to my front end, and ask HTTP traffic to upgrade to HTTPS traffic.

Domain name

Then I needed a domain name for my website, you can think of a domain name as basically a link for the website. I went over the three providers that offer free 1-year in Github student pack, namely Namecheap, Name.com and .TECH. I analyzed what service they provide and how much the domain is going to cost after the 1-year free trial. Namecheap has the cheapest domain name after the free trial, so I decided to use it.

Now I have to configure it to redirect traffic from the domain name to the IP address of my VPS. There are different types of records, the ones that you're most likely to use are an A record or a CNAME record. A records allow you to associate a domain name or a subdomain to an IP address, CNAME records allow you to point your domain name or subdomain to a hostname.

I pointed the subdomain I wanted to the IP address of my VPS, then pointed the www version of the domain to the domain without the www part.

Storing data

For the encryption function of the website, I want people to be able to generate an encryption key locally and have it stored locally. So the data they post to the server will be encrypted. This makes it so that even if the website got hacked, the hackers couldn't read the encrypted content.

Users don't want to enter the key every time they visit the website, I need a way to store the encryption key locally. I was thinking to use cookies for this task, but then I found out that cookies get sent back to the web server for every request, which isn't secure at all. So I searched that there is Local Storage in browsers now, and the information there will not be sent to the web server, so I decided to use it.

It is very easy to use, to write something to local storage, you do localStorage.setItem("name", "Chris");, and to read something, you do let myName = localStorage.getItem("name");.

Svelte can be hosted statically

I don't know how I didn't know this, I always thought Svelte has to be hosted on a VPS, maybe because I developed Django apps before and they require a dedicated host. I found this out because I was talking to a friend about how I am buying a VPS to host the website, and he asked me why didn't I use Github pages instead. I said the website needed to be hosted, but he said it doesn't. I looked into it, and turns out he was right! Svelte doesn't have to be hosted dynamically. When we run npm run dev, it runs the developer version, which is dynamically hosted because it reacts to changes. But if we do npm run build, the page is built and the dist directory. Then we can deploy through any static site hosts, including Github pages.

GitHub pages can be used for public repositories for free accounts and private repositories for Pro accounts, which I have from the GitHub student pack. Moreover, GitHub is a big website, so I'm sure loading a front end there is very fast, as they would have good content delivery. Security isn't a big concern either, since only the front end is hosted there, and the important data is secured in the back end. Therefore I will be using GitHub pages to host my site on second thoughts.

Hosting on GitHub Pages

To host on Github pages, if we want to use the user GitHub page (username.github.io), instead of username.github.io/reponame, we need to have the website in the repository with the name username.github.io . Then we can control what to deploy, we can either set up a GitHub workflow, which is GitHub doing something for you automatically, or just deploy from a branch.

To automate this process, there is an npm package called gh-pages. It enables you to deploy your new version of the build, which is inside the dist folder for Svelte projects, to a new branch. Then if you configure your GitHub page to deploy that branch, this would automatically be deployed.

There are some important things to set up. We have to configure the homepage to the GitHub page website in package.json, as well as set up a new run config called deploy, which runs gh-pages -d dist . If we are using a custom domain, we also need to create a CNAME file inside the public/ directory with the content being the custom domain name. Then when we run npm run build then npm run deploy, our GitHub page will be deployed.

Conclusion

In this blog post, I discussed my experience with self-hosting a Supabase Svelte project. I covered setting up routing with Nginx, obtaining an SSL certificate with Certbot, securing a domain name with Namecheap, and hosting the project on GitHub Pages. Overall, I found that self-hosting my project was a valuable learning experience that allowed me to gain a deeper understanding of web hosting and the tools involved.

Self hosting Supabase

The advantage of Supabase is that it supports self-host, so I don't have to worry about limits and bills. I went ahead and searched for the official tutorial. I expected a very complete tutorial that covers every step, but what I got was a very high-level overview of how to do it, talking about architecture, configuration, database etc. Then there is another tutorial on self-hosting with Docker. To their credit, there is a very concise script for getting the docker running, but that's all you get in terms of a hands-on tutorial. The rest of the article is yet another high-level overview of important things to do. This is fine for experienced developers, but as a new developer, I would have appreciated better guidance.

At that time, I didn't know what is docker-compose, how I deploy the docker from the local environment to the production environment (my VPS), and how to set up the secrets, so I kept on searching for more helpful tutorials.

Luckily, I stumbled upon this Youtube tutorial, huge shout out to Kelvin Pompey. He demonstrated the entire process of self-hosting Supabase with an Ubuntu Server on Digital Ocean, which I was able to follow step by step, and reproduce the working Supabase backend. I didn't use Digital Ocean, I used a VPS from Oxide Host since I've been using them for a few years starting from hosting a Discord bot, and the service is reliable.

I learned how to set up supabase, how to install docker-compose and that docker-compose starts the processes inside the docker-compose.yml file, and to update the secrets inside the .env file before deploying the Supabase.

Setting up Svelte front end

Now that I've set up the backend, I want some example frontend code to test if the backend is working. I was originally thinking about making a React app as it is pretty popular. But I found an officially maintained to-do list app written in Svelte, since it is very similar to a prayer tracker, I've decided to use that as the skeleton code and build on that.

The code is within the Supabase repo, and the Supabase repo is huge, so I don't want to clone the entire thing, just the part I need.

Only cloning a part of a repo

I've learned this from this StackOverflow post. We need the latest version of Git, I failed at first because I was using an older version of git that doesn't support sparse checkout.

I am going to use the Supabase repo as an example, the --depth 1 flag stops git from cloning any files inside folders. and the --filter=blob:none will filter out all blobs (file contents) until needed by Git, and --sparse just tells git you plan to git checkout later.

Then we cd into the directory, and sparse checkout, with set as the path of the content we want to check out.

git clone --depth 1 --filter=blob:none --sparse https://github.com/supabase/supabase
cd supabase
git sparse-checkout set examples/todo-list/sveltejs-todo-list

Then I just needed to run npm install to install the dependencies and npm run dev to run the Svelte project. I also installed nvm to manage the version of Node js.

I also needed to update the .env file to paste the Supabase host address, and the anon_key used in Supabase. Side note: I used the default credentials the first time I used Supabase, when I changed the credentials, the database no longer recognised me, I had to do docker-compose down --volumes to reset everything after I updated the .env credentials.

Enabling third-party authentication

Supabase supports third-party authentication, and it does that by recognising the email address. So let's say you used the same email address example@example.com on Google and Facebook, signing in with either will lead you to log in to the same account.

These third parties use an authentication standard Open Authentication (OAuth). OAuth is a protocol that allows applications to access user information without requiring their password. This allows users to securely share their information with third-party applications, minimizing the risk of a security breach.

Basically, instead of identifying a user using a password, the app asks Google, "do you trust this guy?". If Google trusts them, the app trusts them too.

To enable this, we have to create an account on the OAuth provider. I was able to do it for Google and Facebook. For Apple, the OAuth requires me to create an Apple App, and that is locked behind a paywall. I need to pay 79 pounds a year to access that, so I won't be using that.

Problem with Twitter OAuth

When I try to do it on Twitter, the funniest/most annoying issue happened. On the sign-up page, there is a button that asks do I want to receive marketing emails from them, I said no. Then they said they have sent a confirmation email to my email account, I didn't receive it, so I clicked have it resent. Guess what they said: "There was a problem resending the confirmation email, User has disabled email notification from Twitter...". I checked my Twitter setting and email notifications are on.

I then registered from the Twitter developer account on another Twitter account, this time ticking the marketing email field, this time I was able to get the confirmation screen and do it. So I think this was the issue. The funniest thing is that I was unable to fix this issue, I cannot find the setting to re-enable marketing email on the Twitter developer page.

Email sending

For email sending, originally I thought I can send emails from my VPS. So I searched for How to send emails through VPS. Every result uses Gmail or another managed email service, apparently, we can't simply use a VPS to send an email, I needed something called an SMTP server.

So I went on and search for How to host an SMTP server on a VPS. I found an article that says hosting a mail server is like building a jet, the article isn't trustworthy since it is from an email-sending company and they have a monetary incentive to say this. I wanted to find some open-source solutions that I can install in 1 step and be done, but there aren't any. Therefore I concluded the article was right and I gave up making my own SMTP server.

I had to find an email-sending service. I looked at different ones, SendInBlue has the largest free-email sending quota, 300 emails per day, which means 9000 emails per month. Of course, this is assuming 300 people sign up per day, which is highly unlikely, the number usually fluctuates. Still, this is the largest quota, so I went with them.

Conclusion

In this blog post, I talked about the process of creating a Supabase Svelte project. Including challenges I faced when self-hosting Supabase, setting up Svelte, enabling third-party authentication and email sending. I also wanted to talk about routing, SSL, domain name and storing information inside the web browser, but I realized this article already got very long. I've decided I'll write about them in part 2 of this blog post. So stay tuned.

As Christians, we should pray. it is a way for us to communicate with God and develop a deeper relationship with Him. In the Parable of the Unjust Judge, Jesus taught us not to lose heart when praying. But sometimes, we forget to pray, forget what to pray about, and forget how God answered our prayers. Imagine if our friend forgot to talk to us, or even worse, forgot what we did for them.

That's why we should keep track of what we pray about. It can help us see how God is answering our prayers, as well as help us stay accountable and faithful in our prayer life.

Using a paper journal or a note-taking app to do so is fine, but searching through paper journals is hard, and note-taking apps may not be as convenient as they are not tailor-made for this purpose.

Why I want to make it

Firstly, I see the need for it. While there are some great options out there, such as PrayerMate or the YouVersion Bible app, they are limited to smartphones only, you cannot access them with a computer. Moreover, they are very complex with many functionalities. It would be nice to have a simpler one. Because

“Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.”

― Antoine de Saint-Exupéry, Airman's Odyssey

Secondly, I am interested in trying out different technologies, and this sounds like a cool challenge. I can see this actually being helpful for others. I'm sure I'd learn a lot from doing this.

That's why I've decided to develop it and make it available to the public.

Features of the prayer tracker website

I want to build a website where users can:

1. Store prayer items, including the ability to add, edit, and delete items.

2. Give users a daily prayer list based on the frequency set for each prayer

3. Record how has God answered their prayer, or when the prayer was not granted

4. Encrypt the prayer items optionally

Since it is a website, we can assume the prayer items will be synced, and there will be account control. I choose a website over an app because a website can be accessed by any device connected to the internet, whereas apps must be downloaded and installed on a specific device.

How do I plan to build it

I've been following Fireship on Youtube, so I've got some technologies on my radar that I want to try out. Supabase is one of them, it claims to be an open-source Firebase alternative Backend as a Service (BaaS). I'm happy to see that they have a self-host option, which means I wouldn't be charged a large sum overnight, as I am hosting it myself, I can control how many resources to give it.

For the front end, I'm planning to use Svelte, simply because it is the javascript framework used in the Supabase to-do list example. I'm assuming it is a good framework for the official example to be built on that. I've also looked up the Fireship video on Svelte, and it looks promising.

For hosting the back end and front end, I'm going to use a VPS provider that limits my CPU and connection speed, but doesn't incur any additional cost nor have a data transmission limit, in my case, Oxide Host. It is much less stressful knowing that I won't be charged extra even if something went wrong.

What are the other building blocks

I've decided to use Supabase, Svelte and a VPS as the foundation of my project. Now I have to choose the various building blocks that I'm going to use to build my website.

Before that, I want to talk about the two approaches to building something. The first approach is to start doing first and plan as you go, this leads to quicker development if nothing goes wrong, but it is also possible that you'll encounter a problem that sets you back a lot. The second approach is to plan everything before starting, this reduces the risk of finding problems mid-project and having to redo everything, but this slows down the progress, and there is only so much we can plan for, we may still find unexpected problems. I will use the latter approach and test each individual building block before using them to build the website.

At the time of writing this, I've already tested out all the technologies. The process is very difficult, I've been stuck at problems after problems. This process took me several days, and over 24 hours of figuring things out, so I'm going to spare the details and just explain what I'll be using. Later on, I'll write another blog post to talk about how I tested them.

Conclusion

This is my plan to build this project. I will make a blog talking about my experience with testing the above building blocks, as well as a complete tutorial on how to set up a Supabase Svelte project.

This program was inspired by helping a friend. The friend has to click on links inside documents one by one to check if they still work. Hearing that, I thought to myself, that sounds like a task that can be automated. Therefore I asked for some sample word documents and started testing the concepts.

In face of this big task, I decided to break the task into a few steps, figuring out a way to do each, and then piece them together.

Step 1: Finding links from texts

If I want to extract some patterns from text, the first thing that pops into my mind is Regular Expression. They are a way to find patterns in texts and are often deemed difficult and feared by developers. Therefore I did what any sane developer would do: search for this problem online and copied the Regex from stack overflow.

The regex is (https?:\/\/\S+), which is pretty easy to understand

Problem 1: the succeeding full-stop is also matched

Let's say I have a sentence that ends in a link, such as http://chit.hashnode.com. If I parse this text in the above regex, the last full-stop (.) will also be included, because it is a non-whitespace character.

the solution is to use another regex, http[s]?:\/\/[^\s]+[^. ], here we have a [^. ], which means: Match a single character not period (.) and not whitespace ( ), so it solves the problem of it catching the trailing period.

Problem 2: having a newline character at the end

The previous regex solves the problem of having a period at the end of the line. But what if there is a newline character after it?

Knowledge dump: What is a newline character

The newline character is the invisible character that tells the text editor to go to the next line. For example, let's say \n is the newline character, I am a line.\nI am another line will become

I am a line.
I am another line

Let's use the website regex101 to test it, when we have A sentence ending in https://google.com., the https://google.com will be correctly captured.

But with a newline character after it, it will be caught as well, since the newline character was not excluded in [^. ].

As I am writing this blog post, I figured the solution was to exclude the newline character as well, resulting in the regex http[s]?:\/\/[^\s]+[^. ], but I was not this clear-minded when I was programming, the solution I came up with is to replace all the newline characters with the space bar. Then do the regex matching, which also worked, because there are no longer newline characters.

Problem 3: Different types of links

The third problem is with links with no http, our regex only matches links starting with http, but some links start with www, or maybe even no www.

I am sure more clever regexs can accommodate this, but I didn't want to spend too much time at this stage, so I just used a python library to do the work for me. That's the beauty/problem with Python, there are so many libraries that you can just use one, and not care how it is implemented.

from urlextract import URLExtract
urlextracter = URLExtract()
urls = urlextracter.gen_urls(s)

Step 2 Check the links

Now that we have all the URLs from a text extracted, we want to check them.

Problem 4: links without a protocol specified

But remember how some links don't start with HTTP? I need to add an HTTP in front of them, or else the python library request will have a hard time knowing what protocol is required, so I used the following code to achieve that

import re
def formaturl(url):
    if not re.match('(?:http|ftp|https)://', url):
        return 'http://{}'.format(url)
    return url

urls = [formaturl(url) for url in urls]

here I try to match the regex (?:http|ftp|https)://, it tests if the URL starts with http/ftp/https, if not, we append http:// in front of it

then we use list comprehension to do it on the entire url list.

Actually checking the link

Now we get to the step of actually checking the link, we do that using the requests library. We wrap the requests.get() inside a try-catch block so that if other issues happen, the program will not crash, it will simply return false. and if the status_code of the response isn't 200, then we return false too, else we return true.

200 means all good, so if the website returns standard content, it will return all good.

def check_link(url):
    print(f"checking {url}")

    # Try and see if url have inherit problem
    try:
        response = requests.get(url)
    except:
        return False

    # See if not 200
    if not response.status_code == 200:
        return False

    return True

Step 3: Reading word document text

Now I have a function that extracts URLs, and another to check if the url's website is working. I have to extra the text from a word document.

To do that, I can simply use the save-as function inside word and call it a day. That would save the document in a text format and we can read the text file with the program and get the result.

But that would mean the user has to do more, so I was thinking, is it possible to read the word document directly with Python?

The answer is yes because actually, word documents are just zip files. To know if I'm telling you the truth, install 7zip and unzip a word document, and you will see the following result. A word document is just a zip file containing a lot of XML files.

A way to solve this problem is to read the word document as a zip file and find the links inside the XML files. But since I'm using Python, I decided to find if there are any libraries that can do this for me. I found docx, docx2txt, docx2python. After testing them all, the one I decided to use is docx2python, because it extracts the main text, headers, footers, and even footnotes. The syntax is as below:

from docx2python import docx2python

# extract docx content
def get_text_by_docx2python(path):
    text = docx2python(path).text
    return text

Step 4 Piecing them all together

Now that all parts of the puzzle are here, it is time to implement the main logic, we first extract the text by use_docx2python.get_text_by_docx2python, then urlextracter.gen_urls to get the URLs, after adding http to them, we use check_links.check_link to check the link, and print a helpful message.

Problem: some links show a different result when we request it from the program than when we actually click the site

This is because some sites do not like robots, but since Python requests have the default user agent as python-requests/2.25, the website sees this and forbids the program from getting the result. What we need to do to fix it is to use another user agent, I copied the user agent on my web browser, Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36, then I used response = requests.get(url, headers=HEADERS, stream=True), now the website responds correctly.

Step 5 Doing it asynchronously

Now waiting for the program to do all the things is fine, but it is also too slow. The slowest part is waiting for the websites to be fetched. Since we have to wait for each website to give us the result. This would be faster if we use concurrency, which means doing things at the same time.

Think of it as, burger shop A gives you a burger in 5 minutes, and Boba shop B gives you a boba in 3 minutes. You can get a burger and then a boba in 8 minutes. If you order both at the same time and then collect them when they are ready, you'll only need 5 minutes.

Concurrency code is a bit more complex so I'm not going to explain it here, you can visit RealPython on this topic, they have a great tutorial on this topic.

Step 6 Warp up

At last, I added a code to show a file dialogue to further simplify this for users.

import tkinter as tk
from tkinter import filedialog
file_path = filedialog.askopenfilename()

Conclusion

That's how I did it and I hope you learned something from it.

Mean battery level of scooters

Graph for the entire duration

This is the mean battery level of all scooters around the city during the entire duration of the project. The graphs may look small, you can right-click the image and click open image in new tab to enlarge them.

In general, the mean battery level fluctuated between 55% and 60%.

We can observe that before 7 July, there were flat lines every day. This was because back then scooters went offline at 10 pm, and online at 6 am. During that time my program could not collect data and it assumed the battery level didn't change. After 7 July, you can see that the transitions in the graph were smooth again with no jumps.

A scooter "goes offline" when the API no longer returns information about the scooter. The API that I used is the API that tells users where the idle scooters are. So when someone is using the scooter, when the battery of the scooter is being charged, or when the scooter is sent back to the warehouse, the scooter is considered "offline".

We can also see errors in the data collection. There were two sloped straight lines during June, as back then my program would crash if the API server didn’t return information. This was fixed afterwards by adding a try-catch and retrying after failures. The second error was on 1 July, the graph broke into half. This is because I calculated the mean battery level each month, and the program guesses the battery levels of offline scooters by forward filling, ie. if the scooter went offline with 70% battery, the program assumes it would stay at 70% while being offline. When the new month started, the scooters with unknown battery levels were not included in the calculation, so there was a jump in the value. The reason I used forward filling despite this disadvantage is given in the 7th blog in this series.

We can also see a sharp decline in battery levels around 7 July. This might be because scooters no longer went offline at night and the scooter service area was expanded, so the battery drained even during the night, and with a wider area, it was more difficult to reach them and replace the battery of them.

Graph for weeks

These graphs were weeks starting from 10 July, they all started on Sundays. There was a general trend that the battery level was higher in the second half of the week.

Graph for days

These graphs were some days, they all started at around 0 am. There was a general trend that the battery level would increase in the first half of the day, usually, at 10 am, then start decreasing. This was strange, an increase in battery level should mean someone was swapping out the batteries, or new scooters with a full battery was being introduced. But it was hard to imagine people swapping batteries in the middle of the night.

Available scooter count

Graph for the entire duration

Same as the mean battery level, the days before 7 July had the count drop to zero at night because they were offline. And the number of available scooters also sharply declined from around 1150 scooters to under 500 scooters on 11 July. Presumably, it was because of the change in operation too.

We can also observe that more scooters were put into operation from 26 July, I believed it was due to the Commonwealth Games 2022 on 28 Jul 2022 – 8 Aug 2022. When the Commonwealth Games started, the number of available scooters, along with the mean scooter battery level dropped. It should be because of the increased ridership. This can be confirmed on the news where it said Voi sees record e-scooter ridership during 2022 Commonwealth Games at over 66,000 journeys.

Then the count returned to normal until 16 Aug but the count increased again on 19 Aug when the number of scooters exceeded 1750 scooters.

Graph for weeks

These are graphs of weeks starting from Sunday. There seemed to be a slight trend where the number of scooters in the second half of the week was higher.

Graph for days

These are graphs of days starting at 0 am, usually, the number of scooters increases to about the middle of the day, then decreases.

Ridership

Total graph

This graph corresponded positively to the available scooter count graph, when more scooters became available, more rides happened too. In July, there are fewer rides, but in July and August, the number of rides increased.

At the beginning of the graph, there were two places where the count hit close to zero, that was because of the error of my data collection program. My program crashed during that two times so I wasn't able to recognise rides during that two periods.

Commonwealth games

With this data, we can answer the question: did ridership increase during the Commonwealth Games 2022? The answer is it did. From 28 Jul 2022 to 8 Aug 2022, the ridership peaked at around 6400 rides per day, but the increase didn't stop there, it seemed even after the Commonwealth Games and the increase in scooter count and coverage, the increased ridership was maintained.

The sum of the rides during the Commonwealth Games calculated in my program is 65951, which was super close to 66000 said in this article. There was a discrepancy in the number due to the logic of my program. A ride was counted in my program when a scooter became offline and then online again within 45 minutes. But it did this at a one-minute interval. So if a rider parked a scooter and another rider rented it within a minute, my program would count it as one ride when there were supposed to be two. Another problem was that when a scooter temporarily went offline for a battery swap, my program still considered it a ride since the scooter went offline and then online.

I limited the maximum number of minutes in a ride because scooters might be taken to the warehouse, fixed, and re-released, so I could not count all instances of scooters being offline and then online again as a ride, so I chose 45 minutes as it was the maximum riding duration for a Voi pass.

Per week

Now I want to answer the question, does the ridership correspond with weekdays?

These weeks were the same as the ones with available scooter count. They generally corresponded with each other, days with more available scooters generally result in more ridership. There was no clear pattern in which weekdays had greater ridership.

Per day

The same days were chosen as the ones in the available scooter count graphs. These graphs were generally the opposite of those graphs, the times with the least available scooter were the times with most rides, which makes sense because people were using scooters so there are fewer of them available.

We can also see on most days the ride count was minimum in the morning around 5 am, then increased to peak at 8 am, then fell back again, then increased again from 12 pm to around 8 pm, then slowly decreased. I think the first peak was when people get to work, and the second peak was for people leaving work.

Heatmap

day

I figured it made little sense to measure this over weeks or the entire period, so I decided to only measure this over a day.

The scooters generally became more concentrated on certain spots on the map from 0 to 12 pm, then became more spread out after that.

Battery consumption

During ride

The way I calculated this was the same as ridership, so this analysis suffered from the same problems. However, I think these results would still be very close to the true value, as the calculation only fails in a few edge cases. The average level of battery drop was 0.461% per minute when unlocked. The battery consumption varied in each ride since scooters could be idle while unlocked, or be used in battery-consuming activities such as going up slopes.

I also filtered the battery consumption based on which month the data was collected, and found a surprising result. The battery consumption was decreasing. This is strange since I was taking on average all rides, so there shouldn't be much difference. Some possible explanation for this phenomenon was a change in rider behaviour, a change in battery capacity, or Voi figuring out a way to use less battery on its scooters.

I also filtered the results based on the starting battery level, and there are some interesting observations too. The results for the first half make sense, it is the same for our phones, the battery generally drains slower at a higher battery level. However, the battery drain decreased when it is below 40%, which is strange, usually, the battery drains the quickest when it is closest to zero.

When idle

I also wanted to know what is the battery draining speed when the scooter is sitting there idle. The average battery drained, while the scooter was idle, was 0.006830% per minute.

Starting at different battery levels, the draining speed was different too, the less battery there was, the faster the battery drains, this was what I had expected.

In my experience, lithium batteries, used in smartphones, usually drain faster when the battery level is lower. I wasn't able to pin down why, but I saw some explanations on the internet what I would like to share, one has to do with the lithium discharge voltage curve, in a lower stage of charge the voltage decreases and therefore the current has to be increased to maintain the power output, therefore the battery drains quicker. But I wasn't able to find solid proof of this, so if you know more, please let me know in the comments down below.

This brings us to another question, why during rides, the battery drained on 0-20% wasn’t the quickest, but during idle, the battery drained on 0-20% was the quickest? It could be an error in my data processing. It could also be that Voi's display battery level was not the actual battery level, it displayed a lower level when the battery was near zero so that it could travel further during the supposed near zero battery, to prevent the scooter from running out of battery mid-ride, but this is just a theory.

Conclusion

This project started out with the goal of locating a scooter in the city, which I am now able to do with the data I collected. But I found greater interest in studying the broader trend of scooters around my city. So below are my findings:

I can conclude that the scooter business is doing well. The ridership increased during the Commonwealth Games 2022 and the usage didn't drop back down after the games. The batteries are being replaced in time so their mean battery level and available scooter count are stable. The scooter's battery drain when idle is about 65 times slower than when on a ride, so leaving scooters on the street isn't costing them a lot.

Other than what I learned from the data, I also learned a lot about data analysis in the project, like how to collect and organize data, how to analyse them, and some handy Python modules to make it easier.

Downloading the data

I have been using a VPS (virtual private server) to collect the data regularly and automatically. Therefore I have to first download the data from it using a tool called Termius.

I selected all the data items from the start of July and copied them to my local machine. There are about 2GB of data.

Recap of what data I collect

I collect the data of all Voi scooters around my city every minute by using the API provided by the Voi mobile app. The data includes each available scooter's id, battery level and location. I modify it into a JSON list. Then I save the result to a JSON file every hour. Therefore the data I have looks something like this:

Importing the data and plotting the graph

All data files are in the same folder, so I generate a list of all file paths inside the folder and sort them in ascending order. Then I sanitise the data by trying to parse it using Python's try-catch. If the data file is correct, Python would not catch any error, so if Python caught an error, I know I have to fix the data file. Once that's done, I read all the files in ascending order and stored the time stamp and vehicle count in two separate numpy arrays. I then plot it using matplotlib. The graph looks something like this:

We can observe:

1. Before 7/7, the scooter count always goes to zero at the end of the day. Because scooters are disabled at night, after that, Voi decided that scooters would be available 24/7, therefore the scooter counts never dropped to zero again.
2. Near 12/7, the scooter count dropped to very low, that is just after the change, so maybe Voi did some changes in its workforce so that fewer workers are replacing batteries on scooters, or they are doing some checking on scooters so a lot is brought back for checking. Either way, the scooter count increased and returned to normal.
3. After 30/7, the number of available scooters seems to plummet, I suspect that is because of the Commonwealth games.

Finding the set of all available scooters

Then I run through all files, running the set union function from Python, to find the set of all scooters available throughout July. I do that by the list of all available scooters at each time stamp to a set, and take the union of the set of all scooters registered. So in the end, the union of all sets is the total amount of available scooters.

After running that code, I found out there were 2544 scooters of unique id that were available at a point in July.

Get all dates list

What I want is a gigantic pandas table with scooter ids as columns, and time stamps as rows. I already know the ids, so I have to find the time stamps, I do so by iterating through all the data items and adding the time stamps to a list.

Fill all the tables

I run the code to fill all the tables, it does this by iterating through all the data files, in each individual data item, and adding them to the corresponding pandas table. This code is extremely slow because we have 44499 data of individual time stamps. And in each data, there are 1000 scooters, and for each scooter, we are storing their latitude, longitude and battery level, so we are adding 44499 1000 3 items to tables, which is over one hundred million. The code wasn't able to finish in 2 hours, therefore I had to stop it, and continue the code from where I left off the other day.

Plotting individual scooters

Some scooters are more active, like

Some are decommissioned for some time, maybe for fixing, or just recycling scooters.

But keep in mind that we only know the id of the scooter, the same scooter could have its id switched, we don't know what the id means.

Plotting the mean battery count

now I wish to plot the mean scooter battery count. I do so by first dropping columns that were all null values. then I use the method of forwarding filling to fill in null values. Then at last I plot the graph using the plot function and the mean function with axis = 1. there are null values in my code reasons: when a scooter is rented, it is no longer available, so the value becomes null. when the scooter is called back for fixing or other purposes, it is not available. Before scooters are available all day, at night, all scooters are not available.

Plotting location heatmap

Then I generate the heatmaps using seaborn and matplotlib, I generate one for each hour, then piece them together as a gif file. The negative of having it as a gif file is that a gif file generally has a bigger file size as it compresses images individually, while mp4 is capable of storing the change between images, hence if the images in the video are similar, a lot of storage space can be saved. I convert the gif to mp4 using the answer suggested by this StackOverflow answer by using moviepy. and the result is as below.

Summary and what's next

There are a lot of interesting observations that can be made using these data. But for the length of this blog, I am just going to show the data collected and fewer observations. Again if you are interested in how the data was collected, you can check out Where's my Voi scooter series.

I think there are many conclusions to be drawn, such as where will scooters be concentrated at what time, and around what time most scooters are available.

Solution

I will put the solution on top. If you are interested in how I found it, you can read about it below.

Simple solution for if what you do is lightweight

Lightweight means that it doesn't use much CPU power and time, like a simple print function.

Assuming you want to run the code every 10 seconds.

import time
from datetime import datetime

def do_things():
    print(datetime.now())

while True:
    do_things()
    time.sleep(10)

you should get something like this

2022-08-24 10:59:35.247516
2022-08-24 10:59:45.259224
2022-08-24 10:59:55.272715
2022-08-24 11:00:05.284097
2022-08-24 11:00:15.290823

There you have it, you can replace the sleep time if you want a different interval, and you can replace the content of the do_things() function with what you want the program to do.

Problem with the simple solution

But if you are like me, what you wish to do every interval takes time, whether it is waiting for I/O from another program, or it is doing something very CPU intensive. Then if we still do what we did, we will get inaccuracies in wait time like the below.

import time
from datetime import datetime

def do_things():
    number = 50_000_000
    sum(i*i for i in range(number))
    print(datetime.now())

while True:
    do_things()
    time.sleep(10)

Here I used sum(i*i for i in range(number)) on a huge number to force the CPU to do a lot of work, don't mind the underscore in 50_000_000, it is the same as 50000000. The result of this program is:

2022-08-24 11:07:54.377208
2022-08-24 11:08:07.599538
2022-08-24 11:08:20.838108
2022-08-24 11:08:34.043164
2022-08-24 11:08:47.255688

As you can see, we want the program to execute every 10 seconds, but it took the program about 14 seconds each loop, this is because some extra time is spent computing. This is why you need the advanced solution.

Advanced solution

import time
from datetime import datetime, timedelta

def do_things():
    number = 50_000_000
    sum(i*i for i in range(number))
    print(datetime.now())

while True:
    previous_datetime = datetime.now()
    do_things()
    time_took_running_do_things = datetime.now() - previous_datetime
    remaining_time_in_secs = (timedelta(seconds=10) - time_took_running_do_things).total_seconds()
    time.sleep(remaining_time_in_secs)

and the result is:

2022-08-24 11:14:40.821099
2022-08-24 11:14:50.797054
2022-08-24 11:15:00.806950
2022-08-24 11:15:10.834795
2022-08-24 11:15:20.845102

Explanation

Here you can see that the codes run every 10 seconds even though we are doing the heavy lifting. This is because we compensated for the time taken to run do_things() by waiting for less afterwards. If it took the program 3 seconds to run do_things(), we only wait 7 seconds afterwards, so the program still do things every 10 seconds.

Limitation

This solution is limited by, the action you want to do must to take longer than the interval you want to loop. If you want to do an operation that takes 10 seconds every 1 second, this solution cannot help you. You might want to look into Concurrency from this RealPython article.

How I got there

Now I will talk about how I figured this out.

I am working on a program that sees if I'm playing computer games at regular intervals to remind myself how long I'm playing, but the problem is that it is very inaccurate. I would play a game for 45 minutes, then the program will tell me that I only played for 30 minutes.

I first thought this is a problem with time.sleep(), maybe because the CPU is doing a lot of work while gaming, the wait is wrong.

But this is proven wrong by when I'm not playing games, the program still runs at incorrect intervals.

Then I commented out the things the program does within each interval, only leaving the print time statement, and then the problem was fixed. It turns out the problem is what I do inside the interval is costing above 2 seconds!

I then immediately went on to try to over-engineer the problem. I thought to myself, maybe I want the loop and the do_things() function in separate threads via threading or asyncio, or maybe even separate CPU using multiprocessing. I watched the entire tutorial on Speed Up Your Python Program With Concurrency by Jim Anderson on RealPython until I drew this graph.

I then realize, what if I just do them all linearly, then compute how much time I need to wait? It is then that it dawned on me, that the problem was never this complicated. So I fixed the problem and decided to write this article to help others like me. So there you have it.

Heatmap

Overlapping image

To overlap the image of the map and the heatmap, I need to find out how to have a half-transparent image on top of another image.

I first have to make an image transparent, I do that by using the putalpha(128) to make an image half transparent. It works by putting an alpha, or transparency level of 128 on that image. An alpha level of 0 means completely transparent, and level 255 means completely opaque. So 128 is in the middle, hence half-transparent.

Then I tried using image.paste(another_image) to put the half-transparent image on top of the other. but what happened is the image complete overwrites the other. what I needed was alpha_composite. For testing, I alpha composited the half-transparent map on top of the sat.

I created this map_img by getting the four coordinates I used for the range of the heatmap. and Taking a screenshot with exactly the four corners.

sat_map

from PIL import Image
map_img = Image.open('img/map.png')
map_img.putalpha(128)
sat_map = Image.open('img/sat_map.png')
sat_map.alpha_composite(map_img)
sat_map.save('img/result.png')

result.png

Masking

Now I want to mask the heatmap image on the map image so that we can see the areas with a lot of scooters on the map. I wanted to use alpha composite like the last example, but I figured that using masking would be better. Image masking is putting image A on image B using image C as a guideline. I plan to put a dark image over the map, using the heatmap as a guideline. Places with more scooters will be darker so that we can tell where the scooters are on the map.

The function to do the masking is Image.composite(), it requires all images to have the same size. So I plan to use the size width 600 height 600 for all. I first make a background image of all white, resize the map image to the size of the heatmap using the thumbnail function, and paste it into the background. then I read the heatmap image and fix its size and convert it to L, which means greyscale for some reason. then I make a red image of all red, and I composite all of them together.

bg_img = Image.new("RGBA", (600, 600), 0)
map_img = Image.open('img/map.png')
map_img.thumbnail((474, 546), Image.ANTIALIAS)
bg_img.paste(map_img, (36, 21))
heatmap_img = Image.open('img/heatmap.png').resize((600, 600)).convert("L")
bk_img = Image.new("RGB", (600, 600), 255).convert("RGBA")
final_img = Image.composite(bk_img, bg_img, heatmap_img)
final_img.save('img/result2.png')

heatmap_img

The result is, for the places with more scooters, a red overlay will be on them, so we can tell where the scooters are focused.

What's next

In the next blog, I plan to analyse the Voi scooter data from July, since my data collecting program has been running for over a month already. I'm sure there will be interesting data collected.

My brand

To make social media accounts, I need a brand. I am just going to call it "Chit's Programming Blog". Then I also need an email for all the social media, but I don't want to use my personal email or work email because I don't want to get my emails entangled. So I made a new Gmail account since it is the most trusted. Popular social media may blacklist other email providers because they are less secured and may be used to create spam bots etc.

Profile picture

I need a profile picture for social media. I don't want to use my photos, I'm also concerned about copyright and trademarks, so I cannot just snap an image online and use it. Therefore I need to design my own profile picture.

There are a handful of tools I can use. There are tools where I have to make everything from scratch, like Microsoft Paint, and Photoshop. There are also tools that give stock images and fonts for you, like Canva. There are also tools that claim to use AI to generate your logo if you tell it about what you want, like Tailor Brands, Designs AI, and Adobe Express.

I used Adobe Express since it is the bigger brand, and it says free to use forever on their website.

Social Media Presence

Then I created my social media page on different platforms.

Twitter

Once I created my basic Twitter account, I get to upgrade it into a Twitter Business Account, where I have to tell Twitter what company I am. Since there is no choice for content creator or blogger, I chose Information Technology Company. It also asks me if I am a shop or a creator, which I chose creator.

Then my Twitter account is finished. It gave me a handle "@BlogChit", which I am not so happy with, then I found out on their website that I can change the handle. I want the handle to be memorable and easy to type out. Looking at popular Twitter users, most of them use the same as their display name, so I think I will do that too.

But I discovered my name is too long, so at last, I decided to go with "ChitProgramming"

Instagram

At first, I thought I must use my real name on Instagram. But it turns out I can use my business name too, so I used it for searchability.

When I thought that I'm done making my Instagram account, I found a button to "Switch to professional account". I found this article talking about why should I switch to a professional account, and the point Instagram Business Profiles Can Share Links in Instagram Stories sold me, it means that I'll be able to share my blog post with a single click. So I clicked the button, selected "Creator" to describe me better than "Business", choose "Personal Blog" as my category, and then I have a professional Instagram account.

Facebook

This is basically the same with Instagram since it is owned by the same company, Meta. But one annoying thing is that it makes posts that I don't want. After I created the profile and added images, it made a post about my birthday, how I updated my profile picture and updated my cover photo.

Facebook seems to force you to use your real name to make an account, so I was confused why there are so many business pages on Facebook, it turns out that peoples use "profile", while businesses use "page", and "profile" to create "pages".

So I created indicating that my category is "Information Technology Company", added my blog's information, then it was done.

Reddit

Reddit is the easiest of all, I just have to select a username, and enter my blog's info. I don't have to set up a professional account or anything.

Cross-posting

I thought I had to make a program to automatically post my blog on other websites, then make a canonical link back to the original one. But turns out, that both Dev and Medium have features that help you publish articles from other sources.

Dev.To

Inside the Settings -> Extensions page, there is an option where I can publish to the Dev community from RSS.

I get my RSS on the upper right corner of my blog, then I paste it on Dev.

Then all my posts from Hashnode end up there, that's so cool and useful.

I have to add tags manually, I have to check what tags are trending on Dev. I check this on their tags page

Medium

As for Medium, I cross-post by going to Settings -> Stories -> Import a story.

Then I input the link to my article and click import.

This one gave me some extra spacing between lines, which I have to remove.

Medium doesn't share what tags are trending, it only recommended some topics, so I have to find out in a third-party website.

What's next

In this article, I created social media accounts and cross-posting accounts. Now I can share my blog posts on social media and engage in discussions. And I will cross-post my previous blog posts at a regular interval, to avoid being spammy. As for SEO, since it takes time for Google to index my blog posts, I think I'll have to wait, and post an update once I can find myself on Google.

Possible reasons for low viewer count

I start by investigating the shortcomings of my blog, to see how I can improve.

Time on the platform

I've only written several blog posts. There is a small chance in each of my blog posts to be appreciated and attract followers. So the longer I am on this platform, the higher chance this happens.

My writing skills

I haven't written a blog before, and seldom write essays since I study Computer Science. Therefore my writing may wordy, unclear, and unpleasant to read. To fix this, I need to write more, as well as read good blog posts to learn.

My writing topics

Looking at the trending blog posts on Hashnode, it is clear the trending ones teach you something or tell you something you don't know. But the ones I write document a programming journey, so it is not what most people look for.

However, I do not intend to change my article topics, as my blog is for documenting my programming journey and my target audience is people interested or wanting to learn from my experience.

I also notice a lot of popular blog posts have a number in them, like 6 Habits I have picked up from working in tech for 3 years, 15 Common Beginner JavaScript Mistakes. I thought about using AI, feeding in article topics and views, to see if there is any correlation with it, but I'll save this idea for later.

People lack a chance to see my articles

I watched a video from Youtuber How Money Works, called Why Finance "Gurus" Want You To Hate Them - How Money Works. In that video, the author made an interesting point. The reason why fake gurus - people who give bad financial advice, act so unbearable, is because "any publicity is good publicity", working in conjunction with the sales funnel.

The sales funnel

The sales funnel is each step that someone has to take in order to become your customer, as visualised above. Actually, it is more like a sieve or a filter, since a funnel keeps everything poured in, but some customers will be lost in the process. For the sake of recognisability, I'll keep calling it a sales funnel.

Let's say in each step of my sales funnel, I keep 10% of the customers. If I want more people to enter the last step, follow me, I can: a) do better in each stage to keep more customers, such as making clickbait titles and writing a more interesting first paragraph of my article to increase the percentage of people staying in each stage. b) put more people in the initial stage of the sales funnel, so that more people will enter the final stage.

That's why I came up with the idea of making a program which automatically cross-posts my blogs and also creates social media presence to promote my blog to increase my exposure and put more people inside my sales funnel.

Popular blogging sites

To do that I will have to know where should I cross-post my blog, below are websites that I found online.

• Hashnode, where I'm on
• HackerNoon and FreeCodeCamp, have dedicated editors to review blog posts, so I may only submit high-quality blog posts
• Dev.To, simular to Hashnode
• Medium, a website for all kinds of articles
• Google Blogger / WordPress / Wix..., these sites while giving me a lot of freedom, there is no way for people to find me there, as there is no explore button, so I will not consider these, as at this stage, I want to be discovered
• LinkedIn, a place with professionals and experts, so I need to be careful what I post there
• Tumblr, a mini blogging website with built-in social media capabilities

Social media where I can create a presence

• Twitter
• Reddit
• Instagram
• Facebook
• Snapchat
• Youtube and TikTok, if I ever consider making videos
• Pinterest, seems to be image-based but I don't think I have any images to show

Search Engine Optimization

Ideally, my blog posts should be searchable from Google, so there is an additional way for people to visit my blog. But currently, my blogs cannot be searched on Google.

The first reason is that I have an apostrophe in my blog title and that "chit" is usually used with "chit chat", so Google thinks it was a typo. If I force Google to search "Chit's", my blog appears on top.

The other reason is that I recently changed the name from "Chit's tech blog" to "Chit's programming blog" because I realized that a tech blog is a blog that talks about new technology, which is not what I cover.

When I search for the title of my first article, I cannot find the result. It is because the past me thought it would be a good idea to make a custom SEO title. If I have time, I want to edit all my past blog posts' SEO titles and descriptions, for Google to better index my blog posts.

Also, notice that my article published on DevTo was indexed instead. This is undesirable because my articles on different platforms are fighting for views, what I should do instead is to use a canonical link, to tell Google, that the article you found on DevTo, comes from Hashnode, so Google knows where to send people.

When I search for newer articles, they do not appear in the result, I assume it is because it takes time for Google to index my blog posts.

I also designed an image for social media sharing, so it looks prettier when I share my blog.

I have to say, Canva is really easy to use, and has some nice templates.

Now All I have to do is wait for Google to index my new and improved blog.

Action plan

In the next blog, I want to create a social media presence on the above social media platforms, as well as investigate how can I cross-post to the other blogging platforms.

Finding out the latitude and longitude of my city

Scooter data uses latitude and longitude to store where they are. Latitude and longitude are a way to represent where something is in the world by how far it is from the equator and the prime meridian.

Heatmap

I aim to use a heat map to plot the concentration of the scooters. Heat maps show which part of the map is hot. It is used for three-dimensional data where the x and y axis are for the first two dimensions, and the colour (heat) is for the third. In my case, the x and y axis will be used for the latitude and longitude, and the colour will represent how many scooters are in that area.

Changing data representation

Currently, my scooter location data is split into two dataframe, one for latitude and one for longitude. I want to get all the scooter locations from a timestamp, so I will get the data from the same row from both dataframes.

Then I will fill the data into a dataframe where the latitude and the longitude are the two axes, and the value in the middle means how many scooters are in that coordinate range. So I want to find the 4 corners of the city where there are scooters, so I know how to set the range for the latitude and longitude.

Fixing floating point precision using string formatting

Then I turn the four corners into a range, I use the np.arange() function, however, due to the precision problem of floating point numbers, the numbers are not accurate.

list(np.arange(52.59, 52.40, -0.005))

the output is, [52.59, 52.585, 52.58, 52.574999999999996, 52.56999999999999,...

And the dataframe looks like this

Therefore I used string formatting and list comprehension to turn this list of floating point numbers into a string, which doesn't have the problem of precision.

lat_index = np.arange(52.59, 52.40, -0.005)
str_lat_index = ["%.3f" % i for i in lat_index]

Creating the dataframe

Now I can create the dataframe from the str index and column, I used fillna(0) because at first of creating the dataframe, all values are null values, and since I am counting the number of scooters, everything should be 0 at the beginning.

scooter_area_df = pd.DataFrame(index=str_lat_index, columns=str_lng_column, dtype=np.int64).fillna(0)

Filling the dataframe

Then I extracted the specific row from the dataframe, for the starting index, I took the value at 2022-06-25 at 5 am.

df_starting_index = 26852

latitude_df_row = latitude_df.iloc[df_starting_index]
longitude_df_row = longitude_df.iloc[df_starting_index]

And I fill in the scooter_area_df by going through the latitude dataframe and longitude dataframe, fetching the value from both of them and incrementing the particular cell in the scooter_area_df. Notice that I used the lazy approach, which is to round down the data to the nearest 2 decimal point, this is not accurate because what I should have down is to round up/down depending which one is closer.

for lat, lng in zip(latitude_df_row, longitude_df_row):
    try:
        if not pd.isnull(lat):
            str_lat = "%.2f" % lat
            str_lng = "%.2f" % lng
            scooter_area_df.loc[str_lat, str_lng] += 1
    except KeyboardInterrupt:
        break
    except:
        traceback.print_exc()

Plotting the graph

I then plot the graph using the seaborn library and add the appropriate labels to it.

fun fact: We usually import seaborn as sns because Samuel Norman "Sam" Seaborn is a fictional character portrayed by Rob Lowe on the television serial drama The West Wing.

import seaborn as sns
fig, ax = plt.subplots(figsize=(12, 10))
fig.patch.set_facecolor('white')
ax = sns.heatmap(scooter_area_df, square=True)
plt.xlabel("Longitude")
plt.ylabel("Latitude")
plt.title("The concentration of scooters")

Problems with the current plot

This result is not desirable because:

1. The squares are too big, I wish we could be more precise
2. It may not be geographically accurate, as 1 latitude is usually not as long as one longitude, ideally the heatmap will look the same as if it is plotted on an actual map.

Problem 1: being more precise

To do this, How much we decrease the square size matters a lot, because if we decrease it too much. If there is a bunch of scooters next to each other, they might be separated into different areas, and we cannot see how concentrated they are.

The following is an example of what happened if we decreased the step, which is the square size, by 10 times, you can hardly see the dots.

So I need to find an appropriate amount to decrease the step.

Problem 2: the actual ratio of the map

To do this, I plan to measure the relationship between the distance between the two points, and the differences in latitude and longitude.

The horizontal distance is 13.53 km, and the longitude difference is 0.2

The vertical distance is 15.57 km, and the latitude difference is 0.14

So let's say if I want each square to be 100 meters, then the vertical step will be 0.14/155.7, and the horizontal step will be 0.2/135.3.

Find the closest value in a sorted array

Previously, I took the lazy route and just used string formatting to do the rounding, but since now the intervals are not multiple of 10, I need to find a new way. I aim to find which value from the vertical range or horizontal range is the closest to the value, so a binary search should be useful.

I copied the code from this stackoverflow answer, which uses the bisect library for a quick way to find the nearest result, the code looks like this:

from bisect import bisect_left

def take_closest(myList, myNumber):
    """
    Assumes myList is sorted. Returns closest value to myNumber.
    If two numbers are equally close, return the smallest number.
    """
    pos = bisect_left(myList, myNumber)
    if pos == 0:
        return myList[0]
    if pos == len(myList):
        return myList[-1]
    before = myList[pos - 1]
    after = myList[pos]
    if after - myNumber < myNumber - before:
        return after
    else:
        return before

I also changed the code to fit the code that updates the value inside the scooter_area_df, notice that I used lat_index_reversed, since take_cloest only works on an ascendingly sorted array, I had to make a reversed latitude array to make it work, take_cloest returns a value instead of an index, so I don't have to modify the returned value.

str_lat = "%.3f" % take_closest(lat_index_reversed, lat)
str_lng = "%.3f" % take_closest(lng_column, lng)
scooter_area_df.loc[str_lat, str_lng] += 1

now the heatmap looks like this, which when looking at the Voi app, is very similar to how my city looks

Turning it all into a function

I now made plot_concentration_of_scooters a function, in which I first get the two dataframes outside of the function so it will only be called once, then declare where latitude and longitude start and end.

import seaborn as sns

longitude_df = store['longitude_df']
latitude_df = store['latitude_df']

lat_start_index = 52.54
lat_end_index = 52.40

lng_start_index = -1.99
lng_end_index = -1.79

Then I calculate the vertical step and horizontal step, create the ranges, make the scooter area dataframe, fill it, and plot the result.

def plot_concentration_of_scooters(df_index, square_side_in_meters):
    lat_step = (lat_end_index - lat_start_index) / (15.57 * 1000 / square_side_in_meters)
    lng_step = (lng_end_index - lng_start_index) / (13.53 * 1000 / square_side_in_meters)

    lat_index = np.arange(lat_start_index, lat_end_index, lat_step)
    lat_index_reversed = lat_index[::-1]
    str_lat_index = ["%.3f" % i for i in lat_index]

    lng_column = np.arange(lng_start_index, lng_end_index, lng_step)
    str_lng_column = ["%.3f" % i for i in lng_column]

    scooter_area_df = pd.DataFrame(index=str_lat_index, columns=str_lng_column, dtype=np.int64).fillna(0)

    latitude_df_row = latitude_df.iloc[df_index]
    longitude_df_row = longitude_df.iloc[df_index]

    for lat, lng in zip(latitude_df_row, longitude_df_row):
        if not pd.isnull(lat):
            str_lat = "%.3f" % take_closest(lat_index_reversed, lat)
            str_lng = "%.3f" % take_closest(lng_column, lng)
            scooter_area_df.loc[str_lat, str_lng] += 1

    fig, ax = plt.subplots(figsize=(12, 10))
    fig.patch.set_facecolor('white')
    ax = sns.heatmap(scooter_area_df, square=True)
    plt.xlabel("Longitude")
    plt.ylabel("Latitude")
    plt.title(f"The concentration of scooters on {longitude_df_row.name}")

here is what happens when I use different values

plot_concentration_of_scooters(38339, 500)

Making multiple results into a gif

With a still frame, I can only tell the distribution in a static manner, what if I want to show the change over time? I want to make a gif to show the change, like in a day.

I found this article that is really helpful, it talks about how to make multiple plots into a graph.

What I came up with is to have a temporary folder, then I plot each graph, after that saving them with plt.savefig(), and then using the imageio library to turn them into a gif file, after that deleting the temporary plot images.

import imageio

def make_scooter_concentration_gif(df_start_index, df_end_index, step, square_size_in_meters):
    temp_folder = Path('temp/scooter_concentration_gif/')
    df_index_range = range(df_start_index, df_end_index, step)
    for df_index in df_index_range:
        plot_concentration_of_scooters(df_index, square_size_in_meters)
        plt.savefig(temp_folder.joinpath(f'{df_index}.png'))
        plt.close()

    with imageio.get_writer('graphs/mygif.gif', mode='I', duration=1) as writer:
        for df_index in df_index_range:
            image = imageio.imread(temp_folder.joinpath(f'{df_index}.png'))
            writer.append_data(image)

    for df_index in df_index_range:
        temp_folder.joinpath(f'{df_index}.png').unlink()

the result of make_scooter_concentration_gif(28288, 29304, 100, 200) is

Improving the function

This is good, but one that I have against it is that it loops too quick and it is hard to tell when is the graph ending, I can either change the loop variable for the gif to make it not loop, or I can add a blank frame at the end so the users know if a new cycle begins, I prefer the latter solution.

Also, in each image, the scale for the anchor the colourmap is different, sometimes the max is 20, sometimes 10, so it is hard to tell for a single pixel, whether it is gaining scooters or not, so I modified the code the have a vmax argument, to fix the maximum for the scale.

the result looks like this make_scooter_concentration_gif(28288, 29304, 100, 200)

at another day make_scooter_concentration_gif(36903, 37920, 60, 200, 15)

Final version of the code for reference

def plot_concentration_of_scooters(df_index, square_side_in_meters, vmax=None):
    lat_step = (lat_end_index - lat_start_index) / (15.57 * 1000 / square_side_in_meters)
    lng_step = (lng_end_index - lng_start_index) / (13.53 * 1000 / square_side_in_meters)

    lat_index = np.arange(lat_start_index, lat_end_index, lat_step)
    lat_index_reversed = lat_index[::-1]
    str_lat_index = ["%.3f" % i for i in lat_index]

    lng_column = np.arange(lng_start_index, lng_end_index, lng_step)
    str_lng_column = ["%.3f" % i for i in lng_column]

    scooter_area_df = pd.DataFrame(index=str_lat_index, columns=str_lng_column, dtype=np.int64).fillna(0)

    latitude_df_row = latitude_df.iloc[df_index]
    longitude_df_row = longitude_df.iloc[df_index]

    for lat, lng in zip(latitude_df_row, longitude_df_row):
        if not pd.isnull(lat):
            str_lat = "%.3f" % take_closest(lat_index_reversed, lat)
            str_lng = "%.3f" % take_closest(lng_column, lng)
            scooter_area_df.loc[str_lat, str_lng] += 1

    fig, ax = plt.subplots(figsize=(12, 10))
    fig.patch.set_facecolor('white')
    ax = sns.heatmap(scooter_area_df, square=True, xticklabels=5, yticklabels=5, vmax=vmax)
    plt.xlabel("Longitude")
    plt.ylabel("Latitude")
    plt.title(f"The concentration of scooters on {longitude_df_row.name}")

import imageio

def make_scooter_concentration_gif(df_start_index, df_end_index, step, square_size_in_meters, vmax=None):
    temp_folder = Path('temp/scooter_concentration_gif/')
    df_index_range = range(df_start_index, df_end_index, step)
    for df_index in df_index_range:
        plot_concentration_of_scooters(df_index, square_size_in_meters, vmax=vmax)
        plt.savefig(temp_folder.joinpath(f'{df_index}.png'))
        plt.close()

    with imageio.get_writer('graphs/mygif.gif', mode='I', duration=1) as writer:
        for df_index in df_index_range:
            image = imageio.imread(temp_folder.joinpath(f'{df_index}.png'))
            writer.append_data(image)
        writer.append_data(imageio.imread('temp/scooter_concentration_gif/blank.png'))

    for df_index in df_index_range:
        temp_folder.joinpath(f'{df_index}.png').unlink()

What's next

For the next blog, I aim to incorporate the map into the plots, like making the map as a background image in the heatmap or plotting the scooters at dots on the map.

Deal with missing data

Because of my data collection program crashing, there are some data missing, I now need to find them out, so that I don't mess up my data analysis by making incorrect assumptions.

Finding missing data

I am going to iterate through all the data I have and check the separation between it and the previous data item. The data is collected every 1 minute, so any time longer than that indicates there is something wrong. But since running the program also takes time, the separation should be 1 minute + the duration of each loop. There might also be cases where there is a slight error and the program skips a minute. So I will check for data separated by more than 3 minutes to be safe.

def find_missing_data():
    battery_level_df = store['battery_level_df']
    previous_timestamp = battery_level_df.index[0]
    for current_timestamp in battery_level_df.index:
        if current_timestamp - previous_timestamp > pd.to_timedelta("3 minutes"):
            print(f"Missing data from {previous_timestamp} to {current_timestamp}")
        previous_timestamp = current_timestamp

find_missing_data()

The program finds that there are missing data in the following positions:
Missing data from 2022-06-02 21:59:47.420510 to 2022-06-02 23:24:21.071122
Missing data from 2022-06-04 14:46:03.130303 to 2022-06-05 19:08:46.083742
Missing data from 2022-06-15 09:04:56.485744 to 2022-06-17 21:15:31.558383
Missing data from 2022-06-22 09:05:24.070133 to 2022-06-22 15:09:05.898395

Finding day cuts

Scooters are available from 6 am to 10 pm, so knowing when the day starts and ends is important to our analysis. Since there is no direct way to tell, I need to find them out.

It works by iterating through the timestamp and vehicle count array, then whenever the vehicle count drops the zero, we know it is 10 pm and the day ended, and we set zero_flag to True, then when the vehicle count leaves zero while zero_flag is True, we know it is 6 am, this repeats until the last data item.

def find_day_cuts():
    timestamp_np_array = np.load('cached_data/timestamp_np_array.npy', allow_pickle=True)
    vehicle_count_np_array = np.load('cached_data/vehicle_count_np_array.npy', allow_pickle=True)
    zero_flag = False
    for index, item in enumerate(zip(timestamp_np_array, vehicle_count_np_array)):
        if item[1] == 0 and not zero_flag:
            print("become zero", index, item[0])
            zero_flag = True
        if zero_flag and item[1] != 0:
            print("leave zero", index, item[0])
            zero_flag = False

find_day_cuts()

The program outputs each day like this:\ become zero 594 2022-06-02 23:24:21.071122\ leave zero 930 2022-06-03 05:00:53.124146

Analysis of battery

Individual scooter

I am going to analyse the battery of a scooter. I first read the dataframe from the HDFS storage, then I generate the series from the dataframe, and change the first and last value to zero so that it will not be skipped.

def plot_scooter_battery(starting_index, steps, scooter_id):
    global battery_level_df
    series = battery_level_df.iloc[starting_index: starting_index + steps, battery_level_df.columns.get_loc(scooter_id)]
    series[0] = 0
    series[-1] = 0
    fig, ax = plt.subplots()
    fig.patch.set_facecolor('white')
    series.plot(title=f"Battery of scooter {scooter_id} from index {starting_index} to {starting_index + steps}", xlabel="timestamp", ylabel="battery level", ax=ax)

plot_scooter_battery(930,1400,'npv9')

To illustrate my point, the graph without skipping zero looks like this

By setting the last item in the series as zero, the graph is forced to be extended, so we can see the whole thing.

Critical mistake

Then I realized that by changing the starting and ending value of the series, I actually accidentally edited the dataframe that I am not supposed to edit. This is because by calling the iloc function, the returned series is actually a reference, instead of a copy, so any edit done to that reference will reflect on the dataframe as well.

I was afraid that I have to re-run the function which took an hour to run to make the dataframe again. Luckily, I didn't save my edit to the HDFS storage, so I could load it again.

So I changed the function to make a copy of the series by calling .copy(), and I only edit the series if it is NaN, the updated function is:

def plot_scooter_battery(starting_index, steps, scooter_id):
    global battery_level_df
    series = battery_level_df.iloc[starting_index: starting_index + steps, battery_level_df.columns.get_loc(scooter_id)].copy()
    series[0] = 0 if pd.isnull(series[0]) else series[0]
    series[-1] = 0 if pd.isnull(series[-1]) else series[-1]
    fig, ax = plt.subplots()
    fig.patch.set_facecolor('white')
    series.plot(title=f"Battery of scooter {scooter_id} from index {starting_index} to {starting_index + steps}", xlabel="timestamp", ylabel="battery level", ax=ax)

Other graphs

what happens when that scooter is not available on that entire day

Mean battery level of all scooters

Difficulty

This plot sounds simple at first, I just have to use the mean() function from pandas and everything is done. The challenge is with NaN values, there are scooters that are not available for the entire day, and there are also scooters that are being used.

To illustrate my point, I drew this simulation, there are 3 scooters, scooters A, B and C, the line is its battery level, in between the lines is the time when the scooter is being used, and the value is NaN. At p1, no scooter is being used, so the mean is the mean of all scooters. At p2, scooters A and B are being used, so they are not being used to calculate the mean, so the means is the remaining scooter, scooter C. At p3, all scooters are back, so the mean goes back up.

This isn't the real mean battery of all scooters. This is what the graph looks like without any editing.

Plan of action

Ideally, I want to fill in the blanks with a line going from the previous point to the next point, as if the battery consumption is linear. However, that would be difficult. When I am iterating through the elements, I'll have to find the point when the data stops for a particular scooter, go forward until the data reappear, and fill in the data linearly. That is fine for a scooter, but will be very difficult with so many scooters.

Moreover, with so many scooters, a forward fill, where I fill NaN values with the previous value will work very similarly because of the sample size.

So what I did is to first copy the specific area of the dataframe I want to analyse and copy it, so I won't change the original dataframe. Then I use the dropna() function to drop columns where all the values are NaN, meaning that particular scooter wasn't available that entire day, and we don't need it. Then I looped through all the elements in the copied dataframe to fill in the NaN values.

starting_index = 930
steps = 1945-930

battery_level_df = store['battery_level_df']
selected_area_df = battery_level_df.iloc[starting_index: starting_index + steps].copy()
selected_area_df.dropna(axis='columns', how='all', inplace=True)
selected_area_df_without_filling_nan = selected_area_df.copy()

for row_num, (row_index, row_content) in enumerate(selected_area_df.iterrows()):
    for item_num, (item_index, item_content) in enumerate(row_content.iteritems()):
        if pd.isnull(item_content):
            selected_area_df.iloc[row_num, item_num] = selected_area_df.iloc[row_num - 1, item_num]

Then I plot it using the following code:

plt.figure(facecolor='white', figsize=(15, 8))
plt.plot(selected_area_df.mean(axis=1))
plt.ylim([55,65])
plt.title("Mean battery level over time using my own way to fill na")
plt.xlabel("Timestamp")
plt.ylabel("Battery Level")

But then I realized that I can use the inbuilt fillna function with the ffill method, which forward fills the value. I ran the function and plotted the graph, this function ran faster than my own.

Different results from my implementation and the inbuilt method

But very confusingly, although these two plot looks similar, there are some very minor differences. These two functions are supposed to work the same way, therefore I think I have to investigate what caused the differences.

I suspect it is because in my implementation, I did selected_area_df.iloc[row_num, item_num] = selected_area_df.iloc[row_num - 1, item_num], if the first element is NaN, the value from the end will be used to fill it because of how negative indices works. I investigated and found there are some scooters that were initially unavailable, but then were activated in the middle of the day. That is why the result is different, I forgot about the edge case.

Assumption about the data

This brings me to the problem, what should I assume about the scooters that aren't available at the start? There are 3 possibilities:

1. It was broken and had to be fixed
2. It was out of battery and need a battery replacement
3. It was being used, but this is not possible, because it is just after the scooters are available, so no one has time to unlock scooters yet

I think most of them will be out of battery instead of broken, so I plan to fill the lowerest value of battery to them, I know for a fact that scooters don't get deactivated when they hit zero, they usually are deactivated before that to prevent accidents where a scooter run out of battery mid-ride happen, so the problem is when, when does the scooter get diactivated? I looked through the dataframe, and usually, the scooter gets deactivated when they reach 8 per cent battery, so I will assume scooters that are not available at the start of the day are at 8 per cent battery.

To compare, I kept the same y limit to this graph, as you can see, the average battery count decreased a lot.

My assumptions were incorrect

This is actually still inaccurate, as at the end, let's say a scooter is unlocked with 15 per cent battery left, when it is locked if the battery level is lower than 8 per cent, Voi don't want you to unlock it again, so the scooter will not be available again, in this case, we incorrectly assume the battery is still at 15 per cent because of forward filling, when in fact it should be 8 or less per cent.

I intend to fix it by filling all NaN values at the end with 8 per cent, and backward fill only those values up until they were last unlocked, I believe this will give us a more realistic view of the battery condition.

I am thinking of a shortcut, which is to forward fills the first half and backwards fill the second half, this would be fast, and create mostly what we want, except there will be a jump in the middle

as you can see, this is very problematic.

My assumptions were still incorrect

I found out that some scooters are disabled while having 50% battery or above, which means they are not disabled because of a lack of battery, but for other reasons, that is why assuming scooters end with 8 per cent battery is so wrong.

My solution is to use the previous day to find the last seen battery count. I first compute the one_day_before_index, which is either one day before the starting index, or 0 when we don't have the data for one day before. Then I first get the columns of data used for that specific day, then I use forward fill on the data plus one day before, then cutting the previous day, so that the data is in the desired range.

final code

def plot_mean_battery(starting_index, ending_index):
    one_day_before_index = max(starting_index - 1440, 0)
    starting_diff = starting_index - one_day_before_index
    battery_level_df = store['battery_level_df']
    columns = battery_level_df.iloc[starting_index: ending_index].dropna(axis='columns', how='all').columns
    selected_area_df = battery_level_df[columns].iloc[one_day_before_index: ending_index].fillna(method='ffill')[starting_diff:]
    plt.figure(facecolor='white', figsize=(15, 8))
    plt.plot(selected_area_df.mean(axis=1))
    plt.title(f"Mean battery level over time from {selected_area_df.index[0]} to {selected_area_df.index[-1]}")
    plt.xlabel("Timestamp")
    plt.ylabel("Battery Level")

plot_mean_battery(3539, 13166)

Other days

Now I run this function on other days, and these are the result.

Entire week

For the entire week, from 6/6 Monday to 12/6 Sunday.

What's next

I may try to explain the result of the average battery count, I may also try to start looking into the scooter locations.

Improving the data collection program

The data collection program crashed multiple times. Since data cannot be collected while the program is crashed, I have a lot of missing data, which makes data analysis difficult.

I need a try-except block to catch exceptions instead of letting the program crash. I should have done this long ago, but I was lazy. Now I need to pay the price.

In the main loop, I first ask for a response, then store the response, and wait 60 seconds before making another request, the two possible errors are:

Failing to get a response from the server

When I make a request to the API server, it has happened that it refuses to give me an access token, or response. In this case, I need to cancel future actions and make a request maybe 5 seconds later.

Getting an invalid response from the server

If the response is not what I expected, I will simply not parse the response. and wait for the next iteration.

The fix of the data collection program

The following is the before and after of the main loop. I used two try-except blocks to wrap up the two functions that may crash. In the first block, if the program runs into an exception, I will print out what is the problem, and continue the loop after 5 seconds. In the second block, I simply print out the problem and do nothing. So now if something bad happens, the program will not crash, but instead, wait and try again later. In retrospect, this fix is so easy that I should have done this earlier, so then I don't have to deal with so much missing data.

Before:

while True:
    response = make_api_requests.get_scooter_locations()
    handle_response.store_response(response)
    time.sleep(60)

After:

while True:
    try:
        response = make_api_requests.get_scooter_locations()
    except:
        print("Failed to get scooter locations, retrying in 5 seconds")
        time.sleep(5)
        continue
    try:
        handle_response.store_response(response)
    except:
        print("Failed to store response, will skip storing the response")
    time.sleep(60)

Moving the project to a Jupyter notebook

From my other blog post, I discovered about Jupyter notebook and decided to use it to complete the analysis. As a JetBrains user, I searched for tools from Jetbrains that support Jupyter notebooks. I found DataSpell. It seems useful for me as it can:

• automatically starts a Jupiter server when I start running the code
• allow me to view dataframes and Jupyter notebook variables
• be used for free because of the Github Student Developer plan, so why not

Importing and checking data

The first thing to do is to sanitise the data, I will have to iterate the data files. I copied the functions from my previous blog post.

from pathlib import Path

def get_sorted_json_list():
    path_list = sorted(Path('location_data/').glob('*.json'))
    return path_list

Then I print all files that are not valid json, because if my data-gathering program ran correctly, all data should be valid json, so I want to double-check before doing any analysis.

import json

def check_all_files():
    sorted_file_list = get_sorted_json_list()
    for path in sorted_file_list:
        with open(path, 'rb+') as f:
            try:
                json_data = json.load(f)
            except ValueError:
                print(f"file {path.name} is not valid json")
    print("all file ok")

Two files showed up as invalid json, so I fixed the file and tried again, this time all are valid

Creating numpy array

In the previous blog post in this series, I plotted the graph of vehicle counts, now I want to redo it with the new data I gathered. I first created the numpy array for timestamps and vehicle counts.

import numpy as np
import pandas as pd

timestamp_np_array = np.empty(MAX_DATA_SIZE, dtype=pd.Timestamp)
vehicle_count_np_array = np.empty(MAX_DATA_SIZE, dtype=int)
counter = 0

Then I define the function that gets the vehicle count from a file, it opens the file and fills the two numpy arrays with data. I used the global keyword here so that I don't have to worry about scopes.

import traceback

def get_vehicle_count_list_from_file(path):
    global timestamp_np_array, vehicle_count_np_array, counter
    with open(path, 'r') as f:
        try:
            json_data = json.load(f)
            for individual_data in json_data:
                if counter < MAX_DATA_SIZE:
                    timestamp_np_array[counter] = pd.to_datetime(individual_data['time_stamp'])
                    vehicle_count_np_array[counter] = individual_data['vehicle_count']
                    counter += 1

        except ValueError:
            traceback.print_exc()
            print(f"file {path.name} is not valid json")

Then I execute the operation, truncate the two numpy arrays so they are only as long as the amount of data we have, and then save them so they can be loaded at another time, I am not going to work non-stop on this project, so I need to be able to load the data at another time.

for path in get_sorted_json_list():
    get_vehicle_count_list_from_file(path)

timestamp_np_array = timestamp_np_array[:counter]
vehicle_count_np_array = vehicle_count_np_array[:counter]

print(counter)

np.save('cached_data/timestamp_np_array', timestamp_np_array)
np.save('cached_data/vehicle_count_np_array', vehicle_count_np_array)

Plotting the data

I now plot the graph. I stated how long, because I may not want to plot the entire graph, I also have a step variable so that the graph will not be too packed. There are 39356 data items to fit inside a graph, so not every point has to be considered

import matplotlib.pyplot as plt

def plot_vehicle_count_graph():
    timestamp_np_array = np.load('cached_data/timestamp_np_array.npy', allow_pickle=True)
    vehicle_count_np_array = np.load('cached_data/vehicle_count_np_array.npy', allow_pickle=True)

    fig, ax = plt.subplots(figsize=(15,8), subplot_kw={"title": "Number of available scooters over time",
                                                       "xlabel": "timestamp",
                                                       "ylabel": "Number of scooters available"})
    fig.patch.set_facecolor('white')
    starting_time = 0
    how_long = 39356
    step = 30
    ax.plot(timestamp_np_array[starting_time:starting_time+how_long:step],
            vehicle_count_np_array[starting_time:starting_time+how_long:step])
    plt.show()

plot_vehicle_count_graph()

Note that I put everything in a function, this is so that when I come back to the project, and I need to run previous code blocks to import everything I need, I can do that without doing everything. I could've also separated all the imports and put them at the start of the Jupyter notebook.

Placing data in dataframe

I decided that it is important to have the data in tabular form, because when I investigate battery consumption, having data in tabular form makes it easier to track individual scooters. As it is just going down a column, compared to digging through multiple files.

I precompute everything and put it inside a dataframe, instead of going through the files every time I do the analysis for the scooters.

In that case, I will need to know how many columns and rows there will be, as in the pandas' documentation when I create a dataframe, I need to specify the columns and index.

Generating all vehicles set

We iterate through all the files, then we use list comprehension to accumulate all the ids, and we use the union operation to combine the set with the total set.

vehicle_id_set = set()

def all_vehicle_id_for_a_single_json_file(path):
    global vehicle_id_set
    try:
        with open(path, 'r') as f:
            json_data = json.load(f)
        for time_data in json_data:
            new_set = set([vehicle_data[0] for vehicle_data in time_data['vehicle_data']])
            vehicle_id_set = vehicle_id_set.union(new_set)
    except:
        traceback.print_exc()
        print(f"In all_vehicle_id_for_a_single_json_file: file {path.name} is not valid json")

def add_all_vehicle_id_to_set():
    global vehicle_id_set

    # Iterate through all files
    sorted_json_list = get_sorted_json_list()
    for json_path in sorted_json_list:
        all_vehicle_id_for_a_single_json_file(json_path)

add_all_vehicle_id_to_set()

Generating all timestamp list

Now I find out all the timestamps to use them as indexs.

datetime_np_array = np.empty(shape=(39356,), dtype=pd.Timestamp)
datetime_np_array_counter = 0

def get_all_dates():
    global datetime_np_array_counter
    sorted_json_list = get_sorted_json_list()
    for json_path in sorted_json_list:
        try:
            with open(json_path, 'r') as f:
                json_data = json.load(f)
            for time_data in json_data:
                datetime_np_array[datetime_np_array_counter] = pd.to_datetime(time_data['time_stamp'])
                datetime_np_array_counter += 1
        except:
            traceback.print_exc()
            print(f"In get_all_dates: file {json_path.name} is not valid json")

actually creating the dataframe

Now I create a dataframe with all the vehicle_id as columns, and the number of collected data as rows, this will be a huge dataframe. This is rather straightforward, I just call their constructor with the index, columns and data type specified.

battery_level_df = pd.DataFrame(index=datetime_np_array, columns=vehicle_id_set, dtype=np.int64)
longitude_df = pd.DataFrame(index=datetime_np_array, columns=vehicle_id_set, dtype=np.float64)
latitude_df = pd.DataFrame(index=datetime_np_array, columns=vehicle_id_set, dtype=np.float64)

filling the dataframe

Now that I have this empty dataframe, I need to fill it, I will once again iterate through the json_files, and put the correct data in the correct place when filling in the fields, I can either 1 .assume the order of data is correct and just fill them in one row by row without checking the index

1. check the index every time, which is more expensive I think I will check the index, just to be safe

%%time

def fill_all_df(battery_level_df, longitude_df, latitude_df):
    for json_path in get_sorted_json_list():
        print(f"doing {json_path.name}")
        with open(json_path, 'r') as f:
            json_data = json.load(f)
        for time_data in json_data:
            current_row_datetime = pd.to_datetime(time_data['time_stamp'])
            for vehicle_data in time_data['vehicle_data']:
                battery_level_df.loc[current_row_datetime, vehicle_data[0]] = vehicle_data[1]
                longitude_df.loc[current_row_datetime, vehicle_data[0]] = vehicle_data[2]
                latitude_df.loc[current_row_datetime, vehicle_data[0]] = vehicle_data[3]

fill_all_df(battery_level_df, longitude_df, latitude_df)

I used the %%time to measure how long this operation took, and the result was Wall time: 1h 35min 49s, this proofs that this operation uses a lot of computing resources.

storing the data

Now I wish to store the result of the computation, I want it to store quickly, and load quickly, according to this article from Matthew Rocklin, the cost of serializing text-based data and numeric data is compared, and HDFS store seems to perform well in numeric data, so I decided to use it.

store = pd.HDFStore("store.h5")
store['battery_level_df'] = battery_level_df
store['longitude_df'] = longitude_df
store['latitude_df'] = latitude_df

What's next

I intended to talk about the results of my data analysis in this blog post, but this is already very long, so in the next one, I will use the dataframes to investigate why the number of scooters decreases throughout the day, and other interesting things.