TLDR

If you just want the tool name, Syncthing + Tailscale. If you want to understand how these tools work, this post is for you.

Motivation

I have been using Obsidian as my note-taking tool for a while. So far, the experience is top-notch, I can't compliment it enough.

Lately, I need to travel here and there without my laptop. My phone is the only digital friend I bring along. I usually note some outlines in my phone and elaborate on the details later when I have access to my laptop.

It becomes more tedious as the number of fleeting notes increases. That's the reason I want to synchronize the Obsidian folder on my phone and laptop.

Objectives

1. The folder content must be identical on the devices EVENTUALLY and AUTOMATICALLY. It not need to happen in real-time though.
2. In case of conflicts, I can manually choose which file to keep. Conflicts are unavoidable in synchronization, I need to pick a resolution strategy for those cases.
3. The solution must work as long as the devices are connected to the same network.
   The objectives are pretty straightforward, so I think there should be some ready-to-use solutions on the Internet.

Available solutions

Let's travel around the Internet to find which can help us.
The first solution showing up is buying Obsidian Sync, from the official Obsidian team. Not that I don't want to support the developer team, I also work in the computer field so I can understand why they do that. Everything needs money to survive. Server needs money to continue functioning. Human needs money to continue working.
However, I have financial problems, hence I can't afford that price. Sorry Obsidian team.
**If you can afford it, please buy the Obsidian Sync. **

After wandering around for some time, I notice a name that comes across multiple times, Syncthing. It's free and open-source, perfectly fitting my requirements. I decide to give it a try.

The setting is quite easy and well-documented. What's interesting me more is how Syncthing works under the hood.

How Syncthing works

The team has published a post about the underlying synchronization mechanism. I really appreciate and admire them for doing that. Open and free knowledge is always my dream and philosophy.

Without further ado, I will place a figure here to understand how it works in the big picture.

These steps seem simple, but a lot of big-brain engineers behind them. Let's break down each step.

Connection

First question: Imagine this scenario, you and your friend are working for two secret organizations. You live in different dormitories. The dorm manager assigns you a new room every week. You are only to contact with people outside the dorm through the dorm manager. All of these things are for your safety's sake. How can you and your friend send mails to each other?
P/s: You can have a meet-up with your friend to prepare a plan before working for those organizations.

This is actually the situation where we are facing. Our devices' IP (address on Internet) are concealed and updated regularly by the routers.
Both devices must know the other's address for further communication.

But aren't them on the same network? They can just directly connect without router interference.
You get the point here. In that case, dorm A and dorm B is indeed the same dorm. But we still need to deal with the "rearrange" part. You shouldn't just run into every room and ask "hey, are my friend here?" with the hope that your friend is here. And in reality, you can't do that with routers.

Many habit-level applications solve this issue by manually informing the IP address of one device to the other. But automation is one of our goal so this solution is not viable.

But I am able to send messages from my PC to Android through Messengers. So there must be a way, right?
Absolutely. That is the same technique Syncthing applying.

All we need is a middle man, like a post office, or just the woman who sells drinks in front of your dorminory.
Both of you know exactly where they are. You can simply tell them "if anyone want to find you, tell them I'm at room ABC" after the room rearrangement.
Your friend goes there, asks where you are, and tells them the same thing.
Voila, problems solve.

Wait, wait. Shouldn't we contact with outsider through the manager, does he even allow that type of messages?
Great observation. That where the "p/s" shines. You and your friends must prepare some kind of cipher that only both of you understand beforehand. Indeed, in the world of Internet, this's called "protocol".

However, that's not the end of this nuisance.
The dorm manager also has his own room mapping.
You live in room A, but he will send your mail from room X, and only give you mails sent to room X.

Since that's the case, we have to take an additional step:
Send a mail to the middle man to know our "proxy" room.
The full process is as follows:

Syncthing already has its own middle man.
If you don't trust them, you can setup your own middle man.

I trust them.

Scanning

Before synchronizing, we must know what we already have. That's what scanning does.
"Master yourself, master the enemy".

There are two types of scanning:

• Interval full scanning: once per hour.
• On-update scanning: whenever a file is created, updated or deleted.

One clever optimization from Syncthing team, they do not rescan the whole folder at once but randomly rescan each file at a given time window. That reduces the CPU workload magnificently.

After scanning, all the information such as filename, modified date, size are stored in a hidden table (invisible by default).

This should be enough, don't you think so?
I thought the same thing, but then astonished by the Syncthing team.
They take a step further by annotating which part of a file has changed. A file is split into multiple chunks. Then, each chunk is given an ID (you can search more about hashing).
When synching later, they just send the updated chunk over the Internet.

Not only that, they also assign a version for each file.
I questioned myself why they do that? Isn't modified time enough?
The answer lies in the next section: Comparing.

Comparing

When two devices are connected, they start comparing the scanning tables.
This process is also started on every file changes, too.

Given two scanning tables, how do you compare them?
Or more specifically, how do you know which files to delete, create or update?

Given device A has file X and device B doesn't have it. How do Syncthing know that file is deleted or created?
They don't mention it in the documentation, but I think a deleted file is updated with the deleted time and size 0 in the table.

How about rename? Does Syncthing know that it should rename instead of deleting and creating a new file?
Honestly, I don't know. They don't mention this case either. But I believe they handle this case efficiently, maybe add another column named "old name" to the scanning table.

Now, let's proceed to the most interesting case: update.
Just compare the modified date, right. Where is the interesting part?
Consider this case, you have file X in both devices. You change that file on laptop. Then you fly to another country and your Android changes the time. You update that file on Android because you don't bring your laptop.
Now you have finished your trip, go back to your home, open your laptop, wait for the file on laptop to synchronize with one in Android.
Tada, the file on Android rolls back to the same as in laptop.

Because you updated the file in another timezone, the modified time might not be suitable anymore.
That's why Syncthing performs versioning on each files, like mentioned in the previous section. This can be a number that is monotonic increase.

When comparing, the versions of a file from all versions will be compared, then one global version is chosen (usually the highest one).

If multiple global versions are found, like the example above, users must manually choose what version they want to keep (objecitve 2).

At the end of the comparing process, all files are marked with respective global versions, along with their local versions. Then, synchronization happens.

Synchronization

With that much preparation from scanning and comparing, the synching process can work smoothly now.

When the local version and the global version of a file mismatch, the application compares all chunk IDs of the local file and those of the global file (from the target device). Subsequently, the discrepancy chunks are transferred from the global file to the local one.

That's what happens inside Synchthing. It may take a few minutes to read but the Synchthing team has spend months (or even years) to make it accesible to us.
Let's take a moment to commend their work.

Connection (Part 2)

Everything works flawlessly, the idea is splendid, why do we need this section?
Maybe it works ... but not for me.

Somehow my devices can't find each other. To tell the truth, I don't know the reason.
This seems to be a popular issue, you can easily find many dicussion about this (link or link)

I need to find a more reliable solution. Fortunately, I just need to make my devices visible to each other, all the synchronization are already handled by Syncthing.

That's when I discover Tailscale, a free and open-source application dedicated to connecting devices.

Why does that solve my problem?
To be frank, I don't understand what the problem actually is. I just try and it works.
After reading the Tailscale post, I notice one thing that Syncthing does not mention. Those might be what Syncthing is missing.

I will make use of the dorm analogy again for easier understanding.

The problem is:
The dorm manager does not allow anything sent to your friend, if he hasn't sent anything to you first.
What an annoying man! That means neither you nor your friend can send first.
So... we're stuck? But Tailscale works, they must have some tricks in their hand, right?

Just as you figure, Tailscale does a thing called "NAT hole punching". In this case, after you and your friend have the address, both of you just repeatively send mails to each other. The first time fails, but from the second time onward, it does the trick.

The real challenge here is, the acceptance window is really small. They don't mention the details but I think it only lasts a few seconds.

Conclusion

All the pieces are gathered, I can enjoy note-taking everywhere from now on.
This blog is not as long as the first one, but I also learn a lot from this journey.
Hope you find those knowledge helpful sometime in the future.

Thanks for reading.