HOME02 - My Smart Home's Cloud integration

HOME02 - My Smart Home's Cloud integration


13 min read

Notice: this is a brain dump. it means i was writing this as i was thinking it out. there's a story to it and details can be changing throughout

This will be a multi-part story because I just came up with the ideas and need to note them down before they evaporate into "reality". As I said before, there will be a Raspberry Pi running all kinds of fun stuff. The amount of stuff is growing day by day as I discover use-cases I was not able to do before due to lack of knowledge or because it didn't make sense at the time. Now however, I have the knowledge and the apartment to create a great system - as I do so well.

Intro to Tripper

Tripper (the name of my home server, yes I'm treating it as a pet lol) is currently running Plex, PiHole, Transmission and Home Assistant, so it's already pretty crowded in the Docker host network. I had to use the host network because my VPN connection was erroring bridge network mode and I'm not a Docker pro. If you know the solution, please let me know lol. The next idea that just came to me and inspired this blog post is storage and backup of all my data, and boy do I have a lot of it in a lot of places. Firstly, Time Machine backups for both my Macs. These will require a networked TM compatible solution also packaged in Docker. I remember something like this existing so there's definitely a solution. If not, I'll just make my own networked Time Machine drive with blackjack and hookers and put it on GitHub.

My Data

I also have two external hard drives in an "offline RAID 1" configuration that store all my pictures, videos and Co. from my camera and videos I've created throughout my life. The size is about ~500GB but there's a lot of RAW stuff that can be deleted, which is on the to-do list. These are critical pieces of data and can not be lost under any circumstances, this is why I clone the contents to the two now as well, which is still not too safe but better than a single one.

The Cloud

This is where the Cloud comes in, as all this data will be backed up to S3. There will be a local copy using one of the hard drives connected to the Pi (or later upgrading to a tiny PC local server) and its contents being constantly synced. This is not the only way the Cloud will be integrated into the Smart Home, but this will be the first. As "boring" as this might sound, it actually helps me a ton because I won't have to worry about losing this data ever. It is ultra-cheap storage that is so durable it can survive a nuclear attack. I do not expect to be retrieving these files too often so the storage class for these will be Glacier Deep Archive. In case I do need to restore some files from this backup, the "bulk" retrieval costs only a tiny amount of money, so in the whole backup process the outbound data transfer from AWS will be costing the most, which is pretty funny if you ask me. I also have a good amount of family pictures and videos that will get the same treatment.

Backing Up

My philosophy is; have personal data you don't want to lose stored in two places. Having a home NAS with any kind of RAID configuration is good, but what happens when a fire bursts out? Unfortunately it does happen, but a lightning bolt can even travel down the power socket and brick all of those hard drives. Having two external hard drives in two separate physical locations is not bad but cumbersome to manage and keep up to date, also they do break easier than 3.5" drives. The point is to minimize the chances that both copies of the data disappears at the same time. In the case of my Time Machine backups onto Tripper, backing up to S3 would be pretty redundant and needlessly expensive. The chances of my computers and my home server losing the data at the same time is very slim, and I have iCloud enabled as well on all my devices. The Time Machine backups will be stored on the attached storage to my Pi and they won't be backed up anywhere from there on out.

Not trusting iCloud

I am to this day, against all my love for Apple, very fucking frustrated at how shit iCloud can be at certain things. For example you store all your photos and videos in Photos, which is set to store everything in iCloud and download as you are bringing it up on your phone. This sounds amazing, except that it's unusable because Apple datacenters' outbound speed is literally at 300KB/s. It can take more time to download a video than to watch it, for example. Now, all my photos since I've been using an iPhone are only in there, which is more than the past 6 years of my life. Basically ever since I've been an adult. I have it set to store everything on my devices, because I have space for it, but I don't like carrying ~60GB of this one big bunch of data around. I would so love to just store it in iCloud and be able to use it on a good connection, but no, Apple still hasn't made this usable. But this woudln't solve my problem anyways, because what I'm afraid of is my Apple account. If that ever gets lost - which is unlikely but still - then I'll lose all my data completely. I know there would be the pictures still downloaded on my devices, but still. It's a risk I am not sleeping well because of. Now my thought was S3 for this as well and I think that's where I'll end up. There is an app called PhotoSync which can sync Photos to S3 continuously. I haven't explored it yet however, but I sure have high hopes for it.

Existing Home NAS solutions

I experimented in the past with NextCloud, SeaFile and other home NAS software solutions for the Raspberry but none of them I trust. I don't like the WebUI parts of it, I don't like it just being hella slow all the time, I don't like the "pet-ness" of it all. None of them can be run in Docker as easily as it should be. It feels like taking a Windows 2003 server from an on-premise DC and putting it on AWS in ECS. Sure you went Docker, but are you really a containering tho? But maybe it's just me, which might be the case but I tried so many times I would be surprised it if wouldn't be true.

The confusion

So I am not sure about the networked storage topic. I don't really need it, but it would be great as a replacement for plugging in my external hard drives for misc. files for example. It would also be much easier to maintain and keep for a longer period of time since I do format my hard drives at least every X years because things happen. A networked storage would just be more stable and reachable than the current solution, and with 256GB super fast USB-C USB drives, taking stuff with me isn't going to be an issue either. Worst case I'll VPN into my home network and copy it over the internet when I need something off of it. It has been a long time since I came in contact with networked storage on MacOS however, it will take a bit of time to figure out what works well. (If you have any knowledge on it, please let me know.)

The realization

Now that I think about it, all of the data will have to be accessible over their own network share lmao. Yeah, if the external hard drives are connected to the server, they can only be accessible through a network share, and it's also a great way to access them. Okay, it feels like my thoughts just connected the storage part of the server into a big circle haha.

Closing Words

In essence, everything storage related, except for my Time Machine backups, will be shared as a network drive and its contents backed up to S3 into various storage classes. The local storage will be my current external hard drives bit this will be updated in case I switch from the Pi to a PC as a server, which is very likely in the near future (ideas on which tiny PC to get?). The applications handling all this will - as usual - be dockerized.

The Plug

I have been working at Nordcloud for a couple months now. I have to say I am very happy that I joined. I am able to learn so many things on the daily and our internal skills are amazing. We're growing very rapidly tho, so if you are interested in joining make sure to check out our open positions, of which a lot is EU-remote😉

Almost forgot!

I GOT A HOMEPOD MINI TODAY! It is almost as small as my fist but the sound is amazing. I'm just starting to set up the "Hey Sir, turn on the TV" command. Also did I say I got a TV? Totally forgot. It's a 65" 4K Samsung one, with all the bells and whistles. The projector idea went out the window because the lighting is not fitting for it in the living room and the lack of HDMI CEC was a real deal breaker. Apple TV (now 4K) is still on the table tho, that's staying.