How I built the backend of YTPlay

This is the story of how I built the backend of YTPlay (a YouTube playlist downloader) despite not having any experience in backend development prior.

Inspiration

So I was trying to download a YouTube Playlist that had about 50-60 videos and all the sites I could find wouldn't allow me download all the videos at once, I'd have to click on each video, go to another page and download it there, so I decided to build my own solution.

Building the frontend

I was able to easily come up with a frontend design for the home page, I wanted it to be as simple and straight to the point as possible, and built it with Next.js, Tailwind, Typescript, and a new UI library I had been experimenting with at the time called shadcn.

Nextjs API routes?

Like I mentioned earlier I had no prior experience in backend development at the time I started working on this project so when I needed to build an API for getting the playlist items data instead of spinning up a node/express server I had to use Nextjs API routes with the help of ChatGPT to handle that.

Problem Faced

At this point, I had built the frontend and backend (sort of) for the project but it could only get download links for videos individually so I had solved the problem but not the one I faced and now I had to figure out how to somehow use the download links to create a zip file and send that to the user.

Solution?

After a bit of back and forth and consuming every tech article written on streaming in Nodejs I was ready, I built a simple prototype of hard-coded arrays with a couple of download links and the way it works is it loops through an array, and makes a request to each download link in the array and pipe's the data to the response being sent to the client It also does this async using Promise.all so you wouldn't have to wait for the first one to respond before the others start. Tried with different links and it was working, Until I checked my logs and saw I was getting a warning from Next.js talking about the size of the data I am sending not being over a certain size and then I had to read more on the difference between an API route built with Nextjs and a dedicated server.

Discovery

So Nextjs API routes are actually invoking these things called serverless functions which can be understood as mini-servers that perform one task, they spin up, perform a single task, and then turn back off, unlike a dedicated server that is always on, the duration of these serverless functions and the size of data they can send back depends on the platform it's being deployed to. So obviously I couldn't use something that can't send data over 4 MB (by default) and will turn off after 60s of being invoked, which means I had to use a dedicated server for this, at least the zip streaming part.

Progress?

I was able to (with the help of stack overflow and GPT) set up a simple express endpoint that replicates what I had on my API route, tested it with the dummy data and it was working but I had one issue, how do I get the necessary data to the server?
The way it worked before was with hard-coded data so it was available on the server and I needed to get the actual playlist data because I needed the download links generated to stream the zip file with the videos.

I actually spent a very long time on this (maybe 3 weeks+) my original plan was to get the data to the frontend and pass it to the URL param so I'll serialize the entire array of download links and pass it to /download-zip?data=%5B%22hello%20world%22%2C%20%22did%20you%20actually%20parse%20the%20array%20or%20just%20read%20through%20it%2C%20that's%20weird%22%5D but these were download links and for some reason, they were extremely long maybe 1000 characters per URL from what I read online it's best to have URLs under 2000 characters to be safe it will work on all browsers and older versions of browsers and also I didn't like the idea of parsing the entire (200 - 600 kb) response to get the download links and then append that to the URL so I was searching for another way, I also considered sending a POST request but to start a download you click on a link to a server that responds with a file and your browser downloads it, which means POST requests were off the table. After a bit of not figuring out how to do this, I decided to work on other features as well as other projects and during this time I saw this video by Theo talking about ratelimiting with Redis and upstash. In the video, he talked about Redis and how it was mainly used for caching responses and I thought to myself "Hmm, if this project grows to have a large number of users, it would be a good strategy to have cached the data from my API routes" (I know trying to optimize a dysfunctional product).

Solution

So I set up caching just so that immediately after the data is available to be sent it gets stored in my Redis cache, each Playlist is identified using its playlist ID and that's when it hit me, on my Backend I can just get the data from that cache and repeat the earlier process generating the stream.