TopHN running in a side window so I can see news bubbling up and down in real time while I work away... (Yes, what you see is some actual code from the project - don't laugh!).
Over the past couple of years, I have tried to push my ageing brain constantly, and one of the best ways I've found to do that is to try and learn a new programming language, framework or methodology every month or so, just to keep the skills sharp.
I've always had a love/hate relationship with NoSQL databases, having cut my teeth for many decades on pure SQL systems, so I wanted to get my hands dirty with that. I've also struggled a little bit to get to grips with Javascript front end frameworks, and wanted to improve my skill sets in that area.
So this past weekend, I decided to get 'down and dirty' with Vue.js as well as RethinkDB. There is a lot of good natured banter amongst programmers about React vs Vue vs Angular etc. and I wanted to see for myself which one would suit my programming style better. I had already done a lot of work in Angular v1 with my mobile app development (using Cordova and Ionic), and wanted to see if Angular v2 and the other frameworks I mentioned would be an easy transition.
Long story short, I had a bit of trouble getting my head around Angular v2, as well as React. At the end of the day, Vue.js just seemed more natural, and possibly closer to Angular v1 to me, and I found myself being able to understand concepts and start knocking together a basic app within short order.
RethinkDB has also been in the news lately, with their parent company shutting down, although the database itself looks like it will live on as open source. I've always liked the look of the RethinkDB management console, as well as the ease of installation on various platforms, so I decided to install it on my development Mac and give it a go.
The Project
The big question is - what to build? I wanted to build something actually useful, instead of just another throwaway project. Then, one day last week while I was browsing around Hacker News, it hit me.
Now, I love browsing Hacker News, and catching up with the latest tech articles, but one of the things that I found myself repeatedly doing was (a) refreshing the main 'Top News' screen every few minutes to see what people were talking about, and what had made its way to the Top 30, and (b) checking the messages that I had personally posted recently, to see if there were any replies to them, and (c) constantly checking my Karma balance on the top of the screen to see if there had been a mass of up or downvotes to anything I had posted.
These three things seemed to be my primary activities on the site (apart from reading articles), so I decided to see if I could build a little side project to make it easier. So TopHN was born!
What is TopHN in a nutshell? Well, it is basically a real time display of top news activity on your web screen. To be fair, there are already a LOT of other Hacker News real time feeds available out there, many which are far better than mine - but I wanted my solution to be very specific. Most of the others display comments and other details, but I wanted my solution to be just a 'dashboard' style view of the top, important stuff that was relevant to me (and hopefully most other users too).
First things first, I decided to take a look at the HackerNews API. I was excited to see that this was based on Google's Firebase. I had used Firebase in a couple of mobile programming jobs 2 years ago, and really loved the asynchronous 'push' system they used to publish changes. I debated whether to use the Firebase feed directly, but decided that No, because I was going to be doing some other manipulation and polling of data, that I didn't want to clutter up the Firebase feed directly with more poll requests, but instead would try and replicate the HN data set in RethinkDB.
So I went ahead and set up a dedicated RethinkDB server in the cloud. This was a piece of cake following their instructions. One the same server, I built a small Node.js app (only about 30 lines of code), whose sole purpose was to listen to the HN API feed from Firebase, and grab the current data and save a snapshot of them in my RethinkDB database.
Hacker News actually publishes some really cool feeds - every 30 seconds or so, a list of the top 500 articles are pushed out to the world as a JSON string. Also, they have a dedicated feed which pushes out a list of changes made every 20 to 30 seconds. This includes a list of article and comment ids that have been changed or entered in their system, as well as the user ids of any users who had changed their status (i.e. made profile changes, or had their karma increased/decreased by someone, or posted a comment etc.).
I decided to use these two feeds as the basis for building my replicated data set. Every time the 'Top 500' feed would be pushed out, I would grab the id's of the articles, have a quick look in RethinkDB to see if they already existed, and if they didn't, I would go and ask for the missing articles individually, and plop those in RethinkDB. After a few days of doing this, I ended up with tens of thousands of articles in my database.
I would also sniff out the 'changes' feed, and scan the articles in there to see if I already had them, and copy them if not. Same with the users. Every time a user was mentioned in the 'changes' feed, I would grab their updated profile and save in RethinkDB.
The screenshot above shows the RethinkDB management console, a really cool tool for checking server performance, as well as testing queries and managing data tables and shards.
So far so good. The replicated database was filling up with data every few seconds. Now, the question was - What to do with it?
I was excited to see that RethinkDB also had a 'changes()' feature, which would publish data changes as they happened. But unlike the Firebase tools, these weren't client side only tools, and needed some sort of server platform to engage the features. So what I decided on, was to use another Node.js app as the server back end, and use Vue.js as the front end for the interface elements.
I would also need to build a connection between the two using socket.io. I was a bit disappointed that there didn't seem to be any native way to push/pull the changes from server to client without it, but hey - we are all about learning new things, and building a socket driven app was certainly something I hadn't done before (at least not from scratch).
So, end of the day, this second Node.js app would sit on a different server, and and wait for a user to visit the site. Now, users can do a couple of things. They can simply visit the top level URL of the site, and just see the Top 30 feed in real time. And I mean nearly real time. As new articles are published, or they move up and down the Top 30, the page view will bubble them up and down and show the latest scores and comment counters.
If the user elected to enter in their HN username, the page would additionally also display the user's Karma balance in real time, along with a notation for how much it has changed in the last couple of minutes. Nothing like vanity metrics to keep people excited!
Also, if their username is entered, the page will show their last 10 or so comments and stories they published, so they can keep an eye on any responses to comments etc.
The second Node.js server is essentially a push/pull server. It will silently push Top 30 list changes to all web browsers connected to it. AND it will also set up a custom push event handler for any browsers where the user has specified their username. As you can expect, this take a bit of management, and server resources, so I hope I never get to experience the HackerNews 'hug of death' where a bunch of people log on at the same time, because I am not really sure of how far this will scale before it comes to a screaming halt.
The Vue.js components purely sit there and listen for JSON data packets from the server pushes, and then format them accordingly and display them on the web page without having to refresh.
I haven't gone into the nutty details of how I built this on here, but if there is any interest and I get lots of requests, then I am open to publishing some code snippets and going into deeper detail of how I built the various components.
All in all, I am pretty happy with what amounted to around 4 or 5 days of part time coding. I think this is a useful tool, and as you can see from the header image, I tend to have a narrow Chrome window open off to the side so I can keep an eye on news happenings and watch them bubble up and down. The web page is also totally responsive, and should work on most mobile browsers for portability.
Are you a Hacker News member? Why not check out https://tophn.info and let me know what you think?