Basecamp's house of cards...

Photo by Sigmund on Unsplash

Character is like toothpaste - it really only comes out under pressure (unknown)

I don’t know where I first heard the above quote, but the events of the last week have really shown how true this saying is.

I will start by saying that I have been an interested (but not rabid) fan of the journey that Jason Fried and DHH have been, building their startup called Basecamp. I started on my journey of building online SaaS system 20 years ago, when they first started the company. I had a further 20 years more experience of running a software consulting business before that, so I was intrigued, and a little surprised that two young lads could start an online software business that just seemed to go gangbusters out of the gate.

I guess that was the main thing that caught my attention - that it seemed so effortless, and looked like revenues just snowballed from day 1. Heck they even said that they were earning revenue before they had a chance to build in a billing and payment interface into the system! Something that I had never experienced at that stage.

As a developer myself, I admired DHH’s work with building the Rails framework. Though even back then, I found the framework to be a little too opinionated - much like the founders themselves. I grew to love the Ruby language, but not the Rails framework. Indeed my current startup, which is successful beyond my dreams, is built using Ruby (but on a different framework).

I used to read Jason and David’s blog posts, and over the years, I even bought a few of their books. However I always read their advice with a touch of reservation. They always struck me as being the guys that fell into a pond and came up with a Koi carp in their mouth and went “Oh, hey, let’s tell everyone else how to be great Koi carp fishermen like ourselves”.

Don’t read me wrong - I did respect a lot of the things they did, and how they rattled some of the embedded habits of running a business, but somehow, it all seemed to come from a charmed existence, devoid of any real challenges or problems that required some really creative problem solving.

As far as I know, the never had to deal with situations like:

• Having to make payroll for the team with $0 in the bank

• Having an employee go rogue and either embezzle funds from the company or conduct systemic abuse of another employee over time

• Having a founder/partnership separation that was acrimonious or came at the wrong time

• Having to deal with data loss or accidental customer data deletion due to bad internal practices

I have had to deal with nearly all the above in my 40 odd years of running former businesses. Some of them more than once. I wanted advice that came from the painful viewpoint of having experienced this and and had you staring at the possibility of the end of your company as a consequence. But when you are busy raking in the cash as Jason and David were, it is hard to listed to the “Oh, just do as we do and you will be just as successful” mantra.

The current events at the company have shown the fragility of the foundations that have brought the company to these dizzying heights. Fundamental issues with employee management have been ignored and seemingly plastered over with benefits and reinforcement of “You are lucky to be working at such a great company and being lead by such great leaders!”.

It seems though, that when push came to shove, and employees expressed dissatisfaction with a number of policies and people within the company, it all became too hard, and instead of confronting the problem head on and dealing with the issue, the decision to avoid it and offer band aid solutions was put forward. This seems not to have worked this time around.

Indeed, it seemed that at the ultimate “all hands” meeting, one founder put his normally opinionated personality into hiding and said he would offer no opinions, and the other founder attended from his bed, turned his video off and muted his mic and seemingly posted a slew of anti-Apple retweets during probably what was the most important meeting of his company’s existence.

Basecamp is a wounded beast now, and it will take some time to recover. I genuinely hope that they find their feet again, and that they learn from this, and that their company sticks around for another few decades. I hope that the employee’s who left find gainful and happy employment elsewhere, and I hope that Jason and David approach any future “lessons” they they wish to impart with a lot more humility. I also hope they also see fit to step down from the pulpit every now and then and ask for advice from those that came before them, rather than roll over every business legacy with their own thoughts of how things should be done.

How to conduct a good demo of your SaaS

Photo by Chris Montgomery on Unsplash

Frequently, when we finish a demo session of our HR app, customers will tell us “Wow, that was a great demo!” when signing off. I never really thought about it, other than the fact that I usually had a good time showing them around, and that I was glad they enjoyed it too.

But then this past week, I have been on the opposite end of the stick, as my co-founder and I have been looking around for a new CRM and email campaign system for our SaaS product. We’ve had demos with quite a few vendors now. Some of them quite huge mega companies, and one or two of them who happen to be the darlings of the startup world who are talked up a fair bit on social media.

Our findings? Almost all the demos we have done have been mediocre or terrible experiences. Really. It has been quite a surprise and a bit of a wake up experience. For instance, I am currently typing this while waiting for our third ‘demo’ meeting with one particular vendor. The first was cancelled due to a laptop problem with the sales rep who was to give us the demo. The second was just a meeting with a junior employee who simply asked us a series of qualifying questions (that we had already emailed them before) and couldn’t actually answer our own questions about their product, and this third appointment coming up, I assume we will finally actually get to see what the app looks like finally. And this is a huge company with millions of users worldwide.

Our experiences basically boil down to:

• We’ve been given demos by nervous, hesitant people who don’t seem to know their own product very well

• People giving demos seems surprised, or put out that we ask a lot of questions and pose scenarios to them

• We seem to be treated as an intrusion to most vendors, and they seem to be keen to work out if we are ‘serious’ before they will commit any time to us

All the above is basically the antithesis of what we try and do in our own company.

Our own sales cycle is very heavily demo based. Almost all our new customers generally do a demo session with us before buying, so I understand the importance of this crucial step in building our customer base.

Are our demos particularly fantastic, and do we have a set ‘playbook’ for how we run things? Absolutely not. In fact, I think there are several ways for us to improve (which I outline below), but I was surprised to see that our demo sessions are apparently well above the standards of companies that have received $millions in funding. (As of writing, we have received $0 in funding).

But what IS our philosophy when doing demos? Well, I have a simple set of rules that I always follow, and that I encourage anyone on our sales and customer success teams to do as well. Here they are:

• This is all about the CUSTOMER. The fact that someone wants to see around something I wrote is totally exciting to me, and I cannot wait to show them what we have. No bored monotones here, or rushing through. I am as excited as a new dad showing off my baby - no matter how disinterested YOU may be!

• This is all about the CUSTOMER. They have come to us because they have a problem. Rather than jumping straight into screens, I spend at least the first 10 minutes of the demo time asking them about their business, and what challenges they are facing, so I can best tailor the demo to suit them. Note: We do this at the same demo session. No separate sessions just to ask them questions and wasting their energy to try and schedule a follow up demo time. We try and ask pre-qualification questions in our online demo booking form.

• This is all about the customer. We have customers in over 70 countries, so not everyone is going to work during my 8 to 5 work day. Someone wants a demo at 6am my time, or at 10pm, I will usually arrange things so I can do it. Not all the time, of course, because that would be nuts, but where possible, we try and accomodate them when they are free. Now that we have a larger sales team spread across the world, I don’t have to work crazy hours any more, but I will do so if needed from time to time.

• A demo is not a short lesson. We don’t go through step by steps, but rather focus on the results they can get. For instance, if someone asks us about onboarding checklists, we don’t go through how to set up a checklist from scratch, instead, we jump to an already sent checklist in our demo company to show what it looks like from the employee’s perspective, and then how it looks like to the admin manager when the employee completes it. There are a dozen things in between all of that, but we don’t go into it in detail, but rather skim over it. Benefits over features. Sell the sizzle, not the sausage.

• We do have a ‘standard patter’ of how we walk through the system when showing it off, but that is not a fixed thing and anyone can throw that out the window completely depending on customer feedback and responses during the demo session. It is simply a guideline to set things in motion and eliminate ‘quite spots’, but it is not set in stone.

• We (our sales and customer success teams) sit in on each other’s demos every now and then so that we can all learn from each other, and improve our presentation style or standard patter. I originally set the demo sequence years ago, and my team uses that to a large degree, but recently I asked to sit in on other sessions because I thought that my flow was obsolete, and there could be better ways I could to it myself based on everyone else’s experience.

• We always pause for questions, and if there are non forthcoming, we ask leading questions to eke it out of our customers. Questions are good, and means that they are processing what we are saying and trying to apply it to how they work in their own companies. If they are silent, we try and ask things like “How do you go about recruiting at Acme Co. right now?” or “How do you onboard new hires at the moment?” to try and work out if they have problems that we may have a solution to.

• We never over promise. I was guilty of always saying “Oh, we haven’t got that feature right now, but we can add it next month for sure”. Well, sometimes we can, but often times we cannot, so I have learned not to give unrealistic expectations. This also goes if a customers asks us for something that we have NO plans to build. We never lead them on. We simply state that this isn’t in the larger product vision and then we even recommend competitors for them to talk to. My philosophy here is “If my customer is happy using my product, then I am happy. If my customer is happier using a competitor’s product rather than sad using my product, then I am happy”.

• Be honest. Our system is large and complex and ever changing. Heck, I wrote 99% of our app and even I don’t know if we can do certain things or have a particular feature quite often. Nothing wrong with saying “I don’t know the answer to that, but I can check and get back to you”, but do it with confidence. Nervous looks and hunched body language does not inspire faith in you (and, as a result, your company) at all. Oh, and if you promise to get back to someone with information, please actually get back to them and don’t ignore it or put it in the ‘too hard’ basket and forget about it.

• Respect the customer’s time. They have taken a chunk out of their work day to spend it with you. Do not make them regret that. Actually, this is a problem area for us. Our demo sessions are allotted 45 minutes, but we often go over that, especially if there are a lot of questions. We always pad out the gaps between sessions to allow, and if we see a session is going over the 45 minute mark, we will always pause to ask the customer if they are happy to continue, or if they want to schedule a second session.

• Always follow up. We try to send out a ‘Thank You’ email to the customer within minutes of ending the session, and this means just a simple ‘thanks’ for their time and their questions. We rarely ask for the sale during this follow up email unless they seemed really keen to sign up today. This is all part of being respectful for their time and attention given to us rather than anyone else.

The above is not a definitive list by any stretch, rather, some simple guidelines that we follow here at our startup. It’s not perfect, and we are always changing and updating it to see what works, but I believe it at least contains the essence of what most companies should consider when doing product demos.

Finding and fixing slow queries

Photo by Caspar Camille Rubin on Unsplash

I love dashboards, and I run several on my system to monitor my HR startup in real time. I love collecting data points around subscriptions, user activity, and most importantly, server uptime and performance. Especially now that we are growing to thousands of customers and tens of thousands of employees using our system on a daily basis, I am finding that a lot of things that we built in the early days are starting to show cracks and issues.

Most of this would have been hidden from my knowledge, except for the fact that we have been using AppSignal for our application monitoring for just over a year now, and the AppSignal dashboards and one screen that I keep checking several times a day to ensure that everything is just ticking along fine.

One thing that I have noticed of late though, is the page rendering speed was slowly creeping up. Pages that were taking 1 to 2 seconds to build and render on the server side were now taking 5 to 8 seconds, sometimes 10 seconds plus. This was, of course, unacceptable, and would be severely impacting the user experience for our customers and their employees.

Hunting For The Problem

I was curious as to what was causing these slow downs, and luckily, AppSignal’s fantastic dashboard actually allows me to click on one of these peaks on the chart, and see exactly what was happening at this point, like so:

From this, it will show a breakdown of all the App processes that were running at this particular point in time, making it easy to identity the ‘slow process’:

And furthermore, clicking on the process name above, will show you an ‘execution timeline’, showing all the calls and database activity, along with the execution time for each:

This makes it so easy to see the actual database query that was causing the slowdown in the first place.

This is showing the actual SQL code that is running slowly in this instance.

Identifying The Underlying Cause

So what was the issue? The answer was actually quite easy to deduce by the above database query tracing, and it had a lot to do with how Ruby ORMs (Object Relational Mappers) work. What is an ORM? Well, to simplify the explanation, it is basically a way for Ruby to treat a database store as just another Ruby object, allowing you to manipulate it just like a normal Ruby object without worrying about knowing SQL (Structured Query Language) or any underlying database commands.

To explain it in layman’s terms - lets say you have a database table called ‘Employees’, and you also have another table called ‘Departments’. Now, each employee can belong to exactly one department, so you can set up a relationship between the two tables so that there is a department_id field in the Employees table, which links to the id within the Departments table.

Now, because we have many different companies using our HR system, we need to also have a Company database, and link each Employee, AND each Department to a company by using a company_id field to separate them. Each company can have their set of employees, and each company can also have their own unique set of departments.

Looks a bit complicated right? And handling this in SQL would be a tricky thing, however, the DataMapper Ruby ORM that we use to layer on top of our database actually makes this extremely easy. In fact, to access, for example, all employees in Company ‘Acme Industries’, who are in the Department called ‘Finance’, it would be as easy as executing this:

And it would work straight away. No need to worry about table joining etc. - the ORM takes care of all of that for us, and gives us the results we desire… or does it?

Let’s look at the SQL query that we would normally do to get the same result, using a standard SQL INNER JOIN clause:

A lot more complicated, isn’t it? BUT… it is super fast, and able to pluck out employees from a table with thousands of companies and millions of employees in less than a second. That is the beauty of raw SQL.

However, when I look at the raw SQL being generated by our DataMapper ORM, it was a little different. Now, with SQL, there are many ways to skin a cat, and it looks like DataMapper was taking the easiest pathway. After all, multi table joins are complicated - just look at the query above! What DataMapper was doing was splitting the query into three parts based on the things being asked of it. Here was the resulting code:

It is important to realise that this is actually giving the EXACT SAME result as the raw SQL query above, but it is doing it in a slightly different way.

But why was it running so much slower that it should have? It didn’t really make sense.

But then, we realised it. It was all coming down to Step 2 of the DataMapper query, which was staring at us in the face when we looked at the AppSignal logs.

Fixing The Problem

You see, now that we had thousands upon thousands of customers in the system, almost all our customers had a Department called ‘Finance’ in their companies. So Step 2 was actually returning THOUSANDS of department IDs that Step3 was trying to look for. Of course, most companies only have ONE department called ‘Finance’, so the third step should have only been looking for one company_id and ONE department_id, but it was not. All because we had not specified a qualifier for the Departments table.

Changing the DataMapper query to the following helped to increase the speed of the query quite significantly:

As you can see, we had to add the qualifier:

:company => {:name => “Acme Industries”}

to the Department file as well.

This resulted in the following conversion to SQL:

So while the Ruby method now looks like we are duplicating things a little, it actually works better in terms of database performance.

To show you the difference, here is the AppSignal dashboard for page rendering speed before and after we made the fix:

Look at the drop in average page responses from 4 to 6 seconds, down to 2 seconds or less.

This didn’t stop here. Motivated by these gains, I went ahead and kept looking at later spikes to see if there was anything else on our dashboard that could be optimised. Sure enough, I found a couple of other queries that had the same issue, and I fixed those too:

Most big spikes gone! What a win.

This just goes to show the importance of an application monitoring tools such as AppSignal or any other APM (Application Performance Monitoring) tool for your SaaS. A lot of time, you won’t realise that certain optimisations will be needed when you have a smaller data set. It is only as you grow that you start to see some of these bottlenecks, and a good tool will certainly help narrow down where you should be working on.

Most times, people just upgrade server capacity, or throw more caching solutions in the mix, but all that costs extra money, and if you a bootstrapped startup like us, this sort of fix cost $0. Actually, in terms of AppSignal subscription price, it has pretty much paid for this whole year’s subscription to their service in terms of cost savings from our hosting provider.

Reflections on 2020

I feel kind of conflicted writing this, because 2020 was a horror of a year for most people around the world. Instead, for me, and my family, it was somewhat of a start of new beginnings and some huge wins.

I feel that I have finally stepped out of a decade long black tunnel, and I can finally see the light ahead. You see, back in 2009, I lost my dad, and my wife lost her mum. For us, each losing the parent that we were closest to really took the wind out of our sails, and we both felt that we had lost purpose and a solid goal in life.

I lost interest in my old consulting business, and ended up selling it back in 2015 so that I could focus on launching a brand new startup that meant I could write software that I wanted to write. My wife felt ‘blocked’ in her art, and ended up taking on multiple part time jobs to ensure a steady income for the family while I rebuilt my career (what a crazy thing to do when you turn 50, huh?).

It basically meant a decade of ‘grind’ and living on our savings and on the good will and charity of others. I am forever grateful to family members and close friends who supported us unquestioningly during those difficult days.

But I will be honest and say that because it went on so long, there were days where I despaired that we would never see the light at the end of the tunnel, and that I was just throwing energy into something that wouldn’t last or pan out the way I expected.

People came and went into our lives during that time. Some left us in a better position, but some also left us in a far worse position. After a former co-founder lost interest in my startup and abruptly left in 2018, I ended up in hospital with a stress related heart condition, and I feared that this would mean I could not bounce back again.

But being in these depths was also what forged my inner strength, and bounce back I did. I found a new co-founder and together we built the business up to a far higher level than I ever thought possible. 2020 was the first full calendar year we worked together, and we managed to get to well over a thousand paying customers and we currently manage tens of thousands of employees on our HR SaaS platform.

In some ways, the COVID scare that shut down most of the world was a boon for us, as more people wanted an HR system to manage their suddenly remote or ‘work from home’ teams. For the first time in a long while, I felt that were in the right place at the right time. Well, more accurately, the right place at the wrong time.

I don’t like the idea that we are benefiting from something that is causing so much misery, but I re-frame that as being that we are providing a valuable service to people that want to survive the tough times we are in. In fact, rather than capitalising on this by raising our prices as many mentors have suggested, we instead slashed our prices by half to make it easier for struggling businesses to jump aboard.

My wife managed to go back to just the one part time job that she loves, and she has more time for her art and other creative passions. Our oldest son, who had moved down to Melbourne to pursue his music career also moved back home while the entertainment industry rebuilds itself, so it has been good to have the family back together again.

This week was a milestone, as it marked the first full year of me earning a steady salary from my startup. It was so good not to have to watch our pile of savings diminishing away, and to be able to pay bills using the income earned from something that just started out as an idea in my head all those years ago.

Life is good. 2020 was good, and 2021 will be better.