A Tale of Two Monoliths

How to integrate tech stacks is a big decision

I find myself in a situation where I need to integrate the technology we have at Eight One Books with the internal platform we have built at Eight One Partners. It's an interesting problem that I've been thinking about the past few months and became a more earnest endeavour the past week. Just starting to write this post has given me some insight into how to go about it so I figure I'd finish the post. Hopefully this can help you if/when you encounter a similar situation.

While integrating two tech stacks is not an uncommon situation, how we got here with our technology does feel a bit strange. My business partner and I started off building Eight One Books as a SaaS platform for accounting firms. It cut a number of tasks my partner was doing at his firm from hours to minutes. It allowed him to hire paraprofessionals to do the work at a level of a CPA, which is crucial given the CPA shortage in the US. We thought this would be an easy sell to other firms.

We were very very wrong. We often heard our intended customers tell us "you're going to lower the number of hours I can bill for." It was a punch to the gut. We knew this was going to be an issue, but we assumed people would be happy to re-work their business model. My partner ended up changing his billing to a fixed price every month. This made clients happy and my partner ended up making more per hour than he did when he was hourly. This argument did not sway the people we thought would be our customers.

A couple of startup advisors chastised us. We solved our own problems, but we didn't talk to enough customers to understand what their problems were before we started building. The obvious solution is to go back to the drawing board, talk to other accounting firms, and build something they would actually use.

Except... this exposed the fact that those firms' clients had a problem that needed solving: accountants that didn’t have their best interests in mind. Our pivot wasn't to find new software to build. Our pivot was to focus on building an accounting firm and compete with those firms that we thought were doing their clients a disservice.

By focusing on providing services rather than software, the way we built technology had to change. Eight One Books was designed to have multiple firms, which required a more robust role based access control (RBAC) system than an application that just had one firm's clients. RBAC is still important, but the considerations for a single firm RBAC system are a subset of a multi-firm RBAC system. Eight One Books also had all the trappings of a SaaS: sign up pages, subscriptions, access control on endpoints based on subscriptions, etc. We had held off on integrations as well since many APIs treat external apps differently from internal apps. There is a bit more overhead on building an external app. I once had to argue with a service for weeks over a scope they thought I didn’t need. Yet, if their evaluators had tried using my app for more than 10 seconds, they would have realized how important it was.

The biggest consideration though was that Eight One Books was built to be a cohesive product. Whenever anyone using it had a request, that request had to be framed with the consideration of how other firms would use that feature and respond to that change. Even though we are focusing on building a firm, my partner and I would like the option to go back to a SaaS business when it makes sense so these considerations would still be in play.

Building internal tools requires a different mindset. Prioritizing work is a simple ROI calcuation: number of dollars saved by building something versus number of dollars spend building it. There's no need to worry about whether something fits into a product or how it would be marketed. If we end up with more money in our bank account by building something, it gets built. It doesn't even have to be a fully functioning web app backed by a database. A number of the early tools I built were just Javascript bundles delivered statically. No API calls. No database. They were basically desktop apps, except run from the browser instead of needing to install a desktop app.

There's also a preference for NOT building anything at all. We have purchased software that costs $200 a month that has reduced staffing costs by over $2000. I wish I could do that a hundred more times.

Since the mindset was different, I decided that it would be better if I just started fresh. Make a new repo that just has a collection of all these haphazardly built tools. None of these tools make sense in Eight One Books and they are so bespoke that it would be difficult to think of even a handful of other firms finding them useful.

Eventually, we realized how we needed analytics that were spread across multiple systems that we used. Easy enough. Write some scripts that automatically pulls data from APIs and throw them into a database. Add Metabase on top for easy querying. Problem solved.

Except we started to realize how important some of that data would be to our clients. Example: we have invoice analytics to help us understand how to best collect on outstanding invoices or how to tailor future engagements with various clients. Many of our clients would benefit from those analytics as well.

Metabase has decent RBAC, but not robust enough to allow for multiple clients to have access to that system while making sure they can't view each other's data. Still, that's not a hard thing to build. Take some of the RBAC logic from Eight One Books. Take some of the more useful dashboards from Metabase and recreate them using Plotly. Since this is an internal application, I can also lock access to email addresses or Google SSO accounts from known domains for added security.

The problem is that as soon as we realized we were going to give clients access to the system, I should have immediately gone back to the product mindset instead of staying with the internal tools mindset. I did not and now we have two monolith software applications. That's still fine right?

It was fine... until we found analytics that requires combining the data and logic built into Eight One Books with the data we had in our internal platform (which still doesn't have a name). Eight One Books was built because we did not find any other software that did what we did. That made it unique and we had to use it. There's no option to not integrate our tech stacks. That leaves us with a number of possible options.

1 - The most obvious thing is to open up access to the internal API for Eight One Books to our new platform. The problem is that this gives our staff and clients two places to go to for information. The separation is less based on logic and more just how we landed when we made our pivot. The useability of this model is horrendous.

2 - Copy and paste the code from our new platform into Eight One Books, then take down the new platform (or vice versa). This unifies the application and gives us the option to spin Eight One Books back out as a SaaS if we wanted to. The downsides of copy/pasting are removed since we will end up with one monolith to maintain. However, the reason the new platform was built separately still apply. There are downsides to trying to maintain a cohesive product rather than a set of tools that are useful. Development would slow down significantly.

3 - Include the Eight One Books code base as a git sub-module in the new platform. The new platform would have access to both databases, but it would only access the Eight One Books database with the Eight One Books code. Any changes I would need to make in the Eight One Books code to support our efforts in our internal platform would automatically be pushed out to the Eight One Books server so that both applications would still be viable. For the frontend, I use a variant of Flux and Atomic Design so it is also really easy to just include Eight One Books frontend components in our internal platform's frontend.

That last option seems the most appealing. It gives us the most flexibility from a business perspective while also giving our staff a single place to do all their work. The main problem is that a lot of the multi-firm, subscription, etc code in Eight One Books would be unused for a good while. There are automated tests, but unused code is always a risk even with tests. There would also be a decent amount of refactoring. Multiple paths can be used in PYTHONPATH, but that doesn't help if I have modules named model.client or model.account in both repositories. I would either need to prefix every folder in one of the repos or switch everything to relative paths. This would likely be an easy set of search/replace jobs, but I would be negligient if I didn't test it thoroughly.

There's also the "is it worth it?" consideration. Do we actually need the business flexibility of keeping Eight One Books as it's own SaaS? If we go back to the SaaS business, it would almost certainly look very different than it does now (and would probably require putting in multi-firm logic into the new platform).

When I started writing this post, I realized there was a fourth option. There's no need to treat the backend and frontend of either application as a single unit. Everything that's been built has been build modular. That means I can include Eight One Books as a git sub-module, but only use the frontend components in the new platform. I'd have to modify the API routes a bit, but I could have the web servers for both applications continue to run as is. Adding a second deploy step to deploy to both sets of web servers is easy. It is basically option 3 with less work, but it does still leave the problem with unused code sitting on the Eight One Books server.

As I write this, I like the fourth option more and more. If you have an opinion, I'd love to hear it (though I suspect I'd have probably started work by the time this post gets published).