This is a somewhat-tech focused post but the spirit of the post tells the story that even though you sometimes know what you need to do, taking the time to explore a known (or unknown) promotes new ideas and opportunities.
At Intergreatme we're currently working on our new SME portal. I have been spending some time thinking about a conceptual database design. A lot of the current design work has been focused on the prototype I built in a couple of weeks (using PHP) that was repurposed to work with a larger project between Telkom, Lalela and Angelshack. Having said that, as the solution that is being built goes beyond the current transaction-only approach we have when dealing with RICA transactions, as the core focus of this portal is on FICA.
It's not secret that we use CockroachDB as our database of choice, although I do like to also use SQLite3 for building a quick prototype or analysis work - something I still make use of today (and choose modernc.org's sqlite implementation.)
While I was prototyping some code around the use of blind indexes specifically using SHA256 as the hashing method. This lead me down two different paths of research:
I had previously read up on CipherSweet, a cross-platform library that provdes searchable field level database encryption. We already make use of blind indexes at IGM, but I wondered if there isn't a better way for us to be doing this. It's through some basic searches and prompts to ChatGTP (including CipherSweet's documentation on Blind Indexing) that I also came across the BLAKE2 (and for x64 platforms, BLAKE2b) algo.
Secondly, I started to look at the appropriate data type to use to store this kind of information, and naturally, looked at the CockroachDB docs. We also use UUID's as our unique identifier, generally because the use of a sequentially incrementing ID isn't advised for a clustered database.
Reviewing the documentation on for ID Generation Functions and I noticed there is a ULID function. I've never heard of ULID before, so I figured I'd task ChatGTP to provide me with a summary of what it is as a quick-win way of finding information without needing to search for it.
ULID (specification) is a Universally Unique Lexicographically Sortable Identifier. I'm not going to go in to the properties of what it is, or the differences between UUID and ULID. But what I like about it is that it provides a timestamp component as well as an element of randomness to generate a unique identifier. This is pretty useful in terms of generating unique identifiers that have time as part of the identifier, as we work on an insert-only approach to storing data (which also presents its own challenges when it comes to performance when dealing with millions of records in a table.)
In a second example, I was watching a YouTube video about SvelteKit and Golang when I came across HTMX, which then led me to see Hyperscript (when checking out file upload functionality included in HTMX). This then led me to also find Templ, which is notoriously difficult to locate using search engines, as a different way to build UI's outside of the use of the html/template standard library.
This then led me to find the Hotwire library, which comes from 37signals' and that honestly reminds me of how I would build applications with xajax back in the day.
After some experimentation with HTMX, and the web component library Shoelace.style as well as Hotwire, I have slowly started using Hotwire over HTMX if only because HTMX had some issues with the Shoelace inputs not allowing HTML5 input validation running, as well as not sending the POST data from the form to the server. I'm sure that HTMX does support this, but it doesn't really seem to work "out the box", at least, not with my simple trial and error experiment.
The point?
If I hadn't have taken the time to do some expansive research from a particular topic, I would not have found these other ways of doing. I wasn't specifically looking for BLAKE2, ULID, or even the ID generation functions in CockroachDB. If I wasn't curious about how people are using SvelteKit 2 with Go, I'd never have found HTMX, Hyperscript, Templ and Hotwire. The outcomes were always better than expected: not only did I learn something new, but these are now also possible items for us to discuss and determine if they suit our specific use cases in a way that supports our application design.
As a side note:
Ironically, I have already experimented with building a similar style identifier that provides a timestamp and variable-length random sequence as part of some R&D as part of a link hydration/dehydration strategy. Something else I've learned as part of this process is the use of Crockford's Base32 (used in ULID generation) in which the alphabet excludes the letters I, L, O, and U to avoid confusion and abuse. Of course, for my needs as part of a link hydration/dehyration process, this is largely irrelevant. But still, something learned just from a curious question.
I released the first version of crdbt on 26 October 2022 and since then have only made a couple of minor changes to the solution. I've been using crdbt in production at Intergreatme to help manage the way I update our CockroachDB cluster:
For all intensive purposes, that part of the process works pretty flawlessly.
But as much as that part works, there are a host of other issues with the project which need some TLC. The issues I have with the current code base are around my code structure. It was/is a project to expand my Go skills, after all.
The approach has been simple: create a new Linux VM, install Go (1.21.3), Visual Studio Code, and clone the repo from Github. Working from a clean slate environment has shown me just how lacking (and unhelpful) the tool is when it comes to giving useful feedback to the user. In most cases, there was no feedback: you didn't know if the command had run successfully, or if it had run at all.
I have spent the last two days reorganising the code and doing some hefty refactoring, to the point that ironically the two parts of the application I use the most, are the two parts that are currently broken.
I've removed the previous releases of crdbt, and as soon I have fixed the issues with the update and upgrade commands, I will release a new version.
A lot of focus has gone in to making the possible errors (and error resolution) more clear. I've re-ordered most of the commands to be grouped together. I'm also working on adding in some versioning to crdbt, and want to redesign how and when information gets presented to the user.
Downloading, extracting, and installing is a bit more intelligent. I pay attention to the environment (linux-amd64, linux-arm) to give crdbt support for alternate architectures where supported. As an example, I use linux-amd64 on my desktop at home, but linux-arm on my Mac inside a Linux VM.
Downloading no longer tries to extract if the file is there, it just exits (this was an overcomplication.)
Extracting has been simplified. There's also a spinner for progress (thanks goroutines + channels.)
Moving files around is in its own method. There's better support for prompting for sudo and error handling.
The changes to downloading, extracting and moving files are the reason the update and upgrade commands are currently broken.
The Exec command has changed from Output to RunCombined, providing a more useful output when errors occur.
I need to re-write the list command. It should also take in to consideration pinning (see pin feature below.)
Rather than using a heredoc for the cockroach.service file, I've simply embedded it in the binary, and write it when needed. This could provide more flexibility when trying to run Cockroach in a cluster, vs the current single-node service.
A new install feature is in the works which is around 75% complete. I haven't finalised the process as I am undecided around how interactive the process should be, and how far along the process should go. As in, use command line flags to confirm choices, or prompt the user for them.
Another new feature will be the ability to pin specific versions of CockroachDB. For instance, the current use of the latest command gets, well, the very latest version. But it doesn't take in to consideration that you might be running 21.1.x or 22.2.x which could be quite disasterous to run crdbt upgrade latest (moving from 21.1x to 23.x.x)
By creating a pin feature, it will allow crdbt to pin to a particular version, and only update/upgrade to the latest version in that stream. This in itself creates design concerns: where do I write the configuration file? Single-user, multi-user?
What excites me the most about crdbt is not only how useful it is internally at Intergreatme, but that it gives me the opportunity to learn something new.
|
Updated 28/09/2023 The dust has settled and the savings are over 51% per month. This is a substantial saving of several hundred thousand Rand per year. |
At the begining of this year I gave myself three core objectives at Intergreatme:
Intergreatme has always had a focus on how we protect people's personal information, which is why practically everything is encrypted in our databases. This affords us the concept of "encryption-at-rest", which is a fancy way of saying that the information you give us is encrypted prior to being saved to disk (or database.)
Of course, protecting information goes beyond simply storing information in a secure way, we also need to ensure that the information is retrievable. To do this, the original solution made use of a small Cassandra cluster which was being used as an object storage engine, along with some meta data about the files being stored in PostgreSQL.
Now, we haven't really looked at optimising the storage since the company's inception, largely because some problems you can throw money at, like increasing disk space when necessary. Up until now, it has just been an opportunity cost: either we look to reduce storage to save money, or focus on money-making activities: building new product.
Last year we were thrown a real curve ball by Microsoft Azure in that they gave us notice that they were increasing Azure costs by 15%. Given we have quite the cluster of VM's to ensure we have scalability with our homebrew AI that checks every transaction, along with a cluster of CockroachDB and Cassandra boxes... this naturally became a significant price concern.
I worked with the Dev team to look through the resources we have available to us, and we saw some low-hanging fruit that allowed us to reduce the size of some VM's and optimise disk space. These initial changes were enough to negate the 15% increase in Azure costs and only took about a month for us to realise these savings*.
In January the Intergreatme mobile app suddenly stopped working for iOS users. I was again at a point where there was this opportunity cost thing lurching out at me: spend time focusing on trying to get the iOS app working again, or focus on product? More importantly, I also did a revenue analysis of the app vs. spend vs. estimated time to fix (and the cost of labour associated there) and the result to me was clear: discontinue the app.
I put forward a recommendation to the Board, along with a detailed proposal of what we would do technologically, along with how we would communicate these changes to our app users, while still complying with POPIA. After about a month of development, we released our comms to users and provided them with the ability to download their data, delete it, and obviously opt-out of any further communication.
With this new process, it was decided that we would not store the information in Cassandra, but rather in blob storage. Once complete, we decomissioned the various servers and databases associated with the app. Sadly, this gave us about an 8% reduction in Azure costs (though cumulatively by this time, we've saved 23% on spend.)
I was not quite satisfied with our spend, and started looking at ways to further reduce our spend. Our two main cost areas are storage, followed by virtual machines. And storage is expensive because we have several clusters of machines storing the same encrypted information, which inflates the size of each item we store. So I looked in to the cost of using a different model for storing our documents. I looked at three main options: Amazon S3 buckets, Cloudflare R2, and Azure Blob storage.
Given we already use Azure (for now), it made sense to use Azure Blob storage. Objects are first loaded in to hot storage as these objects are required frequently for AI to analyse documents, as well as being available to our verification portal and customer insight platform (not to mention Enterprise clients that decide to get a copy of these documents.)
Azure provides a way to move these objects to cold storage automatically, so after a month, they are automatically moved, resulting in a further cost saving.
Fortunately, we only have a few API's surrounding file access (upload or download), so there wasn't too much to change in converting from Cassandra to Blob storage. Every object is still encrypted by these services, and I didn't want us to change too much in the system to prevent side-effects, so we just kept using our same encryption methodology even though most of the modern storage options now also automatically encrypt data being stored to them.
The next phase involved stopping new images from being saved to Cassandra, and instead saving them to Blob storage. Finally, we needed to move historic items from Cassandra to Blob storage - an effort that took several days to complete I might add (it's terrabytes of data.)
This migration of data resulted in an obvious cost increase for that period with the influx of data in to the blob storage, but this would only be a temporary cost as it is a once-off exercise. Once shifted to Azure Blob storage, the team started to decomission Cassandra, remove data disks that were no longer necessary, and resize virtual machines.
The impact of these changes is clear with our latest invoice reflecting a cost decreased of 29.41% while it ensures our future spend also remains low.
* I would say it is nearly impossible to really gauge the Azure Spend forecasts as it constantly changes, and seems to run from month-to-month without considering what your billing date is. So I've never been able to successfully read the forecasts properly until I receive an invoice.
I recently engaged with the operations team to discuss how we can change the way we work at Intergreatme. Part of this process of exploration is that we have redefined the way in which we roster work for the manual intervention of KYC transactions.
Everyone loved it! Everyone loves having a more structured week-on, week-off approach.
But the concern raised by the team was: "what can I do during this time to continue to create value for the business".
We recently explored some of the additional ways that they can provide value:
Read. For the past few years I've been building up a library of resources that is accessible to the staff. To me, just reading, learning, and upskilling is a massive benefit to the business.
Update the wiki. The employees of the business are the life blood of information. They work with the products and tools the tech team creates on a daily basis. They understand the in's and out's of the technology. Spending time on this is a worthwhile endeavour.
Lean-in. Get involved in other areas of the business. As much as we're one business, there are two functional areas: operations, and development. The best people to test the product are the users of the product. Lean in and be proactive with getting involved with testing.
Simplify. Think about the role you play in the business, and the processes you currently go through. Are there gaps? Are there ways to short-circuit processes?
Think. Sometimes just taking the time to think about where you are and take stock of the situation can provide value as an outcome of that time spent thinking. You'd be amazed at how powerful your subconscious mind is. One of my methods when I get bogged down with code that I am struggling with is to just stop. Go for a walk. Put your subconscious to work and clear your mind of the issue. Generally, after a few minutes thoughts start bubbling up and voila! A solution to the problem arises. You just need to give yourself the space (and time!) to remove yourself from the situation to think about it.
Ask. Questions are a great way to learn. Why do we do something in a particular way? Why does the KYC engine do this particular thing? Why does this page look like it does? But also, a great way for us to change the way we operate as a business. I've been asking a lot of questions to the employees to try and get them to change the way they work.
Experiment. Quite similar to 'ask' above, but being willing to experiment and try new things. Be willing to explore new ways of doing things. Check in to see if it is working. With every experiment, we have a feedback loop to see what went right, what went wrong, and should we go back or move forward. Experimenting takes time. It isn't something that happens every day, but happens over a period of time. Be willing to make experiments, but make sure you measure their effectiveness, and ensure there is team buy-in.
What does "productivity" mean for you?
As a side note, I asked ChatGPT the same thing, and it came up with similar responses:
I recently made use of Afterpay for a few small purchases given my limit. Here is my take on both the platform, and my experience with buy-now-pay-later.
If you don't know what Afterpay is, it is a platform that allows people to make purchases and pay for them in four interest-free instalments. It means they'd make the payment to the vendor (and take any fees associated with it), while I'd then be paying them the instalments (and any penalties for non-payment.)
Starting out, it's easy enough to do. Find an item you want, buy it, and pay it off over a few weeks. It almost seems effortless, and without any anxiety, right?
Well, you would think!
The reality is, my experience using Afterpay has left me never wanting to use it again. While the idea of having four payments come off your card sounds appealing, the reality is I was anxious of every time I came close to needing to do the payment. I would have to login to my bank account, and double-check that there are sufficient funds.
Secondly, once you have more than one purchase coming off on Afterpay, my anxiety levels increased dramatically. How many extra payments do I need on track one that I'm paying off, vs. how many on track two. I can only imagine if I had three or four things on the go with Afterpay... The cycle actually starts to feel never-ending with a stream of different payments coming off at different times during the week based on when you bought something.
It was terrible. I basically never want to do it again.
I had a few other issues with the platform, that I even went to lengths to try and explain and get resolved. So much for "support your customer".
Issue 1:
I wasn't able to confirm my identity up-front, so I deferred adding in those details. However, when I had all my details at hand, I couldn't get to the appropriate part of the app! The app kept routing me to the help page, instead of routing me to the "add your details" page even though that was the button I was pressing!
The support agent was clearly foreign (from the bad English and grammar) and couldn't make heads or tails of my requests, of which there were two:
The agent was able to get the account activated; but thereafter the wheels fell off. I got to the point of sending them a video of the issue. In the end, I just gave up trying to help them resolve a problem in their own UI.
Issue 2:
Afterpay doesn't clearly mention that they only support using Google Wallet. I mean, is it really so difficult to just do things through NFC?
My support agent was also foreign. My original query was whether Google Wallet is the only wallet that can be used (as much as I use an Android-powered device, Google is not my choice of anything.)
The response was that they were sorry my phone didn't support NFC.
So I sent them a follow up... whereby the response was... even further off the mark! The follow-up response was that if I don't have access to the Google Play store, that I should download the APK. I mean, sure Afterpay, tell people to do an insecure way of getting access to your service.
It is no secret that I am both a fan and user of Cockroach DB. At Intergreatme we have it deployed as a clustered database for our main technology stack, but I have also made use of it as a standalone database on some of the smaller servers that I work with (predominately interfacing with PHP.)
That's one of the things I love about Cockroach DB: download an executable, and you're basically ready to go! It doesn't rely on a bunch of hidden files and configuration settings to operate, which simplifies the whole deployment model. Want it to run as a standalone instance? Sure, provide some basic configurations and you're ready to work. You can even deploy it in an insecure mode to make it even easier to get started.
But once Cockroach is running in production, it becomes a bit of a task to update it.
My current process is:
Download the latest version of Cockroach DB
curl https://binaries.cockroachdb.com/cockroach-v22.1.9.linux-amd64.tgz | tar -xz
Stop Cockroach DB (assuming you are using systemd as described in Setting up Cockroach DB)
sudo systemctl stop cockroach
Replace the binary
sudo cp -i cockroach-v22.1.9.linux-amd64/cockroach /usr/local/bin/
If you are using the custom GEOS libraries, move them now:
sudo cp -i cockroach-v22.1.9.linux-amd64/lib/libgeos.so /usr/local/lib/cockroach/
sudo cp -i cockroach-v22.1.9.linux-amd64/lib/libgeos_c.so /usr/local/lib/cockroach/Start Cockroach DB
sudo systemctl start cockroach
Check that everything is still works
sudo systemctl status cockroach
If you are working in a clustered environment, move to the next instance of Cockroach DB and repeat the process above... Or roll back by replacing the installed version with the previous version.
I have only had one instance that refused to upgrade before, and I had to use the awkward backup and restore methods to destroy the previous version of the database and recreate it in a new version.
Making thing easier for myself
As you can imagine, having to search for the resources to remind myself of the commands I need to enter becomes a bit tedious at times, not to mention the problems that can arise if you are working across different terminals and paste in the wrong command in the wrong sequence or order!
To solve for this, I wrote a simple bash script that effectively emulates the process above. I called it crdb-update.sh and it accepts the version as a command line argument to the version of Cockroach DB you want to use. Because of the simple deployment process, and that certain versions of Cockroach DB are by default backwards compatible, the script allows me to easily upgrade/roll back versions of Cockroach DB with nothing more than a simple command by specifying the particular version I want installed.
I then wrote a few other scripts to handle other use cases that I have with Cockroach DB: like easily accessing the SQL command line interface shell, or viewing the systemd status.
What is crdbt all about then?
I wanted something a bit more flexible than having a series of shell scripts to make interfacing with Cockroach DB a bit more elegant. So, I decided on creating crdbt (Cockroach DB Tools): a Go application that combines all the scripts I make use of in to a single application that is easy to deploy, and acts as a more powerful utility.
While the first version of this utility might seem pretty unordinary, I created crdbt around my existing use cases at Intergreatme, and hope that it provides other users of Cockroach DB a useful tool to help manage their database.
The source is located here, and you're able to download a precompiled binary for Linux here.