What is software latency? Latency is how long it takes for a webpage to load after clicking a button, or for a video to start playing after clicking play. Latency is like a chronic illness; most of the time, people don't notice it, or there's a slight delay that doesn't significantly impact their lives. Therefore, in the eyes of software engineers, latency isn't really a bug. After all, the software has done its job and met the client's needs. No one is complaining, so engineers continue to update the software and add new features.
Latencies are like a chronic illness; they slowly become a problem, but no one can say for sure when they'll turn into one. If you click on a webpage and a new page only appears a second later, is that a problem? If a video takes five seconds to load, is that a bug? Engineers would say: if it takes a second for the webpage to load, we can add a one-second transition animation; if the video takes five seconds, we can add a progress bar. Just like with chronic illnesses, if you're feeling unwell, walk less; if you're short of breath, climb fewer stairs. Solutions are devised by humans; with a little ingenuity, the software still runs, just slower.
As the number of software users grew, website data increased, and the software became increasingly complex and slow, one day everyone suddenly realized it was now a bug! It wasn't a logical bug, but a performance bug. This kind of bug is terrifying because once we decide it's a bug, all of a sudden the software is riddled with bugs. It's like suddenly being diagnosed with high blood pressure; many of our daily habits become questionable. For example, if today we decide that a user wait of more than 10 seconds is a performance bug, then everything from the website front-end to the data back-end that contributes to those 10 seconds could be a bug. That's right, software that worked fine yesterday is filled with bugs today.
Performance bugs aren't just a continuous chain of problems from front-end to back-end; they can also be like bugs within bugs, peeling an onion layer after layer, until you're left in tears and utterly frustrated. For example, a user logging into a website takes 15 seconds. Engineers check the system logs and find that searching for the user's account takes 10 seconds. So, after several days of improvements, they miraculously reduce the time to 0 seconds. So do users login only take 5 seconds now? Wrong! User login still takes 15 seconds. Engineers check the system logs again and discover that while searching for the user's account takes 0 seconds, searching for the user's password still takes 10 seconds. Since users need both their account and password to log in, several days of work yielded no improvement overall. Many software perform parallel computations, and often the system only reports the worst path. Therefore, solving one problem often leads to the discovery of countless other problematic code paths.
The poor engineers, questioned by their manager about the lack of progress after several days, could only stammer that they had discovered new bugs. The manager was not happy: the progress is so slow with bugs being discovered one after another! Therefore, improving software latency is an unpopular task. On one hand, it's not like designing entirely new features where you can freely design and feel a sense of accomplishment upon completion; on the other hand, it's difficult to report progress to your managers or tech leads, since the extent of improvement and the time required are hard to predict.
...like a cooking process, with each engineer taking turns adding their favorite ingredients, stirring it up, hoping it wouldn't explode
I once worked in a team where, due to numerous client demands and deadline pressures, new features were often released one by one without considering the optimal algorithm. So, the entire software, besides being like a chronic illness, was also like a cooking process, with each engineer taking turns adding their favorite ingredients, stirring it up, hoping it wouldn't explode: until one day, it had a chemical reaction, and the whole thing became inedible.
Later, the task of resolving the problem and starting from scratch fell to me. I had to work day and night fixing bugs while also writing documentation to explain to managers why progress was slow and why the bugs kept multiplying. It took a total of six months to reduce the software latency from six minutes (you read that right, six minutes) to 20 seconds. Could it be improved? Possibly, but that's the question for tomorrow when the latency creeps back up...