Time is the only real currency we have
Suggestions for Software Engineers on spending time on things that matter and avoiding things that don't
- A Note on ‘Software Engineering Stuff’
- Stressed And Struggling For Time
- Things That Matter (Time Savers)
- Things That Don’t Matter (Time Sinks)
“Well, that’s the thing right - Time is the only real currency we have”. This was the singularly brilliant summation of a long discussion the company’s founder and I had. It was one of those long walks we occasionally took in the middle of the afternoon - when the weather was still pleasant and there was a lot to talk about. The topic of discussion was Developer Productivity.
We were 1 the fastest growing company in India and had hit a Billion Dollars in valuation with a team of about 20 engineers.
Developer productivity wasn’t just an aspirational goal for us. We needed to be highly productive to even survive the challenges we faced on a daily basis and what lied ahead of us.
I’ve always heard Time == Money, but that never really struck me as vividly as the above statement. I feel it is profound and cuts through all clutter in our head.
Now that I have some more time to reflect on it, I wanted to try and best summarise what I’ve learnt on my journey so far. This is a result of ~8 years of introspection & observations while working in some of the fastest growing companies on the planet . Hopefully you’ll take away something that helps you in daily time management.
A Note on ‘Software Engineering Stuff’
Before we start off, I want to quickly put out a short disclaimer (I consider this important enough to give it it’s own subsection)
All ‘Engineering Stuff’ - from the way we write code, to design choices, to the tools we use come with tradeoffs. This is the only universal law of software development (if there ever was one).
There is no ‘absolute right’. If there were, you and I wouldn’t be needed much.
That applies for this post as well, so don’t come charging towards me with a flame thrower.
But while we don’t have an ‘absolute right’, we are lucky enough to be in a field where logical conversations are generally possible and a question of whether it is raining or not can usually be settled by stepping outside.2
Stressed And Struggling For Time
Off late, when I got talking to a whole bunch of software developers around me, the theme of burn out and never-having-enough-time was surprisingly recurrent (Surprising because, it was the exact same thing I go through as well - I thought I was a special snowflake with the weight of the world boring down upon me. Alas)
When I reflect on it, it seems quite odd.
Think about it - programming these days should be easier. We are a product of decades worth of research and innovation in the field of computer science and software engineering - but most of us work way too hard (or, at least seem to work way too hard) , spend more time than necessary in dealing with bugs and are almost always behind deadlines.
Most of us are working on a significantly higher plane of abstraction - on languages such as Python, Kotlin, Go etc.
We almost never have to worry about memory management (unless you are one of the 8 people on the planet working on improving the garbage collector implementation)
The frameworks of today help us in invariably pointing the gun in the right direction, but we never seem to ‘nail the target’ once and be done with it.
We are always behind our product schedule and we want things to end - but they never do.
My observation, here, is that we end up spending way too much time on things that don’t matter and spending increasingly lesser time on things that do.
Things That Matter (Time Savers)
Learn to type Fast
‘But software engineers spend more time thinking than typing’, I can almost sense you thinking. Agreed.
However, whenever you do choose to grace the keyboard, your fingers must be in a position to do your bidding almost as fast as your brain thinks.
Typing fast is one way to easily get one of the most mundane parts about our jobs become less painful.
You don’t want to be the person who thinks their problem through on a piece of paper, has all the ‘structure ready in the head’ but gets bored halfway through implementing it since it’s a lot to type out and it’s taking a lot of time.
More importantly, learning to type fast will help you interact with your developer environment a lot more efficiently.
If you’re a Jedi, then think of the keyboard as your light saber. Be one with it and you’ll slice through anything that comes in your way.
How to do it? Don’t go out looking for the best keyboard on the market. Just download GnuTypist and do the touch typing course. You’ll find your basic laptop keyboard (butterfly keys or not) is capable of rocket speed with the right training.
TurboCharge your Dev Environment
If you imagine you’re a Jedi3, then your dev environment is the force. You can use it to move mountains by just focussing your energies and leveraging it’s powers.
Master your IDE
I have interacted with many engineers who treat their IDEs as little more than a notepad - using it simply for code navigation, certain auto complete features and use some debugging features.
But modern IDEs have so much more to offer.
Want to rename a method? Change it’s parameters? Add Default values to parameters? All this can be accomplished via a single shortcut.
Want to Toggle Breakpoints? Jump in and out of methods while debugging? Yup, can be done in a single stroke.
How to do it?
Every time you are presented with a task that requires you to do any of the following, spend 5 minutes to check if your IDE has a command for it
- Repeat a change multiple times (like renaming a method everywhere it is being used)
- Switch tabs or windows to get from one place in code to the other (like going back to the method that you edited 10 minutes back)
- Interact with an external tool (say, running your build by switching to the terminal)
- Whenever you are using the mouse - your IDE will have a keyboard shortcut for it
Also, spend 10 minutes a day going through any available tutorial for your IDE. In a month’s time you would have put in a solid 5 hours in learning a tool you use more than 1000 hours a year4.
Master a terminal based text editor
In the age of IDEs, one should not forget the power of the text editor. While most of your coding would happen on an IDE, you’d invariably end up using a terminal based text editor for some part of your work.
It may be while checking log files, or while writing a blog post (like this one) or logging on to a server and direct-editing files there5.
I cannot imagine writing this blog on any other tool apart from my trusted vim. I have markers everywhere (adding a footnote is a keystroke away, for example), move lines from one section to the next in a blink and use the powerful search features to get around easily.
Besides, learning Vim helped me improve how I work on my IDE (Vim Emulation FTW!)
Yup, I use vim. You may use emacs. Here, go look at this, calm your nerves and come back.
There’s a tool for everything
As I mentioned earlier, we are blessed to be at a time and place where we can leverage ideas and hard work of thousands of people who’ve worked before us.
Keep an eye out for things that bother you, think of a probable solution to the problem and chances are that there is already a tool to help you with that.
If you don’t find one, email me and I’ll be happy to help you look for one.
- Using the mouse to copy-paste text from your terminal ? Use Tmux
- Learning something new and tired of switching back-and-forth between browser and IDE? Use i3
- Ever wished you could navigate the browser with the keyboard? Use Vimium
Automate
Here’s an old xkcd post suggesting the RoI of making routine tasks more efficient.
But here’s the kicker - since we mostly work in teams, any effort put in by one person to automate routine tasks saves time for everyone. Going by the above calculation, if one person on a team of 10 engineers spends 3 whole days shaving off even 30 seconds on a task done by everyone only 5 times a day, we would have gotten a complete return on investment over a period of just 6 months. Bring that up in your next sprint planning meet.
People always talk about the prodigious 5X engineer. The reality, though, is a lot more boring than that. 5X engineers, in my opinion, are those that help 5 more developers save some time in daily tasks.
Just like other things, chances are that whatever you’re looking to do, there’s a tool for that. In case you’re having trouble finding one, email me and we can find one together.
Tests
Tests are, unfortunately, one of those contentious ‘Engineering’ topics that I mentioned earlier.
Each time you’ve written a piece of code, or reviewed a piece of code, or have spent time diagnosing a bug - if you have a test case for that method/class/API, you’ll be able to move faster (and this applies to all members for your team, mind you)
I am personally whole heartedly subscribed to having tests at all possible levels one can, but that’s the thing - you don’t need to be. The quest here is to save time, and like with all ‘Engineering Stuff’, there are tradeoffs and there are more than one ways to achieve the same goal.
Continuous Delivery
Continuous Delivery processes help you significantly save time avoiding routine dances before a release.
A word of caution, though - simply deploying to production multiple times a day does not qualify as continuous delivery.
There is more than enough information on the principles and practices of CD on the website (and Jez & Fowler’s books/lectures) that I need not go deeper into it.
Just remember - the Returns of even 1 engineer’s efforts on setting up good CD practices are multiplicative.
Script it out
The power of scripting for low-fi tasks is invaluable.
- Do you routinely check for the same ‘x’ things while diagnosing an issue? Script it out
- Is your company having a lot of new people joining the team and you have to explain the same steps to them, despite there being a documentation for it? Script it out
Know your infrastructure
When I was working as an infrastructure engineer, we routinely had emergency emails asking us to check if the infra is fine. Of course, it was always an emergency since the application was acting up and users were affected; and Of course, it must be the infra which is responsible for it.
It’s easy to suggest,“It must be the kernel [or the network or the database or the infrastructure]”. Don’t be that person. Don’t guess things - learn to diagnose things on your own.
Yes, we deal with complex systems when we deploy our code today. The application code we deploy may end up running in a container which itself is running on a Host Operating System running other containers; the Host OS may itself run as a Virtual Machine which may run on another completely different Operating System sitting in a big box which hosts 10s of such machines. Our code may ‘hit an API’ which results in multiple network hops - perhaps to a machine located in an adjacent building in the same ‘Data Center’.
Every parameter of everything involved in that setup - from the configuration of the OS, to the number of applications running on the host system, to a link flapping on a network cable on the core switch can result in your application behaving weird. This is the reality. While it is almost impossible to know everything about your infrastructure, simple tools will help you navigate a large subset of routinely faced issues.
Learn how things are deployed to production
Figure how/where your application code runs.
Is it running in a container? What is the configuration of the host machine? Do you know where to look to get basic stats on CPU and IO usage (Network & Disk)?
Does traffic come to it via a load balancer? Where do you find the stats for that?
How is a service call to a dependency (datastore or other application) made? Does it involve a network? If yes, how is that traffic routed?
Knowing this would help you isolate issues faster. This will help you apply The Box technique to isolate performance bottlenecks.
Learn more about your datastore
Invariably, a significant part of your application’s job would be either reading from or writing to a myriad of data stores.
Basically, If you’re married to your application and love it, then the datastore is like extended family. You must spend time learning more about it.
A colleague once brought a replication lag of ~3hours down to under 1second on a MySQL slave by simply RTFM on various configurations of the storage engine.
Similarly, I’ve seen a DBA mitigate a crisis (of what everyone thought was a DDOS) by zeroing in on a query and running EXPLAIN for it.
These may seem like wizardry, but it is mostly just people RTFM and spending time to learn more about the datastore their application adores.
Things That Don’t Matter (Time Sinks)
My Language is the best (Or, your language sucks)
I once attended a conference where the founder of a hot-startup proclaimed, on stage, that they chose ‘Functional Programming’ over procedural because it is ‘Theoretically Superior’.
No, it is not. Both Church and Turing proved that.
More importantly, any functional code you write is converted to it’s Turing equivalent - we don’t have a real machine built on church calculus.
Such ideological stances are generally rooted in half baked understanding of things and compounded by the desire of someone to come off as cool by knowing more about esoteric/obscure or even the latest-and-greatest-shiny-new-thing.
Sure, [Modern] functional languages are designed to help provide less ways to shoot oneself in the foot , but a well oiled C++ company6 would have evolved standards and patterns to write as beautiful a code than what a company working on Kotlin or Scala can; so where’s the difference?
It’s all about finding what works best for you, taking early bets and seeing them through.
booking.com used to work on perl; cleartrip.com on lisp; udaan.com works on kotlin ; there are 1000s of successful companies working in Java/C++. It doesn’t matter.
What matters is how productive the organisation is - how effective developers are at making progress on the job at hand - and whether or not it is helping you move forward.
‘X’ doesn’t scale, Lets Switch to ‘Y’
During an interview with a young dev, I enquired as to how he had ‘helped scale up <service x> and made it more performant’. He revealed how his mentor had advised him to switch to python web server because nginx is slow. It is indescribable7 how I felt when I heard this. But after a few brief seconds of an emotional whirlwind, I empathised with him.
In a similar vein, in another interview, someone described how they ‘moved away from JSON to GRPC’ because JSON was bloating their APIs too much and GRPC is more performant.
I do not want to get into the merits/demerits of the choices they took (thereby, abiding with the advise that I am repeatedly giving here).
I do, however, want to draw your attention to some facts that apply to most of us in general:
We live in a time where we can easily procure and deploy very powerful hardware. Unless you have chosen some of the worst written frameworks8, they will ‘scale’ more than you may imagine .
The word scale is also one of those things we notoriously are prone to brag about based on our half baked understanding of things and the desire to be ‘cool’.
I used to be one of those people. Back in the day, we were proudly managing a service with 10k reads-per-second (has your jaw dropped yet?) backed by a 32core 128GB MySql machine.
I was proud of the ‘scale’ we had managed to weather, until a dev with about 15 years of experience joined our team. He casually remarked one day that he could run 3 of his last startups with that much computing power9. I learnt a big lesson that day, that life is not a bragging contest. Also, what we think is scale is probably not. We are prone to boast of our accomplishments but they are largely the result of us standing on the shoulders of giants.
While dealing with and handling scale is beyond the scope of this blog post, I’d leave you with this general advise
- Learn one base tool for the job and learn it really really well
For e.g, pick up nginx/apache web server and I almost guarantee you that these will break much much later than your entire application stack. - RTFM & Benchmark
Benchmark every tool/technology that you are picking up. This way, you’d be more aware of the ‘limits’ of that technology.
How you benchmark is beyond the scope of this post (perhaps even beyond the scope of me) but a simple process can yield fairly good results.
Push a technology to its limits by maxing out the resources (compute / memory / network / disk IO) it is supposed to consume on a common piece of hardware (which you’ll most likely employ in production)
Once you max it out, RTFM and tweak it to ‘improve’ upon the limit that you just hit.
Repeat till you’ve either satisfied your intellectual curiosity, or realised that the solution you’ve chosen can handle 10x more of your predicted scale.
Lets use a <thing-i-just-learnt-about>. Its perfect for the job
A young dev we hired was given the task of writing a simple CRUD application. The Update part of it, I thought, was particularly tricky, so I figured it’d be a project right up his alley and challenge his skills. He was sharp, full of energy & wanted to prove himself - everything you want in a young developer.
He also wanted to use a <distributed-highly-scalable-queue> in the application.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
Me: Why?
YD(Young dev): It will help us scale faster in the future and guard
against the impedance mismatch between the core business logic and the
rate at which clients hit the API.
Me: Sure, but do you really want to take on the complexity of having a distributed queue so soon in the life cycle of your application? It'll probably not hit the scale you mentioned for quite some time
YD: Oh there's no complexity, we can just use the one provided by
<cloud-provider-x>
Me: But what if it looses data? Since your application deals with
money, it is critical we don't loose a single piece of information
YD: How will a queue loose data?
Me: Any distributed system can loose data
YD: How so?
Me: Ever had a payment fail where your money was deducted but the
order failed at the merchant page? That's a distributed system losing
data (Not the best of examples, I know)
YD: That's different. Those are multiple different services and
ecosystems. Queues reside within the same private cloud - probably the
same data center. The problems of the internet don't apply here
(Oh Snap! He got me)
Me: Yes, but there is still a network. And the queue given by
<cloud-provider-x> is yet another piece of software written by people
like you and me. What if it has a bug ?
YD: Oh
I encouraged him to read Distributed Systems for Fun and profit.
I also showed him how the complexity of his system lies in the design of the data model and the API and not in scale (yet).
A good model would also help us ‘scale well’ in the future (when we may, inevitably, have to rely on complicated distributed systems)10
There was yet another instance of a more seasoned developer who wanted to rewrite our order management system and use <the-latest-and-greatest> NoSQL solution offered by <cloud-provider-x>.
It would help us ‘be ready’ for a potentially 100x more scale than what we’re currently facing, he claimed. This in itself was scary (refer previous discussion), but what was scarier was that not once did he mention
- Whether or if going with NoSQL was inherently better since it would provide us with a data model that will prevent future developers from shooting themselves in the foot
- If all the complex business use cases could be handled within the bounds (such as row level atomicity) imposed by the NoSQL data store (the current system relied heavily on RDBMs transactions to stay ‘correct’)
My whole point is this - yes, what we do is generally complex and we live in a world where our tools are constantly evolving. We need to continuously up skill ourselves and learn more to stay relevant.
It is important to have the knowledge of a number of tools in our mental tool set - so we can employ any of them for the job at hand.
It is more important, however, to get an understanding of using the right tool for the job. Our work as engineers is to help the company make money by keeping customers happy. Not to show off the latest-thing-I-just-learnt-about to our peers.
Your way is wrong
As already mentioned, there is no ‘absolute right’ when it comes to Engineering Stuff. Similarly, there is no ‘absolute wrong’.
One of my favourite software stories is unfortunately not a first hand experience. This was from early 2010s when a friend worked for a company that deployed engineers in the client’s team to augment their bandwidth. They were supporting a complicated piece of legacy software. My friend found a piece of code ‘written in an odd way’. He couldn’t intuitively grasp it and thought it could be ‘written better’11. He did what most of us do - googled around and found a better way.
He then cautiously approached the grey-bearded engineer who had originally written that code. My friend pointed to the ‘design pattern’ on the internet and suggested how code could be written better.
To his absolute credit, my friend inquired sincerely as to why the code was written the way it was.
To this, the older engineer promptly replied “Why yes, of course this does seem better. But you know the thing is, when I wrote this, there was no internet”
I absolutely love this story and remind myself each time I look to blame someone for how it’s written.
If you write code for long enough, you will come across code that would want you to become the proverbial psychopath and shoot the original author.
However, it is important to be empathetic instead. It is necessary to have mutual respect towards your peers(if you don’t, you probably shouldn’t be there in the first place12)
If possible, try to understand why things are done the way they were done.
If that’s not possible, it is perfectly fine to poke fun at the original author, make a note of how you’d change it instead and come back to it when you have the time. Move on13. Don’t spend time criticizing others.
I hope you found this useful (assuming you didn’t come here just to see how it ends).
I’d love to hear what you think. We don’t have a comment section yet, so if you want you can email me or comment on HackerNews
Well, they still are - I’m just no longer part of the organisation ↩︎
From ‘The Audacity of Hope’, Barack Obama ↩︎
We all have ↩︎
Well, I’ll be honest - wakatime says I clocked about 700 hours last year. But I’m sure you’re a better developer than I am :) ↩︎
We’ve all done it, admit it ↩︎
C++ is notoriously famous for giving you many ways to shoot yourself in the foot ↩︎
A mixture of irate, sad, I-want-to-end-this-interview-right-now kinda feeling ↩︎
In which case, what were you smoking when you chose that in the first place ? ↩︎
Before the keen reader remarks ‘but were they any good?’. Yes they were - he made enough money that he was working ‘for fun’ now ↩︎
To his credit, he did go through the entire book and did a remarkably good job at iterating over his API design and data model. ↩︎
You can substitute ‘written better’ with your choice of lingo - ‘better designed’, ‘used better abstractions’ or what-have-you. ↩︎
A good place to be is where the feeling of looser-ish-ness is prevalent. If you’re the smartest person in the room, you should look for better places ↩︎
All of us have done this and others would do it for your code as well, because here’s the thing - there is no such thing as perfect code. ↩︎