I’ve spent 7 years working in large tech companies, and I can expect every now and again I’ll need to do some menial work that can take hours or even days. Sometimes by myself, and other times as a group with other engineers. This can be clicking through too many UIs, running through a list of commands and wait times, or going through a spreadsheet of 1000+ rows and typing “good” or “bad” for each row.
This is something that can be surprising to some new hires especially when we have so much sophistication and “magic” available to us today. Tons of tools that can work on their own and in bulk: generating boilerplate, scheduling workloads, coordinating changes across a fleet of machines, aggregating trends from billions of datapoints, not to mention the slew of research going into LLM based methods.
Even so, doesn’t matter your tenure, doesn’t matter your rank, we’re all equal in the face of necessary menial work.
Training your barometer
I like to think that many engineers have common origins:
I like making things and like to see those things used, hence I became an engineer. I’m opinionated in my work, calibrated my “ew, we can do better” barometer, and so build the things that provide satisfaction (and let me be a little lazy).
We pursue a lingering sense of “better”, and it’s hard to accept that “enough” right now is preferable to “better” later. Learning when to lean into our instincts and when to deny them can be tough.
Sometimes there is nothing better
An illustrative case is most early-stage ML work. You don’t have a dataset, so you need to make one. Rows and rows of inputs, outputs, labels. Associated files like images, videos, what have you. These are things that you and/or the people working with you will have to collect and manage by hand.
To save time, you may be able to find something to scrape and might try generating some synthetic data. With the advent of widespread research, services, and tools based on LLMs, the quality of generated synthetic data is as high as its ever been. Heck, half of my current work is based on improving accuracy and reliability in LLM based evaluations. But any and all shortcuts come with risk. Your ML work will only be as good as your underlying dataset, and synthetic data is a lesser substitute for the real thing, likely generating more work for you in the future.
At the end of the day, you may sit there and recognize that these shortcuts were just a time sink and throwaway work. There may be something salvageable, but as an addendum to the original work that is still yet to be done. The required work is constant, just kicked a little further down the line.
Sometimes it’s not worth the resources
Your resources are your time and your money, and you’ll often choose to exchange one for the other. The menial work is an understood cost, something planned for. When you consider spending your resources on simplifying or scaling the menial work, you add cost and risk to your project. It’s easy to see the value for frequent touch points or processes, but often times your current problem isn’t the hottest thing in town. Figuring in a more realistic pattern Consider spending time to improve a workflow or spending money to outsource data annotation. At what point does the benefit recoup your time, or earn its value in dollars? Now measure that against the other work you could be doing.
It may be worth revisiting the idea after, or considering if there are any longer term merits, i.e. recurring work. There’s a good chance that it still isn’t worth it to the business. Your resources are better spent providing value elsewhere.
A simple lesson
This is one of those things that’s super simple at face value. After having a conversation about it with one of my junior engineers, I just kept thinking about it. I was remembering how I had to learn it the stubborn way - I kept sinking time into figuring out “a better way to do this” because it felt wrong to do it any other way. I was willing to stay a little later so I could figure out how to make it “right”, insisting that tediousness is a solvable problem that can be solved with consistent diligence. After knocking my head against every wall I came across, I had to start doing quick cost-benefit analyses to talk myself down, ultimately preserving my sanity. Sometimes the right answer is to plug in your headphones, turn your brain off, and work through a wave of tedium.