Engineering and Responsibility for Your Team
In my first year leading a student team working on a simple web app, we were tasked with integrating a new feature from a volunteer, an AI/LLM specialist. The feature supposedly worked fine… it was tested on his machine and was reported as such on the morning of the event. If it were ready earlier, we could have written a CI/CD pipeline, but we were all part-time volunteers with schoolwork to finish.
The feature ships because a few of us insist on it, for all the work he put into it. I was fine with it. Our event leads advised against it. We could have done perfectly fine on that day without that new feature. The feature ships and some idiot from leadership begs for a pedantic last-minute change. I make the change, and the feature breaks. The company loses participants and loses face. The software team loses face. Even during the failure, leadership was looking for someone to blame. But despite being the one at fault, the sword did not fall on me.
The volunteer who made his debut was never tasked with critical work ever again. I was unable to convince leadership that I was the one at fault. I could have read the code myself and seen that the hard-coded values that I changed in production on that day were his feature’s constraints. I was the one who broke the feature. I hardly care about the failure of our software compared to how that company treated him afterwards.
The Rewrite of “The Everything App”
In the same year, I took on some volunteers. This was the total rewrite of a mess of a webapp that was reliant on three external storage services and one open-source “Database and API all-in-one” self-hosted service that was full of stale data. I still have a selfie with a very very secret API key, discovered in the HTML of a button, on that webapp. We updated all our dependencies. It took a whole week. Leadership decided to put me in charge of rebuilding the whole damn thing.
It’s low-stakes. The company relied on a workforce of students specifically because they valued the learning-by-doing of putting students in charge. That, and we’re cheap. I spent an entire month talking about architecture and design before starting its implementation, like a good boy in grown-up pants.
Being the altruistic (desperate for more hands) leader that I was, I asked volunteers what library, or part of the tech stack they wanted to learn. Go, TypeScript would have been good. Being the anxiety-prone (I believe it now to be a beneficial trait) engineer that I am, I chose Python (FastAPI) for the backend, without ORM. We had less than one API transaction per second even in our most popular seasons.
Go is great, I wanted to deploy some Go too. However, if I were to leave the project, could the company easily find someone as curious as me, that they learn Go within their undergraduate degree? Will they understand what I’ve written? We will leave in two years or less, because we’re graduating, because every one of us is just an undergraduate student that needs a semester to get up to speed.
TypeScript would have been fine too with all the frameworks. But there will be fewer TypeScript users than Python users. For the future workforce of this company, I chose Python for the backend. Python is taught to every student. We can lure volunteers by calling Python experience valuable, and teach them from there.
Actually, Don’t Teach them From There
We lured volunteers by calling Python experience valuable, and tried to teach them from there. Actually, don’t take every student you can find, that was a mistake in my approach. As much as your goal is to teach students, passing intro to Python isn’t an indicator that you’ve found a reliable colleague.
You spend an hour or two onboarding them, making sure they have the perms and the software runs on their machine. You give them a small task and some relevant documentation to get them started. They don’t ask questions. Then you have five volunteers that you ask for updates on their task weekly, until they stop showing up, and the task is late.
These are the people you get in your group projects, but now, everything they give you represents your work. And unlike group projects, you are the one choosing your groupmates. Be more selective, talk to potential volunteers, ask about the repositories on their GitHub profiles. Find the people who are worth investing your time into. At the scale of a real-world project built in your name, you need hands you can rely on.
Engineer Away Competence
Recently, I “fixed” some code. I had a really rough day of carrying a group project. I made a one-line PR to fix some embarrassing code I wrote some months ago. My co-lead said “not even your fault, should have been caught in review”. Well, LGTM! It passed the CI/CD.
Our codebase has two builds. One for development on a local machine, one for deployment on a machine of a different architecture. CI/CD only checks the deployment build. A colleague finds a deployment machine to test my code, it fails to compile, I waste her time.
I think to myself, “If I were a better programmer, I could have never written the wrong code in the first place. If I were less tired, I’d have compiled the code on my machine after I wrote the fix like I usually do. If my reviewer tried the code like he should, my mistake would never have made it to main. But, wouldn’t it be good to have that in CI/CD so that it never happens in the first place?” As soon as I’m done thinking, a fellow engineer already had the PR up to catch those errors in our CI/CD pipeline.
I’ll steal the words of a League of Legends player whom I think could be a great systems engineer.
If you’re smart, you hold yourself accountable for everything. If you’re wicked smart you create a system to hold yourself accountable. Layers on layers of protection.
Give yourself grace when needed. We aren’t machines.
We’re engineers, and our names are on our work. That’s the system that holds us accountable. So how should we prevent our names from being held accountable in a production environment? Wicked smart engineers build systems that both prevent mistakes and slap themselves for making mistakes (beyond attempting to compile in CI/CD; all kinds of testing, static code analysis tools like LSPs, linters, formatters, type-safety, memory safety).
So how should we know to build those systems without first making those mistakes? Learn from everyone else’s, but as the architect who chose the tech stack and its limitations, you’d better have some knowledge of mistake-preventing tooling. That, and you’ll make mistakes anwyays. Give yourself grace.
TL;DR
- You will be responsible for your project and its outcomes
- You must take care of your team. Own the mistakes that happened on your watch
- You will cooperate with people of different skill levels and availabilities
- Engineers will come and go
Reflect the above in leadership, as well as the product’s architecture, stack, CI, and design.