I've spent over 15 years in the software industry, much of it in management, and there are certain conversations that I have either witnessed or participated in repeatedly over the course of my career. Not surprisingly, many of them are related to the concept of the proper way to build software. Management wants to build things fast and cheap; engineers want to write robust, quality code, even if it takes longer. Data programmers want to consolidate business logic in stored procedures; software programmers want business logic in services that can be unit tested and readily debugged. New hires want to use modern technology and rewrite everything; the old guard wants to preserve the status quo because working software is valuable.
A spectrum of development practices
If you're nodding your head, well, I've tricked you a bit - notice that I said "the proper way" - singular. If you believe there's only one proper way to build software, this article is aimed at you. Even if you realize that there are numerous ways to build software, you may still have had some of the same conversations I'm going to reference. I'm here to tell you that you're not alone.
Let's start by dispensing with the notion that there is only one proper way to build software. Let's consider two very different ways to create software:
First, let's acknowledge that this is a spectrum, with extremes at either ends. Second, we admit there are other axes we could add, and they'd likely overlap significantly or entirely. For instance, "no unit tests ↔ 100% code coverage for unit tests", "WET ↔ DRY", or "use the latest, most bleeding edge technology" ↔ "use the most stable, reliable technology"
For now, we'll stick with the "perfect" ↔ "immediately" axis. It's unlikely that either extreme actually exists in the real world, but it's worth imagining such hypothetical approaches.
Placing your organization on the axis
At RawCode, Inc, they ship things immediately - programmers type the code and, presuming it compiles, it's shipped out the door. Maybe this is how 90% of apps get into the Google App Store. Someone thinks of an idea and it's out the door in minutes. It's unlikely to work.
At Pristine Unlimited, nothing is shipped until it is 100% bug-free, tested with 100% code coverage, perfectly conforms to SOLID principles and scales sub-linearly. Perhaps the most realistic real-world example would be for truly mission-critical software where lives are at stake - on a space shuttle or an airplane. But of course they ship incredibly rarely - perhaps once a decade.
Think about the organization you currently work for or ones you have worked for. Where do they fall on this spectrum? Where is Google? Where is Slack? Where is NASA? These are all successful, healthyorganizations - and hopefully the ones you've worked for are, as well. And if you are building software as part of a business, then the idea is that there is not one successful way.
Placing yourself on the axis
The second part of this exercise, then, is to determine where YOU lie on this axis. My guess is that where you place yourself differs somewhat from project to project, depending on a variety of factors, although potentially always within some range. It may also be that your organization has different standards depending on which project is being considered, but most likely both you and the organization are in a fairly narrow range, with the organization being less flexible. If you were to plot your preferences against your organization's, you might up with something like this:
The important part of this exercise - figuring out where you lie on this spectrum, and where your organization lies - is to determine how good a fit you are for your organization. Of course, without quantitative ways of measuring the values on the axis, that could prove problematic. Let's add some annotations to the graph:
How's the fit?
So we've basically drawn two 45-degree lines and shaded in some areas. Let's discuss each:
The sweet spot:
If your preferences are basically in line with your organization's, then everyone's happy! You're developing and shipping software that is up to your personal standards in a time frame that the business requires. If everyone was always in their organization's sweet spot, there'd be no need for this article.
Mismatch area 1:
In this area, the organization wants the software shipped more rapidly than the engineer. Here are some indications that you are in this area:
- Developers are constantly complaining about code quality
- Tech debt is accumulating rapidly
- Frequent need for, or discussion of, rewrites
- Internal discussions that the organization is "short-sighted" or "too sales-driven"
- Developers doing work they aren't proud of
- Constant pressure to lower estimates / developer estimates being ignored by project managers
It's important to remember that all of these symptoms can occur in perfectly functional organizations. It is certainly the case that at some organizations like this, the priorities need to shift, and management needs to tamper its expectations, because they are asking the impossible from software that is at its breaking point. If the mismatch is dramatic enough, but the organization is on solid ground from both engineering and business standpoints, is may be the case that the developer is working in the wrong industry.
Mismatch area 2:
In this area, the organization needs high quality software, but the engineer wants to get things done quickly. Here are some indications that you are in this area:
- High rate of customer-affecting production incidents
- Engineers cannot support or maintain each others' code
- Little to no documentation is available, especially in places where it would be beneficial
- The software does not scale well
- Frequent need to refactor / large amounts of code duplication
- No due diligence performed around use of new technology, resulting in many different packages and frameworks used across the organization to achieve the same purpose
- A lack of design / planning meetings
I'm inclined to say that area 1 is generally more common than area 2, as many businesses underestimate the time it takes to produce quality software.
Getting to the sweet spot
An important point here is that all of these issues surface in any engineering department. As I said above, you are not alone. I'm not advocating that any of these alone is a sign of a mismatch or dysfunction and that you should quit your job immediately. That said, if your priorities as an engineer don't align with your employer's, at some point it might be time to cut your losses. Of course if you actually do think you need to move on - we're always looking for great engineers! Short of that, my intent in writing this is to change the tone of conversations that start out with the idea that software is not being built in the proper way.
If you are having frequent conversations along the lines I mention, or are often frustrated by others not doing things "the right way", take a step back. It may be the case that you just need some perspective, or you should shift expectations about what the organization's priorities are. Maybe talk to management or even sales and try to understand the business needs driving the push for quality or speed. Maybe start the arduous task of changing how your company or engineering department functions: Demonstrate how valuable a few good unit tests can be. Quantify the extra time spent rewriting and debugging poorly-architect code versus how long it would take to refactor it. Help leaders in your organization see that code quality can affect the bottom line.
"Changing the company"?!? That's right: None of the area in blue is fixed. Both you and the company are mutable.
Some developers are very particular about how software should be built. Some don't change over time. Some have no preference and are happy to do whatever the organization wants. No set of preferences is "right", but it's important to acknowledge that you do have preferences that inform your views on writing software.
In my experience, as developers grow, their range of preferences widens, trying to adapt based on the specific business needs of the project they're working on. In general they also seem to move from the left to the right of the axis, usually coming out of a school with a notion of what is "correct" and then realizing that practical engineering requires compromises. They also tend to gain a better understanding of the business requirements, and come to grips with the fact that without the sales department, there is no engineering department.
This growth can be frustrating, as developers believe that they are forced to compromise their ideals and reduce the quality of their work. Often this frustration can be mitigated by providing them more visibility into the company's business operations and priorities. Experience can also bring a greater understanding of YAGNI. We all remember the instances in which we wrote code rapidly and then ended up being forced to rewrite it as requirements changed. We might have even said "I told you so". But it's a lot harder to think of the times where you took a shortcut that made you cringe, and then you never actually revisited the code. Or what about the generic, beautiful solution you wrote that would handle all potential future use cases, and then none of those use cases ever came to fruition?
Similarly, organizations change, although almost always at a slower rate than individuals. Usually young businesses need to ship immediately to start generating revenue. As they get a base of users that rely on the software functioning properly, they move down the vertical axis, less tolerant of revenue-impacting issues.
Organizations grow in two ways - their software matures, requiring becoming more complex and monolithic. This often requires more discipline to handle - knowing when to rewrite or reuse existing code instead of building something new out of the whole cloth. The more mission-critical the software, the more reluctant an organization should be to modify existing code. The other way organizations grow is simply by increasing their size. This increases the need for coordination - documentation, planning, etc. An engineer who has been at a company since its founding may lament additional processes that are put in place to accommodate this growth, thinking that the company is no longer able to get things done quickly, and bureaucracy is increasing. It's important to both minimize unnecessary bureaucracy and explain the purpose of new policies that are put in place.
Where's our "sweet spot"?
As a Director of Engineering, one of my responsibilities is to take these observations and put them into practice. At NGP VAN, we try very hard to make sure that as many engineers as possible are in the sweet spot. That means we need an open dialog between engineers and other stakeholders, to ensure that the company's values and the engineers' approach to coding is in harmony as much as possible. It also requires being honest with ourselves about where we are on the axis as a company. NGP VAN is still a mid-size company, and our clients are political campaigns and non-profit organizations. We need to continually deliver value to them rapidly so that they can win elections and get their message out. We've also been around for nearly two decades, so we do have a lot of engineering experience in this space. That means we know when it's worth it to accumulate tech debt to pay down later, or when it's necessary to do it "correctly", with an eye toward longer-term improvements. We want to hire and grow engineers who understand how to make those tradeoffs on a day-to-day basis.
In my two years here, we haven't moved much on this axis, but we've become a lot more disciplined about making these decisions intentionally. Specifically, we are continually working on enhancing communication between stakeholders and engineers so that we know the implications of the choices we make. If we need to rapidly push something out to satisfy a client, but it'll come at the cost of future tech debt, we want the cost of that debt to be explicitly identified, accounted for, and incurred. Maybe we'll decide to address it later; maybe we won't - but at least we know it's there.
We've also become better about communicating across departments. It's easy to have requirements handed down from "on high" to engineering, and for engineers to feel like the Sales or Product department is telling them to get something done quickly without understanding the implications. That's why the conversation needs to be an actual two-way conversation, and hopefully iterative. Product and Sales tells Engineering about requirements; Engineering presents different ways to achieve those requirements; Product asks about costs and benefits of certain approaches; Engineering provides more detailed estimates, etc.
We're currently working on more clearly articulating a vision statement for the engineering department, including a well-defined and highly visible technical roadmap. Some of our engineers will no doubt feel that our technical roadmap is not comprehensive enough; others will believe that we're making unnecessary changes. But we'll have the dialogue out in the open, so people can see where we are on the axis, and both the organization and each individual engineer can potentially help move us all into the sweet spot.
Properly aligning the goals of the organization and the development ethos of engineers is a difficult task. It requires self-awareness and openness, but successful software organizations must work to hit the "sweet spot" as often as possible.