Engineering Is Not Just About the Cool Stacks

2021-05-25T00:00:00+00:00

As a developer, I love finding new shiny pieces of tech that can help me develop web applications better. To me, it is one of the best parts of the job: Pick up a new programming language, a new database, a new standard, etc., and build cool things with it. But recently, I had the chance to reflect on my experience working on a boring stack at one of my previous workplaces, and it was actually a lot better than I expected.

My journey

I used to work at a company where the teams are free to decide the tech stacks that best fit their requirements, so a wide range of modern tools were used there to solve the problems at hand. Within just 2 years of joining the company, I switched teams four times and touched everything from Java, Scala, Akka, Terraform, AWS ECS, Serverless, API Gateway, AWS Lambda, AWS DynamoDB, React, Gatsby, Vue, Nuxt, TypeScript, Flowtype, and many other tools that really helped advance my technical skills.

But in late 2018, my manager invited me to assist a “less popular” team in the company. The team handled a myriad of complex products that are quite old. The team’s tech stack was not the shiniest compared to the other teams in the company; the products are developed using Java, a language I generally do not prefer due to its verbosity. We just shipped JAR files to the platform’s marketplace, so there were no web services to maintain and hence it didn’t challenge or sharpen our operations skills. The frontend JavaScript code was largely built with jQuery as it came bundled with the platform we are building on top of.

As a developer who loves shiny tools, this team should have been a dreadful one to be in. But to my surprise, as I reflect upon it, it was actually the team that gave me the highest work satisfaction. It was satisfying because there were a lot of problems to solve, and most of the time the harder problems to solve are not the tech stack: the apps work, and customers are paying.

Instead of talking about moving from Java to Kotlin or from jQuery to React, our team spent the time brainstorming ideas on how to increase code quality, how to ensure discipline and follow good principles when crafting software, how to improve our developers’ workflow and processes, and how to best share knowledge and contexts that we had just acquired for future team members. Each problem solved gave me that dopamine boost that made me feel good.

Below are some of the things we did to solve these problems.

Problem #1: Code quality

The products the team were handling had been suffering from some quality degradations in recent years, and these ultimately led to more bugs and regressions, which then leads to an increase in customer complaints, and that leads to us working under pressure, which then leads to even more quality degradations, and the vicious cycle repeats.

When this started to happen, what did we do?

We slowed things down. Instead of having individual engineers work on multiple tickets in parallel, we held more pair programming sessions and reduced the number of tickets in progress to help incorporate collaboration earlier and gain feedbacks on the implementation strategy sooner. At the same time, pairing sessions also helped get the newer engineers up to speed faster. We introduced more quality gates for pull requests to be able to be merged into main. We agreed as a team to scrutinize them harder, e.g. if you do not have tests among the changes, just consider the pull request rejected. We also required all team members to approve the changes. We worked with product managers to slow down shipping new features and instead having more bugs fixed. Hence, the health of the codebases got better and better and the products got more and more stable each day.

We also held a brown-bag meeting reinforcing SOLID design principles to everyone in the team. Knowledge sharing like this helps bring everyone onto the same page while pairing and reduce the long discussion threads on pull requests.

Members of a team come and go, and that is true for all teams. Complex apps like the ones we were maintaining are usually full of legacy decisions that the newer team members will not have the knowledge of if no deliberate efforts were made to ensure that those contexts get shared. So, we took steps to mitigate this by including the contexts in commit messages. We also encouraged writing our findings in the Jira tickets and in the pull request descriptions on GitHub, but we preferred Git commit messages for documenting decisions related to the commit, because Git is the system that is least likely to change. We might switch from Jira or from GitHub to some similar services, so contexts would be lost when that happens, but it is very unlikely that we would switch from Git for version control.

We also started a knowledgebase to contain information that should be shared within the team, including onboarding instructions, development gotchas, legacy contexts, etc. This helps distribute the knowledge to everyone and reduce blockers on specific team members. We also made a habit of documenting a discussion that happens offline, often in our internal communication channels or in our internal knowledgebase.

Problem #3: Developers’ workflow

Improving our process was also a focus. One of the first things we did was rewrite our Jenkins CI/CD pipeline to adopt multibranch pipeline, allowing us to build and test feature branches before they were merged into the main branch.

We also automated several tasks that we identified as repetitive, from as small as notifying everyone that there is a new pull request open on GitHub, to automatically running compatibility checks for all of our products once the platform we were building on released a new beta version. We constantly looked for areas we could automate better and discussed them in our weekly retrospectives.

We also tuned our workflow to better organize our capacity around all the demands that we are getting: from our customers, our roadmaps, and our improvements bucket. We adopted Classes of Service (CoS) to achieve that. CoS is an approach in which you decide different treatments to different types of work. This means the capacity we have is flexible depending on the demand of each type of work.

For example, we introduced a collapsible task buffer called Intangible that holds work whose value customers do not experience directly, such as upgrading Node.js version, etc. It is collapsible in the sense that the developers in this CoS can be reassigned to other CoS should we deem more capacity is required there. Note that CoS (or any other system for that matter) is not without its pros and cons, so be sure to tweak it to fit your circumstances should you consider adopting it.

Not all problems are tech, but all tech has problems.

Sometimes, we identify ourselves through our favorite tech stack.

Hey, I’m a JavaScript developer.

Hey, I’m a Ruby developer.

And that is not wrong per se, but this particular experience of mine serves as a reminder that the essence of software engineering goes deeper than just the tech — it is to solve problems through software. And that is what we do at End Point.

Reference

SOLID Design Principles at Devopedia
Classes of Service:
- On Classes of Service (CoS), a more pragmatic approach towards “squad formation” by Yuen-Chi Lian at ServiceRocket
- Predictability and Classes of Service by Leonardo Campos at ThoughtWorks

The Dollars and Sense of Hiring Software Consultants

2019-08-16T00:00:00+00:00

Photo by Nik MacMillan on Unsplash

I’m often asked by family and friends about End Point’s market and how software consulting fits into the business landscape and I thought I’d write up some thoughts for the general public in how hiring a consultant is actually the smartest thing to do for your business and it makes a ton of sense financially.

Let’s talk some math about hiring an in-house developer:

According to Glassdoor, the average software developer salary is around $50,000–$120,000. Let’s say $85,000 per year. You typically pay for health insurance, other benefits and overhead expenses, so let’s just round the total cost of hiring one person to $110,000 per year (I think that’s actually pretty conservative).
You’d probably need to hire two people to support anything essential to your business, since having a single developer is risky (illness/vacation/employment change, etc.).
Your total cost of getting two developers in-house is around ~$220,000 per year

	Hiring your own developers	Software consulting agency
Senior developers	✓	✓
Diverse skillset	?	✓
Protection from staffing changes	✗	✓
Flexible budget	✗	✓
Staff up and down easily	✗	✓

Consulting Instead of Hiring

Spending less: You could spend $40,000–$50,000 a year in consulting and get excellent coverage, including new feature development and support for your project. Some of our customers even spend substantially less than that.
Skillset: Were you able to hire two people that know all of your software stack? Can they work on your CRM/inventory system and do your website? Can they troubleshoot issues with your Windows Active Directory setup? Are they senior developers with a decade of experience? Or did you have to hire junior developers who need to train up on your project?

When you’re working with an End Point team, you’re able to get expert advice on the parts of your system that you need. For example, you might not need a full time DBA, but we can provide you an expert database resource for a few hours a week. You might not have enough work for a full time network engineer, but you do need someone to help with your SSL certificates, domain names, and caching layer. In that case, one of End Point’s DevOps engineers can offer you help for a few hours a month.

Team resources: What happens if someone doesn’t know something? Do they have a resource to turn to or are they trying to figure it out with a Google search or Stack Overflow article? End Point has 40+ engineers on our team and we work in a collaborative way to solve problems, giving you the benefit of a very wide diversity in technical skillset.
Hiring: Do you have the technical expertise to vet new employees? Hiring, testing, vetting, and onboarding new developers can be really tricky and End Point’s technical management team takes care of that for you. All of our developers have been through a rigorous skills test before employment. Upon joining the company, developers participate in our technical bootcamp to ensure they understand all of our internal procedures for data handling, security, client management, code verification, teamwork, and many other important skills.
Resource changes: If you’ve hired your two software developers, you still have to replace them in the event of an illness or job transition. If a developer leaves, End Point takes care of resourcing your project with new personnel and getting them up to speed. You don’t have to worry about the hiring or replacement of employees. We assign new team members with the appropriate skills to tackle your project, and our employees have our broad and deep team to call on and consult with to leverage their effectiveness.
Staffing up for a big project: One of the challenges faced by any business looking to do a big project either with an internal application or a website is how to staff up for all the work and then figure out what to do with their staff once the big project is over. Hiring a consulting agency lets you put 2, 3, 4, or even more people on a project for a big push and then back off to fewer people once your development is over. You don’t have to worry about temporary employees or a situation where you have to let one of your employees go due to lack of work or budget for development.

Summary

Hiring consultants can reduce cost for your project development, eliminate the stress of hiring knowledgeable staff, and provide you deep technical experience from a team of professionals.

End Point has some verified client reviews at Clutch.co so read about how we’ve helped our clients!

Work philosophy canon

2018-05-07T00:00:00+00:00

Shared culture

Having some shared culture is important for working together well, and we can build that culture on familiar terminology, understanding, experiences, stories, and ideas.

To help give all of us at End Point some common reference points, we have collected a set of valuable articles and books that we encourage everyone to read.

Some of these have been standard reading at End Point for more than a decade, while others have been added over the years since we began doing this. Some are short and simple, others more in-depth. Our list is intentionally general, mostly avoiding specific technologies that only a subset of us use.

No one article or book can be entirely authoritative for all situations, but each gives us more of the wisdom out there to consider and judiciously apply where it fits.

Slow and steady

When new employees start at End Point, we ask them to read the articles during their first week or two, and the relevant books within roughly their first year.

Reading only a little at each sitting and spreading the reading out over time allows the ideas to sink in gradually and be incorporated into our work, rather than overwhelming with new information that cannot all be absorbed at once.

For everyone

Because we work in the software development industry, it is important that not only technical people such as developers, database experts, and system administrators be part of the shared culture. Everyone else at End Point including designers, project managers, sales, marketing, etc. should also be familiar with these articles and the terms and concepts they discuss:

Speed matters: Why working quickly is more important than it seems by James Somers
Please don’t just say hello in chat originally by Brandon High at nohello.com
Learn to Read the Source, Luke by Jeff Atwood—practical reasons free software and open source matter, and why actually using the source code matters
What is free software? by the Free Software Foundation and Richard Stallman
The Open Source Definition by the Open Source Initiative
Josh’s Rules (of Database Contracting) by Josh Berkus
On excellence by Ethan Rowe—about ownership, taking responsibility for the problem at hand
Conventions to know by Jon Jensen—little but important things that ease communication in our work
Selling Without Being Salesy by Jason Parks
Five Whys by Eric Ries (covered in more depth in his book The Lean Startup)
Software Is a Mess and I Feel Fine by Alan Stevens
Choose Boring Technology by Dan McKinley
Getting Real book by 37signals (now Basecamp)

For developers

These articles and books are targeted primarily at developers but enlightening for everyone:

A summary of the international standard date and time notation by Markus Kuhn
The Admin Zen by Michael Prokop—in true DevOps fashion, this system administration advice applies to development too
Things You Should Never Do, Part I by Joel Spolsky—don’t rewrite things from scratch (almost never, anyway)
On Git Commit Messages by Michael Jones; to get a feel for the long history of others sermonizing on the same worthwhile topic, see also:
- How to Write a Git Commit Message by Chris Beams
- On commit messages by Peter Hutterer
- A Note About Git Commit Messages by Tim Pope
Every line of code is always documented by Mislav Marohnić; see also The git pickaxe by Philip Potter
How to Read Other People’s Code—and Why by Christopher Schanck
You don’t know JavaScript by Michael Woloszynowicz—the lessons apply to most technologies
The Pragmatic Programmer book by Andy Hunt and Dave Thomas (see the write-up on the new 20th anniversary edition by Jon Jensen)
Practices of an Agile Developer book by Venkat Subramaniam and Andy Hunt—kind of a sequel to The Pragmatic Programmer (see the review by Ethan Rowe)

Any others?

We are always on the lookout for other readings that are helpful, so feel free to leave a comment with your favorites!

(Updated in early 2021 with articles on Git commit messages and history spelunking, and Choose Boring Technology.)

Designing a Computer Science Program for Free (or Cheap)

2017-05-04T00:00:00+00:00

This blog post is for people like me who are interested in improving their knowledge about computers, software and technology in general but are inundated with an abundance of resources and no clear path to follow. Many of the courses online tend to not have any real structure. While it’s great that this knowledge is available to anyone with access to the internet, it often feels overwhelming and confusing. I always enjoy a little more structure to study, much like in a traditional college setting. So, to that end I began to look at MIT’s OpenCourseWare and compare it to their actual curriculum.

I’d like to begin by acknowledging that some time ago Scott Young completed the MIT Challenge where he “attempted to learn MIT’s 4-year computer science curriculum without taking classes”. My friend Najmi here at End Point also shared a great website with me to “Teach Yourself Computer Science”. So, this is not the first post to try to make sense of all the free resources available to you, it’s just one which tries to help organize a coherent plan of study.

Methodology

I wanted to mimic MIT’s real CS curriculum. I also wanted to limit my studies to Computer Science only, while stripping out anything not strictly related. It’s not that I am not interested in things like speech classes or more advanced mathematics and physics, but I wanted to be pragmatic about the amount of time I have each week to put in to study outside of my normal (very busy) work week. I imagine anyone reading this would understand and very likely agree.

I examined MIT’s course catalog. They have 4 undergraduate programs in the Department of Electrical Engineering and Computer Science:

6-1 program: Leads to the Bachelor of Science in Electrical Science and Engineering. (Electrical Science and Engineering)
6-2 program: Leads to the Bachelor of Science in Electrical Engineering and Computer Science and is for those whose interests cross this traditional boundary.
6-3 program: Leads to the Bachelor of Science in Computer Science and Engineering.(Computer Science and Engineering)
6-7 program: Is for students specializing in computer science and molecular biology.

Because I wanted to stick what I believed would be most practical for my work at End Point, I selected the 6-3 program. With my intended program selected, I also decided that the full course load for a bachelor’s degree was not really what I was interested in. Instead, I just wanted to focus on the computer science related courses (with maybe some math and physics only if needed to understand any of the computer courses).

So, looking at the requirements, I began to determine which classes I’d require. Once I had this, I could then begin to search the MIT OpenCourseWare site to ensure the classes are offered, or find suitable alternatives on Coursera or Udemy. As is typical, there are General Requirements and Departmental Requirements. So, beginning with the General Institute Requirements, lets start designing a computer science program with all the fat (non-computer science) cut out.

General Requirements:

I removed that which was not computer science related. As I mentioned, I was aware I may need to add some math/science. So, for the time being this left me with:

Notice that it says

one subject can be satisfied by 6.004 and 6.042[J] (if taken under joint number 18.062[J]) in the Department Program

It was unclear to me what “if taken under joint number 18.062[J]” meant (nor could I find clarification) but as will be shown later, 6.004 and 6.042[J] are in the departmental requirements, so let’s commit to taking those two which would leave the requirement of one more REST course. After some Googling I found the list of REST courses here. So, if you’re reading this to design your own program, please remember that later we will commit to 6.004 and 6.042[J] and go here to select a course.

So, now on to the General Institute Requirements Laboratory Requirement. We only need to choose one of three:

- 6.01: Introduction to EECS via Robot Sensing, Software and Control
- 6.02: Introduction to EECS via Communications Networks
- 6.03: Introduction to EECS via Medical Technology

So, to summarize the general requirements we will take 4 courses:

6.004
6.042[J]
One course here
6.01:, 6.02:, or 6.03:

Major (Computer Science) Requirements:

In keeping with the idea that we want to remove non-essential, and non-CS courses, let’s remove the speech class. So here we have a nice summary of what we discovered above in the General Requirements, along with details of the computer science major requirements:

As stated, let’s look at the list of Advanced Undergraduate Subjects and Independent Inquiry Subjects so that we may select one from each of them:

Lastly, it’s stated that we must

Select one subject from the departmental list of EECS subjects

a link is provided to do so, however it brings you here and I cannot find a list of courses. I believe that this link no longer takes you to the intended location. A Google search brought up a similar page, but with a list of courses, as can be seen here. So, I will pick one from that page.

The next step was to find the associated courses on MIT OpenCourseWare

Sample List of Classes

So, now you will be able to follow the links I provided above to select your classes. I was not always able to find courses that matched by exact name and/or course number. Sometimes I had to read the description and look through several courses which seemed similar. I will provide my own list in case you’d just like to us mine:

Conclusion

So there you have it, please feel free to comment with any of your favorite resources.

Joe Mastey at Mountain West Ruby Conference 2015

2015-04-17T00:00:00+00:00

A conversation with a co-worker today about the value of improving one’s professional skills reminded me of Joe Mastey’s talk he gave at the 2015 Mountain West Ruby Conference. That then reminded me that I had never finished my write up on that conference. Blogger won’t let me install harp music and an animated soft focus flashback overlay, so please just imagine it’s the day after the conference when you’re reading this. “That reminds me of the time…”

I’ve just finished my second MWRC and I have to give this one the same 5-star rating I gave last year’s. There were a few small sound glitches here and there, but overall the conference is well-run, inclusive, and packed with great speakers and interesting topics. Rather than summarizing each talk, I want to dig into the one most relevant to my interests. “Building a Culture of Learning” by Joe Mastey

I was excited to catch Joe’s talk because learning and teaching have always been very interesting to me, regardless of the particular discipline. I find it incredibly satisfying to improve upon my own learning skills, as well as improving my teaching skills by teasing out how different individuals learn best and then speak to that. There’s magic in that one on one interaction when everything comes together just right. I just really dig that.

Joe’s work as the Manager of Internal Learning at Enova has gone way beyond the subtleties of the one-on-one. He’s taken the not-so-simple acts of learning and training, and scaled them up in an environment that does not sound, on paper, like it would support it. He’s created a culture of learning (“oh hey they just said the title of the movie in the movie!”) in a financial company that’s federally regulated, saw huge growth due to an IPO, and had very real business-driven deadlines for shipping their software.

Joe broke his adventure down into three general phases after refreshingly admitting that “YMMV” and that you can’t ignore the existing corporate culture when trying to build a culture of learning within.

Phase 1—Building Credibility

I would hazard a guess that most software development shops are perpetually at Phase 1: Learning is mostly ad-hoc by way of picking things up from one’s daily work; and has little to no people pushing for more formal training. People probably agree that training is important, but the mandate has not come down from the CTO, and there’s “no time for training” because there’s so much work to do.

How did Joe help his company evolve past Phase 1? Well, he did a lot of things that I think many devs would be happy to just get one or two of at the their company. My two favorites from his list probably appeal to polar opposite personality types, but that’s part of why I like them.

My first favorite is that books are all-you-can-eat. If a developer asks Joe for a tech book, he’ll say yes, and he’ll buy a bunch of extra copies for the office. I like having a paper book to read through to get up to speed on a topic, ideally away from my desk and the computer screen. I’ve also found that for some technologies, the right book can be faster and less frustrating than potentially spotty documentation online.

My second favorite is how Joe implemented “new hire buddies.” Each new hire is teamed up with an experienced dev from a different team. Having a specific person to talk to, and get their perspective on company culture, can really help people integrate into the culture much more quickly. When I joined End Point in 2010, I worked through our new hire “boot camp” training like all new hires. I then had the occasionally-maddening honor of working directly with one of the more senior evil super-geniuses at End Point on a large project that I spent 100% of my time on. He became my de facto new hire buddy and I could tell that despite the disparity in our experience levels, being relatively joined at the hip with someone like that improved my ramp-up and cultural integration time greatly.

Phase 2—Expand Reach and Create Impact

If my initial guess about Phase 1 is correct, it follows that that dev shops in Phase 2 are more rare: More people are learning more, more people are driving that learning, but it’s still mostly focused on new hires and the onboarding process.

Phase 2 is where Joe’s successful efforts are a little more intimidating to me, especially given my slightly introverted nature. The efforts here scale up and get more people speaking publically, both internally and externally. It starts with a more formal onboarding process, and grows to things like weekly tech talks and half day internal workshops. Here is where I start to make my “yeah, but…” face. We all have it. It’s the face you make when someone says something you don’t think can work, and you start formulating your rebuttal immediately. E.g. “Yeah, but how do you get management and internal clients to be OK with ‘shutting down development for half a day’ for training?” Joe does mention the danger of being perceived as “wasting too much time.” You’ll want to be sure you get ahead of that and communicate the value of what you’re spending “all that dev time on.”

Phase 3—Shift The Culture

It would be interesting to know how many shops are truly in Phase 3 because it sounds pretty intense: Learning is considered part of everyone’s job, the successes from the first two phases help push the culture of learning to think and act bigger, the acts of learning and training others are part of job descriptions, and things like FOSS contributions and that golden unicorn of “20% personal project time” actually happen on company time. Joe describes the dangers or downsides to Phase 3 in a bit of a “with great power comes great responsibility” way. I’ve never personally worked somewhere that’s in Phase 3, but it make sense that the increased upside has increased (potential) downside.

At End Point, we have some elements of all three phases, but we’re always looking to improve. Joe’s talk at MWRC 2015 has inspired me to work on expanding our own culture of learning. I think his talk is also going to serve as a pretty good road-map on how to get to the next phase.