Wednesday, December 18, 2013

Software Engineering at DealerOn

During interviews or in conversations with other tech folks, a lot of people want to know what software development is like here at DealerOn. Unlike my colleague Frank DeCaire’s posting “Working at DealerOn” which talks about our technology and growth this one talks a bit about our practices and processes which permit us to deliver software. 


As we have grown our team during all of 2013, it was clear that we needed to formalize our software development practices. We settled on Kanban as our chosen software development methodology. We chose Kanban because it works hand-in-hand with continuous delivery practices (more on that in a moment), it allows us to always work on the most important thing for the business (they change their mind a lot), and it is less estimate / metric driven than other Agile approaches.

The single hardest part about adopting Kanban is moving away from giving firm completion dates. In our pre-Kanban days we used to do this and unsurprisingly we would routinely miss them for the usual reasons of unknown complexities or change in priorities. We now have been doing Kanban long enough to provide forecasts of when something may be done based on our past performance. As priorities change, so does the forecast.

There are certain commitments for customers or things we need to get done by a trade show. This is reflected by prioritizing those items at the top or near-top of the queue so that they will get done before other things.

Bugs and Enhancements vs. Projects

This gets a little bit more into the nuts-and-bolts of our Kanban implementation. We maintain two separate backlogs. One has our projects (and all technical tasks and pre-release bugs chained to them) and another has our post-release bugs and enhancements (which are non-bug small improvements to the software).
Generally everyone on the team works on projects. As a team, we typically review new projects and task them out together. This allows everyone to know what is going on, make suggestions, and contribute to the project. In reality, it works out that one or two people carry a project to completion with limited involvement by other team members, but others can get involved at any time to push it forward to completion. All team members are involved with all areas of our platform (we’re still small enough for this to work) so it is important for everyone to be knowledgeable about every project regardless of whether or not they work on it.
Each day one person (according to daily rotation) does not work on projects. They instead work on bugs and enhancements. Also they serve as the “point of contact” for our customer teams if they have questions. In true Kanban style, the bugs are taken in order of priority and a lack of knowledge on that system is not a pass at fixing it. This has resulted in almost no silos of knowledge about all but perhaps a handful of our legacy systems. One non-Kanban twist is that we flag certain bugs and enhancements as “quick” and action those first which prevents fifteen-minute things (e.g. “Change the label from ‘New vehicles:’ to ‘Vehicles, New:’) from clogging the backlog.

Continuous Delivery 

DealerOn has always been, and continues to be, a continuous delivery company and this goes in hand with Kanban nicely, which has no real notion of release dates. We release the various components of our software once or sometimes multiple times per day. This is enabled through fully scripted packaging and deployment processes which permit us to deploy with a single click of a button to any environment, all orchestrated through Team City. A variety of automated tests at all levels (unit, integration, and end-to-end) as well as server and log monitoring help ensure we don’t introduce any regression bugs. Rollback (or “rolling forward”) is just as easy which serves as an excellent insurance policy in the event of an undetected issue.
The business owners and our customers love this because they can see something that was broken get fixed or something small that they asked for in just a few hours or a day. We never have to say “it is fixed but our next release is on the 1st” which is sure to disappoint.

One issue this can cause is that it is difficult to ensure everyone is aware of all the changes to the software, especially as the teams grow. We need to work on better ways of creating release notes and ensuring that they are digested by everyone.

From a development perspective, this requires a slightly different mindset. Developers commit knowing that their changes will be tested immediately and may go live. There is no “tightening for the release” period often found in classic waterfall and many Scrum teams. Good test coverage, code reviews, and adherence to coding standards -- discipline -- are important to ensure successful releases and maintainability. All of our team members have ownership of the packaging and deployment systems as well as our infrastructure – not just a “release manager” or the IT folks.
We use branches for larger code changes, typically done for projects but is sometimes done for certain bugs. The determination of what gets done on a branch is part art and science, but as a general rule if multiple people need to work on something before something goes live we use a branch. We use Mercurial as our source control system, and it works very well with Team City which conducts our automated builds, testing, and deployment.


I hope that this article provides some insight into what software delivery is like at my current company, and perhaps you'll find it useful if you are looking to learn more about us or you are looking into adopting Kanban practices at your organization.  If you have any experiences to share or have any questions, feel free to get in touch with me.