Lessons I learned about Agile Software Development
Experience report from a project with a big industry company
Note: I drafted this post about 3 years ago. I don’t know why I never published it. Some days ago I already published a post with a similar history. As I want to bring my blogging activity back to life I will keep finishing up and publishing older drafts besides writing completely new ones. I didn’t further edit the draft besides removing todo comments and correcting some typos. Just keep in mind that stuff may be out of date and have fun.
I am so happy to see that more and more big companies, which used to have heavyweight processes regarding software engineering, are moving to more agile software engineering methodologies.
Right now in my job, I have to engineer a piece of software for a customer with a small team trying to be as agile as possible. Fortunately, the customer supports agile software engineering, but still, there we encountered some challenges.
In this article, I want to share my experience realizing that project and the lessons learned from doing that.
Sophisticated Software Engineering
I have a pretty specific idea of how sophisticated software engineering should look, which might differ from your point of view. Short version, I think software should be developed in small cycles where engineers show a DevOps mindset.
Feedback loops
It is all about feedback loops. Feedback loops on different abstraction layers.
- Continuous Integration with automated tests gives feedback about whether new modifications on the codebase are working the existing code.
- Cycles of development time frames in combination with demos for the customer give feedback about whether the requirements match or not.
- Retrospectives give feedback about whether the process itself should be improved.
You apply the mantra Inspect and Adopt and the more cycles you have, the more you can actually and respond to change. I think this is what it means to be agile. You constantly pull yourself out of the comfort zone, or even better you can prevent yourself from even getting to the comfort zone.
DevOps
Yes I know, you are tired of buzzwords. Anyway, I have to mention it, because, in my opinion, it is a very important engineering mindset or culture.
In classic software engineering, software is developed by developers and operated by operators or admins. The problem with that is:
- developers usually have no clue about operating software
- operators usually don’t know anything about the implementation of it.
This makes debugging and releasing the software kind of heavyweight process, which is referred to as Gap between developer and operations.
This gap somehow prevents us from being agile because it enlarges the development cycles, which by the way is the reason why those “Big Bang” deployments are most likely used in classic software engineering.
So … you will probably find several definitions out there of what DevOps exactly is, but we will look at it as a mindset of engineers who control the whole software lifecycle from development to operation.
You build it, you run it!
Challenges
Knowing about how to do sophisticated software engineering, unfortunately, is not enough to also actually do it, there are different challenges you will face especially when working with a customer. I will discuss these in the following sections.
Divergence of software engineering practice
So in the company, I am working for, we know about how to engineer software in an agile way, as we are implementing the Kanban framework for our software engineering process.
Our customer uses Scrum which is also an agile software engineering framework but is slightly more strict and has more ceremonies than Kanban.
Even though there are some similarities in both the Kanban and Scrum framework which can be described as abstract agile principles, basically the principles of the agile manifesto, people tend to stick to specific details of the respective framework instead of following the agile principles behind them.
So the challenge for us here is to somehow synchronize the customer and our team regarding the software engineering process. Not only because of the different approaches and understanding of agile software engineering practices but also because we have different separate boards to visualize our work, this is a very challenging task.
The Gap between Dev and Ops
Even though we know that the gap between developer and operations is a bad thing, working with a customer you will most likely have to face this situation. The reason for that is, that customers will probably want to use their infrastructure while not being able to give you access to it you would need.
In our case the customer uses AWS and we have only restricted access, but they supported us with a such called Infrastructure Engineer who should help us with the tasks regarding the infrastructure.
The challenge here is communication. On the one hand, we have to communicate what we need for the software we are implementing to be operated. But on the other hand, he also has to communicate to us what exactly he is doing, because in the end “we build it, we run it” or “we build, we fix it if it fails”.
The more this knowledge gap is diverging, the more you will fall into old patterns of engineering software and the more rigid your process will be.
Underestimating tasks
Estimation is a hard topic. Some people out there completely rely on the estimation of tasks while mostly having some kind of deadline, while some people say completely refuse to give estimations and work with deadlines.
And here you have the problem. Most customers want you as engineers to give estimations and work with deadlines. But, this contradicts the agile software development approach where you show the progress of the project on a regular base.
To cut a long story short, you will most probably have to estimate and work with deadlines, as we had to, and I guarantee that you will underestimate and overestimate tasks.
Scope Creep
We refer to scope creep as changing of requirements or scopes. We all know (at least I hope so) that it is simply not possible to specify every single requirement of a software system upfront, which has different reasons.
The customer himself will not know them all up front, they will change and even then, information gets lost while communicating them to the engineers, just to name some problems.
Scope Creep will happen and it will happen in the most unpleasant situations, in the middle of a sprint or even some days before a deadline. It will make you feel upset but you will have to somehow deal with it.
Solutions
So, we had a quite tough job to overcome these challenges, but here is how we tried to tackle them.
It is very important to understand that finding solutions for the different challenges resulted in constantly reflecting on the work we are doing and being willing to adopt as well as being honest to ourselves.
Daily Standups
We started the project with two engineers on our side, so we didn’t need a daily synchronization on the current progress or occurring impediments as communication between us was very regular and lightweight.
As soon as we realized that there will be this gap between implementing and operating the software I mentioned before, we decided to do a daily standup to synchronize with the Infrastructure Engineer.
Ops Showcase
As the project proceeded we realized that, even though we synchronized daily with Infrastructure Engineer and also did pairing sessions with him, the gap began to get bigger and bigger.
To tackle that problem we decided to have such called ops showcases, where the Infrastructure Engineer introduces us in the work he did to operate the software we implemented.
Task Grooming
As already mentioned, task estimation is a tricky topic and we found ourselves giving wrong estimates on tasks or even forgetting to specify every single detail about a task while creating tickets on our board.
Therefore we started to do a weekly technical Task Grooming where we went through all tickets and checked for the right description and tried to estimate them right before Sprint Planning. This way we tried to compensate wrong specification and estimation as well as making our work more precise and predictable.
Note: A precondition for this is that the backlog is prioritized by the Product Owner
Sprint Planning
While engineering a piece of software in a customer project, the customer will most likely want to see the current progress of the project and have a certain degree of predictability of how the progress will in the near future.
In the Sprint Planning session, which takes place at the beginning of each sprint we create transparency about all the tickets on our board(especially the ones in the backlog) and let the customer decide which tickets should be worked on in the upcoming sprint. At the same or directly afterward we, as the team implementing the software, decided how much of the chosen tickets we can achieve.
This way has little to no surprise regarding the outcome, the progression of the work we are going to achieve, as we all come to a commitment about the work we think can be achieved in every sprint.
By the way, do you remember what I told you about the connection of feedback loops and the ability to engineer software in an agile way? Well having sprints(with Sprint Planings and Demos) also allows the customer to be agile as he can also inspect and adapt based on the results of every sprint.
Retrospective
Doing Retrospectives on a regular bases is probably the most important ceremony we have in agile software engineering. We want to get feedback about how the team feels and performs, as well as figuring out how we can improve.
The Retrospective usually takes place at the end of each sprint. There are different ways how to do a Retrospective but basically, you want the team to reflect and them to think about ways to optimize their work.
In our Retrospective, we first catch the general feeling about the last sprint of every team member by letting them choose one of the following smileys to express their feelings: :), :/, :(. This way we can get a very basic first impression.
After that, we show data like how many tickets were done and the divergence of the estimated and real effort for every done ticket. Doing that allowed us to get a feeling about the quality of our estimations.
The last part of the retrospective is probably the most important and valuable. Each member notes his/her lessons learned as well as things the team should start, stop, or continue doing. Now we try to extract Action Items out of those notes and assign a team member to it, which is then responsible for the progress of that respective item. On each Retrospective, we also check which of the Action Items of the last Retrospective were achieved to decide if they are still important for the team or can be neglected.
Note: The last part is not a complaining session. Instead of just complaining, try to formulate your complaint positive, as something the team should start doing. This way you avoid team members being offended.
Lessons Learned
This project was an awesome opportunity to move from a DevOps mindset to a NoOps mindset and to play around with serverless technologies. But don’t get me wrong this was a serious project for a big customer which is now used in production.
In my opinion, we not only learned a lot about serverless technology and AWS but also we got a lot of knowledge of how to do agile software engineering in a better way. Let’s summarize the most important things:
- Embrace Transparency
- Try to achieve predictability
- Communication is key
- Fail fast, escalate fast
- Sync, Sync, Sync
