Reference: Sommerville, Engineering Software Products, Chapter 2

 

Software products must be brought to market quickly so rapid software development and delivery is essential. Virtually all software products are now developed using an agile approach. Agile software engineering focuses on delivering functionality quickly, responding to changing product specifications and minimizing development overheads. A large number of 'agile methods' have been developed. There is no 'best' agile method or technique. It depends on who is using the technique, the development team and the type of product being developed.

Plan-driven development evolved to support the engineering of large, long-lifetime systems (such as aircraft control systems) where teams may be geographically dispersed and work on the software for several years. This approach is based on controlled and rigorous software development processes that include detailed project planning, requirements specification and analysis and system modelling. However, plan-driven development involves significant overheads and documentation and it does not support the rapid development and delivery of software. Agile methods were developed in the 1990s to address this problem. These methods focus on the software rather than its documentation, develop software in a series of increments and aim to reduce process bureaucracy as much as possible.

Agile Manifesto

All agile methods are based around incremental development and delivery. Product development focuses on the software features, where a feature does something for the software user. With incremental development, you start by prioritizing the features so that the most important features are implemented first. You only define the details of the feature being implemented in an increment. That feature is then implemented and delivered. Users or surrogate users can try it out and provide feedback to the development team. You then go on to define and implement the next feature of the system.

Agile development principles

• Involve the customer: Involve customers closely with the software development team. Their role is to provide and prioritize new system requirements and to evaluate each increment of the system.
• Embrace change: Expect the features of the product and the details of these features to change as the development team and the product manager learn more about the product. Adapt the software to cope with changes as they are made.
• Develop and deliver incrementally: Always develop software products in increments. Test and evaluate each increment as it is developed and feed back required changes to the development team.
• Maintain simplicity: Focus on simplicity in both the software being developed and the development process. Wherever possible, do what you can to eliminate complexity from the system.
• Focus on people, not the development process: Trust the development team and do not expect everyone to always do things in the same way. Team members should be left to develop their own ways of working without being limited by prescriptive software processes.

The most influential work that has changed software development culture was the development of Extreme Programming (XP). The name was coined by Kent Beck in 1998 because the approach was developed by pushing recognized good practice, such as iterative development, to 'extreme' levels. Extreme programming focused on 12 new development techniques that were geared to rapid, incremental software development, change and delivery. Some of these techniques are now widely used; others have been less popular.

Widely adopted XP practices

• Incremental planning / User Stories: There is no "grand plan" for the system. Instead, what needs to be implemented (the requirements) in each increment are established in discussions with a customer representative. The requirements are written as user stories. The stories to be included in a release are determined by the time available and their relative priority.
• Small releases: The minimal useful set of functionality that provides business value is developed first. Releases of the system are frequent and incrementally add functionality to the previous release.
• Test-driven development: Instead of writing code and then tests for that code, developers write the tests first. This helps clarify what the code should actually do and that there is always a "tested" version of the code available. An automated unit test framework is used to run the tests after every change. New code should not "break" code that has already been implemented.
• Continuous integration: As soon as the work on a task is complete, it is integrated into the whole system and a new version of the system is created. All unit tests from all developers are run automatically and must be successful before the new version of the system is accepted.
• Refactoring: Refactoring means improving the structure, readability, efficiency, and security of a program. All developers are expected to refactor the code as soon as potential code improvements are found. This keeps the code simple and maintainable.

Software company managers need information that will help them understand how much it costs to develop a software product, how long it will take and when the product can be brought to market. Plan-driven development provides this information through long-term development plans that identify deliverables - items the team will deliver and when these will be delivered. Plans always change so anything apart from short-term plans are unreliable. Scrum is an agile method that provides a framework for agile project organization and planning. It does not mandate any specific technical practices.

Scrum terminology:

• Product: The software product that is being developed by the Scrum team.
• Product Owner: A team member who is responsible for identifying product features and attributes. The Product Owner reviews work done and helps to test the product.
• Product backlog: A to-do list of items such as bugs, features, and product improvements that the Scrum team has not yet completed.
• Development team: A small self-organizing team of five to eight people who are responsible for developing the product.
• Sprint: A short period, typically two to four weeks, when a product increment is developed.
• Scrum: A daily team meeting where progress is reviewed and work to be done that day is discussed and agreed.
• ScrumMaster: A team coach who guides the team in the effective use of Scrum.
• Potentially shippable product increment: The output of a sprint that is of high enough quality to be deployed for customer use.
• Velocity: An estimate of how much work a team can do in a single sprint.

Key roles in Scrum:

• The Product Owner is responsible for ensuring that the development team are always focused on the product they are building rather than diverted into technically interesting but less relevant work. In product development, the product manager should normally take on the Product Owner role.
• The ScrumMaster is a Scrum expert whose job is to guide the team in the effective use of the Scrum method. The developers of Scrum emphasize that the ScrumMaster is not a conventional project manager but is a coach for the team. They have authority within the team on how Scrum is used. In many companies that use Scrum, the ScrumMaster also has some project management responsibilities.

In Scrum, software is developed in sprints, which are fixed-length periods (2-4 weeks) in which software features are developed and delivered. During a sprint, the team has daily meetings (Scrums) to review progress and to update the list of work items that are incomplete. Sprints should produce a 'shippable product increment'. This means that the developed software should be complete and ready to deploy.

Key Scrum practices

• Product backlog: A to-do list of items to be implemented that is reviewed and updated before each sprint.
• Timeboxed sprints: Fixed-time (2-4 week) periods in which items from the product backlog are implemented.
• Self-organizing teams: Self-organizing teams make their own decisions and work by discussing issues and making decisions by consensus.

The product backlog is a list of what needs to be done to complete the development of the product. The items on this list are called product backlog items (PBIs). The product backlog may include a variety of different items such as product features to be implemented, user requests, essential development activities and desirable engineering improvements. The product backlog should always be prioritized so that the items that be implemented first are at the top of the list.

Product backlog item states:

• Ready for consideration: These are high-level ideas and feature descriptions that will be considered for inclusion in the product. They are tentative so may radically change or may not be included in the final product.
• Ready for refinement: The team has agreed that this is an important item that should be implemented as part of the current development. There is a reasonably clear definition of what is required. However, work is needed to understand and refine the item.
• Ready for implementation: The PBI has enough detail for the team to estimate the effort involved and to implement the item. Dependencies on other items have been identified.

Product backlog activities:

• Refinement: Existing PBIs are analyzed and refined to create more detailed PBIs. This may lead to the creation of new product backlog items.
• Estimation: The team estimate the amount of work required to implement a PBI and add this assessment to each analyzed PBI.
• Creation: New items are added to the backlog. These may be new features suggested by the product manager, required feature changes, engineering improvements, or process activities such as the assessment of development tools that might be used.
• Prioritization: The product backlog items are reordered to take new information and changed circumstances into account.

PBI estimation metrics:

• Effort required: This may be expressed in person-hours or person-days i.e. the number of hours or days it would take one person to implement that PBI. This is not the same as calendar time. Several people may work on an item, which may shorten the calendar time required.
• Story points: Story points are an arbitrary estimate of the effort involved in implementing a PBI, taking into account the size of the task, its complexity, the technology that may be required and the 'unknown' characteristics of the work. They were derived originally by comparing user stories, but they can be used for estimating any kind of PBI. Story points are estimated relatively. The team agree on the story points for a baseline task and other tasks are estimated by comparison with this e.g. more/less complex, larger/smaller etc.

Products are developed in a series of timeboxed sprints, each of which delivers an increment of the product or supporting software. Sprints are short duration activities (1-4 weeks) and take place between a defined start and end date. Sprints are timeboxed, which means that development stops at the end of a sprint whether or not the work has been completed. During a sprint, the team work on the items from the product backlog.

Sprint activities:

• Sprint planning: Work items to be completed in that sprint are selected and, if necessary, refined to create a sprint backlog. This should not last more than a day at the beginning of the sprint.
  • Establish an agreed sprint goal. Sprint goals may be focused on software functionality, support or performance and reliability.
  • Decide on the list of items from the product backlog that should be implemented.
  • Create a sprint backlog. This is a more detailed version of the product backlog that records the work to be done during the sprint.
  • In a sprint plan, the team decides which items in the product backlog should be implemented during that sprint. Key inputs are the effort estimates associated with PBIs and the team's velocity.
  • The output of the sprint planning process is a sprint backlog. The sprint backlog is a breakdown of PBIs to show the what is involved in implementing the PBIs chosen for that sprint.
  • During a sprint, the team have daily meetings (scrums) to coordinate their work.
• Sprint execution: The team work to implement the sprint backlog items that have been chosen for that sprint. If it is impossible to complete all of the sprint backlog items, the sprint is not extended. The unfinished items are returned to the product backlog and queued for a future sprint.
• Sprint reviewing: The work done in the sprint is reviewed by the team and (possibly) external stakeholders. The team reflect on what went well and what went wrong during the sprint with a view to improving their work process.
  • Sprint review: The product owner has the ultimate authority to decide whether or not the goal of the print has been achieved. They should confirm that the implementation of the selected product backlog items is complete.
  • Sprint retrospective: The team reviews its own process review and reflects on how Scrum has been used. The aim is to identify ways to improve and to discuss how to use Scrum more productively.

The ideal Scrum team size is between 5 and 8 people. Teams have to tackle diverse tasks and so usually require people with different skills, such as networking, user experience, database design and so on. They usually involve people with different levels of experience. A team of 5-8 people is large enough to be diverse yet small enough to communicate informally and effectively and to agree on the priorities of the team. The advantage of a self-organizing team is that it can be a cohesive team that can adapt to change. Because the team rather than individuals take responsibility for the work, they can cope with people leaving and joining the team. Good team communication means that team members inevitably learn something about each other's areas.

The developers of Scrum assumed that teams would be co-located. They would work in the same room and could communicate informally. Daily scrums mean that the team members know what's been done and what others are doing. However, the use of daily scrums as a coordination mechanism is based on two assumptions that are not always correct. Scrum assumes that the team will be made up of full-time workers who share a workspace. In reality, team members may be part-time and may work in different places. For a student project team, the team members may take different classes at different times. Scrum assumes that all team members can attend a morning meeting to coordinate the work for the day. However, some team members may work flexible hours (e.g. because of childcare responsibilities) or may work on several projects at the same time.

External interactions are interactions that team members have with people outside of the team. In Scrum, the idea is that developers should focus on development and only the ScrumMaster and Product Owner should be involved in external interactions. The intention is that the team should be able to work on software development without external interference or distractions.

In all but the smallest product development companies, there is a need for development teams to report on progress to company management. A self-organizing team has to appoint someone to take on these responsibilities. Because of the need to maintain continuity of communication with people outside of the group, rotating these activities around team members is not a viable approach. The developers of Scrum did not envisage that the ScrumMaster should also have project management responsibilities. In many companies, however, the ScrumMaster has to take on project management responsibilities. They know the work going on and are in the best position to provide accurate information and project plans and progress.

• Intro to Scrum in Under 10 Minutes by Hamid Shojaee