How to address data and architecture standards in agile development

One challenge agile leaders and teams face is how to define and follow data and architectural patterns and standards in agile development. There’s a belief that it’s hard to drive data and technical standards because agile teams work in sprints that are usually two to four weeks long, and product owners generally overbook the backlog with prioritized features. Standards take time to develop; following them requires teams to have sufficient time to plan technical implementations.

Agile teams that execute in one sprint and plan only the next one will have a hard time using standards to formulate their development plans. If documented standards aren’t easy to follow or reference, then teams are less efficient, and it’s harder to train new developers on best architecture and data practices. It’s like a team wandering in the forest without a map or a GPS; they might be able to get to the next trailhead but they won’t know if they’re heading down an optimal path to get back to town.

What data and architecture problems need attention

It might help to put data and architecture standards in two categories:

• Standard architectures, such as data models, data pipelines, technology to enable a microservices architecture, standardized CI/CD (continuous integration and continuous delivery) pipelines, or proof of concepts around new technologies. These require upfront engineering work.
• Standard practices, including naming conventions, testing requirements, microservice interface standards, and usability patterns. These guide agile teams on how to implement features and address technical debt. They may also include process standards that define how to extend data models, validate CI/CD pipeline improvements, or document a new microservice endpoint.

When a standard requires engineering work, then it’s best to define this work as epics, features, and stories in the agile backlog and assign them to appropriate teams. These teams should consider other application development teams as their customers and should define acceptance criteria around their work. The product owner for this development can be a data, application, or solution architect who strives to deliver a component that’s easy for agile teams to use and delivers business value.

On the other hand, when standards provide data and architecture guidance to development teams, then these standards should form a basis of how developers implement user stories. This will require teams to have a strong working knowledge of these standards, and perhaps an easy-to-use knowledgebase for team leads and members to review.

Agile development requires continuous planning

Many agile teams hold their planning meetings at the start of the sprint. They review the prioritized user stories, estimate them, and commit to working on them during the sprint.

This approach works when teams’ priorities are making small improvements to existing applications. However, if they are working on new features and want to align with data and architecture standards, then just-in-time planning isn’t sufficient. Teams don’t have enough time to review user stories and align the implementation to standards in one sit-down right before the sprint is about to begin.

Teams driving toward standards need to plan earlier. I prefer calling the meeting at the start of a sprint a “commitment meeting” where teams finalize what stories they are working on. The vital work to plan the implementation of these stories is scheduled in one or more “planning meetings.”

Ideally, teams establish a continuous agile planning process where they review epics, features, and user stories on an ongoing basis. More complex work items requiring more planning time are scheduled more than one sprint before scheduled implementation so that teams can complete thorough development plans; smaller items can go through the process faster. Most importantly, meeting sooner takes the time pressure off teams; thus, they are more likely to consider standards because there is sufficient time to plan the implementation.

Develop reference architectures and data models

One way to align agile teams is to develop reference architectures and data models that depict current state, near-term future state, and longer-term objectives. These diagrams serve as a roadmap for development teams so that they know how best to align their implementations to architecture and data standards.

Documented reference architectures are single-page diagrams with color codes and other symbols that separate current state from future states. To fit them on a single page, architects should define a scope of components that are related and depict the end-to-end servicing of one or more applications.

Reference data models may include multiple diagrams depending on how data is used in the organization. They often include the following:

• A conceptual data model depicting business entities, relationships, and essential transactions.
• An analytic model for how data is centralized in data lakes or data warehouses and used for analytics, AI experiments, and data visualizations.
• A data integration model showing data sources, the critical transformations performed on data loaded from them, and the primary databases where the data is stored.
• A service model showing how microservices and other APIs connect to databases.

These diagrams provide agile teams with a starting point for understanding standards and future directions. Architects should supplement them with more details such as API documentation, data dictionaries, and a listing of subject matter experts for each architecture and data component.

Write acceptance criteria that reference standards

User stories should describe the what, why, and for whom of the requirement. Ideally, the product owner shouldn’t be documenting how the story should be implemented as this is the job of architects, software developers, and testers to define. Teams should be accountable for delivering the functionality and ensuring that the implementation meets architecture, data, security, devops, and other standards.

Just documenting standards and reference architectures is not sufficient to get teams to adhere to them. Teams are under too much pressure to develop code and complete releases, so reviewing standards is not always top of mind. Architects should take on the responsibility of reviewing user stories, meeting with teams to share learning, and aligning the implementation to appropriate standards by writing acceptance criteria into the story.

Also, software development managers should discuss acceptance criteria and standards with their teams to ensure they follow best practices, and that implementations align with future architecture and data standards.

Larger organizations will have to consider multiple approaches to align agile teams to data and architecture standards. Defining standards, planning ahead of the sprint, writing architecture-driven acceptance criteria, and defining responsibilities are some of the practices that can enable teams to deliver new functionality that aligns with architecture.