The development community has recognized for a long time that involving subject matter experts in the software development process is a critical factor to success. Techniques like joint application development (JAD) formalized this almost 30 years ago, and lately the agile methods have rules to have a customer available at all times for consultation on the requirements.
Getting the subject matter experts involved is difficult, because today we only have indirect ways of engaging them in the development process. The communication is based on informal ways like verbal conversations, reading and writing specs, feedback sessions and other non-machine processable ways. Subject matter experts rarely do the actual software programming. Furthermore, many different experts are needed to shape the requirements for more sophisticated applications. For example, an application for a modern weapon system could require experts in dozens of subject matters.
But even if you have the experts involved it is hard for them to have the right impact on the software. Let me share a personal story. The project had been going for about 12 months when I came in as a consultant. The team had developed many prototypes for the customer, but very little in terms of an application the customer could use. The development team complained that the customer did not know what they wanted. The customer, on the other hand, complained that the developers were just showing useless prototypes of screens that had little to do with the business problem the customer was trying to solve (a new insurance sales process). It was obvious that the project was spinning wheels building ever more useless prototypes, and the frustrations on both sides were escalating (as was the budget of the project).
So what did we do? We started to do use cases. This suddenly put the customer in charge of articulating what they wanted out of the system using their own words. As many of you have noticed working with use cases, the subject matter experts really feel comfortable describing their own use cases. Many times the developers jumped up and started to interrupt explaining how the screens, the database and use cases could be built - so we had to restrain them for a short while… The subject matter experts were thus able to articulate what they needed out of the system in their own business terminology almost free from technical jargon. After a while we had a set of use cases that began to stabilize and a shared consensus emerged within the customer group and the development group of what we needed to build. We started on an iteration plan to deliver the functionality in increments. As new functionality was deployed, the use cases also were refined with ever better understanding from the various project stakeholders.
As evident from projects like this, users and subject matter experts have very few ways to express their continuously evolving expectations and needs clearly enough to be directly usable by the software developer team.
Use cases help to give the subject matter expert a tool in context. It is definitely a step forward from the traditional approach in which a requirements document enumerates long lists of seemingly unrelated “shall” requirements. But use cases are still indirect – they require subjective interpretations by the development team of the subject matter experts’ intentions. Use case descriptions, as currently practiced, are not formal enough to be processable.
Subject matter experts do not lack ways of expressing themselves in their own terms. Quite the contrary, many domains use way more sophisticated nomenclature, notations, models and languages than we do in the software development community. We as developers typically do not understand their domain knowledge in enough detail to be able to implement their precise intentions and needs. Similarly, subject matter experts typically do not understand our software in enough detail to be able to communicate the subject matter details in ways that we need in order to do the implementation they want.
This communication gap is one of the most difficult problems in software development, and causes many projects to fail. Two methods of bridging this gap are being tried today.
The first method is to have the experts do their own programming (end user programming). Specialized tools like Excel or CAD systems show that non-programmers can be fairly effective building their own solutions with the right conceptual user models, although these are not truly programming tools. Although there is plenty of research in the field, End User Programming has many hurdles to overcome to be successful in a general way.
The second method is to educate the programmer to become a subject matter expert. Many organizations are able to do this over a long period of time, and it works in some domains. But again, that method is not practical in a larger, general scale.
So how can we cut this Gordian knot of software development? The fact remains that software needs to serve the large scale, general needs of software customers. So the question becomes: “Is there an alternative process by which we can give our subject matter experts more direct, immediate and first hand influence over the software to be built by programmers?”
We believe there is. By separating domain expression and software expression as much as possible, we can give the subject matter experts the right tools to express their business requirements in a way that can be directly used by the programmer. For example, a "super use case editor" could be used to express subject matter experts’ intentions for the software to be built. Similar to the output of a CAD editor or a spread sheet, a super use case editor’s output would not need to be executable in its own right. But it could record information in ways that are processable by the programmers. Obviously such an editor would need to be very domain specific and specialized based on the problem area at hand.
For this process to be feasible, the programmers still need to be actively involved working with the subject matter experts. But rather than interpreting verbal intentions of the subject matter experts and trying to implement them directly in code, we can factor the two issues. First a super use case editor, what we call an intentional editor, could be configured to fit the subject matter experts’ own terminology and other notations. The editor could then record their intentions in a deep semantic way we call domain code. This domain code could be processed with generative programming techniques that define the programmers’ knowledge of how to build the type of application at hand. Obviously, getting this process right the first time is as elusive as in any complex software development, and the need to support an evolving, changing system defined in the domain code and generators is as critical here as in any system.
In a future essay we will discuss this process of perfecting the relationship between subject matter experts and programmers in more detail.
Recent Comments