Getting the experts involved

The development community has recognized for a long time that involving subject matter experts in the software development process is a critical factor to success. Techniques like joint application development (JAD) formalized this almost 30 years ago, and lately the agile methods have rules to have a customer available at all times for consultation on the requirements.

Getting the subject matter experts involved is difficult, because today we only have indirect ways of engaging them in the development process. The communication is based on informal ways like verbal conversations, reading and writing specs, feedback sessions and other non-machine processable ways. Subject matter experts rarely do the actual software programming. Furthermore, many different experts are needed to shape the requirements for more sophisticated applications. For example, an application for a modern weapon system could require experts in dozens of subject matters.

But even if you have the experts involved it is hard for them to have the right impact on the software. Let me share a personal story. The project had been going for about 12 months when I came in as a consultant. The team had developed many prototypes for the customer, but very little in terms of an application the customer could use. The development team complained that the customer did not know what they wanted. The customer, on the other hand, complained that the developers were just showing useless prototypes of screens that had little to do with the business problem the customer was trying to solve (a new insurance sales process). It was obvious that the project was spinning wheels building ever more useless prototypes, and the frustrations on both sides were escalating (as was the budget of the project).

So what did we do? We started to do use cases. This suddenly put the customer in charge of articulating what they wanted out of the system using their own words. As many of you have noticed working with use cases, the subject matter experts really feel comfortable describing their own use cases. Many times the developers jumped up and started to interrupt explaining how the screens, the database and use cases could be built - so we had to restrain them for a short while… The subject matter experts were thus able to articulate what they needed out of the system in their own business terminology almost free from technical jargon. After a while we had a set of use cases that began to stabilize and a shared consensus emerged within the customer group and the development group of what we needed to build. We started on an iteration plan to deliver the functionality in increments. As new functionality was deployed, the use cases also were refined with ever better understanding from the various project stakeholders.

As evident from projects like this, users and subject matter experts have very few ways to express their continuously evolving expectations and needs clearly enough to be directly usable by the software developer team.

Use cases help to give the subject matter expert a tool in context. It is definitely a step forward from the traditional approach in which a requirements document enumerates long lists of seemingly unrelated “shall” requirements. But use cases are still indirect – they require subjective interpretations by the development team of the subject matter experts’ intentions. Use case descriptions, as currently practiced, are not formal enough to be processable.

Subject matter experts do not lack ways of expressing themselves in their own terms. Quite the contrary, many domains use way more sophisticated nomenclature, notations, models and languages than we do in the software development community. We as developers typically do not understand their domain knowledge in enough detail to be able to implement their precise intentions and needs. Similarly, subject matter experts typically do not understand our software in enough detail to be able to communicate the subject matter details in ways that we need in order to do the implementation they want.

This communication gap is one of the most difficult problems in software development, and causes many projects to fail. Two methods of bridging this gap are being tried today.

The first method is to have the experts do their own programming (end user programming). Specialized tools like Excel or CAD systems show that non-programmers can be fairly effective building their own solutions with the right conceptual user models, although these are not truly programming tools. Although there is plenty of research in the field, End User Programming has many hurdles to overcome to be successful in a general way.

The second method is to educate the programmer to become a subject matter expert. Many organizations are able to do this over a long period of time, and it works in some domains. But again, that method is not practical in a larger, general scale.

So how can we cut this Gordian knot of software development? The fact remains that software needs to serve the large scale, general needs of software customers. So the question becomes: “Is there an alternative process by which we can give our subject matter experts more direct, immediate and first hand influence over the software to be built by programmers?”

We believe there is. By separating domain expression and software expression as much as possible, we can give the subject matter experts the right tools to express their business requirements in a way that can be directly used by the programmer. For example, a "super use case editor" could be used to express subject matter experts’ intentions for the software to be built. Similar to the output of a CAD editor or a spread sheet, a super use case editor’s output would not need to be executable in its own right. But it could record information in ways that are processable by the programmers. Obviously such an editor would need to be very domain specific and specialized based on the problem area at hand.

For this process to be feasible, the programmers still need to be actively involved working with the subject matter experts. But rather than interpreting verbal intentions of the subject matter experts and trying to implement them directly in code, we can factor the two issues. First a super use case editor, what we call an intentional editor, could be configured to fit the subject matter experts’ own terminology and other notations. The editor could then record their intentions in a deep semantic way we call domain code. This domain code could be processed with generative programming techniques that define the programmers’ knowledge of how to build the type of application at hand. Obviously, getting this process right the first time is as elusive as in any complex software development, and the need to support an evolving, changing system defined in the domain code and generators is as critical here as in any system.

In a future essay we will discuss this process of perfecting the relationship between subject matter experts and programmers in more detail.

Comments

[Petr Pan.]

In brief (if I understand it well enough), there is a wide gap between the user's/customer's/domain expert's/non-programmer's informal specification and the programmer's formal solution. One way to bridge this gap could be to give the user a tool (domain editor) by which she can express her requirements (use-cases) and thus she can directly participate on the development process that in turn saves programmer's expensive time.

BTW, the same idea was already published, eiher on http://osl.iu.edu/~tveldhui/papers/dagstuhl1998/
or on http://radio.weblogs.com/0114989/2003/03/09.html
and personally, the best it was summarized in this great Sergey's article - http://www.onboard.jetbrains.com/is1/articles/04/10/lop/

I'm sure, you know all these articles. However, what none of them answers and what is the most crucial part, is the WAY to enable this - non-programmers are not usually used to write or speak in formal way.
I'm eager to see what JetBrains and you are up to ;-)

[Magnus Christerson]

Hi Petr,
We have two things going on here:

First, the subject matter experts are usually perfectly capable of expressing their domain knowledge with fine precision as long as it's expressed in ways they are comfortable with, in their comfort zone. They have very detailed domain knowledge, and they are usually happy to express it in fine detail. For example, in the insurance system I mentioned above, the subject matter experts could define the various insurance policies, their commission structures, and the relevant legislations all in extraordinary detail. However, once we ask the subject matter experts to express themselves in ways that is more programmer friendly, or computable, they shy away; this is not their comfort zone, and this is a barrier. Use cases are bridging this gap by having a structure that is both friendly to the subject matter experts as well as directly usable by the programmer. Use case editors have been around for 15 years - I was involved in building the first one in the Objectory tool, and it was very succesful. Subject Matter experts are very comfortable working in semi structured text, complemented with simple graphics. However, the result of these use case editors are not processable in a general way as they rely on text, or simple node-arch type diagrams. So the content needs to be derived by the programmers through a conversation with the subject matter expert. This works, but there is a manual, subjective and non-processable step involved here.

Secondly, the demand for domain specific languages and language oriented programming are coming from the other direction; from the programming side of things. They typically support a computable language or at least processable language. Domain specific languages are already used by programmers, and will be even more so with the emerging language workbenches, to express suitable abstractions for the software to be built.

It is in the intersection of these two trends we see an oppurtunity. By staying in the comfort zone of the subject matter experts, but doing it in a way that is also directly usable by the programmer using generative programming techniques.

[Fidelio]

I think I understand your goals like having domain specific editors, getting in the experts and (my favorite) machine processable domain code:. Last one is the top and #1! Kicking our sales guys from Word and Excel to a well defined representation would be awesome...

Actually, the path and the tools are vague for me, so let me raise some questions:

What is the value of a monolithic intentional editor? What is the difference between the intentional editor and having a separate domain specific editors that outputs (standard?) domain code?

What do you mean on configuring your editor? Having a set of 'mini' editors that is put together for a specific domain or completely separate domain editors within intentional editor? For the latter case I guess you have a meta editor for editing domain editors, which is uhh.. edited by itself:)

Anyway, I can't imagine the intersection of all domain editors. Is there anything more than the operating system features?

thanks, fid

[Magnus Christerson]

Hello Fidelio,

The key to the puzzle is the common underlying representation for all domains - domains are distinguished mainly by the projections in which they are displayed.

"Monolithic" is not the impression that we create to the user, but it is technically correct - once the projections are different, the operations on the underlying representation can be identical in most cases. This makes it possible that a new domain can start with its "own" editor very rapidly, and benefit from undo, change history, names, integration etc.

When we talk about "configuration" we mean mainly the schema definition for the domain including the definition of the projections, the notations, although other aspects of the editor can be made domain specific as well.

[Zoltan "Hypo" Radanyi]

I think Magnus Christerson’s August 9th entry (“Getting the experts involved”) was the best explanation of intentional programming for laypeople so far. It made me wonder, however, whether the design of your web site itself couldn’t be used to clarify and reinforce the intentional software message. Given that design elements themselves are a kind of symbol system, clarity and consistency here might be helpful not just from a usability point of view, but in exemplifying the benefits of non-arbitrary codes and nomenclature. I was also thinking that some interactive illustrations of use cases, perhaps based in part on the graphical models of Alistair Cockburn (whose work I’ve only just found on the Internet) might help give visitors a better idea of the goals of intentional programming. As it stands now, the site doesn’t really generate the excitement you would associate with a programming revolution; it seems directed more at generative and aspect-oriented programmers (the already converted) than at the subject matter experts it’s designed to benefit, and the look and feel is a bit more like the past than the future. There’s got to be a better way to show “what you know but cannot prove” — that there’s a revolution going on in software development.

The site underwent some changes several months ago, and I think they were all for the better. The division of the pages into three parts (with the menu on the left), was one improvement, as was placing introductory excerpts of the most recent news articles on the home page. From a design point of view, I really like the logo, but I think it could be used more, and in more creative ways. For example, the first letter of the title of each page (e.g. “N” for “News” or “T” for “Technology) could reprise the logo by being inverted, boxed or highlighted in the same color as the original. This allusion would help keep the message of visible intentions alive beyond the home page.

Web sites do a better job of capturing and holding the visitor’s attention when the user sees the entire homepage at a glance and scrolling is neither possible nor necessary, the page itself providing the user’s focus. This could be accomplished by optimizing the resolution for the average user, but your home page lacks a central element, the kind of thing you would want users to focus on. (Generally, links to news articles, blogs and the like are not centerpieces, but, as supplemental information, are placed elsewhere.) The visual center could be provided by an image which suggests the values you’re striving for — for example, if Intentional Software is committed to facilitating the free flow of ideas in recognizable form from the minds of experts to programmers’ code and back again, an image could be chosen that would represent fluidity and unimpeded communication — at the simplest level, maybe a picture of a programmer and an architect looking together at a single screen shot, the proverbial “same page” for both parties to “be on”. (It wouldn’t have to be so literal, of course.)

A lot could be done with this site, both in the way the site provides information to the user (i.e., its plot or narrative, which could be adapted so users go through the site in a more determined order, creating both a bit of suspense and some insights into the possibilities of intentional programming), and with regard to the details. Design details often seem like nitpicks, but they’re important: for example, there needs to be consistency concerning font colors and sizes on the one hand, and information type and heading level, on the other. If links are in dark blue throughout the site, then this color shouldn’t be used for blog excerpts. If the home page introduces a color scheme where gray is dominant, the rest of the pages shouldn’t be predominantly black text against a white background. There is no logic here in the variation in font formats, colors and sizes, rendering these variations meaningless and clashing conceptually with the Intentional Software message of meaning made manifest in code (as I implied at the outset). I should add that, from a usability point of view, the light blue text is hard to read against the gray background.

As for the “big picture”, this might better be done by showing rather than telling. A semi-ordered progression through the site with interactive demos and explanation along the way is one possibility. But if an interactive feature involving some kind of intentional or use case editor isn’t feasible, then how about an animated simulation showing subject matter experts how intentional programming streamlines workflow, how collaboration is facilitated when the code needed to execute it isn’t a foreign language? (I’m imagining something like this: a comparative diagram showing two animated use-case flowcharts, one depicting traditional workflow and one depicting “intentional” workflow.)

As a designer without much of a background in programming, I’m all for comprehensible code and helping laypeople work more efficiently. When a friend showed me the “World Question” article eight or nine months ago, I found it interesting, but then when I got on your web site I was a little disappointed. When I went to the site again a few months later, I noticed some improvements and began thinking of ways it could be improved further. I know it’s a lot easier to tear down a house than it is to build one, but I’d imagine anyone who can challenge the whole programming industry can take a little respectful criticism.

Hypo

[Michael Clagett]

One of the questions that keeps surfacing in my mind as I read through your blog and also as I read through Martin Fowler's series of articles and his example of creating a domain-specific language with MPS is this:

It is true that in a great number of domains (perhaps every domain) there is a domain language that practitioners and subject matter experts use to characterize what it is that they do. It is not always so clear, however, what it is that end users want from their software. My experience is that many times they don't even know themselves or that at the very least there is an elaborate dance involved in integrating the visions of multiple parties who are often operating themselves at different levels of abstraction and with different and conflicting vocabularies. And in reality the process of domain engineering is only part of what we do when we capture requirements and drive forward to a picture of an intended piece of software. So not only is it a matter of capturing the language and vocabulary that can be used to express intentions, but also of helping business people refine and elaborate the intentions themselves.

Given this, sometimes the picture that y'all are painting strikes me as a bit of a chicken and an egg thing. Is the idea to work with business people first in a domain engineering capacity to come up with the language and editors that will then permit them and us software engineers to explore their requirements? Or does the work on domain-specific languages and the appropriate editors happen in the course of a larger "problem engineering" effort, hand-in-hand, say with a requirements capturing and elaboration toolset. And if this is the case, wouldn't you need the type of product you are describing to be integrated with good requirements capture and refinement tools, along the lines of what is articulated in the UML standard?

A related question is, if what you are doing is capturing the abstract syntax tree of a problem/solution description and projecting various views of it to your product's users, how can you along the way capture the abstractions and refinements that lead you to that problem/solution description? These will almost by definition not be expressed so cleanly in the language of the domain but will include all sorts of confusions and imperfections that ultimately hopefully will help lead to a precise domain language, but certainly cannot be expected to be at that point from the start?

In a response to one of the comments above, you note that "the subject matter experts are usually perfectly capable of expressing their domain knowledge with fine precision as long as it's expressed in ways they are comfortable with, in their comfort zone." Even if that is true for domain knowledge (which I'm not so sure it always is), the question of hammering out the vision of a software solution is a somewhat separate matter.

So in the end, I guess one of my questions is how you see your products and tools integrating with other tools that might be being used to help model a problem domain and a solution space?