Requirements Engineering - A problem that no software development company wants to solve

In this article I will be sharing by real life experiences interacting with the management of software development companies on topics dealing with requirements engineering, analysis and design. This will also make you understand the probable reasons behind certain behavioral trends shown by the management of software development companies.

I was made to learn most of these facts the hard way by the software industry the moment I started my journey as a young entrepreneur trying to solve one of the most difficult problems in the software industry ie "Requirements Engineering".

These instances will help you understand the hesitance of the top management of software development companies to solve the problems in the space of systems analysis and design.

Instance 1

I will first describe my real life experience which changed my life and made me commit the biggest foolishness of my life.

As a result of a failure of a huge project and because of some extensive research I came across some of the inherent limitations to human beings when dealing with complex systems which to a certain extent are responsible for failure of complex software systems.

We were also able to exploit some fundamental system properties to come out with a mechanism using which requirements of a system can be validated for its completeness and correctness in a domain independent manner (It sounds unbelievable, but yes it is possible).

As requirements engineering has been a problem for ages and nearly every green field project in every company have their share of failures wrt requirements. We thought our software product will be a hot cake. As an young entrepreneur, I thought I have hit the gold mine... but this happiness was short lived.

We were also able to build a software system which can take of these human limitations and can also validate the requirements in a domain independent manner.

When I showed this revolutionary product to certain alumni of mine who were then heading various software development companies in India... the response I got was quite shocking!!!!!!!!!!

As they were my well wishers... they told me that "Hemant agreed you are addressing a very valid problem... and your solution looks great as well.... but tell me who in this world wants this problem to be solved?" Their reasoning was that most of the software development companies work on the time and material model where in the revenues earned by the company is directly dependent on the longetivity of the project and why will the software development companies do something which will improve the quality of the software system, but reduce their revenues. Even though they were my well wishers and were guiding a young entrepreneur about some hidden trade secrets... but still I refused to accept their reasoning. The same reasoning were given by the VCs and investors to shoot the business proposal down. But I still refused to accept this as reality.

I continued marketing my software to various software companies in India, thinking that these guys have a real problem and I can provide a solution to the same. But all of them praised the product a lot but no one was willing to use it. That's where I remembered the word of advice given to me by the investors and my alumni. I started my entrepreneurial journey trying to solve a problem which no software company wants to solve to satisfy this business interests. By this time I realized that the problem that I am trying to address is actually a business catalyst for the software development industry.

Although no software development company will accept this in public... but this is a untold truth of the software development industry.

Instance 2

As a trainer.. I have trained software professionals from 1-20 years of professional experience from fresher to Group Heads, Delivery Managers and Program Managers. In my experience of training, I have come across atleast 2 Group heads, 4 - 5 delivery managers and atleast 10 Project Managers who after the completion of the training, accepted that their system analysis, design and development process has severe lapses and they promised that they will take it up with their top management to look into this and rectify the same. As they are regularly in touch with me, I know they have tried their level best to convince their management to address issues wrt requirements engineering, analysis and design but the people sitting at the level of Vice Presidents or the other higher levels are just not willing to look into the same.

Instance 3

I have been training a renowned service based software development company for the past couple of years on OOAD, UML, Design Patterns and Software Architectures and have conducted more than 50 training programs. Most of the engineers and managers were able to understand and accept that requirements engineering, analysis and design is virtually missing and most of their projects were in crises mode. When I launched Analysis and Design Service, few of them requested me to speak to their top management as the company can greatly benefit from these services. I requested the top management for an appointment and I got an appointment with the top executive of the company. In the meeting when I talked about the various problems addressed by our services, I was told that we know every thing and was told that whenever we have a project which is facing problems with requirements engineering, analysis and design, we will think about you. Friends look into this very proactive approach... "Let the problem happen and then we will address it". I was told this when I knew very well that most of the projects in that company were facing crises every day wrt requirements engineering, analysis and design. A project manager in the that company told me "We do very little requirements gathering, absolutely no analysis and very little design and then directly start with implementation and after sometime the project goes the crises mode.". A Engineer from the same company told me that "We are so used to this hot water (problems wrt requirements, analysis and design) that most of us has accepted this as reality and no longer feels the heat of the same."

Question is "Why is the top management deliberately trying to overlook the problem when most of the projects were facing these problems?"

Instance 4

I came across this experience when I was promoting Analysis and Design Service. I happened to speak to one of my ex boss who is now a part of senior management team for a software development company. He was grateful to give me a good amount of his time and appreciated my work and services in this area. He realized that my services can be tried in one of the projects to understand its effectiveness. As he was a part of the senior management team, he directed few of his project directors to look into these services. On the day of the meeting one of the directors told me that Hemant you will only have 45 minutes for this presentation. Unfortunately out of the 5-6 directors that were present for this meeting only one was really interested in knowing about our services, rest all were just playing around with their BlackBerry's. The one who was really interested told me that "Hemant your vision is our problem statement. We face these problems every other day". And can you guess what was the end result? I never heard from them after that. The reason I had to mention this instance out here is because in this case I realized, requirements engineering, along with analysis and design is one topic that the management just doesn't bother to look into. One can say that I was not called because my services were not up to the mark. But the point is how can one decide about the various aspects of my products and services within half an hour. When I know that most of the participants were just not interested in listening about this topic right from the beginning. The point I also want to emphasize out here is "Does the topic of requirements engineering deserve only 30 minutes wherein virtually most of the projects face this problem every day?"

The beauty of this problem is that engineers and technical managers understand the problems and want them to be solved, but invariably the higher management is not just willing to look into this although they know the problem very well.

This problem can also be understood from the fact that we have written about the criticality of these multifaceted problem of requirements engineering to most of the CEOs of the software development industry in India but none of them is willing to look into it.

Although human limitations itself makes it a very difficult problem to solve, but can be managed by appropriate software systems and best practices, but the strategic facet of the software development companies makes it nearly impossible to solve.

We need to understand that at the end of the day all of these software development companies are profit oriented business systems guided solid business objectives to achieve.

Lets look into the business model of software systems:

Business Model - Service based Software Development Companies.

Software development companies are business systems the responsibility of which is to develop custom software systems for their customers respective business problem. This is a basic behavior of any service based software development company. The structure of this system are the software professionals(I am deliberately not addressing them as a engineers) who are supposed to develop the system.

As service based software development companies are business system, any business will have to make maximum profits to keep its shareholders and investors happy. For the profits to be maximum, the revenues has to be maximum and the cost has to be minimum.

Cost can be minimized by a lot of ways, definitely shifting the development work to countries like India can help a company minimize its cost.

The non functional parameters that govern the projects and business units are "Increase the revenues and reduce the cost". As I have discussed costs can be reduced by ensuring "that salaries paid to the billable resources should be as minimal as possible."

The revenues can be addressed by the following parameters:

  1. Increase the number of projects
  2. Increase the time line of each of the projects
  3. Increase the number of billable resources
  4. Increase the rate of billing rate per hour

Out of the above mentioned parameters for increasing the revenues, option 1 and 4 are very difficult to achieve option 1 depends more on the market conditions and the effectiveness of the sales team while with increased competition option 4 is not feasible.

The only parameters that management can safely play around are Option 2 and Option 3 the complexity of the system is a debatable issue and cannot have any quantified.

This can be achieved by creating more billable resources and by paying them less salaries.

Unfortunately quality of systems doesn't figure in the above mentioned list.

My experience dealing with the management of the software development companies have made me compile this list.

There can be multiple reasons for the top management to ignore this topic:

  1. The top management is not aware of these problems.
  2. The top management is technically sound
  3. The top management doesn't understand these problems.
  4. The top management finds these limitations and their solutions useless.
  5. The top management is arrogant enough to accept that they don't have the capability to solve the same
  6. The top management doesn't want to deliberately solve this problem because of their strategic benefits.
  7. The top management doesn't find any motivation to take the risk of addressing these issues.
  8. The top management knows that even if they want to address this issue, they cannot.

Option 1 is definitely not true as I have been trying convince to most of them about this for the past 6 years, so saying that they are not aware is not true.

Option 2 is definitely the case, most of the people occupying the higher positions are brilliant man manager who have mastered the art of increasing the billable heads at minimum costs, but lack technical expertise

Option 3 is also not true, if their engineers and technical managers understands and appreciate the problems and their solutions.. then how is it possible that their management doesn't understand the same.

Option 4 is also not true, the same reasoning as mentioned above, If their engineers and managers understand and appreciate this, then it cannot be useless.

Option 5 and Option 6 is also true as well.

Option 7 is also a practicality, if you look into their business model, you will see there is no motivation to take the extra headache of sorting this problem out when they are able to get the promotions and hikes even without solving the same.

I will summarize this article by saying that entrepreneur thinking of starting any business related to this field... I just want to warn you... you are dealing with a problem which no software development company can solve.

Hemant Jha
Founder - VPlanSolutions
Researcher, Trainer