Technical explanation behind estimates in software development industry going off by a factor of 100 times or more?

In this article I will try to technically explain the reason behind estimates in software development industry going off by a great factor.

Any one who has worked in the software development industry or have got their software developed through a software development company will be aware of this problem of estimation.

It is very common in the software development industry to come across projects wherein initially the software development company promised the customer that their software will be completely developed within couple of years but in reality the system takes 10-12 years to be used reliably to certain extent.

In my article "Lets not blame the customer - Human limitations wrt software systems" we have discussed the two severe human limitations faced by human beings in general while dealing with complex systems. In my other article "" I have explained how most of the times estimations are done on the basis of past experience rather than proper analysis using modeling.

The problem of estimations going off by a great margin is a result of combination of the lack of knowledge about these human limitations and the best practices to manage the same along with absolute neglect of analysis using modeling.

In the article on "Human Limitations" we have discussed the "Inability of a human brain to comprehend numerous combinations of logical scenarios that can exist in software system." as one of the severe limitations.

Let me quickly explain this limitation by the following case.

To exemplify this problem , let me discuss a question which I have asked to numerous software engineers and analyst in the software industry.

"How many logical scenarios need to be considered and evaluated by an analyst before which he can say that he has completely understood and captured the requirements wrt the reserving a ticket functionality for a railway reservation system for Indian Railways."

Let me ask you to guess among the following answers:

0 - 20 scenarios
20 - 30 scenarios
30 - 100 scenarios
100 - 300 scenarios
300 - 500 scenarios
500+ scenarios

This question is very interesting in itself because if I ask the same question to a software engineer or an analyst .. their response will be around 20 - 30 scenarios.

The reason majority of the respondents talked about 20-50 scenarios is because it is found that a human being generally doesn't have the capability of thinking about more 20-30 scenarios and because of which you can see that your customer may not give more than 20-30 scenarios and your engineer will not codify more then 20-30 scenarios.

If you ask the same question to a laymen he cannot think about more than 20-30 flows because it is found humans in general cannot think about more than 20-30 flows.

But in reality the number of flows in this functionality could be around 500- 700 flows.

Let us technically understand as to how can we find out the number of possible scenarios that needs to be evaluated for a particular functionality.

The mathematical rule says -

The number of logical scenarios that need to be evaluated in a functionality is directly proportional to the combinations of different types of domain entities which are manipulated in the functionality.

For example... Reserving a ticket functionality:

A specific type of user reserves a specific type of ticket, for a specific type of passenger, for a specific type of train, for a specific type of coach, for a specific type of seat, using a specific type of payment mode, by a specific type of customer, for a specific quota between specific types of stations.

The 14 domain entities that are manipulated in this functionality are:
User, Passenger, Customer, Ticket, Train, Coach, Seat, Payment Mode, Station, Quota, Fare, Concession, Route, Meals etc.

We also know that each of these domain entities have multiple types. Lets say on an average each of these domain entities have 3 types each.

Refer to "Best practices of structural modeling using class diagrams".

As each of the domain entities can have 3 types each, the business logic for reservation can differ for every other different combination of domain entities that are supposed to be manipulated in this functionality.

So for 14 domain entities with 3 types each ... there can be 1000+ combinations which are possible.

Even if we discard 50% of the same, still we have at least 500 flows for which we will have different business logic.

Even if we know this rule of finding the number of flows of a functionality , if we start writing the same manually , we might take inordinate amount of time to write it down same and that also we might not be sure as to whether we have captured all the possible combinations.

Hence the task of finding out all the possible logical scenarios that need to be considered is not possible manually because of the sheer number of combination of different types of domain entities with different business logic each.

This is the second limitation faced by a human being wherein the human brain is not capable of thinking about the numerous logical conditions. This is the reason, the customer will not be able to tell you anything more than 20-30 scenarios although he might be aware of other scenarios as well.

In the next section, Using these human limitations, I will try to technically explain the reasons for estimations going hay wire in the software industry by a factor more than 100 %.

As we now have inferred that a human beings cannot think more than 20-30 scenarios or flows, let me try to describe a practical case in software development industry wherein there are multiple stakeholders all doing their work efficiently but manually. As all of the stakeholders are doing the job efficiently ... no one can be blamed for this.

Let say for a behavior like Reserving a ticket for a railway reservation system,

  • The customer thinks about 20-30 logical scenarios and gives the associated business logic to the analyst.
  • The analyst doing his job efficiently thinks bout 20 more logical scenarios that could be present and captures the business logic from the customer and formulates his requirement document.
  • The requirements captured and documented by the analyst then goes to the project manager for estimation who also using all his experience puts a 50% buffer while estimating ie he will now estimate for 60 scenarios.
  • Then it goes to the software engineer or the developer who is also doing his job the right way, he thinks about 20 more scenarios which was not thought by the customer and the analyst and he programs the 60 scenarios during his development.
  • Post development, local testing team who is supposed to test this behavior thinks about 20 more scenarios that were not caught by the previous stakeholders and ensures their development, and hence after this phase the functionality will take care of 80 scenarios.
  • Post local testing, the functionality goes to the production testing team which lets say adds 20 more scenarios to it and finally the functionality gets deployed on the production systems.
  • It means while the functionality was rolled out on the production system only 100 logical scenarios were taken into account while we know in reality more than 500 scenarios were possible.
  • Once the functionality is in production from this point of time all the logical scenarios which existed but were not taken into account (500-100= 400 scenarios) will keep coming as change requests for the next 6-7 years.

Our work in this area shows that when people talk about the problem of customer requirements changes, these changes mostly are the missing logical scenarios that were not considered and taken care of rather than the requirements changes because of customers business changing.

I am sure your quality team might have given you this statistics about the cost of solving a defect in the production stage as compared to requirements stage.

The industry statistics is the cost of solving a defect in the production stage is at least 100 times (some metric say it is around 400 times) more than the cost of solving the defect in the requirements gathering stage.

So let us now multiply the difference in the number of flows ie 500-100 with a factor of atleast 100 and this is what the customer will have to pay for this problem.

This example will technically explain why estimations in software industry goes haywire by more than 100 % or why 70% of the software projects are a failure or why US industry looses more than 30 billion USDs only because of defects in the requirements gathering stage.

Think about it project manager only estimated for 60 scenarios but in reality there were more than 500 scenarios and the multiplication of this number by a factor of 100 makes this number unimaginable.

The beauty of this problem is there is no one to blame. Every stakeholder was doing their job the right way.

The only thing that can be concretely blamed is the limitation of the Humans to think about more than 20 scenarios.

If you look at it from the point of view that it is a human limitation then none of the stakeholders can be blamed.

But if you think about it from technical point of view this problem can be solved very easily by simply writing a software program that takes domain entities and its types as an argument and that can generate the combination of the types of each of the domain entities.

I agree that manually it is extremely difficult to think about the possible combinations, but for a software program it is just a matter of seconds.

According to me, this problem can be very easily solved by the analyst by using a bit of his engineering skills by using a software program to get the combinations of domain entities.

Both of these human limitations are interrelated with each other. For anyone to find out all the combination of different types of domain entities, the prerequisite is that he should be able to understand the structural complexity of the system in a single frame of vision.

The exact way of solving both these problems will be described in the section "Best Practices for modeling the structural complexity using class diagrams".

On one hand the lack of usage of engineering skills by the analyst to handle this complexity is the root cause of this huge problem in this software industry.

On the other hand the way Software development companies make this problem more worse.

Generally in the software industry, all the roles of analyst, designer, engineer and unit level tester is all played a single engineer. As a single person is playing all the roles, the number of logical scenarios found and taken care of is far less than what if all these roles are played by different people or individuals.

Using these examples I have tried to demonstrate that bulk of the problems faced by the software industry are because of some human limitation wrt complex systems and these problems will continue to exist till we don't develop engineering tools and best practices to overcome these limitations.

Hemant Jha
Founder - VPlanSolutions
Researcher, Trainer