9 Examples of Hidden Complexity in Software Development

Software development

“Why can’t you just give us a fixed price for development?”

In our early days we used to provide fixed prices quotes for development projects.

There are a number of reasons we’ve since stopped giving fixed price quotes (and switched to fixed budgets instead), but the main reason is that it’s impossible to accurately estimate the cost and time needed to design and build custom software.

Yes, the more research time we spend during project scoping, the more accurate an estimate will be. However, even if an engineer were to triple or quadruple their time spent scoping a project, there will still be issues they won’t encounter until they actually start development.

That’s why we strongly warn people when they receive a fixed price quote for a custom development project.

Unless a team has built the exact app before, there are always lots of hidden complexities in bespoke software which means the only way to provide an 100% accurate estimate is to go ahead and build the entire application!

While it’s not possible to provide an exhaustive list of such hidden complexities, we thought it’d be helpful to give examples of some of our repeat offenders.

1. Features that end up being much more complex than the client was able to communicate

We recently designed and built a multi-user checklist app that facilitates quality assurance and compliance in the transportation industry. As part of scoping and design, we worked out the requirements for the app, one of which was the need for offline usability (using the app without connectivity to the Internet).

Only during development did the client realise that they needed multi-user authentication and data syncing in offline mode. This meant multiple users would need to be able to log in and out without data access (offline user authentication), and seeing any new updates previous users had made while still offline (local data syncing and storage).

This blew out our original estimates for the offline component of this project as we needed to extend our original intention for cloud-based user authentication (as we assumed users wouldn’t need to log in and out of the app while offline) and build a more complex local data storage (as we needed to sync local changes and then update all those changes to the server, while dealing with any changes that may have been made by online users while the app was offline).

2. Complexities integrating with APIs and other third party services

It’s more common than not, that applications require integration with at least one API. This always adds potential uncertainty to a project, particularly if it’s not a commonly used API.

We recently built a data consolidation and researching tool that required integration with over ten APIs. Not surprisingly it was not the smoothest project we’ve run.

Some APIs we integrated with were straight forward, but other APIs were poorly documented which required multiple phone calls and correspondence with their technical team. Furthermore, one of the APIs was wrongly sold by their sales team which meant we discovered during development we couldn’t actually use it for our purposes (which meant we needed to write a custom application to web scrape the data we needed).

Working with 3rd party APIs or services can be like using a black box. We’re at their mercy because we have no control over how they work or how they communicate back.

Sometimes the API responses we might get back might not be specific. For example “Failed. Error.”, but we have no idea what error or with what part of the interaction it had a problem with, forcing us to try different alternative approaches until it works. This can be very time consuming.

Due to the poor documentation and cryptic error responses that some APIs provided, our original time estimates were greatly underestimated.

3. Changes to third party APIs

A while ago we built mobile app to spec and it was running well in production until suddenly the client started getting charged through their roof when Google decided to make a major change to their Google Maps API pricing.

The app was heavily integrated with this API so we couldn’t simply take out that functionality.

To combat this problem we wrote additional code to cache the data we received, allowing us to reduce the client’s costs by nearly 90%. Although the final result was successful, the time taken to implement the much more extensive caching was not accounted for.

It’s worth noting that any third party API provider (not just Google) can change their policies, pricing and business model at any time, completely outside of the client’s or our control.

4. We are not domain experts

As software designers and developers and not domain experts, we rely on our clients to provide us with the exact specifications.

We recently designed and built a fitness related app which was targeting Te Reo speakers.

While the app functionality was reasonably straight forward, this particular project required extensive data entry time fixing presentation issues, particularly where things like macrons were or were not needed (special characters indicating where particular words should be spoken as a long vowel).

5. Discrepancies on ‘implicit’ behaviour

When building a fairly complex software product for the accounting industry we encountered a number of instances where the client felt that certain functionality was implicit in a feature. For example, the client expected there would be an admin email notification certain events happened. Or, that an address field would give auto-complete suggestions.

In both of the above scenarios, extra programming is required (we needed to write additional logic for the email notifications, and we ended up integrating with a third party provider to handle auto-complete address suggestions).

In and of themselves, these additional features only added a few hours here and there, but throughout the entirety of the project they totalled significant hours which were not given in our estimate.

6. General issues with dependencies

Typically a medium to large piece of software will have a number of dependencies that are needed for it to function properly. This can mean that features that were fully working, are suddenly broken due to a change in an underlying package, a new security requirement, or changes to operating systems (particularly iOS and Android in the case of mobile apps).

This can be a significant time sink if there are multiple dependencies which were previously fully integrated and working together, but due to one dependency breaking, it has a flow on effect requiring code rewrites across multiple components.

7. Production use cases that weren’t part of the design

An app or website can be built to spec, but sometimes you’ll have real world users who haven’t read that spec!

An example could be an application that needed to load menu or index items on a page. Such a page was built and working fine in both development and production, and was accepted as ‘completed’ by the client.

But then one of their users had a use case where they needed to load over 500 menu/index items (more than ever thought about in planning or development).

The page that previously loaded instantly t00k several seconds or more to render (not a great user experience)! Yes, we could have engineered for this use case from the beginning, but without knowing it as a legitimate use case it would have been over-engineering on our part (costing our client more time and money than was necessary).

To solve this issue we needed refactoring that part of the application so it partially rendered the results on page load.

There are always solutions to the problems we encounter, but sometimes those problems aren’t encountered until way down the road after the initial build is shipped and live.

8. Removing features that were originally part of the design spec

Sometimes clients decide that a particular feature wasn’t such a good idea after they see it built and live. While often times it can be very quick to remove a feature, other times that feature may be heavily referenced throughout the codebase, which means additional time needs to be spent resolving the absence of that feature.

It’s frustrating for everyone involved, but removing a feature can sometimes majorly affect the costs of a project compared to its estimate.

9. Edge cases

Edge cases are situations which haven’t been accounted for in the scoping, design or development phases. The bigger the project, and the larger the user base, the more edge cases there will be.

Often months or even years into production, you’ll come up against a scenario that no one prepared for.

Just a few examples could include:

  • Complexities around how to handle leap years or Daylight Saving Time changes.
  • How to handle multi-user situations where two people make a opposing action on an item at the exact same time.
  • Situations where a device is in low power mode, or where connectivity keeps switching between WiFi and cellular data.

A recent example we encountered was for a P2P services app we built (think ‘Uber’ or ‘Doordash’) where an initial Auth transaction for a user was successful, but when the session ended and the application when to complete the transaction the card issuer declined it because the customer had canceled their credit card while the session was in progress (due to another unrelated fraud issue).

We didn’t happen to ask the question during scoping, ‘What happens if, while a customer is in an active session, she finds a fraud transaction on her statement and cancels her credit card?”.

The more complex a feature or project, the more edge cases you’ll discover. The screenshot here from Reddit is a great (and humorous) example of an edge case Apple hadn’t considered or tested prior to launching their crash detection software on the iPhone.

How do deal with hidden complexities

This list isn’t intended to persuade you not to build you bespoke web application or mobile app. But it’s helpful to go into projects expecting to encounter unexpected complexity and reserving a healthy portion of your budget to solve it.

Article written by

Will Schmidt

- Owner

Need a Custom App or Software Tool for your business?

Let’s have a chat! We’d love to understand your needs and design a solution that works for you.
 
Get a Free Quote