Testing 1, 2, 3...

Estimated Reading Time: 5 minutes 26 seconds. Read Later

The Importance of Product Testing

As mentioned in my September blog, Engineered Software will be celebrating 30 years in business in 2012. In that post I listed the core value we have followed to better meet our customer’s needs and expectations, specifically:

1. Create a sustainable company
2. Create product that can be used by a wide variety of people, not just engineers
3. Play well with others.

In this month’s blog I will be sharing with you how our software testing has evolved over the years.

When we started Engineered Software both Carolyn Popp and I (the two founding principles) saw the value of developing a testing program for our software. Our feeling was, if we released a program that had a calculation error, our customers might not trust our new company. In addition, if the customer had trouble entering the necessary data or lost data due to a problem with the program, we were not providing them with the value they expected. For the first version of PIPE-FLO, we developed a series of program features; the formulas the calculations were based on, along with a set of example calculations. These example calculations included a variety of edge cases designed to test the program source code.

I developed a series of test problems from text books, technical publications such as the Crane Technical Paper 410, along with developing a series of calculations performed using an electronic calculator (remember 1982 pre-dated Lotus 123 and Microsoft’s Multi-Plan). A set of calculations sheets were developed complete with the equations, references, all input values, along with intermediate and final calculated results. Prior to every program release, these examples were entered manually into PIPE-FLO and the results validated. Carolyn, a former IBM system engineer, was just as adamant on program testing and validating prior to program release.

Our third & fourth employees, Jaclyn and Kathleen Byron (we often called them J&K), both with mechanical and civil engineering degrees, were hired to improve our program testing, provide customer technical support, and assist in software development. Based on their customer interaction they were able to created test procedures to validate all engineering calculations along with the user interface. They also were responsible for automating the test procedures by developing test scripts for each program feature.

When one of the software developers finished a new PIPE-FLO feature, J&K created an automated test, and checked the results. Any problems with the new feature were brought to the attention of the programmer and resolved. Once we had a PIPE-FLO release candidate, we would perform all automated tests again to validate the software results prior to release. The release testing took approximately 2 – 4 weeks to complete, allowing the correction of last minute program bugs, along with a review of the test results.

As our company grew, we added more programmers and engineers to our development team. In 2006 we decided it was time to update our software development and testing practices by implementing the Agile Software development process.

The major impact was to document the proposed program features and so everyone knows how the feature works. The process starts by engineers creating “stories” for each feature describing how customers used the feature. The application developers would write the program code to meet the requirements listed in the “story”, and the developers in test would develop the testing needed to ensure PIPE-FLO meets the “story” requirements.

Testing is now broken down into unit stages:

• Unit Testing
• Integration Testing

To demonstrate I’ll use an example for calculating the fluid Reynolds number. First engineering first creates a “story” defining the Reynolds number calculation, the data that must be supplied, and a variety of examples with the calculated results.

The application developers create a function for calculating the Reynolds number. The new function is placed in the function library used by all developers that need to calculate the fluid Reynolds number. In addition the developer creates a “unit test” based on the example calculations supplied by engineering in the “story.” The unit test is then placed in the unit test library that is run every time the program is compiled by any developer.

Now let’s say the developer creating the orifice sizing feature needs to calculate the fluid Reynolds number of the orifice. The software developer will use the previously defined Reynolds number function. If the function needs to be modified for the orifice sizing calculations, (say using the orifice diameter instead of the pipe diameter) then the Reynolds number function in the library may be modified by the developer writing the orifice feature. Once a developer compiles the program on their local computer the Reynolds number unit test will be performed to insure nothing was changed that would give bad Reynolds number results for either the pipeline or orifice Reynolds number calculation. If the test fails the developer knows immediately there is a problem in the Reynolds number function and makes the necessary correction.

Once the developer has completed the orifice sizing feature on their computer they check in their code to a central build server. The build server is where all programs are compiled for release. Once the code is checked in on the build server all unit tests are performed on the code, if any problems occur the developer is notified immediately. The build server performs all unit tests each time any one of the application programmers check in their source code.

Integrated Tests are developed by a separate group of programmers that specialize in software testing. The developers in test create the integrated tests to ensure the program’s higher level features work as called out in the “story.” The majority of these tests are developed to test PIPE-FLO’s user interface.

For example in PIPE-FLO the pipeline graph window shows the fluid Reynolds number over a range of flow rates. The developers in test will create an integrated test to check that the graph window is acting as defined in the story. For example the integrated test may check to see that the Reynolds numbers are graphed and displayed and the user also has the ability to change the color.

In the past the integrated testing was conducted when the feature was initially added to the program, and at the completion of the program prior to program release. The time difference between initially adding the feature releasing the program could be measured in months. During that time span a developer may make a change in a new feature that has an effect on a previously tested feature.

With our new automated testing once an integrated test is written to test a feature it is added to the test suite on the testing server. Every night the test server performs all automated tests.

In the evening, the test server takes that day’s program from the build server and loads it to the test server. The test server loads the current program onto multiple computers on the test farm. The test server then assigns the various integrated test to each computer and keeps track of the results. Currently there are about 10 computers in our testing farm and it takes approximately 6 hours to perform the test and review the results.

The next morning each member of the development team can check to see if the code they wrote yesterday has an adverse effect of the program development. If any of the tests fail the developer will be notified and can make the necessary changes that day.

This testing has paid off for both Engineered Software and our customers. In the last two versions of PIPE-FLO we have only had to issue one maintenance release to correct problems associated with the program. A more reliable program means less stress for such important engineering choices, and our aim is to make our customer’s job easier.

I would love it if you left a comment or even sent me an email to blogger @ eng-software.com. Also, we are currently welcoming guest bloggers. If you are interested, just send me a message about becoming a guest blogger, and what you would like to write about. Thanks for reading!