Homework Assignment 6; CS 450

Part 0: Overview

In this assignment, you are going to write a series of test cases. For each test, you will provide the input, the command line to execute and the output.

!!!!!!! FOLLOW THESE DIRECTIONS VERY CAREFULLY !!!!!!
For the first test case, the input file should be called USERNAME.1.TESTNAME.input, the file with the command line should be called USERNAME.1.TESTNAME.command and the output should be called USERNAME.1.TESTNAME.output. USERNAME should be your username and TESTNAME should be a descriptive name for the test.

Subsequent tests should follow this same naming convention changing the test number and TESTNAME (USERNAME.2.TESTNAME.input, USERNAME.2.TESTNAME.command, USERNAME.2.TESTNAME.output). Look here for some examples including a simple TESTPLAN to get you started. Notice that the command files do *not* contain a pathname before the executable name and that the input file is not mentioned in the command file!!! Input will be taken from stdin in our testing script.

It is not essential that your tests are numbered sequentially (i.e. if you delete test 4 you don't have to renumber 5 through N.)

It is very important that you adhere carefully to the naming conventions I have described as we will be automating testing in subsequent homeworks. I will use an automated testing script much like the ones available here: runSortedTests_ksh.sh and runSortedTests_sh.sh .

In addition to writing the tests, you should describe each test in a test plan document called TESTPLAN.USERNAME. You should describe the purpose of each test case and argue that all the test cases cover the full range of functionality. Your testplan should convince the reader that your testing is thorough, appropriate and likely to catch common errors. Neatness and organization will count (a lot !!).

Part 1: Command Line Option Testing

Choose a simple input file and write a set of test cases that exercise all possible combinations of the -l, -t, -d and -c flags on that file. Consider what kind of input file will best help you isolate command line processing errors. (Note: We did not specify the -help output sufficiently for use in automated testing. So do not test this option. ) Give each flag its own hyphen (ex. -l -t -d -c rather than -ltdc) even if your code does not require it. Remember also that flags should occur in order l, t,d, c.

For this set of tests, it is important that you argue that you have exhaustively tested all possible legitimate combinations of the command line arguements. Why is exhaustive testing possible in this instance? How many tests are required to do this?

Part 2: Black Box Testing

You should also write a variety of tests that probe the program's ability to correctly count words according to our specification.

Write a series of 25 black box test cases for any person's program. Within this "limited testing budget", you should do your best to cover the full range of the program's functionality. You do not need to focus on the command line interface processing so you may want to consider using simply cs450words (i.e all options on) for each test.

The test should be small enough that you can verify the correctness of each test by hand, but large enough that you can cover a large percentage of the programs functionality. I recommend thinking of each test according to a theme (e.g. one test that tests all legal inter-word punctuation, one tests that tests illegal inter-word punctuation, one that tests capital to lower case, etc.) Remember that good test cases have a high probabilty of finding an error. Ask yourself, what parts of the functionality are likely to be buggy.

Add a description of each test to your testplan. In this case, exhaustive testing is not possible so it is important that you *argue* that you have used your limited test plan widely to effectively test the programs functionality.

Are there any aspects of the functionality that are not well defined? What? Make sure that those aspects of the functionality are isolated to individual test cases?

Note: You do not need to focus on error handling or performance testing. Error handling is covered by the white box testing in another assignment. Performance testing is covered by the real world/stress tests in Part 3 of this assignment.

Part 3: Real World/Stress Tests

Add 2 more complicated real world test cases to your test suite. You do not have to be able to verify these by hand. They should stress the performance of the program or possibly its ability to handle unusual characters without halting. Do not duplicate any of our standard files (dictionary, huck finn, etc.) Look elsewhere for large or complicated files.

Describe the files you choose (how big, what is in them, etc.). Describe also how your program performs on them. However, in the interest of our disk space limits, do *not* submit these files. We will add some to the class test suite. Even in your private test suite, you may not want to include a .output file corresponding to the test. Since you often cannot hand verify tests like these, they are used primarily to stress performance and robustness than correctness.

Part 4: Submit your assignment.

You must check the following files into /afs/clarkson.edu/class/cs450/students/YOUR_USERNAME/hw6 : TESTPLAN.USERNAME, USERNAME.*.input, USERNAME.*.command, USERNAME.*.output,

Note: Do not submit your code, Makefile, README, etc. Keep any improvements to your code and documentation for homework 7.

When I grade this assignment I will ask myself questions like: Did you follow directions? Do I think your test cases exercise the full range of functionality *and* probe for likely errors? Do you try all combinations of the command line options? Did you think of input files that probe for likely errors in implementing our definition of words? Did you have a couple larger more complicated tests?

Remmber to estimate the time it will take you before you begin and to record the time it actually takes you in your TIMELOG. You will submit the TIMELOG again in the next assignment. Remember also that the names of submitted files *must* be exact!!!