Verification & Validation Results

1. Introduction and overview

This Verification and Validation Results document is an outline for the development team (Design Bug) for analyzing quality assurance and testing activities of the When Animals Attack project. This document is also intended for the client's (Professor Henderson and Brandt) use so they will be informed of the results of our QA procedures and activities.

2. Summary of Results

On an informal level we have spent a reasonable amount of time peering into one another's code.

The majority of the component testing and interfacing is done. Our testing has turned into tuning our AI player. The individule agents in our AI algorithm need to cooperate in a more ordered fashion.

3. Results from reviews, walkthroughs, inspections, and audits

We have informally peered into each other's code. The informal peering has helped us understand potential architecture issues. We have also spent time pair programming This has helped us to avoid conceptual and syntactical bugs from being introduced into the code that we have already found to be reliable.

4. Results from testing

4.1 Summary of component test

Due to the graphical nature of the game, testing becomes much harder than simply writing unit tests for each class, running them, and making sure they pass. User interaction and input will be required for most of the testing. We will be testing the following main components (The testing of smaller components such as MovePhase, PlacePhase, BuildPhase, Tile, etc will be inherently tested as part of the following component tests):

GameControl
Human (Player)
DumbCPU (Player)
SmartCPU (Player)
Board
GameState

4.1.1 GameControl

Test Case Group	GameControl
Features to be Tested	Make sure a game can be correctly setup. Make sure the logical “playing order” of the game is followed.
Testing Approach	Run the program and give user input as required, carefully monitoring how the game plays out.
Pass/Fail Criteria	Passes if the game is setup correctly and the logical order of the game is followed according to the rules.

Passed... there were no significant issues.

4.1.2 Human

Test Case Group	Human
Features to be Tested	Make sure the Human class correctly interprets input from the user to generate each “phase move”
Testing Approach	Setup a Human vs. Human game and give valid and invalid input to make sure it is handled correctly
Pass/Fail Criteria	Passes if invalid input is not accepted and valid input is handled correctly to generate the “phase move”

This seems to be working.

4.1.3 DumbCPU

Test Case Group	DumbCPU
Features to be Tested	Make sure the DumbCPU returns valid “phases”
Testing Approach	Setup a Human vs Computer game and verify that every move made by the computer results in a valid next-GameState, also setup a Computer vs Computer game and let them play a few hundred games
Pass/Fail Criteria	Passes if for every Computer move a new valid GameState is generated

This is working.

4.1.4 SmartCPU

Test Case Group	SmartCPU
Features to be Tested	Make sure the SmartCPU returns valid “phases”
Testing Approach	Setup a Human vs SmartCPU game and verify that every move made by the computer results in a valid next-GameState, also setup a SmartCPU vs SmartCPU game and let them play a few hundred games
Pass/Fail Criteria	Passes if for every Computer move a new valid GameState is generated

We have just recently added an option for the computer to play against itself a given number of games. After the set of games have been played, our software spits out the statistics of what computer player won.

4.1.5 Board

Test Case Group	Board
Features to be Tested	Make sure the board is displayed correctly, and that every change in the GameState is appropriately updated on the Board
Testing Approach	Run the program and make sure everything is placed in the right spot and all the information is displayed. Play the game and make sure that after every click and move the Board is updated appropriately
Pass/Fail Criteria	Passes if the Board always displays the appropriate information at the appropriate time

This seems to be working nicely. There have been a few bugs that have been resolved already.

4.1.6 GameState

Test Case Group	GameState
Features to be Tested	Make sure the game state is always valid according to the game rules
Testing Approach	Run hundreds of Computer vs Computer games. Play Human games and make sure there is a valid GameState after every move
Pass/Fail Criteria	Passes if there is always a valid GameState

This was a large part of our testing. Although there have been a few issues, everything should be working correctly now. More defects may pop up as we continue to test for integration.

4.2 Summary of integration test / testing product as a whole

There are six main system tests and they are all closely related. These tests involve the verification that the following game situations are all played exactly according to the rules of the game:

Human vs Human
Human vs DumbCPU
Human vs SmartCPU
DumbCPU vs DumbCPU
DumbCPU vs SmartCPU
SmartCPU vs SmartCPU

4.2.1 Human vs Human

This will involve setting up and playing several human vs human games and closely monitor them to verify that all the game rules are followed.

We did quite of bit of testing in this area around the time of release one. We will need to do some “checkup” test on this before the product is released.

4.2.2 Human vs DumbCPU

This will involve setting up several human vs DumbCPU games closely monitoring them to verify that all the game rules are followed.

We have done some of this, but it is obviously fairly easy to beat the DumbCPU.

4.2.3 Human vs SmartCPU

This will involve setting up several human vs SmartCPU games closely monitoring them to verify that all the game rules are followed.

We have done some of this, but it is still easy to beat the SmartCPU since it isn't fully implemented yet. This will require more testing after SmartCPU is fully implemented.

4.2.4 DumbCPU vs DumbCPU

This will involve setting up a DumbCPU vs DumbCPU game, telling them to play a few hundred games and then review the logs to make sure all the game rules were followed.

We have been able to automize this portion, and have had the software play a few hundred games without any problem.

4.2.5 DumbCPU vs SmartCPU

This will involve setting up a DumbCPU vs SmartCPU game, telling them to play a few hundred games and then review the logs to make sure all the game rules were followed.

We have been able to automize this portion, and have had the software play a few hundred games without any problem. The statistics indicate, however, that the SmartCPU needs some more work in order to beat the DumbCPU more reliable. This will require more testing after SmartCPU is fully implemented.

4.2.6 SmartCPU vs SmartCPU

This will involve setting up a SmartCPU vs SmartCPU game, telling them to play a few hundred games and then review the logs to make sure all the game rules were followed.

We have been able to automize this portion, and have had the software play a few hundred games without any problem. This will require more testing after SmartCPU is fully implemented.

5. Evaluation of the process

5.1 Evaluation of test cases

We could have spent some more time writing automated unit test cases. The graphical testing has seemed to be adequate for us, but more unit tests could help us as we add more code to the project.

5.2 Results from the defect tracker

N/A. No defect tracking tool was used. Each developer is informed of any defects in his code, probably through another developer, and then takes corrective action to fix the bug. If our code was generating a large amount of bugs, or we had a specialized debugging person(s), then this might be useful.

5.3 Lessons learned

The best way to ensure quality code is produced, is to pair program, and work in close contact with the other pairs in the team.

6. Outcome of acceptance test and delivery

Creating a list of possible game situations, then generating each game with two human players and verifying that all the game rules hold and that there are no unhandled exceptions, or unexpected misbehaviors has been a major portion of our testing procedures so far.
This part involves testing the game with two computer players. There are three possibilities: SmartCPU vs SmartCPU, SmartCPU vs DumbCPU, DumbCPU vs DumbCPU. Each one of these will be tested by setting the game up to play many (thousands or tens-of-thousands) of games. They should be able to play all of these games without any exceptions or unexpected behaviors. We have done some of this testing. More testing on this section is required before our final release.
Playing many games against a “SmartCPU” and verifying that all the game rules hold. We have done some of this testing. More testing on this section is required before our final release.
Playing many games against a “DumbCPU” and verifying that all the game rules hold. We have done some of this testing.

7. Summary of open issues

We need to continue testing as product development furthers. In addition, we need to improve SmartCPU so that it has a wider margin of wins against the DumbCPU.

8. Additional information

Don't spit into the wind... bad idea.