Game Programming and Development Tools

Unit Testing Games – samw3

samw3

Member

Posts: 542
From: Toccoa, GA, USA
Registered: 08-15-2006
Hello All,
I've been a lurker here for a while, and am coder and a Christian myself. I have 20+ years experience coding in many different languages.
Currently I am a research developer for an R & D company doing government contracts. I noticed that this community is highly focused toward writing games. Fine by me.. code is code.
But I have a question for the game coders.
How do you code Unit Tests for Quality Assurance when you are dealing with a user interface for the game?
I can understand black-boxing individual objects, but any ideas on QA for the game interface itself?
The reason I'm asking is we may be submitting a proposal for a "Serious Games" topic and feel a little lost on game programming methodologies.

Thanks for you comments in advance.

Sam

EDIT: I should have noted, I guess what I am interested in is a 3D game interface unit tests mini-howto.

[This message has been edited by samw3 (edited January 05, 2007).]

Realm Master

Member

Posts: 1971
From: USA
Registered: 05-15-2005
quote:
Originally posted by samw3:
Hello All,
I've been a lurker here for a while, and am coder and a Christian myself. I have 20+ years experience coding in many different languages.

[This message has been edited by samw3 (edited January 05, 2007).]


*drops dead*

no... way...

*jaw dropps off...*

well, um... first off, welcome to CCN!

Secondly:
20+!!!??! Holy smokes! You'll be one of the most prized members of CCN!! You'll be gettign a lot of PMs from some serious guys (I'm... not incredibly serious, not the best, but i Still love game programming adn creation. )

------------------
yeah, im a little crazy
Check out my crazy sig that I made:

samw3

Member

Posts: 542
From: Toccoa, GA, USA
Registered: 08-15-2006
er, thanks.. I don't mind answering a few questions. I've done project management as well.
As for fluent languages: Java, C, C#, PHP, shell script
Langs I've done in the past: VB, ASP, Pascal/Delphi, and 6502 8-bit assembler
Langs I dabble with: Ruby, Python, PicMicro assembler

Maybe I should rephrase this thread's question... Do game programmers do Quality Assurance with their code? i.e. Unit Tests/Acceptance Tests?

------------------
Sam Washburn

HanClinto

Administrator

Posts: 1828
From: Indiana
Registered: 10-11-2004
Hi, Sam! Welcome to CCN!

I'm also interested in knowing the answer to this question. I'm a professional programmer by day, and at most of my work places, we don't do much unit testing for quality assurance of our GUI programs. How do you unit-test a GUI? We have some unit-tests in our back-end (object persistance IO and such), but I'm interested in knowing how much unit testing has value in game GUIs.

We certainly do regression testing, where we have a reproducable set of steps for every bug that we have found, and our testers go through the steps to make sure that everything works before release (or something like that), so it's sort of like "unit tests for people", but nothing as comprehensive as a unit test for a 3d math library or something where results are very easily right or wrong and quantifiable.

More on this later, but the coffee shop is closing so I gotta' roll.

I look forward to hearing more people's thoughts on this one!

In Christ,
clint

samw3

Member

Posts: 542
From: Toccoa, GA, USA
Registered: 08-15-2006
These days my code's interfacing with the real world is primarily through web interfaces or through PIAs (interfacing with custom hardware, we do some embedded work).

We currently use Selenium from http://www.openqa.org/ for unit testing for the web gui's. For the PIAs we build a simulation object that models the hardware with custom methods and assertions. We have used Mercury's XRunner and WinRunner for os gui testing in the past, but its expensive.

But Selenium-IDE make it a breeze to build web gui test scripts and the core can play them back on a BUNCH of different browsers and it has some great assert tests. If you work with the web, I highly recommend checking it out.

As for game environs, I guess human regression testing could be a fall back, but would drive the project costs up a bit since we'd have to hire more testers. It would be nice if it could be automated.

Any other ideas out there?

BTW, thanks for the welcome to Realm and Clint.

------------------
Sam Washburn

[This message has been edited by samw3 (edited January 07, 2007).]

AndyGeers

Member

Posts: 45
From: London, UK
Registered: 06-20-2005
We also use Selenium for unit testing our web-based clients - not for long, mind you, but it seems really cool!

I've not got much of a unit-testing vibe from the games development community as a whole, but I know that there are C++ unit testing libraries out there like CppUnit and Boost.Test - not particularly suited to GUI testing though.

The forums at http://www.gamedev.net include a lot of professional game developers, so it might be worth asking around there.

------------------
--
http://www.geero.net/

Mack

Administrator

Posts: 2779
From:
Registered: 01-20-2001
Welcome to CCN Sam
samw3

Member

Posts: 542
From: Toccoa, GA, USA
Registered: 08-15-2006
quote:
I've not got much of a unit-testing vibe from the games development community as a whole

That's a shame, I have found that taking the time to unit test or even just regression testing saves boat-loads of time later when you inevitably tweak and break.
While I don't agree personally with all the methods incorporated in "Extreme Programming", I do like the idea of writing your unit tests first. It takes more time initially, but then kinda provides a project spec of sorts as you have defined "tests to pass".

@Mack: glad to be here.

------------------
Sam Washburn

AndyGeers

Member

Posts: 45
From: London, UK
Registered: 06-20-2005
quote:
Originally posted by samw3:
[QUOTE]I've not got much of a unit-testing vibe from the games development community as a whole


That's a shame, I have found that taking the time to unit test or even just regression testing saves boat-loads of time later when you inevitably tweak and break.
While I don't agree personally with all the methods incorporated in "Extreme Programming", I do like the idea of writing your unit tests first. It takes more time initially, but then kinda provides a project spec of sorts as you have defined "tests to pass".

@Mack: glad to be here.

[/QUOTE]

Yeah, the Unit-Test-As-Spec methodology does help a lot.

Don't know if this helps, but Gamasutra had a few articles on agile/unit testing in games a while back. Here are two I found quickly:

http://www.gamasutra.com/features/20050329/roken_pfv.htm
http://www.gamasutra.com/features/20060628/mcguire_01.shtml

------------------
--
http://www.geero.net/

Calin

Member

Posts: 358
From: Moldova
Registered: 12-04-2006
Nice links andygeers. I finally got to grips with what is being discussed in this thread =].

samw3

Member

Posts: 542
From: Toccoa, GA, USA
Registered: 08-15-2006
Thanks Andy!

This is exactly what I was looking for! The articles have been very helpful in this design. Particularly the one titled "Automated Tests and Continuous Integration in Game Projects".

The author makes a point that a good place to automate game ui testing is in the regression tests. Here is a generalized solution from the part of the article that I needed.

1. Be sure all methods which use randomness can be seeded (predetermined)
2. Build an input capturing layer between ui and system to save play test input as a "script" of sorts.
3. Allow the playback system to screenshot periodically and do a pixel for pixel comparison for that each frame with the original frame (regression test).
4. Make it easy to capture a new test.

One that I thought of, adding a "Save/Load State" feature to the test interface so all variables for the current level could remain consistent.

Thanks again.

Sam

------------------
Sam Washburn