Viewing page 5 of 5 pages.
Previous 1 2 3 4 5
LIMITATIONS AND DRAWBACKS
In their current form, the tools are still quite limited. Neither feels like
it has progressed beyond the beta stage, and there are a multitude of factors
that might be relevant but have not yet been researched enough to know whether
they affect the user's experience. That said, Intel also does not yet consider either
tool a finished product. Work is still being done on both tools, and it will
continue even after they are released. Both tools have
substantial roadmaps for future features and improvements. For the G-CAT, this
means further research on different games (perhaps including new genres), while
the DH-CAT can expect to see more primitive tasks and a completely new category:
Premium Connected.
Many improvements for the G-CAT are planned.
Of the planned improvements to the G-CAT, the most pressing is probably the
need to determine the effect of different game settings. During the presentation
of the tool to the press, many people expressed concern that the settings used
— default graphics options at 1024 x 768 resolution — does not test the full
capabilities of most graphics cards. This is less of a concern that it might
seem — good quality testing is possible no matter what settings are used so
long as the basic test parameters remain the same — but expanding the capabilities
of the tool to encompass more variables could definitely improve the reliability
of the end result.
The amount of research needed to make this
feature possible is substantial. Proper experimental techniques dictates that only
one setting can be changed at a time, which means that testing the effect of
multiple settings requires multiple tests and an increase in the sample size.
In my opinion, the basic settings used in the initial market research are perfectly
adequate for the tool in its present form. The average gamer often does not
even look at the graphics options in his games — he simply plays at the default
settings unless he needs to turn things off for performance reasons.
Some of the features that may show up in future versions of the tool.
Of the features listed above, the most important from the standpoint of user
experience is probably A/V sync. Intel has stated that this feature is in
active development, so we expect to see it sooner than later.
For SPCR, the most interesting feature is "Thermal / Acoustic".
A software tool that could determine whether a system produces an acceptable
level of noise or heat would be invaluable — but it's probably a pipe
dream. Even thermal testing would be difficult to do properly, due to the
variations of thermal monitoring on different systems, and acoustic testing
seems almost impossible. How could a self-contained system measure how it
sounds from a meter away? Even if a sound meter was embedded in the motherboard
chipset, how could it determine how the system sounds from a standard position outside the
case?
CONCLUSIONS
There is much to like about Intel's new Capabilities Assessment Tools. Measuring
performance in terms of subjective user experience is a significant step forward.
So is the idea of measuring a "system" instead of a single piece of hardware.
And, last but not least, evaluating a system in terms of its capabilities rather
than abstract numerical terms is exactly the way we believe it should be evaluated.
So much for the good. The theoretical goals behind the CATs are beyond reproach,
but these lofty goals must contend with the limitations of what can be tested,
and determining what affects user experienceis by no means simple.
In their current form, Intel's tools do not yet feel complete. Intel has recognized
this, and has promised much in the roadmaps for the two tools, but many questions
still remain.
Group perceptual research is useful for determining what
level of performance on specific parameters is satisfactory, but it says very little about what factors
contribute to user experience. Before any market research is done, a specific
theory about what affects user experience must be developed. In other words,
there may be factors that affect user experience that are simply not being measured
because nobody has recognized that they are significant.
This is why it is helpful to examine as many factors as possible. This is especially
important for the DH-CAT, which basically provides the user with a set of checkboxes
for each possible task. The more checkboxes there are, the more likely it is
that the tool will be helpful. This approach is commendable — it moves
away from trying to simplify performance as a single number — but if the
checkboxes don't reflect what users are actually doing with their computers,
it's not much help.
All in all, the Capabilities Assessment Tools reflect a holistic approach to
computer performance. Thinking about performance in
terms of capabilities and actual benefits is a big step towards maturity
in this industry. Perhaps the biggest obstacle that the tools face is not their
actual implementation, but convincing people that this is the way to think about
performance, and helping them adapt to the language of user experience rather
than bandwidth, throughput and latency. In addition, hardware sites will
need to re-think their test methodologies, since the CATs test entire systems,
not separate components.
Another potential obstacle is the fact that the market research is based on
the "average user", while most review sites cater to the enthusiast
market. While the occasional drop below the FPS threshold hardly matters to
a casual gamer, it may be of great concern to a serious gamer who has been
lag-killed. Similarly, the default settings in the G-CAT tool reflect the settings
used by everyday users, not enthusiasts. The same issue applies to the research
behind video quality: How does the opinion of an "average user" apply
to a couch-potato who has trained his eye to notice visual flaws? Has too much
detail been glossed over by calibrating the "acceptable" level of
quality to the average user's tastes?
Intel has some appreciation of the challenges to acceptance of their new performance
assessment approach. They've taken steps to try and include the media as well
as early adopters among PC users by creating a site about the new CATs: http://www.intel.com/performance/newtools.htm.
In the IDF Fall 2005 press conference where much of the info here was presented,
Intel promised to make both of these tools available for free download. They
also strongly urged the hundred or so attendees for feedback on the tools to
make them better. SPCR staff members have been invited to participate in the beta
testing program for both tools. There appears to be an open attitude regarding
the development of these tools in order to ensure that the end results will
be embraced by the review community.
In the end, we can toss in our two cents worth, then wait and see. Once the tools are released, it is a matter of enough testers testing enough systems to verify the accuracy of the tools in predicting user experience. It's safe to predict that flaws will surely be found, and the tools will likely evolve to keep up with changes in PC components.
* * *
EDITOR'S NOTE: We believe Intel's new approach to performance assessment is an important step for the industry. It's possible that they're being developed partly because Intel has been edged consistently by AMD in standard processor benchmarks for the last year or two. A scan through the major gaming and performance oriented web sites shows a broad consensus of praise for AMD's top models as the processors of choice. It's possible that this step down from the performance pinnacle of market perception is a factor. A new way of assessing PC performance that deemphasizes the role of raw speed might be helpful to cool the current enthusiasm for AMD processors among performance nuts. This is conjecture, indeed. In our view, it hardly matters. What does matter is that real users' experience and perception are finally being included in assessment tools being developed by the most powerful entity in the PC hardware world.
We've been invited to visit the labs in Oregon where some of Intel's research on acoustics is being done. One fascinating and ambitious project undertaken by Intel was to measure the ambient noise levels inside certain "standard" types of buildings in selected cities around the globe. This was an effort to answer the question, What is typical ambient background noise? It's an issue that is fundamental to silent computing. In general, a computer only has to be quieter than ambient by some small degree (aha, another research project!) for it to be effectively silent. Some of the cities mentioned were Shanghai, NY, and Berlin. Obviously, there are many methodological challenges, but the simple fact that Intel has embarked on such a research project bodes well for the silent PC future. We'll certainly make a report if this visit materializes.
* * *
Comment on this article in our Forums.
| Help support this site, buy from one of our affiliate retailers! |
|
