Intel's new Capabilities Assessment Tools

The Silent Front
Viewing page 5 of 5 pages. Previous 1 2 3 4 5

LIMITATIONS AND DRAWBACKS

In their current form, the tools are still quite limited. Neither feels like it has progressed beyond the beta stage, and there are a multitude of factors that might be relevant but have not yet been researched enough to know whether they affect the user's experience. That said, Intel also does not yet consider either tool a finished product. Work is still being done on both tools, and it will continue even after they are released. Both tools have substantial roadmaps for future features and improvements. For the G-CAT, this means further research on different games (perhaps including new genres), while the DH-CAT can expect to see more primitive tasks and a completely new category: Premium Connected.


Many improvements for the G-CAT are planned.

Of the planned improvements to the G-CAT, the most pressing is probably the need to determine the effect of different game settings. During the presentation of the tool to the press, many people expressed concern that the settings used — default graphics options at 1024 x 768 resolution — does not test the full capabilities of most graphics cards. This is less of a concern that it might seem — good quality testing is possible no matter what settings are used so long as the basic test parameters remain the same — but expanding the capabilities of the tool to encompass more variables could definitely improve the reliability of the end result.

The amount of research needed to make this feature possible is substantial. Proper experimental techniques dictates that only one setting can be changed at a time, which means that testing the effect of multiple settings requires multiple tests and an increase in the sample size. In my opinion, the basic settings used in the initial market research are perfectly adequate for the tool in its present form. The average gamer often does not even look at the graphics options in his games — he simply plays at the default settings unless he needs to turn things off for performance reasons.


Some of the features that may show up in future versions of the tool.

Of the features listed above, the most important from the standpoint of user experience is probably A/V sync. Intel has stated that this feature is in active development, so we expect to see it sooner than later.

For SPCR, the most interesting feature is "Thermal / Acoustic". A software tool that could determine whether a system produces an acceptable level of noise or heat would be invaluable — but it's probably a pipe dream. Even thermal testing would be difficult to do properly, due to the variations of thermal monitoring on different systems, and acoustic testing seems almost impossible. How could a self-contained system measure how it sounds from a meter away? Even if a sound meter was embedded in the motherboard chipset, how could it determine how the system sounds from a standard position outside the case?

CONCLUSIONS

There is much to like about Intel's new Capabilities Assessment Tools. Measuring performance in terms of subjective user experience is a significant step forward. So is the idea of measuring a "system" instead of a single piece of hardware. And, last but not least, evaluating a system in terms of its capabilities rather than abstract numerical terms is exactly the way we believe it should be evaluated.

So much for the good. The theoretical goals behind the CATs are beyond reproach, but these lofty goals must contend with the limitations of what can be tested, and determining what affects user experienceis by no means simple. In their current form, Intel's tools do not yet feel complete. Intel has recognized this, and has promised much in the roadmaps for the two tools, but many questions still remain.

Group perceptual research is useful for determining what level of performance on specific parameters is satisfactory, but it says very little about what factors contribute to user experience. Before any market research is done, a specific theory about what affects user experience must be developed. In other words, there may be factors that affect user experience that are simply not being measured because nobody has recognized that they are significant.

This is why it is helpful to examine as many factors as possible. This is especially important for the DH-CAT, which basically provides the user with a set of checkboxes for each possible task. The more checkboxes there are, the more likely it is that the tool will be helpful. This approach is commendable — it moves away from trying to simplify performance as a single number — but if the checkboxes don't reflect what users are actually doing with their computers, it's not much help.

All in all, the Capabilities Assessment Tools reflect a holistic approach to computer performance. Thinking about performance in terms of capabilities and actual benefits is a big step towards maturity in this industry. Perhaps the biggest obstacle that the tools face is not their actual implementation, but convincing people that this is the way to think about performance, and helping them adapt to the language of user experience rather than bandwidth, throughput and latency. In addition, hardware sites will need to re-think their test methodologies, since the CATs test entire systems, not separate components.

Another potential obstacle is the fact that the market research is based on the "average user", while most review sites cater to the enthusiast market. While the occasional drop below the FPS threshold hardly matters to a casual gamer, it may be of great concern to a serious gamer who has been lag-killed. Similarly, the default settings in the G-CAT tool reflect the settings used by everyday users, not enthusiasts. The same issue applies to the research behind video quality: How does the opinion of an "average user" apply to a couch-potato who has trained his eye to notice visual flaws? Has too much detail been glossed over by calibrating the "acceptable" level of quality to the average user's tastes?

Intel has some appreciation of the challenges to acceptance of their new performance assessment approach. They've taken steps to try and include the media as well as early adopters among PC users by creating a site about the new CATs: http://www.intel.com/performance/newtools.htm. In the IDF Fall 2005 press conference where much of the info here was presented, Intel promised to make both of these tools available for free download. They also strongly urged the hundred or so attendees for feedback on the tools to make them better. SPCR staff members have been invited to participate in the beta testing program for both tools. There appears to be an open attitude regarding the development of these tools in order to ensure that the end results will be embraced by the review community.

In the end, we can toss in our two cents worth, then wait and see. Once the tools are released, it is a matter of enough testers testing enough systems to verify the accuracy of the tools in predicting user experience. It's safe to predict that flaws will surely be found, and the tools will likely evolve to keep up with changes in PC components.

* * *

EDITOR'S NOTE: We believe Intel's new approach to performance assessment is an important step for the industry. It's possible that they're being developed partly because Intel has been edged consistently by AMD in standard processor benchmarks for the last year or two. A scan through the major gaming and performance oriented web sites shows a broad consensus of praise for AMD's top models as the processors of choice. It's possible that this step down from the performance pinnacle of market perception is a factor. A new way of assessing PC performance that deemphasizes the role of raw speed might be helpful to cool the current enthusiasm for AMD processors among performance nuts. This is conjecture, indeed. In our view, it hardly matters. What does matter is that real users' experience and perception are finally being included in assessment tools being developed by the most powerful entity in the PC hardware world.

We've been invited to visit the labs in Oregon where some of Intel's research on acoustics is being done. One fascinating and ambitious project undertaken by Intel was to measure the ambient noise levels inside certain "standard" types of buildings in selected cities around the globe. This was an effort to answer the question, What is typical ambient background noise? It's an issue that is fundamental to silent computing. In general, a computer only has to be quieter than ambient by some small degree (aha, another research project!) for it to be effectively silent. Some of the cities mentioned were Shanghai, NY, and Berlin. Obviously, there are many methodological challenges, but the simple fact that Intel has embarked on such a research project bodes well for the silent PC future. We'll certainly make a report if this visit materializes.

* * *

Comment on this article in our Forums.



Previous 1 2 3 4 5

The Silent Front - Article Index
Help support this site, buy from one of our affiliate retailers!
Search: