Real Device Testing
From bemoko developer wiki
Contents |
Introduction
Real device testing is the testing of an application or web site on a real device. This can take several forms:
- Real devices in the hands of your test team.
- Real devices in the hands of real users - closely tied into usability testing.
- Nothing beats actual real users using your site, pressing buttons and doing things "they shouldn't". No matter how well planned your test scripts and process is a real user will always often off script and try something unexpected.
- Remote access to real devices[1]
- Outsourcing, such as crowdsourcing[2], of device testing.
Test Strategy
A mobile web site is driven by storyboards and style guides that describe the page flow of the site and the expectations of what each page should look like on a variety of device classes. Device testing should verify that the deployment satisfies these expectations on a range of devices, termed hero devices, that represent the different classes of devices.
Real device testing should take place along side device simulation testing and automated testing. Given that device simulation testing is often more straightforward than real device testing, there are benefits in verifying that the rendering of a web site in a common PC browser looks OK before they are verified in a real device. Furthermore it is useful to verify the functionality of site with automated tests prior to trying it out on a real device. Both the rendering and functionality of a site stand a much better chance of working on a real device if they pass the device simulation and automated tests
... but that does not mean to ignore some real device testing. Separation of concerns is a powerful thing - you can verify the rendering and style guide compliance of site independently of functional testing. The way the bemokoLive UIs are developed in parallel to content integration allows you to verify the static rendering without requiring the underlying functionality to be hooked in. An easy way this can be done is by doing the rendering tests on a site that extends the core site with the plugins stubbed out. This technique is especially helpful when the full system integrates with an internal and secure data service, since the rendering can be developed and tested in isolation of a full stack. With this in mind, there is no excuse in not deploying and testing early.
Dealing with Non-Compliance
Most browsers are non-compliant to web standards in some form or another. This is not surprising since (1) software invariably has bugs and (2) "web standards" is not a well-defined term - which web standards? - and (3) standards evolve, cf. HTML5. To deal with suspected non-compliance issues it's important to understand which standards a browser is aiming to be compliant with, which standards it is partially compliant with and which extensions (sometimes vendor specific) it also supports. The "Web Compatibility Test for Mobile Browsers", ACID2 and ACID3 tests described below help rate a given browsers compliance.
With that in mind, style guide desires are not necessarily possible on all devices. bemokoLive helps with optimisation of the device markup so that you can deliver to the best of the capabilities of a given device, but the capabilities of a given device, may fall short of the desired design. In these cases the style guide should describe how the experience should degrade so that when you test a site on a particular device it is still functional and usable, at the same time as delivering to the capabilities of the device.
Testing Browser Compliance
The WAP 2.0[3] standard, which most modern browsers are compliant with is based on XHTML Mobile Profile[4].
The W3C provides a "Web Compatibility Test for Mobile Browsers" @ http://www.w3.org/2008/06/mobile-test/ (or colour blind friendly version @ http://www.w3.org/2008/06/mobile-test/nored/) which gives you an indication of how compliant a browser is.
Many mobile browsers are going beyond the standards of XHTML Mobile Profile and supporting features in HTML4 or even HTML5. The ACID2[5] test @ http://acid2.acidtests.org/ testing for compliance against HTML4, CSS 2.1 styling, PNG images, and data URIs. The ACID3[6] test goes further and tests for other aspects, including JavaScript and DOM traversal.
References
- ↑ DeviceAnywhere - http://www.deviceanywhere.com/
- ↑ mob4hire - http://www.mob4hire.com/
- ↑ WAP2.0 http://en.wikipedia.org/wiki/Wireless_Application_Protocol#WAP_2.0
- ↑ XHTML Mobile Profile - http://en.wikipedia.org/wiki/XHTML_Mobile_Profile
- ↑ ACID2 - http://en.wikipedia.org/wiki/Acid2
- ↑ ACID3 - http://en.wikipedia.org/wiki/Acid3