Many instructional technologies do not achieve high levels of use in an institution. Others are adopted but quickly abandoned. Technologies fail for myriad reasons, but many failures can be avoided if a comprehensive evaluation process is used before adoption. We analyzed the reasons for failure and, from that, developed the USA2 framework to guide institutions in asking the right questions before adopting a new technology.

The USA2 framework includes six adoption criteria: utility, security, accessibility, usability, scalability, and affordability. Questions related to each of these criteria can be found at https://mycsun.box.com/v/USA2-Framework. We have found that these categories cover the critical issues that schools must be examine when deciding whether to adopt a new technology.

Utility

Utility relates to whether the product provides features users need. A “cool” set of features isn’t worth much if it doesn’t help users do what they want or need to do. The most important considerations are what the tool can do and whether it improves teaching or learning (or both) in ways that tools are already in use cannot. For example, if faculty are happy with PowerPoint, they are unlikely to deem another slide-analogy presentation tool worth the effort to learn unless it provides additional features that they think unique and useful. Likewise, when replacing a tool with a less expensive one, the school must take care to evaluate whether the features frequently used in the original tool are available in the proposed replacement. Even apparently minor features can be important if faculty have grown accustomed to using them. For instance, an institution might replace its classroom and desktop video capture system with a cheaper one that captures only classroom video on grounds that there are many alternatives for doing screen recording. But faculty might balk at using two systems instead of one and may give up on the whole video capture endeavor.

Security

Instructional technology security has three essential components: confidentiality, integrity, and access for authorized users. Confidentiality means that users’ private information is kept from those who have no legitimate reason to access it. Integrity means that data is safe from malicious attack. User access, the third component, is often sacrificed in the interests of the first two. A system is secure when two-factor authentication, which requires the user to enter a password and a code received on a mobile or other device, is used; however, the user may find this process to be unacceptably slow and tedious. In this way, security often strikes a balance between competing interests.

Accessibility

Accessibility refers to the ability of individuals with disabilities to access electronic information. For a technology to be accessible, it must enable all users to access information and complete tasks with essentially the same degree of ease, regardless of the presence of a disability or use of assistive technology. Equity and fairness require schools to ensure that all instructional technology is accessible. Additionally, accessibility is the law. Sections 504 and 508 of the US Rehabilitation Code, as well as the Americans with Disabilities Act, dictate that those with disabilities should have the same access as others.

When evaluating accessibility, a good place to start is the Voluntary Product Accessibility Template (VPAT), which should be available from all vendors based in the United States. The VPAT indicates how information and communication technology products meet standards for accessibility and identifies any areas where standards have not been met. In addition, institutions should ask whether any accessibility testing has been done and whether the results of testing are available. If the school can access the tool prior to purchase, it can assess for important accessibility features. These include ease of navigation without a mouse (many users with vision or mobility impairments may not be able to use a mouse); the presence of alternative text for screen reader users; closed captions for those who cannot hear or have learning disabilities; and appropriate use of color. (Is contrast between text or controls and backgrounds high enough? Are alternatives provided when color is used to impart meaning?)

Usability

Usability relates to how easy and pleasant a product is to use. Jakob Nielsen (2012) defines usability as having five components:

• Learnability: Is the tool easy to learn how to use?
• Efficiency: Can tasks be accomplished quickly with the tool?
• Memorability: Is it easy to remember how to use the tool after not using it for a while?
• Errors: Are errors made frequently? Is it easy to recover from errors?
• Satisfaction: Does the user enjoy using the product?

When evaluating usability, schools can assess for such matters as adherence to common standards or conventions, consistent navigation, and error handling.

Sustainability

Sustainability is the product’s ongoing ability to meet users’ needs. It can include the level of support needed over time to enable users to learn and use the tool. It can also involve scalability—the capacity of the technology to support increased usage over time, with increased demands on hardware, software, data storage, or bandwidth. The vendor must be able to remain available to update and improve the product as needs change.

Finally, when use of the tool must be discontinued—either because of a move to another tool or because the tool is no longer available—there could problems with data loss if any stored data cannot be exported in a usable format. Anyone who has done a learning management system migration understands some of the pain involved with this last point.

Affordability

Affordability includes both the direct and indirect costs of acquiring and using the technology. Direct costs may include licensing, data storage, and user training. Indirect costs include any administration or instructional designer/technologist time necessary to support the technology. Anticipated increases in cost over time must be considered. Affordability can be evaluated by looking at the direct and indirect costs of the tool; consideration should be given to whether the cost is likely to increase significantly over time.

Using the USA2 framework for decision-making

When choosing a new technology, institutions should start by identifying everyone who will have a stake in the decision. Each group—for instance, administration, finance, IT, faculty, and students—may have different and even competing needs and priorities. These stakeholders should help identify the relative importance of the six criteria as they apply to the specific technology and the context in which it will be used. Then the tool should be evaluated on each of the six criteria in the USA2 framework.

Institutions might want to use a weighted decision matrix like the ones provided at https://mycsun.box.com/v/USA2-weighted-matrix. After assigning scores to the criteria according to their relative importance, they can evaluate the tool for how well it meets each of the criteria, again assigning a numerical score. Multiplying the two numbers and then adding for all six criteria will result in an overall score, which a school can use to evaluate a single tool or compare multiple tools. Alternatively, to take a more holistic approach, stakeholders can get together and discuss a tool and which criteria it meets or does not meet, reaching a consensus on whether to adopt it. Either way, we believe that the USA2 framework will help to improve instructional technology decisions.

Reference

Nielsen, J. (2012, January 4). Usability 101: Introduction to usability. Retrieved from https://www.nngroup.com/articles/usability-101-introduction-to-usability

Suzanne David, MA, is the e-learning technology manager and Maria Fernandez, MS, is the assistant director of distance learning at California State University, Northridge.