We have found an interesting case study about how a telco software provider evaluated their software and the methods they used. The company is called Telelogic AB and their software is Telelogic Tau.
Telelogic is a provider of software solutions to the telecommunication industry and has more than 8000 customers and operates in 22 countries worldwide. It was completely acquired by IBM in 2008 and exists under the IBM software group.
Telelogic Tau SDL Suite is a tool for designing and implementing real-time software in the telecommunication industry. It consists of a graphical editor and syntax analyzer, a simulator and validator and several optimized code generators for compilation to executable code.
Two evaluation methods were used to test Telelogic Tau for usability. They are:
- Questionnaire – quantitative results
- Heuristics evaluation – qualitative results
Evaluation method 1:
The SUMI questionnaire (Software Usability Measurement Inventory)
- A commercially available questionnaire (created by Dr Jurek Kirakowski) which is a tested and proven method to measure software quality (user experience) from the end user’s perspective
- The results are used to measure
five usability aspects
- Efficiency – the degree to which users feel that the software assists them in their work.
- Affect – the user’s general emotional reaction to the software.
- Helpfulness – the degree to which the software is self-explanatory. Adequacy of documentation.
- Control – the extent to which the users feel in control of the software.
- Learnability – the ease with which the users feel that they have been able to master the system.
How it was done:
- Questionnaire was shared with 90 selected participants in Europe
- 62 were properly filled out and returned
Results:
- Software did not meet the appropriate standards on several aspects of usability
- Figure below shows that only two of the SUMI scales were above average (Affect and Learnability)
Evaluation method 2
- This is a usability evaluation method used to find usability problems in software which can be then solved as part of an iterative design and development process.
- A set of expert evaluators would examine the interface and assess its compliance with a defined set of heuristics such as Visibility of system status, Error prevention, etc.
How it was done:
- In Telelogic, there weren’t many usability experts therefore, experts on the software from within the organization was picked
- They were given specific scenarios and tasks to perform then, given 1 hour to find specific usability problems
Results:
- 72 usability problems were found (20% highly severe, 65% somewhat severe, 14% less severe)
Key learnings:
- These methods can be used even for companies that do not have a lot of experience in usability and user experience engineering
- Not many resources are required
- The two methods complement each other very well
- Management level support is required to successfully carry out any form of evaluation (to accommodate the time and costs required to conduct them)
- Relatively quick results that are also useful for improvement of the software
- This study was conducted in early 2000’s on an older GUI but the evaluation methods are still applicable for today’s modern interface
Sources
Telelogic Tau: https://www.eit.lth.se/fileadmin/eit/courses/ets150/IntroSDLSuite.pdf
IBM acquisition: https://www-03.ibm.com/press/us/en/pressrelease/23792.wss
About Telelogic AB: https://www.bloomberg.com/profiles/companies/TLOG:SS-telelogic-ab
SUMI Questionnaire: http://sumi.uxp.ie/
Heuristics evaluation: https://www.nngroup.com/articles/ten-usability-heuristics/
Case study used: Johan Natt och Dag, Björn Regnell, Ofelia S. Madsen, Aybüke Aurum. (2001). An Industrial Case Study of Usability Evaluation in Market-Driven Packaged Software Development. Retrieved from: https://www.researchgate.net/publication/228892229_An_industrial_case_study_of_usability_evaluation_in_market-driven_packaged_software_development
Usability and User Experience: What's new? 
Nice article! Seems Telelogic Tau has two ways to do their usability evaluation. Both qualitative and quantitative methods’ analysis impress me! Looking forward to seeing more usability article from you.
LikeLike
2 evaluation methods are used, questionnaire and heuristics evaluation. I felt that based on the results shown in the article, both methods seem to be quantitative?
LikeLike
I think heuristics is viewed as a more qualitative method. It’s more of an observation of how usable the system is, don’t you think?
LikeLike
Even though this case has a sense of time, I can still learn a general framework of UE test from it. It used both quantitative and qualitative methods, and had both expert participation and general user participation, is a more comprehensive test.
LikeLike
As you mentioned above, although Heuristic principles had developed many years ago, they are still relevant today because no matter how technology is unrecognizable, the way people behave hasn’t changed as quickly as technology. Thank you for sharing those interesting used of evaluation methods in modern software development.
LikeLike
As you mentioned above, although Heuristic principles had developed many years ago, they are still relevant today because no matter how technology is unrecognizable, the way people behave hasn’t changed as quickly as technology.
LikeLike
As you mentioned above, although Heuristic principles had developed many years ago, they are still relevant today because no matter how technology is unrecognizable, the way people behave hasn’t changed as quickly as technology. Thank you for sharing those interesting use of evaluation methods in modern software development.
LikeLike
As you mentioned above, although Heuristic principles had developed many years ago, they are still relevant today because no matter how technology is unrecognizable, the way people behave hasn’t changed as quickly as technology. Thank you for sharing this interesting article and the use of evaluation methods in modern software development.
LikeLike
Nice article introducing real world ux testing. End user testing is indeed important.
LikeLike
Hi,
Thanks for your share on the case study about the evaluation method of Telelogic. The detailed questionnaire and heuristics evaluation methods help me deeper my understanding about what we have learnt in the class.
– KitKat
LikeLike
Thank you for sharing! This article is quite verbose and useful. Via your analysis for this case study, we learned two evaluation methods. Really helpful!
LikeLike