 |
| About Bioline | All Journals | Testimonials | Membership | News |
|
||||||
|
||||||
Tanzania Journal of Health Research, Vol. 10, No. 1, January, 2008, pp. 39-45 The reliability and usability of district health information software: case studies from Tanzania J.H.LUNGO Department of Computer Science, University of Dar-es-Salaam, P.O. Box 35062, Dar-es-Salaam, Tanzania Received 10 September 2007 Revised 19 December 2007 Accepted 20 December 2007 Code Number: th08008 Abstract The District Health Information System (DHIS) software from the Health Information System Programme (HISP) based in South Africa is widely implemented in many developing countries as a health data analysis tool. Through the HISP Tanzania project, the DHIS was piloted in five districts in Tanzania. The objective of this study was to qualify and quantify the extent to which district health workers consider the adaptation of the DHIS software to the needs of the routine health management information system. In a period of 14 months (from June 2003 to August 2004) data were collected from health workers trained to use the software through a triangulation of various qualitative data collection techniques including interview, questionnaire, participant observations and retrospective testing. The software was evaluated in terms of reliability, usability and user satisfaction. In general the reliability of the software was rated high but its usability was rated to be low. The software was found not to accommodate some health data from various health programmes and there was a mismatch between the implemented online data entry forms/reports and their respective paper-based forms/reports. The study recommends improved design of the DHIS user interface (forms) and reports to replicate the paper-based forms in order to assure usability and reduce the incidences and impact of human errors in the keying-in of health data. Keywords: district, health information, software, reliability, usability, Tanzania Introduction The World Health Organization (WHO) defines Health Information System (HIS) as a system that integrates data collection, processing, reporting and use of the information necessary for improving health service effectiveness and efficiency through better management at all levels of health services (WHO, 2004). The collection, collation, compilation, analysis and reporting of health data in HIS of most developing countries are riddled with major problems. It is usually mentioned that the data received are often not helpful for health management decision-making because they are incomplete, inaccurate, untimely, obsolete and unrelated to priority tasks and functions of local health personnel (Lungo, 2003; Kimaro, 2006). In a recent situation analysis in Tanzania, it was observed that at the facility and district levels, reporting systems are weak, both in terms of completeness and timeliness (Rumisha et al., 2007). This situation is likely to be exacerbated by gaps in health knowledge and information communication in Tanzania. One such gap is lack of adequate knowledge and information exchange capacities among the health providers and the ability to share that information with the targeted community (Mboera et al., 2007). Wilson (2000) suggests that the creative use of Information Technology (IT) in health-care systems is one of the most promising means of improving the quality, timeliness, clarity, presentation and use of relevant information for primary health care. In this case, IT is regarded as a solution to fill the gap in information exchange and presentation. Kimaro & Twaakyondo (2006) suggest that through the use of IT, healthcare sectors can potentially plan, monitor and evaluate health services as well as communicate more effectively within and across organisational hierarchies. Computer systems could be used to create a single health data repository for all stakeholders such as vertical programmes. The Ministry of Health in Tanzania has long recommended a computerised health information system at district, regional and national levels (MoH, 2000). However, until 2002, health data were processed manually at the healthcare facility and district levels. In 2002, the University of Dar-es-Salaam and the University of Oslo initiated a research project known as Health Information System Programme (HISP) to implement computer software at district medical offices in Tanzania. The District Health Information Software (DHIS) was piloted in five districts: Ilala, Kinondoni, Temeke, Bagamoyo and Kibaha. DHIS, which was originally developed in South Africa, is a Free Open Source Software (FOSS) which gives countries an opportunity to get the software free of charge and to make use of local expertise to customise it according to local needs (Braa & Hedberg, 2002). Software customisation includes mimicking manual health data collection tools to look similar on the software data entry forms, accommodating most routine data elements, and translating the software into the local language. The open source based nature of the DHIS is one of the main factors which promoted the software’s adoption in several countries including Tanzania, Mozambique, India, Malawi and Ethiopia (Braa et al., 2004). Implementing a new system or changing an existing one may have a profound effect on work and service processes. Because HIS is costly and involves a significant amount of staff time, it is important to evaluate these systems critically and learn from ongoing experience in order to improve the utilisation of existing systems (Wyatt & Wyatt, 2003). Evaluation in this context refers to the systematic collection and analysis of information to support decision-making in projects, processes or methods (Häkkinen et al., 2003). Because of likelihood of contextual differences from one country to another, there was a need to evaluate the potential of the DHIS in terms of its reliability and usability in Tanzania. The objective of this study was to determine the adoption of DHIS software in the management of health information at both facility and district levels and to evaluate users’ satisfaction with the performance of DHIS as a surrogate measure of the effectiveness of this software to be used as a health data analysis tool in Tanzania. Materials and Methods Study area The evaluation was carried out in Bagamoyo, Kibaha, Ilala, Temeke and Kinondoni district in eastern Tanzania, where the DHIS was implemented from June 2003 to August 2004. Bagamoyo and Kibaha were the first districts where the DHIS was installed and piloted in 2002, and were supposed to be the blueprint of the DHIS in Tanzania. The strategic approach was therefore to include Kibaha and Bagamoyo and complement the observations from these districts with those from the new sites of the DHIS in the other three districts. The study used a combination of purposeful/ judgemental and snowball/chain sampling procedures (Odhiambo-Otieno, 2005; Silverman, 2006; Wilson & Smith, 1991). The logic and power of purposeful sampling lie in selecting information-rich cases to study in depth (Patton, 2002). There were two categories of samples: study areas n=5 districts (based on purposeful/judgemental sampling procedures) and respondents n=18 health workers (based on outcome of snowball-sampling procedures). Interviewees were identified as a result of snowball-sampling to follow information-rich informants starting with the participants in a DHIS training workshop. The rationale was to get those informants who were actually involved in using the DHIS software, that is, those from the districts where the software was installed and who have been assigned to use the software. Data collection Questionnaires were distributed to DHIS users during a training workshop. Information collected on the questionnaires included age, educational background, computer use experiences and users’ satisfaction with the DHIS: they marked a value on a scale (accuracy, failure rate, training, support, format and content). Then the users left with the questionnaires, which were collected by the investigator after one or two weeks on DHIS on-site visit support rounds. The questionnaire tool only allowed close-ended questions where a user was required to rate his/her level of satisfaction with the DHIS using a value on the scale on the questionnaire for each aspect of the software characteristics (performance, training, support, content and format). Semi-structured interviews with open-ended questions were directed to all respondents. Some of the questions were directed during the DHIS user training sessions and others were randomly asked of different subjects during fieldwork support. During the training sessions, a video-recorder was used to record questions and answers, but during the fieldwork visits, notes were written down in the researcher’s diary. The investigator, who was among the developers of the DHIS software in Tanzania, made observations on the way the DHIS users interacted with the software in their work-place and the way programmers implemented users’requirements in the software. Field notes were recorded in a diary when an interesting event occurred, for example, when novice bugs (bugs owing to inappropriate operation of the software) were identified. A retrospective testing as an evaluation method at the design, coding, testing, and deployment stages of the software life-cycle was done. The testing covered efficiency, effectiveness and satisfaction usability issues through quantitative data. After training sessions, users were given completed quarterly report then asked to enter the data in the DHIS software and generate a similar report from the software. At the end of the assignment session, discussion was done with users who did not manage to accomplish the assignment. This was sought important to explicate difficulties encountered. Videotapes of usability tests sessions were made to collect more information on the participants’ behaviour during the test process. Data analysis Questionnaire data were entered in a computer package known as SPSS version 11.0 for windows (SPSS Inc. 2000) on two different files. The two files were then compared and the consistency between them assessed in order to detect data entry errors. The data were analysed by descriptive statistics where frequency and cross-tabulation were computed. Significant differences were defined as those with P<0.05. Video-recorded interviews were transcribed into a word processor document for Microsoft Office Word 2003 (Microsoft Corporation). The handwrit ten fieldwork notes were also entered into the word processor document. The field notes from participant observations were entered into the same document where the interview notes were transcribed. This provided a richer description of the interview notes. The retrospective tests were mapped to the user characteristics in the SPSS package in order to find out if there was a link between user characteristics and their levels of understanding and working with the software quickly. Scaling of seven intervals was quantified by assigning the values -3, -2, -1, 0, 1, 2, and 3 to the intervals. Data collected from different methods were transcribed and encoded and fitted into the scales. User reaction toward the software was analysed in different through the questionnaire, from observed data, and actual use of the software, then a point from the 7-point semantic differential was selected. The concept of summated scales, which are formed by combining several individual scales into single composite measures, was used. All high scales loadings on factors were combined, and the average score of scales was used as a replacement scale. Total user satisfaction was obtained by averaging the scores on the five scales presented in Table 1. Conceptual model and hypothesis In this study user satisfaction with the DHIS software attributes of reliability and usability was assessed in depth. For the purpose of data analysis, five scales were introduced: accuracy, failure rate, training and support, content and format. The first two scales measure reliability factors and the last three scales measure usability factors. Accuracy intended to measure accuracy of the output of the system while failure test measured the frequency of failure of the software. As regards to usability, the training and support intended to measure effectiveness of training and support delivered to users which contributed to ability to learn. On the other hand, content and format intended to measure adequate coverage of information and clarity of software reports, respectively. All scales were of 7-point semantic differential type adopted from Chin & Lee (2000) model. User characteristics considered in the analysis include factors such as age, education level and computer literacy of end-users (Barki & Hartwick, 1994; Winston & Benjamin, 2000), while software characteristics include aspects of content, accuracy and failure rates (Mayhew, 1999; Nielsen, 1993; Pan, 1999; Sommerville, 2001). Results Results are presented in descriptive statistics and qualitative excerpts of encoded user responses during interviews and retrospective testing. Study respondents’ ages were divided almost equally, with 56% under 40 years old and 44% over 40+ years. Thirty-three percent (N=18) of the respondents had university education and 39% (N=18) of them had acquired a formal computer training (had a certificate or higher qualification in computer training). Reliability The quantitative results of the study indicate that respondents have rated the reliability of the DHIS software with an average of 2.11 out of 3.00 points (70% user satisfaction). The specific scales scored as follows: accuracy scored 1.89 and failure rate scored 2.44 out of 3 points. One interviewee, who entered a completed paper forms and compared it with the report computed by the software, noted “…the software is very accurate. When I summed up the values entered in the software manually, I still got the same results as the software printout” (DHIS User, Kibaha). Usability The usability of the software scored 1.70 points out of 3.00 points (56% user satisfaction). The content of the software was rated poor at 0.89 (i.e. 29% user satisfaction) (Table 1). In terms of the extent to which the DHIS software can accommodate various health data, one user commented that, “No EPI data set, no HIV data, it is missing a lot of health data contents” (DHIS User, Bagamoyo). In addition, users noted that for those forms implemented in the DHIS software their layout did not look like the paper-based form. They noted that although the DHIS’s online data entry form had all required fields as in the paper form, those fields were arranged in a different order. That is, field one on the paper form could be field six on the computer. The same was true of the electronics reports generated by the DHIS (the reports are not designed to look like the paper-based reports). Users explained that it is important for the software developers to keep the same layout design of the forms and reports otherwise health planners would reject them. Generally, the DHIS users and developers agreed that the software is a customisable tool that can fit in with the structure and content of the Health Management Information System in Tanzania. For example, health data being collected at the health facility level are categorised into two groups: routine health data and annual health data. The DHIS development team instructed DHIS users to enter quarterly reported data in March, June, September and December for quarters 1, 2, 3 and 4 respectively. To fully accommodate annual data on health facility infrastructure (hospital beds, cars, tables, staff, etc.), population data with different age groups and coverage level (e.g. district population, health facility catchments area population), the DHIS captures them in a special form called “semi-permanent data”, which is updated annually. In terms of accommodating health indicators, the DHIS software has an indicator definition form and automated spreadsheet pivot-table which givespermanently. This has been reflected in the DHIS software. While registering a health facility in the DHIS software, users are provided with options to indicate whether that health facility is active (reports health data) or passive (does not report health data). In doing this, the DHIS has reflected and accommodated the actual status of a health facility. User satisfaction Results indicate that user age was not positively correlated with user satisfaction with the software attributes (usability and reliability) (Table 2). There was a slight correlation of 0.05 significance between user satisfaction and their education background. Computer literacy played an important role in user satisfaction with ease of use of the software (see cor relation significance of 0.01). Evaluation showed that the training and support activities of the DHIS development team scored the highest user satisfaction with an average value of 2.61. Analysis of interview empirical data, however, indicated that most DHIS software trainees complained that one week-training was too short for new users to become familiar with the software. In retrospective testing, some users demonstrated the ability to register a new health facility, key routine health data and generate standard reports on the last day of the training sessions. Working with advanced features like indicator definition, creating new data elements and working with Microsoft Excel pivot-tables were more slowly adopted during on-site training and support sessions. Discussion The results of this study indicate that the DHIS software was rated as reliable and accurate software. The potential of this software to address the challenges of presenting health data information was also immediately noted by users as positive. However, more work needs to be done, to ensure that the DHIS software computerises all health data that are collected at the district level that would provide a comprehensive district database. This is important because, in addition to the routine Health Management Information System, at the district level vertical disease control programmes such Tuberculosis/Leprosy, Expanded Programme on Immunisation, HIV/AIDS have specific information systems (Mghamba et al., 2004). The DHIS was rated low in terms of its usability. Specifically, the capacity of the software to accommodate many types of health data was the major area that needed improvement. The argument here is that, in order for the software to be used as a health data analysis tool, it should accommodate all health data being reported instead of accommodating just a few types of the records. In this regard, the DHIS software falls short. The problem of incomplete data in the DHIS software is associated with problems of the Health Information System (HIS) itself. The HIS is fragmented in such a way that there is a need for streamlining the health data collection tools of various systems into one before they get computerised in the DHIS software. Since the data generated through the HIS are incomplete, the information entered in and generated from the DHIS software is equally regarded as incomplete (Kimaro & Twaakyondo, 2005). Thus, the problems of HIS in Tanzania are being propagated to the DHIS software. Some of those missing health data types are owned by separate vertical disease programmes over which even the national HIS, which is being computerised through the use of the DHIS software, has no control. All HISP Tanzania training workshops lasted for one week (5 working days) followed by two days of on-site training and monthly support. This duration was found to be effected by most of the users. Though HISP Tanzania training and support was rated as effective, most users felt incompetent in working with the DHIS. Generally, staff with high computer literacy demonstrated the ability to learn the DHIS software faster than their computer novice colleagues. Cole (2004) proposes that if the skills of mouse control, keyboard and Windows navigation requirements is compared that are needed to undertake information literacy then information literacy should be viewed as a broader concept that includes computer literacy. As an example, it is very difficult to undertake a CD-Rom database search (information literacy) on a computer if the user is unable to log on to the computer and use a mouse (Cole, 2004). Leland et al. (2000) propose computer literacy competences (minimum knowledge) to be considered necessary. Use of word processing programs effectively; use of library databases and catalogues to locate print materials; finding information on the Internet and evaluate its reliability and usefulness; being able to write email effectively and appreciating the ethical issues of computing is a minimum computer literacy competences for any one who need to work with computer based information systems (Leland et al., 2000). Since, the DHIS development team is asked to keep the same layout of the paper forms and reports when computerising district data, this is likely to reduce data entry errors and also will shorten learning curves for the software. Those who know how to enter data on the paper forms should be able to master the electronic form when they are available in the same layout. Similar, this is expected to facilitate easy data interpretation by decision-makers who are used to paper-based reports. In general, the DHIS software is perceived as reliable software with just a few failing areas. According to this study, however, the software falls short in terms of contents. With further refinement, the software application has the potential to improve management of health information at district level. The DHIS is free open source software which is distributed with its source codes. It gives the local developers the opportunity to localise the software according to local needs (Câmara & Fonseca, 2007). Reliable and timely health information is the foundation of public health action. However, this is not the case for many developing countries like Tanzania. It is often unavailable due to under-investment in systems for data collection, analysis, dissemination and use (WHO, 2007). With the introduction of open source DHIS software in the country it is expected that to facilitate data management and utilization at the district level would be improved. The study recommends that in customisation of the software mimicking the paper-based data entry forms in it is important. In addition, the lesson is drawn that computer illiteracy is a bottleneck in the implementation of computer systems in healthcare systems. This suggests a re-organisation of user training, where, instead of concentrating on training health workers to manage the computer database systems, the training should include computer literacy courses on operating systems, word processor and spreadsheet software, since it is shown that users with a computer background learn the software faster than their inexperienced colleagues. References
© Copyright 2008 - Health User's Trust Fund (HRUTF) The following images related to this document are available:Photo images[th08008t2.jpg] [th08008t1.jpg] |
| |||||||||
