Objective: In this study we give a detailed description of how to construct verification rules and then evaluate the benefits brought to the laboratory.
Methods: All logic processes and verification rules are constructed in middleware with reference to the CLSI Auto10-A Guideline. There were 569,001 patient test results collected to establish the range of the limit check, delta check, and the consistence rule check.
Results: Daily results show the autoverification (AV) passing rate of all test results to be 92%∼95%. About 80% of test reports can be auto-released.
Conclusions: Individual differences in the verification of test results are eliminated, turnaround time (TAT) is shortened, and full-time equivalent (FTE) are reduced, thus enabling medical technologists to devote more time and effort to handling intercepted test reports, which, in turn, improves the quality of patient care.
Clinical laboratories must respond to challenges such as reducing manpower requirements, increasing service quality, simplifying processes, and decreasing the report release turnaround time (TAT). In addition to the introduction of automated equipment and the development of a laboratory information system (LIS) technology, another way to raise working efficiency is to build an autoverification (AV) system,1–3 by which test reports are AV against report check rules based on the LIS or middleware. Middleware is information software installed between the LIS and the instruments. It delivers information such as the test orders from the LIS to the instrument and the test results of the instrument back to the LIS. In an AV system, the verification rules and the criteria of the test results are built into the middleware, so instead of the results requiring a manual check, they are AV by computer. These verifications include limit check rules, critical values, comparison with former results (delta check), and consistency of related results (consistence check).4 After the check rules are set, each medical technologist performs the test result verifications based on the same judgment platform. In this way, check rules for all test results are standardized.5–7 Furthermore, the AV system can expedite the report check. Central laboratories deal with an enormous number of tests each day and are always under pressure to quickly report the test results. With an AV system, at least 80% of the test reports can be AV without the need of manual intervention, thereby allowing medical technologists to concentrate on the test reports intercepted in middleware. In May 2008, a laboratory automation system (LAS) was installed in our laboratory. After 2 years of operation trials, LAS had taken the place of most of the manual efforts. In 2010, the utilization of middleware in an AV system was planned. In order to validate whether the verification rules could actually be implemented and meet our requirements, a validation and management mechanism based on a CAP checklist and the CLSI guideline4,8,9 was established. This manuscript gives a detailed description of the entire validation and management process, and an evaluation of the benefits brought by the AV system to the laboratory.
Materials and Methods
Collection of Patient Test Results
To define the range of the limit check and the delta check for each test item, 569,001 test results were collected in December 2008. These data were arranged by size, and the distribution percentages of the limit check and the delta check were calculated. These values were then used as the basis for establishing and adjusting the limit check and the delta check of each test item. Also, in order to validate the practicality of the check rules in the AV system, the following was completed: 1) 105,164 patient test results were collected for verification of the check rules and their accuracy in the AV system; 2) 830,233 test results, including 139,650 requisition sheets, were collected for calculation of the AV percentage of all items; and 3) 25,526 test reports were collected for probing into the causes of manual verification (MV) to obtain the true-positivity rate and the false-positivity rate of the check rules in the AV system.
Construction of Information Transfer System
Both the computer algorithm and the technical database of the AV system were built in middleware (DM2; provided by Beckman Coulter, Brea, CA). The construction of the information transfer system is shown in Figure 1. With DM2 as the core center, LIS sends the test orders and patient information to 3 Beckman Coulter DxC 800 biochemical analyzers, 2 DxI 800 immunoassay (IA) analyzers, and an automated track system (PrepLink, Beckman Coulter). The test results from these 5 instruments are sent back to the LIS after verification in DM2, and the LIS then sends the test reports to the hospital information system (HIS).
Computer Algorithm of Autoverification Process
Based on the reference methods of the manual check, which was regularly used in the past, and the CLSI Auto10-A Guideline, the critical value check, limit check, delta check, and consistence check were selected for the verification process (Figure 2), in addition to patient information and an instrument warning flag.
The precondition for using the AV system was that the QC results had to fall within the laboratory’s acceptable limit. Together with test results released from the instruments, patient information and the sample condition were entered into DM2 for verification. The order of validation was the critical value check, followed by the limit check, delta check, and, finally, the consistency check. If the results passed the critical value check but failed the limit check, the delta check was applied, by which the current data were compared with previous data to determine the differences. When the degree of difference fell within the limit of delta check acceptability, a consistency check followed. Test results passing the above check rules were regarded as AV reports, while results that failed any of the above rules were interpreted as manually verified (MV) reports. There are various reasons why checks fail and thus require MV results; hence, medical technologists must make professional judgments based on their experience. When previous data were used for delta check comparisons, the stipulation was that they were to be no older than 7 days.
Construction of Verification Rules
The purpose of using AV was to produce real-time and more accurate test reports with computer verification. Therefore, the infrastructure of the AV rules was constructed so as to increase the precision and efficiency of the data checks.10 The verification rules included a limit check, delta check, and consistency check. In December 2008, 569,001 test results were collected as the basis for constructing the verification rules. The principles for constructing the verification rules were as follows:
Limit Check: Data not falling within the analytical measurement range (AMR) were considered invalid. For a wide AMR, the acceptable range for the limit check was determined using a distribution interval of patient data between 2% and 98%. For instance, 2%–98% of the 14,239 test results of glucose concentration were within the range of 3.6–20.5 mmol/L, while the AMR for the glucose test was 0.2–33.3 mmol/L; 3.6–20.5 mmol/L was thus established as the limit check interval of glucose when the verification rule was applied. The limit check intervals of all 76 items were constructed using the same principle.
Delta Check: Opinions on the scope of acceptability of the delta check are not consistent. False-positive rates rise when the acceptable range is stringent, whereas a high number of false negatives can be expected from the opposite.11,12 Historical delta check data were analyzed, and a relevant difference was assigned as the acceptable range according to the distribution of the delta check of each test item and its clinical specificity in pathological changes. The parameter of the delta check was set between 5% and 200%. For example, 1.) sodium (Na+) differed from the previous results by 5%; 2.) chloride (CL−) by 50%; 3.) potassium (K+) by 20%; 4.) vitamin B12, total thyroxine (TT4), and total triiodothyronine (TT3) by 50%; 5.) there was no delta check for the C-reactive protein (CRP) test when the test value was lower than 8 mg/L, and the difference was 100% when the CRP test value was higher than 8 mg/L; and 6.) the difference with the previous result was 200% for test results lower than 0.5 μg/L, and 50% for test results above 0.5 μg/L for Troponin I.
Critical Value Check As it has been used in operation, the critical value bulletined in our hospital was followed.
Consistence Check As the medical procedures for acute diseases rapidly change, and the clinical test results fluctuate, it was difficult to perform a consistency check on each item, and only portions of the test items were established based on practical and clinical diagnostic criteria. For example, if the test result for thyroid stimulating hormone (TSH) was lower than 0.3 mIU/L, and free thyroxine (FT4) was lower than 18.1 pmol/L, the report was intercepted as an MV report. In addition, at least 30 consistence-check rules were used in the system.
To validate whether the verification rules and their settings in DM2 were able to meet our requirements and could actually be executed, electronic simulated data and special sample validation methods were established as follows:
Electronic Simulated Data Validation In order to support the validity of the limit check, delta check, and consistency check, 25 entries of simulated data were created on simulation software built in DM2. Of these, 13 data entries fell outside the acceptable range in the limit check, delta check, or critical value check, and 12 data entries did not fulfill the consistency check rules or other special rules (data not shown). Through these check rules in DM2, the validation procedure was shown to have a satisfactory performance and to fulfill the necessary requirements.
Data of the 105,164 Test Results Generated From the Trial Run of the Autoverification System
Number of test results
1. Test results that underwent a delta check
(1). Test results failing delta check (MV)
2. Test results failing limit check
(1). Test results failing limit check but without delta check (MV)
(2). Test results failing limit check and delta check (MV)
(3). Test results failing limit check but passing delta check (AV)
Autoverification passing rate % (by test)
Special Sample Validation Special samples included abnormal proficiency test samples and more than 20 patient samples. Most of the test results fell outside the acceptable range of the limit check and the critical value check. These test results of the special samples were selected to validate the AV system’s functionality and the reliability of the reports.
Laboratory Trial Run of Actual Patient Test Results
According to the CLSI Auto10-A Guideline,4 an AV system must be validated using actual patient results upon startup. A total of 105,164 test results from August 18, 2010, to August 26, 2010, were collected and accessed in the AV system to verify the results. The failing and passing rates of the delta checks and the limit checks were then computed and are shown in Table 1.
As shown in Table 1, 11%–17% of all test results underwent a delta check each day, with 10%–14% of the test results failing the delta check. Approximately 2.2% to 3.4% of all test results failed the limit check rules each day and, of these, approximately 71%–83% of the test results did not have previous data available for the delta check. In all the test results failing the limit check, 5%–11% also failed the delta check, whereas 11%–20% passed the delta check. The daily passing rate for AV was 95% to 97%. Due to the uniqueness and specificity of certain medical treatments, there can be numerous associated unpredictable factors; however, the above day-to-day variation was considered acceptable.
Autoverification Passing Rate
There were 76 test items involved in our AV system, with 42 biochemically (BIO) related and 34 IA related. For the purpose of testing the AV passing rate, 830,233 test results containing 139,650 requisition sheets were collected from February 2010 to May 2010. By category, the average passing rate was 96.1% for the BIO-related test items and 93.9% for the IA test items. The overall passing rate for the BIO and IA tests was 95.6%. In terms of the test requisition sheets, the passing rate was 81.5% (Figure 3). These reports required no manual check and could be AV. Because each requisition sheet usually included several test items, MV was necessary if 1 item failed any of the verification rules. Therefore, the AV passing rate calculated in terms of the test requisition sheets was far less than those completed for individual test items (81.5% vs 95.6%). In addition, when compiling the statistics of the 830,233 test results of the 76 test items, the results showed that 62 test items had a passing rate higher than 90%, 9 had a passing rate between 81% and 90%, 3 were between 70% and 80%, and 2 test items were lower than 70% (Figure 4).
Cause Analysis of MV Test Reports
To understand the causes necessitating the requirement of manual intervention and, thus, MV data, a total of 25,526 patient reports were collected, with 4903 reports being classified as MV. The causes of MV reports were analyzed, and the results are shown in Table 2. Of these MV reports, 41 reports (0.8%) had remarkably abnormal data, which may have been due to sample contamination or anticoagulant interference. Another 300 MV test reports (6.1%) were marked with error flags from the analyzer: the data were questionable and all were intercepted for further confirmation. Another 4562 MV test reports (93.1%) were classified as acceptable MV data, since they were released without the need of any modification to the original test results after consultation with a clinician. Hence, it could be estimated the true positivity rate of the intercepted reports was 6.9%.
↵†Acceptable MC data released after communicating with clinician.
In the given validation of the check rules for the test reports, patient test results were used as the main framework of the validation (Table 1), as their representation of the actual patient results spanned the ranges acceptable for most rules of validation. However, some of the test results were found to be exceptionally abnormal and could not be reported by the AV system; as well, such samples were not easily identified in a short period of time. For example, albumin>total protein, creatine kinase-MB>creatine kinase. In order to validate these rules or cases, we used simulation software in the middleware to validate whether the verification rules and process were well implemented. Our results demonstrated the rules and workflow designed could be correctly executed. Table 1 and Figure 3 show that the AV system designed in this study yielded an average passing rate of 95% for all test results, and a higher AV rate, as compared to previously published data, of 73%.13 This could be because different laboratories have different acceptable ranges for the limit check and the delta check for each test item. For example, the range of the delta check was set between 5%–200% in our laboratory, while others may adopt 20%–30% for their acceptable range.13 Furthermore, as shown in Figure 4, 2 test items, ethanol and human chorionic gonadotropin (hCG), were accepted via AV with a passing rate lower than 70% (27% and 68%, respectively). It was concluded that the legal cutoff of the alcohol concentration for illegal drunk driving in Taiwan was 10.8 mmol/L, and requests for blood alcohol tests always involved traffic issues. For this reason, the blood alcohol concentration was usually higher than the legal ethanol concentration cutoff. As the reports might be used for legal purposes, the removal of such test items and the application of manual checks to all ethanol-related reports was considered. A low AV passing rate of hCG was correlated with the analytic measurement range of the analyzer, which was 0.5–1000 IU/L for hCG. Pregnant women usually have an hCG level higher than AMR. Although the instrument might automatically dilute the samples and mark it with a dilution flag, the processing of the result still required manual intervention.
One of the most important functions in the AV system was to hold the samples with errors reports. Table 2 shows that of the 25,526 patient reports, 41 were classified as having remarkably abnormal data and requiring interception or rejection. Further investigation revealed that the abnormalities in the data were due to serious hemolysis during blood sampling (test items such as K, LD, AST, etc), ethylenediaminetetraacetic acid (EDTA) contamination (test items K and CA failed the limit check and the delta check), and severe lipemia or insufficient samples. Also, 6.9% of the MV data was rendered non-reportable due to an error flag marked by the instrument. The remaining 4562 (93.1%) reports were classified as acceptable MV data, and the required manual intervention was identified to be caused by changes during the medical procedure. These test results were reported after a brief communication with a clinician. These samples were retested, and the results were consistent with the previous results. When compiling the statistics according to the panel tests in the test requisition sheets, the AV passing rate was 81.5% at the current stage, which greatly alleviated the burden in report verification.
The stability of the performance of the automated analyzers now in use is relatively better. With the aid of the barcode system, the error rate in the laboratory is greatly reduced. The function of an AV system is not only to detect errors in the test results, but also, by its application, to improve patient clinical care. For example, results in the delta check falling outside the acceptable range could imply a significant change in the patient’s condition. In such cases, the laboratory would inform the physician so the patient can receive appropriate treatment.
Quality control (QC) results should confirm if an item falls within the acceptable range before the AV system starts; otherwise the verification procedure should cease. At present, the QC system is not connected with the AV system of the middleware; hence, manual intervention was adopted to control the mechanism, that determined whether the QC results were acceptable. For this reason, the QC system should be integrated with the AV system, then, when QC failure occurs, the AV of the given test item will automatically stop.
Most publications describe the AV system as a way to shorten TAT in reporting test results and thus reduce the labor burden in the laboratory.13 However, as different laboratories have different working configurations and workflow, it was estimated that the use of an AV system would have a full-time equivalent (FTE) of 3.5 full-time employees per year in our laboratory. The TAT of patient reporting was dramatically reduced because we released the AV test reports immediately instead of the previous batch release. However, our experience has suggested the greatest benefit from the system was the consistency of the test result verifications done by different medical technologists. All of the test results were AV based on the same standard and treated in the same manner, thus ensuring the quality of the reports. However, an AV system is also subject to limitations. Even though the check rules seem flawless, errors can still occur. An example might be the mislabeling of the patient sample. If the data have merely slight differences and are not intercepted by the delta check, incorrect reports might be issued. On the other hand, erroneous data might still be reported when the interferences in the sample, such as a partial clot, have not been detected by the instrument. Technical errors of this kind, although few, remain unavoidable. Another reason for using the system would be to allow medical technologists to spend more time and effort focusing on the handling of MV test reports and, thus, improve the quality of patient care.
The authors thank Dr. Jang Jih Lu for his encouragement and Beckman Coulter for its technical support.
. Laboratory reporting for the future: Linking autoverification to the electronic medical record. American Association for Clinical Chemistry. AACC Continuing Education. Available at: www.aacc.org/access/record/default.stm. Accessed October 2005.
. (Revisited univariate delta check method for hematologic laboratories [I]—Usefulness for detection of specimen mix-up in patients with hematologic disorders). (Japanese) Rinsho Byori. 2002;50:1072–1075.