ºÚÁÏÍø

Production Efficiency of Physicians in Outpatient Consultation of Cardiovascular Cases

Tomoyuki Takura^1* Haruki Itoh²

¹Department of Health Economics and Industrial Policy, Osaka University Graduate School of Medicine, Osaka, Japan

²Sakakibara Heart Institute Hospital and Clinics, Tokyo, Japan

Corresponding Author:

Prof. Tomoyuki Takura, Ph.D
Department of Health Economics and Industrial Policy
Osaka University Graduate School of Medicine
2-2 Yamadaoka, Suita, Osaka, 565-0871, Japan
Tel: +81-6-6879-6550
Fax: +81-6-6879-6549
E-mail: takura@heip.med.osaka-u.ac.jp

Received Date: May 24, 2013; Accepted Date: June 17, 2013; Published Date: June 19, 2013

Citation: Takura T, Itoh H (2013) Production Efficiency of Physicians in Outpatient Consultation of Cardiovascular Cases. J Community Med Health Educ 3:218. doi:10.4172/2165-7904.1000218

Copyright: © 2013 Takura T, et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Visit for more related articles at Journal of Community Medicine & Health Education

Abstract

Background: We evaluated the production efficiency of physicians in outpatient consultation using Data Envelopment Analysis (DEA) to verify the applicability of this method in health care field.
Methods: Clinical accomplishment by outpatient consultation (number of complex cases) and economic contribution to health service income (fee claims) were defined as the contributions of physicians in medical institutions. We investigated consultations and fee claims of 963 outpatients (all aged: 72.9 ± 11.8 years, males aged: 72.0 ± 11.7 years / females aged: 74.0 ± 12.0 years, hypertension: 17.5% / angina: 13.1%) consulted by 15 physicians in a week at a medical institution specialized in cardiovascular disease. The physician’s fee claim per practice time was used as an index in production efficiency of each cardiovascular case. We determined production efficiency of physicians using both of total fee claim and number of complex cases per consultation time using DEA.
Results: There was no statistically significant difference between main disease of outpatients in consultation time (p=0.28). We showed a significant increment of physicians fee claim (point per min) according to the patient condition aggravated from mild to moderate cases in severity (71.4 ± 2.4: p=0.004). The production efficiency also significantly increased from the first consultation to follow-up (71.6 ± 2.4: p=0.048). There was a large variation among the 15 physicians in the consultations, and 4 physicians (26% of all subjects) formed a group with the most favorable production efficiency according to DEA.
Conclusion: We demonstrated evaluation of production efficiency of physicians using clinical and economic accomplishments. The production efficiency is significantly influenced by disease severity and consultation step. Our finding suggests that DEA may be applicable for evaluation of health care services.

Keywords

Cardiovascular diseases, Severity, Consultation step, Fee claim, Production efficiency, Data envelopment analysis

Introduction

Recent health care reform has included discussion in development of a sustainable and robust system to cope with diverse demands with limited resources [1-3]. Reinforcement of the health care system requires adjustment for management of medical institutions in response to internal and external environment. In particular, efficient and flexible use of resources (e.g. specialists and equipment) and maximization of clinical and economic outcomes are needed in medical institutions.

Evaluation of efficiency can be achieved by productivity analysis using resources input to accomplish a target (input variable) and the added value (output variable) used as indices [4-7]. In some fields except health care service [4,5] management of efficiency has been introduced to utilize resources effectively, but it was limited in health care field [6,7] probably due to the unique characteristics. In fact, s were expected to work for social role but not economic benefit, and evaluation of efficiency were more likely to be ignored particularly in Japan. However, this situation has slightly changed due to low economic growth and limited health rescores. Thus, management and improvement of efficiency are required in health care systems.

In outpatient consultation, which is one of the major roles of cardiovascular medical practice, evaluation and treatment of complex cases may be important in production efficiency of physicians, although there are diverse concepts of social role and significance of physicians in management of medical institutions. The clinical outputs through outpatient consultation include the planning of a treatment and facilitation of health recovery as treatment order for outpatients, and therefore, we should place a large value on outpatient consultation of complex cases.

In medical institutions, sound management of complexity in clinical practice is required to balance social demands and limited resources. Thus, Data Envelopment Analysis (DEA) [8-10] using clinical outputs as well as economic output may be appropriate to management social significance of health care field.

In briefly, few studies were conducted on measurement of production efficiency in viewpoint of outpatient physicians, and it was crucial to develop measurement of the production efficiency including and treatment of complex cases, which leads to evaluation of physicians’ contribution to social welfares. Therefore, we aimed to propose how to measure production efficiency of outpatient physician, and then performed basic DEA to determine the production efficiency of physicians in outpatient consultation, to confirm applicability of DEA in the health care field.

Materials and Methods

Study population

Details of a design of this study have been published previously [11]. Briefly, our study was performed in a physician’s team of outpatient department in a medical institution which specialized in cardiovascular diseases (Sakakibara Heart Institute Clinic), through a week (net period of 5 days) in October 2006. The physician’s team included 15 physicians (12 males and 3 females) who had 10 years or longer experience of clinical practice, and they worked once or more per week through the study period in outpatient department. We prospectively investigated a total of 963 outpatient consultations performed by these 15 physicians (certified specialist of cardiovascular medicine).

Informed consent was obtained to conduct our study based on the declaration of Helsinki [12]. The Board Committee of Sakakibara Heart Institute approved this study.

Definitions of production efficiency

We measured production efficiency in viewpoint of outpatient physician. To assess outcome in production efficiency of physicians in an outpatient medical practice, we initially defined two clinical outputs, one economic output and one input of physicians through outpatient consultation as follows.

For clinical outputs, we initially established two definitions in complexity of each cardiovascular case in outpatient consultation to measure importance of clinical outputs in management of medical institutions. The clinical outputs through outpatient consultation includes the planning of a treatment and facilitation of health recovery as treatment order for outpatients, and outpatient consultation in more complex cases should be evaluated as having output more. Therefore, we defined severity (4 grades: mild to emergency cases) and consultation step (3 steps: first consultation, re-consultation and follow-up) of each cardiovascular case as measurement of complexity (Table 1), and collected these information in an outpatient department. We defined complex cases as outpatient consultation in severe to emergency case or first consultation. The severity based on our study protocol was diagnosed by a cardiovascular physician, and after that the classification was reviewed by review committee including 2 cardiovascular physicians and more.

Table 1: Definitions of severity and consultation step.

We measured “total fee claim” (i.e. health service income) as economic output, which included “physician’s fee claim” for technical part of treatment physicians performed (e.g. consultation, instruction, judgment and testing) and “other fee claim” for materials indirectly involved in treatment by physicians (e.g. costs of drugs and materials). Because all of fee claims (i.e. physician’s fee claim and other fee claims) are paid under direction of physicians in health system of Japan, we defined total fee claim (i.e. physician’s fee claim and other fee claims) as economic output, which includes direct and indirect contribution of physicians to the medical institute. Since emergency cases were transited to inpatient treatment, their fee claim was not counted as outpatient treatment.

We also directly measured “consultation time” for outpatient consultation in minutes, which included “practice time” as time directly involving patients (e.g. for inquiry and judgment) and “workup time” as time spent giving other part of medical care (e.g. directions and filling in slips). Because examination and intravenous injection were conducted by paramedical staffs under direction of physicians, the time was not included in practice tame and work-up time. Medical assistants measured practice time and work-up time using a stopwatch (time study method).

In traditional health economic researches, one of the outputs was “income”, and the input was “hours worked” [13]. But, as measurement using only total fee claims could not evaluate contribution to social welfare, we also measured number of complex cases as clinical output. We did not use labor costs of physicians (i.e. hours worked multiplied by payment by the hour), because the payment by the hour may be biased by the labor environment and positions.

Statistical analysis

We conducted two step analyses, to evaluate production efficiency using clinical and economic outputs for each cardiovascular case and that for each physician.

First, to evaluate production efficiency for each cardiovascular case, we examine associations between clinical outputs (i.e. complexity) and economic output per input for each cardiovascular case. Because “physician’s fee claim”, which is associated with “practice time”, directly adds value in each cardiovascular case, we used “physician’s fee claim” as economic output and “practice time” as input. We analyzed the associations using a Wilcoxon rank -sum test and a Spearman rank correlation coefficient or a Kruskal-Wallis test. We also conducted a generalized linear regression model using severity, consultation step and practice time as covariates, to confirm these confounding effects. Because the distribution of physician’s fee claim per practice time was skewed, the logarithmically transformed value was used in this multivariable analysis.

Second, to evaluate production efficiency for each physician, we examine associations between clinical outputs per input and economic output per input for each physician. Because “total fee claim” adds value during consultation time of each physician, we used “total fee claim” as economic output and “consultation time” as input. To evaluate the production efficiency of each physician, we conducted Data Envelopment Analysis (DEA) [8] using logarithmically transformed values in number of complex cases per consultation time (i.e. clinical output per input) and total fee claim per consultation time (i.e. economic output per input), and the efficient frontier line for physicians (i.e. the most favorable condition for the production efficiency) was identified [14]. DEA has been proposed for relative and scientific evaluation of efficiency to apply productivity analysis for decision making in management. DEA can be used for investigation of resources and achievement of many items with different characteristics, and clearly presents an optimum solution (i.e. efficient frontier of envelopment) formed by functional groups assumed to be efficient (i.e. specific physicians in this study).

For missing data analysis, we applied complete-case analysis for evaluation of DEA (second step), and available-case analysis [15] for other analyses (first step). Before starting this study, sample size was estimated according to the monthly survey of fee claims and practice time in the hospital. Specifically, physician’s fee claim per practice time for first consultation group and follow-up group were assumed to be 60 ± 15 and 70 ± 15 with normal distribution, respectively. Ratio of sample size between first consultation group and follow-up group were estimated to be 1/10. Alpha and power were set at 0.05 and 0.90, respectively. As a result, total sample size was estimated to be 234.

All statistical tests were two-sided and a p-value <0.05 was regarded as statistically significant. SAS, version 9.13 (SAS Institute, Inc., Cary, NC, USA) was used for all statistical analyses.

Results

Distribution in number of outpatient consultations among 15 physicians

Table 2 showed the number of outpatient consultations per week among 15 physicians. Mean number (± standard error) of outpatient consultations was 57.0 ± 13.4 cases per week, ranging from 7 to 164 cases per week. The pattern of their work varied from 1 work day per week as part-time to full-time (5 work days per week). The 963 outpatients aged 72.9 ± 11.8 years included 592 males aged 72.0 ± 11.7 years and 371 females aged 74.0 ± 12.0 years. 17.5% of the outpatients were diagnosed as hypertension, and angina (13.1%) was followed it (Table 3). Because all of cases were initially diagnosed and treated by only one cardiovascular physician in outpatient department, there were no cases doubly-counted in the 15 physicians (Table 2).

Table 2: Number of outpatient consultations among 15 physicians.

Table 3: Diagnosis of 963 outpatients.

We examined the difference of consultation time of cardiovascular physicians according to age, gender and main disease of outpatients, and confirmed no statistically significant differences (Table 4). The R-square between age and consultation time was -0.04 (p=0.23), and the consultation time was 12.0 ± 7.2 minutes for male and 12.1 ± 6.7 minutes for female (p=0.35). The respective consultation time was 12.4 minutes for hypertension, 10.3 minutes for angina, 12.4 minutes for atrial flutter, 12.3 minutes for old myocardial infarction and 11.4 minutes for mitral regurgitation (p=0.28).

Table 4: Mean values ± standard deviation (SD) of consultation time of physicians according to characteristics of outpatients.

Physicians’ fee claim accounted for 29.2% of total fee claim, which included basic physicians’ fee for 4.5% and special physicians’ fee for 24.7%. The basic physicians’ fee included first visit fee and re-visit fee, and the special physicians’ fee included fee for health education, fee for examination (mainly biological examination and diagnosis), fee for diagnostic imaging (mainly for diagnosis), fee for medication (mainly for prescription) and fee for treatment (as a part of general treatment). Other fee claim accounted 70.8% total fee claim.

Analysis of factors influencing production efficiency

Figures 1 and 2 showed associations of severity and consultation step (i.e. clinical outputs) with physician’s fee claim per practice time (i.e. economic output per input), which was used as an index of economic production efficiency.

Figure 1: Physician’s fee claim per practice time according to severity of outpatient cases.

Figure 2: Physician’s fee claim per practice time (points/min) according to consultation step of outpatient cases.

The economic production efficiency of cardiovascular cases significantly increased when the patients’ condition aggravated from mild cases to moderate cases (59.0 ± 3.5 versus 71.4 ± 2.4: p=0.004) (Figure 1), but the production efficiency was significantly lower in severe cases than in moderate cases (50.0 ± 5.8 versus 71.4 ± 2.4: p<0.001). This suggests that outpatient consultation of severe and emergency cases reduces the production efficiency of physicians with regard to economic contribution to health service income. These results show that use of the economic contribution to health service income alone leads to low production efficiency of physicians who are treating complex cases.

We also evaluated an association of consultation step with economic production efficiency of cardiovascular cases (Figure 2). There was no significant difference between the first consultation and re-consultation (59.2 ± 5.8 versus 55.1 ± 3.6: p=0.194507), but the production efficiency significantly increased in follow-up in comparison to the first consultation (71.6 ± 2.8 versus 59.2 ± 5.8: p=0.048) and reconsultation (71.6 ± 2.8 versus 55.1 ± 3.6: p<0.001), suggesting that follow-up consultation markedly increases the production efficiency of physicians with regard to the economic contribution to health service income. These results show that consideration of the economic contribution to health service income alone leads to low production efficiency of physicians who are treating unclear conditions.

These results did not substantially altered, when we used generalized linear regression model or complete-case analysis (not shown in figures).

Analysis of production efficiency of physicians using DEA

The above findings show that an index of clinical accomplishment in complex cases is necessary for a proper evaluation of the contribution of physicians in medical institutions, in addition to indices of economic contribution to health service income. Therefore, we performed DEA using two outcomes, including clinical and economic outputs as follows. First, we selected the number of severe and emergency cases or first consultation as complex cases as clinical output through outpatient consultation, because these economic output was relatively low. Second, we selected the total fee claim as economic output.

The production efficiency of physicians was calculated by DEA using these clinical and economic outputs and consultation time as function of efficiency containing clinical and economic outputs (2 indices) per input (1 index). The results visually suggested that 4 physicians (D1, D6, D8 and D9) formed a group with the most favorable production efficiency (Figure 3). The production efficiency of each physician was plotted with the logarithm of the total fee claim per consultation time (points per min) on the vertical axis and the logarithm of the number of complex cases per consultation time (cases per minutes) on the horizontal axis. The 4 physicians (26.6% of all subjects) were distributed in the upper right region, representing the most favorable performance. From the production efficiency distribution of these 4 physicians, the efficient frontier line for the physicians in outpatient treatment (the most favorable condition for the production efficiency) was determined by drawing an asymptotic line, and the equation shown in appendix was obtained. This equation quantifies how much physicians other than the 4 with a favorable production efficiency need to improve the total fee claim, number of complex cases, and consultation time to maximize their production efficiency.

Figure 3: Production efficiency of physicians in outpatient consultation using Data Envelopment Analysis (DEA) including clinical and economic outputs and input. Complex cases included outpatients of severe or emergency cases or first consultation. This analysis visually shows the most favorable condition for production efficiency of the physicians in outpatient consultation (production efficiency function). The figure was determined by drawing an asymptotic line from the plots of production efficiency of 4 physicians with the highest performance (D1, D6, D8, and D9). The equation enables a quantitative evaluation of how physicians other than the 4 physicians can improve their fee claims, numbers of complex cases, and consultation time (i.e., indices of the production efficiency) to maximize their performance.

Discussion

Newhouse identified three important issues in measurement of efficiency in health care service as follows: 1) measurement of the outcome, 2) maintenance of homogeneity in measurement of health care resources, and 3) control of the characteristics of patients who received health care [14].

We showed a bell-shaped association of severity and negative association of consultation step with economic production efficiency (i.e. physician’s fee claim per practice time) as health care service outcomes among physicians specialized in cardiovascular internal medicine. This finding suggests that complexity of outpatient consultation may be independent from increased economic production efficiency, and which leads to the importance of complexity as measurement of the outcome [14] for production efficiency of physicians. Therefore, we developed and investigated the method for measurement of production efficiency of physicians, taking complexity of outpatient consultation into account, with DEA including clinical output as well as economic output (i.e. total fee claim).

Positioning of the study

Since few studies were conducted using DEA which focusing on health care resources (e.g. physicians) [6], we performed basic DEA with diagram analysis of efficiency (i.e. a Charnes, Coopers and Rhodes model: CCR model) 8 in which production efficiency of physicians can be visualized as a pilot study. Our result suggests that DEA is applicable for evaluation of the production efficiency in management of medical institutions.

For efficient management of medical institutions, roles should be assigned based on the characteristics of each health care resource (e.g. physician), for which it is necessary to evaluate costs based on the overall resources of the institution (including team health care). In other words, we aimed, not only to examine the variation in production efficiency of physicians in our model, but also to examine overall optimization of management of medical institutions. This requires evaluation of how physicians share functions, complement each other, and have synergistic effects. Therefore, we suggest that the study results should be interpreted as follows.

First, DEA shows that specific, rather than superior; subjects were present among those judged as efficient. For example, D6 was included in specific subjects (i.e. a physician on efficient frontier line), but the total fee claim for D6 was slightly lower than those for the other physicians. Since D6 was an only physician who used outside pharmacy instead of in-clinic pharmacy, his total fee claim became low due to loss of fee claim for the external prescriptions in institutional context. Although our study was conducted in one institute specialized in cardiovascular disease, which may lead subjects to relatively homogeneous, we need to assess the production efficiency of physicians with deviation of each physician from efficient frontier line, carefully considering roles as well as characteristics and environment of each physician.

Second, after assigning roles, it is important to examine how improvement of the production efficiency toward the efficient frontier line can be achieved. In actual clinical practice, outsourcing and alternative treatment by allied health professionals are used in many situations to increase final outputs and profit performance. The structure of this health care service system depends on the facility characteristics, such as private practitioners and doctors working in hospitals. To evaluate production efficiency of physicians in general, regardless of working conditions, it is necessary to use intermediate output, such as creation of opportunities to acquire profits, as an index or to establish conditions in which physicians complete all their intrinsic practices by themselves (to exclude the influence of alternative work). Thus, we used the “physician’s fee claim”, corresponding to the other fee claim, as an economic output for contribution to health care service outcome in the analytical process. While further studies are needed including physician profiling [16-20] to improve inefficient functions, our approach may contribute to more effective utilization of scarce resources, improvement of the final output of physicians and team health care, and decisions on outsourcing and work with allied health professionals (e.g. choices in management of medical institutions).

Furthermore, it should be noted that these indices represent “intermediate output”, but not “final output” (final value added) in use of these outcomes as a value added in management of medical institutions. Since the cost is not considered in the fee claim, and the fee claim is not an index of profit performance (i.e. profit = income – cost), increasing treatment orders with a low earning rate may increase the deficit balance of the medical institution. Also, the number of consultations does not cover clinical endpoints such as health status and improvement of survival, as all of added values by the consultation. Further studies on medical care cost analysis and utility analysis (e.g. quality adjusted life year: QALY) are required to address these issues.

Interpretation of the findings

The deviations of the production efficiency of physicians and identification of those with the most favorable production efficiency should be addressed for purpose of overall improvement on product efficiency of the medical institute, and the related background factors should be shared within the organization by role assignment. Thus, to perform a “final output”-oriented study, it is important to investigate whether fee claim are paid for overall health care resources in the medical institution, in addition to evaluation of individual treatments in the medical service fee system.

Our result showed a large variation in production efficiency of each physician, which allows us to find efficient frontier line visually, although the samples size was relatively small. In particular, the results showed that the contribution of physicians cannot be evaluated, based only on the number of fee claims (i.e. number of patients they treated), because some of the physicians who treated a lot of patients in Table 2 (i.e. D2 and D3) had low product efficiency in Figure 3.

We used the number of complex cases (severe and emergency cases or first consultation) as clinical outputs and total fee claim as economic output for production efficiency for each physician. Our finding suggested that complexity of outpatient consultation may be independent from increased economic production efficiency, and which leads to the less evaluation of physicians who treated complex cases in fee claims. Thus, these results indicate an importance using complexity of cases as well as economic contribution for production efficiency of physicians [21].

Future direction

We evaluated an association of severity with economic production efficiency in cardiovascular cases calculated by physician’s fee claim and practice time. The production efficiency significantly increased as the patient condition aggravated from mild and moderate cases in severity probably due to orders of expensive biological tests and prescriptions increased, while the practice time was slightly prolonged. Moreover, the production efficiency significantly decreased at moderate cases compared to severe cases due to prolonged practice time, while the median value of physician’s fee claim was similar to that for moderate cases. So, we showed large standard deviation of the physician’s fee claim for severe case, suggesting that specific treatment for severe cases may increase or decrease physician’s fee claim. Moreover, socioeconomic background [22] and complains of patients may affect to the production efficiency. Therefore, we need further studies dividing by characteristics of patients, disease and treatment process for control of the characteristics of patients who received health care [14].

We also need to pay attention to characteristics (e.g. experience and specialty) and environment (e.g. working condition, support and management system) of each physician, which may affect to their production efficiency. In fact, a previous study showed a relationship of their experience and specialty with the consultation time [23]. Moreover, improvement of health care information system or secretariat’s support may enhance the performance of specific physicians. Therefore, evaluation of the production efficiency in physicians requires further studies on their characteristics and environment, in addition to clinical outcome.

Furthermore, other fee claim accounted for 70.8% of total fee claims, which suggests the importance of paramedical staffs and materials in economic evaluation of outpatient department, although physicians contribute to both of physicians’ and other fee claims. When we calculate the profit of medical institute, the cost related to paramedical staffs and materials may be large, which leads to bias in measurement of productivity. To be concrete, it is important how to calculate the input, which leads to management of medical institute, including how to conduct the team medicine and how to buy the medical materials in future study.

A limitation of the current study is that the current study was positioned as a pilot study and was conducted in only one department including 15 physicians through 1 week, which lead to little variation of characteristics and environment of physicians and low generalizability. Second, we used “intermediate output”, but not “final output” (final value added) for evaluation of outpatient physicians, which may lead to surrogate bias. Thus, further studies are required that include these issues in some institutes.

Conclusion

We quantitatively determined the importance of complexity of outpatient consultation including disease severity and consultation step as clinical output for evaluation on production efficiency of physicians, which leads to evaluation of contribution to social welfare of outpatient physicians. We also confirmed that DEA was applicable for the evaluation of physicians. Although further studies are required to evaluate clinical output of physicians’ contributions, the application of DEA using clinical outputs allows us to manage social significance of physicians which is not evaluated by fee claims.

Acknowledgements

We are grateful to the staff of Sakakibara Heart Institute Clinic for their understanding of the significance of the study and for their cooperation during their busy schedules. We have greatly benefited from the advice of Noda Hiroki, M.D.

Source of funding

This study was funded by a 2006 FY grant-in-aid from the Sakakibara Heart Institute.

Appendix

Equation for the efficient frontier line

[Equation determining the most favorable conditions for production efficiency of physicians in outpatient consultation]

max_Y = A•exp (B•max_X)

A= 3,991

B= -0.096

• max-Y represents the group of physicians with the highest production efficiency: “total fee claim per consultation time”

(y) = total fee claim / consultation time

• max-X represents the group of physicians with the highest production efficiency: “number of complex cases per consultation time”

(x) = Number of consultations of complex cases / consultation time

(In this study, complex cases were defined as severe or emergency cases or first consultation.)

• The productive efficiency function (P) involved in m inputs (I) and s outputs (O) of n physicians (entity) is as follows: physicians J (=1,2,3,...,n)

(In this study, n=15, m=1, and s=2. Ur and Vi are weights (coefficients) to maximize the production efficiency.)

References

1. Weinstein MC, Stason WB (1977)
2. Roberts M, Hsiao W, Berman P, Michael Reich M (2008) Getting Health Reform Right: A Guide to Improving Performance and Equity. USA. Oxford University Press, Second edition.
3. Segall M (2003)
4. Yamada N (1998) Productivity analysis of public hospitals by DEA: Main causes of changes in productivity and optimum production scale. Presentation at the 5th Health Research Forum, Pfizer Health Research Foundation, Tokyo
5. Ozcan YA, Cotter JJ (1994)
6. Andes S, Metzger LM, Kralewski J, Gans D (2002)
7. Santos JR, May L, El Haimar A (2012)
8. Charnes A, Cooper W, Rhodes E. (1978) Measuring the efficiency of decision-making units. European Journal of Operational Research 2: 429-444.
9. Gautam S, Hicks L, Johnson T, Mishra B (2013)
10. Bouland DL, Fink E, Dehaan C, Fontanesi J (2012)
11. Takura T, Itoh H, Hamamoto H, Abe M, Kawase M, Horikawa Y, et al. (2011) Influence that characteristics of circulatory disease gives efficiency to outpatient consultation. The Journal of the Japan Medical Association 140: 98-103.
12. Declaration of Helsinki (2008) The World Medical Association (WMA) 22: 1-5.
13. Ding A, Saini S, Berndt ER (2009)
14. Newhouse JP (1994)
15. Little RJA, Rubin DB (2002) Statistical Analysis with Missing Data, A Wiley&Sons, Inc Second Edition. New Jersey.
16. Lasker RD, Shapiro DW, Tucker AM (1992)
17. Welch HG, Miller ME, Welch WP (1994)
18. Balas EA, Boren SA, Brown GD, Ewigman BG, Mitchell JA, et al. (1996)
19. Ahwah I, Karpiel M (1997)
20. Massanari RM (1994)
21. Greene RA, Beckman H, Chamberlain J, Partridge G, Miller M, et al. (2004)
22. Fiscella K, Franks P (2001)
23. Endo H (2001) Evaluation of internal medical care techniques: Applicability of RBRVS. Iryokeizaikenkyu 9: 7-15.