Abstract
At some point in either our training or our practice, we will wonder about the outcomes of our own interventions and how they compare to the published data. One can easily argue that these results would be equally, if not more, relevant than those found by researching the literature. The answer, whether positive or negative, would certainly provide new insights to the practitioner and impact future patient care. If the decision were made to publish the results, it may even benefit others in the field. Such analyses are not beyond the scope of the everyday practitioner and can be performed with minimal time, money, and resources.
Some of the many questions common to clinical practice are, “How can I find outcome data on this new technique I am using?” “Can I predict which patients will do well/poorly?” “How do my results relate to the published results?” All of these questions are highly relevant to our practices, yet often we do not have the answers. From a distance, it might seem like a difficult endeavor to attempt to answer these questions on our own in the setting of a busy practice. Upon closer inspection, however, this is not the case. The analysis of one's own practice is a technique that can be used by any physician and stands to benefit his or her practice efficiency and to improve patient care.
The objective of this article is to present a simple method for the integration of clinical research into routine clinical practice—a method we refer to as “Prospective Analysis of Practice.” In relation to this series of tutorials, the more personal experience one has with analyzing his or her own practice, the more he or she will be able to rapidly assess and understand published clinical research and spot both pitfalls and values. Analyzing one's own practice is a fundamental step in becoming an expert in one's field.
Prospective Versus Retrospective
The process of analyzing one's practice can be done in two different ways—prospectively and retrospectively. Whereas a retrospective analysis might seem like a simpler method with a quicker return, this turns out not to be true. First, you must sort through the medical records and diagnostic codes to find the patients who fit your criteria. Not only does this take significant time, but it often leaves out appropriate patients who may be listed under other codes. It also introduces recall bias and “cherry-picking,” whether conscious or not. You may also discover inconsistencies in how you treat different patients that makes comparison difficult. Finally, no matter how good your documentation is, you will inevitably discover that your records are not as complete or as comprehensive as you need them to be.
What separates the retrospective and prospective methods is the degree of planning. Proper retrospective studies require planning; for example, a well-done case-control study is often retrospective but requires careful planning. However, in prospective studies, you know what you want to examine and what data you need to gather from your practice in advance. You can therefore plan your approach and record the appropriate data. The end result is a more complete analysis done in less time. The question now arises, “But where do I start?”
Steps for the Prospective Analysis of Practice
The first step in prospectively analyzing one's practice depends on his or her starting point, which is dependent on levels of experience and desire. Some of us have no previous experience with data collection and analysis but have a busy practice. Others have some experience but with haphazard retrospective analysis in an unstructured setting. Still others have reasonable experience with structured retrospective analyses and currently desire to start conducting a prospective study. At the peak are those who have been studying their practice and now desire to seek funding of their own to study clinical questions. Most everyday practitioners will fit into the first two categories, so this is where we will begin.
When beginning with a pile of charts, a long list of patients, and no experience, the first step is to decide on a topic of interest and a question that pertains to that topic (Table 1). Finding an appropriate question may be more difficult than it seems. Decide whether you have a novel question for which you do not have an answer, or whether you want to compare your data and outcomes to that in the literature. Broad questions can be broken down into multiple smaller, more specific questions. For those with a desire to research a particular topic, yet do not have a clear question, the next step is to brainstorm. This involves writing down all the thoughts, questions, “factoids,” and other information about the topic that come to mind during free association. Then rapidly browse some general references on the topic without taking notes, but add anything to the brainstorm that stimulates your interest. Next, construct an outline of only the items on the brainstorm—this is your unique, purified logic. You can then do a quick literature search for key articles (method, materials, and results pertaining to the item), studying each item on the outline. Those items without a clear key article are potential novel questions, whereas those with “answers” make an excellent basis for comparison of your personal results.
The next step is to do background work on your chosen subject. To ensure that you are gathering relevant data, examine the literature on this question in detail and speak with colleagues to get their input. Depending on the topic, this may take no more than an hour or two. Whatever the outcome at this point, you have already completed your own informal CME course.
After doing some background work, you can sketch out your current diagnostic or treatment algorithm for the area under study. Make a diagram of how you practice and what decisions you make relative to those questions. Ask yourself “why,” and critically evaluate for biases at each step of your algorithm. You may want to validate your algorithm by looking through the literature, noting that controversy or lack of information makes your question that much more important. You can then add alternative decisions and actions that could be made at each point. Feel free to change the way you do things during this process if something seems more rational. You may even want to compare your old and new methods of diagnosis and treatment.
At this point, you are ready to design your data collection instruments. Determine all of the relevant variables to create an initial encounter form for the chart. Design another form for subsequent visits that will allow you to gather data longitudinally. If there is a series of diagnostic tests, it may be helpful to create an order set. Although this task seems daunting, it is no more than a systematic way of doing a history and physical examination and then follow-up visits. Many physicians have already made their own general encounter forms, making this just a minor adjustment. In fact, this step may lead to increased efficiency in your patient encounters. After piloting these forms in your practice, you can change them as needed for maximum efficiency.
The next step is to develop a way to organize and track the data you are about to accumulate (Table 2). This is done best using a database on a common spreadsheet program such as Excel or Lotus. A database differs from a log. A log usually consists of a minimal number of data entries such as name, ICD-9 code, general diagnosis, and possible standout facts. It can be a quick, simple way to track patients and search for previous information that should be kept for all patients, but it is insufficient for prospective analysis. By contrast, a database is a comprehensive record that contains all pertinent information on a topic such as complications, audiogram results, medications, etc. Creation of a database is the key to good prospective analysis of practice. Depending on the level of complexity of your question, a database can be as large or small as you need it to be. In addition to data entry boxes (“cells”), you will want to set up formula boxes that calculate your current results instantaneously with each new entry. With your standardized encounter forms, it should take only a few minutes at the end of each clinic day to enter the information. Alternatively, if the database is to be large and complex, the practitioner may delegate the job of data entry to office staff or to a college or medical student looking for a few hours of work per week. Whoever does the data entry, it is important to perform occasional spot-checks for accuracy, typically about 10% of the data entries, and to document the results of these spot-checks.
Example Database
Example of database: Excerpt from a mock spreadsheet documenting tonsillectomy outcomes. Many more columns are possible, including days postop, body mass index, anticoagulation, etc. Variables can be entered as a numeric, such as postop bleeding and returns back to the operating room, and use for calculations, such as the proportions seen in the last row. The last row uses the “average” function to calculate the proportions with each new entry. The first four columns would qualify as a log and are useful for patient tracking OSA, obstructive sleep apnea; PTLD, post-transplant lymphoproliferative disorder.
During the process of data collection, it is crucial to analyze your results at predetermined intervals. Decide when you want to look at your results and then sit down and see what you have found. The time interval will depend on the nature of your question and the frequency with which you see relevant patients. Begin by simply looking at numbers of patients and frequencies with which expected results occur. This alone will provide you with meaningful information. Early trends may either be the results of low initial numbers or a real result and should prompt closer evaluation. You may discover early on that you are missing certain information and want to modify your data collection and tracking process. The duration of your analysis is up to you. An easy argument can be made to keep track of your results indefinitely. A calculation box may even become statistically significant at some point.
After collecting and analyzing your data, the final steps of the prospective analysis of one's practice are to apply and share the information. The degree to which you want to share it may vary considerably. You may want to discuss it with a colleague over lunch or share it with your partners at your next meeting. Depending on the quality of the data, you may even be able to formally write it up and submit it for publication. The gained information may be only pertinent to you, so you may just decide to keep it to yourself and modify your practice accordingly. At the least, you will now be able to tell your patients what percent of your tympanoplasties have an air-bone gap of less than 5 dB instead of quoting results from a 10-year-old study.
As one gets more comfortable with the prospective gathering of data and subsequent analysis, he or she may want to take on a greater research role. You may feel that in this new context, the current mass of information is outdated or inadequate and deserves another look. A good next step is to seek counsel with a clinical epidemiologist or statistician or a more experienced colleague and begin to design a small single-center prospective trial. A mentor at this point is very helpful. Some trials can be done rather well without funding. If more time or money is required to expand the trial, it might make sense to attempt to obtain funding by writing a grant. Just the process of writing a grant can be very educational and stimulate further thought on the topic of interest. Your initial clinical analysis may even serve as excellent pilot data.
Example
To understand this process, let us take the example of post-tonsillectomy bleeding. One could determine this retrospectively and look at the number of patients billed for tonsillectomy and for control of oropharyngeal bleeding. The flaws with this approach are many. Among other things, not all bleeding requires a trip to the operating room or someone else might have taken care of the bleeding. Another option is to look through a log of patients receiving tonsillectomies and then search through their charts to see how many had bleeding difficulties. However, if one assumes a 3% bleeding rate, this would mean for every three positive charts, there would be about 97 charts of nothing—a poor use of time. All of this is assuming the bleeding was documented in the chart.
By applying the method of prospective analysis, the question of “What percent of my tonsillectomies have bleeding” is defined at the start. A quick glance at the literature reveals that multiple factors have been implicated in post-tonsillectomy bleeding such as surgical technique and patient age. You can decide in advance what factors are important to you and possibly modify your question to compare techniques or even body mass index and preoperative indication for tonsillectomy. The spreadsheet can be used to determine calculated parameters such as body mass index (from height and weight), increasing simplicity and ensuring accuracy. You then design a template that contains the variables in which you are interested and integrate it into practice. You log all of your tonsillectomies and the pertinent data in a database at the end of each day (Table 2). After 6 months, you might observe that there is a 2% difference between techniques A and B, all occurring an average of 8 days out, with more patients for whom technique A was used requiring a trip back to the operating room. You can then alter your practice as you see fit, and publish your results if you wish.
Conclusion
Critical analysis of one's practice is a fundamental step in becoming an expert in the field. The most efficient and fruitful method is a prospective analysis. Every individual can and should tailor this process to his or her own style and practice. Every step affords the opportunity to scrutinize one's own methods and results. As with many things, there are different levels of involvement associated with different levels of return. It is up to the practitioner to decide how much he or she is willing or able to invest. In the end, there is no wrong answer, as there is a clear benefit at each level. At the very least, experience with using one's practice as a platform for the good practice of medicine and advancement of science enhances efficient and effective reading and understanding of the literature. At the most, you may discover something extraordinary and publish in a major journal. In any case, you will gain insight into your own practice and be able to make adjustments that benefit you, your partners, and, most important, your patients.