Often when I recommend that a research team prepare a formal analysis plan the first response I hear is, "Why? The analysis is not due for weeks and I have too many other things to do."
An analysis plan is not extra work; it's work that makes all the other project tasks flow efficiently. It will help you produce on-time project deliverables. Typically, you develop an analysis plan in parallel with your research instrument (RI) or questionnaire. Like the questionnaire the analysis plan is tied back to the goals and objectives of the study. In addition to the obvious purpose of an analysis plan, producing a plan serves to improve the RI and manage project scope, these benefits alone will pay you back for the time you devote to creating it.
The RI is referred to in an Analysis Plan (AP) and while there are no hard or fast rules and no one right way to structure an AP we can offer some guidelines. The approach described briefly here is as good as any and better than most for quantitative studies.
The first step in this process will be familiar to those of you who have read AtHeath publications from the Market Research Resource Center (MRRC). Specifically, research has the greatest chance of success when the objectives are clearly stated and that is where we begin. Use these five (5) straightforward steps.
1. State the key study objectives clearly at the beginning of the analysis plan (AP) and refer to them through the process.
2. Describe the major comparisons for the analysis (eg, major cross tabulations for the study such as: Customers versus Non-customers, Companies by size, Customers that are Satisfied, Neutral, or Dissatisfied).
3. State how each question is used to answer a specific objective of the study either on its own or in combination with other data points. Think through how you expect to present the results from each question. What statistics, if any, will you use in the analysis? Identify the independent and dependent variables.
4. Write a clear justification for including the information from the question in the study and perform a section by section "So what" litmus test.
5. When the analysis plan is finished, go back and make sure each key study objective has been addressed.
These five steps are the basic approach to the AP template (see it is straightforward). The key is to focus on objectives and think critically about how to execute on the primary goal of the study.
Next, let's dive a little deeper inside the Analysis Plan. Two of the five steps of an Analysis Plan (ie, steps three and four) are reproduced for each section of the questionnaire. Combined, these two steps provide the question-by-question detail of your analysis plan. First, each section of the questionnaire is described in a brief outline format. Next, the analysis requirements are described for all questions in the section. Finally, a 'justification' is written for the questions in this specific section of the questionnaire are needed. This is the "So what" litmus test. The example below may help to demonstrate how steps three and four are implemented.
Example Analysis Plan Steps 3 and 4
Section Q of the Questionnaire:
Topic – Accessibility of information and mechanisms to access information (eg, data):
a. Website features and functions customer depend on and / or like best
b. Perceptions and preferences for push versus pull tactics for receiving information from host firm
Q10, Q11, and Q12 – All focus on the website features and functions customers and prospects value most and the vendors that do the best job of implementing these features and functions. Conduct a feature / function prioritization analysis – multiple response analysis. To optimize the data, recode open-ended questions (Q12) and conduct analysis to classify the best websites.
Q13-16 – For these questions explore the general frequency of website use and specifically responsive 'use websites to make purchases. These are primarily descriptive analyzes with comparison by the major cross tab groups already outlined. In addition, we are likely to use these data in a segmentation analysis, which we will describe later.
Q20-40 – Capture data on "touch" issues eg, pushing information to clients, how frequently and in what ways. Basic descriptive analysis [possible segmentation variable] and cross tabs with significance testing will be applied.
Justification – The first part of this section provides us with competitive information, but more importantly points us to specific implementations that are considered "best in class" by clients and prospects – a very powerful tool for prioritizing and implementing features on our current website and any redesign work we decide to undertake.
The second part gives us general frequency of website use and purchasing, which is nice to know info, but may not be as actionable as other data. It tells us the relative importance of the website across our customers (by type weaknesses) and if we build-in ecommerce functionality, how much it might be used. However, I doubt we would decide to provide or not provide ecommerce functionality based on the study results (optional information).
The touch information is highly actionable and can help guide our efforts and inform decisions on the level of investment to make in these activities. End Section Q.
As you can see from the example a thoughtful description of the analysis work and the value of the results, along with the justification provides a roadmap. Time well spent