Design Diagrams – PhD Dissertation Research Seminar Track 2 (2024)

Design Diagrams

A research design should be able to be diagrammed graphically. This handout will provide basic definitions and descriptions of constructing a research design diagram. For more details, consult William M.K Trochim's Research Methods Knowledge Base. The design diagrams here are adapted from Dr. Trochim's presentation.

Example 1:

Note the following:

• The box on the left describes the research design in words. It says that there will be an experiment involving two groups (an experimental and a control group), and the dependent variable will be measured before and after the treatment.
• The box on the right describes the two groups, the experimental group on top, the control on bottom. Time moves from left to right.
• “R” means “random assignment to groups.” This is the first step and comes first in time.
• “O” is the pretest. Note that both pretests are at the same point in time. This means they are administered at the same time.
• “X” is the intervention—the independent variable—which the experimental group receives but the control group does not.
• “O” is the posttest. Again, it is administered at the same time to both groups.

Sometimes, there are subscript numbers.

Example 2:

In this example, the pretest (“O₁”) is given again as a posttest, but a second measure (“O₂”) is also given as a posttest.

Example 3:

Here, notice the "N" replacing the "R." This means that random assignment to the groups will not be done. Instead, "non-equivalent groups" ("N") are used, usually the result of convenience sampling. There is still the experimental structure—pretest, exposure to the treatment, posttest, but because the groups are not randomly assigned, it is a quasi-experiment.

Example 4:

Here, exactly the same design, except that now the groups are assigned on the basis of some cutoff score ("C") from a measure administered before groups are formed ("O1"). That is, the pool of participants are given some screening measure and their scores determine in which group they are placed. For example, all the lowest scores on the screening tool might be placed in one group, and the highest (based on the cutoff) in the other group.

Example 5:

In each example, notice the words in the left-hand column. There are basically three main lines (additional indented lines are simply continuations of the previous line):

• Line 1: Measures (Pretest-postest, pretest-two posttests, multiple measures, and so on).
• Line 2: Sample (Groups) (Two-group, One-group, Three-group, and so on).
• Line 3: Design type (Experimental, quasi-experimental, and so on).

The examples can be multiplied, but the basic ideas have been presented—for quantitative design diagrams. How would qualitative diagrams look?

Example 6:

Note the similarities and the differences:

• The data collection method remains the first line (“Three-interview”).
• Groups are replaced by the type of sampling (usually “Purposive” or “convenience”). It could be more detailed ( “Purposive stratified”).
• The design is given in the third line.
• Now, under the “Time” line, instead of groups there are “participants,” and the subscripts indicate which one. (“P1,” “P2,” “P3…P10,”). In this case, the diagram indicates that there will be ten participants and that all ten will receive three interviews, with the third coming after a slightly longer interval than the interval between the first two.
• The line-up of the “O” observations suggests that all three waves of interviews will take place at the same time. This immediately suggests a problem in the plan! We’ll talk about that in a moment.
• Although it would be reasonable to have simply a single line:

  P3…10

  O

  O

  O

• But that would signify that all the interviews would be done at exactly the same point in time, which is not usually the case. If individual participants must be interviewed (or otherwise observed) at different times, the diagram should reflect that, as in Example 7.

Example 7:

Here the full interview schedule is laid out graphically. One of the values of design diagrams—in both methodologies—is that they often reveal a design flaw that a verbal description would not capture. In this case, look at the timing of the some of the second and third interviews. They overlap with other interviews (meaning, they occur at the same time). Obviously, that will be impossible for a single researcher, so the diagram could be altered, reflecting the reality.

Example 8:

In qualitative research, second and third interviews are not unheard of. Their purpose is almost always to interview participants about the results of the first interviews to deepen their views. So completing the first wave of interviews before starting the next wave would be essential to the methods, and the design diagram in examples 6 and 7 alerted the researcher to that potential error that could invalidate the study or at least make it quite difficult to carry out.

Let's look at a quantitative diagram that captures a potential design flaw.

Example 9:

Assume that the research question is: Does exposure to condition X increase scores on dependent variable Y in participant sample Z?

Here is the first design diagram:

Suppose Sample Z is children who will be pre-tested on their reading skills in September at the start of second grade, workers who will be pre-tested on particular skills, or clients who will be pre-tested on some measure of functioning. One classroom will receive the special program from October through December, the other will not. (One group of workers will receive a special training, another will not; one group of clients a special intervention, the other group not.) Both groups will be post-tested in May, at the end of the school year (or nine months after the beginning).

Do you see the flaw? Consider what sorts of things naturally happen for a seven or eight year old child over the course of six months (or for workers or clients over a span of time). For example, young children mature significantly over the course of a year, and that might account for changes in their reading ability. Workers on the job may develop skills naturally by using them. Clients might improve their functioning due to other factors. We’ll continue with the example of the children, but the point applies to many similar kinds of studies.

How might this maturation extraneous variable be countered?

Note: The four months between September and December could also affect the maturation of the children, so perhaps adding comprehension measures mid-way through the program would help.

Example 10a:

But perhaps some of the kids matured faster during the first months of the school year. Perhaps adding more measures along the way, during the training period, could eliminate the problem.

Example 10b:

Here the maturation effect could be controlled for more tightly, and that flaw perhaps be eliminated. But of course, there might be other flaws. The diagram indicates another potential threat to validity that could be eliminated. Do you see it?

Example 10c:

You got it! The children (or workers or clients) might have remembered the original test, which was going to be used four times. So changing the test itself a couple of times could prevent that from distorting the actual impact of the training program.

The last few examples illustrate the usefulness of the design diagram in detecting potential validity threats. In qualitative as well as quantitative research, validity is all-important, and the whole point of careful design before conducting the study is to eliminate or reduce any threats to the study's ultimate validity.

Doc. reference: phd_t2_u07s1_h06_diagram.html