Text Transcript with Description of Visuals
| Audio | Video |
|---|---|
| [ No audio ] | A group video call starts and various participants join over the next 11 minutes. No one speaks until 11 minutes in. |
| >> Morning, Alexis. >> Hi Nancy. How's it going? >> Good. How are you? [Laughter] >> Waiting for the rain to stop. >> Is it still raining up there? Yeah, we had a clear day yesterday, but it's rained again overnight. >> Yeah. >> Yeah. Berry farmers are not happy. >> Yeah. I don't think anyone's too happy with how wet it is. >> You can send the rain to us. We'll gladly take it. >> Oh yeah? Right now, you need to take some. >> Yeah. >> We still have another two or three inches of it forecasted for the next week. >> Yeah, well, we'll take them. We're still way behind. >> I'm sorry. All right. So, it looks like it's 7:00 on the West Coast of the U.S, and nice to see all the names on the screen. So, welcome, everybody, to our second -- so informal workshop. Again, this is intended to mostly assist folks on the Onion "Stop the Rot" Bacterial Project, which is funded by the USDA Specialty Crop Research Initiative. But we did open it up to folks who just want to attend to hear a little bit of the discussion around experimental design. And I think one of the reasons that I did this is in some of the discussions we've had in our Onion team meetings is just thinking about aspects of field trials in particular, but other research trials as well that can end up confounding what you're trying to measure and end up detracting from your ability to find significant differences in the treatments you're evaluating. And sometimes you could have done something if you had anticipated ahead of time. Other times it's just sort of awareness of these confounding factors. So, I want to first of all just mention that I'm not a statistician. So, if any of you start to ask complicated statistics questions, I will very quickly run out of ability to answer those. Really, this presentation is derived from a class that I teach in field pathology and just recognizing some really important fundamental aspects of experimental design that are glossed over far too briefly in statistics courses. And I took about five or six statistics -- or maybe three or four statistics courses as a graduate student, and I, you know, got A's in all those courses and yet I still made some major blunders when I ended up carrying out my own research. And thinking back, you know, at some point we did cover these, but it was glossed over so quickly and it was so easy to forget the really fundamental aspects of these principles and how they can impact your ability to get results that you can use for publications and for determining effects of, you know, trying to help growers. So, I'm going to show some slides now. Thank you for -- to Heather for attaching some of these to the calendar. Let me see if I can pull up -- okay. Let me know if you all can see this slide. I'm going to also close my blinds because the sun shines in and I look like a ghost. | Recording of a Zoom call shows various participants on screen as they speak. |
| Can you all see the screen? >> Yeah, we can see it. >> Great. All right. Thank you. So, again, as we go through today, if anyone has questions, just, you know, speak up or raise your hand. Heather will kindly monitor the chat box or the hand raised and so on. This is meant to be interactive. And yeah, hopefully -- hopefully you'll find it of value to you. So, we're going to talk about some basic principles. But before we do that, I'm going to start with a project of my own. Let me -- oops. Let me get the screen down. | She opens a presentation. Text reads, Stop the rot. Onion bacterial project. Experimental design workshop. Lindsey du Toit. Washington State University. Dutoit at W S U dot E D U. 8th of June, 2022. U S D A, N I F A, S C R I, onion bacterial project number 2 0 1 9, 5 1 1 8 1, 3 0 0 1 3. A picture of an onion field. |
| So, I'm going to start with a project that I worked on during my PhD. I studied the disease common smut of maize, sweet corn in particular. And part of my PhD was looking at how do you develop an inoculation method that will consistently give you a good level of disease pressure in order to be able to run experiments like screening for disease resistance in germplasm collections. And so, this was a three-year PhD and we did this one particular experiment over three years. And I -- Heather attached the paper, the publication from 1999 to the -- hopefully you all saw it was in the calendar reservation. | A new slide with a picture of maize with blue grey corn. Text reads, case study of errors despite appropriate experimental design. |
| But I just want to go through a very serious mistake I made and it was a very valuable learning experience, and it wasn't because I wasn't using the correct experimental design. So, one of the things we were testing in this trial and it's on page 728 in the methods of this looking at how the concentration of inoculum of the Ustilago maydis, the fungus that causes common smut, influenced the amount of common smut that we could evaluate. So, both incidence and severity of the galling on the ears and the background picture here shows the common smut from this from some of the plots that we harvested. So, in this particular field trial, we -- the experimental design I used was a split-plot, randomized split-plot design. So, pretty standard design for field trials. And there were four applications, and I did this over three years, 1995, '96, and '97. And in this case, hybrids are assigned to main plots because of the size of the tractor used to plant the varieties. And then the concentrations of inoculum, which range from a thousand to a million spores per mil were applied to split plots. And these inoculations were done by hand. So, I'm going to show you now the results for each of the three years of this experiment. | An excerpt from the case study. Page 728. |
| So, on the left hand, the three figures on the left-and column are from 1995, '96, and '97, the incidence of ears with galls and the right-hand side is severity. So, let's just focus on incidence because we don't need to get too much in the minutiae. But what you'll notice is in 1995, we had some really nice data. On the x-axis is the concentration of inoculum; the y-axis is percentage of ears with galls. And you can see a really nice regression relationship for the eight hybrids that we evaluated in 1995. The one hybrid punch line differentiated a bit from the others, but it's a very nice sort of linear curve, increase in amount of galling as you increase the concentration of inoculum. So, what happened in 1996? | A selection of 6 line graphs appear. |
| Very similar set of cultivars evaluated, same treatment, same experimental design. I didn't change the experimental design, but the results were a complete mess, total inability to figure out what happened. Now, fortunately, even though I used the correct experimental design -- and the one good thing I did was keep really good notes -- is when I was assigning people to do the inoculations, these were big trials. So, I went ahead in each year, 1995 and 1996, the people who are carrying out the inoculations, we use these what's called hog vaccinators and you put on pollination aprons, put the vial of inoculum in your apron and you have these hog vaccinations that you'd fill up and then inject aliquots into the silk channel on each ear. So, it's a fairly labor intensive process and each person was given a concentration of inoculum. So, for example, I might have assigned Chuck Rudolf, he was working with me to the 10 to the 3 or maybe Teresa the 10 to the 4, Heather the 10 to the 5, and Claudia the 10 to the 6 and say go out and inoculate these plots because these are your concentration inoculum. And that was most efficient because people could go out and not have to come back and keep changing inoculum. But that meant I was confounding any differences among the people inoculating with the effect of inoculum concentration because each person had one concentration of inoculum. So, I was no longer blocking inoculum concentration and this is what happened in 1996. So, in 1997, I made sure that each person inoculated an entire block. Okay, so Claudia in this case would get all four concentrations of inoculum and have to inoculate all of one of the reps. Chuck Rudolf would have to do the next rep, Heather the next rep and so on. So, I was no longer confounding people with inoculum concentration. So, if anyone has any questions about this confounding thing, let me know. We'll go through it in quite a bit of detail. But I used the correct experimental design all three years, and what I wasn't aware of in 1995 was the degree to which people's methods, techniques of applying that inoculum into the silk channels and the ears influenced just how well that inoculation took. And so, I assumed everyone would do this pretty well because it wasn't that complicated. You pull out the [inaudible] of inoculum, inject it into the ears. But if you were sloppy because you had a lot of ears to inoculate and you didn't get it right down the center of that silk channel, you didn't get good take. And so, you can see in 1997 when I removed that confounding effect, I got really nice data again. | The 1996 graph's lines have no discernable pattern. |
| And it actually took advantage of this opportunity, which is in this publication in 1999, to test the influence of the experience of the people inoculating on the ability to get decent data. So, in 1997, I ran an experiment where I divided people into the E category. In other words, have had experience for one or two years with inoculation. Usually they were graduate students or the faculty member or the technician, but the people who were inexperienced designated by an [variable] I, and these were usually high schoolers. So, my advisor would carry out about 20 plus acres of sweet corn research trials every summer. So, we had a big team every summer and we had to hire a lot of time sub people in the summer to help with this. And sure enough, you can see here if you look at incidence severity, the experienced folks, at least two of those experienced folks, the line followed the concentration of inoculum on the x-axis. You can see it followed a pretty straight line. But the inexperienced people, particularly for example, the I4, they got lazier and lazier as the day went on. | 2 new line graphs appear. |
| They started at 10 to the 3 and as they got tired and the heat hits, they just got more and more lazy about how they applied inoculum. So, we were able to demonstrate that even though I'd used the correct experimental design in terms of assigning concentrations randomized within each replication and I had the block design, I no longer was blocking in terms of how I was applying the treatments. I was confounding treatment with people. And this really drove home to me something we'd -- I'd learned in the statistics courses and the, you know, experimental design courses about blocking, but I never fully grasped just how important that blocking process is and what it is that you're trying to remove from your natural variation with the differences among treatments. So, it was a big blunder on my part. I was able to make lemonade out of the lemon I created, so I -- and be able to, through keeping good notes, come back in 1997 and test this effect of the inexperienced versus experienced folks doing inoculation, which has an impact on ability to generate good disease levels. So, this was my blunder and it's always stuck with me. And it's really driven home to me how poorly in most statistics courses we really explain in a practical way what we mean by blocking and how to remove unknown sources of variation. | The line goes steadily down the graph. |
| So, this is what I want to cover to a fair degree today is to talk about when you're running an experiment, how do you set up your trial to try and remove these unknown sources of variation or these sources of variation that can found what it is you actually want to measure your treatments. So, I have a picture here of a field at our research center in Mount Vernon. | A new slide with text that reads, blocking for potential sources of variation. Besides the treatments you are testing. |
| And this was a field in which I had to set up a spinach field trial in 2001. So, I want you to, if you're comfortable, speak up and tell me what do you think in this particular field might influence what I was measuring. So, it was a spinach trial and we're looking at leaf spot diseases caused by fungi that are favored by leaf wetness, humidity, you know, high-moisture conditions. What do you think in this field might confound disease pressure and those field conditions that I should take into account when I set up this trial? Anyone willing to speak up or put something in the chat? >> Hi, this is Jane. >> Hi, Jane. >> Hi. Just looking at the tree cover, you could see that there'd be maybe more shade in one area, being moist at a different time of the day. >> Yeah, absolutely, Jane. Yeah. So, those are on the left hand side of that picture. Those are large, very, very tall cottonwood trees. And they're on the -- so it's hard for you to know the direction here because it's cloudy, but those are on the east side of the field. So, east-west runs sort of across the page. And so, the sun would rise on the east and those big cottonwood trees would create a very long period of shading. So, plots that are closer to the trees would be in the shade for much longer than plots on this other side of the field. So, you have a shading effect that runs in an east-west direction. So, how should I set up -- do you think that would impact this, these leaf spot diseases? Obviously, yes, because it's a -- a foliar disease that's influenced by leaf wetness and shading is going to mean that the leaves would dry out much slower. So, in this case, which direction should I put the blocks? Is anyone willing to speak up about how I should lay out the blocks? Should the blocks be running in a north-south direction or east-west direction? If anyone's comfortable speaking up. [Laughter] >> North-south. >> Okay, so you want them in a north-south direction because in the north-south direction, you're going to have the same amount of shading across that entire block. So, every plot in that block will receive a very similar amount of shading. If my blocks ran east to west, any treatments that are on the west side of the plot are going to get less shade and dry out much quicker than plots that are closer to the trees. The whole idea with blocking is you want maximum uniformity within the blocks so every treatment in that block gets exposed to the same conditions. It's okay to have lots of differences between blocks, but you don't want lots of differences within a block in terms of the conditions that your treatments are exposed to. You -- the treatments are what you're trying to measure, not the other things. So, thank you, Jane. So, it's really important when you're trying to set up an experiment to think about what it is you want to measure and what other things might influence what you're trying to measure besides the treatments you're applying. So, another example is plant size. If you have a disease or a treatment that is influenced by the size of the plants, let's say you're growing seedlings and you want to use them for an experiment and some of them happen to grow slower than others for whatever reason, how might you block for that effect? Anyone willing to speak up? >> Well, you'll just have to make sure you've got plants of all sizes in each block. >> So, actually, it's the opposite, Claudia, but that's really important. I'm glad you said that. So, there's one way you could do it is just randomly choose and say that within a block you want every size possible. But if size potentially influenced your disease, then whichever treatment goes to the smaller plant will have a different disease pressure than whatever treatment goes to the bigger plant. So, what you can do then is look at your plants and divide them by size. And let's say you can have four blocks. See if you can divide your plants into four different sizes, you know, approximate four categories of size. And then within each block, every -- every treatment goes to a similar size plant. Because when you run your ANOVA now and you have a block effect, that influence of plant size is now in your block effect and not in your treatment -- confounded with your treatment effect anymore. So, this is a really good example that we frequently encounter in our greenhouse trials. And this is a case where you can select and move plants around. How about if you've got -- you're growing inoculum and it takes a while and you have to grow inoculum over multiple months? This is obviously not for bacterial cultures, but usually things like fungi. And you've got inoculum of different ages. How might you try to account for that if you know that the age of the inoculum could influence the amount of disease or how virulent the inoculum is? So, a similar kind of thing. If you say, all right, I've got four blocks, I've got five reps, and I have three batches of inoculum, can I divide that inoculum up so that within each replication, all of those treatments get the same age of inoculum? Another one that's really difficult to work with, any entomologist in the group will know this, is if you're dealing with, say, a disease that's -- where the insect's vector, a pathogen like a virus or a phytoplasma, and there you don't have necessarily any control of the migration of the insects. So, how might you try to control this? And I've had this really exact example happen in one of our field trials with iris yellow spot virus in onions that's just vector by thrips. And you don't necessarily know which direction the thrips are going to come from. So, do you have any thoughts on how you might account for this in an experimental design? This is a tough one. I had a big experiment one year that was completely, completely a waste of time because I went out to the field and I thought, okay, it's in a growers field. And I thought, okay, I think this is the direction the thrips are going to come from. So, I blocked so that, you know, that one block was closer to where I thought the thrips were coming from because that would have the most inoculum and moving away. And lo and behold, when the trial took place, thrips came from the complete opposite direction, perpendicular to what I anticipated. And so, the disease pressure was completely confounded, totally confounded with the treatments. And I had zero ability to tell my treatment effects at all because the variation associated with the direction that thrips came from was totally counter to the way I had blocked. And the data were a complete waste. I couldn't use it at all. So, very frustrating situation. And this is a tough one when you don't necessarily have control. But if you have an idea, for example, that certain types of vegetation are favored by the insects, insects tend to do well on certain crops or certain native vegetation, you might try to think about that when you're setting up your experiments. And this is where it becomes really tough with field trials. You tend to have less control. Anything that's influenced by, say, soil fertility, if you've got different differences in soil fertility or you've got differences in drainage and you're dealing with a disease that's a wilt disease, it's influenced by soil moisture, and you've got really well-drained areas in the field and poorly drained areas, that's going to influence disease pressure. Sloping ground or crop-rotation history, if you're putting a trial in a field that's had different crops in parts of the field in the past years of research, that can potentially influence how well your crop might grow and potentially how it might influence whatever it is you're trying to measure: yield, disease pressure, food quality, whatever that might be. Really important things to try and think about. Prevailing winds and temperature, one of the things we've run into in some of our research field trials, we do -- have done a lot of work on some wilt diseases. And wilt diseases are impacted by transpiration demands. So, during warm, sunny weather, the need to transpire means symptoms are expressed much quicker because if the roots are clogged by verticillium or fusarium or some other wilt pathogen, when the plant has to transpire a lot in the heat and it's unable to because the vascular system is clogged, symptoms are expressed much quicker. And we learned that when we're reading disease, even though we had a trial already set up, if we started reading in the morning, like at 8:00 in the morning or 7:00 in the morning, and it was nice and cool and the plants had recovered overnight, and we came back to the same plots later that day, at midday or 2:00 in the afternoon, the disease would be completely different. So, we learned we had to rate when symptom expression was consistent across the reps or at least across the treatments because you could confound the time of day that you rated with which plots you were rating. And we had some big trials that would take two days or three days to rate. So, we would wait until 11:00 o'clock in the morning and then go out and rate by block. And if we didn't finish a block in that same consistent amount of wilt pressure, we would stop and finish that block that day and start the next block the next day. And just to avoid that issue of time of day and how it might influence disease expression. So, there's just lots of little nuances that, you know, on paper you know what a block design is and even if you sort of do block correctly in terms of how you assign your treatments, the blocking comes to more than just where you put your treatments. It's everything you do. It's how you apply the treatments. In the case of the smut, I didn't apply it blocked. I applied it by person. It's how you rate it and how you evaluate, how you harvest, and so on. All of those things can influence and make your data more noisy and less able to differentiate your treatment effects. And I think the really important thing to think about in this is if you don't know what sources of variation might come in and influence what you're measuring, you're not going to be able to block for it. So, blocking is only effective when you can predict what might be countering what you're measuring. Okay? It might be influencing what you're measuring and changing your treatment effects. So, this is the tough part about putting out trials if you don't anticipate things that could confirm what your treatments are doing, then your blocking is not going to be effective. So, does this make some sense? It -- to me, you know, when you read the theory about blocking, it makes sense, but the reality is it's counterintuitive. It's like Claudia mentioned about the plant size. I find over and over again that the blocking effect is counterintuitive. You actually have to think in reverse to truly fully understand the importance of blocking and how you block effectively versus ineffectively. | A picture of a green field with trees. Text on screen reads, Other potential sources of variation? plant size Age of pathogen cultures Direction of migration of insect vectors Soil fertility, drainage, sloping ground, crop rotation history Prevailing winds, temperature (e.g., mornings vs. afternoons for rating wilt diseases) You can't block for sources of variation you don't know about / don't anticipate |
| So, I'm going to talk a little bit about kind of getting it back from this thing of blocking and what you're trying to remove to thinking about your data analysis. And what it is is when you're measuring a treatment effect, if let's say you're measuring an effect on yield, if I measure yield of a plot with or without a treatment and I measure the next plot, there's going to be some difference. They're not identical, whether it's in a, you know, plants in a growth chamber and height of the plants or whether it's yield in a field or fruit, whatever it might be, there's random variation. And so, that experimental error is just the random variation in whatever it is you're measuring. So, if you had 20 different plants and you measured something like plant height, they're not all going to be identical, even in a growth chamber. | A new slide with text that reads, experimental error. Random variation among observations or experimental units. |
| And that variation from plant to plant is what we call experimental error. The key thing in analysis of variance is you try to say the variation from plant to plant, how much of it is because of your treatments and how much of it is because of this experimental error, this random error that occurs naturally, the background noise? If you've got a very noisy data, you're going to have less ability to tell your treatments apart. So, and you have to be able to measure that experimental error. So, you have to estimate it. You don't know what the true experimental error is. You're trying to estimate it. This experimental error is from natural variability in the experimental units. As I said, not every plant is going to be identical in height or identical in disease pressure, even if everything was the same. And then it's also associated with lack of uniformity in how you carry out the experiment. And this is where you're trying to challenge yourself to know how can you make your experimental design as uniform as possible to remove this lack of uniformity in carrying out the experiment from the natural experimental error. And we use this experimental error, we have to estimate it to evaluate the results of an experiment. And that's what ANOVA is. It's just saying what's the variation in what I'm measuring, how much of that variation in what I'm measuring -- disease or yield or sweetness of the fruit, whatever it might be -- how much of that is because of experimental error and how much of it is because of the treatments I'm trying to evaluate? If your experimental error is large, you're going to have to have very big differences among your treatments to be able to tell them apart statistically. If you have small experimental error, you're going to be able to pick up smaller differences in your treatment statistically. That's really what ANOVA comes down to, is trying to separate variation due to noise from variation due to your treatments. Okay. All right. So, how many -- how many -- how do you estimate experimental error? And this really is why you replicate and why you randomize, because you're trying to separate experimental error from treatment differences and number of replications and randomization are the critical way in which we measure error. And again, this sounds basic. It's covered in experimental design, but I think we don't -- we're not taught it in a way that truly helps us understand how we translate that to setting up an experiment. So, how many replications do you need? If you had endless money and time and endless resources, you could do a hundred reps, but we know that's not possible because of limits to resources and time and so on. But the important thing is to recognize that the smaller the difference between treatments you have to be able to detect, the more replications you're going to need. So, if you're measuring something that takes a lot of whatever it is to cause a difference, you're going to have to have more reps and the more replications you have, the more precise you're able to estimate that experimental error. And this is what's really important. It doesn't reduce your experimental error. If you've got noisy data, it doesn't take that noise away, but it gives you a more accurate assessment of how much noise is in your data. And you estimate this based on degrees of freedom. It is the number of treatments multiplied by the number of reps minus one. So, the more reps you have, the more degrees of freedom, the more ability to separate out your treatment effects from your noise. And that's influenced by your experimental design. So, it starts to get a little theoretical here, so I apologize. But I think this is really important to keep reminding yourself why we talk about how many reps you need, why we talk about the importance of your experimental design, and why we ask you why degrees of freedom are important, because degrees of freedom reflects the number of reps, which is important ability to estimate your experimental error. These are really good questions for preliminary exams, defense exams for students is what is ANOVA? What are you actually trying to do when you run ANOVA? Why do we say degrees of freedom are important? >> I have a question for you, Lindsey. >> Yeah, yeah. >> What do you think about the plot size and its importance? >> Yeah. >> Because we were going to set up a dahlia trial with four reps, 20-foot plots, and the statistician said, no, just only have seven-foot plots and make six reps instead. That's better. >> Yeah. That's a really good point, Claudia. And I think the issue there is going back to this comment about you want maximum uniformity within a block, okay, a rep, and if you've got lots of variation in some form or another, you want it between your blocks, not within your blocks. So, in that case, the statistician is correct. Obviously, sometimes equipment means if you've got a big piece of equipment, your plot has to be a certain size because you can't turn your tractor around in a seven-foot distance. There's sometimes logistic reasons you have to have a bigger plot, but you want maximum uniformity within a plot, as uniform as possible, which means smaller is more uniform, especially when it comes to field trials. And it's okay to have -- if you have big differences between your reps, your blocks, that means you've blocked correctly. That means that variation between blocks is accounted for in the blocking effect in your ANOVA. So, your statistician is absolutely correct, Claudia. >> Thank you. >> It's better to have more reps and smaller plots because then your plots are more uniform within that plot and within the block, there's more uniformity. >> I was afraid somebody was going to say your plots are too short. >> Well, that's a good question, Claudia, and you have to think about what is it you're measuring? So, for example, if you're measuring something where, you know, in plots, sometimes you have this border around the plot. And so, things like yield can sometimes be influenced by plants on the outside of the plot, are going to have less competition and yield more. And so, depending on what it is you're trying to measure, your questions you're asking, getting too small a plot could influence what it is you're measuring in the wrong way. So, you have to think about what are my questions? What am I testing? Does a small plot start to influence in a way that's not a treatment effect? >> Okay. Thank you. >> Yeah. A really good question, Claudia. Thank you for asking that because, you know, your thought is "I want big plots". Big plots are only good if there's uniformity within the plot and within a whole block. And so, if you can make a plot smaller without having that adverse effect of things like edge effect, then it's better to have more reps and smaller plots. Yeah. Really good question. Thank you. So, that comes down -- you've basically helped address this question at the top of the slide. How do you control error? | Text reads, Sources of experimental error. 1. Natural, inherent, variability in the experimental units or material. Example, variation in soil properties across a field. 2. Lack of uniformity in carrying out the experiment. Example, changing source of a chemical among treatments, different people applying different treatments or measuring different treatments. Experimental error is used to evaluate results of an experiment: Compare the size of the treatment differences with the size of the experimental error. If the experimental error is large, you can only detect large treatment differences! How do you estimate experimental error? 1. Replication 2. Randomization How many replications? 1. Smaller the difference you need to detect, the more reps you need 2. More reps = more precise estimate of experimental error (does NOT reduce experimental error) 3. Degrees of freedom used to estimate experimental error: = (# of treatments) x (# of reps -1) 4. Depends on experimental design |
| So, it's through your experimental techniques, exactly what you said, Claudia. Uniformity in how you carry out the experiments. So, remember how I mentioned when I assigned people to apply the inoculation treatments, even though I had randomized the allocation of concentrations of inoculum within reps and I had applied them, you know, where they go was correct. The way I had people applying it was no longer blocked. It was confounded with people. So, avoiding bias and uniformity in how you carry out the experiment. It's one of the reasons we use standardized methods. So, you write out the protocol, make sure you consistently follow that standardized method. Things like disease assessment keys where you bring in subjectivity and bias in how you're rating as you get tired through the day and so on. You have standard treatments that help calibrate your brain to -- for uniformity. A really important part I find, especially when you're dealing with research that entails a lot of help, is making everyone aware of your potential sources of error. You yourself as a lead researcher or a grad student may know these sources of error, but it's important to tell everyone who's helping you about these potential sources of error. So, when I was doing the smut trial, what I should have said is to everyone, "Please make sure you put the needle of the hog vaccinator right down the center of the silk channel because if you don't, you get it between the husk leaves and not in the silk channel, the inoculum's not going to be around the ear and it's going to have a big impact." And I probably did say that, but we had high schoolers who didn't care. They just wanted to get paid and leave at the end of the day. So, just being aware of those things is really important. When we are setting up trials, for example, we do a lot of wilt trials with spinach, fusarium wilt that takes -- you know, it may take two days to prepare all of our inoculum. And now you've got batches of inoculum being prepared over multiple days, different strains, different treatments, and you potentially introduce error if you use different people to make up different batches of inoculum because of the importance of that uniformity. So, I make sure that whoever does one task stays with that task throughout the whole procedure. You don't switch people midway through a task because you can potentially introduce bias or error. So, all those little things have a huge difference. When we were planting spinach trials, when I first started working with spinach, I wasn't aware of just how important the depth of planting of spinach seed in the greenhouse can make a difference on germination and rate of growth of the plants. And so, I used to say, well, okay, you know, "Heather, you plant parent A and Mike will plant parent B and I'll plant parent C." And so, again, I was confounding parent instead of blocking us by replication. And I learned this really quickly that this was a bad thing. So, now when we set up these experiments, we say, "Okay, Mike, you're planting all of rep one. Heather, you're planting rep two. I'm planting rep three." So, that if one of us is planting too deep or too shallow, that whole block is going to have that same effect. And it's made a world of a difference in our ability to get good statistical analysis of the data. Every step in how you carry out an experiment, you want to think about, am I confounding something with what I'm trying to measure? And so, it takes a lot of thought and planning ahead and then making everyone aware who's helping you set up an experiment, aware of this and that they understand the need to stick with what they're doing and not switch around and so on. So, applying treatments uniformly over reps and then also making sure how you measure is unbiased, things like standardized disease assessment keys and so on that can help remove that bias. It's -- it's really about thinking carefully as what's going to influence what I'm measuring and how do I remove that influence so it's accounted for in the block effect and not confounding the treatment effect. And it takes a lot of thinking, and when you first start working with a disease or a problem or a crop or a system, you have to build up a familiarity to anticipate these things are going to influence what you're measuring. And that's why it's always scary when you start a new system and you're like, oh, my goodness, I didn't think about this because I'm not familiar with this crop or this problem or this issue, this treatment and how to do this. So, but -- but we in our program, we work really hard to set people up in a specific task and they have to stick with it. So, if someone tells me, well, I have to leave at noon today because I have a doctor's appointment at 1:00, so well, then you can't do this because you're going to have to leave midway through a block and someone else will have to take over from you. And now we have potential variation in how something's done. So, we say, sorry, you're not going to be involved. We'll give you something else to do. But you really have to think that way in how you run these experiments. Selecting your experimental materials to be as homogeneous as possible within a block. Okay, that's if you use a block design, which is almost always the case for agricultural research. If there's things that you can measure but you cannot control and you don't have the ability to separate them by block, there is such a thing called covariate analysis. So, for example, if you didn't, you know, going back to the plant age thing, Claudia, if you didn't really have a nice way to divide them up by block, but you can measure plant height, you can measure it. And when you run the analysis, you can treat it as a covariate so that it removes that variation from the treatment effects. So, it's another way to be able to remove the influence of something that you can measure but you can't control and you don't have the ability to separate it by block. So, it's just another way to consider this. So, being really careful in how you select your experimental design and the key thing here is in red: you want as uniform a block as possible, minimal variation within a rep, maximum variation among a rep. I used to sometimes hear students say, "Well, I have a significant rep effect. That's bad." It's like, no, no, no. That means you chose exactly the right design. If you have a significant rep effect, it means you removed variation from your blocks, between your blocks from the variation due to treatment. So, you were effective in separating that variation that is experimental error from your treatment differences. It is important to use the simplest design possible and we'll go through an example to discuss this. Simplest designs have the greatest statistical power because most of your degrees of freedom go to estimating experimental error. When you start adding these constraints around randomization, you start removing degrees of freedom from the error and I'll show you as an example. So, this randomization is really important. It's a physical process of not just assigning treatments to which experimental units or plots they go to, but applying them as well and how you collect your disease, your disease assessments or whatever the variables are that you're measuring. So, you're trying to get an unbiased estimate of your experimental error. Okay. And we've all read this in our stats classes but I think until you do your own research, the full impact of this doesn't hit you. So, you're trying to make sure that no one treatment is favored or handicapped consistently. You're being unbiased and uniform in how you're doing that. And this is really important because also for analysis of variance, you're supposed to -- these errors, the estimate of your errors, these are supposed to be independent and not correlated. | A new slide appears. Text on screen reads, How can you control Error? Experimental techniques a. Uniformity in how you carry out the experiment, avoid bias b. Standardize methods – write them out, standardized images (e.g., disease assessment keys) c. Make everyone aware of potential sources of error or bias d. Apply treatments uniformly over replications e. Take unbiased measurements 2. Select experimental materials carefully – as homogeneous as possible 3. Covariate analysis – Variables you did not apply and cannot control but which affect what you are measure - use the variable as a covariate to remove that variation from treatment effects 4. Careful selection of experimental design a. Group experimental units so you minimize variability within blocks & maximize variability among blocks – remove variation associated with blocks from estimate of experimental error b. Use the simplest design possible Randomization = physical process of assigning and applying treatments to experimental units (plots) to assure your estimate of experimental error is unbiased 1. No treatment is favored or handicapped consistently 2. Validate assumptions of ANOVA for independent errors (not correlated) |
| So, we're going to give you some examples here. Let's say you have a completely randomized design. The simplest design possible. You're not blocking. You have four reps. Same set of treatments. You just apply them randomly to whatever's after. There's no restriction on randomization. The benefit of a simple design like this is it's easy to layout. It's flexible. The data analysis is easy. You have maximum degrees of freedom for estimating experimental error, and I'll show you that in the next slide. The disadvantage is that all the variability among experimental units is lumped into your estimate of experimental error. So, if you have a lot of variability across all your plots, your plants or whatever your experimental plot basis is, it's all going to be confounded with experimental error and it's going to mean more noise and less ability to separate your treatment effects. It's very rarely appropriate for field or greenhouse trials and even growth chamber trials. We've had trials of students in growth chambers you'd think would be about the most uniform you could get where we discovered that the edge of the bench, the shelf in the growth chamber had more wind turbulence from the way that air would move around the shelves and the plants on the edge of the shelf would dry out quicker than the plants in the middle. And we were dealing with treatments that were influenced by that drying effect. So, we had to block even in a growth chamber, which you wouldn't think so because a growth chamber is supposed to be about as uniform as you could possibly get. So, these little things can really influence your -- your -- if you're paying attention, you're keeping good notes, and you're monitoring carefully, you can start to recognize things that are confounding what you're measuring. In contrast, if you look at a block design, which is about the most commonly used in agricultural research, the whole purpose of blocking is to reduce the, you know, get a more accurate estimate of experimental error. So, you have less noise in your experimental error. You're minimizing the variation within blocks, maximizing variation among your blocks. If it's effective, you're removing the differences among your blocks from the estimated experimental error. You have a more precise estimate of experimental error and more ability to separate treatment differences, smaller treatment differences. The disadvantage is if you're blocking is not effective, you actually increase the estimate of experimental error. And I'll show you in the next slide why that is. So, that's why it's really important to have as uniform a block as possible. Blocks are not uniform. There's variability, a lot of variability within your blocks. And if you haven't blocked correctly, you actually weaken your ability to separate your treatment effects. So, let's look at some examples here. | A new slide titled, analysis of variance. Anova. Text on screen appears as host describes. |
| We're going to run an experiment, same experiment both ways, as a CRD and as an RCBD. We have four cultivars, factor A. We have three strains of a pathogen, factor B. And the treatment design we're going to use is factorial. So, we want all possible cultivars inoculated with all possible strains of the pathogen. So, we have 12 treatment combinations, and that's the treatment design. The experimental design, we're going to run it as a CRD, completely randomized, and we're going to run it as a block design. And we have four reps. So, we have a total number of experimental units as A by B bar. So, it's four cultivars by three pathogens by four reps, 48. | An example appears which the host reads out. |
| So, in the CRD, here's what the ANOVA table looks like. And you don't have to get too bogged down by all these formulae, but you've got factor A, the cultivars, factor B, the strains. You've got the A by B interaction, and then you have experimental error. And if you look at degrees of freedom, it's just, you know, the number of treatments for that factor minus one. So, you've got three, two, the combination of six, and experimental error is 36. So, you have 36 degrees of freedom to estimate your experimental error. | The anova table for C R D appears. |
| Okay, this is estimating the variance associated with each of these. The mean square is just the variance divided by degrees of freedom, and the F value is the ratio of the variance for this factor divided by the experimental error. So, you can see that the bigger the degrees of freedom here, the higher the F value there because this is a ratio. All right? | She hovers over the 3rd column, sum of squares. Factor A is S S A. Factor B is S S B. Factor A times b is S S A B. Experimental error is S S E. Total is S S T. |
| If I now run it as a block design, and let's say in this block design, I assign cultivars to main plots, the big plots that cultivars are applied to, and then I apply pathogens to split plots within those main plots. And now look at the ANOVA table. I have a block effect. This is taking some of that experimental error and separating into a block effect. Then I have factor A, and it's -- it's measured based on interaction of blocks with factor A because this is applied to main plots. Factor B is measured by interaction of blocks with factor B because this is applied to split plots. There's the main plots, and then you have experimental error. Look at the degrees of freedom: 18. So, when you look at this number here, you're dividing it by 18. All right? So, you're going to end up with a smaller F value than if you had 36 degrees of freedom, unless your blocking effect is correct in removing that variation. So, why is -- what is the F-test? Why is it important? Well, it's called F because it's named after Fisher. He was a statistician in the early -- in the 1800s. He developed this. And basically, the F-test is a ratio of two variances, and these variances are estimated by the mean squares. And you're looking at the variance between your groups, so between the cultivars, in factor A, divided by variance within the group. So, that's really what an F-test is, a ratio of two different variances. The higher the F value, the better your model. You have a great ability to detect differences among treatments if they are different. If they're not different, it's not going to make a difference. If there's no difference between treatments, it doesn't matter how many reps you have, you're not going to see that. But the higher the F value, the better your model. Any time you restrict randomization, you lose degrees of freedom in your experimental error. So, you start to get a less powerful test, unless that restriction on randomization is taking that error and putting it into your block effect. Okay? So, another important thing to think about when you start to block and if let's say you have two factors like this, and should you use main plots and split plots or should you just completely randomize within each block? Does anyone have any thoughts on which would be better? So, let's say you have a RCD, but it's not split plot. It's just within each block, you completely randomize the 12 treatments versus applying some to main plots and some to split plots. Which do you think would be more important in terms of power of the design? Is anyone willing to speak up? Doesn't matter if you're wrong, because that means more discussion. [Laughter] So, I don't know if you remember my comment about the simplest design is better because if I -- if I have now two -- by having a main plots and split plots, I have two restrictions on randomization: I have a restriction at the main plot level and I have a restriction at the split plot level. So, I'm removing degrees of freedom here and here from experimental error. So, if you don't need to use a split plot, it's better not to. If you know you can block to separate some to have uniform blocks, differences between blocks, but uniformity within blocks, it's better not to use split plots because you're starting to restrict randomization in two places now, not just your blocks, but also the difference between main plots and split plots. So, if you don't block -- if you don't use a split plot, you'll have more degrees of freedom in your error -- experimental error estimate. You'll have more power for separating treatment effects. If you do end up using split plots, the treatments that are applied to the main plots are going to have less power than the treatments applied to split plots. So, it's something to think about. If you do use a split plot, whatever treatments are applied to the split-plot level, they're going to have greater ability to separate those treatment differences because there's more degrees of freedom, more power at the split-plot level than there is at the main-plot level. So, you might think about that when you're setting up an experiment. Sometimes you have to use a split-plot design just because logistically, your equipment doesn't -- let's say you're using sprinkler irrigation. You can't get your sprinkler irrigation size small enough. So, your sprinkler irrigation has to be main plots, and then you have your cultivars within those. So, you have it -- logistically, you have it. You don't have a choice. You have to go that way. But recognize that whatever is at the split-plot level has more power to separate treatment differences than whatever at the main-plot level. So, you might ask yourself, which one is more important for me to measure? Which treatment, factor A or factor B, is the one that I'm most interested in finding treatment differences? So, again, going back to your questions. So, what I wanted to do here is set up a -- let me see if I can do this -- set up a -- of course now I'm not going to be able to do this. Heather, I think I seem to have forgotten -- lost the ability to do the whiteboard. Do you have the whiteboard active, Heather, on your screen? Sorry, you muted. >> Sorry, you'll have to stop sharing your slides. >> Okay, oh, there we go. | An anova table for R C B D with sources of variation being Blocks, factor A, Blocks times A, Factor B, Blocks times B, A times B, experimental error and total. |
| >> And then-- >> All right, I'm going to-- >> Go to your Zoom controls and choose the whiteboard and share your whiteboard. >> Okay, now where did my whiteboard go? Hmm. I've lost the ability to do the whiteboard, Heather. [Laughter] >> So-- >> All right-- >> Go back to your main-- >> Yes-- >> Zoom menu. >> Zoom whiteboard, yeah, there we go. Okay, loading my whiteboard. Sorry, folks, thank you. All right, new whiteboard. All right. All right, let's see. Now if I go to share screen, whiteboard. Okay. Can you all see the screen now? | The presentation closes. |
| >> Yep, I can see it. >> Yep. >> Great, all right. So, what I'm going to do here is draw an example, and if you have cases that you want to discuss, let's do that. | A whiteboard opens. |
| So, I'm going to start by drawing, and I'm going to choose just black. So, let's say I've got a greenhouse, and excuse my wobbly writing with a mouse. Okay, there's my greenhouse. And I'm going to tell you that over on this side of the greenhouse are my heaters to keep the greenhouse warm when the nights get cool or, you know, it's cool days. | She draws a square. She draws a red squiggly line on the left side of the greenhouse. |
| So, on this side of the greenhouse, I have the swamp coolers, which are used for cooling the greenhouse when it gets too hot. So, there's a temperature gradient. Sorry, there's a temperature gradient from this side of the greenhouse to this side of the greenhouse. And I'm setting up an experiment with something that's influenced by temperature. The plants -- the growth of the plants is influenced by temperature, the pathogen I'm inoculating is influenced by temperature, and I want to set up four blocks of my treatments. Which direction should I be blocking? >> Up and down. >> Up and down. Okay, so thank you. Let's see. I'm going to just do a block. So, your block should look like this. Oops, I'm obviously not doing this right. [Laughter] [ Laughter ] There we go. So, your block should look like -- ah. Okay. Thank you, Janine. So, the block should go up and down. Why is it -- there we go. Of course, I can't do it now. | She draws a green squiggly line on the right side of the greenhouse. |
| There we go. So, there's my one -- there's one block and there's your next block. And so, what you see is within this block, every treatment, let's say I've got eight plots, eight plots of plants up and down, they're all going to be exposed to the same temperature. Whereas ones here are going to be closer to the heaters, they're going to be hotter. And ones here are going to be closer to the swamp cooler, so they're going to be cooler. | She draws four vertical blocks going up and down in yellow. |
| If I went the opposite way and I block this way, I would have a really, really bad design because I have really noisy data tied to the influence of temperature. Anything that's on this side is going to be much warmer than anything on this side, and that's going to influence what I'm measuring. So, you see simply by changing the direction of your blocks, you can have a big influence. In this case, I know that there's heaters and these coolers, I know there's a temperature gradient. I've worked in this greenhouse. I'm familiar with that. Does anyone have any other examples they want to go through as -- just to see what we can discuss? Experiments that you've run or you want to run or things you can think about that -- that might -- you might want to try and draw. >> So, in my case, we have the temperature gradient, but perpendicular to that, we have a bit -- it's not uniform, but we have a light gradient, which makes blocking a bit difficult. >> Yeah, really good, really good question. So, let's let me see if I can try to draw that approximately. So, let's say here's your greenhouse. Okay, Janine. | She draws a block going left and right in blue. |
| And the same thing as before, we had the heaters on the side. All right. And then we had the coolers on the side. All right. So, you know you've got a temperature effect, but now you've also got you said a light effect. | She redraws the greenhouse with heaters and coolers. |
| So, you've got bright on the side and you've got dark on the side. So, this is a really good example, Janine. So, now you've got temperature gradient and you've got a light gradient. So, anyone have any thoughts to help Janine know how she should set up this experiment? Let's say you want four reps, Janine. How -- any thoughts from anyone here about how you might set this up? It's a really good example. So, remember you want maximum -- within a block, you want it as uniform as possible. | She draws a yellow line at the top of the greenhouse. She draws a brown line at the bottom of the greenhouse. |
| So, my recommendation, Janine, if you had four blocks, would be-- >> I think -- sorry, sorry, I'm interrupting, but you'll have to block both ways. But you just wrote it. >> Yeah. >> So, you have to separate them both two block. Yes. >> Yeah, exactly. And one thing, a couple of things you can think about Janine and, sorry, Alicia, thank you for speaking up is, you know, if say temperature is more important than light, then you might consider what, you know, one direction versus the other. But if you know they're both important, this type of design is going to give you more uniformity that accounts for both factors than if you did, say, down the length of the page. Does that make sense? >> Yes, thank you. That helps a lot. >> Yeah. So, just keep asking yourself, how do I set up a block so that I -- these, you know, if I have eight parts in here, there's more uniformity than if I did down this way where the light gradient is going to have a huge influence >> Okay. Yeah, we also blocked according to our irrigation system as well, which is something we initially didn't think was going to be important. >> Yeah. >> But, you know. >> Yeah. And that's, as I said, Janine, experience. If you've not used equipment before, you've not used -- it's a new greenhouse, it's a new lab, you know, new field, all these things make it more difficult if you don't have the ability to anticipate what's going to influence your experiment and what you're measuring. >> Yeah. Exactly. Thank you. >> Yeah. Yeah. So, that's a really good example, Janine. Thank you for bringing it up. Anyone else have example they want to cover? >> Lindsey, this isn't an example or question, but just a comment, and I'm definitely not an expert in this, but there are ways to use spatial autocorrelation in your model in addition to blocking so that sometimes if there's patterns you don't expect or patterns that aren't quite simple, you can still tease apart some of that clustering of like variability. >> Yeah, it's a good, good comment. Thank you, Gabe, and yeah, there's more complex ways to analyze data that can account for that. But then you also have to be able to measure that. And it's -- I mentioned earlier on how you can do covariate analysis, which is one form of this autocorrelation. If you have something like, in your case, Janine, if you had a means to measure that light across, let's say you couldn't block for both and you blocked for one, but you could measure the light or the temperature very precisely across the gradient. You can do an autocorrelation. That's another way to run your analysis. It gets a bit more complicated on the analysis, but, yes, thank you, Gabe, for bringing that up. Good. Really good point. We have a couple of items in the chat box. Oh, yeah, Chuck earlier on talked about trees and sloping ground and shade. Thank you. Any -- any other -- other examples people want to draw on the whiteboard? We don't have to continue to use a whiteboard, but I just thought it was kind of fun to make it interactive. Okay, I'll stop sharing that and then just go back to this. Can you all see the slide? | She draws four blocks in each corner of the greenhouse. |
| So, I'm just going to wrap up with a mention that, you know, this -- I should have said this at the beginning, but this workshop is recorded. So, Heather will post this recording along with the handouts on AlliumNet website where "Stop the Rot" project materials and resources are posted. Anyone can also email me if you have questions. And this is just a link to that paper that I discussed on comments, not that I expect you to read it, but I think it's a good example of how I made my own blinders as a grad student, and it drove home some really important principles. So, as my technician likes to say, experience is making mistakes and learning from them. | She opens the presentation back up. A picture of a rotted onion next to Edvard Munch's Scream painting. |
| So, I'm going to stop sharing with that. Does anyone have any questions or comments they want to bring up to help others in this workshop? >> I think when you evaluate for severity or cytotoxicity, you also need to have consistency in the folks that evaluate because some will overestimate, some will underestimate. >> Yeah, there's a whole set of, you know, we could do another workshop just on how to -- how to do disease assessments or rating or whatever it is. How do you measure whatever it is you're trying to measure as objectively, as unbiased as possible? And Claudia's right. You know, one of the things I love to do is when I rate is to have someone with me. We rate in pairs, particularly if I know it's a more subjective process of rating and it's more difficult thing to rate, whether it's disease or cytotoxicity or whatever it might be. If you know, you know, important part as a researcher is to recognize your bias. If you say "I'm not biased", then you've got a problem. All of us is biased. We all, especially if you're familiar with the treatments and you've been applying, so you tend to know which plots of which treatment you're more biased, and you have to try and remove that bias from how you analyze and make your assessments. So, things like assessment keys, having a partner to help you rate. When I rate in pairs, I tell whoever's holding the clipboard with a notepad, they write their number down before I call my number out so that I'm not influencing them, or you have a discussion. You have standardized treatments in your set of treatments that you use to calibrate yourself, your control treatments. You know, you have assessment keys so you can say this is what -- this is my rating. These are my photos that tell me what a five is versus a 10 or, you know, you can keep yourself calibrated, especially as you get tired later on in the day and your mind starts to wonder if you're rating for hours on end. But there's lots and lots of really important things to try and remove that bias from the way you do your measurements. It's extremely important. Thank you, Claudia. And one of the things you learn as an advisor is running a program is you learn your own biases, but then the hard part when you hire someone is learning what their biases are. And biases are not bad. Important part as a research to say what are your biases and how do you remove them from the way you do research? It's really important to acknowledge your biases and then build in a systematic process to remove those biases from the way you carry out the experiment and the way you do your measurements. And Mike Deary, who works with me, is one of the most objective raters I've ever encountered. So, I love rating with him because I see how objective he is. I tend to be more subjective and I know I have to be careful and remove that. Any other comments around that, Claudia, on disease or rating and assessments? Well, thank you, everyone. It's -- it's two minutes before the hour. I appreciate your attendance. I hope this was helpful. As I said, Heather's recorded this and the materials will be posted on AlliumNet if anyone wants access to those documents or to the slides, and I appreciate you attending. I hope it was useful. >> Yes, it was very useful. Thank you. >> Thank you. >> Thank you so much. >> Thanks, Jane. >> Thank you. >> Thanks, Kumbazila [phonetic]. >> Thank you very much. Bye. >> Bye, Janine. | The presentation closes. |