Season 2, Episode 4 | Challenges of Developing Field Sensors for Vineyard Nitrogen Status (in English)

0:00:00 Patty Skinkis

This is the High-Res Vineyard Nutrition Podcast Series, devoted to helping the grape and wine industry understand more about how to monitor and manage vineyard health through grapevine nutrition research. I am your host, Dr. Patty Skinkis, Professor and Viticulture Extension Specialist at Oregon State University.

0:00:22 Patty Skinkis

Many grape growers are interested in the newest information and technology, and precision agriculture and its application in the vineyard sounds fancy, but what is it really? In season 1 of this podcast, I interviewed engineering researchers in the High-Res Vineyard Nutrition Project about precision agriculture and how it is adapted in vineyards. The primary focus is on the sensors or use of sensors or other tools to obtain field-level information and map that information spatially. Basically, to come up with a map--a vineyard map-- that helps growers make decisions. That data comes from sensors that are comprised of different components and may be applied in different ways such as remotely via satellite or in the vineyard through proximal sensing, that is via ground or above ground means. Some growers I know are enamored by the idea of using drones, but drones in themselves are just a vehicle just like a tractor or an ATV. Where the science and technology lies is in the sensor being developed and used to gather the data. Today we're going to explore the data acquisition part of the equation with Dr. Chenchen Kang, a postdoctoral scholar at the Pennsylvania State University. Chenchen is an engineer working in the Department of Agricultural and Biological Engineering at Penn State, and he just started his new position there. He recently earned his PhD from Washington State University in May 2023 and was co-advised by Dr. Qin Zhang and Dr. Manoj Karkee. He worked over the past two years in Prosser, Washington on spectral imaging to detect abiotic stress specifically related to vineyard nutrients and water levels, and his primary research focus was on developing ground-based sensing technology for vineyard nutrient management. Dr. Chenchen Kang has a bachelor's and master's degree in agricultural engineering from China Agricultural University before earning his degree at Washington State University. Thank you for joining us today, Chenchen.

0:02:23 Chenchen Kang

Thanks for having me, Dr. Patty.

0:02:26 Patty Skinkis

Can you tell us a bit about the research that you conducted while in the HiRes Vineyard Nutrition Project?

0:02:34 Chenchen Kang

The main question that I was exploring in my research is if we can determine the nitrogen levels in grapevines and by analyzing how their leaves reflect light. Specifically, I was using a technology called hyperspectral imaging. I can introduce a little bit of background about hyperspectral imaging. So, when we look at grapevines we see many leaves on grapevines. The leaves reflect the sunlight and then the whole spectrum of light from ultraviolet to infrared, but our eyes can only process a small range of variable light, and the regular digital cameras replicate our eyes. They can capture reflectance in this variable range specifically from three color bands, including red, green, and blue. So, from our human vision and the digital cameras, we can tell the difference in leaf color, leaf shape, and leaf size. However, there are changes happening inside the leaves that we can't see with our eyes. For example, a deficiency of nitrogen in the leaves can cause some changes that affect how they reflect light in the spectrum that we cannot see. The hyperspectral cameras can capture light across the much wider range of spectrum including those invisible parts. By analyzing the differences in how leaves reflect light we hope to determine the nitrogen levels in the grapevines.

0:04:11 Patty Skinkis

So, were you specifically looking at certain wavelengths, and if so, what were those or maybe how did you narrow those down? Basically, how many are there and how many are feasible to really focus on?

0:04:25 Chenchen Kang

So, we started using all the wavelength the cameras can capture, and then we try to rank the importance of those we learned. Like how they contributed to the overall performance of the model. We tried to narrow down to use less and less and find only five that are most important that can contribute to estimate nitrogen levels.

0:04:53 Patty Skinkis

So, can you give the audience an idea of just how many of those wavelengths one could look at with those highly sophisticated research cameras? How many we are looking at, and maybe how many you were able to reduce it to?

0:05:09 Chenchen Kang

The overall number of wavelengths--the camera I used has 274 wavelengths, and first we used all wavelengths to build a model to estimate nitrogen content. We got good accuracy and small errors, then I tried to choose the most important 20 we've learned and build another model. I find the accuracy decreased and if I try to use less the accuracy decreased more, so it's like we have to find a balance or tradeoff between accuracy and field number of wavelengths.

0:05:54 Patty Skinkis

So, having the whole bit was unfortunate, I mean it makes sense that the more wavelengths that you're able to use in the total is going to describe more of the environment. That light environment of the actual leaf and so reducing it makes sense that the accuracy of the model will be reduced. Can you tell us what you were comparing it with? What were you looking at in the model to determine accuracy? Were you ground-truthing it with nitrogen concentrations of the leaf tissue?

0:06:25 Chenchen Kang

Yeah, so, specifically for this I was working on nitrogen, but in this project my colleagues in Washington State University also looked at potassium. For my part, for example, today we want to find the relationship between the differences in nitrogen content and the differences in reflectance and first we need to make some differences in nitrogen levels. This variation, the bigger the better, and so we worked together with the viticulture teams in Washington and Oregon, they applied different rates of nitrogen fertilizer in the experimental trials and at specific timing at bloom and veraison. The viticulture teams collected the tissue samples from pre-selected vines. We used our cameras to scan a canopy from the side of vines at the same time and the tissue samples were sent to the lab for nutrient analysis, then they provided accurate nitrogen content of each one. From our side, we extracted the spectra reflectance from hyperspectral images and for each one we have the reflectance, and we also have the accurate nitrogen content. We have many samples with this paired reflectance and nitrogen value and then we use some machine learning techniques to figure out the correlation and build a model--it would be a regression model. Then we can look at the R-squared value, we can look at the root mean square error (RMSE) and mean error as some [evaluation] criteria. For this model we give leaf reflectance as the input and then the model will estimate our nitrogen value. We can always compare the estimated value if it matches the ground-truth or not.

0:08:34 Patty Skinkis

So basically, being able to scan the leaves in those nitrogen trials in Oregon and Washington and see what the results were based on the scanned sensor results and the outcome of the actual tissue samples. So, the tissues being collected by growers are typically taking leaf blades or petioles and analyzed by ICP and carbon and nitrogen analysis at a lab. What was used--as I know as being in the project--so that growers can relate to very easily--so it's being able to put some ground truthing numbers to those reflectance data and so you get the estimated value. Do you know how much longer this work is going to be continued at Washington State and Oregon?

0:09:23 Chenchen Kang

I think since I already graduated, but my colleagues they are continuing collecting the same types of data this year (2023), I think right now it's at blooming stage. They are collecting data because from our previous results we found some challenges. They also have some new ideas to address the challenges this year and I think also the next year (2024). Also, the more data for the model would allow it to be more robust.

0:09:58 Patty Skinkis

Yes, we always like more data. It's helpful. It's just the challenge in getting it all. I believe all of your testing runs were at veraison only; is that correct in the last two years?

0:10:10 Chenchen Kang

Yeah. We collected data both in bloom and veraison. But, because of some uncontrollable factors, we have more data from veraison. That's why we started analyzing from veraison.

0:10:29 Patty Skinkis

Okay, that's great to know that it's being done at both time points. Do you know, is there any plan to do (or maybe you've done this) scanning of the vine canopies or leaves at other times of year?

0:10:42 Chenchen Kang

Yeah, we actually also collected some data from the [early] shoot [growth] stage of the grapevine with only 5 to 6 leaves.

0:10:52 Patty Skinkis

Oh, so that would be at the shoot thinning stage? That's brilliant because that's when growers are going through and doing shoot thinning anyway. So, you're saying you collected whole shoots and did the ground-truthing on that as well?

0:11:06 Chenchen Kang

Yeah. But, because at that stage the leaf is so small. It's much more challenging to gather reflectance and to find that region of interest from the picture, but we have the data. We are trying to figure out a way to get some information from the data from the shoot stage.

0:11:27 Patty Skinkis

That's great. I think ultimately it would be neat to be able to scan vineyards at different times. It just means more data collection more sampling which just incrementally becomes more challenging from manpower as well as just the cost of doing all the analytics on the nutrient samples (for research). But I know that's something that the group is talking about down the road. So, what do you think are the next steps for this work? Specifically looking at nitrogen in particular, what do you think is the next direction that is needed to be able to come up with a robust but practical sensor for use in vineyards?

0:12:08 Chenchen Kang

Yeah, actually I think in this research we have an original goal like we want to develop some sensors but while we really work on it, we find some more new questions and challenges. So, we have to address those challenges first before we can finally achieve our original goal. One of the biggest challenges is the reflectance. It's affected by many factors. Not only the nitrogen levels and because I also worked in another project, it's related to a water stress and the results from that project is reflectance is also highly related to the water stress of the grapevine. So, we have to address these challenges. For example, when we look at the correlation between nitrogen content and spectral reflectance, we collect data from morning to noon during this short period, the water stress tells us have already changed that dramatically. So, if reflectance is altered by the water status, and we do not have a large number of samples, so it's quite a challenging task to extract a consistent relation between the reflectance and the nitrogen. So, this can make it make it more difficult to find a few wavelengths to inform about nitrogen levels.

So, this year the team in Washington state also measured the leaf water potential from the same vines at the same time [as nutrient sampling and scanning] because in an open field with spatial variation, we cannot finish the data collection in a very short time window, and it's hard to make all vines at the same water status. But we are measuring it, so we have the data. If we have enough data I believe some machine learning techniques can find the pattern of this dataset and decouple the multifactor and find out the relationship among reflectance, nitrogen levels and the water status. I think this is something new for this year's data collection. They also set a shaded artificial light conditions using CO₂ to create a more uniform lighting condition to collect data in such a condition.

0:14:38 Patty Skinkis

Yeah that's very interesting--the water and the nutrient component because you know nitrogen or any other nutrient is not taken up unless we have enough water to supply to the grapevine. But, that's a really interesting component. You're talking about how the canopy was responding and the reflectance in particular to the water stress. I'm glad to see that's being added to this year's scans. What measure of water status are you using to determine or ground truth your water status levels? Is it with a pressure chamber? Okay, yeah, so it's taking leaf or stem water potential--is it probably leaf water potential? Okay, yeah, and that many of the listeners will be able to relate to at least in irrigated areas of the world.

Leaf water potentials are a very standard practice, with a pressure chamber, to determine water status especially during midday is very common. So that’s great to see added into the modeling. Do you have an idea at this point what the outcomes would be if you did have water status? Which one do you think will be influencing reflectance more, or is it hard to determine the impact of nutrients or water?

0:15:50 Chenchen Kang

From my experience, I think water is more correlated to the reflectance. But if we can figure out the relationship between water and the reflectance, then we can remove the impact of this factor and figure out the relationship between nitrogen and reflectance. It's multifactor. If we already know how one factor affects, then we can figure out the other factors.

0:16:23 Patty Skinkis

Yeah, absolutely! In your sensor arrays is there any other factor being included such as size of canopy or size of leaf?

0:16:33 Chenchen Kang

Yeah, I think this year the team is also collecting the canopy light interception data and they are also capturing the RGB, it’s a 3-D camera that takes 3-D images using those information they can get all the information about canopy size and canopy volume. They will also collect the 3-D image in the winter to gather the information about the lengths of the shoot and the total volume of the shoots. Yeah, we believe that information is also related to the nitrogen or potassium content of the entire vine.

0:17:20 Patty Skinkis

Absolutely yeah, so you're capturing growth, size or volume, water status and nutrition, that seems pretty nicely inclusive of what's going to be influencing nutrient status. I think the one additional thing that would be really neat to look into the model--and that’s not maybe something not sensed at that initial time but just giving the listener an idea of how we might use this information-- is also pulling in climate data that we can pull from different climate sources. We wouldn't necessarily need to have a weather station on site to interpret that as well, all of those together.

0:17:59 Chenchen Kang

Luckily, we have all the various data from the Ag Weather station especially in Washington State. I'm not sure about Oregon, but in Washington we have all the varieties recorded.

0:18:13 Patty Skinkis

Yeah, we can pull any data we need from the PRISM group no matter where it is in the US. So, that should be easy to do and something we could look to in the future. Is there anything else from the water stress work that you did that you'd like to tell us about?

0:18:32 Chenchen Kang

Yeah, since like I mentioned water status. It's like highly related to the reflectance. So, the accuracy using hyper background image and to estimate the water status it's already very high. Yeah, but we want to move it farther, we want to get higher accuracy and because the leaf angle also is a factor for reflectance. So in that project, I also extract a little leaf angle information from a 3-D camera then combine this leaf orientation information with spectral reflectance to estimate the leaf water status. It got a higher accuracy than using only reflectance data.

0:19:21 Patty Skinkis

That's really important because yeah, that’s not common [data]. You know--the simpler the model, the less accurate you were going to get, but it's good to identify those things that are causing the challenges or influencing them. Let's say, leaf angle-- I think anybody who works in vineyards appreciates that leaf angle is not uniform so that that makes a big difference. It’s interesting to hear that the water stress model is maybe more advanced at this point in terms of field ready. Would you say that's true?

0:19:53 Chenchen Kang

Yeah, but also the same challenges [apply], it's like as in nutrient detection. Also, when we try to use fewer wavelengths, the accuracy decreased. But still, I think it’s still more advanced than the nutrient detection [models].

0:20:10 Patty Skinkis

OK, so what do you think are the broader applications of this work in particular?

0:20:15 Chenchen Kang

Yeah, so actually our automated goal is to develop a few sensors or cameras that each camera only has a few bands that can be used to detect the nitrogen, potassium, and phosphorus, respectively. Since the similar approach is already applicable in other crops, for example code, but as a researcher our job is to fight through the unique challenges or difficulties in vineyards and [determine] the researchable questions to address these challenges. So, we can fight the suitable requirements or range in what kind of conditions this approach will work and also find limitations. If we can provide this scientific understanding to those sensor companies, they can do a better job to make the sensor available to growers.

0:21:17 Patty Skinkis

Yeah, I agree, as a research team we come up with the basic concept then the framework and commercialization are out of the scope of researchers. You're absolutely right, and I think that's good for the listeners to hear that. Other companies have to pick this up and take it to that next level that we won't be promising a field-ready application by the end of our 4-year funded project. But, I'm glad you mentioned that.

So, you're in a new position at Penn State. Do you have any plans for what you are to be working on or new areas of research that you're going to do or crops that you'll work on? If you want to let us know a little bit about that…

0:21:52 Chenchen Kang

Yeah, since we are in area of the agriculture automation, I'm happy to try some new things because in Washington State I worked more on the sensing technologies. Here I will start working on a robotic, like robotics things for the crop load management like robotic flower thinning, green fruit thinning. I'm excited about like these new things I never did.

0:22:23 Patty Skinkis

You mentioned earlier, before we started, that you're going to be working more with tree fruits rather than with grapes in Pennsylvania is that correct?

0:22:31 Chenchen Kang

Yeah, I think mainly on apples, and I also will also work on some projects in the controlled environment—in the greenhouse using some “internet of things” technologies to monitor the growth of vegetables. Yeah, it's quite different compared to what I did in Washington State University.

0:22:53 Patty Skinkis

Very cool. That's exciting. Now you mentioned “internet of things.” Can you tell us just briefly what that is and how it's used?

0:23:01 Chenchen Kang

The “internet of things” it's like we use different type of sensors to gather data from different aspect. For example, soil moisture, pH, and also weather data is part of “internet of things,” and we can use this data to analyze the growth, the environment, and to make a decision to manage the production of either vegetables or fruits. Yeah, just using sensors --get data, analyze data, and make a decision.

0:23:43 Patty Skinkis

Okay, so, kind of like a decision support tool?

0:23:46 Chenchen Kang

Yeah, right.

0:23:47 Patty Skinkis

Great. So, how did you get started in agricultural engineering? Can you tell us a little bit about the path that you took, and you've been an agricultural engineer from your bachelor's, your master's, and now your PhD. So, if you can just tell us a little bit about that.

0:24:92 Chenchen Kang

I started agricultural engineering as a student for the past eleven years, yeah up to last month. But originally when I selected an agriculture engineering program I didn't know much about it. And I was not sure if I would like it. I went to China Agriculture University not because I liked agriculture just because it's a good school. But most of people don't like agriculture in China, and then I could go to a good school if I chose China Agriculture University. I wanted to study engineering because I thought I would be good at it because I was good in math and physics in high school. Since I select the China Agriculture University, I must take advantage to all those best agricultural programs and engineering in agriculture university, that's agriculture engineering. Before I really started my study there I had no idea about agriculture engineering or agriculture automation. I only knew some tractors or maybe some combine harvesters before that. But once I learned more and more, I felt I liked it, and we can contribute a lot to apply the new technologies to agricultural production to make the people's farming much easier. Especially in China-- make it less labor intensive and improve the product productivity and the efficiency.

0:25:36 Patty Skinkis

Very good. I'm glad you enjoy agriculture because it’s fun, but it's always interesting to hear if people come from an agriculture background or not before they get started. In your case, did you have experience? I'm guessing you had no experience in agriculture before you started with ag engineering?

0:25:52 Chenchen Kang

I had some, yeah, but like those experience gave me a impression of the agriculture. It's also like labor intensive and low pay. I didn't think I’d be working in agriculture for such a long time.

0:26;09 Patty Skinkis

Well that's good. You could see that there's different parts of the production and the research that goes into it. So next, the fun question. If you could research anything in the world, what would it be?

0:26:24 Chenchen Kang

Yeah, so if it's anything. It would not be within science technology. So, I'm pretty interested in…So, I was originally from a small village in China. So, my family or my ancestors they have been living in that very small village for hundreds of years, so I'm quite interested about the history of my family and of that small village. So, when I was a kid, I would ask my grandfather a lot of questions about what was it like fifty years ago or like sixty years ago? Where are those buildings around at those time or not? Yeah, I asked my grandfather a lot of questions. Yeah, I'm also interested in the history of my family because we have a family tree. Yeah, we can see all the names of the ancestors I know it's hard to get that information to research. But I'm so interested.

0:27:30 Patty Skinkis

That’s very cool. Yeah, that's interesting. I think It's hard when you know you don't have those deeper connections especially if you come from a lot of immigrant backgrounds. Even in the U.S., you know I know my family came to the U.S. at some point I have no idea exactly when and if you don't ask those questions that information kind of gets lost. So, I think that's very interesting and very cool to have that much back history on your family.

Well, thank you so much for joining us today, Chenchen. Thanks for sharing your research and giving us some information about the research. For those of you listening, you can learn more about the HiRes Vineyard Nutrition Project from our website https://highresvineyardnutrition.com, and we're also on Instagram, LinkedIn and Twitter.