Season 3, Episode 2 | Sensor Snapshot from Satellites to Drones (in English)

00:00 Patty Skinkis

This is the HiRes Vineyard Nutrition Podcast Series, devoted to helping the grape and wine industry understand more about how to monitor and manage vineyard health through grapevine nutrition research. I am your host, Dr. Patty Skinkis, Professor and Viticulture Extension Specialist at Oregon State University.

00:22.00 Patty Skinkis

One of the main focuses of the High-Res Vineyard Nutrition Project has been precision tools for nutrient sensing. The high-resolution title denotes the team's efforts to increase sampling data points that describe an area of land to a greater extent than our standard methods. These methods include visual observations paired with a few tissue samples that we take at specific time points, mainly bloom and veraison. Of course, when we manage vineyard nutrition, we know in a perennial crop like vineyards that this evolution of nutrition is happening all the time, not just at bloom and veraison.

00:59.00 Patty Skinkis

So, in developing this project, we thought… what would it look like to do a better job at nutrient sampling… maybe with higher throughput and recognizing that vineyards are highly variable? We wanted to figure out how to better manage that variability and manage vineyard nutrition overall. And as we started to develop this project, even before it was funded, we recognized that people who know how to work with sensors and engineers, and data management and analytics were required to do this work.

01:31.00 Patty Skinkis

Viticulturists were the ones who came up with these ideas, but we have limitations. We aren't engineers. We aren't massive data analyzers, we look at plant data, but it's very different when you're looking at high resolution data with a lot of points coming from an area of land. So, we paired together with teams of viticulture researchers and engineering teams, across multiple institutions. And often researchers are known to work within their lane and find it difficult to think outside of the box and build these connections with teams in vastly different disciplines. However, our team was really lucky to have a viticulturist, Dr. Terry Bates, on the team who already had strong connections with engineers to tackle real vineyard production issues using sensors. So, we had a ready-made team from Rochester Institute of Technology to work with us and jumpstart our project in 2020.

02:25.00 Patty Skinkis

Now, in past podcasts, we did talk to Dr. Jan van Aardt from RIT and the group from UC Davis and WSU, or Washington State University, but we wanted to speak with the person who is doing a lot of in the field work on this project Dr. Rob Chancia. Rob is a postdoctoral researcher and member of the Digital Imaging and Remote Sensing Laboratory at Rochester Institute of Technology in New York State. He works closely with Dr. Jan van Aardt on numerous projects and his research is focusing on the spectral characteristics of the grapevine canopy as related to various research projects including vineyard nutrition, and he is responsible for all the field-based applications of the remote sensing experiments in New York State. So, we're glad to have you with us today, Rob, to talk about some of the work that you do.

03:17.00 Rob Chancia

Thanks, Patty. Yeah, I am glad to be here.

03:19.00 Patty Skinkis

So, can you give us an idea about the main questions that you address from the data acquisition standpoint in these vineyard trials?

03:29.00 Rob Chancia

The main question we are trying to address is how best we can first sense and then map nutrient status in vineyards. The biggest target is nitrogen. But from my perspective, we already know or knew even before the project began, how to do this at least as good as you can with the current sensor technology and those limitations. This is not a problem that is unique to vineyards at all. People have been trying to do this for decades in various crops, and we have a good idea about how to do it. So given those limitations that exist, which we can talk about a bit later. We're reaching the benchmarks that exist for other nitrogen status in crops that people do with remote sensing via drone or satellite. So what I've been working on for the most part on the project is looking at the suite of existing sensor technologies that exist, from the ground, on a drone, or on a satellite platform, and looking at their ability to do the sensing and the practicality of what is actually going to be implemented in mass…that a grower can get their hands on and use. So that is really sort of the end goal for me with this project is coming up with the framework for getting the imaging data into the hands of the grower, preferably in a platform that integrates seamlessly and is approachable to them. Like “my Efficient Vineyard”, that you have probably heard about in past podcasts.

04:51.00 Patty Skinkis

Yes, absolutely. So, with this work, I know we are looking at nitrogen, phosphorus, and potassium primarily…and some work is also being done in some of the macro/micronutrients, and I know the work that you are working with in the Lake Erie region is largely also looking at potassium. I forget if you are also looking at phosphorus?... Are these technologies able to be used as one single technology for all the nutrients we are interested in, or do you see them being somewhat different?

05:24.00 Rob Chancia

I should just go back to the beginning. We are sensing all macro and micronutrients; I think a little over a dozen in total across the sites that we visited anyway, that Terry [Bates] gathers [data] and Justine Vanden Heuvel gathers [data] in the Finger Lakes region. That is a lot of data, right? Just in the nutrient data alone. We tend to focus on the macronutrients and a lot of those are correlated with each other, to some degree. And separating that out, I think, you know, as a first thing we're sensing, there's something wrong maybe with the vine and not necessarily knowing what that is. And we have been able to build up these correlations with our ground truth dataset for the various nutrients, but just having the data on its own, you might not know. You have to then follow up. What we can do is get you a map that you could then go and look at an area that looks like something is going on. And you would sample there, and find out what is really going on.

06:17.00 Patty Skinkis

Sure, yeah, that's a good clarification point on the trials that we have… there's some trials that nutrients are being modified, and some where it's just being explored, and we see differences. And that is a good point that it is not just about scanning and getting a simple answer. It is about those different areas that you can focus on. So, you talked about different modalities to get the data. So, ground-based, aerial-based, or satellite-based. Can you talk a little bit about the sensors that would be used in those different applications?

06:55.00 Rob Chancia

Sure. So, when I talk about the different platforms like that, they are all using some kind of spectral sensing. I mean, we have other types of sensors on our drone that are getting different information like thermal or structural information from a LiDAR sensor, but the comparison is more the spectral data. So, visible and near infrared, for the most part. We have a shortwave infrared system on one of our drones as well, but that data is actually really hard to work with and very poor and inconsistent.

07:25.00 Patty Skinkis

And which of the applications do you think is the most promising at this point for doing the best job at giving us an idea of our nutrient status?

07:35.00 Rob Chancia

To compare all the visual and near infrared range sensing technologies on all platforms, they are all pretty comparable to some extent. It just comes down to cost and temporal resolution and ease of using the data. Like, if we are coming to a practical solution, that's going to be useful to a grower, yes, you could fly our expensive drone with the expensive sensor on it and map the vineyard with nutrition, but that's not practical, right? Over the course of this project, even inexpensive drones about $5,000 have come to market that have the right bands to do this kind of vegetation monitoring and people can use them. The problem then becomes managing that data. It is not obvious necessarily. From the satellite perspective, you have much coarser resolution data in terms of spatial resolution, but you get high temporal frequency, right? Like this data is being obtained whether you want it to be taken or not. You’ve just got to access it and it's much easier to work with that data because it's fewer pixels. Stuff is mixed in there, there's issues obviously, right? Like you are averaging in the interrow and soil background and all that kind of stuff. So, you're not getting a pure signal of the vine necessarily, but it gives you a good look at the variability within your vineyard.

And, you know your vineyard better than anyone else, right? You know where the soil is different, or where there is low topography or something, or you know…the grass doesn't grow, or something like that. So, you can identify, “oh, this is strange because of this reason, and then this other area is strange for some other reason. We've got to go look there.” That's what the map is telling us.

So, I see, two solutions: one being satellite, because that data is really easy to get. We just need to integrate it into MyEV and you get a nice temporal look at your vineyard variability over time, right? And you'll get used to looking at that and seeing what you expect to see and then things that you aren't expecting to see and that you need to address. And then for more advanced tech savvy individuals who want to spend some money, the drone option where the market already has these drones, right? You can go buy one and you’ve just got to learn how to use it. And they're really easy to use. The hard part is the data.

It is a lot of pixels, right? So, you've got to take that and extract your vines, upload that to something like myEV. So, we're working on the pipeline for that. So, they can fly their mission, process the data up to a certain point using existing software that is becoming more and more cheaply available. And then hand us the map that they get, and we'll get it ready for myEV. Be that through some other web, you know, upload platform or something like that.

10:10.00 Patty Skinkis

So, you're talking about the data management piece. Now this is a piece that I think a lot of growers and producers really don't understand. They hear about drones then they think about the new technology and how easy things are going to be. And I want you to talk a little bit about that data management because I think people think “well, you know, then we'll just, have somebody young who knows how to use computers who will do this.” And I think that's a little short-sighted. So, can you talk a little bit more about what that data management entails?

10:42.00 Rob Chancia

I agree. We want to make it as easy as possible. And I don't think many of these things are at a stage where it's as easy as possible yet. People might like to have you think that, but I think there's some work we need to do. I think the low hanging fruit is the satellite data because that exists in other aspects already. We just need to have a nice tie-in to myEV where they can draw their vineyard block and say, download a satellite map. It imports like a vegetation index that's relevant to nitrogen and they could see that. For the drone, it's much more difficult. They take a bunch of frames over their vineyard…actually, getting the data isn't really that hard, the learning curve isn't that difficult. But processing the data is not trivial at all. Even beyond just making a map, extracting information explicitly from the vine pixels is a really non-trivial thing that we're trying to build a procedure for that we can put online, and they don't have to do that work. They just upload their map with all those pixels, and we do the hard work.

11:45.00 Patty Skinkis

Rob has been talking about MyEV, which is My Efficient Vineyard, a free platform that you can use to map your vineyard and integrate some of these technologies into it. So that's what he's been working on with Dr. Terry Bates and Nick Gunner as they develop that further. We've got an automatic tie-in from the Hi-Res Vineyard Nutrition Project to build that platform out more. So, I know we've done a couple trainings on this. If you're interested in learning more about that, there will be some links with MyEV and on our website https://highresvineyardnutrition.com/. But of course, this same technology could be integrated into other platforms that growers might use in precision agriculture, correct?

12:29.00 Rob Chancia

Yeah, for sure. I think that's sort of the longer lasting impact of this project that we want to contribute to is working that imaging data into MyEV in a useful way that can be used for any crop.

12:42.00 Patty Skinkis

Yeah, so it's that data analytics interpretation, how to do it, and linking that with the nutrition aspects. So, what do you think … what are your thoughts about the biggest finding that you've had from working on this project? What would you say is the “aha” moment you've had in working with this data?

13:00.00 Rob Chancia

Yeah, maybe not for nutrition necessarily, but we found that with the drone data… and it doesn't even have to be a special sensor at all, It could be a really simple, cheap drone for a few hundred bucks. You can map your vineyard structurally and assess pruning weight without having to go measure it, which you know, I'm not a viticulturist, I didn't understand… I'm learning a lot every time we meet and what’s valuable to you guys. And Terry's very excited to not have to measure pruning weight anymore. We just got to come up with a nice little model to assess the canopy volume using drone data for him and then he doesn't have to do this anymore. So that's a good finding.

I know they use pruning weight along with yield to come up with the crop load and that's used in their nutrient modeling as well, but I can't speak to that as much.

13:51.00 Patty Skinkis

Well, I know that I can speak to the pruning weight part. Now, pruning weights aren't that difficult to get from a standpoint of time of year, we do it in the dormant period. But I think the bigger value is that pruning weights have always been used as a proxy for vine size. And if you can fly a drone or use some sort of tool, to get canopy size in the vineyard in real time, that's much more helpful. I mean, pruning weights will still be good, but it's our best way of quantifying canopy size because the alternative is measuring shoots and leaves and that's a killer, nobody wants to do that. So having some way to come up with canopy size is a big part of it that we often times don't have. And in production data, I think most growers have nothing other than qualitative assessment of, “oh yeah, our canopies were big in this year or not.” So, I think that's a real valuable piece obviously, and nutrition relates to that, better nutrition, higher vigor vines, larger canopies. So that's interesting.

What do you think are the next steps in the process? You've already talked a little bit about this with what you think is the most user-friendly, most practical applications. What do you think would be the next step to really confirm some of that?

15:15.00 Rob Chancia

Yeah, so this year [2024] Terry is deploying the variable rate fertilization based on our map in one of his blocks. So, I guess seeing if that works out… putting it to the test, right? Seeing if those vines that we thought needed a little extra nitrogen if it bears out in the sampling that we do.

15:37.00 Patty Skinkis

And that was all based on an integrated approach, as I understand, correct? Like canopy size… basically data layers, it wasn't just nutrition. Is that correct?

15:48.00 Rob Chancia

Yeah, I think he's measuring canopy NDVI two times a year. He's measuring the soils electric conductivity, last year's yield, our map of what we think the nitrogen content is from the drone and combining all those layers into one management map to zone out his vineyard block.

16:10.00 Patty Skinkis

Yeah, so I think it's great to hear that it is integrated. I think that's another thing I fear when people hear there’s a sensor. It's an easy button to say, “yes, this is targeting this one thing,” but really it all comes down to the big picture.

16:25.00 Rob Chancia

People should understand that coming up with that general model of nutrition is very difficult. Like we cannot do that from remote sensing alone. Every vineyard is different. There's different soil, there's different climate, there's different varieties. You all trellis them in different ways and do different interrow management. And like, that isn't built into the model necessarily. Or at least we don't have all that data yet to deal with and integrate all these different things that would matter for the nutrition output. So, it's really hard to come up with a general model just coming from the remote sensing aspect. So, you really need to take all these other layers and context about your vineyard into consideration when you're coming up with a management plan. So, I think that's what Terry's trying to showcase with MyEV to bring in the data that you have and what you understand and come up with your management plan from that.

17:14.00 Patty Skinkis

So, with the model development, can you speak to what goes into that?

17:20.00 Rob Chancia

I don't want to snow people with machine learning type stuff. We are doing a little bit of that, but it's not that crazy. We're tending towards the multi-spectral solutions, which are just a few visible and near infrared bands and just combining those. Like each of those will have some different impact on the results at the end of the day or turning those into a vegetation index or two and using that in the model. We have models for bloom, models for veraison, maybe a model that includes all of the data… taking into account like this image of the vine was from this date and it had this nutrition. Maybe we keep track of growing degree days that correspond to you know… when that was sampled or precipitation data, things like that. And those tend to matter a lot right? Like if you're going to have a general model, you need to include all this information. But I think that modeling is very tuned to our specific data sets, which is not terribly useful in the grand sense of things. I think the best thing that we can really do is get what is effectively a vegetation index map in the hands of growers that they can track over time and look for things that seem odd in their vineyard and, you know, test nutrition where they think it needs to be tested and not waste the time on some other area where it seems like everything's okay.

18:42.00 Patty Skinkis

So, a targeted approach to nutrient testing. And I think that's really valuable because we have in the meantime, always suggested to growers that if you're going to sample, don't just get one sample and get them from healthy and problematic vines and look at those.

18:58.00 Rob Chancia

Yes, you have to get the distribution right.

19:02.00 Patty Skinkis

That's right. And some growers honestly only sample when they think there's a problem. So, that's I think, also a good way of cutting costs and being more hypothesis testing on a farmer level… about what might be going on. And I think that's a good perspective that you bring with the tools that it just helps you select better and in the end get better data because you're not just diluting the sample by guessing where to sample.

So, what do you think are the logicall next steps for application? For example, how far out is it for there to be real recommendations on how to apply this. You mentioned this a little bit with getting a model integrated into MyEV. How long do you think that a model will take to develop or one that could be used in other kind of precision ag platforms?

20:02.00 Rob Chancia

So, from my end, I don't know that we need a model. I just want to get a spatial variability map of their vineyard canopy integrated into MyEV, which I think is really achievable by the end of the project here and something that we're working on. And with that, they can track, if it's publicly available data, they can get… as long as it's not cloudy, they can get near weekly updates on how their vineyard is doing from a satellite perspective. And they need to look at that and the context with what they actually see in their vineyard, and see what it's explaining to them. If there's areas that they're like, “oh, this is just really low and not performing in terms of what we see in terms of vine vigor from the satellite,” is that because of some contextual thing that you can't tell from the satellite? Or is it because, you know, nutrition could be low? I don't think we're going to get a definitive recommends on how much fertilizer to apply in this area of the vineyard from just from one set of imaging data. I think that's unrealistic, unfortunately. But I'm on more of the cynical end.

21:04.00 Patty Skinkis

Yeah, well I think that’s true from a…. I think that the viticulturists on the team looked at it that way. You know, it's one request to say, come up with this sensor. It's another request to know what’s the reality. And it's very difficult, the plants only show symptoms of something, whether it's disease, water stress, nutrient stress in so many ways. And so, it's hard to be very definitive about something that might not be in a very high amount, like some of the nutrients. I think that's a good perspective to share with the listeners about what can come of this work, and precision agriculture in my mind doesn't replace people. We still need people and expertise to be able to know how to use the tools and to make decisions from them rather than just thinking okay, well now we've got this sensor, you just made our life easier. In some places, I think it makes it a little bit more complicated. But as we explore all of this, we learn more, we can understand more scientifically, but also come up with some guidelines for growers.

And I guess that takes me to my next question or comment, and that is I've seen you present a very nice table of all of the technologies, and I love it. In fact, it's one of the things that I would love to be an output from this project. And you verbally spoke about this a little bit, but I really appreciate that from a standpoint of the reality, because sometimes there can be over-promising in academia… about what science can do, but I really appreciated the table that you talked about with respect to what's available for free and what's easy to use. What's your thought about that table in particular, and sharing that with the growers?

23:09.00 Rob Chancia

Yeah, everyone should be aware that I'm talking about this from the most practical perspective possible. What we have now that can address this problem, and I think there are ways you can maybe get a better look at things like nitrogen, but are they really practical, right? The big confounding thing with nitrogen is that most of the information that would be more correlated with it lives in the shortwave infrared part of the spectrum where we don't really have good remote sensing sensor technology to sample. And that problem’s existed for three decades, like since people started trying to do this, it hasn't changed… and probably won't change soon. You could do this at a very pinpoint level with a field spectrometer, which some of the people on the team are doing to get clean data of that region and get good relationships with nitrogen that are better than just chlorophyll relationships but… you know, it comes at a huge cost for labor and for the actual instrumentation and the complexity of the model that you're working with. So, I don't see it as a realistic aspect.

So that's your imaging spectrometers that cost hundreds of thousands of dollars or your field point spectrometer that's, you know, maybe like $50,000 to $60,000. And, you know, it's… very complex data to deal with. On the easy end, you have maybe more tech savvy folks who are ambitious, could get a drone for a few thousand dollars and subscribe to a processing suite for a couple hundred bucks and gather their own data and do the processing themselves. But there's a learning curve there, right? Like it's… they're not going to get that immediately. It is getting better all the time. Even over the course of this project and the four years that we've been working on this, new things have come out… or things are becoming free. It's becoming easier to do. But really, I think the cheapest options are either going to be commercial satellite data, which offers now near daily imagery. So, you're just limited by clouds, and three meters scale can be quite expensive, but if you have a large group or network that is all paying in, it becomes cost effective. This is part of some association dues or something, you could offer this potentially. Or, you know, publicly available satellite data, which is maybe at like the 10 meter resolution. But it's lower data volume, which is great, so it's great. It could just be online platforms like myEV without issue and it's free. Maybe it's only once every week or every two weeks, depending on weather, but that’s going to be the optimal solution in the near term. It's just a matter of getting that data into the hands of the growers and, you know, making little tutorials on the myEV website that show you how easy it is to work with. So that's something we hope to accomplish by the end of the project here on the practical end.

26:07.00 Patty Skinkis

Sure.

26:08.00 Rob Chancia

There's still a ton of stuff to do with crop models and, you know, the most exotic, expensive data, but is that really the near term solution? I just don't feel like it is.

26:23.00 Patty Skinkis

Yeah, that's always the dilemma. You know, we can throw a lot of research effort into some of these things. But if it's not practical and usable… and also not just practical from a grower standpoint, but also practical from a plant response situation, it's hard to make something just appear without the confines of the system you're working in. So, I think that I've said this a number of times within our research team, it's like modifying expectations. And I think that's of our research team as well as of industry for what the technologies can really do. I think we're all learning, we're learning as viticulturists, we're learning as… you know, as the sensor team, et cetera, on how to make these things work within the confines of budgets and human resources and just the plant system itself.

27:14.00 Rob Chancia

I forgot to mention this aspect: So, we're talking about spectral sensing, but there are alternative ways we can get at this that people are actively working on with interrow sensors that are just standard high res RGB cameras. There are teams at Cornell that have an autonomous robot system, which sure is super expensive, but you know, that stuff becomes cheaper and cheaper over time. And a group at Carnegie Mellon that's looking at counting grape clusters and things like that. I think the Cornell group is looking at disease, but we could use the same kind of imagery to quantify visible symptoms of these various nutrient deficiencies. Like you talked earlier about separating out potassium and phosphorus and nutrient deficiencies. We know what those visual symptoms look like for these different grape varieties. So, if we can teach a computer to take that image, detect all the orange or whatever is happening on the leaves, and quantify on a vine level and map spatially where that is. I think that's promising. It's not, you know, predictive of where nutrient deficiency is going to be bad, but it's sort of a “Let's assess spatially the extent of symptoms.” We're not actively tackling that in this project, but it could be something in the future.

28:35.00 Patty Skinkis

I think that's a good point that there's these other uses. I think they all come with the same challenges, though. What do we do with the data and how do we manage all of it and interpret it? So, it's a big question that I think the Carnegie Mellon group, for example, with the crop estimation, that's gone into the Bloomfield group, and they're still working towards that effort, but it's taken years. It doesn't happen overnight. I know our co-PI Manoj Karkee has been working on it too and trying to come up with less expensive options and that's also where, you know as researchers we want to be very definitive, but often times I think industry doesn't have that same expectation of having such definitive information. So, it's how to find that middle ground, and that's our challenge in these projects.

29:41.00 Rob Chancia

I think that's a product of all these new technologies coming out that are doing really miraculous stuff, but with easier questions. I can look at an image and if I can physically with my eyes identify something, you know, maybe a computer can, but I can't look at this vine and the vine next to it and tell you what the nutrient levels are. So, it's really difficult to teach a computer to do that.

30:04.00 Patty Skinkis

Yeah, something internal versus something external.

30:08.00 Rob Chancia

Yeah, exactly.

30:09.00 Patty Skinkis

Canopy size and number of clusters versus internal composition. Yeah, it's a challenge. I know the team has about another year or so in the project, and I know you're actively working on projects this summer (2024). But one of the questions I have for you is a fun question. How did you make your way into the career path you're in?

30:33.00 Rob Chancia

Yeah, so it's a long path. Coming out of high school, I went to school for construction management. It's totally different than this… and worked in that for a little bit and decided I wanted to go back to school. I went for my degree in physics and followed that all the way through grad school. Then for a number of circumstances, I decided I wanted to move to this area in Rochester (NY) for family reasons and was looking for things physics related. I discovered this imaging science department, which is this nice mix the remote sensing aspect is great because you can do field work and you can work on a computer. It's like… both of the things I like doing to spend your time in different ways. So, I joined the department here and I get sent all over the world to collect field data for various projects. And then in the winter I'm in the office processing data, and I guess that's how I find myself here. I've really enjoyed working with the viticulture team, Terry and Manushi and Justine. It's been a lot of fun, working out in the field and doing trunk collects.

31:33.00 Patty Skinkis

So, can you tell us a little bit about the other projects that you've worked on?

31:37.00 Rob Chancia

Yeah. We were collaborating with some folks at the forest service looking at mangroves. So… we deploy a LIDAR sensor in the mangrove forest to measure actually… a creation of the ground material and seeing if the surface in the mangrove forest is keeping up with the rising sea levels. So, we're just basically developing new technology for them to be able to make this measurement… that's a little bit quicker for them to do than their previous method. And then… we've been to South Africa, we work with ecologists, we fly in our drones over these unique biological regions that have a whole bunch of different species and trying to quantify biodiversity in this endangered region. We do all sorts of things with forestry, assessing biomass and stuff like that. Anything that gets me out in the field, I'll sign up for.

32:27.00 Patty Skinkis

So, it's a nice balance of field and data analytics. Do you work on other agricultural crops or just grapes?

32:35.00 Rob Chancia

Mainly grapes, but one of our grad students here is working on a project for Beet Yield with a company called Love Beets. So, I've been a little bit involved with that project, helping them with their field collects and processing some data.

32:48.00 Patty Skinkis

Very cool. Well, thanks for joining us today, Rob.

If you are interested in learning more about the HiRes Vineyard Nutrition Project, you can visit us on our website, hiresvineyardnutrition.com, and social media platforms:

Instagram: https://www.instagram.com/hi_res_vineyardnutrition/

X: https://x.com/HiresVineyard?mx=2

LinkedIn: https://www.linkedin.com/company/hires-vineyard-nutrition/

If you are enjoying the podcast, please complete a brief online survey.

This podcast is funded through the National Institute of Food and Agriculture’s (NIFA) Specialty Crop Research Initiative Coordinated Agricultural Projects (CAP) grant. Project Award Number: 2020-51181-32159.