Standards and data exchange in agriculture

All of our sciences and fields of work abound with norms, standards, and exchange formats. Agriculture is no exception to this (you will find in the appendix a list of organizations and institutions closely or remotely involved in standardization in agriculture). We need to exchange, collaborate, share data and information, so we need to speak a common language. Whether for exchanges between machine tools, machine software, software-software, machine machines, or simply for informal discussions, we will see that agricultural standards and the exchange formats that have been put in place have changed the situation in the field of agricultural digital technology. Some will say that this dynamic and the inertia of agricultural players is very slow; others will say that great collaborations and great efforts have been made.

The subject is extremely vast. So we will not be able to deal with everything here. To make it clear, we will not talk about legal issues (GDPR – General Data Protection Register – and the like), ownership, consent, or data security. We will focus much more on aspects of standardization – therefore more of a technical subject – even though it must be admitted that all these subjects are interconnected to a greater or lesser extent.

As usual, for the readers of the blog, this article is the result of telephone interviews with actors in the sector (whose names you will find at the end of the article) whom I thank for the time they were able to give me. Several readings of scientific articles, forums, and blogs will have allowed me to complete the feedback of interviews.

Welcome to the world of acronyms, semantics, and agricultural interoperability.

Short introduction


Where to start… Perhaps already by reminding us that words are important. Of all the acronyms and abbreviations that will be presented in this article, not all relate to the same thing. In particular, we will find communication protocols, standards/standards/exchange formats, standardization institutes/organizations, data referentials and dictionaries. Since not all members of this large family are really independent of each other, one can quickly get dizzy and not find one’s little ones there.
To get into the subject more simply, I suggest you start by distinguishing between two different large sectors: mobile equipment and fixed equipment:

In the mobile equipment,

  • we can find tractors, machines, and attached/drawn implements (seeder, sprayer, plough…) – and we can roughly summarize the exchanges within this equipment at ISOBUS (we will go a little further than that).

In the fixed equipment, a separation could again be made between

  • fixed equipment in the strict sense (milking robots, aumatic feeder or stationary machines) which do not have an ISOBUS equivalent, but where this is not so important because these machines are not intended to be attached to anything. We will not focus on this in this article,

  • the associated software tools (field management, accounting…) and cloud on which many different data flows can transit. Here, however, there is quite a lot to be said,

Data exchanges in mobile agricultural equipment


Some background information


ISO is the International Organization for Standardization. It is very unlikely that you’ve never seen ISO standards before. They’re often talked about in terms of quality, management, safety, or health issues, and this in many different fields (and you have to admit that it’s not particularly exciting, it’s often synonymous with procedures and administration). In the mobile agricultural equipment industry, there has been a fairly well-respected ISO standard – ISO 11783 – otherwise known as the more classic ISOBUS standard. Why was this standard implemented? At the beginning, simply to be able to standardize, harmonize, and simplify data exchanges between a machine and a tool towed/attached to this machine. The objective is that a machine and an implement should be able to communicate with each other, so as to be able to trace what the implement is doing (the current crop operation) to the tractor terminal and to give instructions to the implement to carry out its crop operation.

But in order for a machine and an implement to communicate with each other, there is no need for a norm or standard such as ISOBUS, right? Where it becomes more complicated is when several implements can be attached to several different machines… This need for standardization was initially born in European countries because, among farmers here, machines and implements are not necessarily of the same brand. Some farmers may have as many as five or even more different brands on their farm. This problem was less present in other countries such as the United States, where certain manufacturers present such as John Deere or CNH could have a long range of agricultural equipment with an associated distribution network (so-called “full-liner” manufacturers). How then can we make sure not to have a multitude of exchange standards? Well, by creating a common standard! Easy, isn’t it? Not so easy… To choose a common standard, all the manufacturers first had to sit around the table and agree. Not easy when the stakes go beyond a simple technical issue (which was far from being simple). For example, some players were not necessarily interested in interoperability with other manufacturers for economic reasons or because they were not present on the market. Others were not necessarily interested in seeing the ISO standard arrive. For the small anecdote, at the time, Germany, which was secretary of the ISO group, had a competing ISOBUS standard at the German Institute for Standardization (DIN – Deutsches Institut für Normung). In 2008, after an initial, somewhat slow dynamic, several manufacturers decided to create the AEF – the Agricultural Industry Electronics Foundation – to ensure that this common standard could emerge under the best possible conditions. The AEF then took charge of the “Plug Fests”, the famous speed dating events organized every two years between manufacturers to test their machine compatibility, and companies began to get involved beyond ISO and communicate with each other. Initially imagined around 6-7 chapters, it took no less than 14 chapters to define this interoperability standard. ISOBUS was born.

ISOBUS


The ISOBUS standard is composed of several features that can be found on the right side of the nice blue AEF thumbnail (Figure 1). For the moment, only 6 features are clearly defined (UT, TECU….), and several others are to come. Before quickly going back to these features, which must be enabled on the consoles and tools, it must be understood that a device is ISOBUS compatible for features and associated versions, i.e. a hardware may be compatible on the UT feature, but not on the TC-GEO feature, and that compatibility is not necessarily ensured if the version of a device or hardware is changed (by buying a new hardware for example). On this aspect of versioning, special attention must be paid to backward compatibility, i.e. the fact that a new hardware will not necessarily be compatible with older hardware. In addition, for a machine and a tool to be compatible, it is necessary that both are ISOBUS certified, again for the desired functionality and ensuring proper versioning compatibility. ISOBUS certification is not free of charge and is usually obtained after successful participation in Plug-Fests and compliance with AEF protocols. Please note, however, that manufacturers do not necessarily certify their updates. In order to check the compatibility between devices, the AEF has made available an ISOBUS database (www.aef-isobus-database.org), also available as a mobile application. In other words, if there is one thing you should keep in mind, it is that it is not a general ISOBUS compatibility.

Figure 1. The ISOBUS thumbnail

ISOBUS operational functionalities :

  • UT: also called universal terminal functionality because it allows to operate a tool from any machine terminal.

  • TECU: to allow the tractor to share its data (e.g. power take-off regime or speed) with the attached implement. It is important to understand that the data exchange takes place from the tractor to the implement.

  • AUX-N: to control the implement towed or attached to the machine with an auxiliary control, e.g. a joystick.

  • TC-BAS: for the management of conventional tasks (crop operations) that are not geo-referenced.

  • TC-GEO: for the management of geo-referenced tasks (crop operations), especially for variable rate applications, unlike the TC-BAS functionality. This functionality therefore requires, not surprisingly, a variable rate map

  • TS-SC: for the management of tasks that may require automatic changes or section controls (as part of a spraying, seeding or other operation) and thus reduce overlaps between successive passes of machines. This functionality requires a plot outline for border management.

ISOBUS is not something engraved in marble, the standard is bound to evolve. Three functionalities are moreover in the course of improvement, and will come to supplement the six preceding functionalities:

  • TIM (Tractor Implement Management): Contrary to the TECU functionality, the TIM functionality allows bidirectional data exchange, i.e. both from the machine to the implement, but also from the implement to the machine, so that the implement can also control certain tractor functions such as its speed or engine speed. In terms of examples, a round baler, once the bale is ready, can stop the tractor and make it eject the bale; thus further simplifying the work of the tractor operator

  • LOG for recording values independently of the task performed by the machine

  • ISB to deactivate one of the ISOBUS-controlled tools when more than one tool is available, so that you can work with the desired tool.

Keep in mind that AEF is not just ISOBUS. Numerous working groups have been set up within this association (https://www.aef-online.org/fr/laef/equipes-de-projet.html#/Aproposde) with, in particular, actions around field management information systems (we will come back to this later), radio communication standards for exchanges between machines, safety issues, or connectivity management between tractor and implement cameras.

NB: I could not find clear figures on the quantity of ISOBUS-compatible French agricultural equipment. One of the interviewees estimated that around 15% of the fleet would be compatible (this figure should be considered with some tweezers)

CANbus but not just that


ISOBUS is still an exchange standard, it does not explain how the data flows between machines and tools. For that, the concept of “bus” must be introduced. A computer bus is an optical fiber or a copper cable (a kind of pipe), within which machine data from the tractor and the attached implement (engine speed, fuel consumption, implement lift, etc.) as well as management messages are transmitted. These machine data are managed by computers, also called “Electronic Control Units” or “ECUs”, which are in fact electronic boxes that link the computer bus and all the electronic components (sensors, actuators, regulation modules….) connected to the box. So here we find the link with the TECU functionality of the isobus (which should rather be read Tractor ECU). The big advantage of having implemented these computer buses is that it allows to have a single pipe to link a tractor and the electronic components of the attached implements. Imagine the huge amount of wiring that would have to be installed if you had to consider one wire for each type of information to be transmitted between the tractor and the implement. With this computer bus system, the point-to-point connection or linkage (one wire per piece of information) is therefore avoided and a more global approach called multiplexing (a single pipe) is preferred. With a multiplexing approach, the tractor terminal can therefore be considered as virtual (we often hear the term virtual terminal) in the sense that it is as if the terminal was specific to each implement attached to the machine and that it was able to control the implement.

Figure 2: Isobus and CAN Bus. Source: Western CUMA

For computer buses, there are quite a few different schools. Some manufacturers have a bus for the engine, a bus for the sprayer, etc… basically one bus for each type of action. Other manufacturers will have everything on the same bus. In this case, we will rather talk about CAN bus or CAN network (Controller Area Network), because they are the most used buses in the field of agricultural equipment. These different schools can be explained quite simply because some manufacturers do not produce all their equipment themselves. For example, a manufacturer may very well buy an engine from an engine manufacturer, and assemble this engine on a machine that already has its own computer bus. And if this manufacturer then adds a sprayer that has its own bus, that makes another new one! Depending on the manufacturer, these buses can be independent and/or communicate with each other. In case someone (a service provider for example) would like to connect a box to the tractor to retrieve machine data (Fieldview or Samsys box for instance), it should either simply connect to the tractor’s diagnostic or ISO plug (a box powerful enough to decode all the Isobus frames would be needed), or it could connect directly to the CAN bus or to several different buses if the machine’s buses are independent.

Within these computer buses, especially the CAN bus, the data transit according to certain protocols. The J1939 protocol is certainly the most standardized, and allows to retrieve a certain number of machine data. However, manufacturers still need to pass data on their bus with this protocol…. Some CAN buses are indeed proprietary (although some manufacturers are ISOBUS certified), as are some protocols, which does not necessarily facilitate the exchange of data and all that is called telematics in general.

If we now turn our attention to cash, we realize that the user will still have to put his hand in his pocket if he wants to have a complete ISOBUS system. I am not saying that it is necessary to have all the features described above, or that it is not worth the effort (we are beginning to know the benefits of geo-positioning technologies in agriculture), but that it is nevertheless an element to be taken into account. It is indeed something to consider:

  • An ISOBUS-certified mounted/drawn implement

  • An ISOBUS console compatible with the mounted implement

  • Potentially a joystick to facilitate the piloting of the tool.

  • The installation of a GNSS antenna connected to the console and a subscription to a geo-positioning correction service (I invite you to reread the blog article on the subject), in particular to use finely the TC-GEO isobus functionalities and to perform variable rate applications.

  • The purchase of the TC-SC and TC-GEO functions to do section control and modulation

However, there are less expensive solutions. It is for example possible to retrofit, i.e. to equip an old implement or tractor with ISOBUS or in other words, to mount the isobus on existing equipment. This approach makes it possible to get out of the ISOBUS high-end equipment. Some manufacturers are already doing this (for example, Reichhardt’s Iso FIT solution). To develop a technical retrofitting solution, there is not much need for it! At least a tractor ECU and a 12-volt plug. Another solution for a tractor to specifically identify an implement, why not simply equip the implement with a Bluetooth tag? Then, of course, the objective here would not be to get precise machine data, but we would already have the means to make identification at very low cost (it would also be possible to add a tractorist tag to re-parameterize the tractor and the implement according to the driver). On the subject of geo-positioning, even if it is not completely the subject here, and to reduce costs even further, it is also possible to go as far as tinkering with your own agricultural self-guidance with the Canadian solution AgOpen GPS, and to use the open source RTK network Centipede, set up by INRAE. It should be noted that the developers of the Centipede network have made sure that it is as compatible as possible with the manufacturer’s hardware

Data exchanges in fixed agricultural equipment


So far, the focus has been on data exchange between machines and tools. There was already quite a lot to be said about this – in particular, we could talk about Isobus and computer buses – but when you take a step back, you realize that in the end there are not so many players around the table (all things considered). There are a few institutions (we’ve talked about the AEF) and manufacturers of agricultural equipment (tools, tractors, etc.), of course, but overall, that’s how we can summarize data exchanges. We could add the companies that offer boxes to connect to the various tractor/tool plugs to upload data, or more simply boxes attached to machines to upload machine usage data, but that doesn’t really increase the number of players involved (and the latter companies don’t necessarily have a say in these standards). When you start to take an interest in data exchanges on fixed agricultural equipment, particularly with farm management information systems and Cloud infrastructures, it starts to hurt… You’ll see, but the field of interoperability in fixed agricultural equipment is extremely broad, with a lot of organizations, initiatives, standards and formats. In the field of animal production, we’re going to touch the breeder, the veterinarian, the council, genetics, mass distribution and so on. In crop production, we will get closer to the farmer, advisors, the collection and storage organizations, agricultural chains, the consumer; in short, it’s starting to make a lot of people. And the more people there are, the more different needs and interests there are, and the more difficult it is to agree on standards and exchange formats (Figure 3).

Figure 3: A multitude of actors involved in data exchange in agriculture. Source: AgGateway.

We will focus here mainly on the exchanges that take place between machines, farm management information systems and other telematic solutions (boxes, web/cloud portal): machine-machine exchanges, machine-software, web-portal software, software-software.)

In the case of mobile agricultural equipment, we had seen that the need for harmonization was mainly due to the fact that many farms had equipment from different manufacturers. In the case of fixed agricultural equipment, in the end, we always have the same problems but there are others! Why would you want to communicate directly with a machine through a farm management information system or a manufacturer cloud? Quite simply to avoid having to work with USB keys that you can lose, that you have to plug into your tractor and your workstation, where you have to unzip folders (which is not easy for everyone) to access the files you want. On the other hand, you only have to look at the diversity of manufacturers, farm management information system and cloud software to be convinced that using agricultural standards could facilitate data exchanges and partnerships.

To get a clearer picture, let’s start by presenting the three main organizations in the field. The AEF’s leanings on fixed equipment are called AgGateway, AgroEDI Europe and AEF (yes, again, AEF). These three organizations, initially oriented towards North America for the first one and Europe for the second one, are public and are each composed of a certain number of member companies. AgGateway, AgroEDI Europe, and AEF are organized in different work groups, all of which are interested in the definition:

  • modeling standards or data models: for example, a crop operation must have a date, an area, a label…

  • dictionaries or data repositories, i.e. the content of the data: for example the crop code or standard name of winter wheat is ;

  • or exchange standards: this has already been discussed with mobile agricultural equipment, but here it is for fixed agricultural equipment.

AgGateway


AgGateway has nearly 200 member structures. For what interests us in this blog article, AgGateway’s projects are organized around three main themes of precision agriculture (Figure 4):

  • the management of crop operations such as seeding, fertilization or spraying (the SPADE1, SPADE2 and SPADE 3 projects)

  • telematics/telemetry applications very much linked to crop operations to make agricultural platforms exchange and communicate more easily (the WAVE and CART projects are nested in the SPADE 3 project)

  • and water management (the PAIL 1 and PAIL 2 projects)

It can also be seen that the SPADE projects also remain linked to the AEF Isobus standard (ISO-11783). For each theme, the working groups are in charge of preparing data models, repositories, or easy ways to access these repositories (via API for example; we will come back to this concept later).

It is interesting to see that many standardization bodies (public and private) are now members of AgGateway and participate in these work groups. For example, from 2001 to 2011, AgXML developed standards for grain and oilseed production processes. In 2012, its member companies have agreed to conduct activities within AgGateway as part of the CART project. The American Society of Agricultural and Biological Engineers (ASABE), which also publishes quite a few scientific articles on precision agriculture, is involved in the development of norms and standards within the PAIL1 and PAIL2 projects around water management. AgGateway develops some of these standards thanks to certain organizations that have joined it, but also uses some existing standards outside the AgGateway community (such as the one of GS1 or others).

Figure 4. AgGateway’s agricultural interoperability working groups. Source: AgGateway.

It was within the SPADE working groups that the “ADAPT” (Ag Data Application Programming Toolkit) initiative emerged, following an initial successful proof of concept initially called the “SPADE Conversion Toolbox”. ADAPT is an open-source solution set up to respond to interoperability problems between machines and electronic control units (ECUs), but also and above all between machine exchanges and farm management information system (FMIS), and exchanges between different FMIS. The ADAPT solution contains both a data model for the crop operations to be carried out on the fields, a set of proprietary plugins provided by the manufacturers to convert proprietary data from one format to another, a plugin manager to manage these proprietary plugins, an open-source ISO plugin to work with ISOXML data, and a set of open-source RESTful APIs to allow adding contextual information (these are ContextItems such as country, region, or specific information about the parcels, people) to the data. To complete on this notion of ContextItems, it should be understood that the ISOXML format, which will be presented a little later, manages universal data definitions such as yield per unit area or seeding density per unit area. The use of ContextItems allows you to go further by providing contextual information, which will no longer allow you to read a yield per unit area but tons per hectare or bushels per acre for example.

It is important to understand that ADAPT is not a standalone program. It is a set of .NET libraries that are integrated with other software (farm management information system for example). In ADAPT, data is not stored or transported. It is important to understand that ADAPT allows you to convert data from one format to another.

AgroEDI


AgroEDI Europe (AEE) is the EDI (Electronic Data Interchange) community of the agricultural sector. It currently brings together more than 280 members from different agricultural sectors. Initially, the exchange of computerized data in agriculture was mainly limited to the commercial sector and the links between suppliers and distributors, using the EDIFACT reference standard, produced by the United Nations agency UN/CEFACT. AgroEDI Europe has worked a lot on the standardization of data in agriculture, and in particular on the transmission of field data by implementing the DAPLOS (Data PLOt Sheet) format, which contains the information relating to the field as well as all the events associated with it (sowing, fertilization, phytosanitary treatments, irrigation, harvesting). This standard exchange format for the plot sheet has been set up on the basis of the EDIFACT language. Each intervention is recognized with a set of numbers and letters that follow each other (Figure 5).

Figure 5: Extract from a Daplos exchange file. Source: Agricultural Outlook, 2008 (Data exchanges, towards a universal language).

At the end of the 2000s, AgroEDI Europe had the ambition to set up a platform to connect agricultural cooperatives and the HCCA (High Council for Agricultural Cooperation) by linking technical data such as DAPLOS and accounting documents (balance sheets, income statements..). This platform project, called RES-AGRI, was therefore to make multi-EDI (Electronic Data Interchange) data exchange through a hub between different actors of the sector and using several standardized formats (ebXML, EDIFACT, EFI, EDI…). In addition to these exchanges between cooperatives and the High Council for Agricultural Cooperation, the RES-AGRI platform was supposed to manage a number of other flows, particularly between the farmer and several entities around him: the Single Payment Agency (AUP) with Telepac, his cooperative or business, and the Chambers of Agriculture. These exchanges had been standardized in the XML AUP, eDAPLOS, and DAPLOS formats (Figure 6). Despite the fact that this RES-AGRI platform is no longer really up to date, some standards are still in use. This is for example the case of the eDAPLOS message, used in the framework of the AgroSyst project led by INRAE to meet the challenges of the ECOPHYTO 2018 plan.

AgroEDI Europe has also contributed to the production and harmonization of agricultural standards, and this is perhaps now what this organization is best known for. For example, we can find data dictionaries set up as part of national standardization projects GIEA and GIEA2 (Farm Information Management) piloted at the time by the Chambers of Agriculture (APCA), referentials of crops, crop stages, pests and agricultural equipment. These reference systems are still very relevant today, with for example the case of the reference systems on pests (AgroObs computerized messages) adopted by the DGAL in their Epiphyt epidemio-surveillance system.

It is also important to keep in mind that AgroEDI has joined the AgGateway community.

Figure 6. RES-Agri platform and associated standardized formats.

AEF


We’ve already talked about the Agricultural Industry Electronics Foundation (AEF) in the context of mobile equipment, particularly with regard to ISOBUS. This organization has more than 200 members. We’re talking about it again here for stationary equipment because one of the AEF’s work groups has implemented the ISOXML exchange format to link the tractor or machine’s task controller to the Farm Management Information System (FMIS) software. It should be understood that ISOXML is also part of the ISO 11783 standard, just like ISOBUS. This file is transmitted in a TASKDATA folder, which you may have dealt with before (Figure 7). The ISOXML format is not used in tractor-tool communication; we saw above that this communication was rather based on the J1939 protocol of the CAN bus. ISOXML is also not used to record the machine tastk report. It is usually done in a proprietary format of the manufacturer and kept in memory, then translated into ISOXML for export.

Figure 7. ISOXML : The XML tags of a TASK Data file

But the initiatives don’t stop there…


Had you had enough? Let me add a few more layers. A French initiative, Numagri, is coming to life. Led in particular by the FNSEA, the Avril group, the agricultural cooperation, and the chambers of agriculture, this project, which is currently being developed, aims to deploy standards (in particular those of GS1, a worldwide standardization organization not specific to agriculture, specializing in barcodes and standardization methodology, but why not also, depending on the need, those of AgroEDI Europe, AFNOR, CEN or ISO) and repositories of information from French farms. Existing standards will be used. The initiative does not aim to create new ones. The AgDataHub consortium will be Numagri’s active arm on the consent (with Orange), exchange (with API-AGRO and Dawex) and data storage (with 3DS Outscale) aspects of agriculture. AgDatahub is not simply involved in data sharing, but in data exchange, providing a legally, contractually and commercially secure transaction environment within a SaaS (Software as a Service) mode exchange platform. On this platform, the emitter can expose its data and distribute it to a very large ecosystem of players, whether in open data, with a fee or in the context of a private partnership.

The manufacturers of agricultural equipment are not left out either. John Deere, Claas and CNH have joined forces under the DataConnect initiative to connect their web portals together. The software editor 365FarmNet is also involved. The MyPLMConnect and AFS Connect (CNH), Claas Telematics (CLAAS), John Deere Operations Center (John Deere), and 365FarmNet clouds will be discussed together. This cloud to cloud initiative allows any user with several machines from different manufacturers to access their machine data (geolocation, consumption, machine performance, etc.) from the platform of their choice. This is not a new platform but the fact that it is possible to access data from existing platforms to access data from competing fleets of machines. For the moment, DataConnect will only allow a manufacturer to display data from another manufacturer; this data will not be shared. So it will initially be just for viewing, not downloading. In the future, however, this information will be exchangeable and retrievable, in particular through the 365FarmNet interface. Within DataConnect, manufacturers will certainly try to define their own standard for cloud to cloud exchange.

For its part, the Bosch company, in partnership with major players in the agricultural sector from various horizons – equipment manufacturers, service and sensor suppliers (Topcon, Amazone, Lemken, Syngenta…) has deployed its NEVONEX open ecosystem. The objective is to enable these agricultural players to send data, variable rate application maps or outputs of decision support tools directly to farmers’ machines, regardless of the brand of machine used. Bosch is truly positioning itself here as an independent and transparent player in the agricultural world, aiming to bring together, within the same open and neutral ecosystem, players with an interest in connecting their tools and services, and this throughout the agricultural chain. Please understand that NEVONEX does not collect and store data, but ensures that they are exchanged in a secure and agreed manner. There is no need here to buy a new machine to exchange and transmit its data, but only to install a NEVONEX control unit on its machines to make them compatible with the NEVONEX system and open them to the cloud (as we have seen with ISOBUS, it is therefore also possible here to retrofit agricultural machines to make them compatible with the NEVONEX system). This control unit allows to act directly on the functionalities of the machine and/or the attached implement, which distinguishes the NEVONEX system from several other initiatives focused on predominantly cloud-based exchanges. Note also that one of the interests of the NEVONEX system is to be able to use tools and sensors already pre-installed on agricultural machines (we can think of weather sensors for example) so as to take into account the data collected by these sensors to improve technical itineraries and field applications.

The DKE-Data consortium has set up the AgriRouteur solution to connect farm management information system and agricultural machinery. It must be understood that AgriRouteur is a kind of data exchange hub (it’s a bit like the Post Office with the mail). DKE-Data has not defined any standards; it does use some (Shape, ISOXML…) but it is not the one who set them up.

In France, the ISOBUS-certified MyEasyFarm platform centralizes field information from farm management information system and data, resources and maps from service providers. Partnerships with DKE-Data (Agrirouteur) and telematics companies (e.g. the BHTronik telematics box) then enable it to make the link with agricultural machinery in the field. MyEasyFarm has also joined the NEVONEX ecosystem.

The Proagrica company, through its F4F division specializing in agricultural standards and connectivity, is positioning itself between farm management information system to make them communicate with each other. F4F also uses AgGateway’s ADAPT system to retrieve data from agricultural equipment.

In the Netherlands, a group of cooperatives has created the non-profit, independent data platform JoinData. This platform seeks to ensure that farmers regain control of their data and can choose with whom their data is shared (research center, service providers, etc.). Actors who want to retrieve data from farmers must also connect to the JoinData platform, which allows farmers to also access several agronomic services (ADO and others). The Dutch initiative is therefore in the same vein as the one proposed by AgDataHub.

The Open Geospatial Consortium (OGC) has also launched a pilot project (Agripilot) around precision agriculture (https://www.ogc.org/projects/initiatives/agripilot2018) to advance interoperability issues in agriculture. There is even a 2009 article in the journal Precision Agriculture, in which researchers had proposed using OGC geospatial standards (notably WFS and WMS formats) for agricultural issues, even though this was not what they were defined for (Nash et al., 2009). The big interest for them was that these standards were adapted to geospatial data, and that precision agriculture data had a strong spatial dimension.

In 2019, the Brazilian Association of Machinery and Equipment Industry (Abimaq) launched its Farmer Collaborative Database (BDCA) platform to centralize machinery and sensor data from Brazilian farmers. The BDCA platform will also set up a data exchange standard (on its own or using an existing standard).

The New Zealanders have also launched their DataLinker initiative to facilitate data exchange in agriculture and agri-food. It includes data models (similar to what AgGateway could do with ADAPT) that are based on New Zealand agricultural standards. Note that DataLinker also includes tools to secure data exchanges, and to ensure that farmers can consent or not to share their data.

Finally, let’s mention the European project IoF2020 (Internet of Food and Farm 2020) which includes the ATLAS project (Agricultural Interoperability and Analysis System – https://www.atlas-h2020.eu/), led by Germany, and which could just as well include standards and exchange systems already developed (ADAPT, AgroEDI repositories…), or not…

A small table summarizing the initiatives below :

InitiativesStructuresMaturité
ADAPT + référentielsAgGatewayOperational
AgDatahubAgDatahubOperational
AgripilotOpen Geospatial Consortium (OGC)Under development
AgriRouteurDKE-DataOpérationnel
BDCABrazilian Association of Machinery and Equipment IndustryUnder development
DAPLOS, eDAPLOS + référentielsAgroEDI EuropeOperational
DataConnectJohn Deere, CNH, Claas, 365FarmNetOperational
DataLinkerRed Meat Profit Partnership (RMPP)Operational
ISOXMLAEFOperational
JoinDataJoinDataOperational
MyEasyFarmMyEasyFarmOperational
NevonexBoschOperational
NumagriFNSEA, Coopération agricole, Groupe Avril, APCAUnder development

Some additional information


We have just seen that in terms of interoperability, there is no shortage of initiatives. There has been a lot of talk about data exchanges, but have you really understood the differences, when there are any, between all the proposed solutions? Even if I hope so, I propose here a small comparative table between ADAPT (AgGateway), ISOXML (AEF), and Agrirouteur (DKE-Data).

ADAPTISOXMLAgrirouteur
Data model and tool for conversion between formatsData formatData gateway
Does not communicate directly with a machine or tool. Uses a plugin to convert to the right format (e.g. the ISOXML plugin, there are others)Communicates with the machine or tool (TASKDATA file). Final communication is done with the J1939Does not communicate directly with a machine or tool. Files are already standardized (shape, ISOXML…)
Centered on the job. Made to manage agronomic or organization dataCentered on the tool and the machine. Is not made to manage agronomic or organization dataCentered on the tool and the machine. Is not made to manage agronomic or organization data
Takes into account the ISOXML scope (with the ISOXML plugin)Does not take into account the ADAPT scope Can transport ISOXML files
Non-exclusive to ISOXMLExclusive to ISOXMLNon-exclusive to ISOXML
Can be used for exchanges with farm management information systems (FMIS) and web portals (clouds)Maybe used for exchanges with farm management information systems (FMIS)Can be used for exchanges with farm management information systems (FMIS) and web portals (clouds)
Retrieves what is done in the field (parameterization, list of tasks, completed tasks…) AND what it was done for (recommendations, observations, history…)Retrieves what is done in the field (settings, list of tasks, completed tasks…)It remains a data gateway
Use of a contextual data system (ContextItems) to have information related to where we are (ex : tons per hectare, bushel per acre…) via RESTful APIGeneral data dictionary (ex : yield per unit area, plant density per unit area. .). Can include ADAPT contextual data (ContextItems)It remains a data gateway
Open SourcePublicPrivate

We have not talked about it much so far, but APIs (Application Programming Interface) are solutions that are increasingly used in agriculture to facilitate exchanges between several partners: service providers, software publishers, etc. Behind an API, we simply need to see an interface to which a user (human, software, etc.) will connect to access a particular service. It’s a kind of front door, a facade for two software programs to interact with each other. Another way of explaining it is that it’s a kind of standardized mill on which we’re going to send data in a certain way and which is going to send something back to us in a standardized way. Via an API, a user can, for example, have access to a database or an Open Data portal, have access to a decision-support tool or an agronomic service, or access graphic elements. It is therefore possible to retrieve any type of information via an API, whether it is a real-time data flow, a map, or a specific request to a decision support tool. Keep in mind that in many cases, we use APIs to make queries (we want to access information), but we can also use them to write information. Also note that APIs can be public or private. In the first case, these APIs are open to everyone, and several levels of services can be offered, with potential restrictions of use. In the second case, they are rather APIs reserved for professional partners, or even sometimes reserved for completely internal use for a company (which would like for example to facilitate access to different data or services for its employees).

APIs are interesting in that they are not very complicated to develop. And a lot of business models can be associated with them, depending for example on a number of requests, a processing time for a request, a number of physical elements (e.g. number of fields processed), or a monthly subscription.

There are also standards for APIs. The most recent are the SOAP and REST standards (for example, we talk about the REST API). REST APIs are currently the most used because they are lighter and more flexible to develop. Communications are mainly done following an HTTP protocol and several data formats are potentially transferable: XML, HTML, plain text, or JSON. The JSON format (and its GeoJSON spatial counterpart) is predominant for several reasons: (1) it is not necessary to have a fixed structure on a JSON which allows you to add as much metadata as you want, (2) it is the language used for NoSQL databases, (3) it is quite readable even for humans, and (4) there is a lot of software to test and visualize these data. Tabular data, such as .csv, are less used because it is harder to verify their integrity (possibly confused with Excel), the parsers are not as good as in JSON, and the .csv format is still a multi-dimensional table that is more difficult to manage. In addition to these standards, it is quite possible to develop an API in your own way, especially for internal use in a company, but it is generally better to follow standards and best practices.

In France, the company API-AGRO has positioned itself on the API sector (as its name indicates), with the will to give access, on the same platform, to a large number of APIs from the agricultural sector. It is thus the positioning of an intermediary or an intermediary that is adopted here. Keep in mind that many agricultural actors are deploying their APIs (the number of APIs is increasing rapidly and it is not going to stop anytime soon) to give access to their services, tools or data, and thus facilitate exchanges with the rest of the ecosystem.

Let’s try to take a step back


So, are you impressed by the diversity of market standards, or are you completely panicked because you feel late? Let me bring you back to reality. There’s theory, that’s what we’ve seen – everything is beautiful, everything looks good – and then there’s practice. Despite all that can be said about it, interoperability in agriculture is not yet a complete panacea. The rhetoric is often ahead of reality. Let’s be clear, no standard has really emerged, and in many cases, everyone has yet to develop their own little trick in their own corner. Some data exchanges are still made by the FTP protocol (with a good old FTP server where files are deposited). A lot of data is transferred in a flat format (Excel, .csv). Two partner companies are almost always forced to map the way the other one has modeled, defined or filled in its data, because both companies use different standards (when they use them). Some companies have internal files that are standardized in their own way, but do not follow market standards, which makes partnerships much less flexible and ultimately locks up the exchange of field data.

But why do we get the impression that it’s slipping and that it’s so slow? Is interoperability still moving in the right direction? Is it really so complicated to follow an exchange standard or a data repository? Can we really make everything interoperable, and does it really make sense? Do some companies have an interest in not moving that fast? Who has the most to gain and who has the most to lose from agricultural interoperability? This topic raises so many questions! After discussing it with several market players, it is clear that the reasons are multi-factorial.

There is often a question of money behind it


One could have the impression that everyone needs to exchange, but some interviewees noted that in the end there would not be so many real needs identified and applicable. It would therefore be necessary to weigh the need to exchange users against the cost of implementing these exchanges, which remains very high. Interoperability pays for itself from an IT point of view, and it can quickly become expensive. All the implementation, development and linking costs are high (especially maintenance costs because norms and standards evolve very quickly) and it is difficult to monetize them with the end user. Some companies may be willing to pay for part of it in order to be compatible with market players (for example because farmers will not be able to finance it), but the end cost is high. Which actors are able to pay for it? For the current price of subscriptions to farm management information system, can we really expect interoperability with a lot of market players? If these exchanges have to be set up for few users, no company will have any interest in spending time on them. In the same way, setting up an exchange with a single player cannot be economically justified. The example of DataConnect is instructive in this sense. By pooling their resources, manufacturers realize that interoperability will cost them less because it is in the use and enhancement of the data exchanged that they will be able to do business. Each manufacturer, taken in isolation, would probably not have the means to impose a particular standard. Generally speaking, market players are in great demand for standardization, because it is all the more important to make savings in terms of development costs.

Companies do not do voluntary work, nor do they do open data. They need to generate revenue and operate in a competitive sector. For the interviewees, the uses are in the real cases. Interoperability will happen if many farmers want it to happen. If all the stakeholders are convinced of the economic interest of an interoperability project – whether it is direct or indirect, whether it increases sales, accelerates the distribution of a tool or even customer loyalty – then there is no reason why it should not happen. Let’s put aside the marketing fantasy that may be behind the announcement of interoperability between two solutions on the market where, in practice, compatibility works under conditions that are so specific and/or complicated that it ultimately concerns only a few people. So, in the end, the nagging question always comes back: Is there an economic interest in the use of this agricultural data? According to some interviewees, the main difficulty lies in the value of digital tools on the agricultural market. The profitability of digital tools in agriculture is very complicated, and even more complicated in France because not all players are looking for profitability. Public or para-public structures can benefit from subsidies. Some manufacturers can finance a digital tool or service through the sale of agricultural equipment. It can be difficult to find alternative ways of exploiting data when, for example, an decision support tool has to be stamped by an institute that has a competing decision support tool.

The economic model behind all this is therefore not easy to find. As a market player, is it better to open your data to others as much as possible or, on the contrary, do you tend to keep your hands on it? It is understandable that companies that have amassed a lot of information may feel that they are losing capital by sharing their data, and are therefore keen to capitalize and protect their assets to keep their competitive edge (in the case of livestock farming, this was the case, for example, when milk control organizations started out, but also when manufacturers of milking robots started out). Those who have this information know what it cost them to acquire customers, and that this information can be a clear differentiator in the market. For example, some players will not want to exchange their data in order to be sure that they can sell their own tool or service based on this data. For example, why give access to data from a yield sensor on a combine harvester if I sell behind the software to be able to view yield maps? Players with a strategy based exclusively on data retention will certainly be more interested in using proprietary formats rather than general market standards and formats. It must also be said that those who tend to want data are often those who have the least access to it, as access to data can be a hindrance to the development of a business. Some will also go so far as to say that the lack of standards is sometimes an excuse for those who do not necessarily have the solutions that go well.

Nevertheless, the general trend is still towards the opening up of data, with market players having the idea that the added value is to be found in the service associated with the valued data (decision support tool, agronomic indicators, etc.). For example, it may be interesting to link local weather data to decision support tool in order to modify or recommend a change in a planned spraying intervention, and thus improve the quality and effectiveness of this intervention. The use of machine data and the exchange of this data cloud to cloud would make it possible to go further in terms of remote machine maintenance, machine performance tests, and the concepts of guarantees, contracts and equipment rental. So we might as well get closer to existing tools or services that know how to extract meaningful BI information from the data collected. This “openness” is not total either, with some players preferring for example to develop exchanges (APIs or others) in close, privileged partnerships. The general trend towards opening up data is also driven by users, who have certainly understood that platforms, clouds and software are flexible. Users use their smartphone to meet several needs and therefore expect these platforms to be able to do the same thing, at the most reasonable price possible. In a world where everything would be interconnected, for example in the case where all the phytosanitary decision support tools would be connected to all the weather stations, the differentiation would then really be made on the quality of service.

Technical reasons and standards are still too permissive.


First thing to realize, on these technical subjects: it is always simpler to start from nothing than to lug around with several years of existing and to make ascending compatibility, that is to say to update to the latest and most used standards. It is important to realize that the inertia of certain players is very important and that choosing one standard rather than another for a company can be synonymous with great success as well as a complete downfall. Choosing a standard that will not be adopted by the community or partners, a standard that may no longer be maintained in the future, or a standard that does not meet its technical and operational constraints, can very quickly turn out to be disastrous for a company. Technologies have evolved enormously, and continue to do so. For example, when ISO started, Windows wasn’t really there yet, so the protocols put in place were those of the time. Few people could have imagined the rise of platforms and clouds at that time.

Interoperability is also complicated by the fact that standards are quite permissive, if not too permissive. Let’s take the example of ISOXML. This standard was originally set up by manufacturers. It was only later that software editors were integrated into work groups. The design of the standard is therefore not 100% adapted to the problems faced by software editors, particularly on the issue of traceability. For example, the exchange of tasks (crop operations) between a farm management information system and a machine is not as simple as that. On a terminal, for example, it is possible to carry out an operation without attaching it to a plot of land. If no identifier has been assigned to a plot, it will therefore not be possible to trace everything that has been done during the cultivation operation. For example, not all the actors have necessarily considered the possibility of having crop successions and stolen crops within the same campaign, or the notion of rank or inter-row for a vineyard parcel.

Apart from rather classic problems (the console is not connected, import/export in shape file is not well done, the console has not been updated….), ISOXML standardization is still implemented in a different way according to the actors. It seems that everyone implements it in his own way (partially or by adding proprietary layers). There can also be specificities depending on the manufacturers or the equipment installed (for example the age of the console). In addition, since the constructors are not specialists in field management, a lot of information has not been considered and is therefore not likely to be easily retrieved, such as the weather or soil characteristics, for example. A lot of information also remains optional, such as tractor consumption. On this specific example, it is not because an exchange is isobus certified on both sides (because the machine and the tool are certified) that the tractor’s consumption will be accessible because some actors do not want to share it. By connecting to a tractor’s diagnostic plug or to the CAN bus, some manufacturers may not respect the classic exchange formats, making it impossible to send information back. Let’s also recall the example of performance data cited above. In this case, the data can be hidden or offended because some manufacturers are in a model where when a machine is purchased, the software has to be purchased in addition.

Generally speaking, it is therefore easier to import data on agricultural equipment than to export it, since it is easier to control what is sent to a machine than what can be retrieved from it. ISOXML remains an exchange format (like others) but the data inside is not necessarily standardized. It is therefore difficult not to mention the question of data referentials or dictionaries that fill the ISOXML format files. Once again, not everyone uses the same ontologies to define their data. For example, the definition of varieties is different between GNIS, CAP (Common Agricultural Policy), AgroEDI, or COMIFER. No ontology has been recorded for fertilizer products. One of the problems is that not all ontologies are open. It is therefore sometimes necessary to be a member of an organization and/or to pay for access, as in the case of AgGateway or AgroEDI Europe for reference systems of all kinds or Lexagri for phytosanitary products. In the case of GS1, on the other hand, it is possible to retrieve the results of work without being a member (but it is necessary to be a member to participate in the work). We can also add the problem of language: working internationally (or even with partners who have international clients) with French attributes can quickly become limiting. A lot of data is useful in agronomy, but each case must be taken separately; there is no standard that defines everything at the same time, at the same time, fertilizers, itineraries, phytos?

From a very operational point of view, let’s put our finger on the fact that ISOXML certification by the AEF is done in a very theoretical way; it is developers who are working on this interoperability issue. But in the field, it can be relatively complicated to identify the conditions under which the link works. How do you know who to call for this kind of problem in the field? The dealer? The cooperative or the trader? The software editor?

Perhaps we can be reassured by the fact that in France, CAP regulations are the same for the whole country, whereas in other countries such as Germany, each region has its own constraints and reporting procedures?

Can we really make everything interoperable?


Can we really harmonize and standardize everything? Is this desirable? From a technological point of view, it is not easy. There will always be someone to create a new solution that is not necessarily adaptable to the standards of the past. For example, some manufacturers claim to be limited by ISOBUS and the capacity of the CAN bus (perhaps even limited by the electrical connection itself) to transmit the mass of data generated by their machines. These manufacturers would face a limit more in the area of communication than in the area of technology. One could nevertheless question the interest of collecting such large masses of machine data (but this is another debate…). In a digital world, everything evolves quickly. Perhaps we should rather focus on a common core in order to be the most flexible and resilient to any evolution of formats or exchange standards. Keeping a readable database for as long as possible could already be a good starting point.

From a business point of view, the answer is not simple either. The example of field definition is perhaps the most telling. Even if the subject has been well advanced in large-scale farming, many actors have a different definition because their needs are different. For example, there is an ISOXML definition for the plot, yet another one for the CAP, and they are not the same definitions for traceability or precision farming needs. For example, some agricultural work companies will consider two agricultural field for traceability reasons, but only one when it is an variable rate application. Farmers are in fact imposed a definition that does not necessarily apply to their working conditions (the tractor carries out an application in a certain way). On this plot definition, the context is even more complicated in the case of viticulture, because digitization is much less important there than in field crops (and the needs are not necessarily felt there). For example, there is a cadastral parcel, a CVI parcel, a PAC parcel, or a parcel at the foot of the vine. Each plot may contain several grape varieties (sometimes mixed randomly), with different years of planting. From a geographical point of view, this can result in layers and sub-layers to be managed; in short, there is nothing simple about it.

Standardizing everything may not make that much sense after all. Standardizing data that meets a concrete need and use would make much more sense.

Exchange or entry of data


Paradoxically enough, the focus seems to be more on how to exchange data rather than how to retrieve it. We have the impression that we send much more information to the machine (descending information) and much less information from the machine (ascending information). We’ve seen all kinds of initiatives and projects to facilitate data exchange, but we still need to have them, and especially qualified ones, that provide relevant information. Having all the field contours of France and making a map of them is good enough, but what do we do with them afterwards? Knowing the interventions planned on a plot is interesting for traceability, but not having any information associated with this plot on the sowing date, or on the yield obtained, can be limiting enough to go further. This information feedback can be done in several different ways, using mobile applications in the field, or automatically through boxes connected to the machines (in the section on the Isobus, we gave the example of connections to the tractor’s diagnostic or ISO sockets, and Bluetooth tags). This input – automatic or not – could also benefit from being associated with an update date (to know for example when the field was last digitized) and a level of reliability to know the level of confidence that can be given to the data. This notion of reliability could be interesting to provide different levels of service and get closer to more operational needs, without over-quality.

As a conclusion


How will the standardization of agricultural data evolve? Anyone who can predict this with precision is very pretentious. We have seen many initiatives and proposals in progress. Maybe that’s not so bad. Would we really have liked to see a single major player arrive with its standard and impose it on everyone? Some interviewees will nevertheless tell me that in France, we would be the specialists not to go and look at what has been done elsewhere and to want to do everything ourselves, even if it meant reinventing the wheel…. Given the cost of data exchange in agriculture, the market will certainly realize that not all players can put so much money on the table and that a few standards will emerge and stabilize over time. ISOXML seems to be quite widely used; the shapefile too, despite what one might have thought. There will probably not be any major events that will completely change the situation on this standardization issue. Implementation will certainly be incremental, in small steps, depending on partnerships with companies in the sector and, hopefully, on the needs of farmers. A sudden change could nevertheless occur with a regulation (national or European). This was, for example, the case with a European directive on the automatic identification of sheep/goats by radio-frequency chip using the ISO standard. There may also be a lot to expect from the European Commission’s “Data Governance Act” on data exchange that will be passed by the end of 2022. The Commission even talks about precision farming in the benefits associated with data exchange in agriculture (https://ec.europa.eu/digital-single-market/en/european-data-governance). This act will be the GDPR penchant for non-personal data, which represents a large part of the data in the agricultural world.

For the moment, one could rather say that farmers have computerized to meet constraints (traceability or other). The relationship is rather unbalanced in the sense that the traceability of field data has been considered by the EU and the French State mainly (to which the actors of the sector whose software publishers have to adapt), without the needs of farmers having been assessed. There are obviously issues of interest to farmers (for example the management of non treatment zones and the high environmental value label which could benefit from automated data feedback) but is the subject of traceability not treated in too complex a manner? In the same way, the deployment of precision farming applications – and therefore these needs for interoperability – seems to meet a need of companies in the sector rather than a real demand from farmers. Standardization can certainly have some very good sides by increasing the amount of data and services available to farmers, but it is important to keep in mind that end-to-end standardization requires a large organization with many players involved. Who will the farmer have to turn to when there is a problem? To his dealer? The manufacturer? The service provider? His cooperative? By dint of making things too simple, we could end up with effects contrary to those initially imagined.

We still consider the problem too much from the point of view of the company – what we can do with the data – and not from the point of view of the farmer, what he can, wants or needs to do with his data. One thing is certain, for the issue of standardization to evolve, farmers will need to become aware of the issues and acculturate themselves about data, to avoid becoming captive to a proprietary platform or cloud (as may be the case now – we are talking about data portability, i.e. being attached to a provider), and understand what is being done with their data. We cannot accept that a farmer can’t simply retrieve data that he would have filled himself. We cannot accept that a farmer cannot know what is being done with their data. We cannot accept that a farmer should not benefit from the data he shares.

But at the end of the day, doesn’t everything that has just been said in the agricultural sector apply to all the other areas that feel the need for more interoperability?

Interviewees


NomStructure
Julien ANCELININRAE
Guilhem BRUNELMontpellier SupAgro
Jean-Christophe CHASSINEClaas
Gaelle CHERUYAgro EDI Europe
Anthony CLENETSMAG
Alexandre DIAZIsagri
Clément FRAIGNEAUPermagro
Gilbert GRENIERAgro Bordeaux
Eric GUEMENE365FarmNet
David JOULINEkylibre
Thomas KIRSTEBosch
Sébastien PICARDARTAPI-Agro / AgDataHub
Julien SAINT LAURENTJohn Deere
François THIERARTMyEasyFarm
Romain TRIBOUTSamsys
Jim WILSONAgGateway

Literature


AgGateway, 2016. Organization Alignment in Standards Development. Louisville

Applegate, D.B. et al. (2016).  Toward geopolitical-context-enabled interoperability in precision agriculture: AgGateway’s SPADE, PAIL, WAVE, CART and ADAPT. ISPA, 13th Conference on Precision Agriculture

Bahlo, C. et al. (2019). The role of interoperable data standards in precision livestock farming inextensive livestock systems: A review. Computers and Electronics in Agriculture, 156, 459-466

Grenier, G. (2010). Bus CAN sur machines agricoles : les technologies de l’information au service de l’agriculture de précision et de la traçabilité. Ingénieries eau-agriculture-territoires, Lavoisier ; IRSTEA ; CEMAGREF, 67-76

Nash, E., Korduan, P., et Bill. R (2009). Applications of open geospatial web services in precision agriculture: a review. Precision Agriculture, 10, 546-560

Appendix


Here is a non-exhaustive list of actors who are involved in standardization in agriculture (some are more specialized than others, some do not develop standards specifically for agriculture but they can be used in agricultural applications). You can find national/international organizations, industrial or professional associations, informal groups… This list is taken from a seminar by Jim Wilson of AgGateway :


AEF: Agriculture Industry Electronics Foundation

AFNOR: French Association for Standardization

AgGateway

AgroConnect (Netherlands)

ANSI: American National Standards Institute

ASABE: American Society of Agricultural and Biological Engineers

CEN: European Committee for Standardization

CNIS: China National Institute of Standardization

DIN: German Institute for Standardization

ICAR: International Committee for Animal Recording

IEC: International Electrotechnical Commission

ISO: International Organisation for Standardisation

ITU: International Telecommunication Union

GODAN: Global Open Data for Agriculture and Nutrition

GS1

OAGI: Open Applications Group

OASIS: Organization for the Advancement of Structured Information Standards

OGC: Open Geospatial Consortium
Standards New Zealand

UN/CEFACT: United Nations Centre for Trade Facilitation and Electronic Business

W3C: World Wide Web Consortium

1 thought on “Standards and data exchange in agriculture

Leave a Reply

Your email address will not be published. Required fields are marked *