Prior to founding Koidra, Kenneth Tran had worked extensively at Microsoft Research on real-world reinforcement learning research with applications in data center energy optimization and controlled environment agriculture.
The cucumber plants, their leaves wide and green and veiny, stood tall in neat rows, basking in the Netherland sunlight shining through the glass panes of their greenhouses. Hopes were high for the plants—a bountiful crop in just four months using as few resources as possible. With the right amount and type of care, they’d produce vegetables for consumers to enjoy. To the casual observer, though, it might have seemed like the plants had been left to their own devices. Greenhouse staff passed through to harvest or adjust cameras and other electrical devices inside, but human contact from those responsible for determining how much water, nutrition, and light the plants received was nonexistent. That was the point.
This spring, Wageningen University & Research and corporate sponsor Tencent challenged researchers, scientists, and experts from across sectors: Build the greenhouse of the future. Motivated by potential strain on traditional methods of food production as a result of a growing world population and seeing a solution in greenhouses that are operational sans on-site human expertise, organizers asked competition participants to use artificial intelligence to maximize cucumber production while minimizing resources—and to do so remotely.
Nine months later, Microsoft Research’s Team Sonoma, led by Principal Research Engineer Kenneth Tran, beat out four other interdisciplinary teams to win the Autonomous Greenhouse Challenge, creating an agent that produced more than 55 kilograms of cucumber per square meter with a net profit of €25/m2.
“This was the first time worldwide cucumbers were grown in greenhouses remotely on AI,” says challenge coordinator Silke Hemming. “We at Wageningen University & Research were excited to collaborate with different teams on this exciting international challenge. Team Sonoma was able to beat the manual-grown reference operated by good Dutch growers. They not only reached the highest net profit, but the jury also ranked them highest on total sustainability.”
With a net profit 17 percent higher, Sonoma was the only AI team to best the reference expert growers, and its net profit was 25 percent higher than that of the second-place team, led by researchers at Tencent AI Lab. Net profit counted the most toward overall performance in the competition while algorithm novelty and capacity accounted for 30 percent and sustainability— based on efficiency in energy, water, CO2, and pesticide usage—accounted for 20 percent.
A greater journey
For Tran and Microsoft, the work demonstrated in the competition is part of a larger commitment to deploying cloud, Internet of Things, and AI technologies to protect and sustain the planet and its natural resources. In July 2017, Microsoft launched AI for Earth to support individuals and organizations doing work in water, agriculture, biodiversity, and climate change with grants, education, and further collaboration. The initiative’s strategic approach and funding has since been expanded, and the gains being made, especially in the area of data-driven farming, have been impressive. FarmBeats is among the projects receiving recognition for its impact, and Tran and another Team Sonoma member, Senior Research Software Engineer Chetan Bansal, are also contributors to that work.
While FarmBeats is improving data collection outdoors with sensors, drones, and other devices for more sustainable farming, Team Sonoma’s work is in controlled environment agriculture (CEA), an enclosed system of farming that allows growers to determine and execute optimal settings for such environmental factors as light, temperature, humidity, and CO2 concentration.
Tran’s interest in CEA as a research area was piqued in 2017, about a year before he had heard about the Autonomous Greenhouse Challenge. As a member of the Reinforcement Learning group with Microsoft Research , he and his colleagues explore the machine learning technology’s potential for real-world application. Not only is CEA’s ability to have meaningful impact attractive—a more efficient, accessible means to meeting the nutritional demands of the earth’s population—but it is also a great training ground for reinforcement learning models. CEA offers contained scenarios in which to work and an abundance of data, the collection of which is relatively quick and easy thanks to sensing technology and IOT.
“The state of the art of reinforcement learning is notoriously data hungry, so it is critical that we focus on new sample-efficient algorithms,” says Tran. “ To faster close the gap, though, we also need environments where we can collect a lot of data easily and affordably.”
The application focus of Sonoma—the project name for Tran and his colleague’s overall work in the area, as well as the name for the greenhouse challenge team—has been greenhouse and vertical farming, both of which have the potential for safer, faster food production with less use of the resources that have been literally the foundation of traditional farming—land and water. According to the greenhouse competition website, indoor cultivation such as greenhouse and vertical farming can decrease water requirements by up to 90 percent, needs one-tenth the space to produce the same amount of crop as more traditional farming, and can thrive on less pesticides and chemicals. This promising solution requires a workforce of indoor farming experts that might be outpaced by the demand for indoor farming, though, and Tran has made it Sonoma’s goal to help create autonomy in the space.
“AI can help both scaling the expert knowledge in developed countries such as the Netherlands to developing countries, but also improve upon the expert growers,” says Tran.
To reach the Sonoma goal, Tran leads with what he describes as a bottom-up, top-down approach.
“By bottom up, we mean doing novel research in reinforcement learning, the very fundamental reinforcement learning research, and also doing the application-centric research simultaneously,” he explains. “Applicable reinforcement learning research is still at a very, very early stage, so there is a lot of ground for new research to be done. For the top-down aspect, our approach is to seek collaboration with domain experts from around the world.”
The Sonoma project members reflect that philosophy: Tran and Bansal, both of Microsoft Research, represent the AI side. On the plant science side and from partnership institutes are research scientist Xiuming Hao of Agriculture and Agri-Food Canada (AAFC) and Chieri Kubota, professor of controlled environment agriculture at The Ohio State University, among other collaborators.
A competition along the way
It was Tran’s strong belief in the power of collaboration that led to a Team Sonoma for the international Autonomous Greenhouse Challenge. He was visiting Wageningen University & Research—a leading partnership in food production—in March 2018 to explore opportunities for collaboration. During the meeting, his Wageningen counterparts mentioned the challenge, and he brought it back to his team at Microsoft Research. They were all intrigued.
“We started reading about the competition and got excited,” recalls Tran. “It was a great opportunity to get our feet wet and quickly. There was already a strong commitment from multiple key players in the area—Wageningen University & Research, Tencent, Intel, and Delphy and AgroEnergy, among them—working toward a shared goal and a shared vision. Plus, there was guaranteed support from Wageningen staff during the course of the competition.”
And Team Sonoma was formed: Tran from the Microsoft Research Redmond lab; Bansal from the Microsoft Research India lab; Thomas Follender Grossfeld and Vincent van Wingerden of Microsoft Nederland; David Katzin, a Wageningen PhD student; and Hong Phan, a PhD student from the University of Copenhagen. Sonoma was one of five teams selected from a pool of 15 for the main event after a pre-competition “hackathon” that included a virtual growing challenge and a presentation of their approach.
Each team received 96 square meters of greenhouse space at the Wageningen University & Research campus in Bleiswijk in the Netherlands. Each greenhouse was outfitted with the same system of actuators, including ventilation, heating and artificial lighting, and sensors to measure, among other things, temperature, moisture, and energy consumption. Teams were also permitted to install additional sensors and monitoring equipment.
Sonoma installed additional cameras but only one additional sensor—a leaf-wetness sensor, which was not among the preinstalled competition sensors. The team chose the sensor for increased monitoring of humidity and moisture, two factors that lead to crop-damaging pests and disease. Teams were permitted inside their greenhouse only once, to set up their extra cameras and sensors. Throughout the competition, from September 1 to mid-December, they ran their AI frameworks remotely.
But for Bansal, the distance from the site of data collection wasn’t as challenging as the means of data collection itself. The team needed to design a system that could account for Murphy’s law—whatever can go wrong will go wrong.
“The data is coming from multiple sources—there are sensor boxes, cameras, the API being used by the greenhouse,” says Bansal. “All of them can and have failed, so the question is how you’re able to detect it and how fast you’re able to react.”
“We had to deal with all of those issues and design a system that was resilient to all of them, but I think that’s part of designing any real-world control system,” Bansal added.
The Sonoma AI approach
Sonoma chose to build its framework around approximate Bayesian model-based reinforcement learning.
“We bet on model-based RL because we think it is sample-efficient and generalizable,” says Tran. “Sample efficiency is critical for real-world applications. Standard RL algorithms require a huge number of trials—in the millions—to train a good agent, even in simple environments. This is not a big deal in games, where RL has shown success, because an agent can play as many games as it needs. In real-world applications, we cannot afford to run millions of failed trials. So we need to think differently about RL.”
For reinforcement learning to be a viable solution for today’s societal challenges, the team determined the agent must be initialized as strong as any existing system and have the ability to learn and improve over time without limits to its capacity to reach optimality and conceived a framework (above) incorporating these features.
The framework begins by training a probabilistic dynamics model. This model learning is analogous to building a simulator, which helps the agent to plan by imagining. In addition, by way of imitation learning, the agent is initialized to perform like an existing expert policy. From there, the agent will operate on a continual model-based policy optimization process, improving its overall performance with every environmental interaction.
From left: Research scientist Xiuming Hao, greenhouse vegetable specialist Shalin Khosla and Tran at Agriculture and Agri-Food Canada’s Harrow Research and Development Centre in Ontario
For the greenhouse challenge, data from around the greenhouse, such as weather conditions, and from sensors and images from inside the greenhouse were input into the agent, which then determined the intensity and distribution of artificial lighting; the amount of water, CO2, and nutrients to give the plants; and greenhouse temperature. The framework chose settings based on what it learned would result in the most biomass.
“The team has successfully incorporated the current best knowledge and best practices on crop cultivation and management and on greenhouse environmental control into its greenhouse AI control system,” says Xiuming Hao, the Agriculture and Agri-Food Canada research scientist who collaborates with the Sonoma project. “The team started with a high plant density system identified from previous model data, and adjusted the AI climate control based on crop performance and weather conditions over the crop growing period to allow for the best performance of this high density/high production system.”
Tran describes the team’s strategy as conservative. The competition setup allowed for only one trial, and there was not much data existing before the challenge; because of that, its strategy relied on a hand-crafted expert policy alone without resorting to reinforcement learning for continuous learning and improvement—yet.
“By working with domain experts and leveraging their knowledge, as well as the capabilities of our AI agent, together we were able to produce better results within a short time frame,” says Tran. “And this is just the beginning; there is a lot of room for growth, literally and figuratively.”
