AWS cloud and machine learning for precision farming

Ballance Agri-Nutrients is a leading provider of innovative fertiliser solutions, dedicated to supporting farmers across New Zealand. Known for integrating advanced technology into agricultural practices, Ballance’s mission is to enhance pasture and crop yields while ensuring environmental protection and sustainability. The company has a sophisticated fertiliser management system designed to help farmers optimise nutrient application throughout the seasons, ensuring regulatory compliance and providing essential financial tracking tools for informed decision-making.

Overcoming inefficient and costly on-premise fertiliser management system

​​Ballance’s fertiliser management system relied on a proprietary software stack based on a leading commercial geospatial platform and bespoke hardware, incorporating GPS tracking and mapping software to enable precise aerial nutrient application. Farmers & Nutrient Specialists would map their properties, designating specific areas for nutrient application, while satellite imagery and machine learning were employed to identify exclusion zones—such as waterways, trees and shrubs—to prevent environmental contamination.

Despite its initial effectiveness, this on-premise system faced several critical challenges. It was limited to processing only one job at a time, which hindered scalability. Maintaining the custom hardware and proprietary software was costly, with significant expenses tied to licensing fees for the commercial geospatial platform. Additionally, the system had evolved from a proof of concept to a widely-used prototype, yet its architecture was not designed for large-scale operations, leading to workflow bottlenecks.

Moreover, the system’s machine learning models were generic and struggled to accurately account for New Zealand’s diverse landscapes. This lack of precision often resulted in errors that required substantial manual intervention. As environmental regulations became more stringent, the need for accurate identification of exclusion zones became even more critical to prevent nutrient runoff into sensitive areas. Furthermore, there was a growing demand for more advanced financial tracking and farm management tools to aid farmers in making better decisions.

Migration to AWS cloud and MLOps enables scalable, cost-efficient machine learning results

Ballance partnered with DiUS to migrate the compute-intensive and bottleneck portion of its nutrient management system to a scalable AWS cloud-based architecture. This move was more than just a transition to the cloud; it involved integrating advanced, region-specific machine learning models to enhance precision in nutrient application, reducing errors and manual intervention.

The cloud migration and MLOps journey began with a proof-of-concept phase, followed by detailed integration, continuous testing, and validation to ensure compatibility with existing workflows. Customisations were made to meet Ballance’s specific needs, with focussed staff training and support provided to facilitate the transition. Key to the project’s success was the close collaboration between the technical teams, which maintained open communication and an agile approach through regular updates and showcases.

The transformation replicated portions of the original system’s functionality while leveraging cloud computing for improved scalability and cost efficiency. DiUS combined traditional machine learning with open-source geospatial software, and redesigned the system architecture to replace the existing geospatial platform with scalable AWS cloud-based solutions. Amazon SageMaker was used for model training and deployment, AWS Lambda for triggering both training and inference jobs, and AWS Fargate for scaling inference tasks. The solution was deployed using Infrastructure as Code (IaC) through AWS CDK, enabling seamless replication across other environments.

DiUS also established a robust MLOps pipeline to handle batch data processing, enabling more accurate and efficient fertiliser application. This included automating both the training and inference pipelines with serverless function triggers to execute jobs based on files uploaded to S3, significantly simplifying operations for non-expert team members. Inference jobs were tagged with an identifier for the model to be applied, thereby allowing multiple models to be selected within a single pipeline. To efficiently process very large datasets, Amazon Elastic File System was used for files exceeding 300GB and volumes were mounted directly into training and inference jobs.

A retry mechanism was implemented using Amazon Simple Queue Service allowing failed jobs to be re-queued and retried automatically if resources were temporarily unavailable, ensuring reliability in both the training and inference processes. This added robustness to the system, helping Ballance maintain operational continuity even during times of high demand or resource constraints.

Security and governance were addressed through AWS’s secure infrastructure, ensuring compliance with Ballance’s stringent cybersecurity standards. Additionally, all jobs, whether training or inference, were logged using Amazon CloudWatch, providing enhanced monitoring and observability. This ensured any issues could be traced back and performance between models could be compared effectively.

A more efficient, accurate, and sustainable fertiliser management system

The cloud migration enabled Ballance to process multiple jobs simultaneously and at lower cost, cutting processing times and enhancing nutrient application accuracy. Machine learning models tailored to specific regions, like Otago, ensured that nutrients were applied precisely where needed, minimising environmental impact and maximising crop yields. The transition further reduced operational costs by eliminating licensing fees for proprietary software and the need for bespoke hardware.

Parallel processing enabled by the more modern infrastructure meant processing times for model inference dropped from up to ten minutes to under a minute, and training times were cut from 35 hours to a more efficient duration of 8 hours. More than 10 training jobs can run concurrently if needed, rather than one at a time. Similarly, inference jobs scaled efficiently, allowing potentially hundreds of jobs to be processed simultaneously with minimal costs. This improvement in model processing speed reduced manual interventions, enabling the team to focus on other critical projects. The transition was seamless, with minimal issues, and the AWS architecture delivered the expected results.​​

The success of this project has positioned Ballance for future growth, enabling it to serve more farmers and explore new markets. The system’s promising early results suggest it will handle increased demand during the busy spring season effectively.