The world population is expected to increase to 9.7 billion by 2050, and the demand for food by 60%. Will the agricultural industry be able to keep up? How will the world ensure food security? AgriTech products are leveraging modern technologies to fill the gap here.
There's enough data generated by the agricultural industry and many adjacent industries to bring predictability to forecasting food production and create contingency plans. This data is scattered, siloed, unstructured, and warehoused by different private and public organizations using different metrics, local languages, and formats. The different types of data (though siloed) are production, demand, supply, crop prices, satellite images, topography reports, weather forecasts, and soil quality reports. There are many companies that are trying to break through these barriers.
We worked with one such company and built a comprehensive data and analytics platform, which provided insights on demand-supply, and climate and soil conditions. It aided stakeholders across the agriculture value chain to make informed decisions, faster.
Zemoso helped them ingest the diverse data sets, analyze it, and build a Software-as-a-Service (SaaS) platform with dashboards and modeling systems.
The data included global and regional data with daily updates gathered through both scraping and Application Programming Interface (API). It also had:
Deep tech solutions applied here include image processing, data pipelines, and distributed cluster architectures.
Our team normalized the data and automated continuous quality checks on the ingested data from external sources. We implemented the computing logic and built the data processing platform for managing the data at scale.
In addition to incrementally processing daily feeds, one recurring challenge was to process massive satellite raster datasets covering the entire globe. This data is vital to analyze the progression of planetary climate and identify patterns that help forecast impending weather conditions.
We helped our client develop innovative processing techniques like overlaying pre-generated geo-rasters for efficient processing, leveraging distributed storage and workers, and scheduling and orchestrating the activities.
Instead of using supra-large hardware, we maximized using smaller instances, which enabled us to iterate and scale rapidly. We built a processing cluster using an open-source distributed scheduler that could be scaled up and down based on the requirement to speed up or slow down work.
We accomplished in days, what would have taken months to process.
Consider this, data that captures every 10m x 10m block of the planet and does this every day for over 15 years.
We did memory optimizations and virtual machine (VM) sizing to ensure that no task was starved of Random Access Memory (RAM). We used libraries like Mapnik C++ to re-render these rasters so that consumers could view them and perform time travel over a region. Additionally, we performed multiple layering and mapping optimizations to enable this capability.
The data (including imagery) was fed to other pipelines to compute metrics, which in turn helped create aggregate visualizations, both map-based, and charts-based. We generated data and map layers to visualize the data and imagery over time and across the world.
The whole platform was packaged as a very customizable platform that allowed users to select and dissect indicators of interest, compare them, visualize them, and share them.
With the essentials firmly in place, Zemoso transitioned the future growth activities for the platform to the client’s internal team.
The platform continues to evolve, incorporating new data feeds and capabilities such as Artificial Intelligence (AI) - based learning models. That coupled with human insights enabled by powerful data visualization capabilities — is a vital industry asset to navigate uncertainty of the global food and agri-market.
P.S. Since we work on early-stage products, many of them in stealth mode, we have strict Non-disclosure agreements (NDAs). The data, insights, and capabilities discussed in this blog have been anonymized to protect our client’s identity and don’t include any proprietary information.
