The exponential shift in data generation and storage capacity to 2040
An excerpt from The Holon Data Report, Part 4
In our Data Report series, we outline the enormous growth in data generation from 2011 to 2021. It took us over 70 years to reach 1 zettabyte (ZB) of annual data generation, but then quickly hit 80 ZB in 2021 as smartphones and IoT devices began to proliferate across the world.
Looking to the next 20 years, Holon believes that global data generation could even reach as high as 75,000 ZB. In this fourth report, we outline the exponential data growth that is expected to come from autonomous vehicles and Internet of Things (IoT) devices, as well as ten new technology platforms that are moving towards commercialisation over the next two decades.
The sources of data
The rate of improvement in the early-stage testing of autonomous electric vehicle (EV) software by Tesla suggests it will become standard across all of its EV models by the late 2020s (Figure 1). Other EV makers will also likely launch equivalent products before the end of this decade. With each vehicle expected to generate 10 gigabyte (GB) of data every second, autonomous EVs will be the largest single source of data generation globally over the next 20 years. Holon’s conservative estimates show that passenger vehicles, autonomous taxis and ride-hailing service EVs could generate over 16,000 ZB of data annually in 2040.
Figure 1: Tesla Level 5 autonomous driving
Source: electrek.co
The Internet of Things (IoT) will also generate an enormous volume of data relative to human creation today. ARM Holdings, the UK chip designer owned by Japan’s Softbank, forecast in 2018 that the cumulative number of IoT devices in operation will reach 1 trillion units by 2035 (we consider this to be at the high end of estimates). IoT devices are increasingly being placed on critical infrastructure like pipelines and bridges, sending constant performance updates that can alert engineers when problems first begin to occur. Their use case is endless. Using an estimate of 1,573 kilobyte (kB) of data generation every second, Holon’s estimate of 930 billion IoT devices in 2040 will generate over 46,000 ZB of data per year.
“Looking to the next 20 years, Holon believes that global data generation could even reach as high as 75,000 ZB.”
Data is set to undergo a substantial change over the next decade as new technology platforms emerge. Most of the data we produce today (referred to as our ‘datasphere’) occurs at the individual level, with personal 4K videos the largest source of data. As a result, 4.7 ZB (zettabyte, equal to one trillion bytes) or 70% of global data storage capacity sits within end-user devices that include mobile phones, personal computers (PCs), and laptops.
The critical issue we see emerging is the lack of sufficient data storage capacity in the world, which will see our ability to store data fall exponentially over the next two decades. By 2025, our estimates indicate that we will be able to store just 0.04% of the data we generate (assuming four copies for redundancy purposes).
Unless we can massively increase our data storage footprint towards 1,000 ZB by 2040, the lack of storage capability will begin to dramatically slow down the rate of innovation for the technology platforms discussed in this report. We estimate the total cost to reach this target at US$50-100 trillion, and discuss an alternate model put forward by Filecoin that offers a potential solution by bringing thousands of storage providers operating on a unified platform.
Holon’s data generation estimates for all technology platforms
In 2020, the world added approximately 0.4 ZB of new enterprise storage capacity, while the total volume of data produced globally reached 64 ZB. Assuming no existing data in enterprise storage was deleted during the year meant the world could save just 0.6% of 2020’s data (we refer to this as data storability). However, we also need to consider that multiple copies of the same dataset need to be stored for robustness of the data as well as for the speed at which that data can be delivered to its users. Assuming four copies are kept, data storability for 2020 fell to just 0.15% of the 64 ZB of data.
We can combine this net additional annual storage capacity growth for autonomous EVs, IoT and the new technology platforms being launched over the next two decades and can estimate the total volume of data that will be generated every year. Our combined data estimates are shown in Figure 2.
Figure 2: Holon data generation estimates, global technology (ZB), 2025 – 2040
Source: Holon
In Figure 3, we have estimated the total net addition in new storage capacity each year (and have provided it for 2025, 2030, 2035 and 2040). Data Report Part 4, Chapter 1 discusses the methodology behind Holon’s estimates for global data centre storage capacity.
Figure 3: Estimated additional storage capacity requirements forecast to 2040
Source: Holon
The second line estimates the net global data storage capacity increase in each of the years for 2025, 2030, 2035 and 2040, which assumes a 27% compound growth rate to reach our forecast global data centre storage capacity requirement of 1,000 ZB by 2040.
Storage hardware (HDDs and SSDs) however have a short replacement span of just 5-6 years. After accounting for the need to replace this hardware, the straight-line compound growth rate over the period from 2020 to 2040 grows to 39% per annum to reach the same 1,000 ZB target in 2040.
An impossible task, by any measure.
Data Report Part 4 in its entirety, can be found here.
Disclaimer: This Article has been prepared by Holon Global Investments Limited ABN 60 129 237 592. Holon Global Innovations Pty Ltd (“HGI”) is a wholly owned subsidiary of Holon Global Investments Limited (together “Holon”). HGI is a Filecoin (FIL) Storage Provider and is positioned as a major player in the FIL decentralised data storage arena. FIL Storage Providers are rewarded in FIL for the provision of data storage capacity. Holon, its officers, employees and agents believe that the information in this material and the sources on which the information is based (which may be sourced from third parties) are correct as at the date of publication. While every care has been taken in the preparation of this material, no warranty of accuracy or reliability is given and no responsibility for this information is accepted by Holon, its officers, employees or agents. Except where contrary to law, Holon excludes all liability for this information.
