Key Considerations in IoT Big Data and Cloud Computing

In 2018, the number of connected IoT devices is projected to grow to over 23 billion, according to Statista. And these devices will create a volume of 400 zettabytes of data by the end of the year, as reported by Datastax. With such an explosion in device-driven data, it’s important to have a strategy for maximizing the business impact of IoT big data, transforming it into actionable intelligence.


Naturally, many organizations are turning to the cloud as a central location to aggregate, analyze, learn from, and store IoT data; in fact, Datastax projects that 90% of all IoT data will live in the cloud by 2020. If your organization is one of the 65% of companies who are predicted to adopt IoT by 2020, read on as we share our key business considerations when planning for a large influx of IoT data.

Consideration 1: Edge Processing

With large amounts of data being generated by end devices, it’d be impractical for every data point to be sent to the cloud. Take for example the connected car. According to a presentation by Intel CEO, Brian Krzanich, the average connected car will produce 4,000 GB of data per day, and that’s just for one hour of driving a day. Organizations must decide which of these data points should be sent to the cloud for additional analysis and which are only important if they, for example, report an anomaly.

Further to the point, some data simply should not make the round-trip to the cloud. For example, it would not make sense for an autonomous vehicle to sense an upcoming stop sign, and send that data to the cloud and back. In addition to it not being timely given potential latency issues, it adds an unnecessary burden to the system.

Clearly defining what data can and should be processed at the edge — either for immediate action like a stop sign or for anomaly detection — is an important first step in an IoT data strategy. Closely following, organizations should seek to identify data that can be aggregated at the edge and sent in that form to the cloud for deeper analysis vs data that needs to be sent in real-time.

Consideration 2: IoT Data is not Traditional Data

Unlike data most organizations are used to working with, IoT data is often unstructured and come from noisy devices that are recording any number of data points — from temperature to time stamp, location to motion and more. To make sense of these data points, it is often helpful to couple them with external data sources.

As a result, cloud platform vendors have created specific solutions, like Amazon’s AWS IoT Analytics, for analyzing IoT big data. These platforms provide tools to effectively analyze massive volumes of IoT data, allowing you to collect only the data you want from your devices, applying mathematical transforms to process the data, and adding device-specific metadata before storing the processed data where it can be analyzed via SQL queries.

The lesson here is that standard analytics or BI tools will likely not be a fit for your IoT data. Prepare in advance for this eventuality by examining IoT-specific analytics solutions like AWS IoT Analytics.

Consideration 3: Cloud-Based Machine Learning

For deeper intelligence where you are able to extract patterns with greater business relevance (and impact), machine learning is a natural next step in the IoT big data journey. Machine learning can help you perform more complex analytics and gain important inferences, such as which devices are about to fail.

Creating a tight, agile feedback loop, machine learnings can then be distributed back to the edge in the form of feature enhancements, upgrades, etc. to further enhance the device’s function and the overall contribution to the business.

Let’s examine a few examples:

IoT  Manufacturing

A customer of ours is a Fortune 500 manufacturer who sought to connect its customers with data insights using an IoT model. The overarching goal: to facilitate continuous data-driven improvement within its customers’ operations — all while complying with EU data privacy laws. With this manufacturer’s competitive advantage revolving around its IoT manufacturing strategy, we created an AWS-based solution that allows it to stream, analyze, store and share data collected by thousands of machines in a secure, EU data compliant cloud architecture. Now its customers can connect and share expertise based on data sent from their machinery to the cloud. This IoT data allows customers to improve their quality and productivity in real time.

IoT Energy

The second customer story we’d like to share with you is Fugro, who collects and provides highly specialized interpretation of geological data. The company was looking to launch OARS (Office Assisted Remote Services), an IoT-based service, to reduce, and potentially eliminate, the need for surveyors onboard sea-going vessels, optimizing project crewing, safety and efficiency.

In addition to building a strong IoT AWS-based infrastructure, Flux7 helped ensure data security, a non-trivial challenge when working with unreliable networks; Fugro’s vessels communicate sensitive data to OARS over satellite, which can be intermittent given the mobility of vessels to remote areas.

Flux7 and Fugro applied networking designed to provide an encrypted channel on top of low bandwidth satellite-backed Internet connections to the vessel. The team designed the system with high availability and infrastructure flexibility to help ensure data would not be lost due to a disaster recovery issue. An IoT cloud platform model coupled with a layered security approach, redundancy and elasticity built in, the OARS project keeps skilled staff onshore and provides faster interpretation of data and decisions, and better access to information across regions.

IoT data analytics is certainly still maturing, with solutions in the industry still evolving to support this growing need. However, with a solid strategy in place, your organization will be positioned to take advantage of important IoT data today, and well into the future. From improving the customer experience to driving safety and cost efficiencies, IoT data will undoubtedly have significant business and customer impact.

To continue learning more how IoT and cloud computing can create sustaining impact to your business, here are several resources to get you started:

• Building the Innovation Friendly IoT Infrastructure
• Succeeding with the IoT-Ready Infrastructure
• The Five Keys to Successful IoT Initiatives
• AWS Case Study: Creating Scalable IOT Infrastructure
• AWS Case Study: Fugro Launches IoT Based Service with Flux7 Cloud Expertise