How does MQTT work in IoT projects?

 

If you think that the internet has changed your life, think again. The IoT is about to change it all over again!

What Is IoT?

The Internet of Things, or IoT, refers to the billions of physical devices around the world connected to each other via the Internet. Collecting and sharing data. Connecting up all these different objects and adding sensors to them adds a level of intelligence to these devices, enabling them to communicate real-time data without involving a human being. These devices range from ordinary household objects like bulbs, thermostats to sophisticated industrial tools, computers, etc. Previously Bluetooth and RF (radio frequency) were used to control IoT applications, but they were limited to a short distance. Adding MQTT capabilities can help in overcoming inter-communication problems by securely automating IoT appliances.

What Is MQTT?

MQTT is one of the most commonly used protocols in IoT projects. MQTT (Message Queuing Telemetry Transport) is a messaging protocol that works on top of the TCP/IP protocol. MQTT can also run on SSL/TLS. SSL/TLS is a secure protocol built on TCP/IP to ensure that all data communication between devices is encrypted and secure. MQTT is a lightweight protocol that uses publish/subscribe operations to exchange data between clients and the server. Furthermore, its small size, low power usage, minimized data packets and ease of implementation make the protocol ideal for the “machine-to-machine” or “Internet of Things” world. Unlike HTTP’s request/response paradigm, MQTT is event-driven, and clients receive published messages. This type of architecture decouples the clients from each other to enable a highly scalable solution without dependencies between data producers and data consumers.

How does MQTT work?

MQTT uses your existing Internet home network to send messages to your IoT devices and respond to the messages.

At the core of MQTT is the MQTT broker and the MQTT clients. The broker is responsible for dispatching messages between the sender and the rightful receivers. An MQTT client publishes a message to a broker and other clients can subscribe to the broker to receive messages. Each MQTT message includes a topic. A client publishes a message to a specific topic and MQTT clients subscribe to the topics they want to receive. The MQTT broker uses the topics and the subscriber list to dispatch messages to appropriate clients. If the connection from a subscribing client to a broker is broken, then the broker will buffer messages and push them out to the subscriber when it is back online. If the connection from the publishing client to the broker is disconnected without notice, then the broker can close the connection and send subscribers a cached message with instructions from the publisher.

MQTT Components:

In MQTT there are a few basic concepts that you need to understand:

Broker – The broker is the server that distributes the information to the interested clients connected to the server. This is the heart of the publish/subscribe protocol. The MQTT Broker is optimally designed to handle many thousands of concurrently connected MQTT clients.

Client – The device that connects to broker to send or receive information. The MQTT Client, be it Subscriber or Publisher (or both in one device) is any device from small Microcontroller up to a fully-fledged server, that has an MQTT library running and is connected to an MQTT Broker over any kind of network.

Topic – Messages make their way from a publisher, through a broker, to one or more subscribers using topics. Topics are hierarchical UTF-8 strings. Clients publish, subscribe, or do both to a topic. In other words, topics are the way you register interest for incoming messages or how you specify where you want to publish the message.

Publish – Clients that send information to the broker to distribute to interested clients based on the topic name.

Subscribe – Clients tell the broker which topic(s) they’re interested in. When a client subscribes to a topic, any message published to the broker is distributed to the subscribers of that topic. Clients can also unsubscribe to stop receiving messages from the broker about that topic.

QoS – Quality of Service. Each connection can specify a quality of service to the broker with an integer value ranging from 0-2. The QoS does not affect the handling of the TCP data transmissions, only between the MQTT clients.

1. specifies at most once, or once and only once without requiring an acknowledgment of delivery. This is often referred to as fire and forget.

2. specifies at least once. The message is sent multiple times until an acknowledgment is received, known otherwise as acknowledged delivery.

3. specifies exactly once. The sender and receiver clients use a two-level handshake to ensure only one copy of the message is received, known as assured delivery.

How to Use MQTT in Home Automation?

In today’s world, automation has become important and is being used in many applications in our daily life. A Home Automation System (HAS) is a system where in home appliances or environment is controlled without much human involvement. It saves power, time and efforts and is more efficient than the conventional systems. Home environmental monitoring is a major Internet of Things (IoT) application, which involves monitoring the inside and outside environment of the home. By using IoT technology, user can create advanced Home Automations Systems that can improve the quality of the life.

Let’s take one such example.

Example:

Let’s say our Home Automation System consists of an electric light bulb that can be controlled with the help of a mobile device. User will use mobile application to toggle light switch and this state will be sent to the mqtt broker. On the other side, electric light bulb with help of microcontroller receives the state sent by user. For this to happen, the mobile device will first define the topic it wants to publish on, then only it will publish the message. Meanwhile, the microcontroller attached to the light bulb subscribes to the same topic. Then once it receives the message that the device has published, it toggles light based on the state. It might also want to publish to another topic so that other clients can monitor the state of that light. Again, the broker role here is to take the message and deliver it to subscribed clients.

Topics are the way you register interest for incoming messages or how you specify where you want to publish the message. Topics are represented with strings separated by a forward slash. Each forward slash indicates a topic level. And also remember topics are case-sensitive. If you want to control multiple light across multiple rooms, you will need to come up with unique topic for these lights. Let’s suppose If we want to toggle bedroom light, the topic will be home/bedroom/lamp.

Now that we have 2 clients the first mobile application will publish to the topic “home/bedroom/lamp” with a message of “on” or “off” every time we push a button from app. In our demo we are using “MyMqtt” app from google play store. The second client will subscribe to “home/bedroom/lamp” and respond to the message by turning a light bulb on or off. And later it will publish with a message of “on” or “off” to another topic like “bedroom/lamp/state” so that other clients can monitor the state of that light.

Cloud for MQTT brokers?

MQTT on-premise broker is a rather time-consuming and demanding solution for any project. If we are talking about the quick launch of the solution for the Internet of things, then launching your own data center with a server for the MQTT broker will require resources for the initial launch and installation. Plus, expanding and scaling the on-premises broker creates a lot of problems when migrating to new servers.

Now let’s compare this solution with a cloud service, where for a minimal cost you can quickly connect your project to high-quality service and start using the MQTT protocol. With a cloud service, you also get support for your project and an already configured security system on the server and the ability to increase your capacities almost without limit. Placing the MQTT broker in the cloud can be a successful strategy both for small projects and for corporate-level projects.

Cloud-based MQTT brokers are many, like:

• Amazon
• Cloud MQTT
• Google Cloud IoT Core
• Heroku
• IBM
• Microsoft Azure IoT

Conclusion

MQTT is a communication protocol based on a publish and subscribe system. It is simple to use and it is great for Internet of Things and Home Automation projects. On the other hand, choosing a right cloud provider to service MQTT also gives you a lot of options now and you can use the message broker in your existing cloud, or choose the most suitable for your task.

Big data with Hadoop

 

Data and Big Data

Tounderstand Hadoop and it’s working, it is necessary to know what Big data is. As the internet grew and with the advent of Web 2.0 & 3.0, demand for data grew. Data collected and shared increased. Particularly with social media and IoT , there is an exponential growth of data stored, transacted and accumulated. Big data is an umbrella term for a large volume of data.

Big Data Analytics

Data is information. Information that is used for planning, for scheduling, forecasting, budgeting, mitigating, etc. The larger the data set, more accurate the information. Hence the importance of Big data. Big data helps an enterprise to uncover information like buying patterns, demographic preferences, trends, correlations, etc. This in turns helps an enterprise discover new streams of revenue, provide better customer service, improve processes, etc. Tools that analyze such large data sets are known as Big data analytics applications.

Hadoop and Big Data

Hadoop is one such big data analytics application. A quick look at its history

Authors: Doug Cutting, Mike Cafarella

Developers: Apache Software Foundation

First Release: April 1st, 2006

Current Release: 3.3.0

Hadoop is a framework for the distributed processing of large data sets across clusters of computers using simple programming models. Whether it’s a single storage server or hundreds of machines across location, every node offers local computation and storage. With this parallel processing happening across hundreds of nodes which are distributed, data processing is faster than conventional databases. So basically rather than store all the data in a single or limited number of datasets, Hadoop clusters multiple nodes which can process even petabytes of data quickly

Hadoop Modules

Hadoop Distributed File System — HDFS is a file system that’s designed for fault tolerance and for deployment to the lowest end of hardware as well. Since Hadoop works with Big data, HDFS file systems are tuned for large data files. A major plus point to HDFS is that it provides mechanisms for applications to perform the computation on large data files where they are located instead of sending these large files to the application

Yet Another Resource Negotiator (YARN) — YARN is to applications what Houston is to Nasa. The control center!. Yarn manages nodes, cluster nodes, job status, scheduling, tasks. It also does extensive monitoring and reports.

MapReduce — Hadoop supports and uses replicated data. There are a number of benefits of this. Such as the availability of data at any time, fault tolerance etc. But data replication can lead to redundancy as well. This is solved by MapReduce. The “Map” in MapReduce job filters and sorts the input data-set into independent chunks which are processed by the map tasks parallely. Once mapping is done, reducing steps in and performs a aggregate or summary function to flush out any redundant data

And finally across the above 3 models and other models in Hadoop, Hadoop common libraries exist which can be used by all modules

Hadoop vs Conventional Database

Actually, Hadoop is not a database, so this comparison might feel like apples and oranges, but Hadoop as we saw is an ecosystem for distributed storage and thus has some similarities with database

Hadoop is ideally used for applications dealing with petabytes of data RDBMS are a good fit for applications with GigaBytes of data

RDBMS are mainly used for structured data. Well defined schemas and data definitions are mandatory for RDBMS. Hadoop works well with structured and unstructured data

Since Hadoop works mostly with semi-structured or unstructured data, it does not need SQL. Hadoop applications do support SQL as well in various lite versions, but it primarily uses HQL or Hive Query Language. Which is particularly good for Metastore. RDBMS use SQL

Hadoop is fully open source, whilst most RDBMS are licensed

Data and Data objects are stored as key-value pairs in Hadoop whereas as the name suggests in RDBMS, data objects are relational

Hadoop is well suited for data types like Audio, Video, Images, etc. RDBMS goes well with relational data such as commonly found in OLTP

Hadoop Ecosystem

Hadoop is not a single library or tool or framework which handles big data. It is essentially a platform or a suite or an ecosystem which provides tools, frameworks, libraries, models, objects to solve, analyse, gleam insight from Big data. The four main modules of Hadoop are listed above. In addition the main components of an Hadoop ecosystem are

Data Storage is a collection of hardware, commodity hardware which are known as nodes or clusters in the Hadoop Ecosystem

Data Processing involves computational techniques which processes the data stored in the data storage. This is built for optimization

Data access tools helps applications to write queries, execute commands on the data sets returned by Data processing tools

Needless to say, the Hadoop Ecosystem is constantly evolving and new tools keep getting added every day. What we have listed above are some of the popular ones

Extras

Hadoop framework is written in the Java programming language, with some native code in C and command-line utilities written as shell scripts.

The MapReduce model can be implemented in any language for any application that uses Hadoop.

Hadoop can be implemented on-premise or on the cloud.

More than half of the fortune 50 companies use Hadoop

https://www.prnewswire.com/news-releases/altiors-altrastar---hadoop-storage-accelerator-and-optimizer-now-certified-on-cdh4-clouderas-distribution-including-apache-hadoop-version-4-183906141.html

Conclusion

Data is only going to increase. With more applications coming up every day, data growth is one-way traffic. In such an environment, with the proven benefits exhibited by Hadoop, enterprises can and should start thinking about the virtues of Big data for their organizations. With the ability to host Hadoop on clouds, enterprises can get the best of both worlds, cloud storage and the power of Big data

Enterprise Digital Transformation Guide in the Post CovidWorld

 Business Transformation

Uncertainty is the new normal today, with no clear end as well. Businesses, small and large, have been besieged with unprecedented challenges. Any business is constantly in a state of improvement, upgrade and transition. This is necessitated primarily by the current pandemic, then changing market economics, customer preferences, government policies etc. Business transformation is inevitable. It can be defined as changes or modification to business operations necessitated by the need to improve or due to external factors such as Covid. Digital transformation is one of the key steps in Business transformation.

What is digital transformation?

Is it simply the adaptation of technology in an organisation?

Using software, tools and other systems to replace manual work?

Does adding more softwares and state of the art hardware transform a business digitally?

Far From it! Adapting technology makes the job easier for sure, but digital transformation is an entirely different subject.

Digital transformation can be defined as a time bound, holistic change in the business culture which involves people, process and physical aspects, driven by technology. It is a mindset that needs to be adopted by a business. To break it down, Digital transformation redefines a business operation, how employees work, how management perceives data and makes decisions. It alters the way of working in a more systematic and automated way with the singular goal of improving the business at all target levels. Digital transformation is not simply adding technology. Like replacing a software with another. Its planning, analysing, conducting & supporting business operations via technology. IDC has predicted that digital transformation will take up 50% of IT budgets by 2023

Need for Digital Transformation

Before we start on defining the need for digital transformation, why does an organisation need digital transformation? Is it mandatory? The answer to that is simply where does the business see itself in a few years down the line? If the goal is to scale, constantly get better, outrank the competition, diversify etc, then digital transformation is a must. Customer experience is being driven by digital mediums at a rapid pace. Cheaper hardware, faster data speed also drive the need for digital transformation. In the post covid world, uncertainty is written large. No one knows for sure if the old ways of working and doing business will return. Take for example RPA or Robotic Process Automation. Gartner has predicted the spending on RPA services to grow to $2.4 billion by 2022

Core Changes across business areas

1. Employee upskill
2. Business Process
3. Client and stakeholders empowerment
4. Partners, vendors
5. Business functions
6. Assets(Moveable and Immoveable)

Digital transformation helps customers, employees and management get information as and when they require. Frictionless experience goes a long way in retaining customers

With information available all the time, efficiency is achieved in process, output and decision making

Digital transformations biggest plus point is the data driven feedback which is priceless when it comes to planning. In a digital work environment, companies can pinpoint which areas, depts are bleeding, which are thriving, which need optimisation

Ease of operations invariably drive lower costs and improved bottom lines

Tools and Frameworks

Digital transformation is applied across a wide array of business functions such as sales, marketing, HR, administration and operations via

Cloud Computing for data storage

Big Data tools to mine data for analytics

Mobility to give customers omni channel experience

Mobility to give field staff best access to resources and sales

CRM tools and process for operations and administration

Digital Road Map

Any organisation that wants to radically change its course of working with Digital Transformation needs to have a Digital road map. This should be time bound and it should clearly list out the aims and objectives of the company in a short period of time frame and long period of time frame.

Prepare a team and leadership that defines the current state of capabilities and builds a road map for the transition. This will be based on company values, shared goals, budget and objectives. List out the benefits, list out the goals, list out the time and levels of transformation desired

Across verticals, identify opportunities, bottlenecks, pivot points and evaluate. Ask questions in this stage. For e.x In a delivery department, try identifying the most time consuming activity for delivery. Or in a marketing department, what’s the average time taken to respond a inquiry email

Identify vendors, evaluate solutions, measure the pros and cons of solutions. Make a comparative study and evaluate which is better. Compare over a large time frame and wide data set

Implement tech driven solutions like automation, RPA, cloud based backup etc at the basic level. Train the employees and field workers on how to adapt to this new mode. Take feedback and incorporate

Update processes to Agile on a per project basis. At each project constantly evaluate the time and efficiency of execution.

Digital transformation for the future

The UK Govt has a beautiful slogan “Digital by Default” which summarises the importance of digital service consumption whether private or public in an increasing digital world. As mentioned above, Digital transformation is not limited to mere adoption of technology in the existing work culture of the organisation. It is a change in the mindset of the organisation in an holistic manner across functions and departments