Introduction to Characteristics of Modern Data Warehouse
In the modern era, we have to handle the following requirement challenges raised by business, along with the four V’s.
Data Velocity
“If an organization is a car, they need real-time data to navigate both external and internal bumps and reach their goals timely and safely.”
—Anonymous
Organizations across all industries want to capture data frequently, fast to faster. As the event happens, management wants to know the event happened, and its intensity and repercussions. As the trends go, reporting data requirements has moved from quarterly to monthly, monthly to weekly, weekly to daily, daily to hourly, to every couple of seconds. It is important to mention that there is no industry left untouched by high-speed reporting of business data. Health care, automobiles, media, environmental, and real-time campaign data are only a few of the many I could mention.
Real-time data relates to the high velocity/speed of data. As the name suggests, data is captured in real time, processed in real time, and reported in real time. This processing can be done near the place where data are getting generated to save microseconds. The storage of data is generally done for a short stint, but you can always hook up a parallel stream to store the data for long-term analysis.
In real-time data, less latency is expected in write operations, but ideally is seen in both read and write operations. This reduces latency in the overall data pipeline, from ingestion to output, in the full data-flow cycle. Latency is defined as the time between when a request for data is sent and when a response is provided back to the requester. As time goes by, with the speed of the internet increasing, the speed at which data moves has increased.
Except for internet speed, computation is the most significant contributor to latency. With the advent of cloud computing, processing speed has increased by leaps and bounds. For example, social media on mobile can share location data, video, and images with other users with ease and speed.
Many industries exhibit this high-velocity usage, such as social networking conversations; GPS feeds; voice-to-text feeds; IOT (Internet of Things) feeds from devices from home like bulb, fan, door, etc.; water supply consumption; robotic manufacturing; face-recognition; cyber security threat assessment , and more.
