The terms “big data”, “machine learning” and “artificial intelligence” have been trending in the drug discovery space for several years, both in mainstream media and academic press. These new technologies are believed to make drug discovery cheaper, faster, and more productive, while also promising to enable personalized medicine approaches (e.g. advanced biomarkers, improved patient stratification, etc).

But how is “magic” happening, after all?

First, let’s briefly review some of the basic concepts in the heart of new technologies.

Image credit: gorodenkoff , iStock

Big Data: Volume, Velocity, and Variety

The term “big data” by itself is more of a marketing nature. It describes an abstract concept of having large volumes of data obtained from various channels in multiple formats, which needs to be arranged in such a way that it can be possible to quickly access, search, update, and analyze it to output useful information.

Today "big data" is a central strategic concept in most industries — mainly because of the exponential rate of data generation globally — nearly 90% of all data currently available on Earth has been created in the last two years. 

For example, a big data analytics project may attempt to forecast the weather by correlating simultaneous data streams from myriads of different sensors, satellites and observation stations. All this information comes in different data types, including structured data, such as SQL database stores (tables of data with columns and rows); unstructured data, such as document files (satellite photos, for example); or streaming data from sensors.

A crucial point of the big data concept is, also, a speed at which the information must be processed and analyzed.

The computing power required to quickly process huge volumes and varieties of data cannot be achieved via traditional data management architectures using a single server or a server cluster. This can potentially demand hundreds or thousands of servers that can distribute the work and operate collaboratively. It is virtually impossible for most of the organizations to create and maintain anything of this kind.

So, although the concept of big data has been around for a long time — a term “information explosion” was first introduced as early as in 1941, according to the Oxford English Dictionary — a real data processing revolution happened much later, when a public cloud computing emerged as a primary tool for hosting big data analytics projects.

Nowadays, a public cloud provider can store petabytes (103 terabytes) of data and scale up thousands of servers for as long as it is needed to accomplish the big data project. This is available for a reasonable price and can be utilized by any organization in the world.

Topics: Emerging Technologies    Industry Trends