Welcome!

Cognitive Computing Authors: Yeshim Deniz, Elizabeth White, Liz McMillan, Pat Romanski, Zakia Bouachraoui

Related Topics: @CloudExpo

@CloudExpo: Article

Big Data – The State of Affairs

Big Data is here to stay, but do we have the tools to efficiently process it?

Many products are available as open source or proprietary products that can handle Big Data. Which one is best fit for this task?

Today's classic RDBMSs and tools are able to quickly load the data, process it and present results in an easy to understand format.  You can use SQL or programmatic interface to process the data randomly or in batch; RDBMS's keep data safe, protected against hardware and software failures.

Standards tools and products are not able to cope with Big Data requirement, which is not dissimilar to  what is involved in processing today's regular data sets, just on a much bigger scale. Mainstream companies like telcos, financials, web companies as well as government are reaching the limit of what  can be efficiently processed by classic RDBMS techhnologies.

When it comes to picking a proper platform and tools to handle your Big Data there are a couple of possible choices:

  • Oracle Exadata - it doesn't fit economical mandate; Exadata's weak link and bottleneck is its reliance on classic Oracle RDBMS
  • NoSQL databases -  too immature, they offer no SQL or similar random access query language ( you are presently forced to write  programs to access your data ); often achieve scale-out by not implementing all elements of ACID, CAP
  • Hadoop/MapReduce and related open source ecosystem ( Pig, Hive, HBase ) -  useful for cheap data storage on commodity hardware and batch processing; they offer no efficient, non-programmatic random access
  • proprietary MPP databases running on commodity hardware ( Vertica, Aster Data, Greenplum )  - very fast and can provide random, SQL  access to big data; their management features and general feature sets are immature
  • proprietary MPP databases running on specialized hardware ( Teradata ) - fairly expensive ( don't run on commodity hardware )
  • new platforms that will or are trying to emulate Google Percolator, Dremel  ( latest Google technologies dealing with big data ACID compliant transactions and reporting ), similarly to how Hadoop originated from  Google GFS and MapReduce.

We would say that there is no single, generic product or platform available today that can handle this task. Depending on your needs you have to deploy  and combinne quite a few of technologies to bring you closer to achieving end-to-end efficient, comprehensive processing of Big Data. You will quite likely have to custom build solutions that will fit your particular needs as off-the-shelf solutions are still immature, incomplete or not available.

Big Data is an area of growth and innovation, so current picture is bound to change as new products and technologies appear, bringing us closer to the ultimate goal of routine, efficient processing of Big Data.

More Stories By Ranko Mosic

Ranko Mosic, BScEng, is specializing in Big Data/Data Architecture consulting services ( database/data architecture, machine learning ). His clients are in finance, retail, telecommunications industries. Ranko is welcoming inquiries about his availability for consulting engagements and can be reached at 408-757-0053 or [email protected]

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


IoT & Smart Cities Stories
DXWorldEXPO LLC announced today that ICOHOLDER named "Media Sponsor" of Miami Blockchain Event by FinTechEXPO. ICOHOLDER gives detailed information and help the community to invest in the trusty projects. Miami Blockchain Event by FinTechEXPO has opened its Call for Papers. The two-day event will present 20 top Blockchain experts. All speaking inquiries which covers the following information can be submitted by email to [email protected] Miami Blockchain Event by FinTechEXPOalso offers sp...
DXWordEXPO New York 2018, colocated with CloudEXPO New York 2018 will be held November 11-13, 2018, in New York City and will bring together Cloud Computing, FinTech and Blockchain, Digital Transformation, Big Data, Internet of Things, DevOps, AI, Machine Learning and WebRTC to one location.
@DevOpsSummit at Cloud Expo, taking place November 12-13 in New York City, NY, is co-located with 22nd international CloudEXPO | first international DXWorldEXPO and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time t...
Headquartered in Plainsboro, NJ, Synametrics Technologies has provided IT professionals and computer systems developers since 1997. Based on the success of their initial product offerings (WinSQL and DeltaCopy), the company continues to create and hone innovative products that help its customers get more from their computer applications, databases and infrastructure. To date, over one million users around the world have chosen Synametrics solutions to help power their accelerated business or per...
Charles Araujo is an industry analyst, internationally recognized authority on the Digital Enterprise and author of The Quantum Age of IT: Why Everything You Know About IT is About to Change. As Principal Analyst with Intellyx, he writes, speaks and advises organizations on how to navigate through this time of disruption. He is also the founder of The Institute for Digital Transformation and a sought after keynote speaker. He has been a regular contributor to both InformationWeek and CIO Insight...
When talking IoT we often focus on the devices, the sensors, the hardware itself. The new smart appliances, the new smart or self-driving cars (which are amalgamations of many ‘things'). When we are looking at the world of IoT, we should take a step back, look at the big picture. What value are these devices providing. IoT is not about the devices, its about the data consumed and generated. The devices are tools, mechanisms, conduits. This paper discusses the considerations when dealing with the...
Machine learning has taken residence at our cities' cores and now we can finally have "smart cities." Cities are a collection of buildings made to provide the structure and safety necessary for people to function, create and survive. Buildings are a pool of ever-changing performance data from large automated systems such as heating and cooling to the people that live and work within them. Through machine learning, buildings can optimize performance, reduce costs, and improve occupant comfort by ...
Digital Transformation is much more than a buzzword. The radical shift to digital mechanisms for almost every process is evident across all industries and verticals. This is often especially true in financial services, where the legacy environment is many times unable to keep up with the rapidly shifting demands of the consumer. The constant pressure to provide complete, omnichannel delivery of customer-facing solutions to meet both regulatory and customer demands is putting enormous pressure on...
Bill Schmarzo, Tech Chair of "Big Data | Analytics" of upcoming CloudEXPO | DXWorldEXPO New York (November 12-13, 2018, New York City) today announced the outline and schedule of the track. "The track has been designed in experience/degree order," said Schmarzo. "So, that folks who attend the entire track can leave the conference with some of the skills necessary to get their work done when they get back to their offices. It actually ties back to some work that I'm doing at the University of San...
CloudEXPO New York 2018, colocated with DXWorldEXPO New York 2018 will be held November 11-13, 2018, in New York City and will bring together Cloud Computing, FinTech and Blockchain, Digital Transformation, Big Data, Internet of Things, DevOps, AI, Machine Learning and WebRTC to one location.