|By Jnan Dash||
|January 16, 2014 02:15 PM EST||
I joined 600 people last night at a session sponsored by Hive to listen to Doug Cutting, the creator of Hadoop. Currently he is the chief architect at Cloudera and a director at Apache Software Foundation. The hall at NetApp facility was overflowing with an eager audience. Doug spoke about the future of data management.
He narrated a brief history of Hadoop, how it was founded and how far it has come. As everyone knows, the pedigree of Hadoop came from Google’s GFS (Google File System, now HDFS) and Map-Reduce programming. Here are the key predictions he made:
- Hadoop has grown to become the de-facto standard for Big Data. He had anticipated IBM and Microsoft to come up with alternative designs to compete with Hadoop, but that never happened. Both companies plus Oracle, HP and other players have endorsed Hadoop as the platform.
- Hadoop will become the center of data management in future. It will not be the original HDFS+MR layers, but a whole new ecosystem called “The Enterprise Data Hub”. There will be an explosion of products surrounding Hadoop (all open systems). He cited examples of Pig, Hive, Sqoop, etc. Currently many SQL implementations over HDFS are coming up.
- Will there be OLTP (Transactional systems) on Hadoop? He said yes. Current implementation of Impala (from Cloudera) has SQL on HDFS with Map-Reduce on top is proving quite efficient in ETL workloads. Several customers have started migrating from legacy world to Impala.
- The new project at Google called Spanner is also leading the way to a future OLTP system distributed across the globe. This work will propel future additions to the Hadoop ecosystem.
- He explained the big advantage of Open systems architecture and why that will become the norm over proprietary systems.
- The future Hadoop ecosystem (Enterprise Data Hub) will be a threat to the current incumbents like Oracle, MySQL, SQL server, DB2, and Vertica. Current challenges of weak security and lack of standardization will be addressed eventually.
Doug is an engaging speaker and clearly showed he knows his subject well. I have my doubts on his future predictions, as DBMS’s take a long time to mature and provide all the critical functions for mission-critical applications. We have learnt that over the last 4 decades. Hadoop is still primarily a batch system doing offline analytics. Moving from there to do real-time production workload is quite a jump and will take many years to accomplish.
Then there are the new breed of highly efficient NoSQL databases like MongoDB that are being deployed to create “systems of engagement” at large enterprises. Also, the incumbents are not sitting idle either with a total market size of $30 Billion dollars. It is funny to remember that our tax records are still managed by Model 204 at IRS, a DBMS created during the 1960s. Switching databases is extremely cumbersome and not for the faint-hearted. Doug did say that future spending will steer more towards Hadoop.
Given the challenges of Big Data and the rapid adoption of Hadoop, we will watch this space as it unfolds over next couple of years.
- Innodisk | Efficiencies for Cloud Hardware at Cloud Expo New York
- Join Gartner, IBM, + AWS at AppSphere and save $200 when you register in August!
- In 2014 Big Data Investments Will Account for Nearly $30 Billion - Eventually Accounting for $76 Billion by 2020 End
- Global Cloud Security Market Growing at 15.7% CAGR to 2020: Forecast & Analysis in Research Report Available at ReportsnReports.com
- Video: DevOps and Security
- Worldwide Indoor Location Market Growing at 46.0% CAGR to 2019 Says a New Research Report Available at RnRMarketResearch.com
- Flexera Software's InstallAnywhere 2014 Simplifies Multi-Platform Installation for Physical, Virtual and Cloud Environments
- Mobility News Weekly – Week of August 3, 2014
- Searchmetrics Drives Over 200% World-Wide Growth As More Business Leaders Begin To Recognize The Value Of Search
- Mobility News Weekly – Week of August 17, 2014
- Digital Transformation's Impact on Enterprise Mobility and App Design Strategies
- Web Analytics Market by Solution (Search Engine Tracking & Ranking, Heat Map Analytics, Marketing Automation, Behavior Based Targeting) & by Services (Professional Services, Support & Maintenance) - Worldwide Forecasts & Analysis (2014 - 2019)
- Mobile Commerce News Weekly – Week of August 3, 2014
- Red Hat To Present At Internet of @ThingsExpo
- Mobile Cyber Security News Weekly – Week of August 10, 2014
- Where Are RIA Technologies Headed in 2008?
- Dolphin Announces Open API With Over 50 Add-ons Including Dropbox and Wikipedia
- Cloud People: A Who's Who of Cloud Computing
- 21st century Modern Alarm systems continue to play a key role in various institutions and industries
- SEO/SEM Tips & Tricks: How and When Should You Submit Your Website to Google?
- Cloud Expo 2011 East To Attract 10,000 Delegates and 200 Exhibitors
- Tips For Press Releases in Reputation Management from Industry Veteran Brandon Hopkins
- Yahoo! to Keynote 4th Cloud Expo: Accelerating Innovation with Cloud Computing
- Google Version 2.0: Googzilla - The Calculating Predator
- ManageWP Powers Over 100,000 WordPress Sites Within Three Months of Launch
- Ulitzer’s Amazing First 30 Days in Public Beta
- Google's Competitive Advantage: It Leverages "The Power of Free"
- Ulitzer vs. Ning - a Quick Review
- AOL To Enhance Video Search Engine by Adding RSS Feeds
- Confessions of a Ulitzer Addict