|By Robert Eve||
|January 3, 2012 06:30 AM EST||
Data Virtualization: Going Beyond Traditional Data Integration to Achieve Business Agility is the first book published on the topic of data virtualization. Along with an overview of data virtualization and its advantages, it presents ten case studies of organizations that have adopted data virtualization to significantly improve business decision making, decrease time-to-solution and reduce costs. This article describes data virtualization adoption at one of the enterprises profiled, Northern Trust.
Northern Trust is a leading provider of innovative fiduciary, investment management, private banking, wealth management and worldwide trust and custody services. Clients include corporations, institutions, families and individuals in more than 40 countries.
Based in Chicago, Illinois, the company has over 13,000 employees. At the end of 2010, the company had assets under custody of $4.1 trillion, assets under management of $643.6 billion and banking assets of $83.8 billion. Annual revenues in 2010 were almost $3.7 billion.
For this case study, we interviewed Leonard J. Hardy, Senior Vice President, Operations and Technology. Hardy is a member of Northern Trust's Rapid Solutions group, which provides a solution architecture and consulting facility to other Northern Trust application development groups.
Hardy also manages the company's Integration Competency Center (ICC), which helps other application areas figure out the best way to integrate data. The ICC supports several tools for this, including data virtualization, ETL and data integration best practices. Hardy participated in this case study project in his ICC role.
The Business Problem
An important line of business for Northern Trust is providing outsourced investment management operations for corporate customers, typically investment management firms and banks.
These institutional customers continue to provide front-office investment management functions for their end clients, but outsource all or some of the middle-office and back-office functions to Northern Trust. By contracting these functions out to Northern Trust, the institution does not have to invest in the resources, assets and skills necessary to provide the functions internally. In return, Northern Trust provides guaranteed levels of quality, service, resilience and value-to-cost criteria and management.
The Northern Trust line of business responsible for investment management outsourcing is called Investment Operations Outsourcing (IOO).
The business problem for IOO was the fact that it had a large number of new institutional customers in the outsourcing pipeline and simply could not implement them fast enough.
As Hardy stated, "If we could get customers on board more quickly, we would realize several key benefits. First, revenue would increase. Second, we could go down the pipeline faster. We had customers already signed up for outsourcing who were waiting to be converted and we had to find a way to do that faster. That, in turn, would allow us to open up the pipeline and take advantage of huge new business opportunities in the marketplace."
The Technical Problem
Existing IOO client reporting capabilities had evolved within a traditional systems infrastructure and were both inefficient and inflexible, thereby extending the time it took to set up the end client reporting function for each customer.
For example, value-added client data was stored in multiple separate databases, including legacy, mainframe systems. In addition, some of the business logic to interpret the data structures and create reports was embedded in stored procedure code. The only way to access the data and interpret it correctly was to go through the stored procedures. There was no data warehouse or data mart that consolidated all this data and business logic for reporting purposes.
As a result, according to Hardy, "the old way of reporting required technology intervention for every single customer implementation. As you can imagine, each institution has its own formats, labels, graphics and fonts. A technology expert proficient in our development tools had to get involved and make changes to our reporting process for every customer. The time-to-market on that was very long. We needed the ability to bring on new customers without having to make technology changes for each one."
Two Options Evaluated
IOO considered two solution architectures to meet their business and technical challenges.
Physical Consolidation: One solution for IOO was to physically integrate the data using ETL and data warehousing technology. But that would require pulling all of the business logic out of the stored procedures and figuring out how to get all of the data into one database and have the structure make sense, a massive project that could take up to a year and a half.
Data Virtualization: Instead, IOO took a more innovative approach and decided to virtually integrate the data using the Composite Data Virtualization Platform as a virtual data warehouse. Placing Composite in the middle between the client reporting tool and the back-end data sources would enable IOO to quickly abstract the data into the data virtualization layer, reuse the existing business logic to access the data and generate all of the data needed for end client reporting. This approach required just seven months.
The Data Virtualization Solution - Architecture
Northern Trust's IOO data virtualization solution is a virtual data warehouse implemented with the Composite Data Virtualization Platform combined with a new client reporting front end. The major architecture components include:
Data sources - There are several physical data stores in the IOO environment.
- Transaction data includes all of the detailed transactions for each end-client account: buys, sells, interest income, dividends, etc.
- Valuation data includes, for each end-client account, how much it is worth today, how much it was worth yesterday, etc., based on the transaction data.
- Enterprise asset data is a reference file that contains data about each asset in the client accounts, such as ticker symbol and investment category (equity, bond, money market, etc.). This data comes from a number of different third-party providers.
- Performance data includes very complicated investment performance calculations that permit performance comparisons between accounts, such as one account against accounts of similar value or with similar investment strategies.
- Transaction, valuation and enterprise asset data is stored in IBM DB2 and accessed via COBOL stored procedures.
- Performance data is in Oracle and accessed via web services that call C code.
Data virtualization layer: The virtual data warehouse built using the Composite Data Virtualization Platform abstracts the data in the IOO data stores and creates a unified, virtual view that makes them appear as a single physical data store.
When the reporting platform issues a query to retrieve data from one or more data stores, Composite queries the data by calling the appropriate stored procedures and/or web services, combines the data (for example, associates all of the transactions that make up a valuation and the important performance data that applies to that valuation) and delivers it to the reporting platform.
Consuming applications: The primary consuming application is the client reporting platform. This is the toolset that IOO's internal business/operations partners now use to build, format and produce the reports for IOO's customers and their clients.
Another benefit of the reporting tool is that it supports a data dictionary and business views of the data that directly map to views within the data virtualization layer. These business views, or building blocks, conform the data into reusable components that the business analysts can use to create custom reports.
A key point here is that the solution has enabled IOO to transfer the time-consuming report customization work that previously had to be done for each new institutional customer by very high-priced and very busy technical resources (applications programmers) to IOO's operations partners (business analysts).
As Hardy summarized it, "We have been successful in our goal of taking the technology out of the outsourcing equation. As long as we supply the right data through Composite, the reporting tool handles all formatting, graphing and other functionality that we used to have to hard code within the application program. One reason we had such a big bottleneck in implementing outsourcing customers was the fact that we didn't have enough programmers in our application development group to handle the customization requirements. Business analysts in our operations group can now easily process the building blocks necessary to customize reports."
By combining the capabilities of data virtualization and the new reporting tool, IOO has been able to navigate through and abstract a very complex set of layers of translation to connect the raw IOO data structures to business metadata and finished customer reports.
The Data Virtualization Solution - Best Practices
Hardy offered some best practices and lessons learn based on his experience working with IOO.
Start with a focused project: Hardy stated that a focused initial effort is a key to success. "Don't start with a project to make all enterprise data available via a service." To date, IOO has completed four institutional customer implementations with over ten more in progress or planned.
Centralize support for data virtualization and development: Giving ICC responsibility for the data virtualization technology provided economies of scale and enabled the company to accelerate up the best practices learning curve in effectively implementing the technology in business solutions.
Understand the data: According to Hardy, data virtualization is no different than any other development or physical data integration tool. "The design and architecture of the solution is where you need to spend time. The old carpenter's adage of ‘measure twice and cut once' applies here. Don't just dive in and start coding views. Make sure you really understand the data. This is the key to success in an effort like this."
Educate and support the business: Hardy said it is also important to allocate time to consult with the business and make sure they understand the data. "In our case, they are the ones designing the reports but they don't necessarily have in-depth knowledge about the data in its raw form. So we need to help them understand the raw data and make sure that when the data gets to the end tool, people understand what they are looking at."
Manage business expectations: Although data virtualization is a capability that can get the job done faster, the organization still has to go through the same steps to implement the solution: analysis, design and architecture. "In our case, we could say it saved us a year and a half, but it did take seven months. So it will take time no matter how you do it."
Summary of Benefits
Hardy described the major benefits Northern Trust has achieved with the new IOO client reporting architecture based on data virtualization.
Faster time-to-market increases customer satisfaction and revenue: Data virtualization has dramatically reduced the time it takes to implement a new outsourcing customer by 50%. Moving customers through the pipeline faster improves overall customer satisfaction and gives IOO the capacity to bring in even more business and revenue.
Single point of access simplifies work and cut costs: The fact that the reporting tool, or any consuming application, has a single data access point regardless of the format, access method (SQL, web service, stored procedure, etc.) or data source significantly reduces application complexity and development and maintenance time and cost.
Reuse benefits grow with adoption: The ability to create reusable data services that can be shared by all applications also reduces costs.
Long run flexibility is improved: Northern Trust is embarking on a three-year project to replace all of its underlying data stores with modern data warehouses and data marts. By putting data virtualization in the middle, thereby decoupling the back-end data from the consuming applications, IOO will not have to change anything that has been defined in the reporting tool when it swaps out the back-end data stores. It will only need to change the views to consume the new data sources.
• • •
Editor's Note: Robert Eve is the co-author, along with Judith R. Davis, of Data Virtualization: Going Beyond Traditional Data Integration to Achieve Business Agility, the first book published on the topic of data virtualization. The complete Northern case study, along with nine others enterprise are available in the book.
The Internet of Things will challenge the status quo of how IT and development organizations operate. Or will it? Certainly the fog layer of IoT requires special insights about data ontology, security and transactional integrity. But the developmental challenges are the same: People, Process and Platform and how we integrate our thinking to solve complicated problems. In his session at 19th Cloud Expo, Craig Sproule, CEO of Metavine, will demonstrate how to move beyond today's coding paradigm ...
Jun. 29, 2016 07:00 PM EDT Reads: 484
The cloud market growth today is largely in public clouds. While there is a lot of spend in IT departments in virtualization, these aren’t yet translating into a true “cloud” experience within the enterprise. What is stopping the growth of the “private cloud” market? In his general session at 18th Cloud Expo, Nara Rajagopalan, CEO of Accelerite, explored the challenges in deploying, managing, and getting adoption for a private cloud within an enterprise. What are the key differences between wh...
Jun. 29, 2016 05:30 PM EDT Reads: 1,045
SYS-CON Events has announced today that Roger Strukhoff has been named conference chair of Cloud Expo and @ThingsExpo 2016 Silicon Valley. The 19th Cloud Expo and 6th @ThingsExpo will take place on November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. "The Internet of Things brings trillions of dollars of opportunity to developers and enterprise IT, no matter how you measure it," stated Roger Strukhoff. "More importantly, it leverages the power of devices and the Interne...
Jun. 29, 2016 04:15 PM EDT Reads: 468
"We work in the area of Big Data analytics and Big Data analytics is a very crowded space - you have Hadoop, ETL, warehousing, visualization and there's a lot of effort trying to get these tools to talk to each other," explained Mukund Deshpande, head of the Analytics practice at Accelerite, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
Jun. 29, 2016 04:15 PM EDT Reads: 436
The idea of comparing data in motion (at the sensor level) to data at rest (in a Big Data server warehouse) with predictive analytics in the cloud is very appealing to the industrial IoT sector. The problem Big Data vendors have, however, is access to that data in motion at the sensor location. In his session at @ThingsExpo, Scott Allen, CMO of FreeWave, discussed how as IoT is increasingly adopted by industrial markets, there is going to be an increased demand for sensor data from the outermos...
Jun. 29, 2016 04:00 PM EDT Reads: 402
UAS, drones or unmanned aircraft, no matter what you call them — this was their week. Our news stream was flooded with updates on the newly announced rules and regulations for commercial UAS from the FAA. So, naturally we have dedicated this week’s top news round up to highlight some of our favorite UAS stories.
Jun. 29, 2016 03:02 PM EDT Reads: 300
Internet of @ThingsExpo has announced today that Chris Matthieu has been named tech chair of Internet of @ThingsExpo 2016 Silicon Valley. The 6thInternet of @ThingsExpo will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
Jun. 29, 2016 03:00 PM EDT Reads: 392
CenturyLink has announced that application server solutions from GENBAND are now available as part of CenturyLink’s Networx contracts. The General Services Administration (GSA)’s Networx program includes the largest telecommunications contract vehicles ever awarded by the federal government. CenturyLink recently secured an extension through spring 2020 of its offerings available to federal government agencies via GSA’s Networx Universal and Enterprise contracts. GENBAND’s EXPERiUS™ Application...
Jun. 29, 2016 02:00 PM EDT Reads: 416
Basho Technologies has announced the latest release of Basho Riak TS, version 1.3. Riak TS is an enterprise-grade NoSQL database optimized for Internet of Things (IoT). The open source version enables developers to download the software for free and use it in production as well as make contributions to the code and develop applications around Riak TS. Enhancements to Riak TS make it quick, easy and cost-effective to spin up an instance to test new ideas and build IoT applications. In addition to...
Jun. 29, 2016 02:00 PM EDT Reads: 673
In addition to all the benefits, IoT is also bringing new kind of customer experience challenges - cars that unlock themselves, thermostats turning houses into saunas and baby video monitors broadcasting over the internet. This list can only increase because while IoT services should be intuitive and simple to use, the delivery ecosystem is a myriad of potential problems as IoT explodes complexity. So finding a performance issue is like finding the proverbial needle in the haystack.
Jun. 29, 2016 01:15 PM EDT Reads: 379
When people aren’t talking about VMs and containers, they’re talking about serverless architecture. Serverless is about no maintenance. It means you are not worried about low-level infrastructural and operational details. An event-driven serverless platform is a great use case for IoT. In his session at @ThingsExpo, Animesh Singh, an STSM and Lead for IBM Cloud Platform and Infrastructure, will detail how to build a distributed serverless, polyglot, microservices framework using open source tec...
Jun. 29, 2016 12:30 PM EDT Reads: 567
Presidio has received the 2015 EMC Partner Services Quality Award from EMC Corporation for achieving outstanding service excellence and customer satisfaction as measured by the EMC Partner Services Quality (PSQ) program. Presidio was also honored as the 2015 EMC Americas Marketing Excellence Partner of the Year and 2015 Mid-Market East Partner of the Year. The EMC PSQ program is a project-specific survey program designed for partners with Service Partner designations to solicit customer feedbac...
Jun. 29, 2016 11:00 AM EDT Reads: 606
Apixio Inc. has raised $19.3 million in Series D venture capital funding led by SSM Partners with participation from First Analysis, Bain Capital Ventures and Apixio’s largest angel investor. Apixio will dedicate the proceeds toward advancing and scaling products powered by its cognitive computing platform, further enabling insights for optimal patient care. The Series D funding comes as Apixio experiences strong momentum and increasing demand for its HCC Profiler solution, which mines unstruc...
Jun. 29, 2016 11:00 AM EDT Reads: 538
The IoT is changing the way enterprises conduct business. In his session at @ThingsExpo, Eric Hoffman, Vice President at EastBanc Technologies, discussed how businesses can gain an edge over competitors by empowering consumers to take control through IoT. He cited examples such as a Washington, D.C.-based sports club that leveraged IoT and the cloud to develop a comprehensive booking system. He also highlighted how IoT can revitalize and restore outdated business models, making them profitable ...
Jun. 29, 2016 11:00 AM EDT Reads: 507
IoT offers a value of almost $4 trillion to the manufacturing industry through platforms that can improve margins, optimize operations & drive high performance work teams. By using IoT technologies as a foundation, manufacturing customers are integrating worker safety with manufacturing systems, driving deep collaboration and utilizing analytics to exponentially increased per-unit margins. However, as Benoit Lheureux, the VP for Research at Gartner points out, “IoT project implementers often ...
Jun. 29, 2016 10:45 AM EDT Reads: 573
"delaPlex is a software development company. We do team-based outsourcing development," explained Mark Rivers, COO and Co-founder of delaPlex Software, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
Jun. 29, 2016 10:30 AM EDT Reads: 525
In his general session at 18th Cloud Expo, Lee Atchison, Principal Cloud Architect and Advocate at New Relic, discussed cloud as a ‘better data center’ and how it adds new capacity (faster) and improves application availability (redundancy). The cloud is a ‘Dynamic Tool for Dynamic Apps’ and resource allocation is an integral part of your application architecture, so use only the resources you need and allocate /de-allocate resources on the fly.
Jun. 29, 2016 10:00 AM EDT Reads: 1,307
Connected devices and the industrial internet are growing exponentially every year with Cisco expecting 50 billion devices to be in operation by 2020. In this period of growth, location-based insights are becoming invaluable to many businesses as they adopt new connected technologies. Knowing when and where these devices connect from is critical for a number of scenarios in supply chain management, disaster management, emergency response, M2M, location marketing and more. In his session at @Th...
Jun. 29, 2016 09:45 AM EDT Reads: 1,279
Machine Learning helps make complex systems more efficient. By applying advanced Machine Learning techniques such as Cognitive Fingerprinting, wind project operators can utilize these tools to learn from collected data, detect regular patterns, and optimize their own operations. In his session at 18th Cloud Expo, Stuart Gillen, Director of Business Development at SparkCognition, discussed how research has demonstrated the value of Machine Learning in delivering next generation analytics to imp...
Jun. 29, 2016 09:45 AM EDT Reads: 961
In his keynote at 18th Cloud Expo, Andrew Keys, Co-Founder of ConsenSys Enterprise, provided an overview of the evolution of the Internet and the Database and the future of their combination – the Blockchain. Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life sett...
Jun. 29, 2016 09:15 AM EDT Reads: 1,469