Author: Josh Bae and Anat Fraenkel | Credera
Author: Josh Bae and Anat Fraenkel | Credera
NRG’s enterprise data and analytics team had a vision to democratize data and act as a catalyst for a data-driven culture. Credera built a data marketplace over NRG’s Databricks and AWS data platform to foster data consumer trust and collaboration as well as build transparency for their leadership team.
By surfacing data assets, ownership, data lineage, and other metadata in an intuitive domain-specific user interface (UI), business managers and data analysts across the organization were better able to find, trust, and leverage data. Additionally, the NRG executive team obtained increased access and understanding of the data products that support their business.
Figure 1: The data marketplace is envisioned to be a one-stop shop for all data across the organization.
NRG’s executive team set out an ambitious strategy to move from a power generator to a bundled provider of home services by 2025. Executing this strategy required a unified data-driven approach across the company’s current and future business units. However, buy in to a centralized data-driven approach varied across business units that had traditionally maintained their own data silos.
To break down these barriers, they chose to build a custom, internal data marketplace that allows their executive leadership team and stakeholders across all business units and brands to easily discover, understand, trust, and use data. An intuitive, NRG-specific experience was important to the organization’s enterprise data and analytics team as they saw this as a critical component to driving adoption.
NRG had built (and continues to enhance) an enterprise data platform on AWS and Databricks, including metadata management and data governance processes using the Collibra data catalog. Credera partnered with NRG to build the data marketplace to serve as the front-door to the lake by providing an enhanced view of the metadata in Collibra.
Using design thinking, Credera defined personas and requirements, designed the front-end user experience, and built a custom-fit marketplace that has led to increased trust in the data lake’s offerings. Executives across the business are now able to find the data products they need, therefore improving decision-making capabilities.
The dedicated front end of the marketplace improved the data shopping experience. It gave the executive team and key businesses leaders across the organization better awareness, access, understanding, and trust of critical data and insights that help drive the business.
To move toward a data-driven culture, NRG needed to increase awareness and adoption of its data lake and improve data consumer trust.
Figure 2: The data marketplace solves several data pain points.
NRG had the vision to create a data-driven culture where team members at every level of the organization were empowered to use, understand, and trust data to drive informed business decisions that better served customers and connected dots across their value chain.
To help the company transition toward a data-driven culture, they needed to establish a robust data foundation that would:
Provide visibility and transparency to key business reports and metrics.
Promote collaboration across its enterprise hub and business unit spokes.
Facilitate the creation of seamless data products that provide valuable insights.
The company initially engaged Credera during the implementation of their AWS Lakehouse and successfully instantiated a centralized data lake and modern data governance process for capturing metadata. The next step was to enable the rest of the organization to take advantage of the newly built enterprise data lake.
The keys to this adoption was transitioning to a hub-and-spoke model and tackling the common challenges the company historically faced with data collaboration, which included:
Heavy reliance on tribal knowledge: People needed to know the right people to ask and the right way to ask to find the data they needed. This created barriers to collaboration and confused new team members.
Inconsistent business and data definitions: Even if people used data from the same source, inconsistent business and data definitions caused multiple versions of the truth when generating models and insights for answering key business questions.
Data silos: Data silos were set up based on specific business units and brands, which hindered collaboration and created barriers to data fluency for non-technical users, who did not know how to access, understand, or use the data.
Low visibility into existing data: New reports and data were made without knowledge of existing reports and data sources, which wasted time and caused confusion due to duplicate data products.
To ensure the data marketplace would be adopted—a key to data democratization and data-culture—Credera adopted a design-thinking approach. Design thinking is a process that focuses on empathizing with the end users to create a product that users are excited to adopt. Following these stages closely enabled us to create and mature a minimum viable product (MVP) that addressed NRG users’ data discovery related pain points.
Figure 3: The double-diamond innovation process used to define an MVP for the marketplace.
Credera started to define the MVP by thinking of the users first through these four steps:
Discover: We refined the problem statement and conducted user research to understand users’ needs and desires.
Define: We ideated and defined key features based on user research and our understanding of the existing data platform architecture. We also defined MVP scope and broke down key features into functional requirements.
Develop: Based on functional requirements, we developed our solution as well as designed and iterated prototypes to develop a best-in-class front-end experience.
Deliver: We built the designed front-end experience with a modern and scalable back-end solution.
Below are detailed accounts of each step we took to define an MVP for the data marketplace.
We interviewed 10 people across a variety of departments and functions within the organization to understand their data discovery journies and perspectives on the organization’s data pain points.
To aggregate user information, we created user profile cards for each interviewee, including:
Their key responsibilities
Potential marketplace features that would benefit them
Their trusted data sources
Their data outputs
Pain points along their data discovery journey
From the user profile cards, we created four persona archetypes that would potentially use the marketplace. Next, we prioritized two initial user personas—executive and “MVP power marketplace user.” These persona archetypes were chosen because they unlocked strategic value in influencing adoption of the marketplace across the organization in the long term.
Figure 4: Persona archetypes that would potentially use the marketplace.
Next, we built the current state user journey for our power MVP user. This helped refine our understanding of data discovery and data usage gaps that existed and enabled us to further define the most valuable and beneficial features needed in the marketplace (i.e., business definitions in addition to data definitions).
Figure 5: The current state user journey and use case for the MVP power marketplace user.
Research taught us we needed to align on what sources the marketplace should wrap around. NRG users were comfortable getting their data and meta-data in business unit-specific data depots, and the process of ingesting and governing all NRG data into the AWS Lakehouse was still in progress. The client opted to wrap only around Collibra knowing their modern governance processes and centralization into the enterprise data lake were rapidly maturing and growing.
Figure 6: Current architecture ecosystem.
We took input from our user research to define and finalize a list of marketplace features and mapped those to our persona archetypes. Next, we classified these features by priority: must have, should have, and nice to have. This enabled us to finalize MVP scope around must-have features; from there, we wrote functional requirements for each feature that were reviewed with the client for approval.
Figure 7: Prioritized marketplace features based on user journey pain points.
Our next steps were to wireframe designs and page flow for the marketplace leveraging the functional requirements and a future customer journey built for the power marketplace user.
Our experience designers used their understanding of the journey and user needs as well as inspiration from best-in-class retail experiences to design the following key components:
Browse experience specific to NRG that allows user to quickly narrow down to the relevant data products based on organizational understanding.
Glossary that allows users to search and filter within all NRG-specific business definitions.
NRG-specific search, including the ability to filter on NRG-specific concepts.
Product detail pages that included key metadata for data understanding and context and call-to-actions for next steps to work with the data.
Data access request where users filled out a simple form to kick off the access approval process.
Add to marketplace request to allow users to request additional data products or raw data needed to drive business insight.
Credera then broke the work into user stories and worked to deliver an MVP, building both the front-end design and the supporting layer. The supporting layer was built using modern serverless architecture, leveraging AWS services, to ensure scalability and minimal cost.
The full integration with Collibra leverages Collibra API to ensure portability and maintainability and allows for quick search and display of metadata and access management information for use in the marketplace. The integration with Collibra is fully configurable to manage and display the most relevant data for the users and stays updated as the needs and structure of data changes.
Integrations are set up with PowerBI, Databricks, and Active Directory so users can immediately see which data assets they have access to and enable a direct access request to assets without access. The integrations with PowerBI and Databricks also allow users to directly access reports or tables they have permission to view via a direct link from the marketplace.
Figure 8: Custom-fit marketplace technical architecture diagram.
Once an MVP was launched, we continued to work with NRG to mature the data marketplace product considering both user feedback and new users to roll out to. The figure below represents our agile product management approach to managing the feature backlog and the release of new features. It also represents the features delivered to date.
Figure 9: Marketplace prioritized backlog and agile product management approach.
Following this agile product approach provided a structure to product development that was user-focused, aligned with business goals, collaborative, and risk-aware.
There were four benefits to relying on Credera’s product management expertise to execute the marketplace feature backlog:
Prioritized the needs of target users first by gathering and analyzing customer feedback, conducting further user research, and identifying trends in usage.
Facilitated the alignment of building and prioritizing marketplace features with business goals and objectives. If new, urgent business objectives came in, the team could quickly pivot to address those needs efficiently.
Enabled effective communication and collaboration between cross-functional teams, including design, development, data governance, and sales readiness.
Identified and mitigated risks and challenges that arose during product development, allowing for continuous and timely feature delivery.
NRG’s overall enterprise data and data lake have seen accelerated awareness, adoption, and understanding across the organization. This has been especially true among NRG’s executive team; its team leaders across the business are better able to find and leverage data products to make informed decisions. Furthermore, key data science teams have begun to use and trust data from the enterprise lake for key models and analyses.
Data democratization is underway, as evidenced by the growing adoption of the marketplace, as measured in Adobe Analytics by an increasing number of unique visitors. We rolled out the marketplace to multiple business units at NRG. Once granted access, users have increased their number of visits and time spent in the tool. This growing adoption among a steady user base is a sign of increasing trust in the data lake’s data offerings.
Additionally, the user experience for finding and leveraging data products across the enterprise has improved as measured by conversions to ‘data check-out’ in the marketplace. What was once a time-consuming process consisting of e-mailing and finding those with tribal knowledge is now simplified. The dedicated marketplace front end has surfaced data product owners, data stewards and source owners, lineage, and more to allow consumers to quickly find and understand what they need. Producers can more efficiently find raw data and share products as well.
Figure 10: Increased page conversion signals a more intuitive user interface
To guide NRG toward its vision of a data-driven culture, Credera partnered with NRG to design and build a custom internal data marketplace. The goal of the marketplace was to alleviate various challenges associated with utilizing data and to demonstrate the benefits of contemporary governance practices.
Our strategy for developing the marketplace revolved around understanding how users intended to leverage data. By utilizing a design-thinking methodology, we devised a minimum viable product that prioritized the user’s needs and experiences above all else.
Our agile product management approach was crucial in developing a sustainable product for the MVP. Rather than merely considering data as reports or queries, we treated it as a product that requires packaging and a marketplace. With the increasing adoption of the marketplace, NRG is moving rapidly toward data democratization, enabling everyone to easily access, trust, and use data for business insights.
Ready to start a conversation about building a data marketplace at your organization? Reach out to us at [email protected].
