Managed File Transfer in Banking: A Complete Guide

As businesses deal with larger and more diverse volumes of data, managing that data has become increasingly difficult. Only 5% of businesses feel they have data management under control, while 77% of industry leaders consider growing volume of data one of the biggest challenges. 

Data vault is an emerging technology that enables transparent, agile, and flexible data architectures, making data-driven organizations always ready for evolving business needs.  

What is a Data Vault?  

A data vault is a data modeling technique that enables you to build data warehouses for enterprise-scale analytics. It allows you to adapt to fast-changing business requirements with an agile and incremental approach, avoiding the need for extensive re-engineering. It also ensures historical tracking and auditing of data, as it stores all the data from the source systems without any updates or deletions.

Data Vault Architecture

A data vault follows a three-layered architecture containing the raw, business, and information vault. This design allows you to build agile and scalable data warehouses resilient to changes in the source systems and business requirements.  

Raw Vault 

The raw vault is the most granular level, and it stores data as close to the source format as possible. It also keeps track of all the audit information from the source. 

Since the raw vault is insert-only, no data is ever deleted or updated. This ensures that the raw vault preserves the full history and traceability of the data. The raw vault composed of hubs, links and satellites. These tables capture business keys, relationships and attributes of your data. 

• Hubs represent core business concepts, such as customers, products, or vehicles. They store the business keys that uniquely identify these concepts, along with some metadata information, such as load date and sequence ID. Users can use the business keys to query information about a hub. For instance, a Customer hub has CustomerID as the business key, and a Product hub will have a ProductID as the business key. 
• Links represent the relationships between hubs. Links store the combinations of business keys, showcasing how those hubs are connected. For example, a link between Customer and Product hubs can be a PurchaseTransactions link table. It will contain data relevant to both these hubs, such as  PurchaseDate, Quantity, TotalPrice.
  To make sure each transaction can be uniquely identified, the data vault will concatenate the CustomerID and ProductID, then generate a hash key based on the concatenated string.
• Satellites store the descriptive information about hubs and links. They contain the attributes that describe the characteristics or states of the hubs and links over time. For instance, the satellite for the Customer hub may store the customer’s FirstName, LastName, Address, PhoneNumber, etc.
  Likewise, the satellite for the PurchaseTransactions link table may store information like PaymentMethod, DeliveryStatus, LoadDate, RecordSource. In this table, PaymentMethod and DeliveryStatus provide additional information about each transaction. LoadDate and RecordSource are metadata fields.  

Business Vault 

The business vault is another layer derived from the raw vault. It applies selected business rules, calculations, data cleansing and data quality functions to the data. The business vault can also contain hubs, links and satellites, but they are usually created based on business needs and are usually not a direct copy of those in the raw vault. The business vault is useful for providing query assistance and facilitating user access to the data. 

Here are some data transformations and data quality rules that might be applied in the Business Vault in our example:

• Data Transformation – Customer Lifetime Value Calculation: You might have a transformation rule that calculates the lifetime value of each customer based on their purchase history. This could be implemented as a new Satellite table in the Business Vault that associates each CustomerID with a calculated LifetimeValue attribute. The LifetimeValue could be calculated as the sum of TotalPrice for all transactions associated with each CustomerID.
• Data Transformation – Product Category Sales: You might want to track the total sales for each product category. This could be implemented as a new Hub and Satellite in the Business Vault that associates each Category with a calculated TotalSales attribute.
• Data Quality Rules: You might have business rules that enforce data quality standards. For example, you might have a rule that flags any transactions where the TotalPrice is less than zero, or where the CustomerID or ProductID does not exist in the respective Hub tables. These rules can be implemented as checks in the data loading process or as constraints in the database schema. 

Information Vault 

The information vault (also known as information marts) is a presentation layer built off the raw vault and business vault to support reporting and analytics. It is composed of user-friendly structures like star schemas that representor data marts. 

The information vault can apply further transformations and aggregations to the data to make it ready for consumption by end users or business intelligence tools. 

Learn how to use information marts to deliver customized and relevant data to your end-users.

The combination of raw vault, business vault, and information marts allows for better data integrity, near-real-time loading, and better accommodation of new business needs without affecting existing structures.  

Benefits of Data Vault  

Flexibility and Adaptability 

Data vault can handle multiple source systems and frequently changing relationships by minimizing the maintenance workload. This means that a change in one source system that creates new attributes can be easily implemented by adding another satellite to the data vault model. 

Similarly, new and changing relationships can be handled by closing one link and creating another one. These examples show the high level of flexibility and adaptability provided by data vault. 

Scalability 

As the data volume grows or more source systems are added, data vault scales easily. You can introduce new hubs, links, and satellites to incorporate additional data sources or entities without disrupting the existing structure. The data warehouse allows you to accommodate more data and more users without compromising performance or quality. 

Historical Data Preservation 

The use of satellite tables in data vault ensures that historical data is preserved. This is critical for tracking changes over time, analyzing trends, and meeting regulatory compliance requirements. For instance, in a clinical data warehouse, it’s important to store historical data understanding for how patient diagnoses or provider specialties have evolved over time.

Data Lineage and Auditability 

Data vault embeds data lineage and auditability in the data vault model. In other words, the data vault stores the load date and data source for every new record, telling us when and where the data came from. Additionally, you can analyze hash keys and hash differences to quickly compare row values and identify changes. These features help to ensure data quality, traceability, and accountability. 

Consistency 

Data vault ensures data consistency by capturing data in a consistent manner even when the source data, or its delivery, is inconsistent. This means that the data warehouse can provide reliable and accurate information for business decisions. Moreover, data vault enables parallel loading of data with the use of hash values, which improves data access speed and user satisfaction. 

Agility 

Data vault supports agile development and evolution of data warehouse requirements by following a methodology that includes several principles of agile work processes. This means that data vault projects have short, scope-controlled release cycles that allow the development team to work closely with the business needs and create a better solution. 

Data Vault vs Traditional Data Warehouse Techniques 

Data vault is a modeling technique to build data warehouses but differs from traditional techniques such as dimensional model and 3NF. It has some key differences in terms of data loading, data modeling, and data agility. 

Data Loading  

Data vault loads data differently compared to traditional data warehousing techniques. Typically, data warehouses follow Extract-Transform-Load (ETL) workflow where data is transformed and validated before being loaded into the warehouse. In this technique, you must update or delete existing records in the warehouse based on the changes in the source systems. 

In contrast, data vault leverages the Extract-Load-Transform (ELT) workflow where the data is directly stored in the raw vault from the source system. Business rules, transformations, or validations are applied later in the process, typically inside the business vault.  

This approach allows you to append new records to the vault without updating or deleting existing records. You can apply business rules, transformations, and validations only when you need to use the data for reporting or analysis. 

Data Modeling 

A traditional data warehouse typically uses a dimensional modeling or a normalized modeling (3NF) to organize the data into facts and dimensions or entities and relationships, respectfully.  

Data vault uses a hybrid modeling technique that combines the best practices of both dimensional and normalized modeling. It leverages a hub-and-spoke schema to represent the data in a historical and flexible way. Whether data vault is  the ideal data modeling for you depends on your requirements. 

Data Agility 

A Data Warehouse typically has a rigid and stable structure that is designed to meet the current and anticipated business requirements. The structure of the warehouse may change over time due to changes in business requirements, source systems, or data quality standards. However, such changes may require significant efforts and resources to modify the existing ETL processes, warehouse schemas, and reports. 

Data vault adapts to fast-changing business requirements by separating a data model’s stable and temporal parts. This means that the core business concepts and their relationships are stored in hubs and links, which rarely change. In contrast, the descriptive attributes and their changes over time are stored in satellites, which can be easily added or modified. 

This way, data vault avoids the need for extensive re-engineering of the data warehouse when new sources or attributes are introduced or existing ones are modified. 

Capacity to Handle High Volumes of Data 

Data vault supports parallel loading of data from different sources, which increases the speed and efficiency of data integration. Data vault is based on agile methodologies and techniques, allowing you to deliver value continuously and iteratively, and collaborate actively with business users.

Data Vault Best Practices  

Data vault can handle complex and dynamic data environments. However, to ensure a successful data vault implementation, following  best practices and avoiding common pitfalls is important. Here are some tips and recommendations from experts in the field: 

• Plan and define business model and requirements clearly before designing the data vault. This helps identify the key business concepts, relationships, and attributes that will form the basis of hubs, links, and satellites.
• Use automation tools to speed up and simplify the development and maintenance of the data vault. Automation tools can help generate code, data models, documentation, and tests based on predefined patterns and metadata. 

• Follow the data vault 2.0 standards and best practices to ensure consistency, quality, and scalability of the data vault. Data vault 2.0 is an updated version of the original methodology incorporating new features such as hash keys, parallel loading, audit columns, and business vault.  
• Avoid overloading the data vault with unnecessary or redundant data. Data vault is designed to store raw data from source systems without applying any transformations or business rules. However, this does not mean one should load everything into the data vault without filtering or validating the data. Users should only load the relevant, accurate, and useful data for their business needs. 
• Monitor and test data vaults regularly to ensure their performance, reliability, and accuracy. Data vault is a dynamic and evolving data structure that can accommodate changes in data sources and business requirements. However, this also means that users must keep track of the changes and their impacts on the data vault.  

Build a Data Warehouse with LIKE.TG  

Data vault offers many benefits, such as scalability, auditability, parallelism, and adaptability, making it a great choice for modern data warehousing needs. LIKE.TG DW Builder is a code-free and automated data warehouse design and ETL/ELT tool that allows users to build data vaults in minutes.

To learn more about LIKE.TG DW Builder and its data vault features, visit the website or request a demo today.