PSD2: Modern Banking API Architectures with the Elastic Stack
Editor's Note (August 3, 2021): This post uses deprecated features. Please reference the map custom regions with reverse geocoding documentation for current instructions.
At Elastic, we :heart: APIs because developers love to work with them to get things done. APIs also have the power to change (or disrupt) an industry quickly and decisively, as is the case with The Revised Payment Service Directive (PSD2). APIs make it possible to seemlessly switch from Web browsers to apps, to deploy content to any platform, and to find the best deals among thousands of suppliers. PSD2 sets out to standardize APIs between EU banks and abolish the existing lock-ins that still exist in the industry. Because while financial institutions are closer to the forefront of the innovation curve than almost any other industry, the point can be made that this has not resulted in wide-spread open access to the core banking ecosystems - namely accounts and transactions. PSD2 is a directive from the European Union that will make banks open up access to their, otherwise private, core banking functions in ways that we have not seen before. PSD2 legislation introduces a breadth of opportunity for retail banks, while also introducing new risk. The Elastic Stack plays a vital role in many of the world’s banks today, and that will especially be true for PSD2 architectures.
A Primer on PSD2 Regulation
In a nutshell, PSD2 stipulates that:
- Banks have to allow a secure way for customer to authorize a third party provider to (1) have direct access to account and transactions data, (2) make and authorize payments via APIs.
- Customers have to be able to trust the privacy and security of their information, hence multi-factor authentication (at least two factors) and granular authorization controls (“entitlements”) have to be in place.
- Member states have until 2018 to create local legislation for PSD2, to come into force likely end of 2018 or early 2019, along with penalties for non-compliance.
Much in-depth content has been written about PSD2 since its inception. For a more thorough discussion of PSD2 we refer you to those resources. The rest of this blog will focus on the strategic choices that banks will need to make, and how PSD2 impacts banking architectures.
Strategic Overview
We need to give you a couple of new acronyms to make sense of PSD2. Warning: they don't really roll off the tongue.
- ASPSP
- Account Service Payment Service Providers, the core capability of retail banks.
- PISP
- Payment Initiation Service Provider, a party in between the customer and the bank, and initiates a transaction. Can be a non-bank entity, like the retailer.
- AISP
- Account Information Service Provider, also known as “the cross-bank service” where customers can get a consolidated picture of their finances.
- XS2A
- Access to Account. The legislative API calls that grant AISPs access to transaction data.
Some things in life are inevitable, while others are entirely optional. Retail banks in the EU have to align their strategy along a range of options that lead from a compliant utility-like bank on one end, to a one-stop shop for anything related to consumer finance on the other end. A bank may choose to simply conform to the PSD2 legislation and continue much like it did before. But there is value to be found in going beyond the compliance of integration and open up financial services on top of XS2A. Value such as using your bank’s platform to consume other banks’ APIs and give them the complete picture of their financial status. Or, implement APIs that go beyond XS2A, like for requesting loans, giving advice around savings, or finding businesses where your users can spend their money on new, shiny things.
Fintech players are disrupting business in all areas: investing, paying, and saving. And they are creating new ones like cryptocurrencies. Users love fintech because they provide their services using qualities that users have come to expect from Google, Facebook, Amazon and the likes: everything online and 24/7, using data intelligently to minimize users actions and maximize value, and have a tremendous user experience through UIs and APIs.
Along the spectrum of strategic options, banks may decide to be a lean, utility-like provider of payments and account services. Or, at the other end of the spectrum, provide a world-class experience that users will use as the focal point of all their financial dealings.
So, banks that operate in the European Union can strategize on three axes:
- API Consumption: Stick to the position in the payments infrastructure that they have today, or additionally consume APIs from other ASPSPs to become an AISP that people crowd around?
- API Exposure: Expose just the necessary APIs (as required), or additionally expose many more value-adding services through them?
- User Experience: Do you want to invest in best web and mobile user experience that is available in the market?
Interactions with our clients indicate that most banks, if not all, opt to go beyond the requirements that PSD2 demands of them to become a single open ecosystem between merchants, banks and users.
A simplified architecture of a PSD2-compliant retail banking ecosystem already shows the important role that APIs will play. A big change is that more APIs will have to be opened up to more external parties. Apart from the obvious security concerns, this also means that you will no longer control the usage of your own APIs, other parties will use them as well.
Having and sharing actionable data will be the Norm for AISPs
AISPs (we’ll repeat: Account Information Service Providers) add value to customers by ‘knowing it all’. To become an AISP, a bank must have a complete picture of a user’s financial transactions and accounts. On top of that, an AISP should know what the user wants to achieve, what merchants the user likes, all with user consent (GDPR, anyone?). On top of that, AISPs should strive to have the best experiences (that includes user interfaces, alerts, brand image, trustworthiness) to get in a position of advising the user. Luckily, APIs also help out to get data from merchants and ASPSPs into the AISP.
Real-time query engines are able to react and predict to users, transactions, and the like. Having offline batch processing is a great way to extract intelligence out of data in some cases, but to get that intelligence online, a fast data store is needed with millisecond response times.
European banks have been experimenting with personal finance features on their platforms for years. But they have always been based on the partial picture of the user’s finances, and were arguably not as functional a users have come to expect in recent years. With PSD2, we expect a surge of new personal finance tools that will be completely automated, intelligent and responsive. It requires real-time analytics and natural language processing (NLP) at scale, such as aggregations, fuzzy queries, multi-language, and predictions.
PSD2 Architectures
Banks already operate using internal APIs that connect modern, scalable front-end applications to core account and payment systems. Typically, the core banking systems are legacy systems that don’t scale effortlessly, so they offload part of their responsibilities to various modern data stores to save cycles on the core systems. PSD2-compliant architectures will have to make those APIs accessible to 3rd parties that are, at best, under the bank’s influence, not control. This means that the APIs will make or break access to bank’s most basic functions.
A Shopping List for PSD2 Architectures
A PSD2-compliant architecture requires at least these major components:
- An API to the fast access layer with proper scaling, throttling, and security in place.
- A fast access layer to offload the core banking applications and provide cheap, scalable data services to the API layer.
- Core banking applications, often legacy systems that are already in place. ACID-compliance, a relational nature and its license models usually hinder scalability.
- An observability platform for all of the above, and the focus of the rest of this blog.
The rest of this blog will focus on executing an API-first banking strategy, starting out with running PSD2-compliant APIs and moving ahead as an AISP to provide users the single, intelligent interface to all their personal finance needs. Part II will focus on a logging platform to monitor the PSD2 architecture.
The Elastic Stack
The Elastic Stack is a use-case agnostic data platform that is very well-suited for running high-traffic, API-driven environments like this one. It excels especially when the mere serving of key/value pairs is not enough. Elastic’s analytic functionalities for both structured as well as unstructured data are necessary for serving dashboards, advisories, transaction histories, integrating with 3rd party data stores and the likes. Elastic makes data come alive, it’s far beyond a bunch of documents waiting to be called by their ID. Often the Core Banking Apps are not suited for the scale that banks will be faced with due to technical and licensing issues. Elastic has scalability built-in from its first lines of code, and there are great benefits to its licensing model which is based on number of logical nodes, not the amount of queries, users or ingested data.
The rest of this blog will focus on executing an API-first banking strategy, starting out with running PSD2-compliant APIs and moving ahead as an AISP to provide users the single, intelligent interface to all their personal finance needs. Part II will focus on a logging platform to monitor the PSD2 architecture.
The Elastic Stack for Smart Banking Data Platforms
The Elastic Stack is perfectly suited to run not just your classical search and logging use cases, but also for serving business data via APIs. In a nutshell, an PSD2-enabled bank that also will become an AISP will need the following high-level architecture. In pink, the minimum PSD2 components. In green, an incomplete list of differentiating value-adding services (once interbanking and intermerchant APIs are in place, the list of imaginable value-adding services becomes huge).
We think that successfully running value-adding platforms successfully, banks will require excellence in these key areas:
- Security and Privacy: This includes corporate and legislative requirements such as encryption, authorizations, audit logging, privacy, and data separation.
- Monitoring and Alerting: The ability to know current and historical status of the service, and be informed of any serious deviations from what is considered normal. The ability to view the inner workings of a system is also called observability.
- Quality of Service: The ability to throttle in case of overloads, to protect underlying systems from DoS attacks and to allow the enforcement of Fair Use policies.
- Easy, far-reaching scalability: Scaling the platforms should be simple and painless.
- Real-time answers: Answer calls fast to support the needs and expectations of users, throughout the user experience.
- Self-learning and self-service: Getting useful intelligence out of the data without having to foresee and manages what exactly is relevant (because who would know beforehand how security breaches or outages might unfold?) requires abilities to find anomalies, create advanced dashboards, and a raw data store. Consumers are expected to mix and match layouts of dashoards and apps to their needs, but they will only do so if the data and UX are very intuitive to work with.
Elastic Architecture for Data Platforms
The Elastic Stack is a complete suite of products for running API architectures:
- Logstash is a dynamic data collection pipeline with an extensible plugin ecosystem and strong Elasticsearch synergy. It can read from slower, write-optimized primary data stores such as RDBMSs. Ingestion can be batch-oriented or near real-time.
- Elasticsearch is a distributed, REST API enabled, JSON-based search and analytics engine designed for horizontal scalability, maximum reliability, and easy management.
- Kibana gives shape to your data and is the extensible user interface for configuring and managing all aspects of the Elastic Stack.
- X-Pack is a single extension that integrates handy features — security, alerting, monitoring, reporting, graph exploration, and machine learning.
- The APIs sit on top of the Stack, as well as custom UIs like mobile apps or websites.
The Elastic Stack logical architecture for APIs combines all these products into an end to end platform with accompanying services, like Consulting and Expert Support. As you have probably read a bunch of times by now, Elastic :heart: APIs. That is why the Elastic Stack products natively supports REST API endpoints for easy integration into any architecture.
Serving PSD2 API Requests
Elasticsearch natively supports REST API endpoints, likely the same technology that will dominate the PSD2 landscape. Expect millisecond response times to queries, in an encrypted, authenticated and authorized ecosystem, including audit logging (GDPR anyone?).
Of course, banks already run private APIs similar to PSD2’s public ones to connect their customers to their data on mobile apps, web browsers, kiosks, or ATMs. Consolidating these private and public APIs is going to make banks more cost-efficient and agile (less code to manage).
Stepping Up Your Game as an AISP
Elastic’s most unique feature for AISPs is the ability to not just store and serve data safely and on immense scale, but to get meaningful, actionable insights from that data. We expect that users will be searching for the AISP that provides that best combination of data intelligence and user experience and handle most day-to-day finances with that AISP. Kibana shows off Elasticsearch’s aggregations and time-series features nicely. Custom UIs can utilize the same features and implement a completely custom presentation.
Scalability is important when you set out to store a breadth of customer-centric financial information. Elasticsearch’s code is setup to be scalable from its very beginnings, resulting in a system that can handle millions of concurrent reads and writes per second. It allows AISPs to provide services with 3rd party data just as easily as with internal data.
Implementing a Customer Financial Advisory
Having a time-series of financial transactions of a customer enables an AISP to provide relevant and timely information to their users. Disregard, for a moment, that having an army of analysts look at all data in an AISP to turn it into meaningful information would be cost-prohibitive, i.e. very expensive. What could they possibly come up with? Customers might want to know when they slowly increase their spending in certain categories, like groceries. Or, they might be interested to hear about how their bank can help them redecorate their homes or buy a new car. If privacy regulation allows, an AISP could also aggregate and sell anonymized market insights to governments and commercial sectors. Facebook already acquired a European payments license.
Going back to the army of analysts. Elastic is investing heavily in new ways to look at data that exists in the open source Stack. Two of them are particulary interesting to behavioral intelligence. X-Pack Machine Learning is a technology that does unsupervised anomaly detection on time-series data. It builds up a sense of “normal” by looking at historical data and by looking at peer data, and then weeds out false positives to give you only relevant, actionable insights. This is your army of analysts.
Some good reads on X-Pack Machine Learning are:
Traversing Relevant Connections
X-Pack Graph is a technology that presents connections in data as a graph. It weeds out irrelevant, ‘boring’ connections by comparing subsets of data with their peers. For instance, because the total population has a certain percentage of their money spent on, say, cappuccinos, we might want to explore which subset of the population spends more than average on cappuccinos? This is trivial even with SQL-era aggregations. But now answer this: if I want to sell more cappuccinos, which other products or services should I relate my marketing to so that I reach new audience that would gladly buy more cappuccinos if given the opportunity? What is uncommonly common about people that drink uncommon amounts of cappuccinos that can help me propel my business to the right locations, the right products, using the right brand message? And how can I get these insights as self-service analytics? Graph will help to get meaningful relations from data using wisdom of the crowd algorithms.
Some good reads on X-Pack Graph are:
- Page: Elastic X-Pack Graph
- Video: Graph Deep Dive with Mark Harwood
- Blog: Using Elastic Graph + Kibana to Analyze Panama Papers
‘Orchestration as a Service’ with Cloud Enterprise
Elastic Cloud is our public SaaS service running on AWS. Using that technology and our expertise, we set out to bring the Elastic Cloud experience to any data center. And that’s what we did with Elastic Cloud Enterprise (ECE). As organizations adopt Elastic across use cases and departments, ECE keeps your focus on building value-adding services on top of your Elastic clusters.
We would love to talk about ECE some more, but this blog is not meant as a comprehensive discussion of Elastic Cloud Enterprise. Luckily, those resources already exist. Some good reads on ECE are:
Huh? I thought Elastic was for Monitoring: Logs, Metrics and Traces!
Elastic has been synonymous with logging for years. Indeed, many global financial institutions ingest logs, metrics and traces of their IT infrastructure to an Elastic platform. That platform then provides dashboards, time-series analytics, anomaly detecting using X-Pack Machine Learning, real-time alerting, fraud detection, root-cause analysis and other services with that data. Part II of this series will focus on running Elastic for monitoring APIs.
Icons made by Freepik from flaticon.com