WASHINGTON–Who’s liable if a credit union’s AI-based decision engine makes a decision on its own, outside parameters the CU put in place? What are the implications of the “arms race” taking place in AI? How can Congress “not screw this up?” Those questions and others were at center of discussion in the Senate as the government—and industry—race to get a framework in place around the rapid developments in artificial intelligence.
During a Senate Banking Committee hearing on AI and financial services, questions around other big issues were also debated, with experts acknowledging there are few answers, including around challenges such as bad actors using AI to more effectively commit fraud by learning from other AI designed to combat that very same problem.
Views expessed by senators during the hearing largely reflected idealogical party differences on the role of government in oversight and markets.
Prior to the testimony of three experts, Senate Banking Committee Chairman Sherrod Brown (D-OH) offered numerous cautions around the risks of artificial intelligence in financial services, saying that while efficiencies are often cited, that often just means bigger profits for big banks and Wall Street, while the technology also poses risks to consumers through fraud and scams.
“We have a responsibility to assess what AI means not just for our financial system, but overall for the American people,” said Brown. “We have the responsibility to set policies to ensure that when and if this tool is used it's used to make our economy work better for consumers and savers, not to exploit them.”
Not Being Luddites
Brown said he wants to see AI used to make financial services more affordable and accessible.
Too often, Brown continued, those who challenge new technologies are criticized for standing in the way of progress and described as modern-day Luddites. Brown said he wants to ensure technology helps build a “fair and transparent economy that works for middle class Americans,” and that if existing laws around consumer protection don’t cover emerging technologies, then Congress must create new laws and guardrails that do, including around credit underwriting decisions.
It’s an issue credit unions and other financial services providers are giving close attention, plunging ahead even as the landscape remains hugely uncertain.
“What’s known as generative AI is creating new ways to remove human decision-making from financial services. These so-called advances make it harder to determine who is accountable when things go wrong…in consumer lending markets where AI models used to determine borrowers’ creditworthiness,” said Brown. “Too often, (automated decision-making models) automate and supercharge biases and exclude Black and Latino Americans. It's hard to eradicate discrimination when even the developers can't really explain how the models get to the decisions they make.”
Brown called for “rigorous testing” of AI models before the technology is deployed in real-world situations.
Senator Talks Both Sides of Fraud
Noting AI has “been around for years,” just under different names, Sen. Mike Rounds (R-SD) said the technology has largely improved the consumer experience, but did raise concerns around how AI and other emerging technologies are contributing to fraud, while also suggesting AI can help solve the challenge.
“Despite the billions spent to protect institutions and the billions paid to buy off attackers, things are only getting worse, with synthetic identity fraud costing banks nearly $50 billion last year,” said Rounds. “As much as 95% of phony identities go undetected. Traditional methods of fraud detection rely on manual verification and human analysis, which, unfortunately, have failed to keep up with more sophisticated fraud schemes. By using AI and machine learning companies can analyze relationships between entities and identify suspicious patterns and visualize intricate connections, revolutionizing fraud detection. This allows for a more proactive approach where AI is used to prevent fraud before it happens as opposed to the traditional reactive approach to fraud detection.”
‘Human Control is Necessary’
Rounds called the financial services industry “uniquely poised” to adapt to emerging technologies, and further said any models used need to be transparent.
“Lending algorithms can't simply exist in a black box and human control is necessary,” said Rounds. “AI is data dependent and the technology is only as useful as the quality of data that goes into its models.”
Rounds also called for ongoing investment in cyber infrastructure to protect databases, saying not doing so could allow bad actors to disrupt the financial system.
In terms of the role Congress will play in the regulation of AI moving forward, Rounds said, “I think it is important that we take a pro-innovation stance that will allow the United States to keep and attract the best and brightest talent. Although we will have many discussions about the dangers, we must also acknowledge that halting progress can be dangerous, especially as our global competitors, such as China, have no intentions of slowing down.”
‘Unintended Consequences’
Rounds said financial regulators must resist the urge to over-regulate new technology, “as they run the risk of unintended consequences…Congress should help shepherd the development of American artificial intelligence that is embodied with key principles to promote confidence and trust, which include a right to privacy and transparency. We are at a crossroads. Artificial intelligence is real and it's not going away. We have the opportunity to shape it in a way that reflects the values that are important to us, and it's up to us whether we harness it to make improvements to our financial system or if we simply fall behind.”
What the Experts Told Congress
Testifying before the committee were:
- Melissa Koide, director and CEO, FinRegLab, former deputy assistant secretary for consumer policy, U.S. Department of the Treasury
- Daniel Gorfine, founder & CEO, Gattaca Horizons, LLC, adjunct professor of law, Georgetown University, and former chief innovation officer, Commodity Futures Trading Commission
- Professor Michael Wellman, University of Michigan, computer science & engineering.
With editing for length, here is what each told the Committee:
Koide: High Risk, High Reward
“The adoption of machine learning in credit underwriting…is an application with high risk, but also has the potential for high-reward opportunities as we think about the 50 million adults in the U.S., as well as the millions of small businesses who cannot be sufficiently credit risk assessed with more traditional methods of underwriting,” said Koide.
Koide outlined how the use of machine learning in credit underwriting becomes even more complicated given the legal requirements on lenders to communicate how credit decisions are made and the factors that went into the decisions, including denials. It’s critical lenders are able to evaluate for disparities in criteria and models, according to Koide, and to ensure they also model for safety and soundness.
What Analysis Found
She said her company, which is a nonprofit, evaluated proprietary and open source machine learning tools used to perform tasks relating to model risk management, adverse action notice generation, and fair lending compliance, and found certain techniques were able to generate reliable explanations as to which features were most important for the model's predictions.
“Even for more complex machine learning algorithms, however, there was no one-size-fits-all approach or technique, and overall it really depended upon the human oversight and engagement and understanding of the models and the data choices that were being used to train those models,” said Koide. “We also found machine learning has the potential to improve fairness and inclusion. Machine learning-automated approaches produced a range of alternative model options with smaller demographic disparities than more traditional methods.”
Caution & Risk
When it comes to generative AI, while interest is “considerable,” Koide said financial services providers are taking a cautious approach, specifically with internal testing of internal report generation, as well as code generation.
“Customer-facing use cases are especially high risk due to the potential for generative AI models to provide inaccurate information, discriminatory or otherwise harmful outcomes, and to expose sensitive information,” Koide said. “Regulatory compliance demands a level of explainability and transparency…that many providers are not confident they can attain at this point.”
To that end, Koide said the financial services ecosystem would benefit from the identification of best practices and safeguards for the use of AI and machine learning, especially around automated decision making.
One Expert’s Advice
Koide offered these additional observations and advice:
- “Model risk management expectations and fair lending requirements do not apply equally to actors across the financial ecosystem, and some stakeholders have pointed out that imposing basic governance expectations on non-bank financial providers could be that beneficial to the broader ecosystem.”
- “An articulation of frameworks to govern AI in financial services consistent with high-level principles or standards could be valuable at this stage, aiming all the financial actors in the same direction.”
- “Careful consideration of data governance practices standards and policies would be helpful. While federal laws provide more detailed and robust protections for consumer financial data than other types of consumer data, key laws like the Graham Leach Bliley Act are 20 years old and haven't been updated.”
- “Congress should dedicate resources to support public research in these areas and ensure engagement by historically underrepresented and under-resourced actors in the financial ecosystem.”
‘We All have to Learn’
“We're all going to have to learn and keep up with how this technology works,” Koide said. “We all have an important role to play in making sure that the evolution of our laws and our market practices are for the good and the benefit of consumers households and the broader financial system.”
Gorfine: Recommendations for Policies
Gorfine noted artificial intelligence is an area of “fierce global competition where the U.S. holds many competitive and first-mover advantages.” It’s also an area, he said, that should be “responsibly fostered through thoughtful policy approaches.”
After noting the advances being brought about, including through generative AI, Gorfine observed, “With any area of innovation, however, there are important risks associated with AI, including the potential for perpetuating bias, infringing on data privacy, failing to operate as expected, and advancing frauds and scams.
“The mere speculative fear or fear of future harm, however, should not broadly block development of AI in financial services, including by those small firms and community banks seeking to remain competitive in an increasingly digital economy,” Gorfine continued.
Gorfine encouraged the Banking Committee to use as its guiding principle the assessment of whether new AI-based models improve on what is already in the market.
Some Recommendations
He offered numerous recommendations he said will help achieve balance in adopting AI and AI policies, including:
- “Encourage innovation but monitor for novel risks. It is important that policymakers use their soft power to encourage ongoing…compliant innovation, especially with respect to small firms and community banks.”
- A framework for enhanced clarity and consistency is needed, especially within existing risk management. “Regulators should also encourage and help foster the development of standards, including through collaboration, recognition of standard-setting organizations and the use of safe harbors and guidance that explicitly encourage adherence to such standards.”
- The federal data privacy framework should be modernized. “Given the centrality of data in AI, it is imperative that Congress work to establish a national framework that governs data privacy, advances cyber security and ensures that consumers have control over how their data and information is being used.”
- Congress should avoid hasty and speculative regulation that can chill innovation. “Given existing regulation it is important for policymakers and regulators to avoid hasty and prescriptive rules based on speculative or hypothetical future risks that have not yet emerged.”
- Generative AI developments should be monitored to inform financial regulation.
- Law enforcement collaboration should be prioritized and new technologies should be developed to combat fraud and scams. Those agencies should work collaboratively with the private sector and regulators to ensure the sharing of information and best practices, to increase tech literacy and the adoption of advanced tools to combat threats, Gorfine said.
Wellman: Understand the ‘Implications’
Wellman, who earned a Ph.D. in 1988 from MIT in artificial intelligence and who has been consistently involved in research around the issue ever since, said he has been focused on understanding the implications of AI for financial markets and the financial system.
“AI promises extraordinary benefits: expanding knowledge, automating onerous tasks and making valuable services accessible and affordable to broad segments of our society,” Wellman said. “AI also poses risks to security from malicious exploitation of AI by (bad) AI behaviors to disrupting how we work…The future path of AI is highly uncertain and if somebody tells you they know where AI technology will be in five years or 10 years or even next year, don't believe them.”
Wellman called for “teasing apart” which AI practices and circumstances help versus those that hurt, while identifying market designs or regulations that promote the beneficial and deter the harmful practices, especially in the stock market.
An ‘Arms Race’
What it taking place broadly is what Wellman termed “an adversarial learning situation, kind of an AI arms race between the regulator and the manipulator. An inherent feature of adversarial learning is that any advance in detection technology can be exploited by the manipulator to evade better. Where this leads in any given situation is an open question. It is also possible that AI-developed trading algorithms could produce manipulative strategies even if not instructed to do so.”
There is another reason Congress and regulators need to move more quickly, according to Wellman.
“Our research has demonstrated automatic learning (can) manipulate a financial benchmark given only the objective of seeking profit,” said Wellman. “Our current regulations (are not) adequate to handle such a situation. Much of the existing law depends on intent to manipulate and how that would apply to an AI that learned on its own is unclear. This is just one example of an AI loophole. Our existing laws, generally speaking, assume that people make the decisions. When AI's are deciders, do our laws ensure accountability for those putting the AIs to work?”
A similar risk that is not well understood, according to Wellman, includes generative AI solutions such as ChatGPT manipulating social media, through a “pump and dump” scheme, for example. “AI systems could inject misleading information,” Wellman cautioned.
How Should Regulators Respond?
When asked how regulators should best respond to developments in artificial intelligence, Koide told the committee financial services providers should have to tell the consumer why they received a certain credit decision or pricing as part of broader governance rules that should be in place.
How Not to ‘Screw This Up’
Rounds asked all three witnesses, plainly, how Congress can “not screw this up?”
‘It’s Important to be principled,” said Gorfine. “In financial services there's the right scaffolding in place around regular regulation and governance. However, you need to monitor for where there are emerging risks and once you identify specific risks, you tailor interventions to solve for those specific risks over broad rule makings. Otherwise, you’ll have unintended consequences. You’ll capture activities that you're not intending to. That will make it very difficult for small firms, for community banks to be able to adopt new technologies. You don't want to be prematurely preemptive; you want to make sure that it's reasoned and principled.”
Disagreement on Issue
Wellman, however, said he doesn’t believe it will be possible to prevent powerful AI from spreading throughout the economy.
“How regulation evolves will shape the economic development,” said Wellman. “I must disagree with my colleague, respectfully. I think responding before the risk materializes is sometimes essential to avoiding really terrible outcomes. We can reasonably understand areas that pose risks and shape the environment so that we're more robust before they happen rather than (waiting until afterward).”
Really Thinking About Data
For her part, Koide emphasized the importance of data and “the need to really focus, broadly, on how do we get consumer data privacy. Right now, do we think about updating our data privacy expectations in financial services? In financial services, the offering is based on risk mitigation, which is based on data information. How are we striking that balance? How are we making sure, importantly, that we're not leaving out marginalized communities because the data doesn't reflect an understanding of them.”