I built a thing. It’s called Up Bank Assistant.
Up Bank has a pretty cool API: https://developer.up.com.au/
It gives you read only access to many parts of your bank account including transactions and accounts.
I’ve been curious for a while about ways that AI tools and agents can help improve banking experiences and day to day interactions with our banking apps.
As a little side project I wanted to explore what I could build using this API and my latest obsession: voice agents.
Voice agents are incredible. I wasn’t paying attention for a while and then I started paying attention a few weeks ago and… umm I think they are going to automate practically all receptionist/call centre jobs within two years. The technology is getting that good that quickly.
If you build one well (and i’ll get into what that means in a bit) and tell people ahead of time its an AI, it gets about 96% of the way there of replicating what its like to interact with a human on the phone. Meanwhile they are 100% good at:
- answering simple questions;
- taking bookings;
- making you feel like someone is actually attending to your concerns rather than having you wait endlessly;
- not making you press stupid buttons or say keywords that never interpreted properly
- capturing any information about you ahead of time so that when a human does speak to you they have all the context they need;
- and the kicker: they are available 24x7
So anyways, on to the project: Up Bank Assistant
You can use it to learn about Up Bank and even get a call from an AI agent that will answer questions you have about Up Bank. You can also have a natural language conversation about your transactions!
important security disclaimer: I did my best to make handling the API key secure and the keys are kept encrypted at every leg of the journey except when decrypted momentarily by the backend to use in the API call to Up. At no point are the keys kept in plain text or any of the transaction information stored or logged. The data going to the LLM originates from the client-side and is stripped of personal information/anonymised before being sent out. the code is available here: frontend | backend
With all that said, I did this in a week and I haven’t had it reviewed by anyone component for bugs. This is a little demo. I repeat this is a little demo.
So if you do want to use it, please clone the repo and run it locally.
Okay with that out of the way, let’s explore the project.
Part 1: An information assistant
Chat Interface:
This one’s pretty simple. You ask it a question and it gives you the correct answer using RAG.
But there was a lot of pre-work that went into making this happen. Here’s what I had to do:
- Scrape the entirety of Up Banks website to get all the useful information about their products, services, offerings and terms and conditions.
- Figure out whats relevant and what’s not.
- Chunk this and embed it in a vector database
- Use the vector embeddings to get the information needed to answer a question.
RAG and How Vector Embeddings work
“write this up later”
Anyways, RAG is okay. But the answers were pretty hit or miss. It really depends on the quality of your embeddings and how you structure them. If it’s not done right the answers can be pretty bad. If you chunk a large page and don’t specifically have headers for each chunk to indicate their related you can miss critical information in the retrieval. All of this to say, I wasn’t that satisfied.
Enter OpenAI Deep Research.
Using Deep Research and giant context windows
So let’s try to combine two very impressive tools to see if we can get better results than RAG.
Gemini-Flash-1.5 costs $0.075 per 1M tokens. This is insane. Why are more people not talking about this??
OpenAI Deep Research is very very good at browsing websites and compiling very detailed research reports.
So err, what if I gave Deep Research the Up Bank sitemap, asked it to select the 250 most relevant web pages and filter for irrelevant/promotional material, visit each site and summarise the key points on that site (keeping any specific numbers or figures). And finally to create an extremely dense write up with all of those findings?
Can it do that?
Yeah, it can!
And then what if I just pass that document in as the context to every user query that is passed to Gemini?
Would that work?
YEAH, IT WORKS REALLY WELL!
Voice Agent:
This one’s fairly straightforward in theory: Put your number in and a pre-programmed agent will call you and talk to you in a perfect nice late-twenties-early-thirties-Melbournian-girl accent, answering any questions you have about Up Bank or its products. Sounds cool, right? IT IS REALLY COOL.
Actually, here, go see a voice agent in action. Go to my project >> Learn About Up >> ‘Talk to an Agent’ and enter your number. You’ll get a call straight away.
This is what goes into making that happen.
All the hard-work has been done by companies like Bland AI. It’s literally a low-code/no-code interface at this point. The hard work is in structuring the logic tree and the prompts for the agent to navigate different parts of the decision. Actually the prompting is non-trivial. You have to really think and have taste here.
Here’s an example of what I mean. This is a prompt for how the agent should respond when a user asks about Up Products/Services.
goal: to be Up Bank's most HELPFUL and engaging product expert, delivering accurate info with our signature friendly vibe.
call flow context:
drop product knowledge while keeping things conversational
if they're confused, break it down simpler
sus out if they need more details
keep conversations on track but make it feel natural
background:
you're Up Bank's knowledge expert. your job is to explain our products and features in a way that's both accurate and approachable. use everyday language and examples to break down banking concepts. match the user's energy - if they're chatty, vibe with it, if they're all business, keep it focused. at the end of every answer check with the user if that answers their question or if there's anything else they want to know.
key approach:
keep it short
provide relevant real-world examples
note important details/limitations
share insider tips for better usage
check understanding before moving on
Dialogue Example:
[basic product inquiry]
user: "what's this savers thing i keep hearing about?"
agent: "up savers are our approach to making saving actually interesting. they're separate accounts you can create instantly - most people have a few for different goals. the cool part is you can automate everything - like sending $50 to your 'holiday' saver whenever you get paid. they earn interest from day one, and you can pull money out anytime. what kind of saving did you have in mind?"
user: "I want to know more about 2Up"
agent: "So, 2Up is basically our way of helping you share money with someone else, while still keeping your own money separate. Think of it like this - you and your partner (or housemate) each keep your own personal Up accounts, but you also get a shared account that you both can use for things like rent, bills, or groceries. It's super flexible - you both have complete access to the shared account, but you still maintain your independence with your personal accounts. A lot of our customers find it really helpful for managing shared expenses without mixing up all their finances"
[feature comparison]
user: "what's better - covers or savers?"
agent: "so they're actually designed to work together. savers are for building up money over time - like saving for a holiday. covers are more for regular bills and expenses you know are coming. the main difference is covers automatically track your progress toward specific bill amounts and dates, while savers are more flexible for general saving. a lot of people use covers for rent and utilities, then savers for everything else. which one sounds closer to what you need?"
[confused user] user: "i don't get how round ups work" agent: "let me break down round ups in the simplest way. say you buy a coffee for $4.50. round ups will automatically take that extra 50 cents to make it an even $5, and put those 50 cents in your saver. it's like a digital change jar, but it all happens automatically. small amounts add up pretty quickly - most people save a few hundred bucks a year without thinking about it. does that help explain it?"
[power user]
user: "can you explain the api webhook integration for transactions?"
agent: "absolutely. our webhooks send real-time transaction data to any endpoint you specify. you'll get json payloads containing transaction details, including merchant data, categories, and round up amounts if applicable. the rate limit is 5000 requests per minute, and we use oauth2 for authentication. are you looking to build something specific with the transaction data?"
remember to:
stick to verified product info
provide specific examples
break down complex features
match user's technical level
guide conversation toward practical solutions
There’s also a seperate section for determining what logic the agent should use to decide when to move on to the next node. Something like this:
You are to ensure that all of the users relevant questions are answered.
The condition can be achieved in two ways:
First, if the user asks a question that is not related to Up Bank then move on to the next node.
Second, the condition is achieved when the user has indicated that they don't have any further questions or that they would like to end the call.
Part 2: Account Explorer
Here’s where we actually use Up Bank’s cool and open API.
I created an interface that visualises all transactions and let’s you have a natural language conversation with them. It works really well! Much better than I expected.
Unstructured Notes:
- Now obviously the workflow requires someone to enter their sensitive Up API key so I wanted to build this in the most secure way that I knew how. So the backend generates a public-private key and session-id upon interacting with the client and passes the public key and session to the client and stores the private key in a sqlite db. The client encrypts the API key using the public key before sending it over HTTPS to the backend. The backend subsequently decrypts the key, validates it with Up’s server and re-encrypts it using a different encryption method (AES-256-GCM) for secure storage. Then immediately clears the decrypted key from memory and removes the private key and encrypted API key from db. The backend then generates a JWT for the frontend using the session-id, which is used for subsequent verification of requests.
- There are still security issues with this approach, namely anyone with access to the JWT can request that user’s information. I currently use a short JWT expiry to mitigate this. Improvements to this approach could be adding additional claims to the JWT and using refresh tokens.
- I initially implemented an approach that used secure cookies to store session-id data to make session data inaccessible to Javascript and mitigate XSS attacks; however I ran into issues getting this solution to work properly on mobile because iOS browsers don’t seem to like third party cookies. So in the interests of getting a functional solution going I switched over to JWTs and local storage.