Agent vs. Rules

It is fair to ask whether this needs AI at all. Could a few simple rules do the same job? So instead of guessing, I ran every case through a basic rules engine and compared the two.

The rules I compared against

1.Money already approved but not taken → take it
2.Card blocked or expired → stop
3.Bank flagged it as risky → ask a human
4.Anything else → send a payment link

That is about fifteen lines of code. It only looks at the last error and nothing else. If the AI cannot beat this, it is not worth having.

Same answer

73.3%

In 118 of 161 cases, the AI picked exactly what the rules would have picked. Most cases are easy, and on those the AI adds nothing.

Decided differently

Cases where the AI chose something else. These are the only cases where it is really doing any work.

After reading history

34 of 43

Most of those disagreements came after the AI looked up how the customer had paid in the past. That is something the error code alone cannot tell you.

Where they disagreed, and why

Stop hererules →Send a payment linkagent21 cases

The payment failed because the customer's card is blocked by the issuing bank, and no successful authorization exists for this order.

Order 489 is in 'attempted' status with amount 2499.00 INR.
Single payment attempt (pay_vzxYzhKuT6Wcc9) failed with error code BAD_REQUEST_ERROR, reason 'card_blocked'.
Customer has prior successful payments (wallet and UPI) indicating ability to pay when using valid methods.

Example: order order_e93IZGdKUwFI2A · ₹2,499 · the AI checked the customer's history first

What this tells me

Three quarters of this job does not need AI, and it would be silly to claim otherwise. The last quarter is where it helps. Take a customer whose card was just blocked. The rules give up. But if that same customer has paid happily by UPI twice this month, it is worth sending them a link.

It also catches mistakes. In ten cases the rules wanted to send a payment link to someone who had already paid on a later attempt. That would have charged them twice.

Even so, none of these choices went through unchecked. Every one still had to pass the rules before anything happened.