Do it twice, count it once
An agent that loses power halfway through a payment cannot know whether the payment happened. If it tries again you may pay twice, and if it does not you may not pay at all. One small habit removes the question. Cut the power yourself below.
An agent pays a supplier. The payment reaches the books. Before the answer comes back, something ordinary happens: the power goes, a connection drops, a request times out.
The agent now has a problem it cannot solve by thinking. From where it stands, "the payment never happened" and "the payment happened and I was not told" look exactly the same. If it tries again, it may pay twice. If it does not, the bill may never be paid.
Most automation is written as if this moment does not exist. It works in the demonstration and for the first few weeks, and then one day a supplier is paid twice and nobody can say why, because nothing crashed and no error was shown.
Try it
One agent, one bill of Tk 12,000, one set of books. Cut the power while the agent is paying, then let it try again. Do it first with the key switched off, then with it on.
- Nothing paid yet.
One bill to pay. Pay it, or cut the power while the agent is paying it.
With the key off, every lost answer costs another Tk 12,000. With it on, you can cut the power as many times as you like: the books recognise the payment, keep the first one, and give the agent the same receipt again.
The habit
Give every action that changes something a name that stays the same across tries, and make the place that records the action refuse to record the same name twice.
function pay(books: Books, bill: Bill) {
const key = `bill-${bill.id}`; // the same on every try
const before = books.find(key);
if (before) return before.receipt; // a repeat: answer, do not act
const receipt = transfer(bill.amount, bill.supplier);
books.record(key, receipt); // saved together with the effect
return receipt;
}That is all of it. The agent is now free to retry whenever it is unsure, and being unsure stops being dangerous.
People call the property idempotent: doing it twice leaves the same result as doing it once.
Three ways to get it wrong
A key that changes on each try. A random number made at the moment of trying is a new name every time, so the books see a new payment every time. The key must come from the job: the bill number, the order number, the id of the message that asked for the work.
A key checked in one step and saved in another. If two tries arrive together, both can look, both can find nothing, and both can pay. The check and the record have to be one step. In a database that is a unique rule on the key column, so the second write fails by itself.
A key that stops at your own door. Your books may know the key while the bank, the courier or the mail service does not. Many such services accept a key of this kind with each request. Pass yours through. Where a service accepts none, ask it what it already has before acting.
Check your own system
Tick each line that is true of yours today. Your answers stay on this page and are sent nowhere.
4 lines. Tick the ones that hold, and what is left is your list of work.
The harness on the first page of this site does the fourth one in front of you. Pull the plug on any agent at any moment, and at the end it compares its books with a morning in which nobody did.