Paying a workforce: batch payments and their failure modes

Graduate · enrolled learners

This lesson opens with MyPayrollHR, September 2019.

What happened

MyPayrollHR was a payroll processing company that ran payroll for some 4,000 client businesses. Every other week it sent Cachet Financial Services, the processor that held the settlement account, a file listing which employee accounts to credit and by how much, and Cachet paid them out of that account. On 4 September 2019, amid a fraud by its operator, MyPayrollHR's file sent about 26 million dollars of client payroll money to an account it controlled at another bank instead of to Cachet, while still telling Cachet to pay the employees; that bank then said MyPayrollHR's account had been frozen. Cachet had already paid the workers with money that never arrived, so it used the same automated clearing house access that delivers direct deposit to reverse the deposits. Its first reversal request was improperly formatted, so it sent a second, and a number of banks processed both, so employees saw a month's worth of pay withdrawn from their accounts, leaving many overdrawn, for pay they had already earned and received. NACHA, which sets the clearing house rules, said about 8,000 employees from 400 companies were affected by the reversals, and in all nearly 35 million dollars of payroll and tax payments was left in limbo. Cachet later canceled the reversals and asked more than 100 receiving banks to reject them, leaving itself covering the 26 million, but the episode showed the failure mode of batch payroll: the processors in the chain hold the power to pay, and to un-pay, thousands of people in one action.

The decision point

Paying a workforce is done in a batch: one processor, given access to move money, pays everyone at once through a system like the automated clearing house. That concentration is efficient and it is the risk, because the same access that sends a batch of payments can reverse or block it, and a failure or fraud at the processor hits every worker simultaneously, not one at a time. MyPayrollHR is the case where the batch ran backwards: pay already delivered was clawed back from thousands of accounts through the very rails that had delivered it. The failure modes of batch payroll are therefore not just a missed run but a mass reversal, a mass block, or a mass misdirection, each landing on the whole workforce at once because the authority is pooled in one place. So the decision when running payroll through a processor is to treat that processor's access as a concentrated risk to be bounded: to know what it can do to funds after they are sent, whether pay can be reversed, under what conditions, and to structure the arrangement so that a processor's failure cannot silently pull back or withhold pay from everyone, because the convenience of paying a workforce in one action is inseparable from the danger of one action going wrong for all of them.

Recorded loss

$35,000,000

Orientation and Year One are open: anyone can read them without an account. From Year Two onward the lessons are for enrolled learners, because progress through the later years only means anything if it is tracked against a record.

It is free. We do not sell the list and there is nothing to buy at the end of it.