A University Told Border Force Its Data Could Not Support a Valid Risk Tool. Four Years Later the Tool Was Rating 80% of Detainees High or Extreme Risk.

Every person held in Australian immigration detention is given a risk rating. The rating decides where they are held and how they are moved — whether they travel to a medical appointment in restraints, and how many officers go with them. The instrument that produces it is named in these documents as the ABF/Serco Secure Risk Assessment Template, the SRAT.

In 2016 an internal audit said the SRAT needed an independent review. The Australian Border Force commissioned one from Griffith University in late 2018. The university reported in late 2019. This release, eighteen pages published on the Home Affairs disclosure log on 2 May 2024, contains what happened next.

What was released#

Five documents, eighteen pages, released in part. Eight of the pages are image-only scans, so the quotations below were read from the rendered pages rather than from a text extraction.

The set runs from March 2019 to November 2023: email correspondence between the university’s research team and Border Force officers about the project’s data (Document 2); a minute of 16 March 2020 to the Group Manager, Immigration Detention Group, reporting the university’s findings (Document 3); a minute of 16 September 2020 proposing to close the audit recommendation (Document 4); and a one-page statistical extract dated November 2023 (Document 5).

Redactions are extensive and are claimed under s. 47E(d) of the Freedom of Information Act 1982 — operations of agencies — and s. 47F(1), personal privacy, with s. 22(1)(a)(ii) used to remove direct contact details. In Document 3 the redactions cover the name of a third-party organisation and its tool. In Document 4 the greater part of two pages is withheld under s. 47E(d). We have not tried to work out what any of it says.

Officers below the Senior Executive Service Band 2 are referred to here by their role rather than their name, which is this desk’s standing practice.

The question the audit asked#

Document 4 reproduces the recommendation that started it. It comes from audit report DAT 03/2016, “Management of Detainee Risk Use of Pat-Down Searches and Restraints During Escorted Medical Appointments”. Recommendation 3B reads:

Through consultation with the Risk Section and FDSP, arrange an independent review of the FDSP Security Risk Assessment Tool (SRAT). The review should assess the SRATs technical methodology for articulating risk and provide greater transparency and assurance over the tools inputs, quality, performance and reliability. The SRAT should be aligned with the Department’s Risk policy, framework and matrix.

Recommendation 3C asked for governance around it: “a formal monitor and review program to document change to the scope and/or application of the tool”.

The audit was not asking whether the tool could be improved at the edges. It was asking whether the thing worked, and whether anyone could show that it worked.

What the university found#

The March 2020 minute sets out the project’s shape: in late 2018 the ABF invited Griffith University to undertake a project on “Improving Risk Assessment of Immigration Detainees”, beginning with a review of the evidence and of available instruments. Stage 2 was to “investigate the viability of building or validating a psychometrically valid and reliable risk assessment tool”. Stage 3 was to build it.

Neither happened. The minute records the finding in a single sentence:

The Griffith University has determined it is not possible to proceed with Stage 2 (investigate the viability of building or validating a psychometrically valid and reliable risk assessment tool) or Stage 3 (build the tool) of this project on the basis that ABF data systems do not have the information required to conduct the necessary analyses to build a psychometrically sound assessment tool.

The obstacle is not that the problem is hard. It is that the department’s own records do not contain what would be needed to test whether its own instrument works.

The next paragraph closes the obvious escape route:

Further, there are no already established tools with sound psychometric properties for the population of detainees.

There is no validated tool for this population to buy, and the data to build one is not there. The minute suggests one possible way forward — engaging with an outside organisation whose name is withheld under s. 47F(1) and s. 22(1)(a)(ii), about a screening tool it had developed — “as resources permit”.

The reason recorded for stopping#

The same minute explains why the work had not moved:

Since December 2019, Detention Operations has and is continuing to experience extraordinary resourcing priorities associated to bushfires and COVID-19 response activities, limiting our ability to progress the outcomes of report in a manner that assists in achieving a new and more reliable risk assessment tool.

The minute is dated 16 March 2020. Australia recorded its first COVID-19 death on 1 March 2020 and the national cabinet met for the first time on 13 March. The pandemic response was thirteen days old when it was cited as a reason work already stalled since December could not proceed.

The minute was signed for the Group Manager, Immigration Detention Group, Kaylene Zakharoff, and by the Commander of Detention Operations.

Closing the recommendation#

Six months later, on 16 September 2020, a second minute went to the Assistant Secretary of the Audit and Assurance Branch. Its title is its content:

PROPOSAL TO CLOSE IMMIGRATION DETENTION GROUP RECOMMENDATION THREE – DAT 03/2016

Its stated timing is “Seeking finalisation prior to the next Audit Committee on 19 November 2020”. The reasoning that accompanies the proposal is withheld under s. 47E(d), and we do not know what it says. What the release does show is the sequence: the review required by the audit had reported that the tool could not be validated, and the recommendation that required the review was put up for closure.

The data the project ran on#

Document 2 is the correspondence from the project’s first year, and it is a record of a dataset that would not hold. In March 2019 the university’s research team wrote to Border Force officers about errors in the incident data supplied to them — first one problem, then, two days later, a second reported to Assistant Commissioner Vanessa Holben of Detention and Offshore Operations Command, with a warning that it might require the project to be rescoped. The correspondence also records that the data arrived two months later than scheduled, and that revised milestones were accepted.

These are the ordinary frictions of a research project. They are also, read against what the university concluded eight months later, the first sight of the finding: the information needed to test the tool was not in the systems.

What the tool was doing in the meantime#

Document 5 is a single page prepared in November 2023 — three and a half years after the minute proposing closure. It gives the current state of play, using the tool that was never independently validated:

As at 15 November 2023, there were 928 people in held detention. Of these: 716 (77.2% of held detention cohort) were categorized as high or extreme risk for placement; and 747 (80.5% of held detention cohort) detention were categorised as high or extreme risk for escort.

Four in five people in immigration detention were rated at the top two levels of risk for escort. Whether that distribution reflects the population or the instrument is precisely the question the 2016 audit asked, and precisely the question the 2019 review found it could not answer.

The redactions the department planned in advance#

One passage in the March 2020 minute is about this release’s own subject matter. Its second stated purpose is:

Seek endorsement for Detention Operations to request FOI redactions of interviewee participation information within the Griffith University’s Final Report – Improving Risk Assessment of Immigration Detainees.

The department sought sign-off on what it would black out of the university’s report before anyone had asked for it. On its face the concern is the privacy of people who took part in interviews, which is a proper thing to protect. What the document shows is the timing: the redaction position was settled as part of receiving the report, not in response to a request for it.

What the documents do not say#

Whether the recommendation was actually closed. The September 2020 minute is a proposal, with tick-boxes for “Agreed” and “Noted” that are blank on the released copy. The Audit Committee’s decision of 19 November 2020 is not in this release.

The reasoning for the closure proposal. The substance of Document 4 is withheld under s. 47E(d). We do not know what argument was made.

Whether the tool was ever reviewed afterwards. Nothing here covers 2021 or 2022. The next document jumps to November 2023.

What the university actually recommended. The final report is an attachment (Attachment A) that is not part of this release. The minute cites “pages 13 through 22” of it. Those pages are not here.

What the risk ratings mean in practice. The November 2023 extract gives proportions, not the consequences that follow from a rating.

Who the third party was. The organisation whose screening tool was suggested as a possible starting point is withheld under s. 47F(1). We have not speculated.

How to check this#

Every quotation is from the single PDF linked at the top of this page. Documents 3, 4 and 5 carry the agency’s own document numbers in the top right corner; page references are to the numbering on those documents, with the PDF page given where it differs. Eight of the eighteen pages have no text layer, so those quotations were read from rendered images at 150 dpi — if you extract text from the file and get nothing for those pages, that is the scan, not the quote.

Eighteen pages is small enough to read in full, and we did.