Automating systems can assist administrative decision-making in a number of ways. For example, they can:

Automated assisted decision-making can help identify:

The benefits of automation are not unlimited. Speaking to the Australian Corporate Lawyers Association in 2010,⁵ then President of the Administrative Appeals Tribunal, the Honorable Justice Garry Downes, anticipated some of the problems that automated decision-making has generated:

Justice Downes was not the first to voice such concerns. In 2004, the Australian Administrative Review Council (“ARC”) was one of the first of its kind to consider the growing use of computers to automate governmental decisions. In its Report No 46, Automated assistance in administrative decision-making, the ARC examined what kinds of decisions are suitable for automation and how errors can be avoided or, if made, addressed.

The ARC’s 27 “best practice” principles for decision-making undertaken with the assistance of automated systems were adopted by the Australian Government Information Management Office’s Automated assistance in administrative decision-making: better practice guide in February 2007.⁶ The Better practice guide was directed to automated systems that “build in and automate administrative decision-making logic into a computer system”:⁷

A hallmark of this kind of automated system is its ability to examine a set of circumstances (data entered by the user) by applying “business rules” (modelled from legislation, agency policy or procedures) to “decide” dynamically what further information is required, or what choices or information to present to the user, or what conclusion is to be reached.

The Better practice guide identifies the practice areas that “require particular care with respect to the development and management of automated systems for administrative decision-making”,⁸ covering the requirements to:

A checklist⁹ (expressed to be advisory not mandatory) summarised items that should be addressed when considering the implementation or update of an automated system for administrative decision-making.

The Better practice guide lauds appropriately designed and managed automated systems as offering agencies “potential benefits in consistency, cost efficiency, time economies and new service provision”:¹⁰

In addition to time and cost savings, automated decision-making has the capacity to diminish or even eliminate the risk that a decision will be invalidated by reason of:

The use of automated systems may raise difficult questions about whether Parliament can authorise decision-making by a non-human agent and the reviewability of such decisions. Where a person purports to exercise public decision-making powers, they must be authorised to do so as a matter of law. Equally, if a technological assistant or automated system is being utilised to make part or all of a decision, the use of that system must be authorised. In a paper presented at the Cambridge Centre for Public Law Conference in 2014, Justice Melissa Perry observed that:¹³

Thus, for example, s 495A(1)  of the Migration Act 1958 (Cth) provides that:

Pursuant to s 495A(2), the Minister is essentially deemed or taken to have made a decision, exercised a power or done something else related to making a decision where that was made, exercised or done by the operation of a computer program. However, as Perry J notes, this raises some unique problems with the concept of “delegating” a decision to an automated system, in whole or in part:¹⁴

The Better practice guide recognises that governmental agencies may find it difficult to model business rules from legislation and that there may be a need to resolve legislative complexity where ambiguity in interpretation exists.¹⁵ The ARC report developed this concern, highlighting four areas in which an error can be made in applying legislation to determine an entitlement:

Automation thus may not eliminate the risk of incorrectly applying statutory requirements, because decisions could be based on a misinterpretation of the applicable legal requirements or an incorrect application of those legal requirements to the facts found by the decision-maker. Laws are interpreted in accordance with statutory presumptions and meaning is also affected by context.¹⁶ One of the greatest challenges is to ensure accuracy in the substantive law applied by automated processes. Through the process of translating laws into code, computer programmers effectively assume responsibility for building decision-making systems. The potential for coding errors is real: as Perry J has observed:¹⁷

In those circumstances, grounds of challenge therefore include that the automated decision-maker:

Australian public law has developed a wide range of grounds justifying judicial intervention to overturn administrative decisions or action.²³ The Better practice guide implicitly recognised there may be a myriad of grounds for challenge, by limiting its guidance on automated systems’ treatment of discretion and judgment.²⁴ The Better practice guide assumes that discretionary decisions and matters of judgment will be left to human decision-makers, and touches only upon these issues at a high level. In so doing, it acknowledges that the potential benefits of automation are complicated by the substantive expectations of Australian public law:²⁵

This limitation was consistent with the ARC’s own caution in setting principles for assessing the suitability of expert systems²⁶ for administrative decision-making:

The Better practice guide advises that human discretion and judgment be permitted “where relevant”, and exhorts system designers to be careful that the system “does not fetter” the decision-maker in exercising any discretion he or she has been given under relevant legislation, policy or procedure.²⁷ It suggests agencies support human discretion and judgment by:

Yet even where discretionary decisions are left to humans, deploying automated procedures can still be problematic. Technology-assisted decision-making may frame a decision-maker’s consideration of the issues and make determinations of what is relevant and irrelevant. If automated systems are programmed to be used when a discretion or judgment should have been reserved for a human decision-maker, not only would there be a constructive failure to exercise the discretion, but predetermined outcomes may be characterised as pre-judgment or bias.²⁸

Generally, only a final, or operative and determinative decision, is administratively reviewable, not the interim steps or conclusions.²⁹ However, those interim steps may be considered to the extent that they affect the final decision. It follows that, where interim steps in a decision-making process are automated, but the final or operative decision is made by a human, there is the potential for the decision to be affected by a jurisdictional error should there have been an error in the automation process. Similarly, failure by an automated system to bring relevant issues or material to a decision-maker’s attention will not absolve the final decision-maker from failing to consider them.³⁰

Critically, automating the process of decision-making will not ipso facto avoid practical injustice, such as may occur if a decision-maker (be they human or automated):

If an automated assisted decision-maker acts on insufficient evidence, the decision may be open to challenge because it:

Recalling Justice Downes’s concerns in 2010, errors in computer programming can result in wrong decisions potentially on an enormous scale. For example, if a statutory test is misconstrued when the technology was developed, then that misconstruction could taint any decision made with the assistance of the technology. Automated assisted decision-making may well be productive of deficient reasoning, because the system design leads the (human or automated) decision-maker to:

In this sense, the great efficiency of automated systems could also be their biggest downfall — a lesson illustrated by the Commonwealth Department of Human Services (“DHS”) and Centrelink in the launch of a new online compliance intervention (“OCI”) system for raising and recovering social security debts in late 2016.

Automated decision-making is authorised by s 6A of the Social Security (Administration) Act 1999 (Cth). The OCI system matched the earnings recorded on a customer’s Centrelink record with historical employer-reported income data from the Australian Taxation Office (“ATO”). If the customer did not engage with DHS (either online or in person), or if there were gaps in the information provided by the customer, the system filled the gaps with a fortnightly income figure derived from the ATO income data for the relevant employment period (“averaged” data).⁴⁵

The Australian Senate Community Affairs References Committee conducted extensive hearings into the OCI system as part of its Inquiry into the Better Management of the Social Welfare System Initiative.⁴⁶ The Committee investigated, inter alia, Centrelink’s OCI system and its compliance with debt collection guidelines and Australian privacy and consumer laws, as well as “the impact of Government automated debt collection processes upon the aged, families with young children, students, people with disability and jobseekers and any others affected by the process”.

Evidence before the Senate Inquiry suggested that the methodology on which the OCI system was based — that is, utilising a data-matching algorithm that measured income on an annual basis and divided it into equal fortnightly instalments — generated misleading data upon which subsequent decision-making was based.

Witnesses expressed concern that the automatic debt-raising process appeared not to allow for the complexities of casual work, and this was arguably worse if a person is paid considerably in arrears.⁴⁷ The system design applied an algorithm that assumed income is distributed evenly over 26 fortnights, with a presumption that work is either full-time or part-time, but with no provision for intermittent, or casual, work.

In its 12 May 2017 submission to the inquiry, Victoria Legal Aid demonstrated how calculating a debt by way of algorithms did not predict an accurate outcome:⁴⁸

The Commonwealth Ombudsman’s Investigation Report published in April 2017 was critical of the debt recovery program generally as to its fairness and reasonableness.⁴⁹ Poor service delivery was also a recurring theme in many complaints received by the Ombudsman. Customers had problems getting a clear and transparent explanation about the debt decision and the reasoning behind it.

The design and operation of the Non Employment Income Data Matching (“NEIDM”) project, the data-matching process designed to enable DHS to match income data it collects from customers with tax return data reported to the ATO, suffered from a limitation of purpose. The program protocol (which became public in May 2017) made clear that:⁵⁰

Evidence given by Mr Jason McNamara, General Manager, Integrity Modernisation, Department of Human Service, on 16 May 2017⁵¹ gave some insight into the limitations of the data matching program, and improvements made following the Ombudsman’s Report:

The OCI system is a complex automated system that was rolled out on a large scale within a relatively short timeframe. The Ombudsman observed that inevitably there would be problems with the rollout of a system of that scale. The Ombudsman was particularly critical of the planning and risk management work undertaken:⁵²

The Ombudsman expressly cited the 2007 Better practice guide to reinforce that key considerations in developing automated decision-making systems are whether the system is consistent with administrative law values of lawfulness, fairness, rationality, openness/transparency and efficiency:⁵³

There is always a risk that administrative reviewers will assume a computer system’s infallibility. The system audit trail can thus provide critical forensic insights into potential coding and system planning errors that will challenge — or make good — such assumptions.

The data-matching Protocol for the NEIDM project did not identify whether it was developed in accordance with the Better practice guide or the OAIC Guidelines on data matching in Australian Government administration.⁵⁴ There was also nothing that identified relevant policy or legislation related to specific income tests that should be applied. Indeed, the NEIDM Protocol only made limited reference to quality controls and audit trails:⁵⁵

An audit trail allows for the research and reasoning processes to be comprehended and tracked by a human mind. Automated administrative decisions will be particularly vulnerable to attack where the system does not provide a clear and sufficiently detailed audit trail to make clear what factors have been taken into account. The Better practice guide acknowledged this issue in declaring that audit capability is “essential to accountability”⁵⁶ and in generating a detailed checklist of measures:

The Ombudsman concluded his report by detecting an additional flaw in the system design, which highlighted that a human decision-maker may have exercised differently a discretionary power to waive a 10% recovery fee which was embedded in the business rules encoded in the OCI system:⁵⁷

The final report of the Senate Committee, delivered on 21 June 2017, recorded that in response to the Commonwealth Ombudsman’s recommendations, the Department ceased the automatic charging of a 10% debt recovery fee, and now provides information on how individuals can apply not to have this fee imposed where they have a reasonable excuse.⁵⁸

In the past, citizens understood that automated systems helped humans apply rules to individual cases. Today, citizens need to appreciate that automated systems can be primary decision-makers, taking human decision-making out of the process. The Senate Committee report concluded that many individuals failed to appreciate the use of automation in the decision-making process, which negatively impacted on their responses to the system’s design, misunderstanding the need to engage and supply information when data-matching identified differences:⁵⁹

The technological landscape has substantially changed from that surveyed by the ARC in 2004 and on which the 2007 Better practice guide was based.⁶⁰ They, too, responded to a world in which automated assisted decision-making systems were grounded in logic and rules-based programs that applied rigid criteria to factual scenarios, responding to input information entered by a user in accordance with predetermined outcomes. They focussed on decision support tools, not expert systems that could replace human discretion and judgment.⁶¹

In no meaningful way did they contemplate the challenges now posed by machine learning, “Big Data”, and the rapidly changing world of data management. Machine learning is a branch of AI that allows computer systems to learn directly from examples, data and experience. Through enabling computers to perform specific tasks intelligently, machine-learning systems can carry out complex processes by learning from data, rather than following pre-programmed rules. As it learns over time, the machine responds to feedback, so that the patterns learnt yield useful predictions or insights.⁶² By increasing our ability to extract insights from ever-increasing volumes of data, machine learning could increase productivity, provide more effective public services, and create new products or services tailored to individual needs.⁶³

Machine learning helps extract value from “Big Data”, which is the phenomenon of “high-volume, high-velocity and/or high-variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight, decision making, and process optimization”.⁶⁴ So called “data mining” (the automatic extraction of implicit and potentially useful information from data) is increasingly used in commercial, scientific and other application areas, powered by increasingly complex Operational Database Management Systems (“ODMS”).⁶⁵

Automated decision-making by computer algorithms based on data from human behaviours is becoming fundamental to our digital economy. Such automated decisions can impact everyone, as they now occur routinely not only in government services but:

Because public sector data can be a key enabler and catalyse a range of economic activity, Commonwealth and State governments have devised whole-of-government strategies to deal with Big Data, its use and sharing. In so doing, governments increasingly appreciate data as an asset, and most recognise it as something to be used for the public good. Thus, the Australian Public Service Big Data Strategy released in August 2013⁶⁶ expressed that government data is a national asset that should be used for public good:

Governments’ use of Big Data and ODMS, like any other form of data or information collection, is subject to a number of legislative controls. Thus, agencies need to comply with the Data-matching Program (Assistance and Tax) Act 1990 (Cth) wherever tax file numbers are used. The use of Big Data is also regulated by the Privacy Act 1988. At a Commonwealth level, other controls also include:

In NSW, the Data Sharing (Government Sector) Act 2015 removed barriers that impede the sharing of government sector data and implements measures to facilitate the sharing of government sector data with a new Data Analytics Centre and between other agencies. The data sharing legislation complements existing NSW legislation. Sharing of personal data is excluded.⁶⁷ All data identified in the Government Information (Public Access) Act 2009 as exempt from public release are also specifically exempt from the data sharing legislation.⁶⁸

The expansive use of data is acknowledged by the NSW Government Open Data Policy,⁶⁹ which expresses an aim to release data for use by the community, research sector, business and industry and to accelerate the use of data to derive new insights for better public services. It adopts an open access licensing framework to support the release and reuse of public information, such as by NSW Transport’s data exchange program, the Land and Property Information’s provision of spatial data, and NSW Education Datahub.

The NSW model contrasts with the legislative environment in South Australia which modelled a Public Sector (Data Sharing) Act 2016 on the NSW approach, but without existing privacy legislation or a modern information access regime. The SA legislation takes a different approach by incorporating privacy and a public interest test for sharing in the Trusted Access Principles that govern the provision of information.⁷⁰ This approach provides an authorising environment to facilitate both open data and data sharing more broadly.

The real value in the use of Big Data lies in its predictive potential. A step beyond data mining, data profiling enables aspects of an individual’s personality or behavior, interests and habits to be determined, analysed and predicted.⁷¹ There are vast sources of data used in profiling to build up a picture of an individual. Technologies that generate tracking data (such as smartphones, credit cards, websites, social media and sensors) offer unheralded benefits. Data sources include (but are not limited to):⁷²

Profiling is not as transparent as other forms of data processing. Profiling can involve predictive elements, which can increase the risk of inaccuracy. Profiling is not always visible, and may take place without an individual’s knowledge. Correlations may include hidden biases that have an unintended or discriminatory effect on certain populations. Profiling can emphasise existing stereotypes, social segregation, and limit individual choice and equal opportunities. There is concern that some humans are particularly vulnerable in this area, for example children⁷³ and those with mental⁷⁴ and physical disabilities.

Profiles also tend to be dynamic and evolving. Profiling may generate new data for an individual based on data relating to other people. Profiling data might also be harvested or mined for information and its commercial value. Profiling where algorithms can discover new correlations may prove useful at a later date.

The following table prepared by the UK Information Commissioner’s Office for its April 2017 consultation on Profiling and automated decision-making highlights some of the more widely recognised benefits and risks of profiling:⁷⁵

The Royal Society in its report published on 27 April 2017, expanded on the social issues associated with machine learning applications:⁷⁶

Internationally, data protection authorities have expressly signaled their intention to closely monitor privacy concerns relating to the use of profiling,⁷⁷ Big Data and the evolution of the Internet of Things.⁷⁸ In 2018, European member states (including the UK) will introduce new regulations that govern how data usage can be challenged.⁷⁹ The General Data Protection Regulation (“GDPR”) replaces the Data Protection Directive 95/46/EC and is designed to harmonise data privacy laws across Europe. It is focused on the outcome of automated decision-making (which could include profiling) rather than the act of profiling itself.

Early drafts of the GDPR enshrined what is called a “right to explanation” in law, although research groups argue the final version as approved in 2016 contains no such legal guarantee, as it will depend on interpretation by national and European courts.⁸⁰ The GDPR applies to all companies processing and holding the personal data of data subjects residing in the EU, regardless of the company’s location. Accordingly, the GDPR will impact globally.

The practical problem with the use of algorithms, especially those used in profiling of human characteristics and behaviours, is that they may be protected by trade secrecy laws, and thus remain impenetrable to outside observers. Algorithms, especially those based on deep learning techniques,⁸¹ can also be so opaque that it will be practically impossible to explain how they reach decisions in any event. Indeed, a characteristic of the Artificial Neural Network (“ANN”) is that after the ANN has been trained with datasets, any attempt to examine its internal structure to determine why and how it made a particular decision is impossible. The decision-making process of an ANN is and remains opaque. Thus, no-one quite knows how the moves of Google DeepMind’s AlphaGo artificial intelligence program beat the world’s top Go player — the feat was “beautiful but puzzling”.⁸²

This is known as “the black box problem”.⁸³ Within computing, “black box” may describe a situation where we can only observe inputs and outputs; what really happens to the inputs occurs “in the dark”. Thus, in completely automated decision-making systems, every input and most outputs are encapsulated in the box too: the information sources representing inputs are not observable. Yet in democracies operating under the rule of law, it is basic that people should enjoy openness and the possibility to make up their own mind and take issue if they disagree with the exercise of governmental powers. Access to information regarding the detailed rules in the system should be publicly available. Can we avoid or manage the “black box problem”?

Because human beings program predictive algorithms, their biases and values are embedded into the software’s instructions, known as the source code and predictive algorithms. Algorithms are also what make intelligent robots intelligent. Intelligent robotics blurs the very line between people and instrument.

This new reality is spurring a branch of academic study known as “algorithmic accountability”,⁸⁴ which focuses on critical questions like:

Some academics argue that “a new concept of technological due process is essential to vindicate the norms underlying last century’s procedural protections”.⁸⁵ Procedural protections could apply not only to the scoring algorithms themselves (a kind of technology-driven rulemaking), but also to individual decisions based on algorithmic predictions (technology-driven adjudication). In their joint work, The scored society: due process for automated predictions, law professors Danielle Keats Citron and Frank Pasquale argued:⁸⁶

Development of artificial intelligence and autonomous systems (“AI/AS”) are also giving rise to many complex ethical problems. Although software engineers initially identify the correlations and inferences programmed into algorithms, Big Data promises to eliminate the human middleman at some point in the process. In time, predictive algorithms may evolve to develop an artificial intelligence that guides their own evolution.

The most ethically challenging and risky type of decision-making systems are arguably those governmental decisions that:⁸⁷

Professor Alan Winfield, a professor of robot ethics at the University of the West of England, accepts that it may be difficult to explain an AI’s decision — and for AI, the “black box problem” may well be intractable. Professor Winfield argues (emphasis in original):⁸⁸

The risk that algorithms will make bad decisions that have serious impacts on people’s lives has also lead to calls for a supervisory or regulatory body to ensure transparency and fairness. Such a body could have the power to scrutinise and audit algorithms, so they could go in and see whether use of predictive algorithms which mine personal information to make guesses about individuals’ likely actions and risks is actually transparent and fair.⁸⁹

Professor of philosophy Samir Chopra — who discussed the dangers of opaque agents in his 2011 book (with Laurence White) A legal theory for autonomous artificial agents⁹⁰ — argues that their autonomy from even their own programmers may require them to be regulated as autonomous entities. He suggests an all-encompassing body — a “Federal Robotics Commission” — could deal with the novel experiences and harms robotics enables, dedicated to the responsible integration of robotics technologies into society.⁹¹

Whatever be the type of regulator, the desire to craft a cohesive regulatory theory to underpin algorithmic accountability is finally gathering pace. Professor Frank Pasquale in his research paper, Toward a fourth law of robotics: preserving attribution, responsibility, and explainability in an algorithmic society⁹² builds on the October 2016 lecture by Yale University Law Professor Jack Balkin, The three laws of robotics in the age of Big Data, in which Professor Balkin advanced a proposal for a set of “laws of robotics” for an algorithmic society:⁹³

Professor Balkin argues that public law obligations of transparency, due process and accountability flow from these three substantive requirements:⁹⁵

Professor Pasquale identifies the cornerstone of Balkin’s proposal as being to create obligations of responsibility in systems that do not necessarily share the human experience of intent. The Royal Society acknowledges that society has “yet to test the boundaries of current models of liability or insurance when it comes to new autonomous intelligent systems”,⁹⁶ and suggests different approaches to addressing this issue, including:

To deal with the requirement for attribution for the purposes of attributing legal responsibility, Professor Pasquale proposes a fourth law to complement Balkin’s first three: “A robot must always indicate the identity of its creator, controller, or owner”⁹⁸ (recognising that there may be multiple potentially responsible parties for any given machine’s development and eventual actions).

Pasquale argues, with some persuasion, that to make Balkin’s principles effective, regulators will need to require “responsibility-by-design” to complement extant models of security-by-design and privacy-by-design. That may involve requiring certain hard-coded audit logs in both closed and open robotics, or indeed licences in open robotics that explicitly contemplate problematic outcomes. Initiatives like these will not simply regulate robotics post hoc, but necessarily influence systems development, by foreclosing some design options, and encouraging others.

One computer science researcher has already generated a general framework for “accountable” algorithms, unifying a suite of tools from cryptography to design processes that enable after-the-fact oversight, consistent with the norm in law and policy.⁹⁹ The work demonstrates that it is possible for accountable algorithms to attest to the valid operation of a decision policy even when all or part of that policy is kept secret.

Professor Winfield, mentioned above, is also leading development of a new standard on transparency in autonomous systems, based on the simple principle that it should always be possible to find out why an AI or robot made a particular decision. Professor Winfield heads a British Standards Institute working group on robot ethics to develop industry standards for AIs that aims to make them transparent and more accountable.¹⁰⁰ The group drafted a new standard BS 8611 Guide to the ethical design of robots and robotic systems, published in April 2016, believed to be the world’s first standard on ethical robots. Also in 2016, the very well regarded IEEE standards association — the same organisation that gave us WiFi — launched a Global initiative on ethical considerations in AI and autonomous systems.¹⁰¹ In December 2106, the IEEE published Ethically aligned design: a vision for prioritising human wellbeing with AI and autonomous systems.¹⁰² The purpose of this initiative is to ensure every technologist is educated and empowered to prioritise ethical considerations in the design and development of autonomous and intelligent systems; in a nutshell, to ensure ethics is “baked in”.

Ethical issues almost always directly translate into concrete legal challenges — or they give rise to difficult collateral legal problems. A challenge for policy-makers will be to develop an approach to assessing the ethical risks associated with automated decision-making systems that will be simple and pragmatic, easily incorporated as part of a standard risk assessment exercise, and can be undertaken during a system’s design phase.

Appropriate action can then help to mitigate the latent risk, for example, by:

Intelligent robotics is shaping up to be the next transformative technology of our times. Their increasing role in our lives merits systematic reassessment and changes to law, institutions, and the legal profession and academy.¹⁰³ Critical evaluation is necessary to avoid addressing policy questions in a piecemeal fashion, with increasingly poor outcomes, and slow accrual of knowledge. The literature on the difficult theoretical questions of AI ethics, algorithmic system ethics, and Big Data and ethics, is now vast, well-established and easily accessible. The legal profession, judiciary and academy should be well placed to support an informed public debate about what we want algorithms, machine learning, and AI to do, and how the benefits can best be distributed.