We are already willing to wear or carry devices that provide much detail about our circumstances to databases.⁴ Our mobile phones are capable of providing real-time spatial location data,⁵ as well as retaining a secret record of every location that we visit.⁶ We have embraced highly contextualised and automated directives in the travel context, eagerly (and sometimes blindly) accepting directions from Google Maps.⁷ The capability of machines to invade human privacy will only increase.⁸

The more convenient an agent is however, the more it needs to know about a person (preferences, timing, capacities, etc). This creates a tradeoff — more help requires more intrusion. The record to date is that convenience overwhelms privacy — autonomy and/or independence will increasingly be sacrificed and replaced by convenience.⁹

New brain imaging techniques point to a future in which our thoughts will not be as private as they are now.¹⁰ People could be scanned for one purpose, for example to see how advertising campaigns affect their brains, while they inadvertently generate information that bears on their racial biases, sexual orientation or other sexual preferences.

AI-enhanced photorealistic pictures and videos, or “deep-fake” technology¹¹ is becoming pervasive. For example, Philip Wang, a software engineer at Uber, developed a website called ThisPersonDoesNotExist.com, that creates an endless stream of fake portraits. The algorithm that powers it is trained on an enormous dataset of real images, and then uses a neural network known as a generative adversarial network (or GAN) to fabricate new examples. In a Facebook post, Wang wrote:¹²

In February 2019, the creators of a revolutionary AI system that can write news stories and works of fiction — nicknamed “deep fakes for text” — took the unusual step of not releasing their research publicly, for fear of potential misuse. OpenAI, a non-profit research company backed by, among others, Elon Musk, stated that its AI model, called GPT2, is so good and the risk of malicious use so high, that it is deviating from its normal practice of releasing the full research to the public in order to allow more time to discuss the ramifications of the technological breakthrough.¹³ GPT2 is fed text — anything from a few words to a whole page — and is then asked to write the next few sentences based on its predictions of what should come next. GPT2 is capable of writing plausible passages that match what it is given in both style and subject.¹⁴

The ability to manipulate and generate realistic imagery at scale will have a huge effect on how modern societies think about evidence and trust. Such software may also be used for creating political propaganda and influence campaigns.¹⁵ The courtroom is not immune to misleading evidence. Fake evidence will inevitably leak into the courtroom with the potential to dupe fact-finders.

Chesney and Citron predict a development stemming from deep-fake evidence — “immutable life logs” as an alibi service.¹⁶ Because deep-fake technology will be able to portray people saying and doing things that they actually never did or said, alibis will become essential for digitally ensnared accused to prove their innocence in the courtroom. Hence, deep-fakes will create a heightened demand for proof of where a person was and what they were doing at all times.

In the United States, both the federal and State governments have outsourced many regulatory and legal decisions to computation. Tax returns are too voluminous for IRS personnel to examine manually; “audit flags” are programmed to determine which returns should receive greater scrutiny or be rejected outright. Homeland Security officials are using big data and algorithms to determine which travellers pose a security risk and who can pass unmolested to their flights. So called “predictive policing” deploys law enforcement resources before crimes are committed. And, once perpetrators are convicted, “evidence-based sentencing” may quantify punishment by using data and algorithms to adjust the length of prison sentences based on myriad factors.¹⁷

Privacy proponents will recoil upon learning that AI is also increasing the effectiveness of State surveillance techniques.¹⁸ Before AI, cameras were useful only to the extent that someone either observed a live feed or reviewed recorded footage. That time has passed. With the assistance of AI, cameras can now navigate three dimensions and make sense of what they “see” — all without any human intervention or assistance. Moreover, AI-augmented cameras are beginning to operate beyond ordinary human capability — they can identify millions of faces and predict human behaviour.¹⁹

The advent of China’s social credit system (“SCS”) is a sign of what may eventuate where an individual’s rights and liberties will be determined by the SCS. This is the Orwellian nightmare realised.²⁰ New technologies make it possible to match images and voices with other types of information, and to use AI on these combined data sets to improve law enforcement and national security. Through its Sharp Eyes program, Chinese law enforcement is matching video images, social media activity, online purchases, travel records and personal identity into a “police cloud”.²¹ This integrated database enables authorities to keep track of criminals, potential law-breakers and terrorists.²²

Facial recognition technology is nothing new. We see it, for example, on the iPhone X with its face-scanning technology.²³ But, thus far, China is the world leader in using facial recognition technology as a surveillance tool. Under the Sharp Eyes program, China’s goal is to recognise all Chinese citizens within seconds of their faces appearing on a camera.²⁴ In May 2018, the Chinese government introduced a travel ban on people with poor “social credit”.²⁵

In the world of technology, facial recognition has become a known commodity. Behaviour prediction, on the other hand, is a recent trend. In addition to recognising who you are, AI-augmented cameras will be “intelligent” enough to predict your behaviour. This technology already exists, and it is improving by the day.²⁶

New AI software is being used in Japan to monitor the body language of shoppers for signs that they are planning to steal. This software, developed by Japanese company Vaak, differs from similar products that match faces to criminal records. Instead, VaakEye uses algorithms to analyse footage from security cameras to spot fidgeting, restlessness and other body language cues that could be suspicious and then alerts shop employees about potential thieves via an app.²⁷

Using AI to apprehend potential shoplifters raises ethical questions. For example, even though the incentive of such software may be to prevent theft, is it legal, or even moral, to prevent someone from entering a shop based on this? To exacerbate these concerns, there is also the potential of AI being used to fuel discrimination. A 2018 study by researchers from the Massachusetts Institute of Technology and Stanford University found that various commercial facial-analysis programs demonstrate skin type and gender biases, depending on the types of data used.²⁸ Technologies that rely on algorithms, particularly in regards to human behaviour, have the potential for discrimination. After all, humans have to train the algorithms what or whom to treat suspiciously.

One way in which police arrest suspects is through arrest warrants, which, in most common law jurisdictions, is based on a “reasonable grounds” standard. If an AI-equipped camera identifies someone as a likely criminal, will that be enough to meet the reasonable grounds standard? If so, and assuming the technology assigns a percentage of criminality to an individual, how much will satisfy reasonable grounds — 90%, 70% or 50%? This, of course, also raises the question of whether it is even ethical let alone lawful to arrest a person before they commit a crime.²⁹

What about the role of this technology as admissible evidence in the courtroom? Would it be too prejudicial to show the fact-finder that AI software determined that an accused is a criminal? What if, instead, prosecutors used the technology during a trial to buttress their arguments? In a closing address, for example, the prosecutor might argue: “Based on all the eye-witness testimony, along with the determination that the accused, considering his facial features, has an 80% likelihood of having committed the crime charged, you should find the accused guilty.”³⁰

These types of arguments could be commonplace in the future. There will be a need for clarity from lawmakers and regulators regarding who will ultimately need to decide in what circumstances the use of this technology will be appropriate or desirable as a matter of public policy.³¹

Developments in technology raise important policy, regulatory and ethical issues.³² For example, how should we promote data access? How do we guard against biased or unfair data utilised in algorithms? What types of ethical principles are introduced through software programming, and how transparent should designers be about their choices? What about legal liability in cases in which algorithms cause harm?³³

It must be remembered that technology is not necessarily neutral and objective. A software design may expressly, through its programming, reflect a preference for certain values over others.

AI systems can also be inadvertently programmed to have bias because of the biases of the programmers or, with the development of self-learning algorithms, actually learn to be biased based on the data it is learning from.³⁴ In addition, AI systems find it more difficult to generalise findings from a narrower dataset, with minor differences from a training set potentially making larger-than-intended impact on a prospective set of data, creating potential bias.³⁵

A 2017 study demonstrated that AI can learn to have racist or sexist biases based on word associations that are part of data it was learning from, and sourced from the internet that reflected humanity’s own cultural and historical biases.³⁶

Algorithms — the set of instructions according to which computers carry out tasks — have become an integral part of everyday life, and they have immersed themselves in the law.³⁷ In the United States, judges in certain States use algorithms as part of the sentencing process to assess recidivism risk.³⁸ Many law enforcement agencies use algorithms to predict when and where crimes are likely to occur (so-called “predictive policing”).

Most algorithms are created with good intentions, but questions have started surfacing over algorithmic bias on employment search websites, in credit reporting bureaux, on social media websites and even in the criminal justice system, where sentencing and parole decisions appear to be biased against African Americans.³⁹ These issues are likely to become exacerbated as machine learning and predictive analytics become more sophisticated, particularly because with deep learning (which learns autonomously), algorithms can quickly reach a point where humans can often no longer explain or understand them.

Moreover, it is very difficult to challenge a computer’s decisions, because whoever owns the algorithms owns the trade secrets associated with them, and is neither going to reveal the source code, nor likely be willing to even discuss the secret source and how it makes the algorithm functions.⁴⁰ What justifies the algorithm from an economic viability perspective is its success or perceived success, which is an entirely different question from whether or not it operates in biased ways.⁴¹

In some instances, certain AI systems are thought to have enabled discriminatory or biased practices.⁴² Racial issues also arise in facial recognition software. Most such systems operate by comparing a person’s face to a range of faces in a database. As pointed out by Joy Buolamwini, a researcher at the MIT Media Lab:⁴³

Unless the databases have access to diverse data, these programs perform poorly when attempting to recognise African-American or Asian-American features. Many historical datasets reflect traditional values, which may or may not represent the desired preferences in a current system. Algorithms embed ethical considerations and value choices into program decisions. As such, these systems raise questions concerning the criteria used in automated decision-making.

As algorithms have become an established part of high-stakes projects, concerns have arisen that they are not adequately transparent to allow for accountability, especially if they are used as the basis for harmful or coercive decisions.⁴⁴ There is growing consensus among computer scientists that it would take aggressive research to cut through algorithmic opacity, particularly in machine learning, where opacity is at its densest.⁴⁵

One of the major problems is that classic values of administrative procedure, such as due process, are not easily coded into software language. In the United States, many automated implementations of social welfare programs, ranging from State emergency assistance to the Affordable Care Act (“Obamacare”) exchanges, have resulted in erroneous denials of benefits, lengthy delays and troubling outcomes.⁴⁶

Depending on how AI systems are set up, they can assist people to discriminate against individuals they do not like, or help to screen or to build lists of individuals based on unfair criteria. The types of considerations that go into programming decisions matter a lot in terms of how the systems operate and how they affect customers. For these reasons, the European Union implemented the General Data Protection Regulation (GDPR) in May 2019.⁴⁷ The GDPR is designed to ensure the protection of personal data and provide individuals with information on how data is collected and operates. All organisations within the European Union, and organisations that collect data related to EU citizens, must be GDPR-compliant.

Machine learning is the ability of a computer to modify its programming to account for new data and modify its operations accordingly. It uses computers to run predictive models that learn from existing data to forecast future behaviours, outcomes and trends.⁴⁸ Machine learning, therefore, is dependent on data. The more data it can access, the better it can learn. However, the quality of the data, the way the data is inputted into the system, and how the system is “trained” to analyse the data can all have dire effects on the validity, accuracy and usefulness of the information generated by the algorithm.

In short, not only can an otherwise perfect algorithm fail to accomplish its set goals, but it may also prove affirmatively harmful.⁴⁹ For example, the algorithm employed by Google to answer user questions erroneously declared that Barack Obama, a Christian, was a Muslim.⁵⁰ The algorithm simply did what it was “trained” to do — it gathered information from the internet, “feeding” on websites that propagated false information. Its data pool was polluted, and the algorithm could not discern between “good” and “bad” data. This was also brought to light, for example, by the Microsoft chatbot, “Tay”, which learned to interact with humans via Twitter. Within 24 hours, the chatbot became racist, because internet trolls had bombarded it with mostly offensive and erroneous data in the form of inflammatory tweets, from which the chatbot had “learned”.⁵¹

Even if the data were accurate, the person “training” the AI could infuse their own biases into the system. This may have been a factor in the crime predicting software that has led to the arrest of an unjustifiably high number of African-Americans and other minorities in the United States⁵² as well as sentencing tools that predict higher rates of recidivism for people with these racial profiles.⁵³

Accordingly, the effective accuracy of an algorithm is dependent on both the programming and the data. This dictates a further, legally troubling conclusion. If there are doubts about the results of an algorithm, one can at least theoretically inspect and analyse the programming that constitutes the algorithm. However, given the sheer volume of data available on the internet, it may be impossible to adequately determine and inspect the data used by the algorithm.⁵⁴

Bias and discrimination are serious issues facing AI. There have been a number of cases of unfair treatment linked to historic data, and steps need to be undertaken to make sure this does not become prevalent in AI. Existing statutes governing discrimination in the physical economy need to be extended to digital platforms. This will help protect consumers and build confidence in these systems as a whole.⁵⁵ More transparency is needed in how AI systems operate.⁵⁶

The question arises whether we have reached the point at which we need to devise a legislative instrument on robotics and AI.⁵⁷ The classic line of thinking is that legislation becomes necessary once a societal or technological change calls for an adequate legal framework. Once every home and business is equipped with an autonomous robot, society will change dramatically. People will work, collaborate, interact, live, and perhaps even fall in love with, highly sophisticated machines. We will need to consider humanity’s place in the face of these technologies.⁵⁸

There are, broadly speaking, two schools of thought on the issue of the regulation of AI. The first is based on the premise that regulation is bad for innovation. Entrepreneurs in this camp do not want the field of AI to be defined too soon, and certainly not by non-technical people. Among their concerns are that bad policy creates bad technology, regulation stifles innovation, and regulation is premature because we do not yet have any clear sense of what we would be regulating.⁵⁹

The other school of thought seeks to protect against potentially harmful creations that “poison the well” for other AI entrepreneurs. Subscribers to this school believe that national governments should act expeditiously to promote existing standards and guidelines or, where necessary, create new guidelines, to ensure a basic respect for the principle of “first, do no harm”.⁶⁰

Innovations such as the internet and networked AI have enormous short-term benefits, along with long-term negative effects that could take decades to become recognisable. AI will drive a vast range of efficiency optimisations, but also enable hidden discrimination and arbitrary penalisation of individuals in areas such as insurance, job seeking and performance assessment.⁶¹ Without significant changes in our political economy and governance regimes, AI is likely to create greater economic inequalities, more surveillance and more programmed and non-human-centric interactions.⁶² As to liberty, there are clear risks. AI affects agency by creating entities with meaningful intellectual capabilities for monitoring, enforcing and even punishing individuals. Those who know how to use it will have immense potential power over those who do not or cannot.⁶³

Rapid innovation in technology far exceeds the ability of the world’s domestic and international legal systems to keep pace.⁶⁴ The key for humanity in general and lawyers specifically, will be to develop the positive aspects of the technology, while managing its risks and challenges.⁶⁵ AI regulation will be a necessity, particularly in the areas of safety and errors, liability laws and social impact. Policy-makers will have to embrace the benefits that AI can bring, but at the same time be sensitive to pre-empt the dramatic and potentially devastating effects of misusing AI.⁶⁶

Countries should develop a data strategy that promotes innovation and consumer protection. Currently, there are no uniform standards in terms of data access, data sharing or data protection.⁶⁷ Almost all the data is proprietary in nature and not shared very broadly with the research community, and this limits innovation and system design. AI requires data to test and improve its learning capacity. Without structured and instructed data sets, it will be nearly impossible to gain the full benefits of AI.⁶⁸

AI may well be a revolution in human affairs and become the single most influential innovation in history.⁶⁹ How AI systems unfold has major implications for society as a whole. It matters how policy issues are addressed, ethical conflicts reconciled, legal realities resolved, and how much transparency is required in AI and data analytic solutions.⁷⁰

Human choices about software development affect the way in which decisions are made and the manner in which they are integrated into organisational routines. Exactly how these processes are executed needs to be better understood, because they will have substantial impact on the general public soon and for the foreseeable future.⁷¹