{"id":495,"date":"2020-03-06T18:48:00","date_gmt":"2020-03-06T18:48:00","guid":{"rendered":"https:\/\/paperprisons.org\/news\/?p=495"},"modified":"2024-10-10T18:39:04","modified_gmt":"2024-10-10T18:39:04","slug":"how-automation-bias-encourages-the-use-of-flawed-algorithms","status":"publish","type":"post","link":"https:\/\/paperprisons.org\/news\/all\/news\/how-automation-bias-encourages-the-use-of-flawed-algorithms\/","title":{"rendered":"How Automation Bias Encourages the Use of Flawed Algorithms (Mar. 6, 2020)"},"content":{"rendered":"\n

From 2013 to June 2017, the U.S. Immigration and Customs Enforcement\u2019s New York Field Office determined that about 47 percent of detainees designated as \u201clow risk\u201d should be released while they waited for their immigration cases to be resolved, according to FOIA data obtained by the New York Civil Liberties Union<\/a>. But something changed in the middle of 2017. From June 2017 to September 2019, that figure fell to 3 percent: Virtually all detainees, the data shows, had to wait weeks or even months in custody before their first hearing, even if they posed little flight risk.<\/p>\n\n\n\n

All that time, ICE used the same software to determine a detainee\u2019s fate: the Risk Classification Assessment tool, which is supposed to consider an individual\u2019s history\u2014including criminal history, family ties, and time in the country\u2014to recommend whether that person should be detained or released after 48 hours of arrest. When ICE introduced the algorithm in 2013, the Intercept reported<\/a> Monday, it gave four options: detention without bond, detention with the possibility of release on bond, release, or referral to an ICE supervisor. In 2015, the algorithm was edited to remove the option for bond. Then, it was changed<\/a> again after the 2016 election to remove the release output. According to the NYCLU and Bronx Defenders<\/a>, the possibility of bond or release has been \u201call but eliminated.\u201d (ICE personnel can still technically override<\/a> the recommendations of the tool, which may explain why that 3 percent of low-risk detainees were still released.)ADVERTISEMENT<\/p>\n\n\n\n

On Feb. 28, the NYCLU and Bronx Defenders filed a lawsuit<\/a> alleging that as a result of this change, ICE has illegally detained virtually all the thousands of people its New York Field Office has arrested over the past three years. ICE, the NYCLU argues<\/a>, has a legal obligation to consider whether release is appropriate within 48 hours of arrest.<\/p>\n\n\n\n

The lawsuit aims to restore due process and end what it calls<\/a> \u201cICE\u2019s manipulation of the legal process,\u201d by requiring ICE to make individual assessments and abandon its \u201chijacked algorithm.\u201d \u201cIf the New York Field Office were actually conducting individualized determinations pursuant to its stated criteria,\u201d the lawsuit says, according to the Intercept, \u201cthe percentage of people released should have actually increased since 2017 because more people arrested qualified for release.\u201d<\/p>\n\n\n\n

ICE\u2019s reliance on a risk assessment tool is not unusual, even as it becomes increasingly clear that algorithmic biases<\/a> tend to affect marginalized communities and people of color<\/a>. Risk-assessment algorithms in particular have been in use for decades and, more recently, have become an integral part of the criminal justice system, from policing to evidence to sentencing. Algorithms have started to replace bail hearings<\/a> to help determine who goes to jail, for instance, and police departments use them to predict future unlawful activity<\/a>, which civil liberties groups say leads to heavier policing of communities of color. In large part, we still don\u2019t know how these algorithms work\u2014many of them are kept secret from the public<\/a>, often in the name of protecting intellectual property<\/a>.<\/p>\n\n\n\n

One reason for the dependence on algorithms is \u201cautomation bias,\u201d in which humans attribute more weight than is sometimes deserved to computer decisions, says Colleen Chien<\/a>, a professor at Santa Clara University School of Law who researches innovation and the criminal justice system. \u201cThere\u2019s been a lot of criticism of risk assessment tools, like this one, particularly in the bail and pre-trial contexts,\u201d Chien said. \u201cBut the reality is that human beings like to get information from what they think is an objective source.\u201dADVERTISEMENT<\/p>\n\n\n\n

This \u201cveneer of objectivity and certainty,\u201d as Chien puts it, is particularly attractive to government agencies. \u201cThe reality is that administrators in governments have to make hard decisions every day,\u201d said Chien, who served in the Obama White House as a senior adviser on intellectual property and innovation. \u201cAnd they are going to use tools that help them do it more efficiently, do it more consistently, and more accurately.\u201d<\/p>\n\n\n\n

Sometimes, as in the case of ICE\u2019s Risk Classification Assessment, these tools can start to feel like an \u201calgorithmic rubber stamp,\u201d Chien said. But algorithms tend to reflect the systems and agencies that use and create them. In this context, the data behind ICE\u2019s tool is perhaps unsurprising, since the agency has escalated its terror tactics and \u201cbecome an arm of Donald Trump\u2019s nativist agenda<\/a>\u201d since 2017.<\/p>\n\n\n\n

Chien stressed, however, that algorithms also hold potential for making systems more efficient and for upholding the presumption of innocence. For instance, \u201c[t]he criminal justice system is notoriously biased in terms of its history,\u201d Chien said. So \u201cwhen you look at what\u2019s the impact of the algorithm, you need to take a baseline, and then you need to measure how are we changing from [that] baseline,\u201d she said. Chien advocates for a clean slate policy<\/a>, which would use algorithms to seal or clear Californians\u2019 criminal records. She also pointed out the potential of facial recognition technologies, which are often criticized<\/a>, for exonerating suspects.<\/p>\n\n\n\n

\u201cI think there\u2019s a lot of ways in which algorithms can be very beneficial in criminal justice,\u201d said Chien. \u201cBut I think the reality is, again, that they\u2019re being used whether or not the public thinks they\u2019re beneficial.\u201d (A recent report by the Administrative Conference of the U.S.<\/a> details how pervasive AI tools are across federal agencies.) \u201cThe stories you read about are just the tip of the iceberg,\u201d said Chien.<\/p>\n\n\n\n

Given the ubiquity of these algorithms\u2014and the secrecy in which they\u2019re currently allowed to operate\u2014it only becomes more necessary to hold government agencies accountable for their algorithmic practices. The ICE case is just the latest, and most public, example of how algorithms can be weaponized, under the guise of impartial justice, to a certain end.<\/p>\n\n\n\n

Future Tense<\/a> is a partnership of Slate<\/a>, New America<\/a>, and Arizona State University<\/a> that examines emerging technologies, public policy, and society<\/p>\n\n\n\n

Read Original Article<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

From 2013 to June 2017, the U.S. Immigration and Customs Enforcement\u2019s New York Field Office determined that about 47 percent of detainees designated as \u201clow risk\u201d should be released while they waited for their immigration cases to be resolved, according to FOIA data obtained by the New York Civil Liberties Union. But something changed in the […]<\/p>\n","protected":false},"author":11,"featured_media":500,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[4],"tags":[],"class_list":["post-495","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-news"],"_links":{"self":[{"href":"https:\/\/paperprisons.org\/news\/wp-json\/wp\/v2\/posts\/495","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/paperprisons.org\/news\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/paperprisons.org\/news\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/paperprisons.org\/news\/wp-json\/wp\/v2\/users\/11"}],"replies":[{"embeddable":true,"href":"https:\/\/paperprisons.org\/news\/wp-json\/wp\/v2\/comments?post=495"}],"version-history":[{"count":4,"href":"https:\/\/paperprisons.org\/news\/wp-json\/wp\/v2\/posts\/495\/revisions"}],"predecessor-version":[{"id":1033,"href":"https:\/\/paperprisons.org\/news\/wp-json\/wp\/v2\/posts\/495\/revisions\/1033"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/paperprisons.org\/news\/wp-json\/wp\/v2\/media\/500"}],"wp:attachment":[{"href":"https:\/\/paperprisons.org\/news\/wp-json\/wp\/v2\/media?parent=495"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/paperprisons.org\/news\/wp-json\/wp\/v2\/categories?post=495"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/paperprisons.org\/news\/wp-json\/wp\/v2\/tags?post=495"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}