NEWS

Selected Mentions of Paper Prisons in the News

How Automation Bias Encourages the Use of Flawed Algorithms

BY CHLOE HADAVAS

From 2013 to June 2017, the U.S. Immigration and Customs Enforcement’s New York Field Office determined that about 47 percent of detainees designated as “low risk” should be released while they waited for their immigration cases to be resolved, according to FOIA data obtained by the New York Civil Liberties Union. But something changed in the middle of 2017. From June 2017 to September 2019, that figure fell to 3 percent: Virtually all detainees, the data shows, had to wait weeks or even months in custody before their first hearing, even if they posed little flight risk.

All that time, ICE used the same software to determine a detainee’s fate: the Risk Classification Assessment tool, which is supposed to consider an individual’s history—including criminal history, family ties, and time in the country—to recommend whether that person should be detained or released after 48 hours of arrest. When ICE introduced the algorithm in 2013, the Intercept reported Monday, it gave four options: detention without bond, detention with the possibility of release on bond, release, or referral to an ICE supervisor. In 2015, the algorithm was edited to remove the option for bond. Then, it was changed again after the 2016 election to remove the release output. According to the NYCLU and Bronx Defenders, the possibility of bond or release has been “all but eliminated.” (ICE personnel can still technically override the recommendations of the tool, which may explain why that 3 percent of low-risk detainees were still released.)

On Feb. 28, the NYCLU and Bronx Defenders filed a lawsuit alleging that as a result of this change, ICE has illegally detained virtually all the thousands of people its New York Field Office has arrested over the past three years. ICE, the NYCLU argues, has a legal obligation to consider whether release is appropriate within 48 hours of arrest.

The lawsuit aims to restore due process and end what it calls “ICE’s manipulation of the legal process,” by requiring ICE to make individual assessments and abandon its “hijacked algorithm.” “If the New York Field Office were actually conducting individualized determinations pursuant to its stated criteria,” the lawsuit says, according to the Intercept, “the percentage of people released should have actually increased since 2017 because more people arrested qualified for release.”

ICE’s reliance on a risk assessment tool is not unusual, even as it becomes increasingly clear that algorithmic biases tend to affect marginalized communities and people of color. Risk-assessment algorithms in particular have been in use for decades and, more recently, have become an integral part of the criminal justice system, from policing to evidence to sentencing. Algorithms have started to replace bail hearings to help determine who goes to jail, for instance, and police departments use them to predict future unlawful activity, which civil liberties groups say leads to heavier policing of communities of color. In large part, we still don’t know how these algorithms work—many of them are kept secret from the public, often in the name of protecting intellectual property.

One reason for the dependence on algorithms is “automation bias,” in which humans attribute more weight than is sometimes deserved to computer decisions, says Colleen Chien, a professor at Santa Clara University School of Law who researches innovation and the criminal justice system. “There’s been a lot of criticism of risk assessment tools, like this one, particularly in the bail and pre-trial contexts,” Chien said. “But the reality is that human beings like to get information from what they think is an objective source.”

This “veneer of objectivity and certainty,” as Chien puts it, is particularly attractive to government agencies. “The reality is that administrators in governments have to make hard decisions every day,” said Chien, who served in the Obama White House as a senior adviser on intellectual property and innovation. “And they are going to use tools that help them do it more efficiently, do it more consistently, and more accurately.”

Sometimes, as in the case of ICE’s Risk Classification Assessment, these tools can start to feel like an “algorithmic rubber stamp,” Chien said. But algorithms tend to reflect the systems and agencies that use and create them. In this context, the data behind ICE’s tool is perhaps unsurprising, since the agency has escalated its terror tactics and “become an arm of Donald Trump’s nativist agenda” since 2017.

Chien stressed, however, that algorithms also hold potential for making systems more efficient and for upholding the presumption of innocence. For instance, “[t]he criminal justice system is notoriously biased in terms of its history,” Chien said. So “when you look at what’s the impact of the algorithm, you need to take a baseline, and then you need to measure how are we changing from [that] baseline,” she said. Chien advocates for a clean slate policy, which would use algorithms to seal or clear Californians’ criminal records. She also pointed out the potential of facial recognition technologies, which are often criticized, for exonerating suspects.

“I think there’s a lot of ways in which algorithms can be very beneficial in criminal justice,” said Chien. “But I think the reality is, again, that they’re being used whether or not the public thinks they’re beneficial.” (A recent report by the Administrative Conference of the U.S. details how pervasive AI tools are across federal agencies.) “The stories you read about are just the tip of the iceberg,” said Chien.

Given the ubiquity of these algorithms—and the secrecy in which they’re currently allowed to operate—it only becomes more necessary to hold government agencies accountable for their algorithmic practices. The ICE case is just the latest, and most public, example of how algorithms can be weaponized, under the guise of impartial justice, to a certain end.

Future Tense is a partnership of SlateNew America, and Arizona State University that examines emerging technologies, public policy, and society.

Read original article.

107 comments

  1. Reading your article helped me a lot and I agree with you. But I still have some doubts, can you clarify for me? I’ll keep an eye out for your answers.

  2. Hi! Do you use Twitter? I’d like to follow you if that would be okay.I’m undoubtedly enjoying your blog and look forward to new posts.I love this!! I am a 16-year-old female I think this article great job!!

  3. Do you mind if I quote a couple of your articles as long asI provide credit and sources back to your website?My blog site is in the very same niche as yours and my users would certainly benefit from some of the information you present here.Please let me know if this okay with you. Thanks!

  4. Very nice post. I just stumbled upon your blog and wanted to say that I’ve really enjoyed browsing your blog posts. In any case I’ll be subscribing to your feed and I hope you write again soon!

  5. Very nice post. I just stumbled upon your blog and wanted to say that I’ve really enjoyed browsing your blog posts. In any case I’ll be subscribing to your feed and I hope you write again soon!

  6. I agree with your point of view, your article has given me a lot of help and benefited me a lot. Thanks. Hope you continue to write such excellent articles.

  7. Very nice post. I just stumbled upon your blog and wanted to say that I’ve really enjoyed browsing your blog posts. In any case I’ll be subscribing to your feed and I hope you write again soon!

  8. Very nice post. I just stumbled upon your blog and wanted to say that I’ve really enjoyed browsing your blog posts. In any case I’ll be subscribing to your feed and I hope you write again soon!

  9. I have read your article carefully and I agree with you very much. This has provided a great help for my thesis writing, and I will seriously improve it. However, I don’t know much about a certain place. Can you help me?

  10. I have read your article carefully and I agree with you very much. This has provided a great help for my thesis writing, and I will seriously improve it. However, I don’t know much about a certain place. Can you help me?

  11. I have read your article carefully and I agree with you very much. This has provided a great help for my thesis writing, and I will seriously improve it. However, I don’t know much about a certain place. Can you help me?

  12. I have read your article carefully and I agree with you very much. This has provided a great help for my thesis writing, and I will seriously improve it. However, I don’t know much about a certain place. Can you help me?

  13. I have read your article carefully and I agree with you very much. This has provided a great help for my thesis writing, and I will seriously improve it. However, I don’t know much about a certain place. Can you help me?

  14. I have read your article carefully and I agree with you very much. This has provided a great help for my thesis writing, and I will seriously improve it. However, I don’t know much about a certain place. Can you help me?

  15. I have read your article carefully and I agree with you very much. This has provided a great help for my thesis writing, and I will seriously improve it. However, I don’t know much about a certain place. Can you help me?

  16. I have read your article carefully and I agree with you very much. This has provided a great help for my thesis writing, and I will seriously improve it. However, I don’t know much about a certain place. Can you help me?

  17. I have read your article carefully and I agree with you very much. This has provided a great help for my thesis writing, and I will seriously improve it. However, I don’t know much about a certain place. Can you help me?

  18. I have read your article carefully and I agree with you very much. This has provided a great help for my thesis writing, and I will seriously improve it. However, I don’t know much about a certain place. Can you help me?

  19. Cool. I spent a long time looking for relevant content and found that your article gave me new ideas, which is very helpful for my research. I think my thesis can be completed more smoothly. Thank you.

  20. Cool. I spent a long time looking for relevant content and found that your article gave me new ideas, which is very helpful for my research. I think my thesis can be completed more smoothly. Thank you.

  21. Very nice post. I just stumbled upon your blog and wanted to say that I’ve really enjoyed browsing your blog posts. In any case I’ll be subscribing to your feed and I hope you write again soon!

  22. Thank you for your sharing. I am worried that I lack creative ideas. It is your article that makes me full of hope. Thank you. But, I have a question, can you help me?

  23. I don’t think the title of your article matches the content lol. Just kidding, mainly because I had some doubts after reading the article.

  24. I don’t think the title of your article matches the content lol. Just kidding, mainly because I had some doubts after reading the article.

  25. I may need your help. I’ve been doing research on gate io recently, and I’ve tried a lot of different things. Later, I read your article, and I think your way of writing has given me some innovative ideas, thank you very much.

Leave a comment

Your email address will not be published. Required fields are marked *

Partners

The Paper Prisons Initiative is a project of Santa Clara University that is made possible through the support of our collaborators and partners

Contact Us

If you wish to contact us, please fill out the entire form below and press the "Send Message" button. If you wish to sign up for email updates from us, please only fill out your name and email in the form below and press the "Sign Up for Updates" button.

info@paperprisons.org

A project of
Santa Clara University

© 2021 Santa Clara University

Privacy Policy | Terms and Conditions | Cookie Policy