NEWS

Selected Mentions of Paper Prisons in the News

How Automation Bias Encourages the Use of Flawed Algorithms

BY CHLOE HADAVAS

From 2013 to June 2017, the U.S. Immigration and Customs Enforcement’s New York Field Office determined that about 47 percent of detainees designated as “low risk” should be released while they waited for their immigration cases to be resolved, according to FOIA data obtained by the New York Civil Liberties Union. But something changed in the middle of 2017. From June 2017 to September 2019, that figure fell to 3 percent: Virtually all detainees, the data shows, had to wait weeks or even months in custody before their first hearing, even if they posed little flight risk.

All that time, ICE used the same software to determine a detainee’s fate: the Risk Classification Assessment tool, which is supposed to consider an individual’s history—including criminal history, family ties, and time in the country—to recommend whether that person should be detained or released after 48 hours of arrest. When ICE introduced the algorithm in 2013, the Intercept reported Monday, it gave four options: detention without bond, detention with the possibility of release on bond, release, or referral to an ICE supervisor. In 2015, the algorithm was edited to remove the option for bond. Then, it was changed again after the 2016 election to remove the release output. According to the NYCLU and Bronx Defenders, the possibility of bond or release has been “all but eliminated.” (ICE personnel can still technically override the recommendations of the tool, which may explain why that 3 percent of low-risk detainees were still released.)

On Feb. 28, the NYCLU and Bronx Defenders filed a lawsuit alleging that as a result of this change, ICE has illegally detained virtually all the thousands of people its New York Field Office has arrested over the past three years. ICE, the NYCLU argues, has a legal obligation to consider whether release is appropriate within 48 hours of arrest.

The lawsuit aims to restore due process and end what it calls “ICE’s manipulation of the legal process,” by requiring ICE to make individual assessments and abandon its “hijacked algorithm.” “If the New York Field Office were actually conducting individualized determinations pursuant to its stated criteria,” the lawsuit says, according to the Intercept, “the percentage of people released should have actually increased since 2017 because more people arrested qualified for release.”

ICE’s reliance on a risk assessment tool is not unusual, even as it becomes increasingly clear that algorithmic biases tend to affect marginalized communities and people of color. Risk-assessment algorithms in particular have been in use for decades and, more recently, have become an integral part of the criminal justice system, from policing to evidence to sentencing. Algorithms have started to replace bail hearings to help determine who goes to jail, for instance, and police departments use them to predict future unlawful activity, which civil liberties groups say leads to heavier policing of communities of color. In large part, we still don’t know how these algorithms work—many of them are kept secret from the public, often in the name of protecting intellectual property.

One reason for the dependence on algorithms is “automation bias,” in which humans attribute more weight than is sometimes deserved to computer decisions, says Colleen Chien, a professor at Santa Clara University School of Law who researches innovation and the criminal justice system. “There’s been a lot of criticism of risk assessment tools, like this one, particularly in the bail and pre-trial contexts,” Chien said. “But the reality is that human beings like to get information from what they think is an objective source.”

This “veneer of objectivity and certainty,” as Chien puts it, is particularly attractive to government agencies. “The reality is that administrators in governments have to make hard decisions every day,” said Chien, who served in the Obama White House as a senior adviser on intellectual property and innovation. “And they are going to use tools that help them do it more efficiently, do it more consistently, and more accurately.”

Sometimes, as in the case of ICE’s Risk Classification Assessment, these tools can start to feel like an “algorithmic rubber stamp,” Chien said. But algorithms tend to reflect the systems and agencies that use and create them. In this context, the data behind ICE’s tool is perhaps unsurprising, since the agency has escalated its terror tactics and “become an arm of Donald Trump’s nativist agenda” since 2017.

Chien stressed, however, that algorithms also hold potential for making systems more efficient and for upholding the presumption of innocence. For instance, “[t]he criminal justice system is notoriously biased in terms of its history,” Chien said. So “when you look at what’s the impact of the algorithm, you need to take a baseline, and then you need to measure how are we changing from [that] baseline,” she said. Chien advocates for a clean slate policy, which would use algorithms to seal or clear Californians’ criminal records. She also pointed out the potential of facial recognition technologies, which are often criticized, for exonerating suspects.

“I think there’s a lot of ways in which algorithms can be very beneficial in criminal justice,” said Chien. “But I think the reality is, again, that they’re being used whether or not the public thinks they’re beneficial.” (A recent report by the Administrative Conference of the U.S. details how pervasive AI tools are across federal agencies.) “The stories you read about are just the tip of the iceberg,” said Chien.

Given the ubiquity of these algorithms—and the secrecy in which they’re currently allowed to operate—it only becomes more necessary to hold government agencies accountable for their algorithmic practices. The ICE case is just the latest, and most public, example of how algorithms can be weaponized, under the guise of impartial justice, to a certain end.

Future Tense is a partnership of SlateNew America, and Arizona State University that examines emerging technologies, public policy, and society.

Read original article.

337 comments

  1. Reading your article helped me a lot and I agree with you. But I still have some doubts, can you clarify for me? I’ll keep an eye out for your answers.

  2. Hi! Do you use Twitter? I’d like to follow you if that would be okay.I’m undoubtedly enjoying your blog and look forward to new posts.I love this!! I am a 16-year-old female I think this article great job!!

  3. Do you mind if I quote a couple of your articles as long asI provide credit and sources back to your website?My blog site is in the very same niche as yours and my users would certainly benefit from some of the information you present here.Please let me know if this okay with you. Thanks!

  4. Very nice post. I just stumbled upon your blog and wanted to say that I’ve really enjoyed browsing your blog posts. In any case I’ll be subscribing to your feed and I hope you write again soon!

  5. Very nice post. I just stumbled upon your blog and wanted to say that I’ve really enjoyed browsing your blog posts. In any case I’ll be subscribing to your feed and I hope you write again soon!

  6. I agree with your point of view, your article has given me a lot of help and benefited me a lot. Thanks. Hope you continue to write such excellent articles.

  7. Very nice post. I just stumbled upon your blog and wanted to say that I’ve really enjoyed browsing your blog posts. In any case I’ll be subscribing to your feed and I hope you write again soon!

  8. Very nice post. I just stumbled upon your blog and wanted to say that I’ve really enjoyed browsing your blog posts. In any case I’ll be subscribing to your feed and I hope you write again soon!

  9. I have read your article carefully and I agree with you very much. This has provided a great help for my thesis writing, and I will seriously improve it. However, I don’t know much about a certain place. Can you help me?

  10. I have read your article carefully and I agree with you very much. This has provided a great help for my thesis writing, and I will seriously improve it. However, I don’t know much about a certain place. Can you help me?

  11. I have read your article carefully and I agree with you very much. This has provided a great help for my thesis writing, and I will seriously improve it. However, I don’t know much about a certain place. Can you help me?

  12. I have read your article carefully and I agree with you very much. This has provided a great help for my thesis writing, and I will seriously improve it. However, I don’t know much about a certain place. Can you help me?

  13. I have read your article carefully and I agree with you very much. This has provided a great help for my thesis writing, and I will seriously improve it. However, I don’t know much about a certain place. Can you help me?

  14. I have read your article carefully and I agree with you very much. This has provided a great help for my thesis writing, and I will seriously improve it. However, I don’t know much about a certain place. Can you help me?

  15. I have read your article carefully and I agree with you very much. This has provided a great help for my thesis writing, and I will seriously improve it. However, I don’t know much about a certain place. Can you help me?

  16. I have read your article carefully and I agree with you very much. This has provided a great help for my thesis writing, and I will seriously improve it. However, I don’t know much about a certain place. Can you help me?

  17. I have read your article carefully and I agree with you very much. This has provided a great help for my thesis writing, and I will seriously improve it. However, I don’t know much about a certain place. Can you help me?

  18. I have read your article carefully and I agree with you very much. This has provided a great help for my thesis writing, and I will seriously improve it. However, I don’t know much about a certain place. Can you help me?

  19. Cool. I spent a long time looking for relevant content and found that your article gave me new ideas, which is very helpful for my research. I think my thesis can be completed more smoothly. Thank you.

  20. Cool. I spent a long time looking for relevant content and found that your article gave me new ideas, which is very helpful for my research. I think my thesis can be completed more smoothly. Thank you.

  21. Very nice post. I just stumbled upon your blog and wanted to say that I’ve really enjoyed browsing your blog posts. In any case I’ll be subscribing to your feed and I hope you write again soon!

  22. Thank you for your sharing. I am worried that I lack creative ideas. It is your article that makes me full of hope. Thank you. But, I have a question, can you help me?

  23. I don’t think the title of your article matches the content lol. Just kidding, mainly because I had some doubts after reading the article.

  24. I don’t think the title of your article matches the content lol. Just kidding, mainly because I had some doubts after reading the article.

  25. I may need your help. I’ve been doing research on gate io recently, and I’ve tried a lot of different things. Later, I read your article, and I think your way of writing has given me some innovative ideas, thank you very much.

  26. I may need your help. I tried many ways but couldn’t solve it, but after reading your article, I think you have a way to help me. I’m looking forward for your reply. Thanks.

  27. I may need your help. I tried many ways but couldn’t solve it, but after reading your article, I think you have a way to help me. I’m looking forward for your reply. Thanks.

  28. I may need your help. I tried many ways but couldn’t solve it, but after reading your article, I think you have a way to help me. I’m looking forward for your reply. Thanks.

  29. I may need your help. I tried many ways but couldn’t solve it, but after reading your article, I think you have a way to help me. I’m looking forward for your reply. Thanks.

  30. I may need your help. I tried many ways but couldn’t solve it, but after reading your article, I think you have a way to help me. I’m looking forward for your reply. Thanks.

  31. This article opened my eyes, I can feel your mood, your thoughts, it seems very wonderful. I hope to see more articles like this. thanks for sharing.

  32. I have read your article carefully and I agree with you very much. This has provided a great help for my thesis writing, and I will seriously improve it. However, I don’t know much about a certain place. Can you help me?

  33. Very nice post. I just stumbled upon your blog and wanted to say that I’ve really enjoyed browsing your blog posts. In any case I’ll be subscribing to your feed and I hope you write again soon!

  34. This article opened my eyes, I can feel your mood, your thoughts, it seems very wonderful. I hope to see more articles like this. thanks for sharing.

  35. Reading your article helped me a lot and I agree with you. But I still have some doubts, can you clarify for me? I’ll keep an eye out for your answers.

  36. Reading your article has greatly helped me, and I agree with you. But I still have some questions. Can you help me? I will pay attention to your answer. thank you.

  37. Very nice post. I just stumbled upon your blog and wanted to say that I’ve really enjoyed browsing your blog posts. In any case I’ll be subscribing to your feed and I hope you write again soon!

  38. Our team of experienced professionals Alanya Escort understands the unique challenges faced by bloggers and knows how to navigate the ever-changing digital landscape. We take the time to understand your blog’s niche, target audience, and goals, allowing us to develop customized strategies that resonate with your readers and drive engagement.

  39. Please let me know if you’re looking for a author for your weblog. You have some really great posts and I believe I would be a good asset. If you ever want to take some of the load off, I’d really like to write some material for your blog in exchange for a link back to mine. Please send me an email if interested. Many thanks!

  40. I am currently writing a paper and a bug appeared in the paper. I found what I wanted from your article. Thank you very much. Your article gave me a lot of inspiration. But hope you can explain your point in more detail because I have some questions, thank you. 20bet

  41. I am currently writing a paper and a bug appeared in the paper. I found what I wanted from your article. Thank you very much. Your article gave me a lot of inspiration. But hope you can explain your point in more detail because I have some questions, thank you. 20bet

  42. I am currently writing a paper and a bug appeared in the paper. I found what I wanted from your article. Thank you very much. Your article gave me a lot of inspiration. But hope you can explain your point in more detail because I have some questions, thank you. 20bet

  43. I am currently writing a paper and a bug appeared in the paper. I found what I wanted from your article. Thank you very much. Your article gave me a lot of inspiration. But hope you can explain your point in more detail because I have some questions, thank you. 20bet

  44. Apoiar ferramentas de apostas e estar equipado com uma plataforma diversificada de transações financeiras, a 20Bet oferece suporte tangível aos jogadores. Este é um lugar onde eles podem apostar com dinheiro real, respaldados por concorrentes de diversas disciplinas esportivas. 20bet

  45. Apoiar ferramentas de apostas e estar equipado com uma plataforma diversificada de transações financeiras, a 20Bet oferece suporte tangível aos jogadores. Este é um lugar onde eles podem apostar com dinheiro real, respaldados por concorrentes de diversas disciplinas esportivas. 20bet

  46. I am currently writing a paper that is very related to your content. I read your article and I have some questions. I would like to ask you. Can you answer me? I’ll keep an eye out for your reply. 20bet

  47. I am currently writing a paper that is very related to your content. I read your article and I have some questions. I would like to ask you. Can you answer me? I’ll keep an eye out for your reply. 20bet

  48. I am currently writing a paper that is very related to your content. I read your article and I have some questions. I would like to ask you. Can you answer me? I’ll keep an eye out for your reply. 20bet

  49. Thank you, I’ve recently been looking for info approximately this subject for a while and yours is the best I’ve found out till now. But, what in regards to the conclusion? Are you sure about the source?

  50. I like what you guys are up too. Such intelligent work and reporting! Carry on the excellent works guys I have incorporated you guys to my blogroll. I think it will improve the value of my website 🙂

  51. hello!,I really like your writing very so much! proportion we communicate
    extra approximately your post on AOL? I need an expert on this area to solve my problem.
    Maybe that is you! Looking forward to peer you.

  52. Attractive section of content. I just stumbled upon your website and in accession capital to assert that I acquire in fact enjoyed account your blog posts. Any way I will be subscribing to your feeds and even I achievement you access consistently fast.

  53. I’m amazed, I have to admit. Rarely do I encounter a blog that’s both equally educative and entertaining, and let me tell you, you’ve hit the nail on the head.
    The issue is something not enough people are speaking intelligently about.
    Now i’m very happy I found this during my search for something concerning this.

  54. Do you mind if I quote a couple of your posts as long as I provide credit and sources back to your website? My website is in the very same niche as yours and my visitors would certainly benefit from some of the information you provide here. Please let me know if this alright with you. Thanks a lot!

  55. Greetings! I found your site while looking into ’emergency pregnancy scan services.’ The content is both reassuring and informative. In need of an ’emergency pregnancy scan,’ I’m interested in any advice or insights you may have. Thank you in advance!

  56. Hello! Your site came up in my search for ‘baby ultrasound services near me.’ Your content is both informative and reassuring. As I’m looking for a reliable ‘baby scan near me,’ any insights or recommendations you could provide would be greatly valued. Thanks in advance!

  57. Greetings! I found your site while researching ‘early pregnancy scan services near me.’ The wealth of information is impressive. Currently on the lookout for an ‘early scan near me,’ I’d love to hear your expert opinion or any suggestions you may have. Thanks a bunch!

  58. Hello! Your site caught my eye as I was researching ‘gender scan services near me.’ The information you provide is incredibly helpful. Currently in search of a reliable ‘gender scan near me,’ I would greatly appreciate any insights or recommendations you could share. Thanks in advance!

Leave a comment

Your email address will not be published. Required fields are marked *

Partners

The Paper Prisons Initiative is a project of Santa Clara University that is made possible through the support of our collaborators and partners

Contact Us

If you wish to contact us, please fill out the entire form below and press the "Send Message" button. If you wish to sign up for email updates from us, please only fill out your name and email in the form below and press the "Sign Up for Updates" button.

info@paperprisons.org

A project of
Santa Clara University

© 2021 Santa Clara University

Privacy Policy | Terms and Conditions | Cookie Policy