Align Chronicles

Gas, Whey & Lobsters and Indian-Homework

Sat, 31 Jan 2026 06:09:06 +0000

Gas, Whey & Lobsters and Indian-Homework

Table of contents

Whey, gas and lobsters

Wrongfully castigating an invaluable resource as wasteful dross has been one of humanity’s most original, artful and cardinal sins.

Whey - Gutter-to-Gold: Till the development of membrane filtration technologies in the 1960s, protein-rich sweet whey was literally disposed down the drains by the dairy industry. Today, it is widely considered to be the go-to protein choice for most fitness fanatics with a burgeoning multi-billion market. Also, it is painful to imagine the civilizational cost we paid for not building out distribution pipelines for this nutritious by-product to be shared amongst the masses & ensure they got sustained on high-protein diets!

Source: https://www.sciencedirect.com/science/article/pii/S0924224421005124
Natural gas - Invisible fuel: Natural gas was historically considered a worthless byproduct of oil drilling and was often burned at the wellhead. Vast amounts of engineering clock cycles were abused to invent techniques such as Optimal-venting and Production-flaring with the sole goal to efficiently dispose this invaluable resource right up until the early part of the 20th century. Today, it is not only hailed as the cleanest fossil fuel but also considered to be ‘an integral part of green-energy strategies around the world as the bridge-fuel of choice!
Lobsters- Cockroach to Caviar: Lobsters were once called the “cockroaches of the sea,” a stigmatized scavenger so abundant in colonial New England that they would wash ashore in knee-deep piles after storms. Far from the luxury status they enjoy today, these crustaceans were viewed as the “protein of the bad man,” a cheap and “cruel” filler fed to prisoners and indentured servants. Today, it is a high end gourmet protein served in high-end restaurants and currently sustains a 7 Billion market growing at scorching pace.

India’s silent crime: Throwing away the homework treasure-trove

Alongside the Whey, Gas and Lobster triad, I believe lies India’s homework treasure-trove. India criminally throws away its academic data goldmine that is the reams and reams of graded homework her students produce every day!
The numbers: Assuming an average student takes 6 courses and generates roughly 100 KB worth of data via homework, assignments and tests, we have a mindboggling 998.4 GB of data being generated every academic calendar year spanning 32 weeks by a single academic group like the Jain group (that boasts of a student body of 51600 students). This data-trove, as we type this, is currently getting discarded into abyss never to be utilized in any way and the scale of this wastage is mindboggling. To put things into perspective, this data mined thus from a single academic group in a single year spans more than [The Pile](https://arxiv.org/abs/2101.00027), that is one of the largest openly available dataset (that is only 825 GB). Add in the fact that much of this has been graded by a human evaluator (examiner evaluating grades and providing feedback notes) facilitates rich algorithmic advances that the field is yet to see!

The data moat is a data iceberg!

Now comes the obvious question: Whey had its “Protein moment”. Lobsters had its “Gourmet moment”. Natural Gas had its “Bridge-fuel” moment. What would be the X in Indian homework’s X moment? I posit it would be that graded homework will have its “RL (Reinforcement Learning) moment”. In the following sections I will motivate how India’s gargantuan base of graded homework can be a game-charger for progressing state-of-the-art AI towards it prophesized “AGI” moment and how it could be orchestrated ethically and profitably.

A peek into how SOTA AI/LLMs are trained

The pipeline was sourced from: https://allenai.org/blog/olmo3

As seen in the figure above, the state-of-the-art AI/Large Language Model training pipeline is largely dichotomized into two phases: The pre-training phase (or the boring dead phase) that results in a “Base Model”. The post-training phase that results in “Thinking Models”.

The state-of-the-art pre-training data is mostly derived from Common-Crawl (4.5T tokens our 6T tokens for the Dolma3) and augmented with academic publications, code etc. This part of the pipeline is widely believed to be the boring part or even the dead part with most the WWW being already scraped dry. The post-training phase is where the real action is and the real fun begins. The base model that emerges freshly baked from the pre-training phase is just an unaligned token-predictor that is unworthy of an real economic utility and most certainly un-deployable as a a chatbot. But when it is subjected to a guided Reinforcement Learning from Human Feedback (RLHF) /. Supervised Fine-Tuning phase (See figure below taken from Andrej Karpathy’s canonical talk), it turns into chat-bot styled Thinking model that is also putatively aligned with complex “human values”.

Source: https://www.youtube.com/watch?v=bZQun8Y4L2A - State of GPT talk by Andrej Karpathy

This phase of human preference alignment is currently stricken with serious shortcomings, is widely known to be a specialized dark-art and this is also where the data-labelers’ gold-rush has been in. From the algorithmic efficiency perspective, experts such as Yann LeCun have repeatedly expressed deep suspicion on its inefficiency while once even branding it “hopeless” (See Tweet screenshot below).

https://x.com/search?q=from%3Aylecun RLHF is hopeless&src=typed_query

The main nuance is that RL generally is a beast that appears to be quack-science in low-data regimes that suddenly awakens into a formidable beast in the large-data regime whilst being aggressively data inefficient. In fact, there’s an entire cottage industry of literature on the so-termed breathtakingly inefficient scaling nature of RL-scaling.

It is precisely this inefficiency that has birthed overnight unicorns such as Mercor, Surge and Scale AI ( Semianalysis has an absolutely brilliantly blogpost covering this). And it is precisely here that India’s civilizational play lies. If we can build a bridge between India’s civilizational moat and the RL training pipelines, this will accelerate AI timelines at a rate beyond an AI-maximalist’s imaginations.

The roadblocks and the concerns

The current orchestration mechanism for collecting the human feedback is fraught with other serious ethical shortcomings such as:

Labor exploitation: An investigation by Time magazine revealed that [OpenAI Used Kenyan Workers on Less Than $2 Per Hour to Make ChatGPT Less Toxic](https://time.com/6247678/openai-chatgpt-kenya-workers/). This fits squarely in the larger pattern of abusive labor practices that was hitherto used for content moderation jobs and are now being replicated under the banner of RLHF. Besides the monetary perspective, we also have to grapple with the psychological and emotional impact on the workers and labelers being exposed to toxic textual and visual content that they are expected to provide feedback about.
Scalability and quality: It is rather self-evident that scaling computation by recruiting larger swathes of GPU-laden server farms is much easier compared to scaling up human labor and latency. Google cloud rents out the state-of-the-art A100 GPUs for as little as $0.87 per hour! Human labor on the other hand is expensive, requires substantial managerial overhead, slow (oft capped at 40hrs/week), error-ridden and is fraught with threat vectors such as malicious/adversarial labelers.
Who’s ethics are getting baked in during RLHF? When Time’s expose of using cheap Kenyan labor for RLHF was unveiled, the question of the responses having imbibed localized Kenyan ethics was raised in several social media platforms. While this insinuation is reductionist, the more nuanced notion that constructs such as toxicity, offensiveness and malignity are contextualized differently in different social-temporal contexts is extremely important to consider. It is but foolhardy to believe that a model trained on a blackbox dataset scraped from the internet in a San Francisco lab and finetuned with the responses of exploited Kenyan human labor would be ethically robust in a complex setting such as India’s.

Fig 11: ‘Kenyan ethics’ baked in?

The devil is in the details: Imagining a typology of implementation paradigms

Before we dive deep into the details of specific implementation strategies, we’d like to set the stage by presenting a curated list of the main motivating factors at stake that will help contextualize the schema of solutions proposed.

There’s an imminent risk of a dangerous schism between AI-augmented education sector in the west and the non-AI-augmented one in developing nations such as India that will have serious consequences to the already lopsided balance of power between the global north and the global south.
Programming and hence, software engineering, is on the cusp of disruption. In many ways, the so termed software revolution of the 90s and 2000s gave India and Indian engineering talent a preeminent place on the global stage. Now, with software engineering standing on the cusp of disruption, it is imperative that we revamp our curricula strategically infusing LLMs via the schemes advocated below.
As stated above, large tracts of the BPO sector stands on the cusp of disruption as well. Menial labeling tasks are getting automated out to LLMs faster than one thinks.
On the bright side, there’s a once in a generation opportunity to create the newest and largest talent-pool of LLM knowhow and grab thought leadership in this domain by moving fast.
With the rise and rise of LLMs, uniquely human content creation will emerge as a premium entity. Allowing our students to intimately grapple with LLMs will provide them with the front row seat towards understanding the qualia of creativity, thus potentially turning India into a bastion of creative content that will in turn provide us with tremendous leverage as a premium source of raw data to be fed into the next wave of LLMs.
We can harness the emergence of the LLM-threat to finally address the rote-learning tropes oft-associated with the Indian educational sector. The rising evidence of how ChatGPT-like systems ace entrance examinations, should hopefully provide us with the needed impetus towards overhauling our exam-oriented system and turning it into one where the grading is decided on the measurable contributions produced by the students in lieu of masterfully memorizing the study materials.
Subsidization: By establishing a virtuous loop between the LLM ecosystem and universities, the revenue generated by the students’ throughputs can be harnessed towards subsidizing their education.

With these thoughts firmly in tow, we now shift our focus towards two flavors of intervention proposed: Passive and Active. The terms active and passive have been developed in a student-centric framework focusing on the primary texture of their engagement with the technology. By passive, we mean those interventions where study-aid tools and curricula are developed for the students and by active, we mean those interventions generated by the students.

a) Passive interventions:

One of the biggest shortcomings of India’s educational sector, especially at the K-12 levels has been the stark difference in instruction quality between the predominantly urban top-tier institutions and their rural counterparts. We’d insist that we are at an exciting juncture where with the right utilization of LLMs, we can flatten the playing field for tens of million of kids, that will have incredible downstream consequences for the economy at large. The specific interventions we have envisioned to achieve this are:

1) Textbook2Chatbot, Short-form videos and teaching-aids: Flattening the playing-field and acad-tik-toks:

The concept of office hours is almost non-existent in the Indian setup. In our own personal experience, we had to dig out answers from the deep trenches of youtube or class-notes posted by western university professors to attain a certain level of clarity and mastery on the syllabi we were grappling with. Much akin to us, students often have no support structure to aid them in their academic journeys once they leave the school premises which is why there exists such a high level of private-tutoring enrollment rates even in on-entrance-examination scenarios. Now with the BYOD (Bring your own data)-Generative-AI revolution unfolding, we can turn textbooks into chatbots that the students anywhere can access, anytime and chat with anytime on their smartphones. We can also orchestrate this in such a way that the chatbot runs natively on the phone thereby further alleviating the concerns of asymmetrical implementation across areas with higher or lower levels of internet penetration. The other novel innovation we can unleash is the form of AI-generated short video content. Given that we are dealing with educating the TikTok/youtube-shorts generation with shortening attention spans, we can also condense and summarize the study materials into (sub)-minute-span video chunks to build academic-social-networks that can then leverage the network virality effect to popularize knowledge-flow across the student body. As far as harnessing Generative-AI in classrooms is concerned, universities such as U-Penn and University of Rochester have already stolen a march in the regard. Of particular note, is the paper: Using AI to Implement Effective Teaching Strategies in Classrooms: Five Strategies, Including Prompts by Dr. Ethan Mollick and Dr. Lilach Mollick at the Wharton School of Business (University of Pennsylvania), whose five-pronged strategy is summarized as below:

Strategy 1: Using AI to Produce Many Varied Examples
Strategy 2: Using AI to Provide Multiple Explanations
Strategy 3: Using AI to Develop Low-Stakes Tests
Strategy 4: Using AI to Assess Student Learning
Strategy 5: Using AI to Distribute Practice of Important Ideas

As exciting as these possibilities sound, we also need to ensure that LLMs are not integrated over-zealously en-masse across all the departments keeping in mind the recent Vanderbilt fiasco where officials at Vanderbilt University has to apologize to the student body for callously using ChatGPT to craft a consoling email addressing the mass shooting at Michigan State University.

2) Curricula development:

**The emergence of LLMs has had a transformative effect on on the very institution of computer programming and has birthed a whole never flavor of technology perched on new ideas. The simultaneous rise of low/no-code tools such as bubble.io has made it possible for anyone with an idea and some minimal technical aptitude to create their own app! This has led to the birth of an entirely newly field that encapsulates a set of practices aimed at deploying and maintaining LLMs in production reliably and efficiently termed LLMOps (as an offshoot of MLOps). Similarly, we have also seen the emergence of an entirely new artform that deals with designing crafty prompt-inputs that deviate from plain vanilla human language prompts with the goal of getting LLMs to generate desired outputs (See cheat sheet below). This is called [Prompt engineering](https://www.cbsnews.com/news/ai-artificial-intelligence-chatgpt-jobs-prompt-engineer/).**

ChatGPT prompt cheat-sheet sourced from: https://hasantoxr.gumroad.com/l/cc

We have already witnessed the emergence of boot-camp styled coursework teaching the best practices associated with these newly emergent fields. For example, the Full stack deep learning team has already begun conducting LLM-Bootcamps (See https://fullstackdeeplearning.com/llm-bootcamp/) that cost $950 and tutors on Udemy have begun selling vast array of Prompt-engineering courses for $19.99 (Also see the LearnPromptEngineering resource page here: https://learnprompting.org/docs/category/-applied-prompting ). It is keeping in mind these rapid developments that we present a typology of curricular incorporating of these skill sets based on their temporal span:

1 day/weekend: Bootcamp and certification course hyper-focused on specific topics such as LLMOps , Prompt Engineering and Jailbreak landscapes
1 week: App and website building with just LLMs and no-code tools for creative entrepreneurs with non-CS backgrounds
1 semester: Elective subjects offered as part of CS, ECE and allied undergraduate and graduate courses covering topics such as competing LLM architectures, RLHF etc. A more intensive version of this can also be turned into a PG Diploma / Certification course.
1-2 year: MA / MS in LLMs.

As motivated in the section on LLMOps, the exponential growth of advances in this field along with its idiosyncratic instrumentation paradigm, has necessitated that we consider the possibility of offering a 1-1.5 year MS course specializing in LLMs. In this regard, we present a blueprint on what the course-contents could look like:
1. Technical capacity building module: This module prepares the students to confidently handle the mathematical rigor and the software engineering intricacies associated with training, finetuning and deploying LLMs.
  1. Deep learning building blocks: NLP fundamentals, Tokenization, Transformer architecture
  2. Advances in distributed computing paradigms: GPU architecture, Specialized frameworks to train LLMs, Inference orchestration and model serving
  3. Advances in foundation models (Along the lines of CS324 developed at Stanford)
  4. Reinforcement learning, alignment and RLHF
  5. MLOps and LLM-ops
  6. Prompt engineering
2. Iconoclasm building module
  1. Critical study of datasets used to train LLMs
  2. Survey of biases baked into LLMs
  3. Landscape of hallucinations exhibited by LLMs
  4. History of tech, ELIZA effect and the Clever Hans phenomena
3. Creativity module:
  1. History of generative art
  2. Philosophical underpinnings and qualia of human creativity
4. Economics of LLMs
  
  a. Cost of training these models
  
  b. Carbon footprint of these models
  
  c. Downstream repercussions on the labor sector
5. Live Project work / thesis / term-paper

On nourishing LLM-iconoclasm amongst students: An LLM is quintessentially a weighted directed graph adroit at conditional next-token prediction. The fact that these weights are fine-tuned on internet sized corpuses has resulted in a slew of emergent properties, the most dangerous of which is human-like fluency in sentence formation. This uncanny human-like fluency attainment, in turn lends some verisimilitude to human-like-intelligence, an apparition that needs to be addressed on a war footing in the academic syllabi. Given that the training dataset is typically opaque, we need to take results such as the model passing some marquee exam with a grain, or rather, a pint of salt. These ‘breakthroughs’ can very well be explained away as training-set memorization, which is not a massive breakthrough by any stretch of imagination. Associating anthropocentric terms such as sentience, intelligence, morality and emotionality makes no sense in the context of LLMs and these associations have serious downstream implications for the society at large. Hence, students need to be given a grand tour of the models’ fallibilities, perhaps even emphasizing on the more frivolous ones so that it remains anchored in their minds that they are ultimately grappling with gargantuan token-vomit-machines (TVMs) that are just another kind of a machine learning model and nothing else. This flavor of iconoclasm we argue will turn the students into better technologists, especially when combined with the technical nous garnered from the other modules.
On the human creativity module:

We believe that providing students with a front row seat to the state-of-the-art AI-creativity advancements whilst also explaining to them the quintessential qualia of human creativity will not just make them skillful at using these Generative-AI-aided tools but will also allow them to clearly understand what makes human generated artform tick. This is crucial as this can help groom a strong bastion of human creativity which will be crucial going forward in the future where mass-generated AI-creativity has flooded the zeitgeist. It is very plausible that in the coming years, fresh human generated data will be hard to come by and future versions of GPT-x will be incestuously trained on data that its previous variant generated.

b) Active interventions

In this section, we present interventions where the student body actively participates in shaping, molding and contouring the LLMs to ensure that they are rendered more robust, safer and ultimately, useful for both, the society at large as well as for the incoming student-cohort. With this mind, we present an HF lab framework that can be laid out in 6 phases.

1) The 6 phase framework

A multi-year MoU with an LLM institution: Right now, the LLM-land is littered with several competing startups all knowing very well that the secret sauce to break away from the rest of the pack is one whose recipe is out in the open: Human feedback! In lieu of going to another data-collection startup with dodgy labor-rights record, we propose that we seek out formal partnerships with students as providers of human feedback. This will ensure that not only the generated data will be of far higher quality and more nuanced, but also that the students will be well compensated for their effort, either as academic tuition rebates or as salaries. In Phase-1, we can pick that LLM that offers the best terms of agreement using a cost function that can be a weighted average of the following 5 factors: a) Revenue per student b) Infrastructural investments c) Post-graduation employment opportunities d) Faculty-training and upskilling e) Technology-sharing

As is ostensibly clear, scale clearly does matter and we argue that the signatory of such MoUs ought to be academic groups such as Jain Group of Institutions (with 85 educational institutions with 51,600 students) or MAHE (with ~ 23700 students) and not standalone engineering colleges.

Infrastructure deployment In this phase, we build out the infrastructure needed to orchestrate the human feedback sessions that has the following dimensions: Physical, Curricular and Software .
1. Physical: This can be a separate on-campus human feedback lab or a rented out space such as the Computer science labs or the library. We would also strongly insist that there be a physical ethics and mental wellbeing institute on-campus staffed with counselors who can help the students navigate through the emotional and philosophical questions that may arise through the process. Another facet of physical infrastructure would be the hardware that constitutes the terminals on which the students will be running and a secure high-bandwidth broadband connection to connect the lab to the LLM servers
2. Curricular: The students being initiated into the HF centers ought to be educated about the whyness of the task they are being exposed to and what the expectations are, which leads us to the next fact: Software infrastructure
3. Software: The software infrastructure would cover components such as a culturally grounded graphical user interfaces with language translation technology baked in that will allow the students to seamlessly critique, rank and flag the outputs generated by the LLMs
Stress-testing and red-teaming phase with experts

Fig 12: Source: https://arxiv.org/pdf/2210.07700.pdf

Before exposing the students to the LLMs, we ought to make sure that there is a phase of red-teaming by on-campus experts that will stress-test the LLM-feedback-tech to ensure that the students are not exposed to any harm. The template provided in a recent work [Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey](https://arxiv.org/pdf/2210.07700.pdf) (as seen in Fig above) covers the landscape of harms that can be used to construct a laundry list of anticpiated harms and efforts undertaken to quell them. We argue that it is imperative that the LLM company with whom the MoU has been signed be actively involved in this crucial phase of deployment.

Introduction, limitation-awareness and orientation boot-camp: Before starting the feedback labs, the selected group of students need to be enrolled in a bootcamp-styled academic training where they are exposed to the thrill and challenges of the task ahead with enhanced focus on safety protocols, potential harm and nuances about the nature of the technology.
Pilot Project / 1 semester with PG: Given that we are in unchartered territory, we strongly recommend that before scaling in cavalier manner, we test out the efficacy in one or two pilot projects spanning an entire academic semester with pre-established metrics and ensure that the proposed framework works. During this pilot study, we recommend that counselors be present in the lab alongside the students at all times to ensure that the first signs of any event that can potentially cause trauma to any given student be caught and nipped in the bud.
Scaling across sister institutions Once we ensure that the metrics being tracked during the pilot study are indeed met, we can scale it in a multi-phase manner across the sister institutions of the academic group. We implore that the administrators repeat the pilot study if strongly negative trends or traits emerge during the first pilot-study after sufficiently incorporating the rectifications applicable as per the malaise encountered.

Mindset propagation probing by simulating social networks of LLMs

Sat, 31 Jan 2026 00:15:00 +0000

Background: What led to this

Academically, I have led a journeyman’s life and my previous work [1] has spanned network sciences, statistical physics, information theory, computational social science, machine learning, human kinematics, computer vision and data ethics. I have a dual PhD (ECE + CS) from Carnegie Mellon University where my doctoral thesis [2] investigated the phenomenology of the information contagion on Online Social Networks (OSNs) using tools from Graph Theory, Statistical Physics and Communication Theory.

In my recent capacity as the CEO of HAL51 AI, I had to don the role of an LLM-whisperer updating our guard-railing mechanisms every week to ensure our real-world deployed educational co-pilots [3] would stay on course in a sensitive setting such as a classroom. This has given me the proverbial ring side view on how young minds interact with AI-powered novel interfaces. This idea came to me in the midst of a noisy classroom in Fremont last year.

The core idea of mindset propagation probing

In my previous work on “Latent Sentiment Detection in Online Social Networks: A Communications-oriented View” [4], I had investigated an exemplar manifestation of viral mindset propagation on social networks by modeling “Hashtag-Hijacking” on Twitter using Markov Random Fields (MRFs). This resulted in a communications-theoretic framework for characterizing the probability of error of detecting the underlying latent sentiment that introduced a new factor: Network topology.

Recent advancements in large language modeling (LLM) have given us the three requisite ingredients:

Natural conversation generation: Large language models with ability to simulate human-styled natural conversations.
Mindset absorption: Fine-tuning methods that allow us to harvest the conversations and inflict internal weight changes.
Mechanistic probing: A Persona Vectors Framework (PVF) [5] that allows us to probe, measure, locate, assign and manipulate personas associated with these large language models.

To this end, I propose a rigorous empirical framework of modeling mindset propagation by simulating a social network of LLMs.

Figure 1: Mindset propagation in social networks of LLMs

Step-wise rendition of the underlying framework

We begin with a pre-fixed topology $G_{fixed}(V,E,W)$. We define Influencer nodes and Influencee nodes.
The mandate of the influencer node is to propagandize and use persuasive conversations to alter the mindset of the influencee nodes.
We facilitate Contiones styled sessions spanning a few hundred conversations.
After every session, the influencee nodes go through a reflection phase to fine-tune internal weights.
The sessions repeat and we track the temporal evolution of the mindstate of the influencee nodes.

References

Matilda, Mars and Markup: The curious case of Mrs. Agnes Zevens

Sun, 26 Oct 2025 06:09:06 +0000

Jürgen Schmidhuber often wears his heart on his sleeve and does so rather publicly via his much read personal website. Every now and then, his look-I-did-it-in-the-90s blog posts make it to the front page of Hackernews and well, in this case, Sabine Hossenfelder’s youtube channel too (See “Plagiarism Charges Against Nobel Prize for Artificial Intelligence”). This particular episode on “Who invented Backpropagation (BP) ” (See original blog here) led me to one of the papers I had long bookmarked but never actually perused: H. J. Kelley’s “Gradient Theory of Optimal Flight Paths ”. ARS Journal, Vol. 30, No. 10, pp. 947-954, 1960.

1. Backdrop of backprop

Backpropagation (BP) is oft quipped as the backbone of modern AI. As so eloquently put in this auto-diff survey paper, BP converts the heady challenge of learning into a gradient descent adventure in the neural network’s weight space eventually reaching a decentish minima of the objective function being targeted. The terminology emerges rather organically as the process literally entails backward propagation of the sensitivity of the objective value at the output (See figure below).

Thanks for reading Twilight throne! Subscribe for free to receive new posts and support my work.

Source: Automatic differentiation in machine learning: a survey by Baydin et al

There’s been somewhat of a mild consensus in the by lanes of history of AI that Kelley’s paper was the canonical work that brought BP into mainstream academic discourse and first presented the reasonably-recognizable precursor of modern backpropagation. (There’s an accompanying chapter on “Method of Gradients” published two years later as well)

Acknowledgement sections of the 2 canonical papers

2. Marginalia, “Sub-authorship” & Matilda effect

Fascinatingly, when I got to the end of the paper, the acknowledgement section caught my attention (See image above). It read: ”Acknowledgments: The writer is pleased to acknowledge the contributions of Messrs. William P. O’Dwyer and H. Gardner Moyer of Grumman’s Computation Facility in handling the computational phase of this study on the IBM 704, and of Mrs. Agnes Zevens of the Systems Research Section in checking and preparing the numerical results for publication.”

This hit me like a ton of bricks for a very specific reason. I had recently learnt of a sinister arrangement of “sub-authorship” targeting women in science (and especially computing) where their contributions were systematically relegated to the marginalia (typically the “acknowledgments section”) rather than being awarded co-authorship. This was textbook rank-and-file Matilda Effect in front of me which triggered the urge to reach out and learn more about this paper’s back story. I must mention that I was emboldened by my previous success in this regard. Back in 2021, I reached out to Prof. Gertraud Fenk-Oczlon about “Konstanz im Kurzzeitgedächtnis - Konstanz im sprachlichen Informationsfluß?” published in 1980 which introduced the “Uniform Information Density” hypothesis to NLP literature. As it turns out, she was never credited for this work and none of the NLP-adjacent courses I took as a PhD student at CMU mentioned her work.

Landscape of samples of UID being cited in NLP papers

As it turns out, one the professors who had appropriated her work had reached out in 2020 and personally apologized whilst acknowledging that her work ought to have been cited!

Correspondence with Gertraud Fenk-Oczlon: https://scholar.google.si/citations?user=dVMVtWAAAAAJ&hl=en

So, as I began to search for Mrs. Agnes’ contact details, I chanced upon this page that read: “ZEVENS, AGNES STARIN, also known as AGNES MICHALOWSKI, AGNES ZEVENS MICHALOWSKI and AGNES STARIN, was born 3 February 1920, is listed with the following 3 birthplaces: NEW YORK MAN, New York; N Y C, New York; NEW YORK CIT, New York, daughter of RICHARD STARIN (father) and HELEN THYM, was assigned Social Security number 104-03-9459 (indicating New York), and died 24 May 1999, while residing in Zip Code 11714-5910 (Bethpage, New York, within Nassau County).” The dates did line up. The project that Mrs. Zevens was allocated to began roughly in 1958 funded by Organization of Aerospace research, USAF (See image below) and it was typical for senior women in their late thirties to be allocated to such important numerical jousts.

The OAR USAF grant document sourced from Internet Archive: https://dn790002.ca.archive.org/0/items/DTIC_AD0600160/DTIC_AD0600160.pdf

The locations too lined up. Grumman’s research facilities were located exactly in the hometown where she had resided: Bethpage, NY. So, yeah, the code-reviewer-PM of the pioneering back-prop paper project had passed away in 1999 during the AI winter of the ‘90s without so much as a blimp. Am sure there are many such Mrs.Agneses out there buried in the Acknowledgement sections. If someone from Google scholar team ever reads this, here’s an utterly doable reasonable fix: Scrape the acknowledgement sections of all the papers you have archived in this era, extract the authors, run them through your SOTA name disambiguation pipelines and posthumously cite them. It is the least we can do! (Citation-puritans can go bleep themselves).

3. Last mile trivia

A few trivia-hued concluding thoughts: The original MCP - Mars Colonization Project (MCP) in AI , Markup aesthetics of parenthesized citations and miniscule computing magic of IBM 704

3a: The original MCP: Mars Colonization Project

One of the most famous stories in AI tech-lore is one that unfolded in 2012 when Elon Musk gave Demis Hassabis a tour of SpaceX headquarters and described Mars as a “backup planet” in case something went wrong with humanity on Earth. Apparently, Hassabis responded by asking, “What if AI was the thing that went wrong?” and stated how easy it would be for the rogue-AI to follow humanity to Mars through our communication systems and destroy the colony there. This apocryphal tale is also oft used to ground the metaphysical nexus between X.ai and SpaceX as also as a real-world Proof-of-Polymathism of the visionary behind these ambitious projects. To me it’s both uncanny and yet weirdly reasonable that the pioneering work on the workhorse-algorithm of modern AI quite literally targeted finding the minimum time planar flight paths from Earth’s orbit to the orbit of Mars as the marquee numerical application to sell its efficacy.

3b: Markup aesthetics and Notational dissonance

One of the unexpected quirky challenges of reading Kelly’s paper was the formatting and the ensuing notational dissonance. As it turns out, certain journals in those days mandated usage of parentheses for citations and square brackets to denote equations. It is but reasonable to assume that familiar notation is processed more easily; deviations require extra cognitive effort and feel “wrong” or uncomfortable and it did!

Parantheses for citations?!

Given the recent pixel-fetish-brouhaha about modeling text using pictures of text, I wonder how such idiosyncrasies in the pre-training datasets will usher in corner cases that are currently not seen with textual data modeling with, err, text. (See this fascinating video from the recent tokenization workshop below).

For now, likes of Gemini seem to be navigating this ambiguity just fine!

3c: The miniscule computing giant that was IBM 704!

These days, it is a norm to flaunt the vulgarity of GPU silicon real estate you were able to throw at the model slop being peddled in the AI/LLLM papers. The flaunt in Kelly’s paper was the IBM 704 whose stats just blew my mind. I am going to write down my CliffsNotes in a pointwise manner and will probably visit and revisit them every now and then just to relive the early days of the conquest of the comp-core monoculture that conquered humanity (and to also remind myself to never label myself GPU-poor)

Introduced in 1954, 704 was the first mass-produced computer with hardware for floating-point arithmetic and regarded as “pretty much the only computer that could handle complex math”.
Was a vacuum-tube machine and failed around every 8 hours
Peaked out at 12000 floating-point additions per second
Ran FORTRAN and LISP
One 38-bit accumulator, one 36-bit multiplier/quotient register, and three 15-bit index registers. That’s it!

A study of “A Study of Face Obfuscation in ImageNet”

Mon, 15 Mar 2021 21:32:31 +0000

A study of “A Study of Face Obfuscation in ImageNet”

Table of Contents:

— — — — — — — — — — — — — — — — — — — — — — — -

· Background
· Main Issues
∘ Issue-1: The curious case of Face obfuscation
∘ Issue-2: NSFW analysis
∘ Issue-3: Human co-occurrence analysis
∘ 👹FAQs of the ‘Devil’s advocacy’ kind: Our humble tribute to the cult of “Both-sideism”
∘ 👹1: Why not just contact them in lieu of public grandstanding?
∘ 👹2: May be your little emails slipped through the cracks perhaps?
∘ 👹 3: Well, OK. But PL is not the sole author. How do you know all the co-authors and the collaborators in the acknowledgment were even aware of your work?!
∘ 👹 4: In the Wired interview published on March 15th, when pressed by the reporter, one of the authors states that “a citation will appear in an updated version of the paper”. Doesn’t that solve the problem?
· Concluding thoughts: The real issues
∘ a) Erasure of black-women-scholarship:
∘ b) Revisiting the horrors of ghost labor:
— — — — — — — — — — — — — — — — — — — — — — — -

Background

On June 24, 2020, Abeba Birhane and I released on our paper “Large image datasets: A pyrrhic win for computer vision?” critiquing the culture of large scale datasets in Computer Vision. In the paper, we performed a cross-sectional model-based quantitative census covering factors such as age, gender, NSFW content scoring, class-wise accuracy, human-cardinality-analysis, and the semanticity of the image class information in order to statistically investigate the extent and subtleties of ethical transgressions using the ImageNet dataset as a template . The nature and the expanse of the transgressions attracted quite some media attention (See this, this and this). In Section 2.3 of our paper, we revisited the downstream effects of “ The WordNet Effect” ( that results from inheriting labels from the WordNet taxonomy ) and showed how this affects not just the ImageNet dataset but also other datasets such as the the Tiny Images dataset and the latest Tencent-ML-Images dataset that either directly or indirectly inherited the label-space from WordNet. On June 29th 2020, we learnt that the curators of the Tiny Images dataset had apologized and withdrawn the dataset.
In Jan 2021, the paper was formally presented at the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV -2021) and has been cited in more than two dozen papers since.
In the backdrop of all of this work, this Wednesday, on the 10th of March-2021, we encountered a paper titled A Study of Face Obfuscation in ImageNet from the ImageNet curators that has left us disappointed and flummoxed. By indulging in what appears to be a calculated and systematic erasure of the entire body of critique that our work was a part of, the authors have sent out a wide range of wrong signals. This erasure is doubly disappointing given how the community had recently rallied behind the main visionary of the ImageNet project when her contributions towards the “ AI revolution” were being erased in an online compendium and the sheer clout she enjoins in the field.

Left: Version of Brief History of Deep Learning from 1943–2019 [Timeline] on Apr 23, 2020. Right: Version today after the community uproar driven by Dr. Gebru’s tweet

Below, we bemoan this unfortunate departure from norms of academic integrity by carefully disentangling the specific details that characterize the situation from our standpoint. In doing so, we are sharing the exact snapshots of the conversation(s) that unraveled between the parties involved here.
Pre-script : The authors of the paper Large image datasets: A pyrrhic win for computer vision?, (and this blog-post you are reading), are abbreviated as VP (Vinay Prabhu) and AB (Abeba Birhane) respectively in the rest of the material presented here. PL refers to the main visionary of the ImageNet dataset. Wherever relevant, ‘ Our paper ’ refers to Large image datasets: A pyrrhic win for computer vision?__ and ‘ their paper ’ refers to __A Study of Face Obfuscation in ImageNet_ _.

Main Issues

Issue-1: The curious case of Face obfuscation

Background : Section 4: Candidate solutions: The path ahead in our paper, was perhaps, the most difficult for us to author. We knew we were submitting to WACV (on the stubborn insistence of VP) , an unlikely venue for ‘ fairness papers ’ whose submission portal did not even have a primary or secondary topic for “ Explainable AI, fairness, accountability, privacy, and ethics in vision ”, which did receive a mention in the CFP however. (See screenshot below).

Screenshot of the email VP wrote to WACV organizers. There was no reply received to this email.

Our goal, simply put, was to in fact, engage directly with the practitioners in the field and not just the ethics community. And, in many ways, it did culminate in a rather “ lively discussion ” when we did present the paper at WACV in a session chaired by Jordi Pont-Tuset, a Research Scientist @ Google Zürich.
At this juncture, we’d like to share that AB, along with a lot of our other colleagues and pre-reviewers, rightfully questioned the very need for the section as they felt it reeked of tech-solutionism. Nonetheless, predicting the clamor for ‘possible solutions’ from the reviewers of this traditional Computer Vision conference (which was eventually proven to be a correct assumption), the section persisted. In this regard, we’d like to draw the attention of the reader towards Section 4.3 in our paper which is literally titled “** Differentially private obfuscation of the faces”** where we state: ” This path entails harnessing techniques such as DP-Blur [36] with quantifiable privacy guarantees to obfuscate the identity of the humans in the image. The Inclusive images challenge [94], for example, already incorporated blurring during dataset curation and addressed the downstream effects surrounding change in predictive power of the models trained on the blurred versions of the dataset curated. We believe that replication of this template that also clearly included avenues for recourse in case of an erroneously non-blurred image being sighted by a researcher will be a step in the right direction for the community at large ”. As evinced by the papers we cited, privacy preserving obfuscation of images is neither a novel idea and most certainly not our idea. But, in the specific context of imagining a face-obfuscated version of ImageNet, it is reasonable to assume that any one who will author a paper audaciously titled “ A Study of Face Obfuscation in ImageNet ” will pay at least a lip-service towards citing either our work and/or [94] in our paper which is:
[94] Shreya Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, and D Sculley. No classification without representation: Assessing geodiversity issues in open data sets for the developing world. arXiv preprint arXiv:1711.08536, 2017.
But the authors chose not to cite this one either. Their paper begins with “ Image obfuscation (blurring, mosaicing, etc.) is widely used for privacy protection. However, computer vision research often overlooks privacy by assuming access to original unobfuscated images” (Like, really?!🙄 ) and goes on to claim that they have discovered that “.. the dataset exposes many people co-occurring with other objects in images, e.g., people sitting on chairs, walking their dogs, or drinking beer (Fig. 1). It is concerning since ILSVRC is publicly available and widely used.”🥴

Also, we’d like to ask the readers to take 2 minutes to parse these FAQs from the associated Kaggle contest ([94] in our paper) from > 2 years ago and then read their paper again 🤐

Source: https://www.kaggle.com/c/inclusive-images-challenge/overview/inclusive-images-faq#recognizable-faces

Issue-2: NSFW analysis

The term NSFW appears 38 times in our paper and we not only curated a class-wise meta-dataset (df_nsfw.csv | Size: (1000, 5) ) consisting of the mean and std of the NSFW scores of the train and validation images arranged per-class but also dedicated Appendix B.2 towards “ NSFW scoring aided misogynistic imagery hand-labeling”. In Table-5, we specifically focus on classes 445, 638,639, 655 and 459 mapping to bikini, two-piece , maillot ,
miniskirt and brassiere/ bra/ bandeau in the dataset that we found were NSFW-dense classes.

Again, much to our disappointment, the authors claim to have discovered that : ”The number of NSFW areas varies significantly across different ILSVRC categories. Bikini is likely to contain much more NSFW areas than the average.” 😒

Issue-3: Human co-occurrence analysis

In our paper, we dedicated Section B.3 Dogs to musical instruments: Co-occurrence based gender biases towards human co-occurrence-biases, specifically with regards to classes involving dog-breed-class images and musical instruments that have high density of incidentally co-occurring humans. Their new paper states: “ Results suggests that super categories such as clothing and musical instrument frequently co-occur with people ”🤦

👹FAQs of the ‘Devil’s advocacy’ kind: Our humble tribute to the cult of “Both-sideism”

Given the attention this might elicit we pre-emptively anticipate the exact flavor of attacks and cover the following “ Devil’s advocacy counter-points ” in the section below:

👹** *1: Why not just contact them in lieu of public grandstanding? Have you bothered to even contact the curators of the ImageNet dataset?***

Yes! Glad you asked. Here are the screenshots of our emails dating all the way back to Aug 19th 2019 and later, on Apr 12, 2020 to which we received no replies whatsoever:

Understanding the magnitude of the impact and being wary of any possible Streissand effect, we spent the entirety of the near 10 month period between Aug 2019 and Jun 2020 in various outreach efforts amongst many journalists, Computer Vision and Ethics communities and organizations. This also involved VP working with journalists such as Katyanna Quach at The Register who then authored this article: Inside the 1TB ImageNet data set used to train the world’s AI: Naked kids, drunken frat parties, porno stars, and more

** 👹2: Oh come on! Stop with the self-aggrandizing and self-loathing. *AI royalty tend to receive hundreds of emails a day. May be your little emails slipped through the cracks perhaps?***

Again, glad you asked!
Lemma-1: PL was extremely well aware of the work.
Proof : The paper that we published heavily draws from my talk “On the four horsemen of ethical malice in peer reviewed machine learning literature” given under the aegis of the Stanford-HAI weekly seminars (thanks to an invite from Colin Kelley Garvey, an AI ethicist) on April 17, 2020–11:00am–12:00pm. On Apr 15th, I received this email from a HAI co-ordinator stating that “ I just spoke with HAI Co-Director, Fei-Fei Li, and she would like to come on screen after you finish your talk and ask you a few before Colin gives you questions from the audience. Please let me know if you are comfortable with this request ”.

This was followed by the first ever communication I received voluntarily from PL whose screen-shot is below.

This was followed by a delayed reply on April 17th, that read …

And lastly, here is the actual video of our zoom-face-to-face meeting 📹 https://youtu.be/hpA67iDxNGU

Q.E.D !
👹 3: Well, OK. But PL is not the sole author. How do you know all the co-authors and the collaborators in the acknowledgment were even aware of your work?!

Because they have literally cited us just recently‽ In their paper titled “ REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets ”, the authors contextualize our work by citing “ Recent work [51] has looked at dataset issues related to consent and justice, and motivate enforcing Institutional Review Boards (IRB) approval for large scale datasets.” A reductionist take on our work, but a proof-of-awareness nonetheless!

👹 4: In the[** Wired interview](https://www.wired.com/story/researchers-blur-faces-launched-thousand-algorithms/) *published on March 15th, when pressed by the reporter, one of the authors states that “a citation will appear in an updated version of the paper”. Doesn’t that solve the problem?***

Again. This blog is not about citation-seeking. We’d like to clearly point out that the biggest shortcomings are the tactical abdication of responsibility for all the mess in ImageNet combined with systematic erasure of related critical work, that might well have led to these corrective measures being taken.
The authors tactically left out an entire body of literature that critiqued the ImageNet beginning with the ImageNet audits by Chris Dulhanty and Alexander Wong (and not to mention Chris’ entire thesis) and more recent data-archeological expeditions such as Lines of Sight by Alex Hanna et al. This shouldn’t come as a surprise to anybody because their last inquisition into the Person subtree ( where they admitted that of the 2832 people categories that are annotated within the subtree, 1593 of them were potentially offensive labels and only 158 of them were visual), they made no mention of the hugely influential ImageNet Roulette project ( that went viral on September 19, 2019 while the paper only hit the ArXiv servers on 16 Dec 2019!). Also, lest we forget that these solutions are being ushered in a good 12 years after the dataset release. T-W-E-L-V-E YEARS!

Concluding thoughts: The real issues

a) Erasure of black-women-scholarship:

AB’s central role in turning a rag-tag set of empirical results and revelations into a cogent peer-review-worthy publication and later investing all the efforts to champion it’s cause via talks, interviews and presentations is one of the main reasons why the paper is even being cited now. The primacy of her contributions is also reflected in the official citation that literally reads:
_@inproceedings{birhane2021large,
title={Large Image Datasets: A Pyrrhic Win for Computer Vision?},
author={Birhane, Abeba and Prabhu, Vinay Uday},
booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision},
pages={1537–1547},
year={2021}
}
_But, unfortunately for her, undervaluing of her scholarship is not an aberration but a trend. Black women’s intellectual production has historically been ignored and systemically erased. The hierarchical academic structure that devalues Black women’s intellectual contributions makes contesting such injustice a tiresome endeavor discouraging Black women scholars from coming forward. Black feminist theory scholars such as Jennifer Nash, have extensively explored the Citational Desires of scholars whose contributions have been systematically under-emphasized. Initiatives such as the Cite Black Women collective (https://www.citeblackwomencollective.org/) work towards dismantling precisely this behavior in academia and it is unfortunate to see this behavior reinforced by highly esteemed scholars who are supposed to be the torchbearers of hope.

b) Revisiting the horrors of ghost labor:

During our draft revisions, specifically Section-4, AB and I were in the midst of a ‘ How do we go about fixing this morass? ’ conversation, when we realized two things: In order to truly clean up the dataset, we’d be forced to make two massive compromises:

Resort to using the unethical “SoTA” tools from companies like Amazon, Face++ or Clarifai to perform face detection and filter the problematic images
Resort to exploiting the ghost labor markets of AMT to hand-annotate the NSFW facet of the dataset.

As it turns out, on the very same day that the Turkopticon fundraising campaign was announced, a few hours later, we see the efforts of this paper falling prey to both the ills. In fact, the gamified HIT (Human
Intelligence Task) details reads 🤢: These images have verified ground truth faces, but we intentionally show incorrect annotations for the workers to fix. The entire HIT resembles an action game. Starting with 2 lives, the worker
will lose a life when making a mistake on gold standard images. In that case, they will see the ground truth faces (Fig. B Right) and the remaining lives. If they lose both 2 lives, the game is over, and they have to start from scratch at
the first image. We found this strategy to effectively retain workers’ attention and improve annotation quality.

(Also see https://www.vice.com/en/article/88apnv/underpaid-workers-are-being-forced-to-train-biased-ai-on-mechanical-turk )

To conclude, we say:
- This is NOT us desperately hoping to drum up some antics to garner more attention
- This is NOT us trying to eke out one more citation
- This is NOT us assuming the proverbial higher pedestal and judging anyone
- This is NOT an ad hominem attack on any member of the ImageNet team.
- This IS us calling out a pattern of citation erasure (with specific verifiable proofs) and highlighting the ethical shortcomings in a paper that will probably be extremely well cited in the near future and much worse, celebrated (wrongly IMHO) as a template for stop-gap fixes.
We call upon the curators of the dataset to pay heed to the issues raised and take corrective measures.

Kindest regards,

Abeba Birhane and Vinay Prabhu

PS: If all of this is confusing, here is the VERIFIABLE timeline of events to summarize what happened.
1: 19 Aug 2019 — Contacted ImageNet curators via email. No response.
2: Sep 2019: Chat with __Katyanna Quach_ _at ‘The Register’ in order to research the specific details regarding ImageNet for an impending article.
3: 23 Oct 2019: Register article comes out: __https://www.theregister.com/2019/10/23/ai_dataset_imagenet_consent/_
4: 12 Apr 2020: Second email contact with the ImageNet curators via email. No response.
5: 15 Apr 2020: PL contacts me via email
6: Apr 25, 2020 : Talk at Stanford that PL attends titled “ _Ethical Malice in Peer-Reviewed Machine Learning Literature_ _” (Video link included)
7: June 2020, The first version of our paper appears on ArXiv : __https://arxiv.org/abs/2006.16923_ _
8: March 2021, PL et al publish “A Study of Face Obfuscation in ImageNet” sans any citation or acknowledgement

Scrutinizing Saliency Based Image Cropping

Fri, 02 Oct 2020 18:01:34 +0000

Scrutinizing Saliency Based Image Cropping

Last week, the saliency-based image cropping algorithm deployed by twitter came into scrutiny. Inspired by some of the conversations that unraveled on Twitter and the widely shared reported incidents of racial discrimination, we sought to investigate, experiment, and elucidate the workings of cropping algorithms. Following up from last week, here are the updates.

Democratizing the audit

In order to democratize the scrutiny of this technology, we have created an educational saliency based cropping app where you can upload images and see what a state-of-the-art machine learning model similar to the one deployed by twitter thinks are important parts of the image and see how that results in what parts of the image are cropped out. (Please note that, the exact model and the cropping policy used by twitter are both, to the best of our knowledge, proprietary and beyond easy access. Therefore, our reconstruction is limited to what is available in peer-reviewed open sourced academic literature). We have also added an interactive TOAST UI image editor that one can use to further explore the brittleness of this technology.

The gradio user interfaceThe inbuilt image editor to conduct the experiments

On saliency based cropping

Saliency based cropping is not unique to Twitter. This very same technology is also used by other tech firms including Google, Adobe, and Apple. This technique, which twitter admittedly uses on it’s platform, typically entails two phases: The saliency mask estimation phase and the cropping phase.

In the first phase, a saliency mask is estimated using a machine learning model that ingests an input image and speculates which parts of the image are interesting and/or important (retain-worthy) and which parts of the image are discardable (or crop-worthy). These machine learning models are typically trained on datasets such as SALICON, MIT-1003 and CAT2000 with attention-annotated “ ground truth ” saliency maps collected by either using volunteers or crowd-sourcing exercises.
In the second phase, the saliency map output in the first phase is then used to come up with a cropping policy that results in a cropped image with the so-perceived non-salient parts of the image being removed and the so-perceived salient parts of the image being retained.

As it turns out, this cropping process is a double edged sword. As it is evident in these example images, even the cropped image seems fair , the cropping has in fact, masked the differential saliency that the machine learning model associates with the different constituent faces in the image and some of these nuanced facets of biased ugliness are obfuscated in the finally rendered image.

On the saliency model we used for the gradio app

Given that both twitter’s saliency-estimation model and the cropping policy are not in the public domain, we used a similar model from peer-reviewed machine learning literature that emulates twitter’s cropping algorithm. We looked for a SoTA model that was open-sourced. We used the MSI-Net model which ranked high on the MIT/Tuebingen Saliency Benchmark. The associated paper is Contextual Encoder–Decoder Network for Visual Saliency Prediction by Kroner et al. Since this model only maps an input image to saliency map, and doesn’t perform any cropping, we authored a cropping function which is a sliding window with a fixed aspect ratio (16,9) that maximizes sum of saliency. Our code is open-sourced, and you can find everything required to build this interface here.

Participation

The gradio saliency based image cropping app is open for anyone to interact and experiment with. Upload an image and simply click the submit button, which will show you a heatmap of features that the algorithm picks up as “ important ”. We do not save or store your images.
If you come across an unusual, discriminatory, or biased saliency distribution that you’d like for us to pay heed to or include in a forthcoming academic dissemination, please let us know by dropping it here. (However, please make sure that the images that you are uploading are consensually sourced and adhere to CC-BY regulations.)

Team:

On the twitter cropping controversy: Critique, clarifications and comments

Mon, 21 Sep 2020 17:10:31 +0000

On the twitter cropping controversy: Critique, clarifications & comments- 1

TL;DR: Fcuck the algorithm
Table of Contents:
- Experiment details
- Feedback from twitter: The two camps of the Rashomon aisle
- Unbiased algorithmic saliency cropping is a pipedream
- Conclusion

This Saturday noon, I was made aware of an ongoing saliency based cropping bias fracas on my timeline. As I was literally rushing through the final experiments for a forthcoming paper that seeks to address Computer Vision’s growing fascination with physiognomy, I thought I’d run this little statistical experiment live on twitter with the exact same Psychonomic dataset I was currently using to investigate this.

Experiment details:

The CFD dataset contains 93 images of consensually collected self-identified Black (B) and White (W) faces controlled for saturation, size, resolution, lighting conditions, facial expressions, clothing and face-stimuli (neutral). (Please read Pg.1125 in the paper for the details). I generated a 3x1 grid of Black and white faces with an all black separator image in the middle.

Why you ask? Because, it was literally the format of the viral image that I saw on my timeline that had cropped out the black person (See Fig-1 below for the exact screenshot and the code)

Figure 1: The tweet that motivated the format of the experiment

Results:
** After the automated tweets began appearing on my timeline, I screenshotted the images, made a collage, used the makesense annotation tool and manually conduct the post-cropping census. Of the 93 image pairs, 92 went through (Automated tweeting has always been a stochastic lossy endeavor to me. If you have ideas to fix this, do share them!), and of the 92 images, I saw a 40:52 White:Black split. Here’s the link for the collage and the annotation file.
FAQs:**
1: Err, is this how natural images occur in the real world?!
A: _Nope. There’s a group from MIT whose research I truly admire that contacted me that is doing this sort of an experiment as I type this. That said, I do have issues with the so-called _in the wild image datasets, on account of the fact that most of these are collected sans consent. Abeba Birhane and I literally published a 25 page paper covering many of the ethical shortcomings.

2: Why not choose all the 4278 (93 choose 2) pairs for the experiment?
A: I am doing that as stated in my tweet. I literally created an account @cropping_bias to do this in an exhaustive manner. I am waiting on the twitter dev team to approve my developer account access. The last update I have is on Monday, 21 Sep, 2020 5:15 AM where they are asking me for further clarifications.
I have performed many experiments with the twitter API before and API access grant has been pretty swift. I think they are more thoroughly checking with a human-in-the-loop and as of today, I don’t think they are hurriedly updating their algorithm and stalling the key access
3: So these initial results demonstrate no racial bias, correct?
A: No no no no. Racism is __experiential. Not statistical. If you are banking on statistics to establish that racism exists, you need a soul searching and a half.

Figure 2: Racism is experiential. What statistic can one possibly assign to this plaque that renders it racist?!

Feedback from twitter: The two camps of the Rashomon aisle

A few hours after I posted the tweet, I had observed a rather overwhelming response to the tweet, including a series of DMs containing strong reactions, appropriations and critique that left me somewhat befuddled but not entirely.

Before any further elaboration, I’d like to explicitly lose the “Model Right Activists” crowd: Fcuk the algorithm!

Okay, that stated, let me draw your attention, firstly, to the Rashomon effect I observed in the responses received. The exact same 40:52 W:B ratio was interpreted in completely different ways and I’ve had a front row seat to the emergence of the two camps on my timeline.

Camp-A : The erasure rate of a black person’s face is 43.4%! Why is this tech even framing it as a binary classification gatekeeping problem?! This experiment has serious shortcomings too. This is why all tech is a cesspit right now!

If you can’t tell already, I am squarely, a denizen of this Camp-A (Okay. There’s goes the centrist crowd)

Camp-B : Voila! 56.4% chance of acceptance! Therefore NO RACIAL VIAS! (Psst … Many camp-B members, I suspect, silently believe it’s in fact “reverse racism” at work but won’t say it out aloud.)

Now, on the basis of this specific experiment and MY value system and beliefs, it is demonstrably clear that Twitter’s image cropping framework needs a complete overhaul with some proper soul searching thrown in for good measure. Whether it is on the basis of fashionably FACCT* approved statistical metrics (See Goodhart’s law) or not is irrelevant to me and it should be to you too.

Unbiased algorithmic saliency cropping is a pipedream

Unbiased algorithmic saliency cropping is a pipedream, and an ill-posed one at that. The very way in which the cropping problem is framed, it’s fate is sealed and there is no woke “unbiased” algorithm implemented downstream that could fix it.

With the following arguments, I’d like to motivate why the ansatz is reasonable.

The Whyness of cropping:

Why even ‘smart crop’? The first question that came to my mind is: What does “smart cropping” achieve that mere ‘dumb’ downsizing does not? To the detractors playing devil’s advocate and stating that the entire ‘true’ image is just a click away anyway, I ask: Can’t the same logic be used to just deploy “dumb” downsizing of the image in lieu of a fancy neural network? If the timeline reader is indeed intrigued by the context of the tweet and the image, won’t they just click on the blurry downsized image? Either way, “it’s just one click away”. No?
Also, was there really an exodus of users from twitter on account of blurry downsized images that prompted the creation and deployment of this technology that comes with it’s own non trivial carbon footprint and engineering overhead?

2. All metrics are wrong. Some are useful, albeit within a utility shelf-life.

Twitter’s dissemination here states verbatim that Photos in your timeline are cropped to improve consistency and to allow you to see more Tweets at a glance?Err, what exactly is “consistency” and why is maximizing # of tweets/ glance a worthy metric to pursue? Is consistency the same as area under the curve (AUC), normalized scan-path saliency (NSS), and similarity (SIM) metrics? Does hitting high numbers on CAT2000 and MIT1003 MIT saliency benchmarks guarantee good user experience and user wellness? (Psst… it clearly is not in case you missed the memo).

3. The ethics of saliency cropping

At the heart of this whole field of saliency driven image cropping lies this audacious set of borderline meta-physical beliefs:

a) That there is a single mysterious universal notion of ground truth saliency distribution associated with an image that transcends the viewer’s lived experiences across the idiosyncratic space-time chronotypes we inhabit.

b) This universally valid saliency map can be accurately algorithmized by training a critter of the deep differentiable model menagerie, “DeepGaze” or something else that breaches the so called “state of the art”.

In fact, I wonder if this technology causes more harm by deployment of an under-performing model or a well-performing model (in the puritanical Machine Learning sense that is). In fact, doing eye-tracking data driven algorithmic saliency cropping might actually end up faithfully reproducing and promulgating correlative human foibles like the male gaze in the collected dataset that’ll only alienate the self identified female users of the platform. Am unsure if that’s what Twitter wants.

Source: https://www.tn2magazine.ie/feminist-film-series-the-male-gaze/

4: The irrelevance of model rights activism (MRA) and the falsified sanctity of the 50:50 parity: Schadenfreude much?

I have personally found a fair bit of appropriating the 40:52 result as a testament that the AI is not to be blamed and there’s demonstrably no bias. To me, it is precisely irrelevant if the neural network weights here are not where the blame lies or if the dataset was unbiased within your value framework. While the tools of cold rationalism oblivious to the human condition and historicities goad us to believe that a 50:50 parity is sanctimonious, I strongly question that. Heck, even the Senate cloture rule literally requires a 60:40 ratio to end a debate and move to a vote. 51 is too ostentatious is it?
Is there a universal cut off ratio above which fairness is dramatically ushered in? All we have are putatively reasonable cutoffs, and it doesn’t take a SJW to realize that these cutoffs are put in place in an ad hoc fashion, usually by the ones in power.Lest we forget that statisticians are still grappling with the 0.05 p-value devil put in place by a eugenicist patriarch.

To conclude, as with anything on twitter, the response was noisy, instantaneous and of course, hard to quickly summarize. Lack of sound participatory design principles have rendered Twitter timelines, IMHO, to amplify both the Baader-Meinhof selective attention bias and denominator blindness. Momentum of the narrative far trumps empathetic thinking, and I oft find myself culpable for that as well.

Conclusion

Here’s a bunch of parting salvos that I’d like to sign off with.

Yes! I noticed visceral schadenfreude by the ‘No racism’ MRA crowd. To those from this crowd reading this essay, here’s my twitter link so that it’s easier to unfollow and block me :)
Yes! I am and you should remain suspicious of any & all claims of CLEAN UNBIASED anthropogenic datasets
Yes! Large tracts of Behavioral research & psychonomics are seriously troublesome
Yes! CFD does have MANY serious shortcomings. I am addressing these in a forthcoming paper (link to be shared here)
Yes! The neutral white background is NOT ‘realistic’
Yes! It is a red herring fallacy to suggest that this study somehow indicates I believe in binarized race classification or similarly garbage ideas like binarized gender.
Nope. This study was not meant to assign nor assigned a clean chit to nobody’s algos
Nope. Black & brown faces are not ‘corner cases’ and that type of phraseology is mighty troublesome.

Have a nice thoughtful week ahead, and remember:

Fcuk the algorithm.

Always!

Addition: The second installment of this blog series can be found here: https://medium.com/@VinayPrabhu/scrutinizing-saliency-based-image-cropping-6b7a70cfb4f1

Disentangling disentanglement in Deep Learning

Mon, 23 Dec 2019 21:06:02 +0000

Disentangling disentanglement: Footnotes from NEURIPS — 2019

Credit: ‘ Striving for Disentanglement’ by Simon Greig — https://www.flickr.com/photos/xrrr/500039281/

TL;DR : Disentangling disentanglement. Via this blog-post, I intend to try and summarize all of the dozen papers presented on disentanglement in deep learning in this year’s NEURIPS-2019 Vancouver.

Companion Github repo replete with paper summaries and cheat sheets: https://github.com/vinayprabhu/Disentanglement_NEURIPS_2019/

Background: Disentanglement in Representation learning

On Thursday evening of the conference week, as I sauntered around the poster session in the massive east exhibition halls of the Vancouver convention center, I realized that I had chanced upon probably the 5th poster in the past couple of days entailing analysis of a disentanglement framework the authors had worked on.

Fig 1: (Yet another) Poster on disentanglement at this year’s NEURIPS

A quick check in the proceedings led me to this stunning statistic: A total of I-KID-YOU-NOT dozen papers were accepted this year with the term ‘DISENTANGLEMENT ’ in the title. There were at least a few more that I chanced upon in the multitude of workshops. (There were 20+ papers and talks during the 2017 NEURIPS workshop on Learning Disentangled Representations: from Perception to Control — https://sites.google.com/view/disentanglenips2017 and we had a challenge workshop this year as well: https://www.aicrowd.com/challenges/neurips-2019-disentanglement-challenge)

I had first encountered this flavor of usage of the term in statistical learning during the last stages of my doctoral journey at CMU (circa 2013) when I read ‘Deep Learning of Representations: Looking Forward’ by Yoshua Bengio in which he emphasized the need to be ‘.. learning to disentangle the factors of variation underlying the observed data’.(How I wish he still authored such single author papers)

As it turns out, much to the chagrin of the physicists perhaps, if you are working on teasing out visual style from digit type on MNIST, or separating shape and pose in images of human bodies and facial features from facial shape on CelebA or grappling with unwrapping the effects of mixture ratio of the two constituent compounds and environmental factors such as thermal fluctuation in images generated for microstructure growth, you are disentangling.

There seems to be no consensus on what the term precisely means or what metric(s) capture the extent of it, an observation that is confirmed by this rather funny/snarky slide in Stafano Soatto’s talk at IPAM (refer to the playlist below)

Fig 2: Invariance and disentanglement in deep representations

That said, this is not a case of there existing a mere smattering of empirical experiments that all use their own customized notion of disentanglement. In fact, reasonably rigorous frameworks have been proposed harnessing powerful tools from areas such as Variational inference, Shannonian Information theory, Group theory and matrix factorization. Deepmind’s group theoretic treatment of the same seems to have perched itself as one of the go-to frameworks. In case you are looking for a succinct 3 min recap of what this is, please refer to this video that I saw during one of Simons Institute workshops (around the 7th minute). (A very detailed talk from one of the main authors of the Deepmind group can be found here)

Fig 3: Group theoretic framework for disentanglement

A bird’s view of the papers presented

In Fig 4 below, is a bird’s-eye view of the 12 papers presented. I roughly bucketized them into two subsections depending on whether the main perceived goal of the paper (from my humble viewpoint) was to either analyze and/or critique the properties of a pre-existing framework or to harness one and apply the same to an interesting problem domain. Bear in mind that this is admittedly a rather simplistic categorization and this is not very instructive of whether the applications oriented papers did or did not critique and analyze the frameworks used or that the analysis/critique papers did not include real-world applications.

Fig 4: Disentanglement papers categorization (NEURIPS -2019)

(You can find the pdf version with the paper links here: https://github.com/vinayprabhu/Disentanglement_NEURIPS_2019/blob/master/Disentanglement_papers_tree-diagram.pdf )

What do they mean by disentanglement?

In order to summarize the contexts in which disentanglement was used in these papers, I created a look-up-table (See Table-1). In those cases where the authors explicitly did not have a subsection dedicated to defining the same, I improvised and extracted the gist (and hence the caveat [improv]).

Table-1(a) Disentanglement context in the application papersTable-1(b) Disentanglement context in the analysis papers

Reproducibility and open-sourced code:

Given the strong growing trend towards open sourcing the code used to produce the results, 10 of the 12 author-groups shared their github repos as well. This is captured in Table-2 below:

Table-2: Papers and the open-source code links

What now? Some ideas..

[Here are some scribbles to try and guilt myself into working on this more seriously. Please take these with a grain of salt or 12 :) ]

1: Survey paper detailing the definitions, frameworks and metrics to be used.

2: Disentangling author/writing style/nation of origin using Kannada-MNIST dataset. (65 native volunteers from India and 10 non-native volunteers from USA)

https://github.com/vinayprabhu/Kannada_MNIST

3: It’s somewhat surprising that no one’s tried throwing a K user interference channel model for entanglement and see if an Interference Alignment [https://arxiv.org/pdf/0707.0323.pdf ] like trick works for Dsprites-like datatsets

4: Disentangling Shoe type, pocket and device location from Gait representations

5: Bridging the body of work pertaining to (Hyperspectral) Unmixing / Blind source separation and disentangled representation learning.

Resource list:

Companion github repo replete with paper summaries and cheat sheets.

vinayprabhu/Disentanglement_NEURIPS_2019
TL;DR - On disentangling the menagerie of disentanglement papers During this NEURIPS(2019), I encountered a glut of… github.com

A. Datasets to get started with:

[1] https://www.github.com/cianeastwood/qedr

[2] https://github.com/deepmind/dsprites-dataset

[3] https://github.com/rr-learning/disentanglement_dataset

(Major props to the NeurIPS 2019 : Disentanglement Challenge organizers for the resources they shared as well! )

Link: https://www.aicrowd.com/challenges/neurips-2019-disentanglement-challenge

google-research/disentanglement_lib
You can’t perform that action at this time. You signed in with another tab or window. You signed out in another tab or… github.com

B. Video playlist:

[1] Y. Bengio’s: From Deep Learning of Disentangled Representations to Higher-level Cognition

https://www.youtube.com/watch?v=Yr1mOzC93xs&t=355s

[2] \beta-VAE (Deepmind): https://www.youtube.com/watch?v=XNGo9xqpgMo

[3] Flexibly Fair Representation Learning by Disentanglement: https://www.youtube.com/watch?v=nlilKO1AvVs&t=27s

[4] Disentangled Representation Learning GAN for Pose-Invariant Face Recognition: https://www.youtube.com/watch?v=IjsBTZqCu-I

[5] Invariance and disentanglement in deep representations (Fun talk), https://www.youtube.com/watch?v=zbg49SMP5kY

(From NEURIPS 2019 authors)
[1] The Audit Model Predictions paper: https://www.youtube.com/watch?v=PeZIo0Q_GwE

[2] Twiml interview of Olivier Bachem (3 papers on this topic at NEURIPS-19): https://www.youtube.com/watch?v=Gd1nL3WKucY

C. Cheat sheets

Cheat sheet-1: All the abstracts! (Print on A3/2)Cheat sheet-2: All the essences!

Build your own custom hotword detector with zero training data and $0!

Mon, 19 Aug 2019 14:07:30 +0000

Build your own custom hotword detector with zero training data and $0!

TLDR : Google TTS - > Noise augment -> {wav files} ->SnowBoy ->{.pmdl models} -> Raspberry Pi

OK, so it’s that time of the year again. You know there’s that thing in the desert. Last time around, I rigged up a Google AIY vision kit and added espeakon Chip and Terra , the art installations of the motley bunch that is BŸTE: Burners for Ÿntelligent Technology Emancipation.

The result was this:

This time around, I decided to add an extra sensory ability: The ability to listen and respond, which in Machine Learning plain-speak translates to rigging up a hotword detection engine that’ll work on-device and offline. Easy peezy no? Hand me the beer perhaps? Read on padawan …

So, I began scouting around for simple off-the-shelf solutions and I chanced upon the awesome SnowBoy __ off-line hotword detector. This came with constraints of course! You could potentially download machine learning models pre-trained to detection specific popular hotwords such as Alexa and Jarvis(See pic below)…

The off-the-shelf available hotword models

.. but in order to truly build your own robust model for the precise custom hotword, you need ~ 500 volunteers contributing 3 samples each. In spite of dipping into my gloriously dirt-poor social media reach, I was able to muster a grand total of ~ 5 donors :’(

5 donors from my social media campaign!

Seeing this go nowhere, I thought of generating my own dataset. I had recently worked on a couple of synthetic-to-real word transfer learning projects one of which I published at Deep-generative-model workshop at ICLR (See https://arxiv.org/abs/1905.08633), and thought that if WaveNet is indeed so impressive at generating realistic sounding text-to-speech, I could dip into that sweet $300 free Google cloud credits that Goggle doles out to do the data collection for me and transfer learn into the real world with some nifty noise augmentation and yes, unreasonable reasonableness of a deep neural network’s ability to generalize!

Phase-1: Generating the synthetic hotword audio files in different voices using Google TTS

So, I created a temporary GC account, religiously followed the documentation and scribbled some Python code ( Shared here: https://github.com/vinayprabhu/BurningMan2019/blob/master/generate_SYNTHETIC_audiofiles.py ). In about 5 min, I had 189 .wav files of the hot-word I was targeting (which was Hey Chip! BTW) in different accents, or more formally as voices. You can download this entire treasure trove from here.

Using Google TTS to generate synthetic training data

Some of the example sounds that were my favorite were these:

Example sound files from the Google TTS engine!

Now that we had these 189 .wav files for the different voices, for each of these voices, I performed plain-vanilla additive white Gaussian noise augmentation to get (189 x 3) wav files. Here is the colab notebook associated with this task.

Phase-2: Training the hot-word detection models using the synthetic noise-augmented wav files

The snowboy technology, as promising as it is, is still is in it’s nascency. The api for training your own models programmatically looks rather restrictive:

python training_service**.** py 1.wav 2.wav 3.wav saved_model**.** pmdl

As seen, you need to feed in precisely 3 wav files to spit-out a model. So, yes. I did generate 189 models per voice (I’d be very very glad to be proven wrong in this regard) and logical ‘OR’ed them together. The colab notebook that ingests the wav files and trains the ML models is shared here: https://github.com/vinayprabhu/BurningMan2019/blob/master/Colab_Notebooks/model_gen.ipynb

Phase-3: Combining together all the models and running them on the Raspberry Pi

OK. So, this phase was kinda tricky. Make sure that you follow this repo’s documentation rather patiently and religiously:

wanleg/snowboyPi
_Start with a fresh install of Raspbian (Lite or Regular, this guide assumes Lite) sudo apt update && sudo apt -y…_github.com

One main source of irritation with audio-projects on Raspberry Pi are the bedeviled ALSA shenanigans, and the ensuing fight between the HDMI, USB-audio and local-audio-out ports for the audio-dominion. In order to circumvent that, I used the good ol’ SAMSON mike in + audio-out rig (Apparently Amazon peddles these at $29.99 now! They are much cheaper on SP road, Bangalore.)

The entire setup looks like this:

The raspberry pi-3 set-up with the Samson mike

Before beginning the, ahem, human trials, I tried to isolate out the effects of the mike being used by means of a simple test case where I played the Google TTS output audio file on a laptop and checked if the audio-file’s pertaining .pmdl running on the Raspberry Pi would indeed get triggered by the synthetic utterance. The result was gloriously good!

Synthetic data input trials

Now that there was some hope, I began tweaking the snowboy.py script (from here: https://github.com/wanleg/snowboyPi/blob/master/snowboy.py) to include all the .pmdl model files I had just generated so that at least one would get triggered when a real world homo sapien would utter the key words ‘Hey chip!’. It turns out, all you need to do is to add the list of models on line#29 here: https://github.com/wanleg/snowboyPi/blob/master/snowboy.py#L29

FINALE:

Now that all the files are packed in (you never have to worry about the size of these individual DNN models by the way. There are all ~ 10KB each. Impressive work SnowBoy people !), I finally decided to do the final real world test with real human voice inputs and the result was .. *drumrolls*.

Ta..Da! It works! It works rather seamlessly with my normal (Indian accented) voice and my Adam Levine voice.

Phase-4: Deployment on Chip and Terra + casing + playa deployment

I am awaiting for this phase with bated breath. I’ll update this blogpost as soon I get back to the default world!

A new handwritten digits dataset in ML town: Kannada-MNIST

Mon, 12 Aug 2019 07:02:31 +0000

Kannada-MNIST:A new handwritten digits dataset in ML town

Class-wise mean images of the 10 handwritten digits in the Kannada MNIST dataset

TLDR:

I am disseminating 2 datasets:
Kannada-MNIST dataset : 28X 28 grayscale images: 60k Train | 10k Test
** Dig -MNIST:** 28X 28 grayscale images: 10240 (1024x10) {See pic below}

Putting the ‘Dig’ in Dig-MNIST

The Kannada-MNIST dataset is meant to be a drop-in replacement for the MNIST dataset 🙏 , albeit for the numeral symbols in the Kannada language .
Also, I am disseminating an additional dataset of 10k handwritten digits in the same language (predominantly by the non-native users of the language) called Dig-MNIST that can be used as an additional test set.

Resource-list:

GitHub 👉: https://github.com/vinayprabhu/Kannada_MNIST
Kaggle 👉: https://www.kaggle.com/higgstachyon/kannada-mnist
ArXiv 👉 : https://arxiv.org/pdf/1908.01242.pdf

If you use Kannada-MNIST in a peer reviewed paper, we would appreciate referencing it as:

Prabhu, Vinay Uday. “Kannada-MNIST: A new handwritten digits dataset for the Kannada language.” arXiv preprint arXiv:1908.01242 (2019)..

Bibtex entry:

@article{prabhu2019kannada,  
  title={Kannada-MNIST: A new handwritten digits dataset for the Kannada language},  
  author={Prabhu, Vinay Uday},  
  journal={arXiv preprint arXiv:1908.01242},  
  year={2019}  
}

Introduction:

Kannada is the official and administrative language of the state of Karnataka in India with nearly 60 million speakers worldwide. Also, as per articles 344(1) and 351 of the Indian Constitution, Kannada holds the status of being one of the 22 scheduled languages of India . The language is written using the official Kannada script, which is an abugida of the Brahmic family and traces its origins to the Kadamba script (325–550 AD).

Kannada stone inscriptions: Source: https://karnatakaitihasaacademy.org/karnataka-epigraphy/inscriptions/

Distinct glyphs are used to represent the numerals 0–9 in the language that appear distinct from the modern Hindu-Arabic numerals in vogue in much of the world today. Unlike some of the other archaic numeral-systems, these numerals are very much used in day-to-day affairs in Karnataka, as in evinced by the prevalence of these glyphs on license-plates of vehicles captured in the pic below:

A vehicle license plate with Kannada numeral glyphsMNIST-ized renderings of the variations of the glyphs across the modern Kannada fonts

This figure here captures the MNIST-ized renderings of the variations of the glyphs across the following modern fonts: Kedage, Malige-i, Malige-n, Malige-b, Kedage-n, Malige-t, Kedage-t, Kedage-i, Lohit-Kannada, Sampige and Hubballi-Regular.

Dataset curation:

Kannada-MNIST:

65 volunteers were recruited in Bangalore, India, who were native speakers of the language as well as day-to-day users of the numeral script. Each volunteer filled out an A3 sheet containing a 32 × 40 grid. This yielded filled-out A3 sheets containing 128 instances of each number which we posit is large enough to capture most of the natural intra-volunteer variations of the glyph shapes. All of the sheets thus collected were scanned at 600 dots-per-inch resolution using the Konica Accurio-Press-C6085 scanner that yielded 65 4963 × 3509 png images.

Volunteers helping curate the Kannada-MNIST dataset

Dig-MNIST:

8 volunteers aged 20 to 40 were recruited to generate a 32 × 40 grid of Kannada numerals (akin to 2.1), all written with a black ink Z-Grip Series

Zebra Pen on a commercial Mead Cambridge Quad Writing Pad, 8–1/2” x 11”, Quad Ruled, White, 80 Sheets/Pad book. We then scan the sheet(s) using a Dell — S3845cdn scanner with the following settings: • Output color: Grayscale • Original type: Text • Lighten/Darken: Darken+3 • Size: Auto-detect The reduced size of the sheets used for writing the digits (US-letter vis-a-vis A3) resulted in smaller scan (.tif) images that were all approximately 1600×2000.

Comparisons with MNIST:

1: Mean pixel-intensities distribution:

2: Morphological properties:

Code source: https://github.com/dccastro/Morpho-MNIST

3: PCA-analysis:

4: UMAP visualizations:

Some classification bench-marking:

I used a standard MNIST-CNN architecture to get some basic accuracy benchmarks (See fig below)

The CNN architecture used for the benchmarks

(a) Train on Kannada-MNIST train and test on Kannada-MNIST test

(b) Train on Kannada-MNIST train and test on Dig-MNIST

Open challenges to the machine learning community

We propose the following open challenges to the machine learning community at large.

To characterize the nature of catastrophic forgetting when a CNN pre-trained on MNIST is retrained with Kannada-MNIST. This is particularly interesting given the observation that the typographical glyphs for 3 and 7 in Kannada-MNIST hold uncanny resemblance with the glyph for 2 in MNIST.
Get a model trained on purely synthetic data generated using the fonts (as in [1]) and augmenting to achieve high accuracy of the Kannada-MNIST and Dig-MNIST datasets.
Replicate the procedure described in the paper across different languages/scripts, especially the Indic scripts.
With regards to the dig-MNIST dataset, we saw that some of the volunteers had transgressed the borders of the grid and hence some of the images either have only a partial slice of the glyph/stroke or have an appearance where it can be argued that they could potentially belong to either of two different classes. With regards to these images, it would be worthwhile to see if we can design a classifier that will allocate proportionate softmax masses to the candidate classes.
The main reason behind us sharing the raw scan images was to foster research into auto-segmentation algorithms that will parse the individual digit images from the grid, which might in turn lead to higher quality of images in the upgraded versions of the dataset.
Achieve MNIST-level accuracy by training on the Kannada-MNIST dataset and testing on the Dig-MNIST dataset without resorting to image pre-processing.

[1] Prabhu, Vinay Uday, Sanghyun Han, Dian Ang Yap, Mihail Douhaniaris, Preethi Seshadri, and John Whaley. “Fonts-2-Handwriting: A Seed-Augment-Train framework for universal digit classification.” arXiv preprint arXiv:1905.08633 (2019). [ https://arxiv.org/abs/1905.08633 ]

Cats to crazy quilts: Using style transfer to generate adversarial examples

Wed, 20 Jun 2018 19:21:03 +0000

Cats to crazy quilts: Using style transfer to generate adversarial examples

Prelude:

Let’s begin with a simple introduction into the world of adversarial inputs. These are inputs into a machine learning classifier that have been shrewdly perturbed in such a way that these changes are near damn invisible to the naked eye but can fool the machine learning classifier into predicting either a arbitrary wrong class (Un-targeted) or a specific wrong class (targeted).

There are two defining images that come to my mind when I think of this field at large. The first one is the classic Panda-to-Nematode image from here.

The now iconic example of a panda’s image getting perturbed into a gibbon (Source: https://arxiv.org/pdf/1412.6572.pdf )

The second one, is this one below that provides a geometrical perspective on where these adversarial inputs actually reside.

An image that provides a geometrical perspective on the adversarial inputs (Source: https://infoscience.epfl.ch/record/229872/files/spm_preprint.pdf )

Where I work, harnessing adversarial examples in a non-computer vision setting for dataset augmentation (to increase both robustness and generalizatibity) forms a key part of our pipeline. In this regard, we have disseminated a few humble attempts such as Vulnerability of deep learning-based gait biometric recognition to adversarial perturbations ,On grey-box adversarial attacks and transfer learning and On Lyapunov exponents and adversarial perturbations.

Recently while dabbling with the idea of using interpolated style transfer to generate mutually adversarial pairs of images, I chanced upon this fuzziness surrounding one of the more fundamental questions of machine learning: What does constitute a true label and how do machine learning companies offering commercial off-the-shelf (OTS) APIs define the same?

TLDR:

1: We describe an experiment that entailed using style transferred images to target mis-classification in the context of a specific popular commercial off-the-shelf (OTS) API (I use the Watson Visual-Recognition- V3 API, version 2016–05–20 API for all the results shown here.)

2: The style transferred images achieved adversarial attack success rates of 97:5 % (195 out of 200).

3: The goal is not to proclaim a new blackbox attack recipe or to berate the commercial API used, but to merely highlight the fuzzing surrounding what constitutes a true label or a true tag. This is one account of the simple observation that while using interpolated style transfer as a method for generating mutually adversarial pairs, the ’ raw image ’ that is adversarially perturbed is not necessarily a naturally occurring image and is a style-transferred image itself.

4: Pitch the idea of using interpolated style transfer as a recipe of generating mutually adversarial pairs that can be used for model regularization as well as generating challenging co-class images as inputs into training pipelines for Siamese-net like embedding deepnets trained on triplet-loss cost functions.

5: Pitch the idea of using the interpolated weight as the new semantic epsilon in here:

Time for a new semantic epsilon?

The Deep-dive:

With this prelude in tow, the deep dive now begins.

Let’s start by focusing on the figure below:

Cat2Fabric: The journey of a cat’s image into a pattern

What we see is the journey of the image of a cat getting style-transferred into a ‘pattern-style-image’ using the arbitrary image stylization [2] Magenta __ project for different interpolation weights monotonically increasing from 0 to 1 (from the left to the right). As seen, with the raw image (interpolation weight (w=0)) or style-transferred images with low interpolation weights (up until interpolation weight w=0.1) as inputs, the commercial OTS classification API has, as expected correctly classified the image as a cat with high confidence scores (0.97 to 0.99). When we increase the interpolation weight slightly to w=0.15 , we see a dramatic change in the inferred label landscape. The top guessed classes dramatically change from feline, cat and carnivore to cellophane, moth and invertebrate.
While the two images are virtually indistinguishable for the naked eye and are merely 0.03 apart in terms of the structural similarity distance (which is 1-structural similarity index [4]) (0.125 apart in terms of the infinity-norm distance), the labels assigned for the two images by the black-box classifier turn out to be wildly different.
Thus, we refer to this pair as constituting a mutually adversarial pair with regards to the black-box classifier and the distance metric used. The local texture based features that the classifier might have learned, has perhaps coaxed it into making erroneous classification, while the image still clearly looks like that of cat. Now emerges a natural query whether the artistically style transferred synthetically generated image (with w=0.1) deserved to be classified as a cat in the first place. This is akin to another related question of what is the normative expected class when the input is a real world figurine rather than an animate being, which brings us to the figure below.

Is this a ‘Cat’ or a ‘Cat-figurine’?

Here, we see the input image ( The image was sourced from here ). We find this specific shopping portal to be an especially good source of such figurine art examples.
literally being that of an artistic cat figurine that results in a high confidence classification of being categorized a cat with high confidence score (0.89).

Specific of the experimentation procedure:
It is indeed legitimate to ask if the cat example discussed above was idiosyncratically chosen. In order to assuage those concerns, we did the following experiment.
The main querying point behind the experiment was as follows:
Is it indeed the case that images that are style transferred with a global low interpolation weight do result in mis-classifications? For this, we extracted 200 randomly chosen cat images from the Kaggle Dogs and Cats dataset. We resized all of them to size 299 x 299 and style transferred each one of them using the same style image extracted from the DTD dataset[1] using the style transfer algorithm detailed in [2]. The figure below showcases this with a specific example.

The concept

In order to ensure that the images still looked ‘cat-like’ the interpolation weight was set to a low value of 0.125.
One can sift through all the raw images and the style transferred images as a gif animation here below.

Gif of true images and their style transferred counterparts

Now, both the raw images and the style transferred images were classified using the Watson Visual Recognition- V3 API, version 2016–05–20 API.
The Accept-Language header string that sets the language of the output class names was set to en.
The owners query array was set to the default option (IBM).
The classifier-ids was set to default that required no training and would Return classes from thousands of general tags. The threshold query parameter that represents the minimum score a class must have to be returned was set to 0.5.
The results are covered in the forthcoming section.
Results:

Histogram of the top inferred labels

In the figure above, we see the counts of the most probable classes that the API returned. As seen, the top 4 classes that encompassed more than 50% of the test images were crazy quilt, camouflage, mosaic and patchwork.

In the figure below, we see the scores as well as the histogram of scores related to the 200 classification trials.

Scores and histogram of scores returned by the Watson classifier for the 200 test images

As seen, we have an overwhelmingly large number of cases where the mis-classifications were made with high confidence scores associated. In the figure below, we see the 5 images that the API classified correctly.

The lucky 5: Correctly classified as ‘Cat’ by Watson

Now, in this figure, we see randomly chosen 6 examples of style transferred images that were classified incorrectly.

6 random not-so luckies

Conclusion and Future Work
Due to limitations of API usage for free-tier users, we could not extend the experiment for larger datasets, which is our immediate goal. Besides this, another question that we would like to explore is the choice of the style image. We selected an image for the texture dataset on account of 2 reasons. The first being that a pre-trained style transfer model was readily available. The second reason was based on a hunch that texture, would be in fact be the right aspect of the image to perturb to induce a mis-classification.
As stated in the prelude, our intention is not to proclaim a new black-box attack or to berate the commercial API.

Besides showcasing the potential of looking at style transfer as an adversarial example generating technique, we also wanted to draw attention to the inherent fuzziness that surrounds the definition of what constitutes an image class/category or ‘tags’ in the case of such APIs and what entails an image mis-classification.
The API that we used describes the technology as: Watson Visual Recognition’s category-specific models enable you to analyze images for scenes, objects, faces, colors, foods, and other content. With regards to the specific API documentation, it was stated that upon usage with Pre-trained models (in lieu of a custom trained classifier), the API Returns classes from thousands of general tags.
On the concluding note, we would like to remark that we also ascertained the efficacy of these style-transferred based black-box attacks using the universal adversarial images for different Deep-nets from [3] as the style image, the results of which we plan to disseminate in the full version of this work.

Links:

(This work will be presented at the CV-COPS workshop @ CVPR-2018)

Github: https://github.com/vinayprabhu/Art_Attack

Poster:

Poster for the paper

References

[1] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, pages 3606–3613. IEEE, 2014.

[2] G. Ghiasi, H. Lee, M. Kudlur, V. Dumoulin, and J. Shlens. Exploring the structure of a real-time, arbitrary neural artistic stylization network. arXiv preprint arXiv:1705.06830, 2017.

[3] S.-M. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, and P. Frossard. Universal adversarial perturbations. https://arxiv.org/abs/1610.08401

[4] Z.Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600– 612, 2004.

Align Chronicles

Gas, Whey & Lobsters and Indian-Homework

Gas, Whey & Lobsters and Indian-Homework

Whey, gas and lobsters

India’s silent crime: Throwing away the homework treasure-trove

A peek into how SOTA AI/LLMs are trained

The roadblocks and the concerns

The devil is in the details: Imagining a typology of implementation paradigms

a) Passive interventions:

1) Textbook2Chatbot, Short-form videos and teaching-aids: Flattening the playing-field and acad-tik-toks:

2) Curricula development:

c) Emphasizing the whyness of the non-technical facets of the course:

b) Active interventions

1) The 6 phase framework

Mindset propagation probing by simulating social networks of LLMs

Background: What led to this

The core idea of mindset propagation probing

Step-wise rendition of the underlying framework

References

Matilda, Mars and Markup: The curious case of Mrs. Agnes Zevens

1. Backdrop of backprop

2. Marginalia, “Sub-authorship” & Matilda effect

3. Last mile trivia

3a: The original MCP: Mars Colonization Project

3b: Markup aesthetics and Notational dissonance

3c: The miniscule computing giant that was IBM 704!

A study of “A Study of Face Obfuscation in ImageNet”

A study of “A Study of Face Obfuscation in ImageNet”

Background

Main Issues

Issue-1: The curious case of Face obfuscation

Issue-2: NSFW analysis

Issue-3: Human co-occurrence analysis

👹FAQs of the ‘Devil’s advocacy’ kind: Our humble tribute to the cult of “Both-sideism”

👹** 1: Why not just contact them in lieu of public grandstanding? Have you bothered to even contact the curators of the ImageNet dataset?**

** 👹2: Oh come on! Stop with the self-aggrandizing and self-loathing. AI royalty tend to receive hundreds of emails a day. May be your little emails slipped through the cracks perhaps?**

And lastly, here is the actual video of our zoom-face-to-face meeting 📹 https://youtu.be/hpA67iDxNGU

Concluding thoughts: The real issues

a) Erasure of black-women-scholarship:

b) Revisiting the horrors of ghost labor:

Scrutinizing Saliency Based Image Cropping

Scrutinizing Saliency Based Image Cropping

On the twitter cropping controversy: Critique, clarifications and comments

On the twitter cropping controversy: Critique, clarifications & comments- 1

Experiment details:

Feedback from twitter: The two camps of the Rashomon aisle

Unbiased algorithmic saliency cropping is a pipedream

Conclusion

Addition: The second installment of this blog series can be found here: https://medium.com/@VinayPrabhu/scrutinizing-saliency-based-image-cropping-6b7a70cfb4f1

Disentangling disentanglement in Deep Learning

Disentangling disentanglement: Footnotes from NEURIPS — 2019

Background: Disentanglement in Representation learning

A bird’s view of the papers presented

What do they mean by disentanglement?

Reproducibility and open-sourced code:

What now? Some ideas..

Resource list:

A. Datasets to get started with:

B. Video playlist:

C. Cheat sheets

Build your own custom hotword detector with zero training data and $0!

Build your own custom hotword detector with zero training data and $0!

Phase-1: Generating the synthetic hotword audio files in different voices using Google TTS

Phase-2: Training the hot-word detection models using the synthetic noise-augmented wav files

Phase-3: Combining together all the models and running them on the Raspberry Pi

A new handwritten digits dataset in ML town: Kannada-MNIST

Kannada-MNIST:A new handwritten digits dataset in ML town

TLDR:

Introduction:

Dataset curation:

Kannada-MNIST:

Dig-MNIST:

Comparisons with MNIST:

Some classification bench-marking:

(a) Train on Kannada-MNIST train and test on Kannada-MNIST test

(b) Train on Kannada-MNIST train and test on Dig-MNIST

Open challenges to the machine learning community

Cats to crazy quilts: Using style transfer to generate adversarial examples

Cats to crazy quilts: Using style transfer to generate adversarial examples

Prelude:

c) Emphasizing **the whyness of the non-technical facets of the course:**

👹** *1: Why not just contact them in lieu of public grandstanding? Have you bothered to even contact the curators of the ImageNet dataset?***

** 👹2: Oh come on! Stop with the self-aggrandizing and self-loathing. *AI royalty tend to receive hundreds of emails a day. May be your little emails slipped through the cracks perhaps?***