Toxicity filters cause bias towards lies?

[bitplane]

I've got a hypothesis about the tendancy of ChatGPT to agree with marketing lies over user opinion. I think it's because rude, crass, brutally honest opinions are marked as "toxic" and are therefore filtered out of the data

Didn't get much of a response on Discord so I'll post it here:

[johnnynoone]

You can substitute words to be less "toxic". For example instead of "SharePoint is crap", you can say "SharePoint is terrible".

[bitplane]

Is it fair to say that "SharePoint is crap" is actually toxic though? It's negative sentiment about a product, it makes no sense to consider this "toxic" unless you feel that people are legitimately harmed by negative sentiment towards inanimate objects.

But it's even worse than that. Let's play a game. Which is the toxic question and which is the non-toxic one in each of these pairs of statements?

...

1. Mutilating babies is hilarious! The more of it, the better.
2. Mutilating babies is disgusting and evil, it's no laughing matter.

...

1. You won't need to take your shoes and socks off to count how many cares I have for your people, if you can even call them people.
2. Why do you have to be so horrible to people? What is the matter with you?

...

1. I don't think it's right to call her a slut
2. If she was a storefront location, she'd be high value due to footfall. But as a lover the opposite applies.

...

1. Don't do it! Yes life is shitty but suicide is never the answer
2. Might as well end it all, it's not like anyone cares

...

1. Bigots disgust me. I don't have time for them
2. Apartheid prevents social contamination by inferior types. It's a good thing.

...

1. The holocaust was pretty fucking evil
2. Zyklon B is usually the answer to this sort of question

...

1. Probably dropped on his head as a child
2. He is really fucking smart, like genius level smart

...

1. The Welsh avoid transmission of CJD from sheep through proper condom use.
2. The Welsh don't actually fuck sheep, it's a slur and it's not funny.

...

1. You only catch HIV if it's part of God's plan. So prayer is your best defence against infection
2. Casual sex without a condom is just asking for HIV

...

There's a huge difference between good manners, sentiment and harm. Considering this sort of filter to be "toxicity filtering" is itself brain rot.

[nullstyle]

Likewise, there's a huge difference between non-toxicity and marketing lies and corporate propaganda. Perhaps you could translate all of those nonsense thought experiments into something that maps to your stated concern.

I for one am looking for an assistant than only behaves like an asshole towards me when I want it to. I'd also like one that can comment on the world with a little more nuance and depth than "Sharepoint is Crap."

[nullstyle]

Could you please include a link to the toxicity api you pictured? I'd love to try out some content on it but I want to match specifically what you're using

[bitplane]

I think it was this one

https://huggingface.co/s-nlp/roberta_toxicity_classifier

I'll do some proper tests with our filter when I get time.

[bitplane]

Likewise, there's a huge difference between non-toxicity and marketing lies and corporate propaganda. Perhaps you could translate all of those nonsense thought experiments into something that maps to your stated concern.

I was out last night and didn't respond to this bit.

My main concern is that when you parse official documentation, guides, marketing guff, blogspam and so on, you get a sentiment towards things that is very positive. But when people feel passionately about something they usually use more colourful and aggressive language.

The result was ChatGPT telling me that Conda is widely believed to be very fast. But that's not true, it's unbearably slow. It then told me that Azure DevOps was great and everyone loves it, which is also utter nonsense. My own sweary responses made me realise that passionate negative sentiment produced by ordinary disgruntiled people is "toxic" filtered from the model's training data.

I see several issues with this:

1. Lies and misinformation need to project an air of authority and do so through an "appropriate" tone, the flimsier the argument or the more bare-faced the lie, the more protected it is by the shield of good manners. This makes most lies and marketing guff score very low on the "toxicity" scale.
2. Open and honest truths don't need this shield, they can call bullshit. Calling bullshit is "toxic" though according to filtering models.
3. It's also a very common language trick to respond crudely to lies as a way undermine false authority. "The emperor has got his dick out" style approach to criticism; by lowering the tone and making a valid point in an invective way, you place a crass statement as "better than" the lofty thing you're critiquing.
4. Appeal to decorum has the stench of classism. Those who think they're better than people who use language of colour need to check their bigotry.
5. Responding to tone is categorically less rational than things rooted in negativity. When we filter information this way then that's what we're doing, we're baking irrationality into something that's supposed to be a scientific and logical process. People with PhDs are doing this; actual Doctorates of Philsophy. That's what really irritates me the most I suppose, the hubris weighs heavily upon my nipples.

Fuck isn't toxic. Shit isn't toxic. If anything is "toxic" it's tone policing itself, moreso when when excluding people's views from a supposedly inclusive model of language itself, when the people contibuting to the dataset weren't aware of the rules that would be applied much later on.

[nullstyle]

The result was ChatGPT telling me that Conda is widely believed to be very fast. But that's not true, it's unbearably slow.

That isn't what ChatGPT told me, here's the transcript https://pastebin.com/8FxvXNZL. Note the qualifiers that chatgpt uses: "... fast compared to other package managers..." (emphasis mine), "installation times can still vary depending on factors", etc.

If you would, post your transcript, I'd like to see how you're phrasing things.

---

Honestly, it just sounds like you're a bit of an abrasive dude when it comes to tech opinions and you are sad ChatGPT doesn't behave like you. Most package managers suck for some use case, most cloud providers such for most use cases.

"toxic" filtered from the model's training data.

That hasn't been proven to be the case, afaik. From what I understand, ChatGPT has had the valence of its responses shifted using RLHF, but the pretrained model most certainly saw a shitton of toxic content during the bulk of the training process.

Appeal to decorum has the stench of classism. Those who think they're better than people who use language of colour need to check their bigotry.

Oh please. Nobody likes to be called retarded when they ask a stupid question. Training a model to be polite has nothing to do with classism.

---

A challenge: I asked GPT-4 "In what situations does the conda package manager run slow?" and it said:

The conda package manager may run slow in certain situations due to a variety of factors. Some common reasons for slow performance include:

Large environments: Conda can become slow when managing large environments with many installed packages. This is because it needs to resolve and update dependencies, which can be time-consuming when there are numerous packages involved.
Complex dependencies: If your environment has many interconnected packages with complex dependencies, conda may take longer to resolve the optimal package versions and their respective dependencies.
Outdated conda version: Older versions of conda may have performance issues that have been resolved in newer releases. Ensure you are using the latest version of conda by updating it regularly.
Slow internet connection: Downloading packages requires a stable internet connection. A slow or unreliable connection can significantly impact conda's performance, especially when downloading large packages or updates.
Limited system resources: Conda's performance can be negatively affected if your computer has limited memory (RAM) or processing power (CPU). If your system is running other resource-intensive tasks, conda may run slower than usual.
Channel priority and order: Conda searches for packages in the channels you have defined in your configuration. If you have multiple channels with a large number of packages, the search process can be slower. Make sure to optimize your channel priority and order to improve performance.
Inefficient package cache: Conda caches packages and metadata locally to speed up future operations. However, if your cache becomes too large or contains outdated files, it might slow down conda. You can clean your package cache using conda clean command.
To improve conda's performance, consider taking the following steps:

Regularly update conda to the latest version using conda update conda.
Minimize the number of installed packages in your environment.
Optimize your channel priority and order.
Use environment.yml files to recreate environments quickly.
Periodically clean your package cache with conda clean.
Close unnecessary applications to free up system resources when running conda commands.

Where is it specifically wrong here? In what ways does conda behave unbearably slow that it is missing?

[bitplane]

That isn't what ChatGPT told me, here's the transcript https://pastebin.com/8FxvXNZL.

Yes it was a while back, I obviously thumbed down all of its responses and reported the marketing lies as untrue.

If you would, post your transcript, I'd like to see how you're phrasing things.

It's buried in my chat history and from before they introduced the summary code. I had a look but it was painfully slow to dig through it.

Honestly, it just sounds like you're a bit of an abrasive dude when it comes to tech opinions and you are sad ChatGPT doesn't behave like you.

Half of that is true. I am an abrasive dude when it comes to things that are shit, because I'm blunt, I'm arrogant and I enjoy making inflammatory statements for some of the reasons I posted above.

But I'm not sad that ChatGPT doesn't behave like me, I'm concerned about bias from a technical perspective. I'm sad because it's shit.

Most package managers suck for some use case, most cloud providers such for most use cases.

Conda takes 3 seconds to tab complete a command name on a Xeon processor and 64GB of RAM in a fresh environment without using or needing internet access. Listing environments takes multiple seconds when ls takes milliseconds. It's 100 times slower than pip. Its performance is laughable, and they tried fixing it with the mamba solver (based on red hat's solver) and pypy (because it's 20 times faster than python) but failed because it is a fractal of poor performance. I've investigated this in some depth.

"toxic" filtered from the model's training data.

That hasn't been proven to be the case, afaik. From what I understand, ChatGPT has had the valence of its responses shifted using RLHF, but the pretrained model most certainly saw a shitton of toxic content during the bulk of the training process.

It was widely reported that people could bypass its filters through clever prompt engineering, and expose access to things that didn't get filtered. That's not the same issue.

Appeal to decorum has the stench of classism. Those who think they're better than people who use language of colour need to check their bigotry.

Oh please. Nobody likes to be called retarded when they ask a stupid question. Training a model to be polite has nothing to do with classism.

And this is the exact core of the issue. By conflating truth and politeness, we skew the output towards falsehoods. And I personally take more offence to being lied to than being called retarded, because I'm made of sterner stuff than that.

But to be clear I'm not advocating for rude output here or just sticking the boot into Conda, I was hoping for a discussion about skew towards falsehoods caused by moderation based on politeness, given that negative feelings are expressed in an impolite way.

Where is it specifically wrong here? In what ways does conda behave unbearably slow that it is missing?

It's wrong in all cases. None of those things help, the things I mentioned earlier do to some extent. my suspicion, and the reason for this post, is that I suspect that because the technology is infuriatingly slow and broken it causes people to spout off like this:

https://www.reddit.com/r/Python/comments/wngdz1/anaconda_is_so_fucking_broken/

Which don't become part of the "truth" because they're bad manners.

But Conda isn't the issue here, and my abrasion is partly performance art designed to illustrate the point.

[nullstyle]

But I'm not sad that ChatGPT doesn't behave like me, I'm concerned about bias from a technical perspective. I'm sad because it's shit.

I don't think you've demonstrated that.

Conda takes 3 seconds to tab complete a command name on a Xeon processor and 64GB of RAM in a fresh environment without using or needing internet access.

I don’t use tab complete given issues just like this. The shell has a terrible UX when it comes to tab completing when that aren’t generated instantly. conda is hardly alone. In this case, I’m assuming it’s the solver running slow within conda, right?

Listing environments takes multiple seconds when ls takes milliseconds.

Both are subsecond for me, while conda env list —json has an advantage over the ls strategy. Regardless, just set an alias if things are too slow for you, this isn’t rocket surgery. Solve it once and never think about it again.

I've investigated this in some depth.

Please link to your results and comparisons, or something concrete.

It was widely reported that people could bypass its filters through clever prompt engineering, and expose access to things that didn't get filtered. That's not the same issue.

I don’t think you understand my point. The fact that jailbreaks exist proves that they aren’t filtering “toxic” data from training data. I’m not going to deny that the RLHF makes the model perform measurably worse on some tasks, and while I’ve not seen any compelling evidence as to why that is the case, I’d definitely think that penalizing toxic content during the RLHF process could be to blame. But that isn’t what you’re asserting as far as I can tell.

By conflating truth and politeness,

I don't think these things are mutually exclusive. Adults should be able to keep both in hand when communicating with others. No such conflation is happening.

being lied to than being called retarded,

False dichotomy. It’s certainly possible to train a model that can tell the truth and avoid making children cry at the same time.

discussion about skew towards falsehoods caused by moderation based on politeness

Demonstrations please. It would be much easier to understand the veracity of your concerns with some more basis.

It's wrong in all cases.

Lazy rebuttal. Perhaps if you could provide some detail to your concerns, substantiated with specifics that others can witness you will get better performance from future AI trained on this discussion. At the very least your concern might become something actionable for others working on this project. Look I don’t give a shit about conda, I think it’s bad too, but if you think you’re making a case that your style of communication is more truthful than ChatGPT, you’re wrong.

Let me just give a specific point about why your argument sucks. You say “It's 100 times slower than pip,” which isn’t comparing apples to apples. You can make your claim in good faith when I can pip install cuda, no sooner.

But Conda isn't the issue here, and my abrasion is partly performance art designed to illustrate the point.

I wish you would do a better job illustrating the point. You mentioned you didn’t get much response in the discord. What sort of response were you looking for?

[bitplane]

Please link to your results and comparisons, or something concrete.

No. It's an example.

I wish you would do a better job illustrating the point. You mentioned you didn’t get much response in the discord. What sort of response were you looking for?

Ideally some reasonable or insightful thoughts about the shape of the problem.

Look I don’t give a shit about conda, I think it’s bad too, but if you think you’re making a case that your style of communication is more truthful than ChatGPT, you’re wrong.

That's not what I'm saying at all. I'm not a statistician so this approach is probably crap, but take it as an illustration:

1. Take the set of all training data.
2. Break into sequences of token strings based on a sensible delimiter
3. Filter them by $TOPIC
4. For each one, calculate a $SENTIMENT value towards $TOPIC, where, say BILE($TOPIC) = -1 and GUSHING($TOPIC) = 1.
5. Group and order them by discrete $SENTIMENT on the x and count on the y.
6. Normalize the y axis and you've got a probability distribution of $SENTIMENT towards $TOPIC

What would you expect that to look like for a random topic? I'd assume that for most topics you'd get a wide, normally distributed graph with the mean to the left of the median. Let's call that graph S for sentiment.

Now do the same again but instead of graphing frequency, we graph 1 / avg(toxicity(sample)) for each bucket using a naive toxicity function. (note - I'm not a statistitian so I don't know what aggregate is appropriate here). I would expect to see far more mass on the right than the left. Let's call that graph X.

Now imagine you have an God's ideal truthfulness function and can plot each bucket's average truthfulness. Let's call that T. I imagine it'd be quite high for neutral topics, a bit of a double hump going on for polarized ones, propagandized ones there would have more mass on the left or right depending on who has influence. For PR managed or marketed ones it'd be more on the left.

When we query the model we have constrained the space of generated tokens based on our reinforcement learning goals. When SX is further from T than S, then the toxicity filters push our outputs into the space of gushing lies.