This is working as designed. When a column color is set using the "Use value as color" property, the color applies to all columns to its right until another column is set using the property.
If the column is categorized:
The column that you want to set the text color for is a categorized column. In the Column Properties of your first color column (the one on the far left), if you choose the option to sort it as Ascending, this interferes with the sorting of the column.
To fix this, add another column before the first color column to display the same field as your categorized field and sort THAT column. See the screenshot below.