[Woocommerce] Woocommerce variable products problem

I have added variable products to my Woocommerce store. Some products are only available in one color per size despite having many colors in smaller sizes.

For example this product:

The product is available in four sizes:
3-6 (10 colors)
6-12 (10 colors)
12-18 (10 colors)
18-24 (only in black)
So total of 31 variable products.

The biggest size is only available in black and I have deleted all the other colors from this size. Despite that when I select "Black" in the front end, I still see the other colors as well. This is bad for the customer and would like to fix this.

How can I hide the non existing color/size combinations from front end?

Thanks in advance!