How do Upfront Builder theme settings for colors work?

When you create a new theme from scratch, after specifying 10 colors, the upfront panel just shows 11 colors. Then it just keeps adding more and more colors. And we can't delete that colors, so after some time it gets messy. Can you explain how the colors panel works and is there a way to delete them?