The problem is that xLights is expecting the correct number of channels to be configured. In other words, it has to be configured in a multiple of three for this function in RGB to work properly. The first few times I used the test function in RPG, I simply defined an entire DMX universe and is would crap out with this error. Then yesterday, I was experimenting with a small string of 20 square nodes and in that case I only defind 60 channels. Then to my suprise, the function that would not work for me before suddenly worked perfectly. Then my little brain kicked in and I figured out what was probably stopping me before. As a test, I defined one more channel and like magic, they stopped working again with this error.
So yes, there is a bug in the software but for now you can get around it by making sure that the channels defined are in multiples of three.