software choices-time for me to make changes

[rrowan]

sorry my bad. Yes its Matt of xLights. I will blame by back pain meds 

I am concern that Joe of HLS (I hope that is correct) that he hinted he might change the file format because of larger files are taking too long with the xml format. I think it was in a video I was watching about folders but I am not sure.

Rick R.

[rwave]

I am also in that group looking for software tools for a 2013 show.  I spent 2012 getting hardware via coops (got lucky, participated in all coops that I wanted) and now want to start playing with software.

Unlike most, I have not used any sequencers/nutcracker so I have nothing to "unlearn".  Also, I have no nodes yet, but am interested in Ray's flat RGB.  I plan to use 100% RGB as I also have no legacy lights.

Being a slow learner, I don't have the time to "play" with all the choices, but instead need to pick one, bite the bullet and learn! 

I am leaning toward HLS and Sean's Nutcracker, but am open to anything.  Yes, I have conductors, but I'm sure a solution will exist when I get to it. Any additional thoughts?

Richard (from SoCal)   

[smeighan]

sorry my bad. Yes its Matt of xLights. I will blame by back pain meds 

I am concern that Joe of HLS (I hope that is correct) that he hinted he might change the file format because of larger files are taking too long with the xml format. I think it was in a video I was watching about folders but I am not sure.

Rick R.

XML is a problem, nutcracker and HLS hlsnc files are NOT xml, they are space delimited.

I do believe for large RGB displays we should not have the data as xml, or maybe do the hybrid xml like vixen vix files. Everything xml except for teh section that has the data.


Here is some measurements i made.

I was testing xLights to produce a vixen and a LOR file.

3 minute song, Wizards in Winter

Vixen had 30 nutcracker effects it needed to generate for the song.

xLights to generate song for vixen: 70 seconds to create the 58mbyte file
When xLights generated the exact same sequence for LOR as a lms file: 430 seconds and 697mbytes.

The reason LOR is 10 times bigger than the Vixen vix file is that Vixen 2.1 does not store the data as xml, it instead stores it as a big block of base64.

The save file as hlsnc was 40mbytes.

When creating the UserPatetrns.xml file for LSP this same sequence was 1.2 gbytes.

Now a xml file compressed through 7zip will reduce the file size down by 20 to 30x. You then need to unzip it so you can use it.

LSP msq files are just the xml file compressed along with a few other files.

[Gary]

I use Vixen 2.1.  ...  I'm too cheap to pay for anything.   

Did you steal your controllers and lights from the guy down the street?   

[kgustafson]

sorry my bad. Yes its Matt of xLights. I will blame by back pain meds 

I am concern that Joe of HLS (I hope that is correct) that he hinted he might change the file format because of larger files are taking too long with the xml format. I think it was in a video I was watching about folders but I am not sure.

Rick R.

XML is a problem, nutcracker and HLS hlsnc files are NOT xml, they are space delimited.

I do believe for large RGB displays we should not have the data as xml, or maybe do the hybrid xml like vixen vix files. Everything xml except for teh section that has the data.


Here is some measurements i made.

I was testing xLights to produce a vixen and a LOR file.

3 minute song, Wizards in Winter

Vixen had 30 nutcracker effects it needed to generate for the song.

xLights to generate song for vixen: 70 seconds to create the 58mbyte file
When xLights generated the exact same sequence for LOR as a lms file: 430 seconds and 697mbytes.

The reason LOR is 10 times bigger than the Vixen vix file is that Vixen 2.1 does not store the data as xml, it instead stores it as a big block of base64.

The save file as hlsnc was 40mbytes.

When creating the UserPatetrns.xml file for LSP this same sequence was 1.2 gbytes.

Now a xml file compressed through 7zip will reduce the file size down by 20 to 30x. You then need to unzip it so you can use it.

LSP msq files are just the xml file compressed along with a few other files.

I just used the name Rick used.  I guess I should have checked it (sorry Matt).

The most efficient storage would have 1 bytes for each 'cell' without compression (as each color value can be stored in a small byte of information  -- 8 bits).  The current HLS and NC formats use an ASCII representation of an integer value along with a space seperator.  This means that the storage space of each color value can be 3 bytes itself -- one byte per character -- along with the one space separator for NC means that a given 'cell' can be up to 4 bytes in length.  If we wanted to reduce the current size of the NC files by 1/4 (or so) we could make them hex files storing raw data.  However, this added space efficiency is probably not desired as it reduces readability of the NC file itself without using a translation program to convert from raw hex to readable text.

Kurt

[smeighan]

The most efficient storage would have 1 bytes for each 'cell' without compression (as each color value can be stored in a small byte of information  -- 8 bits).  The current HLS and NC formats use an ASCII representation of an integer value along with a space seperator.  This means that the storage space of each color value can be 3 bytes itself -- one byte per character -- along with the one space separator for NC means that a given 'cell' can be up to 4 bytes in length.  If we wanted to reduce the current size of the NC files by 1/4 (or so) we could make them hex files storing raw data.  However, this added space efficiency is probably not desired as it reduces readability of the NC file itself without using a translation program to convert from raw hex to readable text.

Kurt

I vote for the easier to read data, space delimited 24bot words. This also allows people to generate their own effects or use excel to generate a spreadsheet.

Ascii data is easier to make exchangeable between sequencers. base64 can store a 3 byte rgb in 4 ascii characters. This is why Vixen 2.1 format is pretty compact. The issue is it is harder for someone to generate base64 from excel, vb, and such.

I want to develop an intermediate exchange format that make sit easy for people to develop plugisn for nutcracker. I am hoping to get LSP to also import a nc file some day. This would make it really fast to import data and the file size would be in the 30-50mbytes instead of > 1 gbyte data.

[charles59]

Ascii data is easier to make exchangeable between sequencers. base64 can store a 3 byte rgb in 4 ascii characters. This is why Vixen 2.1 format is pretty compact. The issue is it is harder for someone to generate base64 from excel, vb, and such.

Not that it is probably a concern for this discussion, but although base64 is readily available as a library call for windows developer, it isn't on a Mac.  One has to code the translation themselves.  Not terribly difficult, but just another task one has to do. A another contributing factor perhaps along with ability to generate from excel, etc.

[smeighan]

i think space delimited 24bit numbers is fine. a 3 minute song with 12K channels is around 40mbytes. Since the xml files of LOR lms 700 mbytes and LSP 1.5gbytes are so large, 40mbytes is probably just fine,

I was just saying base64 was more space efficient. I have read thru the nutcracker ffiles when debugigng by viewing them in notepad++. Cant do that with base64.

I am working on a architecture spec of how to write effects for nutcracker. i want to expand the number of effects by making it easy for people to write it.

[Ron]

My bad on following Rick, sorry Matt, I should have known better.

Who the heck is Dave, Rick?

I'm constantly amazed at the work you guys put into Nutcracker.  Thank you many times over.

[rrowan]

My bad on following Rick, sorry Matt, I should have known better.

Who the heck is Dave, Rick?

I'm constantly amazed at the work you guys put into Nutcracker.  Thank you many times over.

There is a Dave for LSP or was?

Rick R.

[Ron]

There is a Dave for LSP or was?

Rick R.

A likely story.

[kgustafson]

i think space delimited 24bit numbers is fine. a 3 minute song with 12K channels is around 40mbytes. Since the xml files of LOR lms 700 mbytes and LSP 1.5gbytes are so large, 40mbytes is probably just fine,

I was just saying base64 was more space efficient. I have read thru the nutcracker ffiles when debugigng by viewing them in notepad++. Cant do that with base64.

I am working on a architecture spec of how to write effects for nutcracker. i want to expand the number of effects by making it easy for people to write it.

Good discussion.  I am not against the format as it stands and it is very readable and easy to grasp the structure.  Just exploring options to see if we can't squeeze efficiency somewhere. 

Kurt