Video synthesis with Pure Data/Gem

bananeurysm · Post by **bananeurysm** » Tue Apr 10, 2012 2:58 pm

I just fired up PureData/Gem the other day and had a lot of fun with simple stuff. (My first time using GEM objects.)

Anyone else playing with this stuff? I was able to get pretty interesting results (imo) in under an hour.

I'd love to hear any tips/tricks ppl might have for this stuff!

<edit>
change the attachment extension to .pd or use this link which I've updated as of 8.9.13
http://www.anestheticaudio.com/video_synth.pd

bananeurysm · Post by **bananeurysm** » Tue Apr 10, 2012 3:24 pm

I guess I should add.. if you've never used pure-data, you can easily (and freely) install it from this link: http://puredata.info/downloads/pd-extended

Then my patch works as a nice free standalone video synth if ya like.

(No pd knowledge is required to run/use it)

lizlarsen · Post by **lizlarsen** » Tue Apr 10, 2012 3:31 pm

Nice! I'll try to check out the patch, but you should post a quick clip showing what it looks like if you have time.

bananeurysm · Post by **bananeurysm** » Tue Apr 10, 2012 5:15 pm

Ok - I made a video. It's pretty crappy and low res. It looks WAY better if you run the patch. But here's a preview anyway

<edit: somehow the framerate got messed up too.. it should be smooth, not choppy)>

lizlarsen · Post by **lizlarsen** » Tue Apr 10, 2012 6:32 pm

Cool! I tried out the patch a bit earlier too! I hope you continue to develop it. Next step would be creating some sort of matrix mixer between the VCOs and the RGB.

Matos · Post by **Matos** » Tue Apr 10, 2012 7:40 pm

Super tornado! Thanks for this. Never really explored pd but this is very cool. Looking at the patch trying to understand what's going on. Any simple way to resize the video output to full screen?

barto · Post by **barto** » Wed Apr 11, 2012 11:28 am

how long before theres a video equivalent of Reason and VST's?

lizlarsen · Post by **lizlarsen** » Wed Apr 11, 2012 11:35 am

Well, there are many different tools to do things like that already... PureData, as seen here, Max/Jitter, VVVV, Processing, OpenFrameworks, etc...
I guess the real question is if "patchable modular video synth simulator" patches will become more popular.

there are already some for Rutt-Etra, but I don't think they are patchable.

VVVV allows you to have a modular environment between graphical nodes/video and control elements, that is the only one I've messed around with much, and it was a few years ago.

Modular realtime video requires more processing power than audio, is one thing.

daverj · Post by **daverj** » Wed Apr 11, 2012 11:52 am

Jitter, GEM, and various VJ programs are already the equivalent of Reason. Programmers create "objects" for some of those programs, which are the equivalent of VSTs.

As Lars says, the processing power is the issue. With 1000 times the data rate, it takes 1000 times the processing power to do video vs audio. When you go through more than a couple of processing functions you start having to give something up, like resolution or frame rate. Or you need to start throwing more computers at the problem.

bananeurysm · Post by **bananeurysm** » Wed Apr 11, 2012 6:43 pm

creatorlars wrote:Cool! I tried out the patch a bit earlier too! I hope you continue to develop it. Next step would be creating some sort of matrix mixer between the VCOs and the RGB.

Good idea! I should also throw some filters in maybe. It's fun stuff - my first foray into any type of video anything.

@Matos - You can resize the video window is to send a message with your desired dimensions to the |gemwin| object. That object is inside the |pd gemwin| subpatch. I don't know if you know pd.. but if you cmd(or cntrl) click on the |pd gemwin| object, you can open it up. Then it's a matter of creating a message box containing this message: dimen 1024 768. (Or whatever dimensions you're after.)

Really I should just put some nice, easy to use resolution buttons in my patch.

Cool that there is some interest. Mostly I just wanted to play around with some simple stuff to create visuals for an experimental dance/multimedia project I'm working on. (My main stuff is audio.)

bananeurysm · Post by **bananeurysm** » Wed Apr 11, 2012 6:50 pm

creatorlars wrote: I guess the real question is if "patchable modular video synth simulator" patches will become more popular. there are already some for Rutt-Etra, but I don't think they are patchable.

It would be fairly straightforward to make make video module-like abstractions in pd, and setup a live patchable environment that way. I'm somewhat interested in putting something like that together, maybe I'll give it a shot.

What Rutt-Etra simulators are you referring to? (Like I said, I'm brand new to video)

radams · Post by **radams** » Wed Apr 11, 2012 7:31 pm

you can get some max4live tools here http://zealousy.com/2011/01/vizzable/
at Bob Zeal's site - he created these
I've only dabbled a bit but they seem to work pretty well

Matos · Post by **Matos** » Wed Apr 11, 2012 11:04 pm

bananeurysm
Thanks! I'm just dabbling in processing, and know nothing of pd, but I'll try it out as see if I can get full screen to send to the lzx for further fun. This patch is great. Keep it up.

There's an awesome rutt patch for quartz composer.

bananeurysm · Post by **bananeurysm** » Fri Apr 13, 2012 5:57 pm

Sweet - I will!

I'll post an update when I get around to sitting down with this again. Maybe modularize it, and certainly I'll put some resolution buttons in.

bananeurysm · Post by **bananeurysm** » Wed Apr 25, 2012 1:59 pm

I've been using that little video synth I made in pd for some improvisations/rehearsals with a dancer I'm collaborating with. I figured I'd share in case anyone is interested:

http://vimeo.com/album/1905584

The sound is also improvised with pd.

Matos · Post by **Matos** » Wed Apr 25, 2012 10:19 pm

Looks and sound great. Is their a specific concept or theme for the piece?

bananeurysm · Post by **bananeurysm** » Thu Apr 26, 2012 1:20 pm

Thanks!

There are some themes and concepts we are playing with - but it's all in very early improvisatory stages. Nothing is set enough to really talk about yet.

bananeurysm · Post by **bananeurysm** » Thu Jun 21, 2012 12:39 pm

We're performing 12 minutes of the piece we've been working on in Austin tomorrow night, and the next night if anyone's interested! We're part of the big range dance festival. http://www.spankdance.com/festival.html

Come on out if you're in the neighborhood.

Ciciliani · Post by **Ciciliani** » Sun Jun 24, 2012 3:13 am

Isadora (http://troikatronix.com/) should be mentioned in this context as well.
Unfortunately it is not free but it has an easier learning curve than PD/GEM or Max/MSP/Jitter, it's very powerful and reliable. A few years ago I did a workshop on Isadora and in under an hour I was able to make my own patches.

Also a few years back I experimented with PD/GEM but found it very unstable. I heard that nowadays it's pretty crash-free, though. Can anybody confirm this?

By the way, has anybody experience in how Jitter and Processing compare in terms of efficiency? The latter really frustrated me when it came to playing back .mov files. I wonder if Jitter handles this better...

I can't make up my mind what video software to go for. I like code, therefore I was drwn to Processing until I tried to play back .mov files. And I'm intimidated by Open Frameworks as I don't know C++....

maxime.annah · Post by **maxime.annah** » Sun Jun 24, 2012 4:22 pm

continuum · Post by **continuum** » Mon Jul 30, 2012 2:46 pm

Ciciliani wrote:Also a few years back I experimented with PD/GEM but found it very unstable. I heard that nowadays it's pretty crash-free, though. Can anybody confirm this?

I'm one of the GEM developers and made the GEM code pretty stable on OSX for running HD video installations, one of which is in the National Portrait Gallery permanent collection. I'd say it's pretty stable.

continuum · Post by **continuum** » Mon Jul 30, 2012 3:09 pm

daverj wrote:Jitter, GEM, and various VJ programs are already the equivalent of Reason. Programmers create "objects" for some of those programs, which are the equivalent of VSTs.

As Lars says, the processing power is the issue. With 1000 times the data rate, it takes 1000 times the processing power to do video vs audio. When you go through more than a couple of processing functions you start having to give something up, like resolution or frame rate. Or you need to start throwing more computers at the problem.

Modern PCs with good GPUs can do a ton of video processing. In 2007, I made a Pd/GEM based video piece that used a Quad G5 which captured 1080i60 video, displayed it on screen, compressed it to disk and played back 4 other 1080i60 videos at the same time. There was contrast, saturation and brightness on each HD layer plus compositing on the GPU shaders. The frame rate was about 60 fps, so there was a lot of processing power left. Systems are a lot more powerful now.

A current Nvidia or AMD/ATI GPU can crank through hundreds and even thousands of frames of HD resolution video in a second using shaders. The latest cards do TERAFLOPS and would have been the fastest supercomputers in the world in the 90s!

GEM, like Jitter and other apps, allows the running of GLSL shaders for maximum performance.

daverj · Post by **daverj** » Mon Jul 30, 2012 5:34 pm

That's cool. It sounds like the choice of graphics card is even more important than the choice of CPU for that kind of processing.

Speaking of GPUs, I know somebody has ported PD to the Raspberry Pi. I'm curious if there's any thoughts of working with the Pi GPU gurus to port GEM to it. I know they're kind of tight lipped about the GPU because of non-disclosure agreements with Broadcom, so it seems like there's only a few people involved in the Pi that can do the GPU porting for speeding up video programs there. At least that's the impression I get reading the forums over there.

continuum · Post by **continuum** » Mon Jul 30, 2012 8:19 pm

The Pi has some issues although it does run a gimpy Pd right now. I hadn't heard about the GPU situation so that probably leave GEM out in the cold. I think running custom code and a RTOS would be better than a full OS, drivers, X11 and an app on top of it all anyway.

There are better platforms for video DSP or even GL in low power hardware. We can discuss putting some serious processing into the modular format if you want.

daverj · Post by **daverj** » Mon Jul 30, 2012 11:39 pm

I was thinking more about a very low cost platform for use in installation art. For modules I'm more into classic analog processing, of which I already have a backlog of about 20 modules under development.

MOD WIGGLER

MOD WIGGLER

Video synthesis with Pure Data/Gem

Video synthesis with Pure Data/Gem