Processing + NyARToolkit + multiple marker tracking

For various reasons, I need to do multiple marker tracking in processing with NyARToolkit.  However, with the default NyAR4psg layer between these two, multiple marker tracking is downright hard, and when you get it working, it’s not quite what you expect. After a few days of Java hacking, during which I was very pleasantly surprised with eclipse, I am now pleased to present to you my modifications to the NyAR4psg that makes multiple marker tracking easy! See here:

Standard hiro and kanji markers tracked simultaneously with augmented reality sphere and cube. In the background some artwork by my daughter!

I’ve called it NyARMultiBoard, and you can use it instead of the default NyARBoard if you want to track multiple markers.

Download a ZIP file containing everything (source code, jar files) from this directory.  If you unpack this into your processing sketchbook/libraries directory, it should work out of the box.  It’s a drop-in replacement for NyAR4psg, so you don’t need to have that installed as well. There is an example to get you started in NyAR2/example/NyARMultiTest.  Note: This uses the GSVideo capturing stack as I explain here, you should easily be able to change it back to processing defaults (just change GSCapture to Capture).

Please let me know in the comments if this works (or doesn’t) for you!

I made this screencast to demonstrate the multiple marker tracking, assisted by TNR:

I also made this really bad screencast (old webcam + night time lighting + transcoding):

If you’re really into the details

I’ve just added two new classes NyARMultiBoard and NyARMultiBoardMarker to the default NyAR4psg distribution. Very importantly, NyARToolkit itself needs to be patched with one extra method in NyARDetectMarker, see the NyARMultiBoard comments.

Update on 20110304

I’ve fixed the problematic frame bug in gsvideo that many of you have been running into. See this post.

Update on 20110305

I’ve updated NyAR2 so it works with the P3D renderer as well, which is often faster for blitting the webcam image onto the display. The updated zip file is named NyAR2-20110305.zip, and it can be downloaded from the usual directory. My changes are based on NyAR4psg 0.3.0 and NyARToolkit 2.5.2.

144 thoughts on “Processing + NyARToolkit + multiple marker tracking”

    1. On the processing forum you state that you just get a blank window. That means that the camera capture is not working.

      Can you get the basic example that ships with GSVideo working? It’s called GettingStartedCapture*.pde.

  1. Hi,
    Yes, ‘GettingStartedCapture.pde’ is OK to capture from my webcam, here is the pic to verify:
    http://dl.dropbox.com/u/25433933/ok.jpg
    All other samples that use webcam can show the result without problem. But this script just does not show, if I change the ‘background (#FF0000)’, it can show the background in red, just no cam view.

    Hope this can be resolved as expected ;).

    1. So now go through the code and compare the camera-related code between the working and non-working script. The nya script should at least show a camera image, even if the detection is not working.

  2. I tried, and this script still shows blank view. While at the same time
    GettingStartedCaptureWin is OK to show.

    I paste the 2 scripts here:
    …….NyARMultiTest.pde………………………
    import codeanticode.gsvideo.*;
    import jp.nyatla.nyar4psg.*;
    import processing.opengl.*;

    GSCapture cam;
    NyARMultiBoard nya;
    PFont font, font2d;

    void setup() {
    size(640,480,P3D);
    colorMode(RGB, 100);
    font=createFont(“FFScala”, 32);
    font2d = createFont(“FFScala”, 10);
    cam=new GSCapture(this,width,height);
    String[] patts = {“patt.hiro”, “patt.kanji”};
    double[] widths = {80,80};
    nya=new NyARMultiBoard(this,width,height,”camera_para.dat”,patts,widths);
    print(nya.VERSION);

    nya.gsThreshold=120;
    nya.cfThreshold=0.4;
    }

    void drawMarkerPos(int[][] pos2d)
    {
    textFont(font,10.0);
    stroke(100,0,0);
    fill(100,0,0);

    for(int i=0;i<4;i++){
    ellipse(pos2d[i][0], pos2d[i][1],5,5);
    }
    fill(0,0,0);
    for(int i=0;i<4;i++){
    text("("+pos2d[i][0]+","+pos2d[i][1]+")",pos2d[i][0],pos2d[i][1]);
    }
    }

    void draw() {
    if (cam.available() !=true) {
    return;
    }
    cam.read();

    hint(DISABLE_DEPTH_TEST);
    image(cam,0,0);
    hint(ENABLE_DEPTH_TEST);

    if (nya.detect(cam))
    {
    hint(DISABLE_DEPTH_TEST);
    for (int i=0; i < nya.markers.length; i++)
    {
    if (nya.markers[i].detected)
    {

    drawMarkerPos(nya.markers[i].pos2d);
    }
    }

    hint(ENABLE_DEPTH_TEST);
    PGraphicsOpenGL pgl = (PGraphicsOpenGL) g;
    for (int i=0; i < nya.markers.length; i++)
    {
    if (nya.markers[i].detected)
    {
    nya.markers[i].beginTransform();
    translate(0,0,20);

    if (i == 0)
    {
    stroke(255,200,0);
    box(40);
    }
    else
    {
    stroke(0,200,255);
    sphere(25);
    }
    nya.markers[i].endTransform();
    }
    }

    }
    hint(DISABLE_DEPTH_TEST);
    textFont(font2d,10.0);
    textMode(SCREEN);
    fill(100,100,0);
    text("frame rate = " + frameRate, 10, 10);
    textMode(MODEL);
    hint(ENABLE_DEPTH_TEST);
    }

    …….GettingStartedCaptureWin.pde……………….

    import codeanticode.gsvideo.*;

    GSCapture cam;

    void setup() {
    size(640, 480);

    String[] cameras = GSCapture.list();

    if (cameras.length == 0)
    {
    println("There are no cameras available for capture.");
    exit();
    } else {
    println("Available cameras:");
    for (int i = 0; i < cameras.length; i++) {
    println(cameras[i]);
    }
    cam = new GSCapture(this, 640, 480, cameras[0]);
    cam.start();

    /*
    // You can get the resolutions supported by the
    // capture device using the resolutions() method.
    // It must be called after creating the capture
    // object.
    int[][] res = cam.resolutions();
    for (int i = 0; i < res.length; i++) {
    println(res[i][0] + "x" + res[i][1]);
    }
    */

    /*
    // You can also get the framerates supported by the
    // capture device:
    String[] fps = cam.framerates();
    for (int i = 0; i < fps.length; i++) {
    println(fps[i]);
    }
    */
    }
    }

    void draw() {
    if (cam.available() == true) {
    cam.read();
    image(cam, 0, 0);
    // The following does the same, and is faster when just drawing the image
    // without any additional resizing, transformations, or tint.
    //set(0, 0, cam);
    }
    }

    1. I see a huge difference between the camera initialization. Point is, one script is working on your setup and the other is not. Both of them work on my setup. The logical deduction is that you have to start transplanting code from the working to the non-working script until the latter starts working on your setup, or until you get error messages that give me more information to work with.

  3. One thing to mention, I need to paste the NyAR2/*.jar folder into the libraries of Processing for NyAR2’s scripts to run(though blank ), or it will show error ‘can’t find a class ‘NyARMultiBoar”‘ , and after I paste, the regular NyAR scripts will show error if I run ‘can’t find a class MultiMarker. it seems 2 libraries conflict to each other.

  4. OK. Since both use different libraries:
    NyAR4psg.jar ..MultiMarker
    NyAR2.jar ..NyARMultiBoard

    It will be difficult for transplanting from one to the other, right ?
    And it seems the code is not a problem, maybe my web-cam has. Just why those using NyAR4psg.jar can show the view, whileas this using NyAR2.jar can’t.

    1. Aah, that could be part of the problem.

      From the web page above: ” It’s a drop-in replacement for NyAR4psg, so you don’t need to have that installed as well.”

      Delete any and all traces of NyAR4psg. You should only have the whole NyAR2 folder, including the example and library subfolders, in your processing sketchbook directory, and NOTHING from NyAR4psg.

      1. Hello cpbotha, I already tried deleting NyAR4psg from my libraries folder and still can’t use NyAR4psg.jar ..MultiMarker. Please help, what could I be doing wrong?

        Thanks

    1. That could very well have to do with changes between nyar2 and nyar4psg. I forked at that point from their 0.3.0 version if I remember correctly. NyAR2 definitely returns a value from nya.detect(), so it almost looks like the version you are calling is coming from a completely different library.

      If you want to solve this problem, you’re going to have to sort out your processing libraries installation.

  5. Hi cpbotha,

    Getting an error msg:
    begintransform.begin() only supported for P3D and OPENGL renderers
    when running the NyARMultiTest.pde

    Need yr advise running on 64 bit system…. any issue?

    1. change to processing version to 1.5.1.using 2+ version may appear the problem “begintransform.begin() only supported for P3D and OPENGL renderers”

  6. Hello cpbotha, I tried to run a sketch that used to work with your library and it gives me this message : “cannot find a class or name called “MultiMarker””
    What can I do ?

  7. Hello cpbotha,
    When I run an Augmented Reality sketch with NyAR2 it wont recognize this command: nya.setARPerspective(); which is vital in order to bound the 3D model to the marker.
    What can I do?
    I am desperate

    1. Can you double-check for me that you are running processing 1.5.1 (the latest stable) and NOT a 2.0 beta? The multi-marker tracking code has been thoroughly tested with 1.5.1, and not with 2.0.

      beginTransform() and endTransform() (from your mail) definitely should work, even the multimarker example uses them.

      1. Dear cpbotha,
        I downloaded de 1.5.1 (latest stable) and all my problems were solved
        Thanks a lot man \ m / (>.<) \ m /

        1. Glad you could fix it!

          Unfortunately, the processing site makes it look like 2.0 is already stable (it’s BETA), whilst 1.5.1 is actually the last stable releas (right at the bottom of the downloads page).

  8. Thanks for the reply!
    Yes, I have a 2.0 version :( I will search for the 1.5.1 and let you know how it goes, I’m having some trouble finding older versions of processing, any clues?
    About beginTransform() and endTransform() everytime I run those commands I keep receiving the following error: “beginTransform() only supported for P3D and OPENGL renderers “

  9. Hallo,
    I am a beginner in Augmented Reality and used Nyartoolkit at first ( For One Patt its good). I wanted to make an Applet as a project with Multiple Markers, but i don’t know how to use these pde Files. Could you help me pls. I searched on google found some tuts, but they don’t help me. I use Eclipse or JME3 + JMF + J3D+NyARToolkit. Maybe an example how to use the functions as java document, because pde looks like a scripting typo and it a bit divicould for me to make a java code out of it.

    1. You’re going to have to take a look in my NyAR2 source (in the zip file). This is simply a translation layer making the NyARToolkit multiple marker tracking functions available to processing and hence the .pde sketches.

  10. Hi,
    I’m currently having trouble getting GSVideo working with the NYARMultitest. I can get it running using processing.video* and changing the GSCapture class to Capture, however I would like to see it working with GSVideo.
    I also always seem to get the message “Display 0 does not exist, using the default display instead” not quite sure what thats means…

    I’m running Processing v1.51 and I’ve confirmed GSVideo is working on its own by running the simple example.
    Any ideas?

  11. Hey there,

    I have little Java experience(FYI).
    I am running Processing 2.0b7
    running GSVideo 1.0.0
    & running the most recent NyAR2 – from March 5th 2011,
    all on a MacBook Pro w/ Mac OS X 10.7.4

    I first tried to run the NyARTest.pde and saw a blue screen with some digital-fragmented squares, and then it flickered to a pure blue screen, and saw this message in the console :
    NyAR4psg/0.3.0;NyARToolkit for java/2.5.0+;ARToolKit/2.72.1DVFreeThread – CFMachPortCreateWithPort hack = 0x135f310, fPowerNotifyPort= 0x135ee80
    DVFreeThread – CFMachPortCreateWithPort hack = 0x115cde0, fPowerNotifyPort= 0x1136e50

    I then started to look in the comments and saw your suggestion of testing if GSVideo to see if that worked – to try and find the source of the problem. On running the GettingStartedCaptureMac.pde I saw a grey screen.
    I tried switching
    cam = new GSCapture(this, 640, 480); to
    cam = new GSCapture(this, 640, 480, “0”); and
    cam = new GSCapture(this, 640, 480, “1”); as the comments within the code seem to suggest, I also tried
    cam = new GSCapture(this, 640, 480, “USB Video Class Video:0”); and
    cam = new GSCapture(this, 640, 480, “USB Video Class Video:1”); just for the sake of trying, with no results.

    I am running a MacBook Pro with Mac OS X version 10.7.4 and an Intel core i5, so I am assuming 64bit, although to be honest after all these years I still don’t know what governs that.

    Like I said before, my knowledge of Java is elementary (I have more background in design than coding) but I am hoping to work with what little knowledge I have to try and make some cool AR, could you help me figure out how to my camera working?

    1. I tried to run the GettingStartedCaptureMac.pde again today (not sure why, you probably have heard the “definition of insanity before”) but for some it worked, as is (processing is running at version 2.0b7).

      Seeing that it worked, I tried adding the cam.start(); method call that was in GettingStartedCaptureMac.pde into the NyARTest.pde under the
      cam=new Capture(this,width,height); line
      and also the
      import codeanticode.*; line just for trial and error sakes.

      After running this, I got a screen with a bunch of noise and the text in the console read as

      DVFreeThread – CFMachPortCreateWithPort hack = 0x1f0dc590, fPowerNotifyPort= 0x1f0db300
      DVFreeThread – CFMachPortCreateWithPort hack = 0x16940a0, fPowerNotifyPort= 0x16941f0
      DVFreeThread – CFMachPortCreateWithPort hack = 0x1693810, fPowerNotifyPort= 0x1692df0

      So I switched to Processing 1.5.1. I ran the GettingStartedCaptureMac.pde and saw this message in the console

      Display 0 does not exist, using the default display instead.
      Feb 6 15:35:59 C-406C8F4C7CF2 java[642] : CGContextGetCTM: invalid context 0x0
      Feb 6 15:35:59 C-406C8F4C7CF2 java[642] : CGContextSetBaseCTM: invalid context 0x0
      Feb 6 15:35:59 C-406C8F4C7CF2 java[642] : CGContextGetCTM: invalid context 0x0
      Feb 6 15:35:59 C-406C8F4C7CF2 java[642] : CGContextSetBaseCTM: invalid context 0x0
      GSVideo version: 1.0.0

      although the webcam again showed up seemingly fine again.

      So I the import codeanticode.gsvideo.*; and the cam.start(); under the
      cam=new Capture(this,width,height); line, although I got a message that .start() is not a recognized method, nor is .play(), despite the fact that .start() worked in the GettingStartedCaptureMac.pde in 1.5.1

      I am getting the feeling that the versions of Processing I am using, and the libraries I am downloading aren’t compatible, but I don’t know what versions of everything I should be using.

      1. Dear Ian,

        This has been tested extensively with the latest stable version of processing, which is 1.5.1.

        If you’re just interested in getting it working, I’d suggest taking a clean processing 1.5.1 and then taking the zip archive mentioned on http://graphics.tudelft.nl/Courses/TI1100A/SoftwareInstallatie (that’s for a CS first years introductory project I’ve been teaching for 3 years) and unpacking it in your sketchbook directory. The zip includes everything you need, including GSVideo.

        1. Ok cool, thanks for the link, I had just got it working too through some messing around, so that is exciting, but the additional libraries are appreciated very much!

          Like I said in my second response to my own post, I will probably run into future problems.

          Thanks for the response!

  12. OK, I am sorry for these constant updates, I am running into problems I can’t seem to figure out, post for help, and then figure it out… partially.

    I was able to get the Hiro and Janji markers and projections (unsure of proper term) to work correctly now with some tinkering, using the most recent update of your NyAR library, GSVideo 1.0.0, and Processing 1.5.1 (would not work in 2.0b7)

    Although I will probably run into other problems in the future…

  13. Hi, thank you so much for making such a terrific set of code – I can count on less than one hand the number of times I’ve been able to immediately run an open source project of this complexity with zero problems. Really great stuff.

    One question – I am noticing a lot of jitter to the 3d shapes. The red ellipses placed at the corners of the fiducials are very stable (one or two points might jitter by 1-2 units), but the 3d shapes shake quite noticeably. Any suggestions?

    1. Dear Jason,

      Thanks for the compliment, I think you just know how to read the manual. ;)

      With regard to the jitter: It’s a known problem. Better lighting and better cameras improve the situation to some extent, but the transformation matrix is re-estimated at every single frame, so its susceptible to small changes in the image. I think the corner detection is just more stable than the actual matrix estimation.

      I think the most practical solution would be to bolt on a hysteresis function, doing a weighted add between the current estimation and the accumulation of all previous estimation. I have never gotten around to doing this. :)

      1. That makes sense. I’ll just need to dig into the API a bit more to understand implement this function – I am new to using NyARToolkit (and by extension, your extension!).

  14. Hello,

    I’m hoping you can help me. I’m working on an AR application and I need to be able to switch beween 3D models on just one marker. I’ve seen some videos on youtube where a marker is introduced and the model projected on the other marker changes. How can I achieve this? Any help will be much appreciated. (There’s an example in the video link below. Thanks

  15. Hi Cpbotha,

    I’ve been following your work and I just want to say without you, I would have been pretty lost trying to implement most of things I have so far in terms of achieving Augmented Reality with Processing. I’m currently working on an application to help teach students about the brain anatomy and would have loved to send you my source codes someway so you could give it a look. I’m hoping you can give me some tips. I’ve done some really cool stuff so far like creating my own custom markers, implementing your wonderful multiple marker tracking so I can have one marker display a section of the brain and another help rotate it at the same time. I still thinking up more ideas :)

    Right now, I’m having a bit of an issue trying to implement sounds without them repeating. I was trying to create a scenario where upon presenting a marker, it loads an audio file describing any section of the brain that has been presented. Now, it’s logical that once I take off the marker that helps load the audio file from tracking range, it would stop. But if my intended users don’t do this, the sound would keep repeating as it is still in range of tracking. Is there a possible fix to this? I’ve tried loads of methods, nothing seems to work. I would be willing to send you my source codes if you don’t mind.

    Thanks again for making the multiple marker tracking a reality :)

  16. dear Cpbotha,

    i have used your library for a project 2 years ago: and everything was ok.

    and now, i dont know why, it does not work anymore

    symptoms : blank window, not any image

    i have found “why” (!!!) : if i change the renderer (open gl or P3D) it works or not; it works with P3D but with a blue flicking, i does not work with openGL

    — i can solve the blue flicking changing your code (not “return” but only if (cam.available() cam.read…)

    but after that i get the error:: multiboard class cannot be found…

    —i have this library (NYAR2, last version)– and imported it
    — i have deleted every nyAr library since 2 years
    — i cannot understand???

    So i think::

    —- problems (old) with capture && open gl
    —- problems with your code (“return”) — i dont know why, beacause it seems normal but!!!— here i am sure…
    —- problems with the library structure (???)

    using processing 1.5.1
    mac os x 1.6.8

    since it worked what changes???
    update java 1.6 to 1.7

  17. dear cp B,

    i think i have have found by myself the solutions for some problems (not all!!!)

    in order to run the pde multiboard i had to::

    1) paste a code folder in the multiboard example directory, with the 2.jars from your library, else i get the missing class message for multiBoard nya;

    2) add cam.start() if using gsVideo library
    else the capture is missing

    3) change (as said) the code inside draw():: if(cam.available){
    cam.read(();}
    else i get some flickering & blue picture

    Then everything seems to be ok, P3D or OPENGL

    Note 1 :: that is tested only with osX 10.6.8 and p5 1.5.1

    Note 2: if using processing video capture instead of GSvideo:
    1) of course you have to import the processing video library
    2) not working with OPENGL, you have to change to P3D

    best wishes

    1. Hi, you seem to be quite good with the MultiARToolkit. Would you like to work on a paid project for me? If yes, give me an email I can reach you on.

      Happy New Year

  18. hi,
    i really interest on your project (from https://www.youtube.com/watch?v=5qAMUM7Z1_4)
    i’ve got an error its error

    “java.lang.RuntimeException: java.lang.NoSuchMethodError: processing.core.PApplet.registerDispose(Ljava/lang/Object;)V
    at processing.opengl.PSurfaceJOGL$2.run(PSurfaceJOGL.java:443)
    at java.lang.Thread.run(Thread.java:745)
    Caused by: java.lang.NoSuchMethodError: processing.core.PApplet.registerDispose(Ljava/lang/Object;)V
    at codeanticode.gsvideo.GSCapture.initGStreamer(Unknown Source)
    at codeanticode.gsvideo.GSCapture.(Unknown Source)
    at NyARMultiTest.setup(NyARMultiTest.java:50)
    at processing.core.PApplet.handleDraw(PApplet.java:2374)
    at processing.opengl.PSurfaceJOGL$DrawListener.display(PSurfaceJOGL.java:731)
    at jogamp.opengl.GLDrawableHelper.displayImpl(GLDrawableHelper.java:692)
    at jogamp.opengl.GLDrawableHelper.display(GLDrawableHelper.java:674)
    at jogamp.opengl.GLAutoDrawableBase$2.run(GLAutoDrawableBase.java:443)
    at jogamp.opengl.GLDrawableHelper.invokeGLImpl(GLDrawableHelper.java:1293)
    at jogamp.opengl.GLDrawableHelper.invokeGL(GLDrawableHelper.java:1147)
    at com.jogamp.newt.opengl.GLWindow.display(GLWindow.java:759)
    at com.jogamp.opengl.util.AWTAnimatorImpl.display(AWTAnimatorImpl.java:81)
    at com.jogamp.opengl.util.AnimatorBase.display(AnimatorBase.java:452)
    at com.jogamp.opengl.util.FPSAnimator$MainTask.run(FPSAnimator.java:178)
    at java.util.TimerThread.mainLoop(Timer.java:555)
    at java.util.TimerThread.run(Timer.java:505)

    please help me
    holy.gemmologo@gmail.com

    thanks!

Leave a Reply

Your email address will not be published. Required fields are marked *