FLARToolkit Single & MultipleMarker Detection

Posted on March 23, 2009
Filed Under Actionscript 3.0, experiments, Opensource, Papervision | 15 Comments

What you will need:
1. Photoshop (any will do)
2. Papervision3D rev. 817 or later
3. FlarToolkit
4. ArToolkit Marker Generator (flash player 10)
5. webcam
6. Fash CS3 or Higher

There are allready some great sites experimenting and writing tutorials about Augmented reality in flash. This is nothing new, just my experience in writing the code to create a simple version so I can learn from the others who started it. So before I start credits go out to squidder.com for some amazing tutorials and examples and saqoosha.net for starting the FlarToolkit.

For creating the best markers go and see the tutorial at squidder.com
For the starters kit you can get flartoolkit via libspark repository or download the starterskit at saqoosha.net.

Example of a Single Marker Detection:

Code Single Marker detection:

  1. package nl.wisman.codexperiments.augmented
  2. {
  3.  // papervision
  4.  import org.papervision3d.materials.utils.MaterialsList;
  5.  import org.papervision3d.objects.primitives.Cube;
  6.  import org.papervision3d.lights.PointLight3D;
  7.  import org.papervision3d.materials.shadematerials.FlatShadeMaterial;
  8.  import org.papervision3d.render.LazyRenderEngine;
  9.  import org.papervision3d.scenes.Scene3D;
  10.  import org.papervision3d.view.Viewport3D;
  11.  
  12.  // flash
  13.  import flash.display.PixelSnapping;
  14.  import flash.display.Bitmap;
  15.  import flash.net.URLRequest;
  16.  import flash.events.Event;
  17.  import flash.net.URLLoaderDataFormat;
  18.  import flash.net.URLLoader;
  19.  import flash.display.BitmapData;
  20.  import flash.media.Video;
  21.  import flash.media.Camera;
  22.  import flash.display.Sprite;
  23.  import org.papervision3d.objects.DisplayObject3D;
  24.  
  25.  // libspark
  26.  import org.libspark.flartoolkit.core.FLARCode;
  27.  import org.libspark.flartoolkit.core.param.FLARParam;
  28.  import org.libspark.flartoolkit.core.raster.rgb.FLARRgbRaster_BitmapData;
  29.  import org.libspark.flartoolkit.core.transmat.FLARTransMatResult;
  30.  import org.libspark.flartoolkit.detector.FLARSingleMarkerDetector;
  31.  import org.libspark.flartoolkit.pv3d.FLARBaseNode;
  32.  import org.libspark.flartoolkit.pv3d.FLARCamera3D;
  33.  
  34.  public class FlarSingleMarkerDemo extends Sprite
  35.  {
  36.   // flar properties
  37.   private var _camParameters:FLARParam;
  38.   private var _marker4Pattern:FLARCode;
  39.   private var _raster:FLARRgbRaster_BitmapData;
  40.   private var _detector:FLARSingleMarkerDetector;
  41.   private var _flarCam3D:FLARCamera3D;
  42.   private var _resultMat:FLARTransMatResult;
  43.   private var _baseNode:FLARBaseNode;
  44.  
  45.   // flash properties
  46.   private var _cam:Camera;
  47.   private var _vid:Video;
  48.   private var _capture:Bitmap;
  49.   private var _loader:URLLoader;
  50.  
  51.   // papervision3d
  52.   private var _vp:Viewport3D;
  53.   private var _scene:Scene3D;
  54.   private var _renderer:LazyRenderEngine;
  55.   private var _light:PointLight3D;
  56.   private var _cube:Cube;
  57.  
  58.   public function FlarSingleMarkerDemo()
  59.   {
  60.    loadCamera();
  61.   }
  62.  
  63.   private function loadCamera():void
  64.   {
  65.    _loader = new URLLoader();
  66.    _loader.dataFormat = URLLoaderDataFormat.BINARY;
  67.    _loader.addEventListener(Event.COMPLETE, onLoadCamParam);
  68.    _loader.load(new URLRequest("camera_para.dat"));
  69.   }
  70.  
  71.   // camera parameters are loaded, load the marker you want to use
  72.   private function onLoadCamParam(event:Event):void
  73.   {
  74.    _loader.removeEventListener(Event.COMPLETE, onLoadCamParam);
  75.    _camParameters = new FLARParam();
  76.    _camParameters.loadARParam(_loader.data);
  77.    _camParameters.changeScreenSize(500, 500);
  78.  
  79.    _loader.dataFormat = URLLoaderDataFormat.TEXT;
  80.    _loader.addEventListener(Event.COMPLETE, onLoadCode);
  81.    _loader.load(new URLRequest("yourmarker.pat"));
  82.   }
  83.  
  84.   // marker now also loaded, so remove _loader and create a FlarCode
  85.   private function onLoadCode(event:Event):void
  86.   {
  87.    initWebcam();
  88.  
  89.    _marker4Pattern = new FLARCode(4, 4, 65, 65);
  90.    _marker4Pattern.loadARPatt(_loader.data);
  91.  
  92.    _loader.removeEventListener(Event.COMPLETE, onLoadCode);
  93.    _loader = null;
  94.  
  95.    // create bitmap and bitmapdata where we can draw the webcam into
  96.    var bmd:BitmapData = new BitmapData(500, 500, false, 0);
  97.    _capture = new Bitmap(bmd, PixelSnapping.AUTO,false);
  98.    _capture.width = 500;
  99.    _capture.height = 500;
  100.    addChild(_capture);
  101.  
  102.    _raster = new FLARRgbRaster_BitmapData(_capture.bitmapData);
  103.    _detector = new FLARSingleMarkerDetector(_camParameters, _marker4Pattern, 65);
  104.  
  105.    initFlar();
  106.    initPV3D();
  107.    initListeners();
  108.   }
  109.  
  110.   private function initWebcam():void
  111.   {
  112.    _cam = Camera.getCamera();
  113.    _cam.setMode(500, 500, 60);
  114.    _vid = new Video(500, 500);
  115.    _vid.attachCamera(_cam);
  116.   }
  117.  
  118.   private function initFlar():void
  119.   {
  120.    _flarCam3D = new FLARCamera3D(_camParameters);
  121.    _resultMat = new FLARTransMatResult();
  122.    _baseNode = new FLARBaseNode();
  123.   }
  124.  
  125.   private function initPV3D():void
  126.   {
  127.    _vp = new Viewport3D(500, 500);
  128.    addChild(_vp);
  129.  
  130.    _scene = new Scene3D();
  131.    _scene.addChild(_baseNode);
  132.  
  133.    _light = new PointLight3D();
  134.  
  135.    // add a cube or whatever we want to show when pattern is recognised
  136.    // in the basenode (FLARBaseNode, which extends from a DisplayObject3D)
  137.    var fmat:FlatShadeMaterial = new FlatShadeMaterial(_light, 0x7DC202, 0xCCCCCC);
  138.    _cube = new Cube(new MaterialsList({all: fmat}), 40, 40, 80);
  139.    _cube.z = 20;
  140.    _cube.name = "cube";
  141.    _baseNode.addChild(_cube);
  142.  
  143.    _renderer = new LazyRenderEngine(_scene, _flarCam3D, _vp);
  144.   }
  145.  
  146.   private function initListeners():void
  147.   {
  148.    addEventListener(Event.ENTER_FRAME, render);
  149.   }
  150.  
  151.   private function render(event:Event):void
  152.   {
  153.    _capture.bitmapData.draw(_vid);
  154.  
  155.    if (_detector.detectMarkerLite(_raster, 65) && _detector.getConfidence() > 0.5)
  156.    {
  157.     // when deteactmarker finds the marker you created show the baseNode
  158.     _detector.getTransformMatrix(_resultMat);
  159.     _baseNode.setTransformMatrix(_resultMat);
  160.     _baseNode.visible = true;
  161.    }
  162.    else
  163.    {
  164.     _baseNode.visible = false;
  165.    }
  166.  
  167.    if(_baseNode.visible)
  168.    {
  169.     Cube(_baseNode.getChildByName("cube")).roll(1);
  170.    }
  171.  
  172.    _renderer.render();
  173.   }
  174.  }
  175. }

Example of a Multiple Marker Detection: (its still a work in progress, still shifts cube from one pattern to another but he…)

Code Multiple Marker detection:

  1. package
  2. {
  3.  // bulkloader
  4.  import br.com.stimuli.loading.BulkLoader;
  5.  
  6.  // papervision
  7.  import org.papervision3d.materials.utils.MaterialsList;
  8.  import org.papervision3d.objects.primitives.Cube;
  9.  import org.papervision3d.lights.PointLight3D;
  10.  import org.papervision3d.materials.shadematerials.FlatShadeMaterial;
  11.  import org.papervision3d.render.LazyRenderEngine;
  12.  import org.papervision3d.scenes.Scene3D;
  13.  import org.papervision3d.view.Viewport3D;
  14.  
  15.  // flash
  16.  import flash.display.PixelSnapping;
  17.  import flash.display.Bitmap;
  18.  import flash.net.URLRequest;
  19.  import flash.events.Event;
  20.  import flash.net.URLLoaderDataFormat;
  21.  import flash.net.URLLoader;
  22.  import flash.display.BitmapData;
  23.  import flash.media.Video;
  24.  import flash.media.Camera;
  25.  import flash.display.Sprite;
  26.  import org.papervision3d.objects.DisplayObject3D;
  27.  
  28.  // libspark
  29.  import org.libspark.flartoolkit.core.FLARCode;
  30.  import org.libspark.flartoolkit.core.param.FLARParam;
  31.  import org.libspark.flartoolkit.core.raster.rgb.FLARRgbRaster_BitmapData;
  32.  import org.libspark.flartoolkit.core.transmat.FLARTransMatResult;
  33.  import org.libspark.flartoolkit.detector.FLARMultiMarkerDetector;
  34.  import org.libspark.flartoolkit.pv3d.FLARBaseNode;
  35.  import org.libspark.flartoolkit.pv3d.FLARCamera3D;
  36.  
  37.  public class FlarMultiMarkerDemo extends Sprite
  38.  {
  39.   // flar properties
  40.   private var _camParameters:FLARParam;
  41.   private var _raster:FLARRgbRaster_BitmapData;
  42.   private var _detector:FLARMultiMarkerDetector;
  43.   private var _flarCam3D:FLARCamera3D;
  44.   private var _resultMat:FLARTransMatResult;
  45.  
  46.   // flash properties
  47.   private var _cam:Camera;
  48.   private var _vid:Video;
  49.   private var _capture:Bitmap;
  50.   private var _loader:URLLoader;
  51.   private var _allMarkers:Array;
  52.  
  53.   // papervision3d
  54.   private var _vp:Viewport3D;
  55.   private var _scene:Scene3D;
  56.   private var _renderer:LazyRenderEngine;
  57.   private var _light:PointLight3D;
  58.   private var _bLoader:BulkLoader;
  59.  
  60.   public function FlarMultiMarkerDemo()
  61.   {
  62.    _allMarkers = [
  63.          {markerSource: "path/your_first_marker.pat", segments: 4, size: 65},
  64.          {markerSource: "path/your_second_marker.pat", segments: 4, size: 65},
  65.          {markerSource: "path/your_third_marker.pat", segments: 4, size: 65},
  66.         ];
  67.  
  68.    loadCamera();
  69.   }
  70.  
  71.   private function loadCamera():void
  72.   {
  73.    _loader = new URLLoader();
  74.    _loader.dataFormat = URLLoaderDataFormat.BINARY;
  75.    _loader.addEventListener(Event.COMPLETE, onLoadCamParam);
  76.    _loader.load(new URLRequest("camera_para.dat"));
  77.   }
  78.  
  79.   // camera parameters are loaded, load the marker you want to use
  80.   // camera parameters are loaded, load the marker you want to use
  81.   private function onLoadCamParam(event:Event):void
  82.   {
  83.    _loader.removeEventListener(Event.COMPLETE, onLoadCamParam);
  84.  
  85.    // setup camera parameters
  86.    _camParameters = new FLARParam();
  87.    _camParameters.loadARParam(_loader.data);
  88.    _camParameters.changeScreenSize(500, 500);
  89.  
  90.    // reset loader
  91.    _loader = null;
  92.  
  93.    _bLoader = new BulkLoader("markerLoader");
  94.    _bLoader.addEventListener(Event.COMPLETE, markersLoadedHandler);
  95.    // now load multiple markers (for now its 3)
  96.    for (var i:int = 0; i < _allMarkers.length; i++)
  97.    {
  98.     _bLoader.add(_allMarkers[i].markerSource, {id: "marker_" + i.toString() });
  99.    }
  100.    _bLoader.start();
  101.   }
  102.  
  103.   private function markersLoadedHandler(event:Event):void
  104.   {
  105.    var codes:Array = new Array();
  106.    var sizes:Array = new Array();
  107.    for (var i:int = 0; i < _allMarkers.length; i++)
  108.    {
  109.     var code:FLARCode = new FLARCode(_allMarkers[i].segments, _allMarkers[i].segments, _allMarkers[i].size, _allMarkers[i].size);
  110.     code.loadARPatt(_bLoader.getContent( "marker_" + i.toString()));
  111.     codes.push(code);
  112.     sizes.push(_allMarkers[i].size);
  113.    }
  114.  
  115.    initWebcam();
  116.  
  117.    // create bitmap and bitmapdata where we can draw the webcam into
  118.    var bmd:BitmapData = new BitmapData(500, 500, false, 0);
  119.    _capture = new Bitmap(bmd, PixelSnapping.AUTO,false);
  120.    _capture.width = 500;
  121.    _capture.height = 500;
  122.    addChild(_capture);
  123.  
  124.    _raster = new FLARRgbRaster_BitmapData(_capture.bitmapData);
  125.    _detector = new FLARMultiMarkerDetector(_camParameters, codes, sizes, codes.length);
  126.  
  127.    initFlar();
  128.    initPV3D();
  129.    initListeners();
  130.   }
  131.  
  132.   private function initWebcam():void
  133.   {
  134.    _cam = Camera.getCamera();
  135.    _cam.setMode(500, 500, 60);
  136.    _vid = new Video(500, 500);
  137.    _vid.attachCamera(_cam);
  138.   }
  139.  
  140.   private function initFlar():void
  141.   {
  142.    _flarCam3D = new FLARCamera3D(_camParameters);
  143.    _resultMat = new FLARTransMatResult();
  144.   }
  145.  
  146.   private function initPV3D():void
  147.   {
  148.    _vp = new Viewport3D(500, 500);
  149.    addChild(_vp);
  150.  
  151.    _scene = new Scene3D();
  152.  
  153.    _light = new PointLight3D();
  154.  
  155.    // add a cube or whatever we want to show when pattern is recognised
  156.    // in the basenode (FLARBaseNode, which extends from a DisplayObject3D)
  157.    for (var i:int = 0; i < _allMarkers.length; i++)
  158.    {
  159.     var fmat:FlatShadeMaterial = new FlatShadeMaterial(_light, 0x7DC202 * Math.random(), 0xCCCCCC);
  160.     var cube:Cube = new Cube(new MaterialsList({all: fmat}), 40, 40, 80);
  161.     cube.z = 20;
  162.     cube.name = "cube_" + i.toString();
  163.     var baseNode:FLARBaseNode = new FLARBaseNode();
  164.     baseNode.name = "baseNode_" + i;
  165.     baseNode.addChild(cube);
  166.     _scene.addChild(baseNode);
  167.    }
  168.  
  169.    _renderer = new LazyRenderEngine(_scene, _flarCam3D, _vp);
  170.   }
  171.  
  172.   private function initListeners():void
  173.   {
  174.    addEventListener(Event.ENTER_FRAME, render);
  175.   }
  176.  
  177.   private function render(event:Event):void
  178.   {
  179.    _capture.bitmapData.draw(_vid);
  180.  
  181.    for (var i:int = 0; i< _allMarkers.length; i++)
  182.    {
  183.     if(_detector.detectMarkerLite(_raster, 65) &#038;&#038; _detector.getConfidence(i) > .5)
  184.     {
  185.      _detector.getTransmationMatrix(i, _resultMat);
  186.      FLARBaseNode(_scene.getChildByName("baseNode_" + i)).setTransformMatrix(_resultMat);
  187.      _scene.getChildByName("baseNode_" + i).visible = true;
  188.     }
  189.     else
  190.     {
  191.      _scene.getChildByName("baseNode_" + i).visible = false;
  192.     }
  193.    }
  194.  
  195.    _renderer.render();
  196.   }
  197.  }
  198. }

And last but not least a flash version to play with:

Try it yourself, but don’t forget to download the patterns first.

Comments

15 Responses to “FLARToolkit Single & MultipleMarker Detection”

  1. Andre Anaya on March 25th, 2009 10:24 pm

    Pretty cool!!
    I didn’t knew that FlarToolkit works with multiple markers

  2. Jan Willem on March 26th, 2009 1:15 pm

    He Andre, definately checkout John Lindquist blog where he has some amazing freaky stuff going on!

  3. leresteux j on April 24th, 2009 7:42 pm

    wonder
    thank u for the “MultipleMarker Detection”

  4. Nin on August 26th, 2009 9:16 pm

    Very good, but does not associate at the 3D object.
    Example: Suppose that the application has 3 patterns. By pointing the pattern 3 to the camera will be displayed in the 3D object shown in the set panttern 1.

    It not is equal to FLARManager linking pattern to object 3D.

    But thanks for sharing!

  5. WIetse on February 27th, 2010 2:07 am

    Very nice. Is there a way to use collada models instead of cubes? I know you can use one collada model if your only using single marker detection but how about the multiple marker detection?

  6. Martijn on March 22nd, 2010 5:15 pm

    THanks for sharing. I still get errors but okay. What I was wondering: The bulkloader ( at the top ) what is that? And is that a class you have to download or is that comming with the flartoolkit?

    Thanks,

    Martijn

  7. Martijn on March 22nd, 2010 5:39 pm

    Sorry for the second message but the error’s are in line:

    159
    183
    189
    These are the messages:

    1008: Attribute is invalid. Flex Problem
    1083: Syntax error: else is unexpected. Flex Problem
    1084: Syntax error: expecting identifier before 0xCCCCCC. Flex Problem
    1084: Syntax error: expecting identifier before bitwiseand. Flex Problem
    1084: Syntax error: expecting rightbrace before rightparen. Flex Problem
    1084: Syntax error: expecting rightparen before 7. Flex Problem
    1084: Syntax error: expecting rightparen before semicolon. Flex Problem
    1093: Syntax error. Flex Problem
    1093: Syntax error. Flex Problem
    1093: Syntax error. Flex Problem

    I don’t know If someone can help me?

  8. Meir on April 10th, 2010 9:55 pm

    Martijn, note that in line 158 you have the string:
    0×7DC202
    the symbol there is non printable that looks like and x , change it to 0x7DC202 and it should work (you might have an extra parenthesis there too).

  9. Meir on April 10th, 2010 9:55 pm

    Sorry, meant, change it to:

    0x7DC202

  10. Meir on April 10th, 2010 9:56 pm

    Okay, the double post explains it… the site formats 0X11111 numbers and puts this strange char

  11. syzen on June 15th, 2010 11:12 am

    sorry..
    i didnt get it, still having error on line
    183 1084
    189 else is unexpected

  12. CHICAN0 on December 16th, 2010 10:55 pm

    can you load flv with Bulkloader to play them over the marker ?

    Thanks

  13. dvd ripper review on June 18th, 2011 5:33 pm

    Thank you for sharing that, it’s very useful to me! Thanks again！

  14. beckji on March 29th, 2012 1:28 pm

    As previously stated by WIetse on February 27th, 2010, I am also interested in the use collada models instead of cubes.
    I can currently produce the code for one collada model but do you have a simple 2 collada model to show multiple marker detection in action.

  15. Tetrahedron on April 23rd, 2012 12:13 am

    I also have issues on line 183 and 189 as syzen and Martijn.

    183 1084: Syntax error: expecting rightparen before semicolon.
    183 1008: Attribute is invalid.
    183 1084: Syntax error: expecting rightbrace before rightparen.

    189 1083: Syntax error: else is unexpected.

-->