[v0.7 TODO] (-) IntegralImageThresholdAdapter is running too slowly. test w/o FLARToolkit... ( ) away3d lite support ( ) add matt reynolds' hybrid threshold adapter (email subject: "another thing...") ( ) support for inverted markers (per email from jimalliban) ( ) adaptive smoothing ( ) motion extrapolation ( ) patternToBorderRatio ( ) test FLARTransMat.transMatContinue with new code uploaded by saqoosha (X) email mark with items for coding assistance ( ) clean up FLARManager by parceling out certain processes to separate classes: ( ) config loading and parsing can go into FLARManagerConfigLoader controller class: pass FLARManager instance, most configurations have public setters. for those that don't (e.g. FLARCameraSource within FLARManager), something else... ( ) full-res video setting to 640x480, downsampleRatio=1, sampleBlurring=2 seems to work well; need to benchmark. thinking was that labeling process is where things slow down with full-res, which implies there are somehow more distinct dark areas in the downsampled full-res image. so, blur them a bit more. but this might have negative impact on marker detection. ( ) fix bug: width < height causes strangeness in PV3D ( ) fix z-drift in Flash3D is it possible this is related to FLARCameraParams.dat? camera FOV maybe? related to lack of FLARCamera3D? (in mail, sent to/from me, subj: away3d lite support and flash 3d) ( ) get non-4:3 aspect ratios working for Flash3D (-) create new camera_para.dat for built-in iSight (-) create new camera_para.dat with no correction (could maybe do this with a longer focal length lens?) [v0.6 release notes] revised FLARCameraSource camera selection; added FLARCameraSource.cameraIndex better camera selection better 2D support: added FLARMarker.x, .y, .z, .rotation2D, .scale2D, .matrix2D updated FLARManagerTutorial_2D updated FLARManagerExample_2D (MarkerOutliner) added FLARMarker.matrix3D, but Flash3D still isn't working well. updated FLARManagerExample_Flash3D corrected bug (incomplete implementation) in vector3D added support for pattern size to flarConfig.xml updated away3D to 3.3.4 updated pv3D to 2.1.920 minimumLabelSize settable via FLARManager.minimumLabelSize and as attribute to . native mirroring support improved stability of some examples/tutorials (Flash3D, Tutorial2D, 3D, Collada: added UPDATED handler and set object.visible = true, to avoid "since new markers are added..." problem described below.) accessor for FLARManager.isActive, instead of de/activate (check if this is v0.6 or earlier) FLARManager.dispose FLARManager.verbose modularized adaptive thresholding, additional options for AT changed flarConfig.xml to match modular smoothing and threshold adapter. note: changes to adaptive thresholding will break pre-v0.6. here's how you fix it. [v0.6 TODO] (X) dispose method for FLARManager (per forum thread from luca volta) (X) complete dispose method for all classes necessary (X) test with profiler (X) verbosity: dump settings changes to console. this is started in FLARManager and FLARCameraSource, but not complete. (X) modularize adaptive thresholding (X) add to dispose() (X) enable setting via flarConfig: (X) enable setting smoother via flarConfig (sim to thresholdAdapter) (X) commit changes to FLARMultiMarkerDetector (X) implement matt reynolds' adaptive thresholding (X) implement eugene zatepyakin's adaptive thresholding (X) update PV3D. sounds like collada handling is better in latest version, and those changes may have impacted the way FLARPVGeomUtils converts FLAR matrices.... (X) better camera handling: choose default camera via Camera.getCamera() (no param). may still want to select USB Video Class Video for OSX. platform detection: if (flash.system.Capabilities.os.substr(0, 3) == "Mac") monitor camera activityLevel; if it stays at -1, open a dialog that allows selection of the camera. while camera is being selected, FLARManager should deactivate. when camera is selected, FLARManager may need to be reinited (?); regardless, it should be activated. if no cameras found, display dialog that asks for camera to be connected, with selection drop down. dialog and dropdown happens in FLARManagerSetupDialog, which implements IFLARSetupDialog. devs can implement their own IFLARSetupDialog, and set it via FLARManager.setupDialog. (X) if no camera found (catch block in FLARManager.initFlarSource), don't continue FLARManager initialization, and provide way to clear out FLARManager instance for reinitialization after user sets up camera. (X) provide way to init FLARManager by passing in an xml object. (X) what is minConfidence if not specified? if it's 0, that's a problem... (X) implement and streamline/optimize FLARMarker.matrix2D (in SequencAR) (X) implement and streamline/optimize FLARMarker.matrix3D (in SequencAR) (X) give access to x/y/z as FLARMarker.x/y/z (X) finish implementing FLARMarker.scale2D and rotation2D (X) update FLARManagerExample_Flash3D to use .matrix3D (X) zip and post v0.6. (X) add bit about marker pattern resolution to tutorials somewheres... (also to source comments of some file(s)?) definitely to documentation page, in flarConfig.xml section... (X) table of contents in blog pages using anchor links. (X) clean up comments for ASDocs (X) rebuild docs (X) write Customizing FLARManager page, and add links to documentation page (from smoother and thresholdAdapter). (X) document support for unscaledMarkerWidth as @size attrib of node (X) document minimumLabelSize in flarConfig (X) document process of developing custom smoother / threshold adapter, including specifying custom params in flarConfig.xml. (X) document changes to flarConfig.xml for smoother / threshold adapter. (X) update build.xml with all new classes (X) SEO - "flash AR", "flash augmented reality", "flar", etc (X) add license agreement checkbox on download/svn url page (X) put 301 redirect on http://transmote.com/flar (X) test deactivation -- holde reports high CPU consumption even while deactivated. (X) modularize adaptive thresholding (see above) (-) create flash IDE version (.fla) (X) expose minimum labeling area in FLARManager (-) standardize setParams in FLARCamera3D across all 3D frameworks (X) update FLARToolkit, fix 3D framework imports to point at new 'support' package. (X) update away3D to 3.3.3 (-) fix ADDED before REMOVED bug (X) get native mirroring working (X) support manipulating FLARLabeling_BitmapData.minimumLabelSize via flarConfig.xml and FLARManager. [v0.5 release notes] created example launcher application class examples for Away3D, Alternativa3D, and Sandy3D added support for Alternativa3D, and Sandy3D to FLARToolkit source compiled external libraries into SWCs FLARToolkit remains as source, due to dependencies on the 3D libraries, and to allow devs to poke around FLARProxy memory and performance testing and optimization adaptive thresholding marker removal delay collada tutorial removed FLARMarkerOutline FLARManager is now inited by passing the flarConfig path directly into the ctor; legacy loading via FLARManager.initManual. [v0.5 TODO] (X) write InsideRIA article about tweaking performance with FLARManager: combining high markerUpdateThreshold with markerRemovalDelay for fast-moving markers using sampleBlurring to balance performance and accuracy (explain labeling()) adjusting adaptiveThresholdingSpeed and Bias for different lighting conditions viewing thresholded source with thresholdSourceDisplay to assess thresholding and blurring impacts (this can all be done via flarConfig.xml!) (X) run careful tests to determine effects of adaptive thresholding and blur filter. (X) email flartoolkit list: comments added to FLARLabeling_BitmapData.labeling() (X) post FLARManager support on forums for other 3D engines (X) implement pointlight and shaded material for away3D example, per holde's email (X) upload new html_docs along with v0.5. (X) update FLARManager page. (X) include scout.txt file in resources/assets (X) put project import process on Inside FLARManager > Getting Started (per email to shachar oz friday) (X) update http://words.transmote.com/wp/flarmanager/flarmanager-documentation/ (FLARProxy, what else?) (X) write collada tutorial (just a slight modification of basic 3D) http://words.transmote.com/wp/flarmanager/inside-flarmanager/loading-collada-models/ (X) update existing tutorial writeups with flarConfig in FLARManager ctor change. (X) update build.xml doc classes list (X) support for other 3D engines: (X) away3D (X) sandy3D (X) alternativa3D (X) build SWCs for FLARToolkit and PV3D note, can't build a FLARToolkit SWC because it has away3D and pv3D dependencies. anyway, i think it's better to leave FLARToolkit source in the download -- it's good for devs to be able to look through it. got the PV3D SWC built; one strange catch, had to set the link type for the /libs folder to "Merged into code". specifying "External" caused a runtime error. (X) remove deprecated bits (?): (X) complete migration to xml config file initialization of FLARManager (X) remove FLARMarkerOutline completely (X) FLARProxy as IFLARSource, specified in flarConfig.xml (X) run profiler, ensure no memory leaks memory results: FLARManager occupies .75-1% of total memory FLARMarkers are cleaned up properly; no loitering objects. there are some inefficiencies, but they are within FLARToolkit and PaperVision3D. performance results: FLARManager uses ~0.6% of CPU consumed every frame by FLARToolkit FLARLabeling_BitmapData.labeling() is consuming: ~77% of CPU per frame (when idle) ~70% of CPU per frame (with marker) ArrayUtil.createJaggedArray: ~2.8% of all total calls NyObjectStack.prePush: ~1.6% of all total calls (~100 times per frame) FLARColorPatt_O3.updateExtpat (with marker) has longest self-time by far: 4.95ms (-) find opportunities for optimization (X) downsampleRatio is affecting away3D example? (email from holde, subject: Re: FLARManager v0.4) (X) apply BlurFilter prior to thresholding, to speed labeling process (X) marker removal delay (X) adaptive thresholding (X) gradually increase adaptive thresholding step size after markers are lost (X) per heman's emails to flartoolkit link, provide super-simple example for loading collada, with instructions on how to swap marker (.pat) and collada. (-) might want to create a simpler multi-marker sample, per this comment: http://words.transmote.com/wp/flarmanager/inside-flarmanager/basic-augmented-reality/ [v0.4 tutorial release] (X) post about flarConfig options on FLARManager documentation page (X) test Vector3D and adjust comment accordingly: probably based on (0,0) at screen center, while centerpoint3D is adjusted to (0,0) at upper-left determine if both are needed. (X) add tutorial urls to class comments in tutorial classes (X) clean up and post ASDocs (X) complete links to documentation pages from within tutorial articles (i.e. link to FLARMarker doc page from 2D tutorial.) [v0.4 TODO] (X) test all examples with all combos of aspect ratio and downsampleRatio (X) 2D (X) 3D (X) PV3D fix: (X) 2D, 3D, PV3D: <4:3, any dsRatio (X) 3D: downsampleRatio changes make a difference in scale (X) test resultsToDisplayRatio in both FLARCameraSource and FLARLoaderSource (X) update FLARProxy to match changes to downsample/resultsToDisplayRatios (X) downsampleRatio should be relative to source, not display (X) remove downsampleRatio from FLARManagerExample_Flash3D (X) non-4:3 aspect ratios are broken in FME_Flash3D / MarkerOutline3D (X) width < height causes strangeness in PV3D (X) clean up all the testing junk in SimpleCubes.as (X) recompile ASDocs (X) clean up / port to flarConfigFile existing examples (X) mirroring (X) non-4:3 example (X) non 4:3 sizes (custom FLARCameraSource) distorting PV3D scene/viewport not centering vertically when 4+:3 (widescreen) (X) would be nice to pass source/displayW/H in via xml, but then flarSource will not be available for addition to display list immediately on instantiation. one way around this is to make FLARManager a Sprite, and just add it, similar to Loader. would like to keep things composition-based though...hm... (-) basic example, with no setup, empty FM ctor, no config xml file. basically, a quickstart. (X) remove Mothership tests (X) test all examples (X) test without (full and partial) flarSourceSettings node in flarConfig.xml (X) FLARLoaderSource specified via flarConfig.xml (X) test mirroring in all examples (X) test aspect ratios in all examples (X) should all use flarConfig.xml, this is now preferred. [v0.4 changes] - commercial license - changes to FLARCameraSource - true mirroring (does not require mirroring container object) (TODO: not yet working for PV3D...) (TODO: working for flash 3D, but turned off (in FLARMarker) to avoid interfering with PV3D) - full-res video (TODO: doesn't work well with low light, so defaults are still 320x240.) - captureW/H vs. displayW/H - downsampleRatio clarified -- applied to sourceWidth/Height to determine size of BitmapData sent to FLARToolkit ACTUALLY, mostly removed, replaced with captureToDisplayRatio - encapsulated downsampleRatio within FLARManager classes, so FLARManager apps don't need to know about it (TODO: this is not yet complete in Flash3D example...) - similar changes to FLARLoaderSource - full-res display - captureW/H vs. displayW/H - matrix-level smoothing allows developers to set the amount of smoothing, and to easily implement their own smoothing algorithms - many application settings can now be specified in xml file, without recompiling (use FLARManager.initFromFile): - mirroring and smoothing - external file paths, e.g. camera params and pattern files - changes to FLARMarker - consolidated FLARMarkerOutline into FLARMarker; deprecated FLARMarkerOutline - provides vector3D, rotationX/Y/Z, what else? - sessionId - applySmoothing (see above) - better disposal - changes to FLARManager - initFromFile - exposed markerUpdateThreshold - smoothing / smoother (see above) - Flash3D working, except for non-4:3 aspect ratios - 12 new sample patterns - ASDocs - there are a LOT of new features here, so there are likely bugs! please help me squash them. - PV3D and FLARToolkit now distributed as SWC instead of individual files, to make downloading and updating from the repo faster - known bugs: specifying a FLARCameraSource with width < height with PV3D causes strangeness (-) since new markers are added before old markers are removed, moving a marker fast enough for it to register as a new marker in a new location can cause a problem where a handler that receives the removed marker second can set a Sprite's visibility to false after the marker is re-added in its new location, thus causing a marker to continue to update, but appear to be gone. solution is to cache added markers in FLARManager.detectMarkers, and only dispatch their added events *after* all removed events have been dispatched. not sure how this will tie in with marker management Vectors within FLARManager; ideally, only thing that changes is order of dispatchEvents. repro: FLARManagerTutorial_2D: move marker fast, circle will disappear, though updated events are still being sent to console. actually, this cannot be fixed within FLARManager without more advanced tracking/ prediction of fast-moving marker locations. problem happens because removal is no longer immediate, but is delayed by markerRemovalDelay frames. so the fast-moving marker reappears somewhere else before it's removed, a couple frames later. since there is no longer any way to tell if the new marker is the same as the old (because it's too far away from the removal position), this cannot be accounted for. solution, then, lies in the app using FLARManager -- always create a new container when a marker appears, do not just toggle the visibility of the container. ( ) test FLARTransMat.transMatContinue with new code uploaded by saqoosha ( ) oddly, still can't really go over 320x240 for capture. even though BitmapData being sent to FLARToolkit is definitely downsampled to 320x240, FLARLabeling_BitmapData.labeling() iterates about 3x as long with a 640x480 capture. can't figure out why... ( ) create camera_para.dat with as little distortion as possible. (use long focal width unibrain lens?) distortion is quite noticeable when using very long or very tall aspect ratios. ( ) move config loading and parsing out of FLARManager into its own class, FLARManagerConfigLoader. ( ) documentation! (+ ASDocs) ( ) reduce jitter by using an ease value that corresponds to the distance the marker moved b/w frames. if small motion (i.e. jitter in a mostly-stationary marker), use slow(low) ease value. if large motion (i.e. moving marker), use fast(high) ease value. NOTE: can do this fairly easily now by changing FLARManager.smoothing on-the-fly; more specifically, by using FLARManager.smoothing as a base, and by sending a value for numFrames to each detected FLARMarker that deviates from FLARManager.smoothing based on the amount of motion in the FLARMarker between current and previous frames. or better yet, based on amount of motion averaged over previous N frames. NOTE: have to check for motion in rotation as well as position. perhaps possible to check for motion in matrix values? ( ) calculate velocity and acceleration of markers and extrapolates when markers disappear momentarily. (i.e. better fast-moving marker detection.) ( ) add patternToBorderRatio to FLARMultiMarkerDetector: use FLARDynamicRatioColorPatt_O3 instead of FLARColorPatt_O3. (X) remove mothership tests!! - MarkerOutliner (X) allow ability to specify patterns in xml file, like MythManagerCMS / FGAR. (X) redesign FLARProxy so it can be used as a source by FLARManager, so any application can just init a FLARManager instance like normal, but specify a FLARProxy as a source (instead of the default FLARCameraSource, for example). also, allow FLARProxy usage to be set via flarConfig.xml. (X) ensure FLARMarker instances are being disposed properly after being copied to new (updated) instances, and after removal. (X) move cameraParamsPath and patterns params from FLARManager ctor into a separate init method, and move flarConfigPath into ctor params. (i.e. make flarConfigPath the default way to instantiate FLARManager.) (X) implement converters for other 3D engines, from makc's links on the google group http://groups.google.com/group/flartoolkit-userz/web/links (look at this: http://makc.googlecode.com/svn/trunk/flash/alternativa3d_flar/SimpleTest.as for info on converting to alternativa3d and also on dissecting FLARParam) also, saqoosha just added Away3D support -- grab matrix conversion from that and add to FLARManager. (X) FLARMarker and FLARMarkerOutline are confusing. some doubled information, may only need one class...assess. yeah, this should be done. just move all accessors from FLARMarkerOutline up to FLARMarker. possibly leave FLARMarker.outline available for one revision, and add trace in outline getter that reads "this is deprecated". (X) dynamic thresholding: take a look at makc's solution; also, ARToolkitPlus method for auto-thresholding: http://studierstube.icg.tu-graz.ac.at/handheld_ar/artoolkitplus.php take avg b/w darkest and lightest pixels within detected marker. perhaps just grab ARToolkitPlus source and integrate into FLARToolkit? (X) compile SWC of libspark and pv3d for distrib w/ FLARManager (X) add easing (optional, set with flag, on by default) to transformation matrices (X) add z to FLARMarkerOutline.centerpoint (is there a native FP10 Point3D?) (X) can i get rid of downsampleRatio? it's a complicated concept, and could probably be automated by just specifying screen size and source size, sim to what i did in...? VideoDownsamplerTest? (-) mirroring FLARCameraSource messes with marker detection. rewrite to flip transmats if source is mirrored (?) (x) detect marker rotation for 2D tracking, and add to FLARMarker.as (x) x/y of FLARMarker.outline should already be scaled according to downsampleRatio when the FLARMarker is instantiated. (X) modify FLARCameraSource to output video at full-res, but downsample according to downsampleRatio. see VideoDownsamplerTest for this, or my modded WebcamFaceDetector. (X) specify settable delay (in frames) before removing a marker. (i.e. "stickier" marker detection.) make available via flarConfig.xml. (-) provide example that draws objects with pv3D this example should demonstrate addition and removal events with some special animation on each event. how bout a simple box that spins up out of the marker on addition, and down into nothing on removal? (-) add an article to sanflashcisco.com (X) contact alex about presenting next month: FLARToolkit maaash.jp (face detection) soulwire's motion detection (subtractive analysis) (X) recreate patterns at higher res; the 3x3 ones all get confused for one another. (X) get new version of FLARToolkit, with ratio params in FLARCode ctor. (X) test FLARPattern._patternToBorderRatio (with tarotaro's marker generator) (X) patternToBorderRatio is working, but only in FLARSingleMarkerDetector. it gets divided by 10, and then passed into FLARDynamicRatioColorPatt_O3 ctor. follow that through further to be sure it's getting used correctly. (X) figure out FLARMarker.direction, and add to method doc need to figure out what 0-3 map to (i think up.left.down.right, same as .pat file, but verify) (X) build new .pat files for all the new patterns i created (X) support different resolution pat files, not just 16x16 (will have to have error checking on patt load; i don't think FLARToolkit can handle different sizes in the same session.) it cannot. FLARMultiMarkerManager will throw a FLARException if any of the loaded patterns are not the same width/height as the first loaded pattern. edit this to be a more descriptive error. (X) support mirrored camera output [optimization opportunities] ( ) replace "jaggedArrays" with Vectors. try to eliminate multidimensionality as much as possible; FLARMatchPatt_Color_WITHOUT_PCA.evaluate, for example, uses a 3D array to compare patterns. ( ) convert Arrays to Vectors ( ) use PixelBender to run averaging (synchronously). ( ) create b/w-only tracking mode (3x speed increase here...) ( ) FLARBitmapDataReader is currently using BitmapData.getPixel. could use BitmapData.getVector instead, particularly in FLARBitmapDataReader.getPixelSet(). also, if there's a way to update a source image as Vector instead of BitmapData, may be gains to be had there as well...?