Mitchell Doughty

Mitchell Doughty

Mitchell Doughty is a PhD student in Medical Biophysics at the University of Toronto. His research interests focus on improving image-guidance in cardiac procedures using augmented reality.

ArUco marker tracking on the HoloLens

1 minute read

Published:

The ArUcoDetectionHoloLens-Unity sample enables marker detection and tracking using the HoloLens photo-video camera and the ArUco library in OpenCV.

Detected marker visualized through the HoloLens display

Marker detection is an important part in implementing an augmented reality solution; enabling the relation of virtual content to real objects in the scene. This occurs through establishing point correspondences between 3D world points and 2D image points and having knowledge of the intrinsic parameters of a camera, K. With known camera intrinsics, 3D world points and corresponding 2D projections in the image systems can be related through the perspective-n-point relation allowing for estimation of camera pose (position and orientation), [R t].

Once the pose of the calibrated camera is known, it becomes possible to align virtual content with real content in the scene.

Running the sample

ArUco marker tracking is made available for use in Unity through IL2CPP Windows Runtime support.

Incorporates:

Requirements

ArUco Detection Sample

  1. Open HoloLensForCV sample in VS2017 and install included OpenCV.UWP.411 NuGet package to HoloLensForCV project
  2. Build the HoloLensForCV project (x86, Debug or Release)
  3. Copy all output files from HoloLensForCV output path (dlls and HoloLensForCV.winmd) to the Assets->Plugins->x86 folder of the ArUcoDetectionHoloLensUnity project
  4. Open ArUcoDetectionHoloLensUnity Unity project and build using IL2CPP, ensure that unsafe code is allowed under Unity Player Settings->Other Settings
  5. Open VS solution, build then deploy to device
  6. When looking at an aruco marker from the correct ArUco dictionary: this sample uses the 6x6 (50, 100, 250, 500) dictionary by default, a virtual cube, which is scaled to the size of the ArUco marker, should appear on top of the marker

You May Also Enjoy

Bounding box drawing utility in Unity

3 minute read

Published:

I was surpised to find that there is no simple method in Unity to draw a rectangle (or bounding box) on a canvas. This project extends the Texture2D methods in Unity, providing a basic sample which demonstrates how to draw lines between specified 2D coordinates on a canvas and construct a bounding box, complete with label and confidence, around a cat! See the project on my personal Github repo for the implementation.