Matlab Fanuc Kinect

Special thanks for Andrew Schoer, Joshua Remba, Pradosh Kharel and Ian Sekiguchi for contributing to this page.

• MatlabFanucKinect.zip
• Software Description:
• Calibration.m:

% Script to calibrate the Kinect Camera to the Fanuc Robot % Moves robot to 64 locations and locates X in Kinect image and then computes % the transformation matrix between the Kinect and Robot. Creates CalibrationDone.mat % which contains transformation matrix T. Stores image data for post processing in % calibrationInput.mat

• Kinect2SharedMem2Matlab.m:

% Example script to acquire images from the Kinect camera in Matlab % ------------------------------------------------------------------------- % Kinect class uses a slightly modified Microsoft Kinect SDK application % DepthWithColor-D3 and SharedMemDLL.dll to stream images from the Kinect camera % to this script. This script initializes the shared memory and then launches % DepthWithColor-D3.exe. DepthWithColor-D3 connects to the % shared memory % and starts streaming RGB and depth images to it. % This is preferable to using the KinectForWindowsExample.m because % DepthWithColor-D3 aligns the RGB and Depth images.

• Matlab2Fanuc.m: Send a series of movements to the Fanuc Robot using TCP/IP. Run LabVIEWMatlabFanuc\LabVIEWMatlabFanuc.vi and LV_COM_FRM on the Fanuc Robot first.

% Example script to move robot with Matlab commands. Runs on computer % connected to Robot or on a remote computer % ------------------------------------------------------------------------- % Launch the teaching pendant program LV_COM_FRM before launching this script. % See: http://classes.engineering.wustl.edu/ese497/index.php/Fanuc_How_Tos,_FAQ,_etc.#FAQ % for detailed instructions. % ------------------------------------------------------------------------- % Messages are sent from this script to the LabVIEWMatlabFanuc.exe to % communicate the new position of the robot. The first 4 bytes of % the message are the length of the message in ASCII and that the message % is floating point numbers delimited by commas. You need to wait until it % receives an 'Ack' message back from LabVIEW. % ------------------------------------------------------------------------- % If this is the computer that also communicates with the robot, then set % host to 'localhost'. In this case, the script will launch % LabVIEWMatlabFanuc.exe. This is the application that receives message % over TCP/IP from this script and communicates the coordinates to the % program running on the robot. % If this is NOT the computer that communicates with the robot, then set % host to 'fanucomp.seasad.wustl.edu' or whatever the IP is for that % computer. You will have to manually run LabVIEWMatlabFanuc.exe on that % computer and set the IP to this computer's address. You have to stop the % application to change the IP. The application also starts minimized so % you'll have to find it in the taskbar. % ------------------------------------------------------------------------- % Matlab to LabVIEW communication based on % http://www.mathworks.com/matlabcentral/fileexchange/42567-matlab-and-labview-data-exchange-over-tcp-ip % Modified to pre-pend the length the the xmit string and to return the % received message as a string % -------------------------------------------------------------------------

• Miscellaneous Functions
  • Kinect.m:

% Class for communicating with the Kinect Camera. Uses SharedMemDLL.dll and DepthWithColor-D3D.

• • proj2cart.m: Convert from Kinect projection to cartesian coordinates. Best FOVx,FOVy is 63.1,49.6.

% Converts from Camera Pixel Coordinates and Real World Depth to % RealWorld x,y,z

• • RGB Depth Registration.docx

% Document with pictures of depth registration using KinectForWindowsExample.m and my Kinect class

• • absor.m

% Tool for calibrating 2 coordinate systems from % Author: Matt Jacobson % Copyright, Xoran Technologies, Inc. http://www.xorantech.com

• • DepthWithColor-D3D

% Slightly modified Microsft Kinect SDK example based on their example code DepthWithColor-D3D % This is preferable to the KinectForWindows.m example because it solves the RGB/Depth registration % issues.

• • LabVIEW2Fanuc

% LabVIEW source code and Application to pass commands from Matlab over TCP/IP to the Fanuc Robot using % the NI Ethernet/IP toolkiit. LV_COM_FRM must be launched on the Teaching Pendant to complete the communication

• • SharedMem\SharedMemDLL

% Microsoft C++ built DLL to pass images in a large shared memory between 2 applications

• • TeachingPendantProgram

% Source code for LV_COM_FRM that is launched from the teaching pendant to communicate with the LabVIEW2Fanuc Application

• • CreateTemplates.m

% Creates templates for X and + images

• • DepthFilter.m

% Median filter for depth

• • Double2CSV.m

% Converts array of doubles to commma separated string

• • exchangeData.m

% transmits data over TCP/IP to LabVIEW program. Used to send robot coordintates.

• • Kinect Projection to Cartesian Conversion.docx

% Details of proj2cart.m function

• My Work
  • readme.txt - this document
  • ApproachVector.m
  • FindMarkers.m
  • proj2PointCloud.m
  • KinectForWindowsExample_Modified.m: Matlab example code modified for near mode (40 cm)
  • markers1.mat: Kinect images of pile of markers
  • RGB Depth Registration.docx
• FindBestFOVs.m: Brute force search for Field of View constants for Kinect camera by minimizing the error of the transformation matrix
  • Coordinate Transformations.docx
  • ExcerptFromDigitalImageProcessing.pdf
  • IMG_1188.JPG
  • Kinect2SharedMem2Matlab.mat
  • Adobe Premiere - video of Robot picking up marker