Difference between revisions of "Matlab Fanuc Kinect"

From ESE497 Wiki
Jump to navigationJump to search
 
(18 intermediate revisions by the same user not shown)
Line 1: Line 1:
−
Special thanks for Andrew Schoer, Joshua Remba, Pradosh Kharel and Ian Sekiguchi for contributing to this page.
+
Source Code for communication between Kinect Camera and Fanuc Robot
−
*[[media:MatlabFanucKinect.zip|MatlabFanucKinect.zip]]
+
 
−
* Software Description:
+
*Special thanks for Andrew Schoer, Joshua Remba, Pradosh Kharel and Ian Sekiguchi for contributing to this page.
 +
* Before you launch Matlab, as administrator:
 +
** Open a Command Prompt
 +
** cd c:\Program Files (x86)\Windows Resource Kits\Tools
 +
** Type C:\Program Files (x86)\Windows Resource Kits\Tools>ntrights -u ACCOUNTS\<wustl key> +r SeLockMemoryPrivilege
 +
* Use H:\* instead of \\warehouse.* for your Matlab working directory. If you browse to your home directory on the H: drive and double click on Kinect2SharedMem2Matlab.m this will happen automatically.
 +
*[http://classes.engineering.wustl.edu/ese497/BigFiles/MatlabFanucKinect.zip Zip file ]
 +
<!-- Zip file from ESE497_UGradResearch\MatlabFanucKinect -->
 +
Source Code Description:
 
*Calibration.m:  
 
*Calibration.m:  
−
% Script to calibrate the Kinect Camera to the Fanuc Robot
+
**% Script to calibrate the Kinect Camera to the Fanuc Robot
−
% Moves robot to 64 locations and locates X in Kinect image and then computes
+
**% Moves robot to 64 locations and locates X in Kinect image and then computes
−
% the transformation matrix between the Kinect and Robot. Creates CalibrationDone.mat
+
**% the transformation matrix between the Kinect and Robot. Creates TransMatrix.mat
−
% which contains transformation matrix T. Stores image data for post processing in  
+
**% which contains transformation matrix T. Stores image data for post processing in  
−
% calibrationInput.mat
+
**% calibrationInput.mat
−
 
 
 
*Kinect2SharedMem2Matlab.m:  
 
*Kinect2SharedMem2Matlab.m:  
−
% Example script to acquire images from the Kinect camera in Matlab
+
**% Example script to acquire images from the Kinect camera in Matlab
−
% -------------------------------------------------------------------------
+
**% -------------------------------------------------------------------------
−
% Kinect class uses a slightly modified Microsoft Kinect SDK application
+
**% Kinect class uses a slightly modified Microsoft Kinect SDK application
−
% DepthWithColor-D3 and SharedMemDLL.dll to stream images from the Kinect camera
+
**% DepthWithColor-D3 and SharedMemDLL.dll to stream images from the Kinect camera
−
% to this script. This script initializes the shared memory and then launches  
+
**% to this script. This script initializes the shared memory and then launches  
−
% DepthWithColor-D3.exe. DepthWithColor-D3 connects to the % shared memory  
+
**% DepthWithColor-D3.exe. DepthWithColor-D3 connects to the **% shared memory  
−
% and starts streaming RGB and depth images to it.  
+
**% and starts streaming RGB and depth images to it.  
−
% This is preferable to using the KinectForWindowsExample.m because
+
**% This is preferable to using the KinectForWindowsExample.m because
−
% DepthWithColor-D3 aligns the RGB and Depth images.
+
**% DepthWithColor-D3 aligns the RGB and Depth images.
−
 
+
*Matlab2FanucExample.m: Send a series of movements to the Fanuc Robot using TCP/IP. Run LabVIEWMatlabFanuc\LabVIEWMatlabFanuc.vi and LV_COM_FRM on the Fanuc Robot first.
−
 
+
**% Example script to move robot with Matlab commands. Runs on computer
−
*Matlab2Fanuc.m: Send a series of movements to the Fanuc Robot using TCP/IP. Run LabVIEWMatlabFanuc\LabVIEWMatlabFanuc.vi and LV_COM_FRM on the Fanuc Robot first.
+
**% connected to Robot or on a remote computer
−
% Example script to move robot with Matlab commands. Runs on computer
+
**% -------------------------------------------------------------------------
−
% connected to Robot or on a remote computer
+
**% Launch the teaching pendant program LV_COM_FRM before launching this script.
−
% -------------------------------------------------------------------------
+
**% See: http://classes.engineering.wustl.edu/ese497/index.php/Fanuc_How_Tos,_FAQ,_etc.#FAQ
−
% Launch the teaching pendant program LV_COM_FRM before launching this script.
+
**% for detailed instructions.
−
% See: http://classes.engineering.wustl.edu/ese497/index.php/Fanuc_How_Tos,_FAQ,_etc.#FAQ
+
**% -------------------------------------------------------------------------
−
% for detailed instructions.
+
**% Messages are sent from this script to the LabVIEWMatlabFanuc.exe to  
−
% -------------------------------------------------------------------------
+
**% communicate the new position of the robot. The first 4 bytes of  
−
% Messages are sent from this script to the LabVIEWMatlabFanuc.exe to  
+
**% the message are the length of the message in ASCII and that the message  
−
% communicate the new position of the robot. The first 4 bytes of  
+
**% is floating point numbers delimited by commas. You need to wait until it  
−
% the message are the length of the message in ASCII and that the message  
+
**% receives an 'Ack' message back from LabVIEW.
−
% is floating point numbers delimited by commas. You need to wait until it  
+
**% -------------------------------------------------------------------------
−
% receives an 'Ack' message back from LabVIEW.
+
**% If this is the computer that also communicates with the robot, then set
−
% -------------------------------------------------------------------------
+
**% host to 'localhost'. In this case, the script will launch  
−
% If this is the computer that also communicates with the robot, then set
+
**% LabVIEWMatlabFanuc.exe. This is the application that receives message
−
% host to 'localhost'. In this case, the script will launch  
+
**% over TCP/IP from this script and communicates the coordinates to the
−
% LabVIEWMatlabFanuc.exe. This is the application that receives message
+
**% program running on the robot.
−
% over TCP/IP from this script and communicates the coordinates to the
+
**% If this is NOT the computer that communicates with the robot, then set
−
% program running on the robot.
+
**% host to 'fanucomp.seasad.wustl.edu' or whatever the IP is for that
−
% If this is NOT the computer that communicates with the robot, then set
+
**% computer. You will have to manually run LabVIEWMatlabFanuc.exe on that
−
% host to 'fanucomp.seasad.wustl.edu' or whatever the IP is for that
+
**% computer and set the IP to this computer's address. You have to stop the  
−
% computer. You will have to manually run LabVIEWMatlabFanuc.exe on that
+
**% application to change the IP. The application also starts minimized so
−
% computer and set the IP to this computer's address. You have to stop the  
+
**% you'll have to find it in the taskbar.
−
% application to change the IP. The application also starts minimized so
+
**% -------------------------------------------------------------------------
−
% you'll have to find it in the taskbar.
+
**% Matlab to LabVIEW communication based on
−
% -------------------------------------------------------------------------
+
**% http://www.mathworks.com/matlabcentral/fileexchange/42567-matlab-and-labview-data-exchange-over-tcp-ip
−
% Matlab to LabVIEW communication based on
+
**% Modified to pre-pend the length the the xmit string and to return the
−
% http://www.mathworks.com/matlabcentral/fileexchange/42567-matlab-and-labview-data-exchange-over-tcp-ip
+
**% received message as a string
−
% Modified to pre-pend the length the the xmit string and to return the
+
**% -------------------------------------------------------------------------
−
% received message as a string
+
*TestCalibration.m:
−
% -------------------------------------------------------------------------
+
**% Script to test the calibration of the Kinect Camera to the Fanuc Robot
−
 
+
**% Moves robot to random locations and locates X in Kinect image.
−
 
+
**% The calculated position of the Fanuc, based on the Kinect and Transformation matrix,
 +
**% is then compared to the position that the Fanuc was sent to.
 
*Miscellaneous Functions
 
*Miscellaneous Functions
 
**Kinect.m:  
 
**Kinect.m:  
−
% Class for communicating with the Kinect Camera. Uses SharedMemDLL.dll and DepthWithColor-D3D.
+
***% Class for communicating with the Kinect Camera. Uses SharedMemDLL.dll and DepthWithColor-D3D.
 
**proj2cart.m: Convert from Kinect projection to cartesian coordinates. Best FOVx,FOVy is 63.1,49.6.
 
**proj2cart.m: Convert from Kinect projection to cartesian coordinates. Best FOVx,FOVy is 63.1,49.6.
−
% Converts from Camera Pixel Coordinates and Real World Depth to  
+
***% Converts from Camera Pixel Coordinates and Real World Depth to  
−
% RealWorld x,y,z
+
***% RealWorld x,y,z
−
** RGB Depth Registration.docx
+
***% In order to convert Kinect Coordinates to FANUC Coordinates:
−
% Document with pictures of depth registration using KinectForWindowsExample.m and my Kinect class
+
****% 1.) Convert Kinect Coordinates to Cartesian using proj2cart function
 +
****% 2.) Transponse Cartesian Coordinates to Column Vectors
 +
****% 3.) Append One as 4th element in Column Vector
 +
****% 4.) Multiply Column Vector(s) by Transform Matrix, T
 +
**Kinect Projection to Cartesian Conversion.docx
 +
***% Details of proj2cart.m function
 
** absor.m
 
** absor.m
−
% Tool for calibrating 2 coordinate systems from  
+
***% Tool for calibrating 2 coordinate systems from  
−
% Author: Matt Jacobson
+
***% Author: Matt Jacobson
−
% Copyright, Xoran Technologies, Inc.  http://www.xorantech.com
+
***% Copyright, Xoran Technologies, Inc.  http://www.xorantech.com
−
 
+
**CreateTemplates.m
 +
***% Creates templates for X and + images
 +
**DepthFilter.m
 +
***% Median filter for depth
 +
**Double2CSV.m
 +
***% Converts array of doubles to commma separated string
 +
**exchangeData.m
 +
***% transmits data over TCP/IP to LabVIEW program. Used to send robot coordintates.
 
** DepthWithColor-D3D
 
** DepthWithColor-D3D
−
% Slightly modified Microsft Kinect SDK example based on their example code DepthWithColor-D3D
+
***% Slightly modified Microsft Kinect SDK example based on their example code DepthWithColor-D3D
−
% This is preferable to the KinectForWindows.m example because it solves the RGB/Depth registration
+
***% This is preferable to the KinectForWindows.m example because it solves the RGB/Depth registration
−
% issues.
+
***% issues.
−
 
 
 
** LabVIEW2Fanuc
 
** LabVIEW2Fanuc
−
% LabVIEW source code and Application to pass commands from Matlab over TCP/IP to the Fanuc Robot using
+
***% LabVIEW source code and Application to pass commands from Matlab over TCP/IP to the Fanuc Robot using
−
% the NI Ethernet/IP toolkiit. LV_COM_FRM must be launched on the Teaching Pendant to complete the communication
+
***% the NI Ethernet/IP toolkiit. LV_COM_FRM must be launched on the Teaching Pendant to complete the communication
−
 
 
 
** SharedMem\SharedMemDLL
 
** SharedMem\SharedMemDLL
−
% Microsoft C++ built DLL to pass images in a large shared memory between 2 applications
+
***% Microsoft C++ built DLL to pass images in a large shared memory between 2 applications
−
 
 
 
** TeachingPendantProgram
 
** TeachingPendantProgram
−
% Source code for LV_COM_FRM that is launched from the teaching pendant to communicate with the LabVIEW2Fanuc Application
+
***% Source code for LV_COM_FRM that is launched from the teaching pendant to communicate with the LabVIEW2Fanuc Application
−
 
+
** RGB Depth Registration.docx
−
**CreateTemplates.m
+
***% Document with pictures of depth registration using KinectForWindowsExample.m and my Kinect class
−
% Creates templates for X and + images
 
−
**DepthFilter.m
 
−
% Median filter for depth
 
−
**Double2CSV.m
 
−
% Converts array of doubles to commma separated string
 
−
**exchangeData.m
 
−
% transmits data over TCP/IP to LabVIEW program. Used to send robot coordintates.
 
−
**Kinect Projection to Cartesian Conversion.docx
 
−
% Details of proj2cart.m function
 
−
 
 
−
 
 
−
*My Work
 
−
** readme.txt - this document
 
−
**ApproachVector.m
 
−
**FindMarkers.m
 
−
**proj2PointCloud.m
 
−
**KinectForWindowsExample_Modified.m: Matlab example code modified for near mode (40 cm)
 
−
**markers1.mat: Kinect images of pile of markers
 
−
**RGB Depth Registration.docx
 
−
*FindBestFOVs.m: Brute force search for Field of View constants for Kinect camera by minimizing the error of the transformation matrix
 
−
** Coordinate Transformations.docx
 
−
** ExcerptFromDigitalImageProcessing.pdf
 
−
** IMG_1188.JPG
 
−
**Kinect2SharedMem2Matlab.mat
 
−
** Adobe Premiere - video of Robot picking up marker
 

Latest revision as of 17:17, 13 January 2017

Source Code for communication between Kinect Camera and Fanuc Robot

  • Special thanks for Andrew Schoer, Joshua Remba, Pradosh Kharel and Ian Sekiguchi for contributing to this page.
  • Before you launch Matlab, as administrator:
    • Open a Command Prompt
    • cd c:\Program Files (x86)\Windows Resource Kits\Tools
    • Type C:\Program Files (x86)\Windows Resource Kits\Tools>ntrights -u ACCOUNTS\<wustl key> +r SeLockMemoryPrivilege
  • Use H:\* instead of \\warehouse.* for your Matlab working directory. If you browse to your home directory on the H: drive and double click on Kinect2SharedMem2Matlab.m this will happen automatically.
  • Zip file

Source Code Description:

  • Calibration.m:
    • % Script to calibrate the Kinect Camera to the Fanuc Robot
    • % Moves robot to 64 locations and locates X in Kinect image and then computes
    • % the transformation matrix between the Kinect and Robot. Creates TransMatrix.mat
    • % which contains transformation matrix T. Stores image data for post processing in
    • % calibrationInput.mat
  • Kinect2SharedMem2Matlab.m:
    • % Example script to acquire images from the Kinect camera in Matlab
    • % -------------------------------------------------------------------------
    • % Kinect class uses a slightly modified Microsoft Kinect SDK application
    • % DepthWithColor-D3 and SharedMemDLL.dll to stream images from the Kinect camera
    • % to this script. This script initializes the shared memory and then launches
    • % DepthWithColor-D3.exe. DepthWithColor-D3 connects to the **% shared memory
    • % and starts streaming RGB and depth images to it.
    • % This is preferable to using the KinectForWindowsExample.m because
    • % DepthWithColor-D3 aligns the RGB and Depth images.
  • Matlab2FanucExample.m: Send a series of movements to the Fanuc Robot using TCP/IP. Run LabVIEWMatlabFanuc\LabVIEWMatlabFanuc.vi and LV_COM_FRM on the Fanuc Robot first.
    • % Example script to move robot with Matlab commands. Runs on computer
    • % connected to Robot or on a remote computer
    • % -------------------------------------------------------------------------
    • % Launch the teaching pendant program LV_COM_FRM before launching this script.
    • % See: http://classes.engineering.wustl.edu/ese497/index.php/Fanuc_How_Tos,_FAQ,_etc.#FAQ
    • % for detailed instructions.
    • % -------------------------------------------------------------------------
    • % Messages are sent from this script to the LabVIEWMatlabFanuc.exe to
    • % communicate the new position of the robot. The first 4 bytes of
    • % the message are the length of the message in ASCII and that the message
    • % is floating point numbers delimited by commas. You need to wait until it
    • % receives an 'Ack' message back from LabVIEW.
    • % -------------------------------------------------------------------------
    • % If this is the computer that also communicates with the robot, then set
    • % host to 'localhost'. In this case, the script will launch
    • % LabVIEWMatlabFanuc.exe. This is the application that receives message
    • % over TCP/IP from this script and communicates the coordinates to the
    • % program running on the robot.
    • % If this is NOT the computer that communicates with the robot, then set
    • % host to 'fanucomp.seasad.wustl.edu' or whatever the IP is for that
    • % computer. You will have to manually run LabVIEWMatlabFanuc.exe on that
    • % computer and set the IP to this computer's address. You have to stop the
    • % application to change the IP. The application also starts minimized so
    • % you'll have to find it in the taskbar.
    • % -------------------------------------------------------------------------
    • % Matlab to LabVIEW communication based on
    • % http://www.mathworks.com/matlabcentral/fileexchange/42567-matlab-and-labview-data-exchange-over-tcp-ip
    • % Modified to pre-pend the length the the xmit string and to return the
    • % received message as a string
    • % -------------------------------------------------------------------------
  • TestCalibration.m:
    • % Script to test the calibration of the Kinect Camera to the Fanuc Robot
    • % Moves robot to random locations and locates X in Kinect image.
    • % The calculated position of the Fanuc, based on the Kinect and Transformation matrix,
    • % is then compared to the position that the Fanuc was sent to.
  • Miscellaneous Functions
    • Kinect.m:
      • % Class for communicating with the Kinect Camera. Uses SharedMemDLL.dll and DepthWithColor-D3D.
    • proj2cart.m: Convert from Kinect projection to cartesian coordinates. Best FOVx,FOVy is 63.1,49.6.
      • % Converts from Camera Pixel Coordinates and Real World Depth to
      • % RealWorld x,y,z
      • % In order to convert Kinect Coordinates to FANUC Coordinates:
        • % 1.) Convert Kinect Coordinates to Cartesian using proj2cart function
        • % 2.) Transponse Cartesian Coordinates to Column Vectors
        • % 3.) Append One as 4th element in Column Vector
        • % 4.) Multiply Column Vector(s) by Transform Matrix, T
    • Kinect Projection to Cartesian Conversion.docx
      • % Details of proj2cart.m function
    • absor.m
      • % Tool for calibrating 2 coordinate systems from
      • % Author: Matt Jacobson
      • % Copyright, Xoran Technologies, Inc. http://www.xorantech.com
    • CreateTemplates.m
      • % Creates templates for X and + images
    • DepthFilter.m
      • % Median filter for depth
    • Double2CSV.m
      • % Converts array of doubles to commma separated string
    • exchangeData.m
      • % transmits data over TCP/IP to LabVIEW program. Used to send robot coordintates.
    • DepthWithColor-D3D
      • % Slightly modified Microsft Kinect SDK example based on their example code DepthWithColor-D3D
      • % This is preferable to the KinectForWindows.m example because it solves the RGB/Depth registration
      • % issues.
    • LabVIEW2Fanuc
      • % LabVIEW source code and Application to pass commands from Matlab over TCP/IP to the Fanuc Robot using
      • % the NI Ethernet/IP toolkiit. LV_COM_FRM must be launched on the Teaching Pendant to complete the communication
    • SharedMem\SharedMemDLL
      • % Microsoft C++ built DLL to pass images in a large shared memory between 2 applications
    • TeachingPendantProgram
      • % Source code for LV_COM_FRM that is launched from the teaching pendant to communicate with the LabVIEW2Fanuc Application
    • RGB Depth Registration.docx
      • % Document with pictures of depth registration using KinectForWindowsExample.m and my Kinect class
Retrieved from "http://classes.engineering.wustl.edu/ese497/index.php?title=Matlab_Fanuc_Kinect&oldid=2834"