Started Labbook 2019.
Labbook IMAV
December 13, 2018
- Received the RealSense D435i. Tested it on nb-dual under Windows, but the depth_example.m, but this fails on:
Error using realsense.colorizer/colorize (line 17)
Expected input number 2, depth, to be a depth_frame
Error in depth_example (line 26)
color = colorizer.colorize(depth);
- Under Ubuntu 18.04, the RealSense Viewer (v2.17.0) could recognize the D435i and the motion module. The Accel stream is updated with 63 FPS, the Gyro stream with 200 FPS:
- Under Ubuntu the matlab-wrapper can access the depth_frames, although the frame doesn't make sense (should have been a picture of me):
December 4, 2018
- realsense-viewer (v2.17.0) still works on nb-dual under Ubuntu 18.04
- Continue with matlab wrapper. Matlab license in now activated. In Create librealsense from source is described that you should create a build directory first. Run sudo apt-get install libglfw3-dev libgl1-mesa-dev libglu1-mesa-dev. Then make .. -DBUILD_MATLAB_BINDINGS:BOOL=ON works. Back to ~/git/librealsense. Did make. Received some warnings of deprecated dynamic exception specifications, but further the make seems to work. Also sudo make install works, but the CMakelist.txt of the matlab/wrapper indicates #TODO: additionally, the compiled mex file and all the .m files need to be copied into a folder called `+realsense` at the install site
- Performed the install manually in ~/Documents/MATLAB/+realsense. Calling realsense.depth_example failed on Invalid MEX-file '/home/arnoud/Documents/MATLAB/+realsense/librealsense_mex.mexa64': Gateway function is missing.
- mexa64 is created with command /usr/bin/c++ -fPIC -fPIC -pedantic -g -Wno-missing-field-initializers -Wno-switch -Wno-multichar -Wsequence-point -Wformat-security -mssse3 -pthread -std=c++11 -std=c++11 -shared -Wl,-soname,librealsense_mex.mexa64 -o librealsense_mex.mexa64 CMakeFiles/librealsense_mex.dir/librealsense_mex.cpp.o CMakeFiles/librealsense_mex.dir/Factory.cpp.o -L/usr/local/MATLAB/R2018b/bin/glnxa64 -Wl,-rpath,/usr/local/MATLAB/R2018b/bin/glnxa64:/home/arnoud/git/librealsense -lmx -lmex ../../librealsense2.so.2.17.0 -lmat (make with VERBOSE=1)
- Making librealsense_mex.mexa64 from the commandline with mex librealsense_mex.cpp Factory.cpp /usr/local/lib/librealsense2.so solved this issue.
December 2, 2018
- Continued with trying to get the matlab interface running. Followed the matlab wrapper instructions, but cmake failed. First my matlab registration didn't work (not connected by vpn?), and cmake should be run from a higher directory?!.
- Saw that the Matlab developer was already installed under Windows with the SDK. It is default installed at C:\Program Files (x86)\Intel RealSense SDK 2.0\matlab\+realsense.
- Added the path to matlab (without +realsense. depth_example.m failed because Frame didn't arrived within 5000, which is the same error as rviz. Tried the viewer, which complained about invalid value in rs2_get_option(option:Output Trigger Enabled). Maybe I shouldn't have done the firmware update :-) The Depth stream works, the color stream fails. Reconnecting the camera solves this.
- Running the matlab depth_example gave the error: camera already streaming. Then I receive a licensing error. Strange, I seem still to work with the R2018b prelease.
- Downloading the R2018b release for Windows. Complains that already an previous version was installed in C:\Programs\MATLAB\R2018b, including the new Deep Learning Toolbox. Installer is also removing the Communication Toolbox and Deep Learning Toolbox 11.2.
- After upgrading (license will work to the end of 2018), I finally could run the depth_example.m from my matlab-prompt under Windows on nb-dual:
- There is also a rosbag example in the +realsense directory.
November 30, 2018
- Continue with testing the D435 at nb-dual, now under Ubuntu 18.04 (bionic).
- The realsense-viewer works nicely. Continued with realsense2.
- The realsense2_camera only contains instructions for ROS Kinetic, with bionic I should do the installation with ROS Melodic.
- Had some problems with catkin_ws still pointing to kinetic, so moved that directory to catkin_kinetic and created a fresh catkin_ws. catkin_make clean complains on several missing packages, but by installing sudo apt install ros-melodic-*missing-package* I could solve that. At the end catkin complains that the realsense SDK is not installed, which is true (only installed the viewer).
- Installed sudo apt-get install librealsense2-dev, which creates /usr/include/librealsense2/hpp. Yet, there seems to be a version conflict, because first process could not be found in the processing_block scope (which I could repair with using an inbetween step with rs2::filter. Yet, the processing_block also has no corresponding on_frame_function.
- Yet, the rs_processing.hpp is the same in /usr/include and ~/git/librealsense2. Removing line 643 solves the build problem (but code doesn't applies any filter).
- Continued with usage instructions and installed sudo apt-get install ros-melodic-rgbd-launch. Started roslaunch realsense2_camera view_d435_model.launch. But this failed on missing xacro resource. Installed sudo apt-get install ros-melodic-xacro. Process started, but couldn't launch robot_state_publisher and rviz. After I installed also those, view_d435_model worked, but only showed the sensor and its mesh. Launching rs_camera.launch or rs_rgbd.launch gave an error that no realsense device was found. Also the realsense-viewer didn't saw the sensor anymore!? I am trying a reboot. Received an kernel-panic!
- After the restart realsense-viewer worked (had even a newer version). Yet, with rs_camera.launch I don't get any rviz. The log reports that the sensor is up, but the backend-v4l2 complaints that no Frames did arrive within 5 seconds. Also the view_d435_model failed, but that is because the urdf library from src is not copied to catkin_ws/install/share/realsense2_camera/urdf. Copying the urdf and meshes directory solved this issue.
- The missing stream is probably the missing line with filter->process(). The downloaded code was realsense-2.1.1.tar.gz.
November 29, 2018
- With a USB-3.1 cable, I was able to perform the Firmware update from v5.8 to v5.10.
- Added C:\Programs\Microsoft\VisualStudio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin to my systems PATH. Now I was able to perform cmake -DBUILD_SHARED_LIBS:BOOL=OFF -DBUILD_WITH_TM2:BOOL=ON in git\librealsense with my VS2017 command prompt.
- Continued with matlab wrapper instructions, but there is no realsense2 target, so step 5 has to be skipped. Step 6 fails on missing mex.h file.
- Added C:\Programs\MATLAB\R2018b\extern\include and C:\Programs\MATLAB\R2018b\extern\lib\win64\microsoft to VC++ directories of librealsense_mex project, but program now fails on missing realsense2.lib (step 5).
- A librealsense2.sln can be found two directories up. That solution contains many projects, including examples, external projects like libusb, but most important the library realsense2. Doing a build on the full solution gave 6 succeeded projects, 23 failed and 2 skipped.
- Performing a rebuild on the realsense2 library (both Debug and Release) created the library in git\librealsense\Release
- Was also missing realsense-file, libtm and libusb. After adding those, librealsense_mex complains that those libraries are win32, while librealsense_mex is x64. Unfortunatelly, librealsense doesn't have x64 platform settings, and copying those results in "module machine type 'x64' conflicts with target machine type 'x86'".
- Try solution as suggested by Steve Lionel and remove all /Machine switches. Making the librealsense_mex with win32 is not an option because no win32 libraries are provided by Matlab. Found the problem in Librarian->Additional Options, where explicitly Machine X86 was defined, which conflicts with the Target Machine option further down.
November 28, 2018
- Running the provided code samples with the Realsense. HelloWorld fails in wait_for_frames, on an exception h=-858993460, which is not handled, but seems to be a memory error (frame too large?).
- Trying instead code snippet:
device d;
std::vector sensors =d.query_sensors();
for (rs2::sensor element : sensors) {
std::string module_name = element.get_info(RS2_CAMERA_INFO_NAME);
}
still fails with a runtime error, probably because I first need a pipe and a context.
- The pipe is not needed, the context is:
rs2::context cxt;
auto device = cxt.query_devices().back();
std::vector sensors = device.query_sensors();
std::cout << "Device has " << sensors.size() << " sensors. \n";
for (rs2::sensor element : sensors) {
std::string module_name = element.get_info(RS2_CAMERA_INFO_NAME);
std::cout << "\nCamera Name: " << module_name << "\n";
std::string product_name = element.get_info(RS2_CAMERA_INFO_PRODUCT_ID);
std::cout << "Product ID: " << product_name << "\n";
}
this code works. Ran this code also on the RS200, but there it fails on the call query_devices().
- Saw an issue on the D435 and USB3.2. Yet, with my USB-C cable only USB2.0 is recognized (and no RGB). This user also uses the viewer v2.16.3, while I downloaded last week v2.16.1. Used instead of the flexible USB-C cable the Thunderbolt cable, and finally I had a depth image:
- Checked the other part of the code. The USB type is still a get_info which shouldn't be asked, but the number of sensors is now two, and with the RGB sensor active also the hello_world application works (pipe starts and distance to nearest obstacle is reported). You could give the pipe.start() a configuration as parameter, but the hello_world also works with the default config.
- Cloning github's librealsense, which contains a wrapper for Matlab.
- Should try to use cmake, but even my VisualStudio prompt didn't know this command.
- Followed the instructions and run .\realsense_metadata_win10.ps1 -op install to add meta-data in the Window10 registry.
November 27, 2018
- Received the RealSense 435 (without IMU). Interface is USB-C, but a USB-C to USB-B is provided. Connected the R435 to nb-dual and tested the RealSense viewer under Windows10 (via KY-888). Got a Infrared image in 2D, didn't see any point cloud in 3D.
- Downloaded latest firmware and FirmwareUpdate Tool, but device disconnects before update can be performed.
November 22, 2018
- Interesting new 3D 360 degree camera from DreamVu Inc.
- Checked Julian's progress. Detectron, Facebook AIR's Research software system, is trained on COCO dataset, which contains 4562 pictures of dogs. There is at least one image from above:
- Installing realsense packages on nb-ros (Ubuntu 14.04 Trusty) failed. Not clear if this is due to a missing key or that trusty directories do not exist.
- On Ubuntu Tutorial, the suggestion is made to install from source.
- Found in this post, that for Trusty Tahr you should use the xenial repository.
- It is even hard to find the reference manual. There is a developer zone, but that site only contains blogs. The best option seems to be SDK 2016 R2 Documentation.
- With using xenial for trusty the command sudo apt-get install librealsense2-dkms works. sudo apt-get install librealsense2utils fails on libglfw3, which is a known issue.
- Followed the developer installation instructions and run scripts/install_glfw3.sh. Tried again with sudo apt-get install scripts/install_glfw3.sh, because the dependency was still not resolved.
November 21, 2018
- Pre-ordered an RealSenseD435i.
- Followed Windows Installation Instructions for the RealSense 200. With my DA300 hub the plugin was not detected, with KY-888 USB-hub the drivers were installed and the sensor became visible in the DeviceManager:
- Still, the R200 was not visible as Source in the RealSenseViewer. Same is truth for the Depth Quality Tool.
- Installed the tools in packages\RealSense, but had no choice on the location of the SDK, so the SDK is installed in C:\Program Files (x86)\Intel RealSense SDK 2.0\, including a copy of the tools.
November 19, 2018
- ROS has two realsense packages: realsense and realsense2. The first seems a bit older and supports the R200, F200, SR300 and D400 cameras. The later supports explicitly the SR300, D415 and D435 on ROS Kinetic on Ubuntu 16.04
November 15, 2018
- xioctl error already occurs when starting realsense-viewer without the realsense connected, so it seems that the viewer try to connect to the laptop's webcam.
- In this post there is a lot of discussion on uvc-streams meta-data and kernel patches. Discussion is on kernel v4.16+, while my 18.04 has v4.15 according to uname -r. Instruction on upgrading the kernel can be found here, but that seems a last resort.
- Checking lsusb -v | grep Intel gave this information:
Couldn't open device, some information will be missing
Bus 004 Device 004: ID 8086:0a80 Intel Corp.
idVendor 0x8086 Intel Corp.
iManufacturer 1 Intel Corp
iProduct 2 Intel RealSense 3D Camera R200
iFunction 4 Intel(R) RealSense(TM) 3D Camera (R200) Left-Right
iInterface 4 Intel(R) RealSense(TM) 3D Camera (R200) Left-Right
iInterface 4 Intel(R) RealSense(TM) 3D Camera (R200) Left-Right
iFunction 5 Intel(R) RealSense(TM) 3D Camera (R200) Depth
iInterface 5 Intel(R) RealSense(TM) 3D Camera (R200) Depth
iInterface 5 Intel(R) RealSense(TM) 3D Camera (R200) Depth
iFunction 6 Intel(R) RealSense(TM) 3D Camera (R200) RGB
iInterface 6 Intel(R) RealSense(TM) 3D Camera (R200) RGB
iInterface 6 Intel(R) RealSense(TM) 3D Camera (R200) RGB
Running sudo lsusb solved the couldn't open device message.
- Running dmesg | tail -80 showed:
[195080.086256] WARNING: CPU: 0 PID: 32415 at /var/lib/dkms/librealsense2-dkms/1.3.3/build/4.15.0/drivers/media/v4l2-core/v4l2-ioctl.c:1297 v4l_enum_fmt+0x1117/0x1430 [videodev]
[195080.086258] Modules linked in: uvcvideo(OE) videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_core videodev(OE) media hid_sensor_accel_3d(OE) hid_sensor_gyro_3d(OE) hid_sensor_trigger industrialio_triggered_buffer kfifo_buf hid_sensor_iio_common hid_sensor_hub industrialio btrfs zstd_compress xor raid6_pq ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs libcrc32c cpuid hid_generic usbhid cdc_ether usbnet r8152 mii rfcomm ccm pci_stub vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) thunderbolt cmac bnep snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic arc4 binfmt_misc nls_iso8859_1 snd_soc_skl snd_soc_skl_ipc snd_hda_ext_core snd_soc_sst_dsp snd_soc_sst_ipc snd_soc_acpi snd_soc_core snd_compress hid_multitouch ac97_bus snd_pcm_dmaengine dell_laptop dell_smbios_smm dcdbas
[195080.086353] intel_rapl snd_hda_intel x86_pkg_temp_thermal intel_powerclamp snd_hda_codec snd_hda_core coretemp snd_hwdep kvm_intel snd_pcm kvm ath10k_pci irqbypass snd_seq_midi ath10k_core snd_seq_midi_event crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc ath mac80211 snd_rawmidi aesni_intel btusb dell_smbios_wmi dell_wmi aes_x86_64 btrtl dell_smbios crypto_simd btbcm glue_helper btintel cryptd snd_seq intel_cstate bluetooth intel_rapl_perf input_leds snd_seq_device ecdh_generic cdc_acm snd_timer cfg80211 joydev intel_wmi_thunderbolt rtsx_pci_ms idma64 wmi_bmof serio_raw dell_wmi_descriptor memstick virt_dma snd soundcore mei_me intel_lpss_pci processor_thermal_device mei shpchp intel_lpss intel_pch_thermal intel_soc_dts_iosf int3400_thermal acpi_thermal_rel int3403_thermal intel_hid int340x_thermal_zone
[195080.086445] acpi_pad mac_hid sparse_keymap sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables autofs4 rtsx_pci_sdmmc i915 i2c_algo_bit drm_kms_helper syscopyarea nvme sysfillrect psmouse sysimgblt fb_sys_fops nvme_core drm rtsx_pci wmi i2c_hid hid video pinctrl_sunrisepoint [last unloaded: videobuf2_memops]
[195080.086491] CPU: 0 PID: 32415 Comm: realsense-viewe Tainted: G W OE 4.15.0-36-generic #39-Ubuntu
[195080.086494] Hardware name: Dell Inc. XPS 13 9370/0F6P3V, BIOS 1.5.1 08/09/2018
[195080.086507] RIP: 0010:v4l_enum_fmt+0x1117/0x1430 [videodev]
- Continue with v4l2-ctl, as June 10, 2017.
November 14, 2018
- Tried to install RealSense SDK with the instructions from Intel. Those are outdated (xenial = 16.04).
- The instructions at github are more recent, but still fail because amazon aws denies access.
- I am not the only one with this problem. Adding sudo apt-key adv --keyserver keys.gnupg.net --recv-key 6F3EFCDE works, but still not trusted.
- Removing the old records, as suggested by github, solves this.
- Could install the demos with sudo apt-get install librealsense2-dkms and sudo apt-get install librealsense2-utils.
- Verified the installation with realsense-viewer. The viewer starts but asks to connect a RealSense camera.
- The viewer gives the following error:
14/11 17:57:52,937 ERROR [140355706025728] (types.h:180) xioctl(VIDIOC_G_CTRL) failed Last Error: Invalid argument
14/11 17:57:52,937 WARNING [140355706025728] (sensor.cpp:722) Exception was thrown when inspecting properties of a sensor
14/11 17:57:52,937 ERROR [140355706025728] (types.h:180) xioctl(VIDIOC_G_CTRL) failed Last Error: Invalid argument
14/11 17:57:52,938 WARNING [140355706025728] (sensor.cpp:722) Exception was thrown when inspecting properties of a sensor
14/11 17:57:53,204 ERROR [140356099080064] (tm-context.cpp:34) Failed to create TrackingManager
August 28, 2018
- Trying to create an obstacle race with one of the Mini drones. Using the Mambo. Downloaded FreeFlight mini. Drone requested an update (from Mambo v2.6.6 to v3.0.17)
- The Flypath is coupled to FreeFlight mini, and works quite good. Turning is left index + left jointpad. Forward backwards is right jointpad.
- Replaced a white wing of Orad, it is now flying safely.
- For the Jumping drones I need another app. I was not able to connect to the TukTuk. I also don't see it on BlueTooth, should I connect with Wifi?
- Connecting to the Jumping Sumo worked, why is the Tuk Tuk not visible? Driving around was really hard, because the Sumo is very responsive (but speed is also on 100%). The FlyPath doesn't work.
Previous Labbook
Related Labbooks