DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This communication is responsive to the correspondence filled on 07/08/2025.
Claims 1-23 are presented for examination.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the claims at issue are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); and In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on a nonstatutory double patenting ground provided the reference application or patent either is shown to be commonly owned with this application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO internet Web site contains terminal disclaimer forms which may be used. The filing date of the application will determine what form should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission.
Claims 1 are rejected on the ground of nonstatutory obviousness-type double patenting as being unpatentable over claims 1 of US Pat. 12375791 B2.
Even though instant application does not claim “wherein the computing system and the image sensors are configured to operate together to measure a relative position of an object in images captured by the image sensors to provide the underwater depth perception, wherein the computing system is configured to select a feature between two image sensors of the image sensors to calculate a distance between the selected feature and the object and to provide the underwater depth perception, wherein the computing system is configured to perform the measurement of the relative position of the object in the images using triangulation, wherein the two sensors are configured to observe the object according to respective rays beginning at respective starting points at the two sensors and extending towards a selected point on a surface of the object, wherein the computing system is configured to use the respective starting points of the rays and the selected point on the surface of the object to define a spatial triangle for the triangulation, wherein the computing system is configured to: use the known distance between the two sensors as a base of the spatial triangle; determine angles between the base and the respective rays corresponding to the respective observations of the object; use the determined angles to determine the intersection point of the respective rays to provide a spatial coordinate of the selected point on the surface of the object according to triangular relations; and use triangular relations to determine the distance between the selected feature and the point on the surface of the object and to provide the underwater depth perception, wherein the computing system is configured to generate an underwater three-dimensional model based on the data captured by the respective image sensors, wherein the model comprises a plurality of distances between the selected feature and objects underwater, and wherein the plurality of distances between the selected feature and the objects underwater comprise the determined distance between the selected feature and the point on the surface of the object.”, however not claiming this does not provide instant application a patentable distinction. Because lack of limitation makes the claim broad obvious variation of US Pat. 12375791 B2.
Instant Application 19/262,992
US Pat. 12375791 B2
1. A system, comprising:
two image sensors,
arranged on different axes and configured to complement each other,
and each one of the image sensors comprising any type of sensor that captures an image based on reflected or emitted electromagnetic radiation or mechanical waves;
a computing system, configured to provide underwater depth perception based on data captured by the two image sensors
a submersible device, comprising a holder,
the holder configured to hold the two image sensors and the computing system,
and wherein computing by the computing system to provide the underwater depth perception occurs onboard the submersible device.
and triangulation, the triangulation using:
(1) respective observations of an object captured by the two image sensors, which is part of the captured data,
(2) a known distance between the two image sensors that defines a base, and
(3) respective angles between the base and rays corresponding to the respective
observations of the object; and
1. A system, comprising:
image sensors,
arranged on different axes and configured to complement each other,
and each one of the image sensors comprising any type of sensor that captures an image based on reflected or emitted electromagnetic radiation or mechanical waves;
a computing system, configured to provide underwater depth perception based on data captured by the image sensors;
and a submersible device, comprising a holder,
the holder configured to hold the image sensors and the computing system, and
wherein computing by the computing system to provide the underwater depth perception occurs onboard the submersible device,
wherein the computing system and the image sensors are configured to operate together to measure a relative position of an object in images captured by the image sensors to provide the underwater depth perception, wherein the computing system is configured to select a feature between two image sensors of the image sensors to calculate a distance between the selected feature and the object and to provide the underwater depth perception, wherein the computing system is configured to perform the measurement of the relative position of the object in the images using triangulation,
wherein the computing system is configured to perform the triangulation, and wherein the triangulation is based on respective observations of the object by the two image sensors,
a known distance between the two image sensors that defines a base,
and respective angles between the base and respective rays corresponding to the respective observations of the object,
wherein the two sensors are configured to observe the object according to respective rays beginning at respective starting points at the two sensors and extending towards a selected point on a surface of the object, wherein the computing system is configured to use the respective starting points of the rays and the selected point on the surface of the object to define a spatial triangle for the triangulation, wherein the computing system is configured to: use the known distance between the two sensors as a base of the spatial triangle; determine angles between the base and the respective rays corresponding to the respective observations of the object; use the determined angles to determine the intersection point of the respective rays to provide a spatial coordinate of the selected point on the surface of the object according to triangular relations; and use triangular relations to determine the distance between the selected feature and the point on the surface of the object and to provide the underwater depth perception, wherein the computing system is configured to generate an underwater three-dimensional model based on the data captured by the respective image sensors, wherein the model comprises a plurality of distances between the selected feature and objects underwater, and wherein the plurality of distances between the selected feature and the objects underwater comprise the determined distance between the selected feature and the point on the surface of the object.
9. Limitations of remaining claims of instant application are obvious over US Pat. 12375791 B2 in view of prior art discussed under Claim Rejections – 35 USC § 103 of this office action. Same motivation described under Claim Rejections – 35 USC § 103 of this office action is applicable for combining US Pat. 12375791 B2 and stated prior arts. Please note 35 U.S.C. 101 allows only one patent from one patent application or invention. In that aspect all dependent claims of instant application are obvious variation of independent claim 1.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 6-20 and 23 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ichiki (U.S. Pub. No. 20250117900 A1), in view of Becker (U.S. Pub. No. 20210041222 A1).
Regarding to claim 1 and 19-20:
1. Ichiki teach a system, comprising: (Ichiki Fig. 1) two image sensors, arranged on different axes and configured to complement each other, and each one of the image sensors comprising any type of sensor that captures an image (Ichiki FIG. 2 shows a left camera 10L and a right camera 10R are spaced apart have different axes. [0045] The stereo camera 10 is an image sensor that captures an image of an underwater measurement target. The stereo camera 10 includes, as illustrated in FIG. 2, a left camera 10L and a right camera 10R. FIG. 2 illustrates an example of a visual field range 11L of the left camera 10L and a visual field range 11R of the right camera 10R.) based on reflected or emitted electromagnetic radiation or mechanical waves; (Ichiki [0051] The left laser 30L is disposed on a left side of the left camera 10L, for example. The right laser 30R is disposed on a right side of the right camera 10R, for example. Note that the number of the laser 30 is not limited to two. To improve a processing speed for calibration, for example, other two lasers may further be installed in upper and lower directions, in addition to the left laser 30L and the right laser 30R. [0054] As laser light [emitted electromagnetic radiation] is irradiated undersea, backscattering light occurs due to plankton, for example. Observing this backscattering light with the stereo camera 10 makes it possible to make visible a light path of laser light. As the scan mechanism 31 controls scanning of the two line laser light beams L1 and L2, the line laser calibration unit 32 acquires information regarding an intersection point of the two line laser light beams L1 and L2 for each of all pixels in the stereo camera 10)
a computing system, (Ichiki Fig. 1, Fig. 2 and Fig. 11) configured to provide underwater depth perception based on data captured by the two image sensors (Ichiki [0049] The stereo matching unit 22 corresponds to a depth image generation unit that generates a depth image on the basis of the captured image of the measurement target, which is corrected by the distortion correction unit 20. The stereo matching unit 22 performs stereo matching processing on a captured image captured by the left camera 10L, which has undergone distortion correction, and is thus corrected by the distortion correction unit 20L, and a captured image captured by the right camera 10R, which has undergone distortion correction, and is thus corrected by the distortion correction unit 20R, to generate a depth image including information regarding three-dimensional measurement.)
and triangulation, (Ichiki [0101] The image processor according to the fifth embodiment is suitable in a case where a distance to a planar object such as a wall 70 of a dam is to be measured, as illustrated in FIG. 15. Irradiating the two line laser light beams L1 and L2 to the wall 70 and causing the stereo camera 10 to capture an image makes it possible to estimate, through triangulation, the distance Z to the wall 70 on the basis of a plurality of points irradiated with the two line laser light beams L1 and L2 irradiated to the wall 70. In stereo distance measurement using block matching, a range within which it is possible to perform distance measurement is generally limited due to a reason of a limited memory resource and a reason of achieving high-speed operation. Further limiting a range of search by the stereo matching unit 22 by using the estimated distance to the wall 70 then makes it possible to achieve a further resource reduction and more prompt operation. Furthermore, as one advantage of limiting a range of search by the stereo matching unit 22, it is possible to expect improved robustness including prevention of an error in estimating a distance due to erroneous matching.)
the triangulation using: (1) respective observations of an object captured by the two image sensors, which is part of the captured data, (Ichiki Fig. 2 shoes overlapping view from two cameras and [0065] FIG. 6, an intersection point of the two line laser light beams L1 and L2 lies at a coordinate (X.sub.L, Y.sub.L) on a captured image captured by the left camera 10L and a coordinate (X.sub.R, Y.sub.R) on a captured image captured by the right camera 10R in a case where angles in the scan mechanism 31 are respectively set to (θx, θy) and (Φx, Φy), identically to the case illustrated in FIG. 5, in a case where a water depth is deeper)
a submersible device, (Ichiki [0098] FIG. 14 schematically illustrates a configuration example of the image processor according to the fifth embodiment. FIG. 15 schematically illustrates a configuration example of the autonomous underwater robot 1 to which the image processor according to the fifth embodiment is applied) comprising a holder, the holder configured to hold the two image sensors and the computing system, (Becker [0122] FIG. 8A is a perspective view of a three-dimensional tactile probing system 5100 that includes a camera bar 5110 and a probe assembly 5140. The camera bar includes a mounting structure 5112 and at least two triangulation cameras 5120, 5124. It may also include an optional camera 5122. The cameras each include a lens and a photosensitive array, for example, as shown in the lens 2564 of FIG. 5A. The optional camera 5122 may be similar to the cameras 5120, 5124 or it may be a color camera. The probe assembly 5140 includes a housing 5142, a collection of lights 5144, optional pedestals 5146, shaft 5148, stylus 5150, and probe tip 5152.)
and wherein computing by the computing system to provide the underwater depth perception occurs onboard the submersible device. (Ichiki [0066] In a case where calibration is to be performed in accordance with withstand pressure in the image processor according to the first embodiment, the line laser calibration unit 32 controls the scan mechanism 31, causes the two line laser light beams L1 and L2 to be outputted, uses a captured image acquired from the stereo camera 10, performs line laser calibration, and uses the method described above to acquire a correction parameter. The parameter correction unit 21 and the distortion correction unit 20 perform distortion correction on the captured image acquired from the stereo camera 10 on the basis of the captured image acquired from the stereo camera 10, the generated calibration parameter serving as the reference for calibration, which is generated in advance, and the correction parameter described above. The stereo matching unit 22 performs stereo matching from the captured image having undergone the distortion correction to generate a depth image. [0098] FIG. 14 schematically illustrates a configuration example of the image processor according to the fifth embodiment. FIG. 15 schematically illustrates a configuration example of the autonomous underwater robot 1 to which the image processor according to the fifth embodiment is applied. [0101] The image processor according to the fifth embodiment is suitable in a case where a distance to a planar object such as a wall 70 of a dam is to be measured, as illustrated in FIG. 15. Irradiating the two line laser light beams L1 and L2 to the wall 70 and causing the stereo camera 10 to capture an image makes it possible to estimate, through triangulation, the distance Z [depth] to the wall 70 on the basis of a plurality of points irradiated with the two line laser light beams L1 and L2 irradiated to the wall 70. In stereo distance measurement using block matching, a range within which it is possible to perform distance measurement is generally limited due to a reason of a limited memory resource and a reason of achieving high-speed operation. Further limiting a range of search by the stereo matching unit 22 by using the estimated distance to the wall 70 then makes it possible to achieve a further resource reduction and more prompt operation. Furthermore, as one advantage of limiting a range of search by the stereo matching unit 22, it is possible to expect improved robustness including prevention of an error in estimating a distance due to erroneous matching)
Ichiki do not explicitly teach (2) a known distance between the two image sensors that defines a base, and (3) respective angles between the base and rays corresponding to the respective observations of the object;
However Becker teach (2) a known distance between the two image sensors that defines a base, and (Becker [0128] FIG. 9 is an isometric view of a detachable six-DOF target assembly 910 coupled to a triangulation scanner 210. The targets on the six-DOF target assembly 910 may be measured with a camera bar, such as the camera bar 5110 of FIGS. 8A-8C. Alternatively, the targets on the six-DOF target assembly may be measured with two or more cameras separately mounted in an environment, which is to say, not attached to a common bar. A camera bar includes two or more cameras spaced apart by a camera-bar baseline. Triangulation is applied to the images of the targets obtained by the two cameras to determine the six degrees of freedom of the six-DOF target assembly and scanner 210. Additional geometrical values such as camera-bar baseline and orientation of the cameras on the camera bar are used by a processor in the triangulation calculation)
(3) respective angles between the base and rays corresponding to the respective
observations of the object; and (Becker [0096] The line segment that connects the perspective centers is the baseline 2588 in FIG. 5A and the baseline 4788 in FIG. 5B. The length of the baseline is called the baseline length (2592, 4792). The angle between the projector optical axis and the baseline is the baseline projector angle (2594, 4794). The angle between the camera optical axis (2583, 4786) and the baseline is the baseline camera angle (2596, 4796). If a point on the source pattern of light (2570, 4771) is known to correspond to a point on the photosensitive array (2581, 4781), then it is possible using the baseline length, baseline projector angle, and baseline camera angle to determine the sides of the triangle connecting the points 2585, 2574, and 2575, and hence determine the surface coordinates of points on the surface of object 2590 relative to the frame of reference of the measurement system 2560. To do this, the angles of the sides of the small triangle between the projector lens 2572 and the source pattern of light 2570 are found using the known distance between the lens 2572 and plane 2570 and the distance between the point 2571 and the intersection of the optical axis 2576 with the plane 2570. These small angles are added or subtracted from the larger angles 2596 and 2594 as appropriate to obtain the desired angles of the triangle. It will be clear to one of ordinary skill in the art that equivalent mathematical methods can be used to find the lengths of the sides of the triangle 2574-2585-2575 or that other related triangles may be used to obtain the desired coordinates of the surface of object 2590.)
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to modify Ichiki, further incorporating Becker in video/camera technology. One would be motivated to do so, to incorporate (2) a known distance between the two image sensors that defines a base, and (3) respective angles between the base and rays corresponding to the respective observations of the object. This functionality will improve efficiency with predictable results.
Regarding to claim 2:
2. (Canceled)
Regarding to claim 4-5:
4. (Canceled)
5. (Canceled)
Regarding to claim 6 and 23:
6. Ichiki teach the system of claim 1, wherein the two sensors are configured to observe the object according to the respective rays beginning at respective starting points at the two sensors and (Ichiki FIG. 2 [0101] The image processor according to the fifth embodiment is suitable in a case where a distance to a planar object such as a wall 70 of a dam is to be measured, as illustrated in FIG. 15. Irradiating the two line laser light beams L1 and L2 to the wall 70 and causing the stereo camera 10 to capture an image makes it possible to estimate, through triangulation, the distance Z to the wall 70 on the basis of a plurality of points irradiated with the two line laser light beams L1 and L2 irradiated to the wall 70. In stereo distance measurement using block matching, a range within which it is possible to perform distance measurement is generally limited due to a reason of a limited memory resource and a reason of achieving high-speed operation.)
Ichiki do not explicitly teach extending towards a selected point on a surface of the object, and wherein the computing system is configured to use the respective starting points of the rays and the selected point on the surface of the object to define a spatial triangle for the triangulation.
However Becker teach extending towards a selected point on a surface of the object, and wherein the computing system is configured to use the respective starting points of the rays and the selected point on the surface of the object to define a spatial triangle for the triangulation. (Becker [0096] The line segment that connects the perspective centers is the baseline 2588 in FIG. 5A and the baseline 4788 in FIG. 5B. The length of the baseline is called the baseline length (2592, 4792). The angle between the projector optical axis and the baseline is the baseline projector angle (2594, 4794). The angle between the camera optical axis (2583, 4786) and the baseline is the baseline camera angle (2596, 4796). If a point on the source pattern of light (2570, 4771) is known to correspond to a point on the photosensitive array (2581, 4781), then it is possible using the baseline length, baseline projector angle, and baseline camera angle to determine the sides of the triangle connecting the points 2585, 2574, and 2575, and hence determine the surface coordinates of points on the surface of object 2590 relative to the frame of reference of the measurement system 2560. To do this, the angles of the sides of the small triangle between the projector lens 2572 and the source pattern of light 2570 are found using the known distance between the lens 2572 and plane 2570 and the distance between the point 2571 and the intersection of the optical axis 2576 with the plane 2570. These small angles are added or subtracted from the larger angles 2596 and 2594 as appropriate to obtain the desired angles of the triangle. It will be clear to one of ordinary skill in the art that equivalent mathematical methods can be used to find the lengths of the sides of the triangle 2574-2585-2575 or that other related triangles may be used to obtain the desired coordinates of the surface of object 2590)
Regarding to claim 7:
7. Ichiki teach the system of claim 6, use triangular relations to determine the distance between the selected feature and the point on the surface of the object and to provide the underwater depth perception. (Ichiki [0101] The image processor according to the fifth embodiment is suitable in a case where a distance to a planar object such as a wall 70 of a dam is to be measured, as illustrated in FIG. 15. Irradiating the two line laser light beams L1 and L2 to the wall 70 and causing the stereo camera 10 to capture an image makes it possible to estimate, through triangulation, the distance Z to the wall 70 on the basis of a plurality of points irradiated with the two line laser light beams L1 and L2 irradiated to the wall 70. In stereo distance measurement using block matching, a range within which it is possible to perform distance measurement is generally limited due to a reason of a limited memory resource and a reason of achieving high-speed operation. Further limiting a range of search by the stereo matching unit 22 by using the estimated distance to the wall 70 then makes it possible to achieve a further resource reduction and more prompt operation. Furthermore, as one advantage of limiting a range of search by the stereo matching unit 22, it is possible to expect improved robustness including prevention of an error in estimating a distance due to erroneous matching.)
Ichiki do not explicitly teach wherein the computing system is configured to: use the known distance between the two sensors as a base of the spatial triangle; determine angles between the base and the respective rays corresponding to the respective observations of the object; use the determined angles to determine the intersection point of the respective rays to provide a spatial coordinate of the selected point on the surface of the object according to triangular relations;
However Becker teach wherein the computing system is configured to: use the known distance between the two sensors as a base of the spatial triangle; determine angles between the base and the respective rays corresponding to the respective observations of the object; (Becker [0096] The line segment that connects the perspective centers is the baseline 2588 in FIG. 5A and the baseline 4788 in FIG. 5B. The length of the baseline is called the baseline length (2592, 4792). The angle between the projector optical axis and the baseline is the baseline projector angle (2594, 4794). The angle between the camera optical axis (2583, 4786) and the baseline is the baseline camera angle (2596, 4796). If a point on the source pattern of light (2570, 4771) is known to correspond to a point on the photosensitive array (2581, 4781), then it is possible using the baseline length, baseline projector angle, and baseline camera angle to determine the sides of the triangle connecting the points 2585, 2574, and 2575, and hence determine the surface coordinates of points on the surface of object 2590 relative to the frame of reference of the measurement system 2560. To do this, the angles of the sides of the small triangle between the projector lens 2572 and the source pattern of light 2570 are found using the known distance between the lens 2572 and plane 2570 and the distance between the point 2571 and the intersection of the optical axis 2576 with the plane 2570. These small angles are added or subtracted from the larger angles 2596 and 2594 as appropriate to obtain the desired angles of the triangle. It will be clear to one of ordinary skill in the art that equivalent mathematical methods can be used to find the lengths of the sides of the triangle 2574-2585-2575 or that other related triangles may be used to obtain the desired coordinates of the surface of object 2590.)
use the determined angles to determine the intersection point of the respective rays to provide a spatial coordinate of the selected point on the surface of the object according to triangular relations; and (Becker [0091] FIG. 4 shows elements of a laser line probe (LLP) 4500 that includes a projector 4520 and a camera 4540. The projector 4520 includes a source pattern of light 4521 and a projector lens 4522. The source pattern of light includes an illuminated pattern in the form of a line. The projector lens includes a projector perspective center and a projector optical axis that passes through the projector perspective center. In the example of FIG. 4, a central ray of the beam of light 4524 is aligned with the projector optical axis. The camera 4540 includes a camera lens 4542 and a photosensitive array 4541. The lens has a camera optical axis 4543 that passes through a camera lens perspective center 4544. In the exemplary system 4500, the projector optical axis, which is aligned to the beam of light 4524 and the camera lens optical axis 4544, are perpendicular to the line of light 4523 projected by the source pattern of light 4521. In other words, the line 4523 is in the direction perpendicular to the paper in FIG. 4. The line strikes an object surface, which at a first distance from the projector is object surface 4510A and at a second distance from the projector is object surface 4510B. It is understood that at different heights above or below the plane of the paper of FIG. 4, the object surface may be at a different distance from the projector. The line of light intersects surface 4510A (in the plane of the paper) in a point 4526, and it intersects the surface 4510B (in the plane of the paper) in a point 4527. For the case of the intersection point 4526, a ray of light travels from the point 4526 through the camera lens perspective center 4544 to intersect the photosensitive array 4541 in an image point 4546. For the case of the intersection point 4527, a ray of light travels from the point 4527 through the camera lens perspective center to intersect the photosensitive array 4541 in an image point 4547. By noting the position of the intersection point relative to the position of the camera lens optical axis 4544, the distance from the projector (and camera) to the object surface can be determined using the principles of triangulation. The distance from the projector to other points on the line of light 4526, that is points on the line of light that do not lie in the plane of the paper of FIG. 4, may similarly be found. [0094] As explained herein above, light from a scanner may be projected in a line pattern to collect 3D coordinates over a line. Alternatively, light from a scanner may be projected to cover an area, thereby obtaining 3D coordinates over an area on an object surface. In an embodiment, the projector 510 in FIG. 3C is an area projector rather than a line projector. An explanation of triangulation principles for the case of area projection is now given with reference to the system 2560 of FIG. 5A and the system 4760 of FIG. 5B. Referring first to FIG. 5A, the system 2560 includes a projector 2562 and a camera 2564. The projector 2562 includes a source pattern of light 2570 lying on a source plane and a projector lens 2572. The projector lens may include several lens elements. The projector lens has a lens perspective center 2575 and a projector optical axis 2576. The ray of light 2573 travels from a point 2571 on the source pattern of light through the lens perspective center onto the object 2590, which it intercepts at a point 2574.)
Regarding to claim 8 and 9:
8. Ichiki teach the system of claim 7, wherein the computing system is configured to generate an underwater three-dimensional model based on the data captured by the respective image sensors, (Ichiki [0039] FIG. 1 schematically illustrates a configuration example of an image processor according to a first embodiment of the present disclosure. [0040] For example, the image processor according to the first embodiment is mounted on an autonomous underwater robot, and is used for performing three-dimensional measurement underwater such as undersea. [0077] The image processor according to the first embodiment measures a change in distortion in a captured image in a real time manner, and provides feedback to the distortion correction unit 20, making it possible to constantly perform three-dimensional measurement using a correctly corrected image. At that time, it is possible to easily perform calibration on a captured image at a portion to which three-dimensional measurement is to be performed. Even if distortion that actually occurs differs from that measured beforehand, making it possible to perform calibration at that time (in a real time manner) makes it possible to perform correct correction. [0078] The image processor according to the first embodiment makes it possible to perform calibration on a captured image on the basis of actual data even under a plurality of types of withstand pressure, making it possible to correctly correct an image, and making it possible to finally calculate a correct depth image through stereo matching. In a case where a stereo camera style is applied as a technique of high resolution, highly accurate three-dimensional measurement undersea, it is possible to address an issue of a distance measurement error due to image distortion that occurs due to negative effects of withstand pressure by using the laser 30 and the scan mechanism 31 to perform calibration, making it possible to easily perform correct correction in a real time manner.)
wherein the model comprises a plurality of distances between the selected feature and objects underwater, and (Ichiki [0098] FIG. 14 schematically illustrates a configuration example of the image processor according to the fifth embodiment. FIG. 15 schematically illustrates a configuration example of the autonomous underwater robot 1 to which the image processor according to the fifth embodiment is applied.
[0099] The image processor according to the fifth embodiment has a configuration where the pattern projector 40 and a Z-distance estimation unit 60 are added in the configuration of the image processor according to the first embodiment described above (FIG. 1). [0100] The Z-distance estimation unit 60 estimates a distance Z to a measurement target on the basis of a captured image of the line laser light L1 and the line laser light L2, which is captured by the stereo camera 10 in the state where withstand pressure is applied (at the second water depth). The stereo matching unit 22 sets, on the basis of the distance Z estimated by the Z-distance estimation unit 60, a range of search for the measurement target for generating a depth image)
wherein the plurality of distances between the selected feature and the objects underwater comprises the determined distance between the selected feature and the point on the surface of the object. (Ichiki [0101] The image processor according to the fifth embodiment is suitable in a case where a distance to a planar object such as a wall 70 of a dam is to be measured, as illustrated in FIG. 15. Irradiating the two line laser light beams L1 and L2 to the wall 70 and causing the stereo camera 10 to capture an image makes it possible to estimate, through triangulation, the distance Z to the wall 70 on the basis of a plurality of points irradiated with the two line laser light beams L1 and L2 irradiated to the wall 70. In stereo distance measurement using block matching, a range within which it is possible to perform distance measurement is generally limited due to a reason of a limited memory resource and a reason of achieving high-speed operation. Further limiting a range of search by the stereo matching unit 22 by using the estimated distance to the wall 70 then makes it possible to achieve a further resource reduction and more prompt operation. Furthermore, as one advantage of limiting a range of search by the stereo matching unit 22, it is possible to expect improved robustness including prevention of an error in estimating a distance due to erroneous matching)
Regarding to claim 10:
10. Ichiki teach the system of claim 1, wherein the image sensors consist of a pair of sensors (Ichiki [0045] The stereo camera 10 is an image sensor that captures an image of an underwater measurement target. The stereo camera 10 includes, as illustrated in FIG. 2, a left camera 10L and a right camera 10R. FIG. 2 illustrates an example of a visual field range 11L of the left camera 10L and a visual field range 11R of the right camera 10R. Left and right camera creates stereo pair. Camera has sensor.) that sense reflected electromagnetic energy. (Ichiki [0051] The left laser 30L is disposed on a left side of the left camera 10L, for example. The right laser 30R is disposed on a right side of the right camera 10R, for example. Note that the number of the laser 30 is not limited to two. To improve a processing speed for calibration, for example, other two lasers may further be installed in upper and lower directions, in addition to the left laser 30L and the right laser 30R. [0054] As laser light [emitted electromagnetic radiation] is irradiated undersea, backscattering light occurs due to plankton, for example. Observing this backscattering light with the stereo camera 10 makes it possible to make visible a light path of laser light. As the scan mechanism 31 controls scanning of the two line laser light beams L1 and L2, the line laser calibration unit 32 acquires information regarding an intersection point of the two line laser light beams L1 and L2 for each of all pixels in the stereo camera 10)
Regarding to claim 11:
11. Ichiki teach the system of claim 1, further comprising a stereo camera that comprises the image sensors and that is configured to capture three-dimensional images using the image sensors. (Ichiki [0049] The stereo matching unit 22 corresponds to a depth image generation unit that generates a depth image on the basis of the captured image of the measurement target, which is corrected by the distortion correction unit 20. The stereo matching unit 22 performs stereo matching processing on a captured image captured by the left camera 10L, which has undergone distortion correction, and is thus corrected by the distortion correction unit 20L, and a captured image captured by the right camera 10R, which has undergone distortion correction, and is thus corrected by the distortion correction unit 20R, to generate a depth image including information regarding three-dimensional measurement.)
Regarding to claim 12:
12. Ichiki teach the system of claim 11, wherein the image sensors consist of a pair of image sensors of the camera. (Ichiki [0045] The stereo camera 10 is an image sensor that captures an image of an underwater measurement target. The stereo camera 10 includes, as illustrated in FIG. 2, a left camera 10L and a right camera 10R. FIG. 2 illustrates an example of a visual field range 11L of the left camera 10L and a visual field range 11R of the right camera 10R. Left and right camera creates stereo pair)
Regarding to claim 13-15:
13. Ichiki teach the system of claim 1, wherein the submersible device comprises a submersible mobile machine. (Ichiki [0002] In an autonomous underwater robot (or autonomous underwater vehicle or AUV), for example, there has been a demand for a technique of performing three-dimensional measurement underwater using a stereo camera. [0020] FIG. 11 is a configuration diagram schematically illustrating a configuration example of an autonomous underwater robot to which the image processor according to the second embodiment is applied.)
Regarding to claim 16:
16. Ichiki teach the system of claim 15, wherein the submersible vehicle is a submarine or a submersible. (Ichiki [0040] For example, the image processor according to the first embodiment is mounted on an autonomous underwater robot, and is used for performing three-dimensional measurement underwater such as undersea. [0081] FIG. 10 schematically illustrates a configuration example of the image processor according to the second embodiment. FIG. 11 schematically illustrates a configuration example of an autonomous underwater robot 1 to which the image processor according to the second embodiment is applied. [0098] FIG. 14 schematically illustrates a configuration example of the image processor according to the fifth embodiment. FIG. 15 schematically illustrates a configuration example of the autonomous underwater robot 1 to which the image processor according to the fifth embodiment is applied.)
Regarding to claim 17:
17. Ichiki teach the system of claim 16, wherein the image sensors and the computing system are configured to operate effectively while the submersible device and the holder are completely submerged. (Ichiki [0037] It has been demanded such a technique of performing three-dimensional measurement undersea for performing inspection and for achieving autonomous movement of an autonomous underwater robot, for example. It has been demanded, for performing inspection, in particular, a technique of high resolution, highly accurate three-dimensional measurement. As a technique of achieving high resolution and high accuracy, a more probable idea is a stereo camera style that is used for on-the-ground purposes in many cases. However, as a water depth increases, withstand pressure applied to a robot or a camera housing increases, and a captured image is more distorted, making it difficult to correctly perform three-dimensional measurement.)
Regarding to claim 18:
18. Ichiki teach the system of claim 1, wherein the computing system is further configured to provide underwater depth perception based on raw data captured by the image sensors. (Ichiki [0049] The stereo matching unit 22 corresponds to a depth image generation unit that generates a depth image on the basis of the captured image of the measurement target, which is corrected by the distortion correction unit 20. The stereo matching unit 22 performs stereo matching processing on a captured image captured by the left camera 10L, which has undergone distortion correction, and is thus corrected by the distortion correction unit 20L, and a captured image captured by the right camera 10R, which has undergone distortion correction, and is thus corrected by the distortion correction unit 20R, to generate a depth image including information regarding three-dimensional measurement. [0050] The laser 30 includes, as illustrated in FIG. 2, a left laser 30L and a right laser 30R. The left laser 30L serves as a first laser light source that emits the line laser light L1 as first laser light underwater. The right laser 30R serves as a second laser light source that emits the line laser light L2 as second laser light underwater from a position that differs from a position at which the left laser 30L emits the light. FIG. 2 illustrates an example of a light path of the line laser light L1 emitted from the left laser 30L and a light path of the line laser light L2 emitted from the right laser 30R.)
Claims 3 and 21-22 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ichiki (U.S. Pub. No. 20250117900 A1), in view of Becker (U.S. Pub. No. 20210041222 A1), further in view of Grunnet-Jepsen (U.S. Pub. No. 20170186166 A1).
Regarding to claim 3 and 21-22:
3. Ichiki teach the system of claim 1, between the selected feature and the object and to provide the underwater depth perception. (Ichiki [0101] The image processor according to the fifth embodiment is suitable in a case where a distance to a planar object such as a wall 70 of a dam is to be measured, as illustrated in FIG. 15. Irradiating the two line laser light beams L1 and L2 to the wall 70 and causing the stereo camera 10 to capture an image makes it possible to estimate, through triangulation, the distance Z to the wall 70 on the basis of a plurality of points irradiated with the two line laser light beams L1 and L2 irradiated to the wall 70. In stereo distance measurement using block matching, a range within which it is possible to perform distance measurement is generally limited due to a reason of a limited memory resource and a reason of achieving high-speed operation. Further limiting a range of search by the stereo matching unit 22 by using the estimated distance to the wall 70 then makes it possible to achieve a further resource reduction and more prompt operation. Furthermore, as one advantage of limiting a range of search by the stereo matching unit 22, it is possible to expect improved robustness including prevention of an error in estimating a distance due to erroneous matching.)
Ichiki do not explicitly teach wherein the computing system is configured to select a feature between the two image sensors of the image sensors to calculate a distance.
However Grunnet-Jepsen teach wherein the computing system is configured to select a feature between the two image sensors of the image sensors (Grunnet-Jepsen [0043] Nevertheless, using correspondence processing, given two or more images of the same three-dimensional scene, taken from different points of view via the two or more lenses (right camera 105 and left camera 110) of the stereo camera, the correspondence processing identifies a set of points in one image which can be correspondingly identified as the same points in another image by matching points or features in one image with the corresponding points or features in another image.) to calculate a distance (Grunnet-Jepsen [0132] In accordance with another embodiment of method 600, processing circuitry determines the depth to the object in the scene by determining correspondence for each of a plurality of points in the captured left and right images and triangulating a distance to each of the plurality of points in the captured left and right images using disparity)
The motivation for combining Ichiki and Becker as set forth in claim 1 is equally applicable to claim 3. It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to modify Ichiki, further incorporating Becker and Grunnet-Jepsen in video/camera technology. One would be motivated to do so, to incorporate the computing system is configured to select a feature between the two image sensors of the image sensors to calculate a distance. This functionality will improve quality with predictable results.
Closely related prior art
Examiner notes teaching of Cheramie (U.S. Pub. No. 20170074664 A1) and
Kobayashi (U.S. Pub. No. 20230045358 A1) is/are pertinent to the independent claim(s), however is not used because dependent claims are better covered by cited references.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NASIM N NIRJHAR whose telephone number is (571) 272-3792. The examiner can normally be reached on Monday - Friday, 8 am to 5 pm ET.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, William F Kraig can be reached on (571) 272-8660. The fax phone number for the organization where this application or proceeding is assigned is (571) 273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NASIM N NIRJHAR/Primary Examiner, Art Unit 2896
Prosecution Insights