Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Submission dated 03/18/2026 amends claims 1, 4, 5, and 19-20. Claims 1-20 are pending.
Response to Arguments
Applicant’s arguments with respect to the independent claim(s) have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Us patent application publication no. 2019/0392632 to Han et al. (hereinafter Han) in view of jp patent application publication no. 2013038640 to Miyake et al. (hereinafter Miyake).
For claim 20, Han as applied teaches a non-transitory computer readable medium encoded with a plurality of instructions that, when executed by a computer processor, perform a method (see, e.g., pars. 60-62 and FIG. 1), the method comprising:
receiving from a camera system, a first image of an object captured from a first perspective and a second image of the object captured from a second perspective (see, e.g., pras.31-33, 38-48, 79-82 and FIGS. 1-, which teach acquiring plurality of images of an object at different viewpoints); and
determining, by at least one processor of the camera system, a pose of the object based, at least in part, on a first set of sparse features associated with the object detected in the first image and a second set of sparse features associated with the object detected in the second image (see, e.g., pars. 49-54, 61, 84-89 and FIG. 1, which teach estimating a pose of the object based on the features extracted from the images, wherein the extracted features such as keypoints/vertices and geometric feature data representing an appearance shape/silhouette of the object; the examiner interprets features corresponding to the keypoints/vertices and outline/shape of the object as the claimed sparse features), wherein:
the camera system includes a first camera module and second camera module, the first camera module and the second camera module being separated by a first distance (see, e.g., par. 32 of Han, which teaches using a multi-view stereo camera; the examiner interprets the teaching of a stereo camera to implicitly teach, if not suggest, having first and second, e.g., left and right or top and bottom cameras/lenses separated by a baseline).
Han as applied does not explicitly teach that “the object is located at a second distance from the camera system, the second distance being one to five times the first distance.” Miyake in the analogous art teaches instances where a distance to the object is between three to five times the baseline length (see, e.g., the first two entries in the last row of the table in FIG. 6, where the imaging distance may be 2 meters and the baselines may be 60 and 45 centimeters).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the baseline such that the imaging distance is one to five times the baseline as taught by Miyake because doing so would allow finding more appropriate baseline for the respective photographing mode (see, e.g., par. 60 of Miyake).
Claim(s) 1-4 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Han in view of Miyake and further in view of us patent application publication no. 2022/0121837 to Cesic et al. (hereinafter Cesic).
For claim 1, Han as applied teaches a method of determining a pose of an object sensed by a camera system of a mobile robot, the method comprising:
acquiring, using the camera system, a first image of the object from a first perspective and a second image of the object from a second perspective (see, e.g., pras.31-33, 38-48 and 79-82 and FIG. 1, which teach acquiring plurality of images of an object at different viewpoints); and
determining, by a processor of the camera system, a pose of the object based, at least in part, on a first set of sparse features associated with the object detected in the first image and a second set of sparse features associated with the object detected in the second image (see, e.g., pars. 49-54, 61 and 84-89 and FIG. 1, which teach estimating a pose of the object based on the features extracted from the images, wherein the extracted features such as keypoints/vertices and geometric feature data representing an appearance shape/silhouette of the object; the examiner interprets features corresponding to the keypoints/vertices and outline/shape of the object as the claimed sparse features),
wherein: the camera system includes a first camera module and second camera module, the first camera module and the second camera module being separated by a first distance (see, e.g., par. 32 of Han, which teaches using a multi-view stereo camera; the examiner interprets the teaching to implicitly teach, if not suggest, having first and second, e.g., left and right or top and bottom, cameras/lenses separated by a baseline).
Han as applied does not explicitly teach that “the object is located at a second distance from the camera system, the second distance being one to five times the first distance.” Miyake in the analogous art teaches instances where a distance to the object is between three to five times the baseline length (see, e.g., the first two entries in the last row of the table in FIG. 6, where the imaging distance may be 2 meters and the baselines may be 60 and 45 centimeters).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the baseline such that the imaging distance is one to five times the baseline as taught by Miyake because doing so would allow finding more appropriate baseline for the respective photographing mode (see, e.g., par. 60 of Miyake).
Han in view of Miyake does not explicitly teach that the device is a part of a mobile robot. Cesic in the analogous art teaches autonomous mobile robots having pose determination functions (see, e.g., abstract and par. 2 of Cesic).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teaching of Han in view of Miyake to implemented in a robot as taught by Cesic because Han as applied teaches that its pose estimation system may be a part of a mobile device (see, e.g., par. 28 of Han) and doing so would yield predictable results of providing autonomy and mobility to the system, making it more versatile and adaptable (see par. 28 of Cesic and MPEP 2143(I)(D)).
For claim 2, while Han in view of Miyake does not explicitly teach (although it teaches using a feature extraction algorithm/model in pars. 84-89 and 119 of Han), Cesic in the analogous art teaches processing the first image and the second image with at least one machine learning model to detect the first set of sparse features and the second set of sparse features, respectively (see, e.g., pars. 50-57 and 101-103 of Cesic, which teach applying a machine-learned model to identify features of the object).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Han in view of Miyake to implement the feature extraction algorithm/module with a machine learning model as taught by Cesic because doing so would yield predictable results of automating the feature extraction process, improving the accuracy and enabling systems to adapt to diverse data (see MPEP 2143(I)(D)).
For claim 3, while Han in view of Miyake does not explicitly teach, Cesic as applied teaches that
the at least one machine model is configured to output a location and a confidence value associated with each sparse feature in the first set and the second set (see, e.g., pars. 53-56 and 101-104, which teach identifying feature and determining their visibility confidence scores), and
determining the pose of the object based, at least in part, on the first set of sparse features and the second set of sparse features is performed only when each sparse feature in the first set and the second set is associated with a confidence value above a threshold value (see, e.g., pars. 50-57 and 101-104, which teach determining a pose of the object when the confidence score meet or exceed the threshold).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Han in view of Miyake to determine the pose when the threshold is met as taught by Cesic because doing so would yield predictable results of reducing a number of features to be processed and hence making the process more energy efficient (see MPEP 2143(I)(D)).
For claim 4, Han in view of Miyake and Cesic applied teaches the first camera module and the second camera module have overlapping fields-of-view (see, e.g., pars. 31-36 and 78-82, which teach that the camera module may be a multi-view stereo camera, which include first and second sensors/lenses), the first image is acquired using the first camera module, and the second image is acquired using the second camera module (see, e.g., pars. 31-36 and 78-82, which teach capturing the object images with the sensors).
For claim 19, Han as applied teaches a mobile robot, comprising:
a camera system a first camera module and second camera module, the first camera module and the second camera module being separated by a first distance (see, e.g., see, e.g., pars. 30-39 and FIGS. 1 and 2, which teach a portable electronic device including a camera module, such as a multi-view stereo camera; the examiner interprets the teaching of a stereo camera to implicitly teach, if not suggest, having first and second, e.g., left and right or top and bottom, cameras/lenses separated by a baseline); and
at least one processor (see, e.g., pars. 30 and 53-59 and FIGS. 1-2, which teach a portable electronic device including a processor module) programmed to:
control the camera system to capture a first image of an object in an environment of the mobile robot from a first perspective and capture a second image of the object from a second perspective (see, e.g., pras.31-33, 38-48 and 79-82 and FIG. 1, which teach acquiring plurality of images of an object at different viewpoints); and
determine a pose of the object based, at least in part, on a first set of sparse features associated with the object detected in the first image and a second set of sparse features associated with the object detected in the second image (see, e.g., pars. 49-54, 61 and 84-89 and FIG. 1, which teach estimating a pose of the object based on the features extracted from the images, wherein the extracted features such as keypoints/vertices and geometric feature data representing an appearance shape/silhouette of the object; the examiner interprets features corresponding to the keypoints/vertices and outline/shape of the object as the claimed sparse features).
Han as applied does not explicitly teach that “the object is located at a second distance from the camera system, the second distance being one to five times the first distance.” Miyake in the analogous art teaches instances where a distance to the object is between three to five times the baseline length (see, e.g., the first two entries in the last row of the table in FIG. 6, where the imaging distance may be 2 meters and the baselines may be 60 and 45 centimeters).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the baseline such that the imaging distance is one to five times the baseline as taught by Miyake because doing so would allow finding more appropriate baseline for the respective photographing mode (see, e.g., par. 60 of Miyake).
Han in view of Miyake does not explicitly teach that the device is a part of a mobile robot. Cesic in the analogous art teaches autonomous mobile robots having pose determination functions (see, e.g., abstract and par. 2 of Cesic).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the teaching of Han in view of Miyake in a robot as taught by Cesic because Han as applied teaches that its pose estimation system may be a part of a mobile device (see, e.g., par. 28 of Han) and doing so would yield predictable results of providing autonomy and mobility to Han’s system, making it more versatile and adaptable (see par. 28 of Cesic and MPEP 2143(I)(D)).
Claim(s) 5-7, 9, 13, 14, and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Han in view of Miyake and Cesic and further in view of Us patent application publication no. 2022/0189062 to Seo et al. (hereinafter Seo).
For claim 5, while Han in view of Miyake and Cesic teaches acquiring depth information from images captured during the first and second object scanning (see, e.g., pars. 38-48 and 111-114 of Han), it does not explicitly teach using two separate cameras, each with a depth sensor.
Seo in the analogous art teaches that the first camera module includes a first depth sensor configured to acquire first depth information associated with the first image, and the second camera module includes a second depth sensor configured to acquire second depth information associated with the second image (see, e.g., pars. 40-47 of Seo, which teach that each camera has RGB and depth sensors with matching fields of view).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Han in view of Miyake and Cesic to employ a camera system as taught by Seo because doing so would allow usage of inexpensive cameras, and increase the accuracy of calibration and generate a high quality 3D volumetric model (see pars. 12 and 29 of Seo).
For claim 6, Han in view of Miyake and Cesic and Seo teaches that each of the first set of sparse features and the second set of sparse features include locations of a plurality of points associated with the object in the first image and the second image, respectively (see, e.g., pars. 49-55, 61 and 84-97 of Han, which teach extracting features of the images, such as keypoints and global and local features, and tracking their locations).
For claim 7, Han in view of Miyake and Cesic and Seo teaches that the plurality of points associated with the object comprise a plurality of corners of the object (see, e.g., pars. 93-95 and 120 of Han, which teach calculating vertex map from the extracted features).
For claim 9, Han in view of Miyake and Cesic and Seo teaches:
projecting the sparse features in the first set into a 3-dimensional (3D) space based on the first depth information to produce a first initial 3D estimate of the object (see, e.g., pars. 71, 90-102, 112-113, and 117-123 and FIGS. 2-4 of Han, which teach reconstructing a depth image-based 3D object based on the first feature data from the depth images); and
projecting the sparse features in the second set into the 3D space based on the second depth information to produce a second 3D estimate of the object (see, e.g., pars. 71, 90-102, 115, and 126-139 and FIGS. 2, 3, and 5-6 of Han, which teach reconstructing a depth image-based 3D object based on the second feature data from the depth images).
For claim 13, while Han in view of Miyake and Cesic does not explicitly, Seo in the analogous art teaches that each of the first depth sensor and the second depth sensor is an indirect time-of-flight sensor (see, e.g., par. 54 of Seo, which teaches using a time-of-flight sensor (ToF)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Han in view of Cesic to use a ToF sensor as taught by Seo because doing so would allow usage of an inexpensive sensor (see par. 54 of Seo).
For claim 14, while Han in view of Miyake does not explicitly, Cesic in the analogous art teaches:
determining whether a location of at least one sparse feature in the first set is inaccurate due to an occlusion of the object by another object sensed by the camera system (see, e.g., pars. 50-57 and 101-104 of Cesic, which teach determining whether a feature is visible to the robot/sensor due to occlusion); and
determining the pose of the object based, at least in part, on the first set of sparse features and the second set of sparse features is performed only when it is not determined that the location of the at least one sparse feature in the first set is inaccurate due to an occlusion of the object by another object sensed by the camera system (see, e.g., pars. 50-57 and 101-104 of Cesic, which teach determining a pose of the object only when there are visible features).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Han in view of Miyake to determine the pose when the threshold is met as taught by Cesic because doing so would yield predictable results of prevent the resource from being wasted on processing invisible features and hence making the process more energy efficient (see MPEP 2143(I)(D)).
For claim 17, while Han in view of Miyake does not explicitly teach, Cesic in the analogous art teaches:
determining whether a location of at least one sparse feature in the first set is inaccurate due to a partial occlusion of the object by another object sensed by the camera system (see, e.g., pars. 50-57 and 101-104 of Cesic, which teach determining whether a feature is visible to the robot/sensor due to occlusion); and
identifying one or more valid sparse features in the first set of sparse features, the one or more valid sparse features not being occluded in the first image (see, e.g., pars. 50-57 and 101-104 of Cesic, which teach determining that a feature is visible to the robot/sensor when its confidence score meets or exceed the threshold),
wherein determining the pose of the object is further based, at least in part, on the one or more valid sparse features in the first set of sparse features and the second set of sparse features associated with the object detected in the second image (see, e.g., pars. 50-57 and 101-104, which teach determining a pose of the object only when there are visible features).
Since Han as applied teaches determining a pose of the object based on the first and second sets of features (see, e.g., pars. 49-54, 61, 84-89 and FIG. 1 of Han, which teach estimating a pose of the object based on the features extracted from the images, wherein the extracted features include global geometric feature data representing an appearance shape/silhouette of the object), it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Han in view of Miyake to validate feature in one of the image sets as taught by Cesic because doing so would yield predictable results of reducing a number of features to be processed and hence making the process more energy efficient (see MPEP 2143(I)(D)).
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Han in view of Miyake, Cesic and Seo and further in view of Us patent application publication no. 2020/0302207 to Perkins et al. (hereinafter Perkins).
For claim 8, while Han in view of Miyake, Cesic and Seo does not explicitly teach, Perkins in the analogous art teaches that the object is a box and the plurality of points associated with the object comprise corners of a face of the box (see, e.g., pars. 22, 51-60, and 64-65 and FIGS. 1, 2A-F and 4 of Perkins).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Han in view of Miyake, Cesic and Seo to detect boxes as taught by Perkins because doing so would yield predictable results of utilizing the Han and Seo’s teaching in manufacturing, transportation, hazardous environments, exploration, and healthcare industries and logistics (see pars. 2 and 20-21 of Perkins and MPEP 2143(I)(D).
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Han in view of Miyake, Cesic and Seo and further in view of us patent application publication no. 2024/0221210 to Wen.
For claim 10, while Han in view of Miyake, Cesic and Seo does not explicitly teach, Wen in the analogous art teaches:
generating a refined 3D estimate of the object based on the first initial 3D estimate, the second 3D estimate and a cost function that includes a plurality of error terms, the plurality of error terms including at least one reprojection error term (see, e.g., pars. 124-127 and 163-168 of Wen, which teach optimizing 3D space points of the images based on the initial 3D space points of the image by determining a reprojection error based on the image).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Han in view of Miyake, Cesic and Seo to perform the optimization as taught by Wen because doing so would improve the accuracy of the pose estimation (see pars. 100 and164 of Wen).
Allowable Subject Matter
Claims 11, 12, 15, 16, and 18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
In regard to claim 11, when considered as a whole, prior art of record fails to disclose or render obvious, alone or in combination:
“… generating the refined 3D estimate comprises:
reprojecting each sparse feature from the 3D space into the 2D image space to determine a corresponding reprojected location for each sparse feature; and
defining a vector from the reprojected location of each sparse feature to its corresponding detected location in 2D image space,
wherein the cost function includes a reprojection error term for each sparse feature corresponding to a length of the defined vector for the sparse feature.”
In regard to claim 12, when considered as a whole, prior art of record fails to disclose or render obvious, alone or in combination:
“the plurality of error terms includes at least one pitch error term.”
In regard to claim 15, when considered as a whole, prior art of record fails to disclose or render obvious, alone or in combination:
“determining whether a location of at least one sparse feature in the first set is inaccurate due to an occlusion of the object by another object sensed by the camera system comprises:
acquiring, using the camera system, depth information corresponding to the first image of the object; and
determining that another object is causing an occlusion of the object in the first image when a histogram of values in the depth information has a bimodal distribution.
In regard to claim 16, claim 16 depends on objected claim 15. Therefore, by virtue of its dependency, claim 16 is indicated as objected subject matter.
In regard to claim 18, when considered as a whole, prior art of record fails to disclose or render obvious, alone or in combination:
“identifying the one or more valid sparse features comprises:
performing pose optimizations of different valid combinations of sparse features to determine combination candidates;
filtering the combination candidates based on one or more thresholds to generate one or more acceptable combination candidates;
ranking the acceptable candidates based on one or more heuristics; and
identifying the one or more valid sparse features based, at least in part, on the acceptable candidate having a highest rank.
Additional Citations
The following table lists several references that are relevant to the subject matter claimed and disclosed in this Application. The references are not relied on by the Examiner, but are provided to assist the Applicant in responding to this Office action.
Citation
Relevance
Noro et al. (wo pat. pub. 2012/127924)
Describes a stereoscopic image capturing device and an electronic apparatus including the stereoscopic image capturing device, and more particularly to a portable information terminal that is an electronic device and a small stereoscopic image capturing device built in a mobile phone. In one embodiment, when the assumed distance from a first lens to a subject is set to Ds (>1250) [mm], a disparity on a 35-millimeter film is set to k [mm], and the focal lengths of first to fourth lens groups are set to f [mm], a base length (L1) within a stereoscopic image capture range of 2500 [mm] and a base length (L2) within a stereoscopic image capture range of 1500 [mm] are represented as follows. L1 = k×(Lmax1×Lmin1)/(Lmax1-Lmin1)/f L2 = k×(Lmax2×Lmin2)/(Lmax2-Lmin2)/f Lmax1: Ds+1250 Lmin1: Ds-1250 Lmax2: Ds+750 Lmin2: Ds-750 k: 1.2 [mm] f: 105 [mm] A base length (LLR) [mm] represented by the distance between the optical axis center (OL) of a first lens for the left eye and the optical axis center (OR) of a first lens for the right eye satisfies the condition of L1LR2. Consequently, although compact, a stereoscopic image capturing device can capture natural and stereoscopic images even at a telescopic end and a wide-angle end.
Tanaka (us pat. pub. 2024/0114121)
Describes a moving body, a moving body control method, and a program. In one embodiment, to enable appropriate imaging of both an object located at a short distance and an object located at a long distance by a stereo camera depending on the situation, a moving body includes: a stereo camera including a first imaging unit and a second imaging unit; a base length changing unit that moves at least one of the first imaging unit and the second imaging unit so as to change a base length between the first imaging unit and the second imaging unit; and a moving body control unit that controls the base length changing unit such that the base length is adjusted to a target base length according to a state variable indicating information of a variable imaging state of the first imaging unit and the second imaging unit.
Table 1
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. See attached form 892 and Table 1.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WOO RHIM whose telephone number is (571)272-6560. The examiner can normally be reached Mon - Fri 9:30 am - 6:00 pm et.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henok Shiferaw can be reached at 571-272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WOO C RHIM/Examiner, Art Unit 2676
/Henok Shiferaw/Supervisory Patent Examiner, Art Unit 2676
Prosecution Insights