DETAILED ACTION
Notice of AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers submitted under 35 U.S.C. 119(a)-(d), which papers have been placed of record in the file.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 06/18/2026 has been entered.
Response to Arguments
Applicant’s arguments, see remarks, filed 06/18/2026, with respect to claims 1-11, have been fully considered, but they are not persuasive.
The applicant argues on page 7, “The applied art, whether taken alone or in any permissible combination, is not seen to disclose or to suggest the features of Claims 1, 10 and 11, and in particular, is not seen to disclose or to suggest at least the features of, wherein in the case in which the accuracy is outside of the predetermined range, the notification unit is configured to notify by jointly displaying the image, feature points detected from the image in estimation of the self-position, the accuracy in estimation of the self-position, and information indicating the object detected by the image analyzing unit.”
In response, the Office does not find this argument to be persuasive. In light of the applicant’s amendment and the breadth of the claim language, the interpretation and application of the previously applied combination of references has changed. In regards to claim 1, HOLZ et al. (US 20200242396 A1), explicitly teaches wherein in the case in which the accuracy is outside of the predetermined range (Fig. 1, #120 called a robotic device. Paragraph [0084]- HOLZ discloses Table 1: Stale Localization - Localization system is unable to determine robotic device pose and/or localization certainty estimate [wherein certainty estimate is accuracy] has exceeded bounds [wherein bounds is predetermined range]. Recovery strategy states robotic device will halt and notify human operator. The operator can intervene by manually driving robotic device to a location for re-localization and reengaging.),
HOLZ further teaches in estimation of the self-position (Fig. 7, Paragraph [0094] – HOLZ discloses location and orientation of robot 700 relative to the environment may be estimated, enabling the robot to navigate through the environment accordingly. See also Fig. 1, Paragraph [0047].)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI of having an information processing device comprising at least one processor or circuit configured to function as: a self-position estimating unit configured to estimate a self-position on the basis of an image; an accuracy determining unit configured to determine accuracy in estimation of the self-position; and an image analyzing unit configured to detect an object from the image, with the teachings of HOLZ having and a notification unit configured to notify information on the object detected by the image analyzing unit in a case in which the accuracy is outside of a predetermined range, wherein in the case in which the accuracy is outside of the predetermined range, in estimation of the self-position.
Wherein having TSURUMI’s information processing device wherein having and a notification unit configured to notify information on the object detected by the image analyzing unit in a case in which the accuracy is outside of a predetermined range, wherein in the case in which the accuracy is outside of the predetermined range, in estimation of the self-position.
The motivation behind the modification would have been to obtain an information processing device with a mechanism capable of estimating a self-position more accurately, since both TSURUMI and HOLZ relate to localization processing, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while HOLZ has a system for simultaneous localization and calibration (SLAM) wherein a computing system may perform SLAM or similar processes to determine a location of a robot while also detecting positions of markers within the environment. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and HOLZ (US 20200242396 A1), Paragraph [105].
TSURUMI in view of HOLZ fail to explicitly teach the notification unit is configured to notify by jointly displaying the image, feature points detected from the image, the accuracy, and information indicating the object detected by the image analyzing unit.
However, WATANABE explicitly teaches wherein the notification unit (Fig. 10, #103 called display apparatus, Paragraph [0087] – WATANABE discloses the operation screen is presented to the user on the display apparatus 103)
is configured to notify by jointly displaying the image (Fig. 10, Paragraph [0086] – WATANABE discloses FIG. 10 is a diagram of an exemplary configuration of an operation screen to execute the image retrieval by using the image retrieving apparatus 104. Paragraph [0037] – WATANABE further discloses image retrieving apparatus 104 executes retrieving processing to retrieve an image which matches the retrieval condition from the image database 108 by using the retrieval condition specified by a user from the input apparatus 102 and present the information on the display apparatus 103.),
feature points detected from the image (Fig. 10, Paragraph [0090] – WATANABE discloses the feature points in FIG. 10 are, for example, 0: head, 1: neck, 2: right shoulder, 3: right elbow, 4: right wrist, 5: left shoulder, 6: left elbow, 7: left wrist, 8: right waist, 9: right knee, 10: right ankle, 11: left waist, 12: left knee, 13: left ankle. When the pose information is input, the feature points may be independently moved, and the plurality of feature points may be moved in conjunction with each other in consideration of connection relations. Furthermore, control points and the like other than the feature points may be added.),
the accuracy (Fig. 10, Paragraph [0089] – WATANABE discloses information displayed in the retrieval result display region 1004 is output to the display apparatus 103 by the retrieval result displaying unit 112 [wherein accuracy is degree of similarity shown in Fig. 10 within retrieval result display region 1004]. See also Paragraph [0079].),
and information indicating the object detected by the image analyzing unit (Fig. 10, Paragraph [0089] – WATANABE discloses information displayed in the retrieval result display region 1004 is output to the display apparatus 103 by the retrieval result displaying unit 112. Paragraph [0049] – WATANABE further discloses the data obtained by the retrieving processing is related to the object. Paragraph [0065] – WATANABE discloses pose estimating unit 106 detects a person region from the input image [wherein pose estimating unit is the image analyzing unit].).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI in view of HOLZ of having an information processing device comprising at least one processor or circuit configured to function as: a self-position estimating unit configured to estimate a self-position on the basis of an image; an accuracy determining unit configured to determine accuracy in estimation of the self-position; an image analyzing unit configured to detect an object from the image; and a notification unit configured to notify of information on the object detected by the image analyzing unit in a case in which the accuracy is outside of a predetermined range, wherein in the case in which the accuracy is outside of the predetermined range, in estimation of the self-position, with the teachings of WATANABE having the notification unit is configured to notify by jointly displaying the image, feature points detected from the image, the accuracy, and information indicating the object detected by the image analyzing unit.
Wherein having TSURUMI’s information processing device wherein having the notification unit is configured to notify by jointly displaying the image, feature points detected from the image in estimation of the self-position, the accuracy in estimation of the self-position, and information indicating the object detected by the image analyzing unit.
The motivation behind the modification would have been to obtain an information processing device with an enhanced and more accurate method of object detection, since both TSURUMI and WATANABE relate to image processing and pose estimation, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while WATANABE has an image retrieving apparatus and an image retrieving method capable of improving a retrieval accuracy and a retrieval efficiency by generating a retrieval query reflecting pose information on a retrieval target. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and WATANABE (US 20190147292 A1), Paragraph [0105].
The applicant argues on page 9, “amended independent Claims 1, 10, and 11 are believed to be allowable over the combination of Tsurumi, Holz, and Watanabe, and reconsideration and withdrawal of the rejections are requested.”
In response, the office does not find this argument to be persuasive. In light of the applicant’s amendment and the breadth of the claim language, the interpretation and application of the previously applied combination of references has changed. In regards to amended claim 10, HOLZ et al. (US 20200242396 A1), explicitly teaches wherein in the case in which the accuracy is outside of the predetermined range (Fig. 1, #120 called a robotic device. Paragraph [0084]- HOLZ discloses Table 1: Stale Localization - Localization system is unable to determine robotic device pose and/or localization certainty estimate [wherein certainty estimate is accuracy] has exceeded bounds [wherein bounds is predetermined range]. Recovery strategy states robotic device will halt and notify human operator. The operator can intervene by manually driving robotic device to a location for re-localization and reengaging.),
HOLZ further teaches in estimation of the self-position (Fig. 7, Paragraph [0094] – HOLZ discloses location and orientation of robot 700 relative to the environment may be estimated, enabling the robot to navigate through the environment accordingly. See also Fig. 1, Paragraph [0047].)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI of having an information processing method comprising: estimating a self-position on the basis of an image; determining accuracy in estimation of the self-position; detecting an object from the image; with the teachings of HOLZ of having and notifying of information on the object detected in the detecting of an object in a case in which the accuracy is outside of a predetermined range, wherein in the case in which the accuracy is outside of the predetermined range, in estimation of the self-position.
Wherein having TSURUMI’s information processing method wherein having and notifying of information on the object detected in the detecting of an object in a case in which the accuracy is outside of a predetermined range, wherein in the case in which the accuracy is outside of the predetermined range, in estimation of the self-position.
The motivation behind the modification would have been to obtain an information processing method with a mechanism capable of estimating a self-position more accurately, since both TSURUMI and HOLZ relate to localization processing, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while HOLZ has a system for simultaneous localization and calibration (SLAM) wherein a computing system may perform SLAM or similar processes to determine a location of a robot while also detecting positions of markers within the environment. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and HOLZ (US 20200242396 A1), Paragraph [105].
TSURUMI in view of HOLZ fail to explicitly teach the notifying of information includes notifying by jointly displaying the image, feature points detected from the image, the accuracy, and information indicating the object detected in the detecting of the object.
However, WATANABE explicitly teaches the notifying of information includes notifying by jointly displaying the image (Fig. 10, Paragraph [0086] – WATANABE discloses FIG. 10 is a diagram of an exemplary configuration of an operation screen to execute the image retrieval by using the image retrieving apparatus 104. Paragraph [0037] – WATANABE further discloses image retrieving apparatus 104 executes retrieving processing to retrieve an image which matches the retrieval condition from the image database 108 by using the retrieval condition specified by a user from the input apparatus 102 and present the information on the display apparatus 103.),
feature points detected from the image (Fig. 10, Paragraph [0090] – WATANABE discloses the feature points in FIG. 10 are, for example, 0: head, 1: neck, 2: right shoulder, 3: right elbow, 4: right wrist, 5: left shoulder, 6: left elbow, 7: left wrist, 8: right waist, 9: right knee, 10: right ankle, 11: left waist, 12: left knee, 13: left ankle. When the pose information is input, the feature points may be independently moved, and the plurality of feature points may be moved in conjunction with each other in consideration of connection relations. Furthermore, control points and the like other than the feature points may be added.),
the accuracy (Fig. 10, Paragraph [0089] – WATANABE discloses information displayed in the retrieval result display region 1004 is output to the display apparatus 103 by the retrieval result displaying unit 112 [wherein accuracy is degree of similarity shown in Fig. 10 within retrieval result display region 1004]. See also Paragraph [0079].),
and information indicating the object detected in the detecting of the object (Fig. 10, Paragraph [0089] – WATANABE discloses information displayed in the retrieval result display region 1004 is output to the display apparatus 103 by the retrieval result displaying unit 112. Paragraph [0049] – WATANABE further discloses the data obtained by the retrieving processing is related to the object. Paragraph [0065] – WATANABE discloses pose estimating unit 106 detects a person region from the input image.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI in view of HOLZ of having an information processing method comprising: estimating a self-position on the basis of an image; determining accuracy in estimation of the self-position; detecting an object from the image; and notifying of information on the object detected in the detecting of an object in a case in which the accuracy is outside of a predetermined range, wherein in the case in which the accuracy is outside of the predetermined range, in estimation of the self-position, with the teachings of WATANABE having the notifying of information includes notifying by jointly displaying the image, feature points detected from the image, the accuracy, and information indicating the object detected in the detecting of the object.
Wherein having TSURUMI’s information processing device wherein having the notifying of information includes notifying by jointly displaying the image, feature points detected from the image in estimation of the self-position, the accuracy in estimation of the self-position, and information indicating the object detected in the detecting of the object.
The motivation behind the modification would have been to obtain an information processing method with an enhanced and more accurate method of object detection, since both TSURUMI and WATANABE relate to image processing and pose estimation, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while WATANABE has an image retrieving apparatus and an image retrieving method capable of improving a retrieval accuracy and a retrieval efficiency by generating a retrieval query reflecting pose information on a retrieval target. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and WATANABE (US 20190147292 A1), Paragraph [0105].
The applicant argues on page 9, “amended independent Claims 1, 10, and 11 are believed to be allowable over the combination of Tsurumi, Holz, and Watanabe, and reconsideration and withdrawal of the rejections are requested.”
In response, the office does not find this argument to be persuasive. In light of the applicant’s amendment and the breadth of the claim language, the interpretation and application of the previously applied combination of references has changed. In regards to amended claim 11, HOLZ et al. (US 20200242396 A1), explicitly teaches wherein in the case in which the accuracy is outside of the predetermined range (Fig. 1, #120 called a robotic device, Paragraph [0084]- HOLZ discloses in some embodiments, robotic device(s) 120 can be subject to one or more failure conditions. Examples of those failure conditions and related recovery strategies are described in Table 1 below. Table 1: Stale Localization - Localization system is unable to determine robotic device pose and/or localization certainty estimate [wherein certainty estimate is accuracy] has exceeded bounds [wherein bounds is predetermined range]. Recovery strategy states robotic device will halt and notify human operator. The operator can intervene by manually driving robotic device to a location for re-localization and reengaging),
in estimation of the self-position (Fig. 7, Paragraph [0094] – HOLZ discloses location and orientation of robot 700 relative to the environment may be estimated, enabling the robot to navigate through the environment accordingly. See also Fig. 1, Paragraph [0047].).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI of having a non-transitory computer-readable storage medium configured to store a computer program comprising instructions for executing following processes: estimating a self-position on the basis of an image, determining accuracy in estimation of the self-position, detecting an object from the image, with the teachings of HOLZ of having and notifying of information on the object detected in the detecting of an object in a case in which the accuracy is outside of a predetermined range, wherein in the case in which the accuracy is outside of the predetermined range, in estimation of the self-position.
Wherein having TSURUMI’s non-transitory computer-readable storage medium wherein having and notifying of information on the object detected in the detecting of an object in a case in which the accuracy is outside of a predetermined range, wherein in the case in which the accuracy is outside of the predetermined range, in estimation of the self-position.
The motivation behind the modification would have been to obtain an information processing device with a mechanism capable of estimating a self-position more accurately, since both TSURUMI and HOLZ relate to localization processing, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while HOLZ has a system for simultaneous localization and calibration (SLAM) wherein a computing system may perform SLAM or similar processes to determine a location of a robot while also detecting positions of markers within the environment. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and HOLZ (US 20200242396 A1), Paragraph [105].
TSURUMI in view of HOLZ fail to explicitly teach the notifying of information includes notifying by jointly displaying the image, feature points detected from the image, the accuracy, and information indicating the object detected in the detecting of the object.
However, WATANABE explicitly teaches the notifying of information includes notifying by jointly displaying the image (Fig. 10, Paragraph [0086] – WATANABE discloses FIG. 10 is a diagram of an exemplary configuration of an operation screen to execute the image retrieval by using the image retrieving apparatus 104. Paragraph [0037] – WATANABE further discloses image retrieving apparatus 104 executes retrieving processing to retrieve an image which matches the retrieval condition from the image database 108 by using the retrieval condition specified by a user from the input apparatus 102 and present the information on the display apparatus 103.),
feature points detected from the image (Fig. 10, Paragraph [0090] – WATANABE discloses the feature points in FIG. 10 are, for example, 0: head, 1: neck, 2: right shoulder, 3: right elbow, 4: right wrist, 5: left shoulder, 6: left elbow, 7: left wrist, 8: right waist, 9: right knee, 10: right ankle, 11: left waist, 12: left knee, 13: left ankle. When the pose information is input, the feature points may be independently moved, and the plurality of feature points may be moved in conjunction with each other in consideration of connection relations. Furthermore, control points and the like other than the feature points may be added.),
the accuracy (Fig. 10, Paragraph [0089] – WATANABE discloses information displayed in the retrieval result display region 1004 is output to the display apparatus 103 by the retrieval result displaying unit 112 [wherein accuracy is degree of similarity shown in Fig. 10 within retrieval result display region 1004]. See also Paragraph [0079].),
and information indicating the object detected in the detecting of the object (Fig. 10, Paragraph [0089] – WATANABE discloses information displayed in the retrieval result display region 1004 is output to the display apparatus 103 by the retrieval result displaying unit 112. Paragraph [0049] – WATANABE further discloses the data obtained by the retrieving processing is related to the object. Paragraph [0065] – WATANABE discloses pose estimating unit 106 detects a person region from the input image.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI in view of HOLZ of having a non-transitory computer-readable storage medium configured to store a computer program comprising instructions for executing following processes: estimating a self-position on the basis of an image; determining accuracy in estimation of the self-position; detecting an object from the image; and notifying of information on the object detected in the detecting of an object in a case in which the accuracy is outside of a predetermined range, wherein in the case in which the accuracy is outside of the predetermined range, in estimation of the self-position, with the teachings of WATANABE having the notifying of information includes notifying by jointly displaying the image, feature points detected from the image, the accuracy, and information indicating the object detected in the detecting of the object.
Wherein having TSURUMI’s non-transitory computer-readable storage medium wherein having the notifying of information includes notifying by jointly displaying the image, feature points detected from the image in estimation of the self-position, the accuracy in estimation of the self-position, and information indicating the object detected in the detecting of the object.
The motivation behind the modification would have been to obtain an information processing device with an enhanced and more accurate method of object detection, since both TSURUMI and WATANABE relate to image processing and pose estimation, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while WATANABE has an image retrieving apparatus and an image retrieving method capable of improving a retrieval accuracy and a retrieval efficiency by generating a retrieval query reflecting pose information on a retrieval target. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and WATANABE (US 20190147292 A1), Paragraph [0105].
The office suggests the applicant amend the claim language to clarify that the joint display is shown in one imaging display or monitor such that all of the listed features of “the image, feature points detected from the image in estimation of the self-position, the accuracy in estimation of the self-position, and information indicating the object detected in the detecting of the object” are shown at once.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
Claims 1, 6, and 11 recite limitations that use words like “means” (or “step”) or similar terms with functional language but do not invoke 35 U.S.C. 112(f):
Claim 1; recites the limitation, “at least one processor or circuit configured to …,” [Lines 1-2].
Claim 6; recites the limitation, “at least one processor or circuit is further configured to …,” [Line 2].
Claim 11; recites the limitation, “a non-transitory computer-readable storage medium configured to …,” [Line 1].
Such claim limitation(s) is/are:
(i) “processor or circuit…” has a structure associated with it (i.e. physical hardware being a processor or circuit).
(ii) “a non-transitory computer-readable storage medium…” has a structure associated with it a memory.
Note: The office understands in claims 1 and 6, all units are understood to be a processor or circuit based on the claim language “at least one processor or circuit configured to function as:”.
Because this/these claim limitation(s) is/are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are not being interpreted to cover only the corresponding structure, material, or acts described in the specification as performing the claimed function, and equivalents thereof.
If applicant intends to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to remove the structure, materials, or acts that performs the claimed function; or (2) present a sufficient showing that the claim limitation(s) does/do not recite sufficient structure, materials, or acts to perform the claimed function.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 4, 5, 7, 10, and 11 are rejected under 35 U.S.C. 103 as being unpatentable over TSURUMI (US 20240412408 A1), hereinafter referenced as TSURUMI in view of HOLZ (US 20200242396 A1), hereinafter referenced as HOLZ, further in view of WATANABE (US 20190147292 A1), hereinafter referenced as WATANABE.
Regarding claim 1, TSURUMI teaches an information processing device (Fig. 5, #100 called an information processing apparatus. Paragraph [0054])
comprising at least one processor or circuit (Fig. 5, #100 called an information processing apparatus. Paragraph [0054] – TSURUMI discloses the information processing apparatus 100 is realized by, for example, a processing circuit such as a central processing unit (CPU) or a graphics processing unit (GPU)) configured to function as:
a self-position estimating unit (Fig. 5, #132 called a self-position estimation unit. Paragraph [0098] - TSURUMI discloses as illustrated in FIG. 5, the control unit 130 includes a map generation unit 131 and a self-position estimation unit 132.)
configured to estimate a self-position on the basis of an image (Fig. 5. Paragraph [0137]- TSURUMI discloses the self-position estimation unit 132 estimates the self-position of the terminal apparatus 300 using at least either the mask space or the feature point map and the captured image acquired by the terminal apparatus 300.);
an accuracy determining unit (Fig. 28-29, #500 called a server apparatus. Paragraph [0245]- TSURUMI discloses the server apparatus 500 collects accuracy information from each information processing apparatus 100 (Step S401))
configured to determine accuracy in estimation of the self-position (Fig. 28, Paragraph [0237]- TSURUMI discloses the server apparatus 500 determines whether the accuracy of the self-position estimation satisfies the desired accuracy according to the extraction accuracy with which the information processing apparatus 100 extracts the feature point P2 from the display area 200R, and determines the shift to the second method based on the determination result.);
an image analyzing unit (Fig. 6, #1316 called an object detection unit. Paragraph [0100 and 0117])
configured to detect an object from the image (Fig. 6. Paragraph [0117]- TSURUMI discloses the object detection unit 1316 detects a display area 200R of the display apparatus 200 by detecting the fixed pattern image from the captured image.);
Although TSURUMI teaches the image analyzing unit (Fig. 6, #1316 called an object detection unit. Paragraph [0100 and 0117]),
TSURUMI fails to explicitly teach and a notification unit configured to notify of information on the object detected by the image analyzing unit in a case in which the accuracy is outside of a predetermined range, wherein in the case in which the accuracy is outside of the predetermined range, in estimation of the self-position.
However, HOLZ explicitly teaches and a notification unit (Fig. 1, #120 called a robotic device, Paragraph [0084]- HOLZ discloses in some embodiments, robotic device(s) 120 can be subject to one or more failure conditions. Examples of those failure conditions and related recovery strategies are described in Table 1 below. Table 1: Pallet Detection Failure - Robotic device expected to discover a pallet [wherein the pallet is the object] at commanded location; no pallet was found - Recovery strategy states robotic device will send message to a control service that includes sensor data relative to where the pallet was expected to be discovered. The control service [wherein the control service is notification unit] will notify human operator and optionally may send pallet pose information manually).
configured to notify of information on the object detected (Fig. 1, #120 called a robotic device. Paragraph [0084]- HOLZ discloses in some embodiments, robotic device(s) 120 can be subject to one or more failure conditions. Examples of those failure conditions and related recovery strategies are described in Table 1 below. Table 1: Pallet Detection Failure - Robotic device expected to discover a pallet [wherein the pallet is the object] at commanded location; no pallet was found - Recovery strategy states robotic device will send message to a control service that includes sensor data relative to where the pallet was expected to be discovered. The control service [wherein the control service is notification unit] will notify human operator and optionally may send pallet pose information manually.)
by the image analyzing unit in a case in which the accuracy is outside of a predetermined range (Fig. 1, #120 called a robotic device. Paragraph [0084]- HOLZ discloses Table 1: Stale Localization - Localization system is unable to determine robotic device pose and/or localization certainty estimate [wherein certainty estimate is accuracy] has exceeded bounds [wherein bounds is predetermined range]. Recovery strategy states robotic device will halt and notify human operator. The operator can intervene by manually driving robotic device to a location for re-localization and reengaging.),
wherein in the case in which the accuracy is outside of the predetermined range (Fig. 1, #120 called a robotic device. Paragraph [0084]- HOLZ discloses Table 1: Stale Localization - Localization system is unable to determine robotic device pose and/or localization certainty estimate [wherein certainty estimate is accuracy] has exceeded bounds [wherein bounds is predetermined range]. Recovery strategy states robotic device will halt and notify human operator. The operator can intervene by manually driving robotic device to a location for re-localization and reengaging.),
HOLZ further teaches in estimation of the self-position (Fig. 7, Paragraph [0094] – HOLZ discloses location and orientation of robot 700 relative to the environment may be estimated, enabling the robot to navigate through the environment accordingly. See also Fig. 1, Paragraph [0047].)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI of having an information processing device comprising at least one processor or circuit configured to function as: a self-position estimating unit configured to estimate a self-position on the basis of an image; an accuracy determining unit configured to determine accuracy in estimation of the self-position; and an image analyzing unit configured to detect an object from the image, with the teachings of HOLZ having and a notification unit configured to notify information on the object detected by the image analyzing unit in a case in which the accuracy is outside of a predetermined range, wherein in the case in which the accuracy is outside of the predetermined range, in estimation of the self-position.
Wherein having TSURUMI’s information processing device wherein having and a notification unit configured to notify information on the object detected by the image analyzing unit in a case in which the accuracy is outside of a predetermined range, wherein in the case in which the accuracy is outside of the predetermined range, in estimation of the self-position.
The motivation behind the modification would have been to obtain an information processing device with a mechanism capable of estimating a self-position more accurately, since both TSURUMI and HOLZ relate to localization processing, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while HOLZ has a system for simultaneous localization and calibration (SLAM) wherein a computing system may perform SLAM or similar processes to determine a location of a robot while also detecting positions of markers within the environment. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and HOLZ (US 20200242396 A1), Paragraph [105].
TSURUMI in view of HOLZ fail to explicitly teach the notification unit is configured to notify by jointly displaying the image, feature points detected from the image, the accuracy, and information indicating the object detected by the image analyzing unit.
However, WATANABE explicitly teaches wherein the notification unit (Fig. 10, #103 called display apparatus, Paragraph [0087] – WATANABE discloses the operation screen is presented to the user on the display apparatus 103)
is configured to notify by jointly displaying the image (Fig. 10, Paragraph [0086] – WATANABE discloses FIG. 10 is a diagram of an exemplary configuration of an operation screen to execute the image retrieval by using the image retrieving apparatus 104. Paragraph [0037] – WATANABE further discloses image retrieving apparatus 104 executes retrieving processing to retrieve an image which matches the retrieval condition from the image database 108 by using the retrieval condition specified by a user from the input apparatus 102 and present the information on the display apparatus 103.),
feature points detected from the image (Fig. 10, Paragraph [0090] – WATANABE discloses the feature points in FIG. 10 are, for example, 0: head, 1: neck, 2: right shoulder, 3: right elbow, 4: right wrist, 5: left shoulder, 6: left elbow, 7: left wrist, 8: right waist, 9: right knee, 10: right ankle, 11: left waist, 12: left knee, 13: left ankle. When the pose information is input, the feature points may be independently moved, and the plurality of feature points may be moved in conjunction with each other in consideration of connection relations. Furthermore, control points and the like other than the feature points may be added.),
the accuracy (Fig. 10, Paragraph [0089] – WATANABE discloses information displayed in the retrieval result display region 1004 is output to the display apparatus 103 by the retrieval result displaying unit 112 [wherein accuracy is degree of similarity shown in Fig. 10 within retrieval result display region 1004]. See also Paragraph [0079].),
and information indicating the object detected by the image analyzing unit (Fig. 10, Paragraph [0089] – WATANABE discloses information displayed in the retrieval result display region 1004 is output to the display apparatus 103 by the retrieval result displaying unit 112. Paragraph [0049] – WATANABE further discloses the data obtained by the retrieving processing is related to the object. Paragraph [0065] – WATANABE discloses pose estimating unit 106 detects a person region from the input image [wherein pose estimating unit is the image analyzing unit].).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI in view of HOLZ of having an information processing device comprising at least one processor or circuit configured to function as: a self-position estimating unit configured to estimate a self-position on the basis of an image; an accuracy determining unit configured to determine accuracy in estimation of the self-position; an image analyzing unit configured to detect an object from the image; and a notification unit configured to notify of information on the object detected by the image analyzing unit in a case in which the accuracy is outside of a predetermined range, wherein in the case in which the accuracy is outside of the predetermined range, in estimation of the self-position, with the teachings of WATANABE having the notification unit is configured to notify by jointly displaying the image, feature points detected from the image, the accuracy, and information indicating the object detected by the image analyzing unit.
Wherein having TSURUMI’s information processing device wherein having the notification unit is configured to notify by jointly displaying the image, feature points detected from the image in estimation of the self-position, the accuracy in estimation of the self-position, and information indicating the object detected by the image analyzing unit.
The motivation behind the modification would have been to obtain an information processing device with an enhanced and more accurate method of object detection, since both TSURUMI and WATANABE relate to image processing and pose estimation, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while WATANABE has an image retrieving apparatus and an image retrieving method capable of improving a retrieval accuracy and a retrieval efficiency by generating a retrieval query reflecting pose information on a retrieval target. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and WATANABE (US 20190147292 A1), Paragraph [0105].
Regarding claim 2, TSURUMI in view of HOLZ and WATANABE teach the information processing device according to claim 1,
Although TSURUMI further teaches by the image analyzing unit (Fig. 6, #1316 called an object detection unit. Paragraph [0100 and 0117]).
TSURUMI fails to explicitly teach wherein the notification unit notifies of a factor on the basis of the object detected by the image analyzing unit in a case in which the accuracy is lower than the predetermined range.
However, HOLZ explicitly teaches wherein the notification unit (Fig. 1, #120 called a robotic device. Paragraph [0084]- HOLZ discloses Table 1.)
notifies of a factor (Fig. 1, #120 called a robotic device. Paragraph [0084]- HOLZ discloses in some embodiments, robotic device(s) 120 can be subject to one or more failure conditions. Examples of those failure conditions and related recovery strategies are described in Table 1 below. Table 1: Pallet Detection Failure - Robotic device expected to discover a pallet [wherein the pallet is the object] at commanded location; no pallet was found - Recovery strategy states robotic device will send message to a control service that includes sensor data [wherein sensor data is factor] relative to where the pallet was expected to be discovered. The control service [wherein the control service is a notification unit] will notify human operator and optionally may send pallet pose information manually) on the basis of the object detected (Fig. 1, robotic device 120, Paragraph [0084], Table 1).
by the image analyzing unit in a case in which the accuracy is lower than the predetermined range (Fig. 1, #120 called a robotic device. Paragraph [0084]- HOLZ discloses in Table 1: Stale Localization - Localization system is unable to determine robotic device pose and/or localization certainty estimate [wherein certainty estimate is accuracy] has exceeded bounds [wherein bounds is predetermined range]. Recovery strategy states robotic device will halt and notify human operator. The operator can intervene by manually driving robotic device to a location for re-localization and reengaging).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI in view of HOLZ and WATANABE of having information processing device comprising at least one processor or circuit configured to function as: a notification unit configured to notify of information on the object detected by the image analyzing unit in a case in which the accuracy is outside of a predetermined range, with the teachings of HOLZ having wherein the notification unit notifies of a factor on the basis of the object detected in a case in which the accuracy is lower than the predetermined range.
Wherein having TSURUMI’s information processing device comprising an image analyzing unit wherein the notification unit notifies of a factor on the basis of the object detected in a case in which the accuracy is lower than the predetermined range.
The motivation behind the modification would have been to obtain an information processing device with a mechanism capable of estimating a self-position more accurately, since both TSURUMI and HOLZ relate to localization processing, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while HOLZ has a system for simultaneous localization and calibration (SLAM) wherein a computing system may perform SLAM or similar processes to determine a location of a robot while also detecting positions of markers within the environment. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and HOLZ (US 20200242396 A1), Paragraph [105].
Regarding claim 4, TSURUMI in view of HOLZ and WATANABE teach the information processing device according to claim 1,
Although TSURUMI further teaches an information processing device (Fig. 5. Paragraph [0054]),
TSURUMI fails to explicitly teach wherein the notification unit performs notification in a case in which the accuracy is higher than the predetermined range.
However, HOLZ explicitly teaches wherein the notification unit (Fig. 1, #120 called a robotic device. Paragraph [0084]- HOLZ discloses Table 1.)
performs notification (Fig. 1, #120 called a robotic device. Paragraph [0084]– HOLZ discloses Pallet Detection Failure - Robotic device expected to discover a pallet at commanded location; no pallet was found - Recovery strategy states robotic device will send message [wherein sending a message is a notification] to a control service that includes sensor data relative to where the pallet was expected to be discovered. The control service will notify human operator and optionally may send pallet pose information manually).
in a case in which the accuracy is higher than the predetermined range (Fig. 1, #120 called a robotic device. Paragraph [0084] Table 1 – HOLZ discloses Stale Localization - Localization system is unable to determine robotic device pose and/or localization certainty estimate [wherein certainty estimate is accuracy] has exceeded bounds [wherein bounds is predetermined range]. Recovery strategy states robotic device will halt and notify human operator. The operator can intervene by manually driving robotic device to a location for re-localization and reengaging).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI in view of HOLZ and WATANABE of having information processing device comprising at least one processor or circuit configured to function as: a notification unit configured to notify of information on the object detected by the image analyzing unit in a case in which the accuracy is outside of a predetermined range, with the teachings of HOLZ having wherein the notification unit performs notification in a case in which the accuracy is higher than the predetermined range.
Wherein having TSURUMI’s information processing device wherein the notification unit performs notification in a case in which the accuracy is higher than the predetermined range.
The motivation behind the modification would have been to obtain an information processing device with a mechanism capable of estimating a self-position more accurately, since both TSURUMI and HOLZ relate to localization processing, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while HOLZ has a system for simultaneous localization and calibration (SLAM) wherein a computing system may perform SLAM or similar processes to determine a location of a robot while also detecting positions of markers within the environment. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and HOLZ (US 20200242396 A1), Paragraph [105].
Regarding claim 5, TSURUMI in view of HOLZ and WATANABE teach the information processing device according to claim 1,
Although TSURUMI further teaches notifies of a value associated with the accuracy (Fig. 29. Paragraph [0246]- TSURUMI discloses the server apparatus 500 determines whether the accuracy of the second method satisfies desired accuracy (Step S402). For example, the server apparatus 500 compares the above-described recognition accuracy with a threshold value, and determines that the desired accuracy is satisfied when the recognition accuracy is equal to or larger than the threshold value).
TSURUMI fails to explicitly teach wherein the notification unit.
However, HOLZ explicitly teaches wherein the notification unit (Fig. 1, #120 called a robotic device, Paragraph [0084], Table 1- HOLZ discloses Pallet Pose Estimation Failure – Robotic device could not determine pose of pallet relative to robotic device at high confidence. Recovery Strategy: Robotic device will send message to a control service that includes sensor data relative to where the pallet was expected. The control service [notification unit] will notify human operator and send pallet pose information manually.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI in view of HOLZ and WATANABE of having information processing device comprising at least one processor or circuit configured to function as: a notification unit configured to notify of information on the object detected by the image analyzing unit in a case in which the accuracy is outside of a predetermined range, with the teachings of HOLZ of having wherein the notification unit.
Wherein having TSURUMI’s information processing device wherein the notification unit notifies of a value associated with the accuracy.
The motivation behind the modification would have been to obtain an information processing device with a mechanism capable of estimating a self-position more accurately, since both TSURUMI and HOLZ relate to localization processing, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while HOLZ has a system for simultaneous localization and calibration (SLAM) wherein a computing system may perform SLAM or similar processes to determine a location of a robot while also detecting positions of markers within the environment. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and HOLZ (US 20200242396 A1), Paragraph [105].
Regarding claim 7, TSURUMI in view of HOLZ and WATANABE teach the information processing device according to claim 1,
Although TSURUMI further teaches wherein the self-position estimating unit (Fig. 5, self-position estimation unit 132, Paragraph [0098] – TSURUMI discloses as illustrated in FIG. 5, the control unit 130 includes a map generation unit 131 and a self-position estimation unit 132),
TSURUMI fails to explicitly teach wherein the self-position estimating unit generates a map on the basis of the image.
However, HOLZ explicitly teaches wherein the self-position estimating unit generates a map (Fig. 10, Paragraph [0135]- HOLZ discloses the computing system [wherein computing system is self-position estimating unit] may also simultaneously develop a map of the markers that specifies locations of the detected makers within the environment. Similar to updating the pose graph, the computing system may also refine the map of markers as new sensor measurements are received from the sensor of the robot.)
on the basis of the image (Paragraph [0133]- HOLZ discloses the computing system may receive sensor data from other types of sensors coupled to a mobile robot, such as camera images, RADAR, etc).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI in view of HOLZ and WATANABE of having information processing device comprising at least one processor or circuit configured to function as a self-position estimating unit configured to estimate a self-position on the basis of an image; with the teachings of HOLZ of having wherein the self-position estimating unit generates a map on the basis of the image.
Wherein having TSURUMI’s information processing device wherein the self-position estimating unit, wherein generates a map on the basis of the image.
The motivation behind the modification would have been to obtain an information processing device with a mechanism capable of estimating a self-position more accurately, since both TSURUMI and HOLZ relate to localization processing, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while HOLZ has a system for simultaneous localization and calibration (SLAM) wherein a computing system may perform SLAM or similar processes to determine a location of a robot while also detecting positions of markers within the environment. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and HOLZ (US 20200242396 A1), Paragraph [105].
Regarding claim 10, TSURUMI teaches an information processing method (Fig. 25-26, Paragraph [0213]- TSURUMI discloses an outline of a self-position estimation method performed by the information processing apparatus 100 in the present modification will be described. The information processing apparatus 100 estimates the self-position using at least one of a first to third methods) comprising:
estimating a self-position on the basis of an image (Paragraph [0221]- TSURUMI discloses in the second method [wherein the method is of self-position estimation], the information processing apparatus 100 performs recognition processing of the display apparatus 200 at the time of map creation, detects the display area [wherein display area is image] 200R of the display apparatus 200, and sets the mask space S. Paragraph [0222]- TSURUMI further discloses in the second method [wherein the method is of self-position estimation], as described above, it is desirable to estimate the self-position of the terminal apparatus 300 with predetermined accuracy in order to classify the feature point with desired accuracy);
determining accuracy in estimation of the self-position (Fig. 28, #500 called a server apparatus Fig. 29. Paragraph [0237]- TSURUMI discloses the server apparatus 500 determines whether the accuracy of the self-position estimation satisfies the desired accuracy according to the extraction accuracy with which the information processing apparatus 100 extracts the feature point P2 from the display area 200R, and determines the shift to the second method based on the determination result);
detecting an object from the image (Fig. 6, #1316 called an object detection unit, Paragraph [0117]- TSURUMI discloses the object detection unit 1316 detects a display area 200R of the display apparatus 200 by detecting the fixed pattern image from the captured image. Paragraph [0118]- TSURUMI further discloses, returning to FIG. 6, the object detection unit 1316 outputs the display area information regarding the detected display area 200R to the mask space calculation unit 1317);
TSURUMI fails to explicitly teach and notifying information on the object detected in the detecting of an object in a case in which the accuracy is outside of a predetermined range, wherein in the case in which the accuracy is outside of the predetermined range, in estimation of the self-position.
However, HOLZ explicitly teaches and notifying information on the object detected (Fig. 1, #134 an obstacle detection subsystem, Paragraph [0052]- HOLZ discloses obstacle detection subsystem 134 can determine whether one or more obstacles [objects] are blocking a path and/or a trajectory of robotic device 120. Examples of these obstacles can include, but are not limited to, pallets, objects that may have fallen off a pallet, robotic devices, and human operators working in the environment. If an obstacle is detected, obstacle detection subsystem 134 can provide one or more communications indicating obstacle detection to path-following subsystem 138.);
in the detecting of an object (Fig. 1, #134 called an obstacle detection subsystem, Paragraph [0052]) in a case in which the accuracy is outside of a predetermined range (Fig. 1, #120 called a robotic device 120, Paragraph [0084]- HOLZ discloses in some embodiments, robotic device(s) 120 can be subject to one or more failure conditions. Examples of those failure conditions and related recovery strategies are described in Table 1 below. Table 1: Stale Localization - Localization system is unable to determine robotic device pose and/or localization certainty estimate [wherein certainty estimate is accuracy] has exceeded bounds [wherein bounds is predetermined range]. Recovery strategy states robotic device will halt and notify human operator. The operator can intervene by manually driving robotic device to a location for re-localization and reengaging),
wherein in the case in which the accuracy is outside of the predetermined range (Fig. 1, #120 called a robotic device. Paragraph [0084]- HOLZ discloses Table 1: Stale Localization - Localization system is unable to determine robotic device pose and/or localization certainty estimate [wherein certainty estimate is accuracy] has exceeded bounds [wherein bounds is predetermined range]. Recovery strategy states robotic device will halt and notify human operator. The operator can intervene by manually driving robotic device to a location for re-localization and reengaging.),
HOLZ further teaches in estimation of the self-position (Fig. 7, Paragraph [0094] – HOLZ discloses location and orientation of robot 700 relative to the environment may be estimated, enabling the robot to navigate through the environment accordingly. See also Fig. 1, Paragraph [0047].)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI of having an information processing method comprising: estimating a self-position on the basis of an image; determining accuracy in estimation of the self-position; detecting an object from the image; with the teachings of HOLZ of having and notifying of information on the object detected in the detecting of an object in a case in which the accuracy is outside of a predetermined range, wherein in the case in which the accuracy is outside of the predetermined range, in estimation of the self-position.
Wherein having TSURUMI’s information processing method wherein having and notifying of information on the object detected in the detecting of an object in a case in which the accuracy is outside of a predetermined range, wherein in the case in which the accuracy is outside of the predetermined range, in estimation of the self-position.
The motivation behind the modification would have been to obtain an information processing method with a mechanism capable of estimating a self-position more accurately, since both TSURUMI and HOLZ relate to localization processing, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while HOLZ has a system for simultaneous localization and calibration (SLAM) wherein a computing system may perform SLAM or similar processes to determine a location of a robot while also detecting positions of markers within the environment. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and HOLZ (US 20200242396 A1), Paragraph [105].
TSURUMI in view of HOLZ fail to explicitly teach the notifying of information includes notifying by jointly displaying the image, feature points detected from the image, the accuracy, and information indicating the object detected in the detecting of the object.
However, WATANABE explicitly teaches the notifying of information includes notifying by jointly displaying the image (Fig. 10, Paragraph [0086] – WATANABE discloses FIG. 10 is a diagram of an exemplary configuration of an operation screen to execute the image retrieval by using the image retrieving apparatus 104. Paragraph [0037] – WATANABE further discloses image retrieving apparatus 104 executes retrieving processing to retrieve an image which matches the retrieval condition from the image database 108 by using the retrieval condition specified by a user from the input apparatus 102 and present the information on the display apparatus 103.),
feature points detected from the image (Fig. 10, Paragraph [0090] – WATANABE discloses the feature points in FIG. 10 are, for example, 0: head, 1: neck, 2: right shoulder, 3: right elbow, 4: right wrist, 5: left shoulder, 6: left elbow, 7: left wrist, 8: right waist, 9: right knee, 10: right ankle, 11: left waist, 12: left knee, 13: left ankle. When the pose information is input, the feature points may be independently moved, and the plurality of feature points may be moved in conjunction with each other in consideration of connection relations. Furthermore, control points and the like other than the feature points may be added.),
the accuracy (Fig. 10, Paragraph [0089] – WATANABE discloses information displayed in the retrieval result display region 1004 is output to the display apparatus 103 by the retrieval result displaying unit 112 [wherein accuracy is degree of similarity shown in Fig. 10 within retrieval result display region 1004]. See also Paragraph [0079].),
and information indicating the object detected in the detecting of the object (Fig. 10, Paragraph [0089] – WATANABE discloses information displayed in the retrieval result display region 1004 is output to the display apparatus 103 by the retrieval result displaying unit 112. Paragraph [0049] – WATANABE further discloses the data obtained by the retrieving processing is related to the object. Paragraph [0065] – WATANABE discloses pose estimating unit 106 detects a person region from the input image.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI in view of HOLZ of having an information processing method comprising: estimating a self-position on the basis of an image; determining accuracy in estimation of the self-position; detecting an object from the image; and notifying of information on the object detected in the detecting of an object in a case in which the accuracy is outside of a predetermined range, wherein in the case in which the accuracy is outside of the predetermined range, in estimation of the self-position, with the teachings of WATANABE having the notifying of information includes notifying by jointly displaying the image, feature points detected from the image, the accuracy, and information indicating the object detected in the detecting of the object.
Wherein having TSURUMI’s information processing device wherein having the notifying of information includes notifying by jointly displaying the image, feature points detected from the image in estimation of the self-position, the accuracy in estimation of the self-position, and information indicating the object detected in the detecting of the object.
The motivation behind the modification would have been to obtain an information processing method with an enhanced and more accurate method of object detection, since both TSURUMI and WATANABE relate to image processing and pose estimation, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while WATANABE has an image retrieving apparatus and an image retrieving method capable of improving a retrieval accuracy and a retrieval efficiency by generating a retrieval query reflecting pose information on a retrieval target. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and WATANABE (US 20190147292 A1), Paragraph [0105].
Regarding claim 11, TSURUMI teaches a non-transitory computer-readable storage medium (Fig. 33, #1400 called an HDD, Paragraph [0288]- TSURUMI discloses the HDD 1400 is a computer-readable recording medium that non-transiently records a program executed by the CPU 1100, data used by the program, and the like.)
configured to store a computer program (Fig. 33, #1400 called an HDD, Paragraph [0288]- TSURUMI discloses the HDD 1400 is a computer-readable recording medium that non-transiently records a program executed by the CPU 1100, data used by the program, and the like. Specifically, the HDD 1400 is a recording medium that records a program for the information processing method according to the present disclosure that is an example of the program data 1450) comprising instructions for executing following processes:
estimating a self-position on the basis of an image (Paragraph [0221] – TSURUMI discloses in the second method [wherein the method is of self-position estimation], the information processing apparatus 100 performs recognition processing of the display apparatus 200 at the time of map creation, detects the display area 200R of the display apparatus 200 [wherein the display area is from the captured image], and sets the mask space S. Paragraph [0222]- TSURUMI further discloses in the second method [wherein the method is of self-position estimation], as described above, it is desirable to estimate the self-position of the terminal apparatus 300 with predetermined accuracy in order to classify the feature point with desired accuracy. See also Paragraph [0117]);
determining accuracy in estimation of the self-position (Fig. 28, #500 called a server apparatus, Fig. 29, Paragraph [0237]- TSURUMI discloses the server apparatus 500 determines whether the accuracy of the self-position estimation satisfies the desired accuracy according to the extraction accuracy with which the information processing apparatus 100 extracts the feature point P2 from the display area 200R, and determines the shift to the second method based on the determination result.);
detecting an object from the image (Fig. 6, #1316 called an object detection unit, Paragraph [0117]- TSURUMI discloses the object detection unit 1316 detects a display area 200R of the display apparatus 200 by detecting the fixed pattern image from the captured image. Paragraph [0118]- Tsurumi further discloses, returning to FIG. 6, the object detection unit 1316 outputs the display area information regarding the detected display area 200R to the mask space calculation unit 1317.);
TSURUMI fails to explicitly teach and notifying of information on the object detected in the detecting of an object in a case in which the accuracy is outside of a predetermined range, wherein in the case in which the accuracy is outside of the predetermined range, in estimation of the self-position.
However, HOLZ explicitly teaches and notifying of information on the object detected (Fig. 1, #134 called an obstacle detection subsystem, Paragraph [0052]- HOLZ discloses obstacle detection subsystem 134 can determine whether one or more obstacles [objects] are blocking a path and/or a trajectory of robotic device 120. Examples of these obstacles can include, but are not limited to, pallets, objects that may have fallen off a pallet, robotic devices, and human operators working in the environment. If an obstacle is detected, obstacle detection subsystem 134 can provide one or more communications indicating obstacle detection to path-following subsystem 138)
in the detecting of an object (Fig. 1, #134 called an obstacle detection subsystem, Paragraph [0052])
in a case in which the accuracy is outside of a predetermined range (Fig. 1, #120 called a robotic device, Paragraph [0084]- HOLZ discloses in some embodiments, robotic device(s) 120 can be subject to one or more failure conditions. Examples of those failure conditions and related recovery strategies are described in Table 1 below. Table 1: Stale Localization - Localization system is unable to determine robotic device pose and/or localization certainty estimate [wherein certainty estimate is accuracy] has exceeded bounds [wherein bounds is predetermined range]. Recovery strategy states robotic device will halt and notify human operator. The operator can intervene by manually driving robotic device to a location for re-localization and reengaging),
wherein in the case in which the accuracy is outside of the predetermined range (Fig. 1, #120 called a robotic device, Paragraph [0084]- HOLZ discloses in some embodiments, robotic device(s) 120 can be subject to one or more failure conditions. Examples of those failure conditions and related recovery strategies are described in Table 1 below. Table 1: Stale Localization - Localization system is unable to determine robotic device pose and/or localization certainty estimate [wherein certainty estimate is accuracy] has exceeded bounds [wherein bounds is predetermined range]. Recovery strategy states robotic device will halt and notify human operator. The operator can intervene by manually driving robotic device to a location for re-localization and reengaging),
in estimation of the self-position (Fig. 7, Paragraph [0094] – HOLZ discloses location and orientation of robot 700 relative to the environment may be estimated, enabling the robot to navigate through the environment accordingly. See also Fig. 1, Paragraph [0047].).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI of having a non-transitory computer-readable storage medium configured to store a computer program comprising instructions for executing following processes: estimating a self-position on the basis of an image, determining accuracy in estimation of the self-position, detecting an object from the image, with the teachings of HOLZ of having and notifying of information on the object detected in the detecting of an object in a case in which the accuracy is outside of a predetermined range, wherein in the case in which the accuracy is outside of the predetermined range, in estimation of the self-position.
Wherein having TSURUMI’s non-transitory computer-readable storage medium wherein having and notifying of information on the object detected in the detecting of an object in a case in which the accuracy is outside of a predetermined range, wherein in the case in which the accuracy is outside of the predetermined range, in estimation of the self-position.
The motivation behind the modification would have been to obtain an information processing device with a mechanism capable of estimating a self-position more accurately, since both TSURUMI and HOLZ relate to localization processing, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while HOLZ has a system for simultaneous localization and calibration (SLAM) wherein a computing system may perform SLAM or similar processes to determine a location of a robot while also detecting positions of markers within the environment. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and HOLZ (US 20200242396 A1), Paragraph [105].
TSURUMI in view of HOLZ fail to explicitly teach the notifying of information includes notifying by jointly displaying the image, feature points detected from the image, the accuracy, and information indicating the object detected in the detecting of the object.
However, WATANABE explicitly teaches the notifying of information includes notifying by jointly displaying the image (Fig. 10, Paragraph [0086] – WATANABE discloses FIG. 10 is a diagram of an exemplary configuration of an operation screen to execute the image retrieval by using the image retrieving apparatus 104. Paragraph [0037] – WATANABE further discloses image retrieving apparatus 104 executes retrieving processing to retrieve an image which matches the retrieval condition from the image database 108 by using the retrieval condition specified by a user from the input apparatus 102 and present the information on the display apparatus 103.),
feature points detected from the image (Fig. 10, Paragraph [0090] – WATANABE discloses the feature points in FIG. 10 are, for example, 0: head, 1: neck, 2: right shoulder, 3: right elbow, 4: right wrist, 5: left shoulder, 6: left elbow, 7: left wrist, 8: right waist, 9: right knee, 10: right ankle, 11: left waist, 12: left knee, 13: left ankle. When the pose information is input, the feature points may be independently moved, and the plurality of feature points may be moved in conjunction with each other in consideration of connection relations. Furthermore, control points and the like other than the feature points may be added.),
the accuracy (Fig. 10, Paragraph [0089] – WATANABE discloses information displayed in the retrieval result display region 1004 is output to the display apparatus 103 by the retrieval result displaying unit 112 [wherein accuracy is degree of similarity shown in Fig. 10 within retrieval result display region 1004]. See also Paragraph [0079].),
and information indicating the object detected in the detecting of the object (Fig. 10, Paragraph [0089] – WATANABE discloses information displayed in the retrieval result display region 1004 is output to the display apparatus 103 by the retrieval result displaying unit 112. Paragraph [0049] – WATANABE further discloses the data obtained by the retrieving processing is related to the object. Paragraph [0065] – WATANABE discloses pose estimating unit 106 detects a person region from the input image.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI in view of HOLZ of having a non-transitory computer-readable storage medium configured to store a computer program comprising instructions for executing following processes: estimating a self-position on the basis of an image; determining accuracy in estimation of the self-position; detecting an object from the image; and notifying of information on the object detected in the detecting of an object in a case in which the accuracy is outside of a predetermined range, wherein in the case in which the accuracy is outside of the predetermined range, in estimation of the self-position, with the teachings of WATANABE having the notifying of information includes notifying by jointly displaying the image, feature points detected from the image, the accuracy, and information indicating the object detected in the detecting of the object.
Wherein having TSURUMI’s non-transitory computer-readable storage medium wherein having the notifying of information includes notifying by jointly displaying the image, feature points detected from the image in estimation of the self-position, the accuracy in estimation of the self-position, and information indicating the object detected in the detecting of the object.
The motivation behind the modification would have been to obtain an information processing device with an enhanced and more accurate method of object detection, since both TSURUMI and WATANABE relate to image processing and pose estimation, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while WATANABE has an image retrieving apparatus and an image retrieving method capable of improving a retrieval accuracy and a retrieval efficiency by generating a retrieval query reflecting pose information on a retrieval target. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and WATANABE (US 20190147292 A1), Paragraph [0105].
Claim 3 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over TSURUMI (US 20240412408 A1), hereinafter referenced as TSURUMI in view of HOLZ (US 20200242396 A1), hereinafter referenced as HOLZ, further in view of WATANABE (US 20190147292 A1), hereinafter referenced as WATANABE, and further in view of BHUTA (US 11295471 B1), hereinafter referenced as BHUTA.
Regarding claim 3, TSURUMI in view of HOLZ and WATANABE in further view of BHUTA teach the information processing device according to claim 1,
Although TSURUMI further teaches by the image analyzing unit (Fig. 6, #1316 called an object detection unit, Paragraph [0117-0118]),
TSURUMI fails to explicitly teach wherein the notification unit notifies of a countermeasure for increasing the accuracy on the basis of the object detected by the image analyzing unit in a case in which the accuracy is lower than the predetermined range.
However, BHUTA explicitly teaches wherein the notification unit notifies of a countermeasure (Fig. 1B, # 174 called a corrective actions data, Col. 4, Lines [53-54]- BHUTA discloses corrective actions data 174 can include data or instructions [wherein data or instructions are countermeasures] for how to correct a poorly-placed pallet. Fig. 5, Col. 13, Lines [52-59] – BHUTA discloses where the detected pallet position is unacceptable, the user interface can include a corrective action that shows and describes how to correct the placement of the pallet 109. While the user interface can include a graphical user interface generated for display, the user interface can also include indicator light instructions, directional light instructions, tone based audio queues, spoken audio instructions, and other user interface types.)
for increasing the accuracy on the basis of the object detected (Fig. 5, Col. 13, Lines [50-54] - BHUTA discloses the user interface can also indicate whether the detected pallet position is acceptable or unacceptable. Where the detected pallet position is unacceptable, the user interface can include a corrective action that shows and describes how to correct [wherein correction is increasing the accuracy of] the placement of the pallet 109 [wherein the pallet is the object])
by the image analyzing unit (Fig. 1B, Col. 12, Lines [61-63] – BHUTA discloses the pallet placement application 154 can perform image processing on an image captured by the camera 103.)
in a case in which the accuracy is lower than the predetermined range (Fig. 8, Col. 16, Lines [27-31] – BHUTA discloses the pallet placement application 154 can determine a detected pallet position based on the largest bound area defined by the orthogonal intersections. In some cases, the largest bound area is only accepted as a detected pallet position if it is greater than a predetermined threshold.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI in view of HOLZ and WATANABE of having the information processing device comprising at least one processor or circuit configured to function as: a notification unit configured to notify of information on the object detected by the image analyzing unit in a case in which the accuracy is outside of a predetermined range, with the teachings of BHUTA of having wherein the notification unit notifies of a countermeasure for increasing the accuracy on the basis of the object detected by the image analyzing unit in a case in which the accuracy is lower than the predetermined range.
Wherein having TSURUMI’s information processing device wherein the notification unit notifies of a countermeasure for increasing the accuracy on the basis of the object detected by the image analyzing unit in a case in which the accuracy is lower than the predetermined range.
The motivation behind the modification would have been to obtain an information processing device with a mechanism capable of estimating a self-position more accurately, since both TSURUMI and BHUTA relate to the navigation of automated guided vehicles (AGVs), wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while BHUTA describes aspects of camera-based pallet placement detection and notification that can prevent AGV failures and increase the efficiency of using AGVs as well as manual powered industrial truck (PIT) equipment. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and BHUTA (US 11295471 B1), Col. 2, Lines [1-9].
Regarding claim 6, TSURUMI in view of HOLZ and WATANABE teach the information processing device according to claim 1,
Although TSURUMI further teaches wherein the at least one processor or circuit is further configured to function as a storage unit (Paragraph [0097]- Tsurumi discloses the control unit 130 is realized by a processor executing various programs stored in a storage device inside the information processing apparatus 100 using a random access memory (RAM) or the like as a work area)
TSURUMI in view of HOLZ and WATANABE fail to explicitly teach wherein the at least one processor or circuit is further configured to function as a storage unit configured to store notification details correlated with the object.
However, BHUTA explicitly teaches wherein the at least one processor or circuit (Fig. 9, #900 called computing device, Col. 17, Lines [61-63] – BHUTA discloses the computing device 900 includes at least one processing system, for example, having a processor 902)
is further configured to function as a storage unit configured to store notification details correlated with the object (Fig. 1B, Col. 4, Lines [1-5] – BHUTA discloses various application components and/or other functionality may be executed in the computing environment 149 according to various embodiments. Also, data 164 may be stored in one or more memory or data store devices and be accessible to the computing environment 149.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI in view of HOLZ and WATANABE of having the information processing device comprising at least one processor or circuit configured to function as: a notification unit configured to notify of information on the object detected by the image analyzing unit in a case in which the accuracy is outside of a predetermined range, with the teachings of BHUTA of having wherein the at least one processor or circuit is further configured to function as a storage unit configured to store notification details correlated with the object.
Wherein having TSURUMI’s information processing device wherein the at least one processor or circuit is further configured to function as a storage unit configured to store notification details correlated with the object.
The motivation behind the modification would have been to obtain an information processing device with a mechanism capable of estimating a self-position more accurately, since both TSURUMI and BHUTA relate to the navigation of automated guided vehicles (AGVs), wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while BHUTA describes aspects of camera-based pallet placement detection and notification that can prevent AGV failures and increase the efficiency of using AGVs as well as manual powered industrial truck (PIT) equipment. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and BHUTA (US 11295471 B1), Col. 2, Lines [1-9].
Claims 8 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over TSURUMI (US 20240412408 A1), hereinafter referenced as TSURUMI in view of HOLZ (US 20200242396 A1), hereinafter referenced as HOLZ, further in view of WATANABE (US 20190147292 A1), hereinafter referenced as WATANABE, and further in view of LEE (US 20230343228 A1), hereinafter referenced as LEE.
Regarding claim 8, TSURUMI in view of HOLZ and WATANABE teach a device according to claim 1,
Although HOLZ further teaches wherein the notification unit determines that the accuracy is lower than the predetermined range (Fig. 1, #120 called a robotic device, Paragraph [0084], Table 1- Holz discloses Stale Localization - Localization system is unable to determine robotic device pose and/or localization certainty estimate [wherein certainty estimate is accuracy] has exceeded bounds [wherein bounds is predetermined range]. Recovery strategy states robotic device will halt and notify human operator. The operator can intervene by manually driving robotic device to a location for re-localization and reengaging)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI in view of HOLZ and WATANABE of having an information processing device comprising at least one processor or circuit configured to function as: a notification unit configured to notify of information on the object detected by the image analyzing unit in a case in which the accuracy is outside of a predetermined range, with the teachings of HOLZ of having wherein the notification unit determines that the accuracy is lower than the predetermined range.
Wherein having TSURUMI’s information processing device wherein the notification unit determines that the accuracy is lower than the predetermined range.
The motivation behind the modification would have been to obtain an information processing device with a mechanism capable of estimating a self-position more accurately, since both TSURUMI and HOLZ relate to localization processing, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while HOLZ has a system for simultaneous localization and calibration (SLAM) wherein a computing system may perform SLAM or similar processes to determine a location of a robot while also detecting positions of markers within the environment. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and HOLZ (US 20200242396 A1), Paragraph [105].
TSURUMI in view of HOLZ and WATANABE fail to explicitly teach in a case in which the number of feature points in estimation of the self-position is equal to or less than a predetermined number.
However, LEE explicitly teaches in a case in which the number of feature points in estimation of the self-position (Fig. 9, Paragraph [0213]- LEE discloses the estimated self-position reliability calculation unit 107 calculates the reliability of the self-position estimated by the self-position estimation unit 105 using correspondence relationship data between the feature point score and the estimated self-position reliability illustrated in FIG. 9)
is equal to or less than a predetermined number (Figs. 7, 9, 11, 13, Paragraph [0322]- LEE discloses the estimated self-position reliability calculation unit 107 compares the calculated reliability of the estimated self-position with the predefined threshold, for example, the “allowable reliability threshold” illustrated in FIG. 7, 9, 11, or 13, and displays a warning on the information display unit 121 of a controller held by a control center or a user (pilot) in a case where the calculated reliability of the estimated self-position is less than the “allowable reliability threshold”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI in view of HOLZ and WATANABE of having an information processing device comprising at least one processor or circuit configured to function as: a notification unit configured to notify of information on the object detected by the image analyzing unit in a case in which the accuracy is outside of a predetermined range, with the teachings of LEE having in a case in which the number of feature points in estimation of the self-position is equal to or less than a predetermined number.
Wherein having TSURUMI’s information processing device where the notification unit determines that the accuracy is lower than the predetermined range in a case in which the number of feature points in estimation of the self-position is equal to or less than a predetermined number.
The motivation behind the modification would have been to obtain an information processing device with a mechanism capable of estimating a self-position more accurately, since both TSURUMI and LEE relate to the navigation of mobile devices, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while LEE describes an information processing apparatus, an information processing system, an information processing method, and a program that enable safe flight or travel of a mobile device, for example, a drone or the like. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and LEE (US 20230343228 A1), Paragraph [0017-0026].
Regarding claim 9, TSURUMI in view of HOLZ and WATANABE teach a device according to claim 1,
HOLZ further teaches wherein the notification unit determines that the accuracy is lower than the predetermined range (Fig. 1, #120 called a robotic device, Paragraph [0084], Table 1- Holz discloses Stale Localization - Localization system is unable to determine robotic device pose and/or localization certainty estimate [wherein certainty estimate is accuracy] has exceeded bounds [wherein bounds is predetermined range]. Recovery strategy states robotic device will halt and notify human operator. The operator can intervene by manually driving robotic device to a location for re-localization and reengaging)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI in view of HOLZ and WATANABE of having an information processing device comprising at least one processor or circuit configured to function as: a notification unit configured to notify of information on the object detected by the image analyzing unit in a case in which the accuracy is outside of a predetermined range, with the teachings of HOLZ of having wherein the notification unit determines that the accuracy is lower than the predetermined range.
Wherein having TSURUMI’s information processing device wherein the notification unit determines that the accuracy is lower than the predetermined range.
The motivation behind the modification would have been to obtain an information processing device with a mechanism capable of estimating a self-position more accurately, since both TSURUMI and HOLZ relate to localization processing, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while HOLZ has a system for simultaneous localization and calibration (SLAM) wherein a computing system may perform SLAM or similar processes to determine a location of a robot while also detecting positions of markers within the environment. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and HOLZ (US 20200242396 A1), Paragraph [105].
TSURUMI in view of HOLZ and WATANABE fail to explicitly teach in a case in which a likelihood of feature points in estimation of the self-position is equal to or less than a predetermined threshold value.
However, LEE explicitly teaches in a case in which the likelihood of feature points in estimation of the self-position (Fig. 9, Paragraph [0213]- LEE discloses the estimated self-position reliability calculation unit 107 calculates the reliability of the self-position estimated by the self-position estimation unit 105 using correspondence relationship data between the feature point score and the estimated self-position reliability illustrated in FIG. 9)
is equal to or less than a predetermined threshold value (Figs. 7, 9, 11, 13, Paragraph [0322]- LEE discloses the estimated self-position reliability calculation unit 107 compares the calculated reliability of the estimated self-position with the predefined threshold, for example, the “allowable reliability threshold” illustrated in FIG. 7, 9, 11, or 13, and displays a warning on the information display unit 121 of a controller held by a control center or a user (pilot) in a case where the calculated reliability of the estimated self-position is less than the “allowable reliability threshold”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI in view of HOLZ and WATANABE of having an information processing device comprising at least one processor or circuit configured to function as: a notification unit configured to notify of information on the object detected by the image analyzing unit in a case in which the accuracy is outside of a predetermined range, with the teachings of LEE having in a case in which a likelihood of feature points in estimation of the self-position is equal to or less than a predetermined threshold value.
Wherein having TSURUMI’s information processing device where the notification unit determines that the accuracy is lower than the predetermined range in a case in which a likelihood of feature points in estimation of the self-position is equal to or less than a predetermined threshold value.
The motivation behind the modification would have been to obtain an information processing device with a mechanism capable of estimating a self-position more accurately, since both TSURUMI and LEE relate to the navigation of mobile devices, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while LEE describes an information processing apparatus, an information processing system, an information processing method, and a program that enable safe flight or travel of a mobile device, for example, a drone or the like. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and LEE (US 20230343228 A1), Paragraph [0017-0026].
Conclusion
Listed below are the prior arts made of record and not relied upon but are considered pertinent to applicant`s disclosure.
Komuro et al. (US 20230039143 A1) - An own-position estimating device for estimating an own-position of a moving body by matching a feature extracted from an acquired image with a database in which position information and the feature are associated with each other in advance, includes an estimating unit estimating the own-position of the moving body by matching the feature extracted by the extracting unit with the database, and a determination threshold value adjusting unit adjusting a determination threshold value for extracting the feature, in which the determination threshold value adjusting unit acquires the database in a state in which the determination threshold value is adjusted, and adjusts the determination threshold value on the basis of the determination threshold value linked to each of the position information items in the database, and the extracting unit extracts the feature from the image by using the determination threshold value adjusted by the determination threshold value adjusting unit.…….. Fig. 1, Abstract.
Ono et al. (US 20250069301 A1) - The present disclosure relates to an information processing device, an information processing method, and a program that make it easy to use an application program to which XR technology is applied in a plurality of client devices. When local maps generated by client devices are combined to generate a global map, and SLAM initialization processing for the global map is executed in a server, and, in a case where the combination of the local maps fails, the server transmits guide information for solving the failure of the combination according to a cause of the failure to the client device and presents the guide information to a user. The present disclosure is applicable to an application program using XR technology.….. Fig. 1, Abstract.
Ono et al. (US 20220297717 A1) - A display device includes an electronic control unit configured to: obtain a traveling plan from an automatic driving system of a vehicle, the traveling plan including a course of the vehicle under automatic drive control of the automatic driving system; obtain, from the automatic driving system, a system confidence level of the automatic drive control calculated based on at least an external environment around the vehicle; and display, on a display, a pointer as an image indicating the course during the automatic drive control in a display mode set based on the system confidence level.….. Fig. 1, Abstract.
YOSHIMURA et al. (US 20220291686 A1) - A self-location estimation device includes a first self-location estimation unit, a second self-location estimation unit, and a first integration unit. The first self-location estimation unit estimates a current first self-location of an autonomous mobile body based on current image information acquired by an image sensor and environmental map information stored in an environmental map information storage unit. The second self-location estimation unit estimates a current second self-location of the autonomous mobile body based on the current image information and a learned parameter learned using the environmental map information. The first integration unit estimates a current self-location of the autonomous mobile body by integrating the first self-location and the second self-location...….. Fig. 1, Abstract.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BEZAWIT N SHIMELES whose telephone number is (571)272-7663. The examiner can normally be reached M-F 7:30am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BEZAWIT NOLAWI SHIMELES/Examiner, Art Unit 2673
/CHINEYERE WILLS-BURNS/Supervisory Patent Examiner, Art Unit 2673
Prosecution Insights