DETAILED ACTION
Notice of AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers submitted under 35 U.S.C. 119(a)-(d), which papers have been placed of record in the file.
Response to Amendments
Applicant’s remarks, see page 7, filed 02/11/2026, with respect to objections to minor informalities found within the specification, submitted in the non-final office action dated 11/25/2025, have been fully considered and are persuasive due to amendments in accordance with Examiner’s suggested corrections. Thus, objections to minor informalities found within the specification have been withdrawn.
Response to Arguments
Applicant’s arguments, see remarks, filed 02/11/2026, with respect to claims 1-11, have been fully considered, but are moot because the arguments do not apply to the current references and current combinations of references being used in the current rejection.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
Claims 1, 6, and 11 recite limitations that use words like “means” (or “step”) or similar terms with functional language but do not invoke 35 U.S.C. 112(f):
Claim 1; recites the limitation, “at least one processor or circuit configured to …,” [Lines 1-2].
Claim 6; recites the limitation, “at least one processor or circuit is further configured to …,” [Line 2].
Claim 11; recites the limitation, “a non-transitory computer-readable storage medium configured to …,” [Line 1].
Such claim limitation(s) is/are:
(i) “processor or circuit…” has a structure associated with it (i.e. physical hardware being a processor or circuit).
(ii) “a non-transitory computer-readable storage medium…” has a structure associated with it a memory.
Note: The office understands in claims 1 and 6, all units are understood to be a processor or circuit based on the claim language “at least one processor or circuit configured to function as:”.
Because this/these claim limitation(s) is/are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are not being interpreted to cover only the corresponding structure, material, or acts described in the specification as performing the claimed function, and equivalents thereof.
If applicant intends to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to remove the structure, materials, or acts that performs the claimed function; or (2) present a sufficient showing that the claim limitation(s) does/do not recite sufficient structure, materials, or acts to perform the claimed function.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 4, 5, 7, 10, and 11 are rejected under 35 U.S.C. 103 as being unpatentable over TSURUMI (US 20240412408 A1), hereinafter referenced as TSURUMI in view of HOLZ (US 20200242396 A1), hereinafter referenced as HOLZ, further in view of WATANABE (US 20190147292 A1), hereinafter referenced as WATANABE.
Regarding claim 1, TSURUMI teaches an information processing device (Fig. 5, #100 called an information processing apparatus. Paragraph [0054])
comprising at least one processor or circuit (Fig. 5, #100 called an information processing apparatus. Paragraph [0054] – TSURUMI discloses the information processing apparatus 100 is realized by, for example, a processing circuit such as a central processing unit (CPU) or a graphics processing unit (GPU)) configured to function as:
a self-position estimating unit (Fig. 5, #132 called a self-position estimation unit. Paragraph [0098] - TSURUMI discloses as illustrated in FIG. 5, the control unit 130 includes a map generation unit 131 and a self-position estimation unit 132.)
configured to estimate a self-position on the basis of an image (Fig. 5. Paragraph [0137]- TSURUMI discloses the self-position estimation unit 132 estimates the self-position of the terminal apparatus 300 using at least either the mask space or the feature point map and the captured image acquired by the terminal apparatus 300.);
an accuracy determining unit (Fig. 28-29, #500 called a server apparatus. Paragraph [0245]- TSURUMI discloses the server apparatus 500 collects accuracy information from each information processing apparatus 100 (Step S401))
configured to determine accuracy in estimation of the self-position (Fig. 28, Paragraph [0237]- TSURUMI discloses the server apparatus 500 determines whether the accuracy of the self-position estimation satisfies the desired accuracy according to the extraction accuracy with which the information processing apparatus 100 extracts the feature point P2 from the display area 200R, and determines the shift to the second method based on the determination result.);
an image analyzing unit (Fig. 6, #1316 called an object detection unit. Paragraph [0100 and 0117])
configured to detect an object from the image (Fig. 6. Paragraph [0117]- TSURUMI discloses the object detection unit 1316 detects a display area 200R of the display apparatus 200 by detecting the fixed pattern image from the captured image.);
Although TSURUMI teaches the image analyzing unit (Fig. 6, #1316 called an object detection unit. Paragraph [0100 and 0117]).
TSURUMI fails to explicitly teach a notification unit configured to notify of information on the object detected by the image analyzing unit in a case in which the accuracy is outside of a predetermined range.
However, HOLZ explicitly teaches a notification unit (Fig. 1, #120 called a robotic device, Paragraph [0084]- HOLZ discloses in some embodiments, robotic device(s) 120 can be subject to one or more failure conditions. Examples of those failure conditions and related recovery strategies are described in Table 1 below. Table 1: Pallet Detection Failure - Robotic device expected to discover a pallet [wherein the pallet is the object] at commanded location; no pallet was found - Recovery strategy states robotic device will send message to a control service that includes sensor data relative to where the pallet was expected to be discovered. The control service [wherein the control service is notification unit] will notify human operator and optionally may send pallet pose information manually).
configured to notify of information on the object detected (Fig. 1, #120 called a robotic device. Paragraph [0084]- HOLZ discloses in some embodiments, robotic device(s) 120 can be subject to one or more failure conditions. Examples of those failure conditions and related recovery strategies are described in Table 1 below. Table 1: Pallet Detection Failure - Robotic device expected to discover a pallet [wherein the pallet is the object] at commanded location; no pallet was found - Recovery strategy states robotic device will send message to a control service that includes sensor data relative to where the pallet was expected to be discovered. The control service [wherein the control service is notification unit] will notify human operator and optionally may send pallet pose information manually.)
by the image analyzing unit in a case in which the accuracy is outside of a predetermined range (Fig. 1, #120 called a robotic device. Paragraph [0084]- HOLZ discloses Table 1: Stale Localization - Localization system is unable to determine robotic device pose and/or localization certainty estimate [wherein certainty estimate is accuracy] has exceeded bounds [wherein bounds is predetermined range]. Recovery strategy states robotic device will halt and notify human operator. The operator can intervene by manually driving robotic device to a location for re-localization and reengaging.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI of having an information processing device comprising at least one processor or circuit configured to function as: a self-position estimating unit configured to estimate a self-position on the basis of an image; an accuracy determining unit configured to determine accuracy in estimation of the self-position; and an image analyzing unit configured to detect an object from the image, with the teachings of HOLZ having wherein a notification unit configured to notify information on the object detected in a case in which the accuracy is outside of a predetermined range.
Wherein having TSURUMI’s information processing device comprising at least one processor or circuit wherein a notification unit configured to notify information on the object detected in a case in which the accuracy is outside of a predetermined range.
The motivation behind the modification would have been to obtain an information processing device with a mechanism capable of estimating a self-position more accurately, since both TSURUMI and HOLZ relate to localization processing, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while HOLZ has a system for simultaneous localization and calibration (SLAM) wherein a computing system may perform SLAM or similar processes to determine a location of a robot while also detecting positions of markers within the environment. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and HOLZ (US 20200242396 A1), Paragraph [105].
TSURUMI in view of HOLZ fail to explicitly teach wherein the notification unit is configured to notify by jointly displaying the image, feature points detected from the image, the accuracy, and information indicating the object.
However, WATANABE explicitly teaches wherein the notification unit (Fig. 10, #103 called display apparatus, Paragraph [0087] – WATANABE discloses the operation screen is presented to the user on the display apparatus 103)
is configured to notify by jointly displaying the image (Fig. 10, Paragraph [0086] – WATANABE discloses FIG. 10 is a diagram of an exemplary configuration of an operation screen to execute the image retrieval by using the image retrieving apparatus 104. Paragraph [0037] – WATANABE further discloses image retrieving apparatus 104 executes retrieving processing to retrieve an image which matches the retrieval condition from the image database 108 by using the retrieval condition specified by a user from the input apparatus 102 and present the information on the display apparatus 103.),
feature points detected from the image (Fig. 10, Paragraph [0090] – WATANABE discloses the feature points in FIG. 10 are, for example, 0: head, 1: neck, 2: right shoulder, 3: right elbow, 4: right wrist, 5: left shoulder, 6: left elbow, 7: left wrist, 8: right waist, 9: right knee, 10: right ankle, 11: left waist, 12: left knee, 13: left ankle. When the pose information is input, the feature points may be independently moved, and the plurality of feature points may be moved in conjunction with each other in consideration of connection relations. Furthermore, control points and the like other than the feature points may be added.),
the accuracy (Fig. 10, Paragraph [0089] – WATANABE discloses information displayed in the retrieval result display region 1004 is output to the display apparatus 103 by the retrieval result displaying unit 112 [wherein accuracy is degree of similarity shown in Fig. 10 within retrieval result display region 1004]. See also Paragraph [0079].),
and information indicating the object (Fig. 10, Paragraph [0089] – WATANABE discloses information displayed in the retrieval result display region 1004 is output to the display apparatus 103 by the retrieval result displaying unit 112. Paragraph [0049] – WATANABE further discloses the data obtained by the retrieving processing is related to the object.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI in view of HOLZ of having an information processing device comprising at least one processor or circuit configured to function as: a self-position estimating unit configured to estimate a self-position on the basis of an image; an accuracy determining unit configured to determine accuracy in estimation of the self-position; an image analyzing unit configured to detect an object from the image; and a notification unit configured to notify of information on the object detected by the image analyzing unit in a case in which the accuracy is outside of a predetermined range, with the teachings of WATANABE having wherein the notification unit is configured to notify by jointly displaying the image, feature points detected from the image, the accuracy, and information indicating the object.
Wherein having TSURUMI’s information processing device wherein the notification unit is configured to notify by jointly displaying the image, feature points detected from the image, the accuracy, and information indicating the object.
The motivation behind the modification would have been to obtain an information processing device with an enhanced and more accurate method of object detection, since both TSURUMI and WATANABE relate to image processing and pose estimation, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while WATANABE has an image retrieving apparatus and an image retrieving method capable of improving a retrieval accuracy and a retrieval efficiency by generating a retrieval query reflecting pose information on a retrieval target. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and WATANABE (US 20190147292 A1), Paragraph [0105].
Regarding claim 2, TSURUMI in view of HOLZ and WATANABE teach the information processing device according to claim 1,
Although TSURUMI further teaches by the image analyzing unit (Fig. 6, #1316 called an object detection unit. Paragraph [0100 and 0117]).
TSURUMI fails to explicitly teach wherein the notification unit notifies of a factor on the basis of the object detected by the image analyzing unit in a case in which the accuracy is lower than the predetermined range.
However, HOLZ explicitly teaches wherein the notification unit (Fig. 1, #120 called a robotic device. Paragraph [0084]- HOLZ discloses Table 1.)
notifies of a factor (Fig. 1, #120 called a robotic device. Paragraph [0084]- HOLZ discloses in some embodiments, robotic device(s) 120 can be subject to one or more failure conditions. Examples of those failure conditions and related recovery strategies are described in Table 1 below. Table 1: Pallet Detection Failure - Robotic device expected to discover a pallet [wherein the pallet is the object] at commanded location; no pallet was found - Recovery strategy states robotic device will send message to a control service that includes sensor data [wherein sensor data is factor] relative to where the pallet was expected to be discovered. The control service [wherein the control service is a notification unit] will notify human operator and optionally may send pallet pose information manually) on the basis of the object detected (Fig. 1, robotic device 120, Paragraph [0084], Table 1).
by the image analyzing unit in a case in which the accuracy is lower than the predetermined range (Fig. 1, #120 called a robotic device. Paragraph [0084]- HOLZ discloses in Table 1: Stale Localization - Localization system is unable to determine robotic device pose and/or localization certainty estimate [wherein certainty estimate is accuracy] has exceeded bounds [wherein bounds is predetermined range]. Recovery strategy states robotic device will halt and notify human operator. The operator can intervene by manually driving robotic device to a location for re-localization and reengaging).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI in view of HOLZ and WATANABE of having information processing device comprising at least one processor or circuit configured to function as: a notification unit configured to notify of information on the object detected by the image analyzing unit in a case in which the accuracy is outside of a predetermined range, with the teachings of HOLZ having wherein the notification unit notifies of a factor on the basis of the object detected in a case in which the accuracy is lower than the predetermined range.
Wherein having TSURUMI’s information processing device comprising an image analyzing unit wherein the notification unit notifies of a factor on the basis of the object detected in a case in which the accuracy is lower than the predetermined range.
The motivation behind the modification would have been to obtain an information processing device with a mechanism capable of estimating a self-position more accurately, since both TSURUMI and HOLZ relate to localization processing, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while HOLZ has a system for simultaneous localization and calibration (SLAM) wherein a computing system may perform SLAM or similar processes to determine a location of a robot while also detecting positions of markers within the environment. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and HOLZ (US 20200242396 A1), Paragraph [105].
Regarding claim 4, TSURUMI in view of HOLZ and WATANABE teach the information processing device according to claim 1,
Although TSURUMI further teaches an information processing device (Fig. 5. Paragraph [0054]),
TSURUMI fails to explicitly teach wherein the notification unit performs notification in a case in which the accuracy is higher than the predetermined range.
However, HOLZ explicitly teaches wherein the notification unit (Fig. 1, #120 called a robotic device. Paragraph [0084]- HOLZ discloses Table 1.)
performs notification (Fig. 1, #120 called a robotic device. Paragraph [0084]– HOLZ discloses Pallet Detection Failure - Robotic device expected to discover a pallet at commanded location; no pallet was found - Recovery strategy states robotic device will send message [wherein sending a message is a notification] to a control service that includes sensor data relative to where the pallet was expected to be discovered. The control service will notify human operator and optionally may send pallet pose information manually).
in a case in which the accuracy is higher than the predetermined range (Fig. 1, #120 called a robotic device. Paragraph [0084] Table 1 – HOLZ discloses Stale Localization - Localization system is unable to determine robotic device pose and/or localization certainty estimate [wherein certainty estimate is accuracy] has exceeded bounds [wherein bounds is predetermined range]. Recovery strategy states robotic device will halt and notify human operator. The operator can intervene by manually driving robotic device to a location for re-localization and reengaging).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI in view of HOLZ and WATANABE of having information processing device comprising at least one processor or circuit configured to function as: a notification unit configured to notify of information on the object detected by the image analyzing unit in a case in which the accuracy is outside of a predetermined range, with the teachings of HOLZ having wherein the notification unit performs notification in a case in which the accuracy is higher than the predetermined range.
Wherein having TSURUMI’s information processing device wherein the notification unit performs notification in a case in which the accuracy is higher than the predetermined range.
The motivation behind the modification would have been to obtain an information processing device with a mechanism capable of estimating a self-position more accurately, since both TSURUMI and HOLZ relate to localization processing, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while HOLZ has a system for simultaneous localization and calibration (SLAM) wherein a computing system may perform SLAM or similar processes to determine a location of a robot while also detecting positions of markers within the environment. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and HOLZ (US 20200242396 A1), Paragraph [105].
Regarding claim 5, TSURUMI in view of HOLZ and WATANABE teach the information processing device according to claim 1,
Although TSURUMI further teaches notifies of a value associated with the accuracy (Fig. 29. Paragraph [0246]- TSURUMI discloses the server apparatus 500 determines whether the accuracy of the second method satisfies desired accuracy (Step S402). For example, the server apparatus 500 compares the above-described recognition accuracy with a threshold value, and determines that the desired accuracy is satisfied when the recognition accuracy is equal to or larger than the threshold value).
TSURUMI fails to explicitly teach wherein the notification unit.
However, HOLZ explicitly teaches wherein the notification unit (Fig. 1, #120 called a robotic device, Paragraph [0084], Table 1- HOLZ discloses Pallet Pose Estimation Failure – Robotic device could not determine pose of pallet relative to robotic device at high confidence. Recovery Strategy: Robotic device will send message to a control service that includes sensor data relative to where the pallet was expected. The control service [notification unit] will notify human operator and send pallet pose information manually.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI in view of HOLZ and WATANABE of having information processing device comprising at least one processor or circuit configured to function as: a notification unit configured to notify of information on the object detected by the image analyzing unit in a case in which the accuracy is outside of a predetermined range, with the teachings of HOLZ of having wherein the notification unit.
Wherein having TSURUMI’s information processing device wherein the notification unit notifies of a value associated with the accuracy.
The motivation behind the modification would have been to obtain an information processing device with a mechanism capable of estimating a self-position more accurately, since both TSURUMI and HOLZ relate to localization processing, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while HOLZ has a system for simultaneous localization and calibration (SLAM) wherein a computing system may perform SLAM or similar processes to determine a location of a robot while also detecting positions of markers within the environment. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and HOLZ (US 20200242396 A1), Paragraph [105].
Regarding claim 7, TSURUMI in view of HOLZ and WATANABE teach the information processing device according to claim 1,
Although TSURUMI further teaches wherein the self-position estimating unit (Fig. 5, self-position estimation unit 132, Paragraph [0098] – TSURUMI discloses as illustrated in FIG. 5, the control unit 130 includes a map generation unit 131 and a self-position estimation unit 132),
TSURUMI fails to explicitly teach wherein the self-position estimating unit generates a map on the basis of the image.
However, HOLZ explicitly teaches wherein the self-position estimating unit generates a map (Fig. 10, Paragraph [0135]- HOLZ discloses the computing system [wherein computing system is self-position estimating unit] may also simultaneously develop a map of the markers that specifies locations of the detected makers within the environment. Similar to updating the pose graph, the computing system may also refine the map of markers as new sensor measurements are received from the sensor of the robot.)
on the basis of the image (Paragraph [0133]- HOLZ discloses the computing system may receive sensor data from other types of sensors coupled to a mobile robot, such as camera images, RADAR, etc).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI in view of HOLZ and WATANABE of having information processing device comprising at least one processor or circuit configured to function as a self-position estimating unit configured to estimate a self-position on the basis of an image; with the teachings of HOLZ of having wherein the self-position estimating unit generates a map on the basis of the image.
Wherein having TSURUMI’s information processing device wherein the self-position estimating unit, wherein generates a map on the basis of the image.
The motivation behind the modification would have been to obtain an information processing device with a mechanism capable of estimating a self-position more accurately, since both TSURUMI and HOLZ relate to localization processing, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while HOLZ has a system for simultaneous localization and calibration (SLAM) wherein a computing system may perform SLAM or similar processes to determine a location of a robot while also detecting positions of markers within the environment. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and HOLZ (US 20200242396 A1), Paragraph [105].
Regarding claim 10, TSURUMI teaches an information processing method (Fig. 25-26, Paragraph [0213]- TSURUMI discloses an outline of a self-position estimation method performed by the information processing apparatus 100 in the present modification will be described. The information processing apparatus 100 estimates the self-position using at least one of a first to third methods) comprising:
estimating a self-position on the basis of an image (Paragraph [0221]- TSURUMI discloses in the second method [wherein the method is of self-position estimation], the information processing apparatus 100 performs recognition processing of the display apparatus 200 at the time of map creation, detects the display area [wherein display area is image] 200R of the display apparatus 200, and sets the mask space S. Paragraph [0222]- TSURUMI further discloses in the second method [wherein the method is of self-position estimation], as described above, it is desirable to estimate the self-position of the terminal apparatus 300 with predetermined accuracy in order to classify the feature point with desired accuracy);
determining accuracy in estimation of the self-position (Fig. 28, #500 called a server apparatus Fig. 29. Paragraph [0237]- TSURUMI discloses the server apparatus 500 determines whether the accuracy of the self-position estimation satisfies the desired accuracy according to the extraction accuracy with which the information processing apparatus 100 extracts the feature point P2 from the display area 200R, and determines the shift to the second method based on the determination result);
detecting an object from the image (Fig. 6, #1316 called an object detection unit, Paragraph [0117]- TSURUMI discloses the object detection unit 1316 detects a display area 200R of the display apparatus 200 by detecting the fixed pattern image from the captured image. Paragraph [0118]- TSURUMI further discloses, returning to FIG. 6, the object detection unit 1316 outputs the display area information regarding the detected display area 200R to the mask space calculation unit 1317);
Although TSURUMI teaches detecting an object (Fig. 6, #1316 called an object detection unit Paragraph [0117, 0118]),
TSURUMI fails to explicitly teach and notifying information on the object detected in the detecting of an object in a case in which the accuracy is outside of a predetermined range.
However, HOLZ explicitly teaches and notifying information on the object detected (Fig. 1, #134 an obstacle detection subsystem, Paragraph [0052]- HOLZ discloses obstacle detection subsystem 134 can determine whether one or more obstacles [objects] are blocking a path and/or a trajectory of robotic device 120. Examples of these obstacles can include, but are not limited to, pallets, objects that may have fallen off a pallet, robotic devices, and human operators working in the environment. If an obstacle is detected, obstacle detection subsystem 134 can provide one or more communications indicating obstacle detection to path-following subsystem 138.);
in the detecting of an object (Fig. 1, #134 called an obstacle detection subsystem, Paragraph [0052]) in a case in which the accuracy is outside of a predetermined range (Fig. 1, #120 called a robotic device 120, Paragraph [0084]- HOLZ discloses in some embodiments, robotic device(s) 120 can be subject to one or more failure conditions. Examples of those failure conditions and related recovery strategies are described in Table 1 below. Table 1: Stale Localization - Localization system is unable to determine robotic device pose and/or localization certainty estimate [wherein certainty estimate is accuracy] has exceeded bounds [wherein bounds is predetermined range]. Recovery strategy states robotic device will halt and notify human operator. The operator can intervene by manually driving robotic device to a location for re-localization and reengaging).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI of having an information processing method comprising: estimating a self-position on the basis of an image; determining accuracy in estimation of the self-position; detecting an object from the image, with the teachings of HOLZ having and notifying of information on the object detected in the detecting of an object in a case in which the accuracy is outside of a predetermined range.
Wherein having TSURUMI’s information processing device comprising at least one processor or circuit and notifying of information on the object detected in the detecting of an object in a case in which the accuracy is outside of a predetermined range.
The motivation behind the modification would have been to obtain an information processing method with a mechanism capable of estimating a self-position more accurately, since both TSURUMI and HOLZ relate to localization processing, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while HOLZ has a system for simultaneous localization and calibration (SLAM) wherein a computing system may perform SLAM or similar processes to determine a location of a robot while also detecting positions of markers within the environment. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and HOLZ (US 20200242396 A1), Paragraph [105].
TSURUMI in view of HOLZ fail to explicitly teach wherein the notifying of information includes notifying by jointly displaying the image, feature points detected from the image, the accuracy, and information indicating the object.
However, WATANABE explicitly teaches wherein the notifying of information includes notifying by jointly displaying the image (Fig. 10, Paragraph [0086] – WATANABE discloses FIG. 10 is a diagram of an exemplary configuration of an operation screen to execute the image retrieval by using the image retrieving apparatus 104. Paragraph [0037] – WATANABE further discloses image retrieving apparatus 104 executes retrieving processing to retrieve an image which matches the retrieval condition from the image database 108 by using the retrieval condition specified by a user from the input apparatus 102 and present the information on the display apparatus 103.),
feature points detected from the image (Fig. 10, Paragraph [0090] – WATANABE discloses the feature points in FIG. 10 are, for example, 0: head, 1: neck, 2: right shoulder, 3: right elbow, 4: right wrist, 5: left shoulder, 6: left elbow, 7: left wrist, 8: right waist, 9: right knee, 10: right ankle, 11: left waist, 12: left knee, 13: left ankle. When the pose information is input, the feature points may be independently moved, and the plurality of feature points may be moved in conjunction with each other in consideration of connection relations. Furthermore, control points and the like other than the feature points may be added.),
the accuracy (Fig. 10, Paragraph [0089] – WATANABE discloses information displayed in the retrieval result display region 1004 is output to the display apparatus 103 by the retrieval result displaying unit 112 [wherein accuracy is degree of similarity shown in Fig. 10 within retrieval result display region 1004]. See also Paragraph [0079].),
and information indicating the object (Fig. 10, Paragraph [0089] – WATANABE discloses information displayed in the retrieval result display region 1004 is output to the display apparatus 103 by the retrieval result displaying unit 112. Paragraph [0049] – WATANABE further discloses the data obtained by the retrieving processing is related to the object.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI in view of HOLZ of having information processing method comprising: estimating a self-position on the basis of an image; determining accuracy in estimation of the self-position; detecting an object from the image; and notifying of information on the object detected in the detecting of an object in a case in which the accuracy is outside of a predetermined range, with the teachings of WATANABE having wherein the notifying of information includes notifying by jointly displaying the image, feature points detected from the image, the accuracy, and information indicating the object.
Wherein having TSURUMI’s information processing device wherein the notifying of information includes notifying by jointly displaying the image, feature points detected from the image, the accuracy, and information indicating the object.
The motivation behind the modification would have been to obtain an information processing method with an enhanced and more accurate method of object detection, since both TSURUMI and WATANABE relate to image processing and pose estimation, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while WATANABE has an image retrieving apparatus and an image retrieving method capable of improving a retrieval accuracy and a retrieval efficiency by generating a retrieval query reflecting pose information on a retrieval target. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and WATANABE (US 20190147292 A1), Paragraph [0105].
Regarding claim 11, TSURUMI teaches a non-transitory computer-readable storage medium (Fig. 33, #1400 called an HDD, Paragraph [0288]- TSURUMI discloses the HDD 1400 is a computer-readable recording medium that non-transiently records a program executed by the CPU 1100, data used by the program, and the like.)
configured to store a computer program (Fig. 33, #1400 called an HDD, Paragraph [0288]- TSURUMI discloses the HDD 1400 is a computer-readable recording medium that non-transiently records a program executed by the CPU 1100, data used by the program, and the like. Specifically, the HDD 1400 is a recording medium that records a program for the information processing method according to the present disclosure that is an example of the program data 1450) comprising instructions for executing following processes:
estimating a self-position on the basis of an image (Paragraph [0221] – TSURUMI discloses in the second method [wherein the method is of self-position estimation], the information processing apparatus 100 performs recognition processing of the display apparatus 200 at the time of map creation, detects the display area 200R of the display apparatus 200 [wherein the display area is from the captured image], and sets the mask space S. Paragraph [0222]- TSURUMI further discloses in the second method [wherein the method is of self-position estimation], as described above, it is desirable to estimate the self-position of the terminal apparatus 300 with predetermined accuracy in order to classify the feature point with desired accuracy. See also Paragraph [0117]);
determining accuracy in estimation of the self-position (Fig. 28, #500 called a server apparatus, Fig. 29, Paragraph [0237]- TSURUMI discloses the server apparatus 500 determines whether the accuracy of the self-position estimation satisfies the desired accuracy according to the extraction accuracy with which the information processing apparatus 100 extracts the feature point P2 from the display area 200R, and determines the shift to the second method based on the determination result.);
detecting an object from the image (Fig. 6, #1316 called an object detection unit, Paragraph [0117]- TSURUMI discloses the object detection unit 1316 detects a display area 200R of the display apparatus 200 by detecting the fixed pattern image from the captured image. Paragraph [0118]- Tsurumi further discloses, returning to FIG. 6, the object detection unit 1316 outputs the display area information regarding the detected display area 200R to the mask space calculation unit 1317.);
Although TSURUMI further teaches detecting an object from the image (Fig. 6, #1316 called an object detection unit, Paragraph [0117-0118]);
TSURUMI fails to explicitly teach and notifying of information on the object detected in the detecting of an object in a case in which the accuracy is outside of a predetermined range,
However, HOLZ explicitly teaches and notifying of information on the object detected (Fig. 1, #134 called an obstacle detection subsystem, Paragraph [0052]- HOLZ discloses obstacle detection subsystem 134 can determine whether one or more obstacles [objects] are blocking a path and/or a trajectory of robotic device 120. Examples of these obstacles can include, but are not limited to, pallets, objects that may have fallen off a pallet, robotic devices, and human operators working in the environment. If an obstacle is detected, obstacle detection subsystem 134 can provide one or more communications indicating obstacle detection to path-following subsystem 138)
in the detecting of an object (Fig. 1, #134 called an obstacle detection subsystem, Paragraph [0052])
in a case in which the accuracy is outside of a predetermined range (Fig. 1, #120 called a robotic device, Paragraph [0084]- HOLZ discloses in some embodiments, robotic device(s) 120 can be subject to one or more failure conditions. Examples of those failure conditions and related recovery strategies are described in Table 1 below. Table 1: Stale Localization - Localization system is unable to determine robotic device pose and/or localization certainty estimate [wherein certainty estimate is accuracy] has exceeded bounds [wherein bounds is predetermined range]. Recovery strategy states robotic device will halt and notify human operator. The operator can intervene by manually driving robotic device to a location for re-localization and reengaging),
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI of having a non-transitory computer-readable storage medium configured to store a computer program comprising instructions for executing the following processes: estimating a self-position on the basis of an image, determining accuracy in estimation of the self-position, detecting an object from the image, with the teachings of HOLZ of having and notifying of information on the object detected in the detecting of an object in a case in which the accuracy is outside of a predetermined range.
Wherein having TSURUMI’s non-transitory computer-readable storage medium wherein having notifying of information on the object detected in the detecting of an object in a case in which the accuracy is outside of a predetermined range.
The motivation behind the modification would have been to obtain an information processing device with a mechanism capable of estimating a self-position more accurately, since both TSURUMI and HOLZ relate to localization processing, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while HOLZ has a system for simultaneous localization and calibration (SLAM) wherein a computing system may perform SLAM or similar processes to determine a location of a robot while also detecting positions of markers within the environment. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and HOLZ (US 20200242396 A1), Paragraph [105].
TSURUMI in view of HOLZ fail to explicitly teach wherein the notifying of information includes notifying by jointly displaying the image, feature points detected from the image, the accuracy, and information indicating the object.
However, WATANABE explicitly teaches wherein the notifying of information includes notifying by jointly displaying the image (Fig. 10, Paragraph [0086] – WATANABE discloses FIG. 10 is a diagram of an exemplary configuration of an operation screen to execute the image retrieval by using the image retrieving apparatus 104. Paragraph [0037] – WATANABE further discloses image retrieving apparatus 104 executes retrieving processing to retrieve an image which matches the retrieval condition from the image database 108 by using the retrieval condition specified by a user from the input apparatus 102 and present the information on the display apparatus 103.),
feature points detected from the image (Fig. 10, Paragraph [0090] – WATANABE discloses the feature points in FIG. 10 are, for example, 0: head, 1: neck, 2: right shoulder, 3: right elbow, 4: right wrist, 5: left shoulder, 6: left elbow, 7: left wrist, 8: right waist, 9: right knee, 10: right ankle, 11: left waist, 12: left knee, 13: left ankle. When the pose information is input, the feature points may be independently moved, and the plurality of feature points may be moved in conjunction with each other in consideration of connection relations. Furthermore, control points and the like other than the feature points may be added.),
the accuracy (Fig. 10, Paragraph [0089] – WATANABE discloses information displayed in the retrieval result display region 1004 is output to the display apparatus 103 by the retrieval result displaying unit 112 [wherein accuracy is degree of similarity shown in Fig. 10 within retrieval result display region 1004]. See also Paragraph [0079].),
and information indicating the object (Fig. 10, Paragraph [0089] – WATANABE discloses information displayed in the retrieval result display region 1004 is output to the display apparatus 103 by the retrieval result displaying unit 112. Paragraph [0049] – WATANABE further discloses the data obtained by the retrieving processing is related to the object.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI in view of HOLZ of having a non-transitory computer-readable storage medium configured to store a computer program comprising instructions for executing following processes: estimating a self-position on the basis of an image; determining accuracy in estimation of the self-position; detecting an object from the image; and notifying of information on the object detected in the detecting of an object in a case in which the accuracy is outside of a predetermined range, with the teachings of WATANABE having wherein the notifying of information includes notifying by jointly displaying the image, feature points detected from the image, the accuracy, and information indicating the object.
Wherein having TSURUMI’s non-transitory computer-readable storage medium wherein the notifying of information includes notifying by jointly displaying the image, feature points detected from the image, the accuracy, and information indicating the object.
The motivation behind the modification would have been to obtain an information processing device with an enhanced and more accurate method of object detection, since both TSURUMI and WATANABE relate to image processing and pose estimation, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while WATANABE has an image retrieving apparatus and an image retrieving method capable of improving a retrieval accuracy and a retrieval efficiency by generating a retrieval query reflecting pose information on a retrieval target. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and WATANABE (US 20190147292 A1), Paragraph [0105].
Claim 3 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over TSURUMI (US 20240412408 A1), hereinafter referenced as TSURUMI in view of HOLZ (US 20200242396 A1), hereinafter referenced as HOLZ, further in view of WATANABE (US 20190147292 A1), hereinafter referenced as WATANABE, and further in view of BHUTA (US 11295471 B1), hereinafter referenced as BHUTA.
Regarding claim 3, TSURUMI in view of HOLZ and WATANABE in further view of BHUTA teach the information processing device according to claim 1,
Although TSURUMI further teaches by the image analyzing unit (Fig. 6, #1316 called an object detection unit, Paragraph [0117-0118]),
TSURUMI fails to explicitly teach wherein the notification unit notifies of a countermeasure for increasing the accuracy on the basis of the object detected by the image analyzing unit in a case in which the accuracy is lower than the predetermined range.
However, BHUTA explicitly teaches wherein the notification unit notifies of a countermeasure (Fig. 1B, # 174 called a corrective actions data, Col. 4, Lines [53-54]- BHUTA discloses corrective actions data 174 can include data or instructions [wherein data or instructions are countermeasures] for how to correct a poorly-placed pallet. Fig. 5, Col. 13, Lines [52-59] – BHUTA discloses where the detected pallet position is unacceptable, the user interface can include a corrective action that shows and describes how to correct the placement of the pallet 109. While the user interface can include a graphical user interface generated for display, the user interface can also include indicator light instructions, directional light instructions, tone based audio queues, spoken audio instructions, and other user interface types.)
for increasing the accuracy on the basis of the object detected (Fig. 5, Col. 13, Lines [50-54] - BHUTA discloses the user interface can also indicate whether the detected pallet position is acceptable or unacceptable. Where the detected pallet position is unacceptable, the user interface can include a corrective action that shows and describes how to correct [wherein correction is increasing the accuracy of] the placement of the pallet 109 [wherein the pallet is the object])
by the image analyzing unit (Fig. 1B, Col. 12, Lines [61-63] – BHUTA discloses the pallet placement application 154 can perform image processing on an image captured by the camera 103.)
in a case in which the accuracy is lower than the predetermined range (Fig. 8, Col. 16, Lines [27-31] – BHUTA discloses the pallet placement application 154 can determine a detected pallet position based on the largest bound area defined by the orthogonal intersections. In some cases, the largest bound area is only accepted as a detected pallet position if it is greater than a predetermined threshold.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI in view of HOLZ and WATANABE of having the information processing device comprising at least one processor or circuit configured to function as: a notification unit configured to notify of information on the object detected by the image analyzing unit in a case in which the accuracy is outside of a predetermined range, with the teachings of BHUTA of having wherein the notification unit notifies of a countermeasure for increasing the accuracy on the basis of the object detected by the image analyzing unit in a case in which the accuracy is lower than the predetermined range.
Wherein having TSURUMI’s information processing device wherein the notification unit notifies of a countermeasure for increasing the accuracy on the basis of the object detected by the image analyzing unit in a case in which the accuracy is lower than the predetermined range.
The motivation behind the modification would have been to obtain an information processing device with a mechanism capable of estimating a self-position more accurately, since both TSURUMI and BHUTA relate to the navigation of automated guided vehicles (AGVs), wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while BHUTA describes aspects of camera-based pallet placement detection and notification that can prevent AGV failures and increase the efficiency of using AGVs as well as manual powered industrial truck (PIT) equipment. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and BHUTA (US 11295471 B1), Col. 2, Lines [1-9].
Regarding claim 6, TSURUMI in view of HOLZ and WATANABE teach the information processing device according to claim 1,
Although TSURUMI further teaches wherein the at least one processor or circuit is further configured to function as a storage unit (Paragraph [0097]- Tsurumi discloses the control unit 130 is realized by a processor executing various programs stored in a storage device inside the information processing apparatus 100 using a random access memory (RAM) or the like as a work area)
TSURUMI in view of HOLZ and WATANABE fail to explicitly teach wherein the at least one processor or circuit is further configured to function as a storage unit configured to store notification details correlated with the object.
However, BHUTA explicitly teaches wherein the at least one processor or circuit (Fig. 9, #900 called computing device, Col. 17, Lines [61-63] – BHUTA discloses the computing device 900 includes at least one processing system, for example, having a processor 902)
is further configured to function as a storage unit configured to store notification details correlated with the object (Fig. 1B, Col. 4, Lines [1-5] – BHUTA discloses various application components and/or other functionality may be executed in the computing environment 149 according to various embodiments. Also, data 164 may be stored in one or more memory or data store devices and be accessible to the computing environment 149.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI in view of HOLZ and WATANABE of having the information processing device comprising at least one processor or circuit configured to function as: a notification unit configured to notify of information on the object detected by the image analyzing unit in a case in which the accuracy is outside of a predetermined range, with the teachings of BHUTA of having wherein the at least one processor or circuit is further configured to function as a storage unit configured to store notification details correlated with the object.
Wherein having TSURUMI’s information processing device wherein the at least one processor or circuit is further configured to function as a storage unit configured to store notification details correlated with the object.
The motivation behind the modification would have been to obtain an information processing device with a mechanism capable of estimating a self-position more accurately, since both TSURUMI and BHUTA relate to the navigation of automated guided vehicles (AGVs), wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while BHUTA describes aspects of camera-based pallet placement detection and notification that can prevent AGV failures and increase the efficiency of using AGVs as well as manual powered industrial truck (PIT) equipment. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and BHUTA (US 11295471 B1), Col. 2, Lines [1-9].
Claims 8 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over TSURUMI (US 20240412408 A1), hereinafter referenced as TSURUMI in view of HOLZ (US 20200242396 A1), hereinafter referenced as HOLZ, further in view of WATANABE (US 20190147292 A1), hereinafter referenced as WATANABE, and further in view of LEE (US 20230343228 A1), hereinafter referenced as LEE.
Regarding claim 8, TSURUMI in view of HOLZ and WATANABE teach a device according to claim 1,
Although HOLZ further teaches wherein the notification unit determines that the accuracy is lower than the predetermined range (Fig. 1, #120 called a robotic device, Paragraph [0084], Table 1- Holz discloses Stale Localization - Localization system is unable to determine robotic device pose and/or localization certainty estimate [wherein certainty estimate is accuracy] has exceeded bounds [wherein bounds is predetermined range]. Recovery strategy states robotic device will halt and notify human operator. The operator can intervene by manually driving robotic device to a location for re-localization and reengaging)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI in view of HOLZ and WATANABE of having an information processing device comprising at least one processor or circuit configured to function as: a notification unit configured to notify of information on the object detected by the image analyzing unit in a case in which the accuracy is outside of a predetermined range, with the teachings of HOLZ of having wherein the notification unit determines that the accuracy is lower than the predetermined range.
Wherein having TSURUMI’s information processing device wherein the notification unit determines that the accuracy is lower than the predetermined range.
The motivation behind the modification would have been to obtain an information processing device with a mechanism capable of estimating a self-position more accurately, since both TSURUMI and HOLZ relate to localization processing, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while HOLZ has a system for simultaneous localization and calibration (SLAM) wherein a computing system may perform SLAM or similar processes to determine a location of a robot while also detecting positions of markers within the environment. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and HOLZ (US 20200242396 A1), Paragraph [105].
TSURUMI in view of HOLZ and WATANABE fail to explicitly teach in a case in which the number of feature points in estimation of the self-position is equal to or less than a predetermined number.
However, LEE explicitly teaches in a case in which the number of feature points in estimation of the self-position (Fig. 9, Paragraph [0213]- LEE discloses the estimated self-position reliability calculation unit 107 calculates the reliability of the self-position estimated by the self-position estimation unit 105 using correspondence relationship data between the feature point score and the estimated self-position reliability illustrated in FIG. 9)
is equal to or less than a predetermined number (Figs. 7, 9, 11, 13, Paragraph [0322]- LEE discloses the estimated self-position reliability calculation unit 107 compares the calculated reliability of the estimated self-position with the predefined threshold, for example, the “allowable reliability threshold” illustrated in FIG. 7, 9, 11, or 13, and displays a warning on the information display unit 121 of a controller held by a control center or a user (pilot) in a case where the calculated reliability of the estimated self-position is less than the “allowable reliability threshold”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI in view of HOLZ and WATANABE of having an information processing device comprising at least one processor or circuit configured to function as: a notification unit configured to notify of information on the object detected by the image analyzing unit in a case in which the accuracy is outside of a predetermined range, with the teachings of LEE having in a case in which the number of feature points in estimation of the self-position is equal to or less than a predetermined number.
Wherein having TSURUMI’s information processing device where the notification unit determines that the accuracy is lower than the predetermined range in a case in which the number of feature points in estimation of the self-position is equal to or less than a predetermined number.
The motivation behind the modification would have been to obtain an information processing device with a mechanism capable of estimating a self-position more accurately, since both TSURUMI and LEE relate to the navigation of mobile devices, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while LEE describes an information processing apparatus, an information processing system, an information processing method, and a program that enable safe flight or travel of a mobile device, for example, a drone or the like. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and LEE (US 20230343228 A1), Paragraph [0017-0026].
Regarding claim 9, TSURUMI in view of HOLZ and WATANABE teach a device according to claim 1,
HOLZ further teaches wherein the notification unit determines that the accuracy is lower than the predetermined range (Fig. 1, #120 called a robotic device, Paragraph [0084], Table 1- Holz discloses Stale Localization - Localization system is unable to determine robotic device pose and/or localization certainty estimate [wherein certainty estimate is accuracy] has exceeded bounds [wherein bounds is predetermined range]. Recovery strategy states robotic device will halt and notify human operator. The operator can intervene by manually driving robotic device to a location for re-localization and reengaging)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI in view of HOLZ and WATANABE of having an information processing device comprising at least one processor or circuit configured to function as: a notification unit configured to notify of information on the object detected by the image analyzing unit in a case in which the accuracy is outside of a predetermined range, with the teachings of HOLZ of having wherein the notification unit determines that the accuracy is lower than the predetermined range.
Wherein having TSURUMI’s information processing device wherein the notification unit determines that the accuracy is lower than the predetermined range.
The motivation behind the modification would have been to obtain an information processing device with a mechanism capable of estimating a self-position more accurately, since both TSURUMI and HOLZ relate to localization processing, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while HOLZ has a system for simultaneous localization and calibration (SLAM) wherein a computing system may perform SLAM or similar processes to determine a location of a robot while also detecting positions of markers within the environment. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and HOLZ (US 20200242396 A1), Paragraph [105].
TSURUMI in view of HOLZ and WATANABE fail to explicitly teach in a case in which a likelihood of feature points in estimation of the self-position is equal to or less than a predetermined threshold value.
However, LEE explicitly teaches in a case in which the likelihood of feature points in estimation of the self-position (Fig. 9, Paragraph [0213]- LEE discloses the estimated self-position reliability calculation unit 107 calculates the reliability of the self-position estimated by the self-position estimation unit 105 using correspondence relationship data between the feature point score and the estimated self-position reliability illustrated in FIG. 9)
is equal to or less than a predetermined threshold value (Figs. 7, 9, 11, 13, Paragraph [0322]- LEE discloses the estimated self-position reliability calculation unit 107 compares the calculated reliability of the estimated self-position with the predefined threshold, for example, the “allowable reliability threshold” illustrated in FIG. 7, 9, 11, or 13, and displays a warning on the information display unit 121 of a controller held by a control center or a user (pilot) in a case where the calculated reliability of the estimated self-position is less than the “allowable reliability threshold”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of TSURUMI in view of HOLZ and WATANABE of having an information processing device comprising at least one processor or circuit configured to function as: a notification unit configured to notify of information on the object detected by the image analyzing unit in a case in which the accuracy is outside of a predetermined range, with the teachings of LEE having in a case in which a likelihood of feature points in estimation of the self-position is equal to or less than a predetermined threshold value.
Wherein having TSURUMI’s information processing device where the notification unit determines that the accuracy is lower than the predetermined range in a case in which a likelihood of feature points in estimation of the self-position is equal to or less than a predetermined threshold value.
The motivation behind the modification would have been to obtain an information processing device with a mechanism capable of estimating a self-position more accurately, since both TSURUMI and LEE relate to the navigation of mobile devices, wherein TSURUMI has an information processing system that estimates at least either the position or the orientation of the imaging apparatus using at least either the three-dimensional mask area or the feature point map and the feature point extracted from the captured image of the imaging apparatus, while LEE describes an information processing apparatus, an information processing system, an information processing method, and a program that enable safe flight or travel of a mobile device, for example, a drone or the like. Please see TSURUMI (US 20240412408 A1), Paragraph [0076], and LEE (US 20230343228 A1), Paragraph [0017-0026].
Conclusion
Listed below are the prior arts made of record and not relied upon but are considered pertinent to applicant`s disclosure.
Eckman et al. (US 10984378 B1) - In one implementation, a system for automatically profiling pallets includes a frame defining an opening that is sized and shaped for a pallet to pass through, and cameras mounted to the frame, the cameras being configured to capture images of a pallet as it passes through the frame. The system further includes a profiling computer system that is configured to receive the images captured by the cameras and to automatically profile the pallet based, at least in part, on analysis of the images. Automatically profiling the pallet includes generating a point cloud representing the pallet based on the images…….. Fig. 3, Abstract.
Jones et al. (US 20190213438 A1) - A mobile cleaning robot includes a cleaning head configured to clean a floor surface in an environment, and at least one camera having a field of view that extends above the floor surface. The at least one camera is configured to capture images that include portions of the environment above the floor surface. The robot includes a recognition module is configured to recognize objects in the environment based on the images captured by the at least one camera, in which the recognition module is trained at least in part using the images captured by the at least one camera….. Fig. 3, 4, Abstract.
Taylor et al. (US 20190196480 A1) - An example method includes determining a path to be followed by a vehicle through an environment. The path includes an ordered sequence of positions. The method also includes determining an intersection between a first object in the environment and a first area planned to be occupied by the vehicle while moving along the path and, in response, sequentially testing the ordered sequence of positions to identify a first ordinal position in the ordered sequence of positions, where the first ordinal position corresponds to a second area planned to be occupied by the vehicle while moving along the path, and where the second area is within a threshold distance of the first object.….. Fig. 1, Abstract.
Mullins et al. (US 20180261012 A1) - A system and method for offloading object detection are described. A server receives first sensor data from a first sensor of an augmented reality (AR) display device. The first sensor data indicates a pose of the AR display device relative to a first reference coordinate system. The server detects a physical object using second sensor data received from a second sensor of the AR display device. The server determines, based on the second sensor data, a pose of the physical object relative to the AR display device. The server then determines the pose of the physical object relative to the first reference coordinate system based on the pose of the physical object relative to the AR display device and the pose of the AR display device relative to the first reference coordinate system..….. Fig. 1, Abstract.
Liao et al. (US 20170277197 A1) - A system and method are provided for autonomously navigating a vehicle. The method captures a sequence of image pairs using a stereo camera. A navigation application stores a vehicle pose (history of vehicle position). The application detects a plurality of matching feature points in a first matching image pair, and determines a plurality of corresponding object points in three-dimensional (3D) space from the first image pair. A plurality of feature points are tracked from the first image pair to a second image pair, and the plurality of corresponding object points in 3D space are determined from the second image pair. From this, a vehicle pose transformation is calculated using the object points from the first and second image pairs..….. Fig. 1, 2, Abstract.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BEZAWIT N SHIMELES whose telephone number is (571)272-7663. The examiner can normally be reached M-F 7:30am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BEZAWIT NOLAWI SHIMELES/Examiner, Art Unit 2673
/CHINEYERE WILLS-BURNS/Supervisory Patent Examiner, Art Unit 2673
Prosecution Insights