DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Remarks
The claims being considered in this application are preliminary amended claims submitted on 09/19/2025. Claims 1-6 are pending.
Priority
The applicant’s claim to priority of JP2023-058133 on 03/31/2023 is acknowledged.
Information Disclosure Statement
The information disclosure statement(s) filed on 09/19/2025 has been annotated and considered.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
In Claim 1:
“an object information holding section holding object information…”
“an estimated image information generating section generating estimated image information…:
“a comparison section comparing the estimated image information…”
“a control switching section that performs an operation…”
In Claims 3-5:
“the image information update section causes…”
“the image information update section selects…”
“the image information update section synthesizes…”
In Claim 6:
“the viewpoint change instruction section causes…”
Structure/support for the 112f language of the claims was found in applicant’s specification in at least ¶0032: “The control device 21 is provided as a computer that includes: an arithmetic section (a processor such as a CPU) 22; a storage section (a ROM or the like) 23; and an input/output section” and ¶0036 via “The arithmetic section 22 includes, as a function section realized by the program: an object information holding section 22a; an estimated image information generating section 22b; a comparison section 22c; an image information update section 22d; a viewpoint change instruction section 22e; and a control switching section 22f.”
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 and 3 are rejected under 35 U.S.C. 103 as being unpatentable over Park et. al. (US 20080008353 A1) in view of Ishihara et. al. (US 20190143525 A1).
Regarding Claim 1, Park discloses:
A robot control system to control, based on image information of an object, (See at least ¶0011 via "a mobile robot including the system that can detect the moving object even while the mobile robot is moving by predicting an image after the movement from an existing image and comparing the predicted image with the present image" and ¶0038 via "The system 100 for detecting a moving object includes an image obtaining (capturing) unit 110 that obtains (captures) the image information of the obstacle 30")
an object information holding section holding object information including distance information of the object, the distance information being calculated based on the image information obtained by imaging by the imaging device located at a position having a predetermined distance (See at least ¶0033 via "At this time, the camera sensor 20 is disposed in relationship to the light source 10 to maintain a constant distance d from the light source 10 and to obtain the image information." and also ¶0039 via "The image predicting unit 120 may include a data converting unit 122 that converts the distance calculated from the image information into the distance data predicted from the moving information of the system 100 and a data extracting unit 124 that extracts the data deviated from the visual angle a of the light source 10 from the predicted distance data after the system 100 is moved." and ¶0040 via "obtain a first image by using the light source 10 in the image obtaining unit 110 and calculate the distance data between the light source 10 and the obstacle 30 by the triangular method". Also see ¶0068 via "exemplary embodiments can also be implemented by executing computer readable code/instructions in/on a medium/media…executed by a computing device")
an estimated image information generating section generating estimated image information of the object based on the object information held in the object information holding section, the estimated image information being estimated to be obtained by imaging by the imaging device at a predetermined position (See at least ¶0039 via "The image predicting unit 120" and ¶0040 via "… obtain a first image by using the light source 10 in the image obtaining unit 110 and calculate the distance data between the light source 10 and the obstacle 30 by the triangular method as described above. On the basis of the distance data of the first image and the moving information (moving vector and rotational angle) of the system 100 after obtaining the first image, the distance data of a second image that is an image at the moved position may be predicted by calculation through the data converting unit 122 and the data extracting unit 124 in the image predicting unit 120. The second image may be obtained (captured) by using the light source 10 in the image obtaining unit 110. Accordingly, it may detect the moving object by comparing the distance data calculated from the second image and the distance data of the predicted second image in the comparing/analyzing unit 130." as well as ¶0053 via "When comparing the distance data of the predicted second image with the distance data of the second image, if the moving object does not exist, then both the distance data of the predicted second image and the distance data of the second image will have an approximately equal value.". Also see ¶0068 via "exemplary embodiments can also be implemented by executing computer readable code/instructions in/on a medium/media…executed by a computing device")
a comparison section comparing the estimated image information generated by the estimated image information generating section to actual image information of the object, the actual image information being obtained by imaging by the imaging device at the predetermined position closer to the object; and (See at least ¶0040 via "The second image may be obtained (captured) by using the light source 10 in the image obtaining unit 110. Accordingly, it may detect the moving object by comparing the distance data calculated from the second image and the distance data of the predicted second image in the comparing/analyzing unit 130." and ¶0053 via "When comparing the distance data of the predicted second image with the distance data of the second image, if the moving object does not exist, then both the distance data of the predicted second image and the distance data of the second image will have an approximately equal value.". Also see ¶0068 via "exemplary embodiments can also be implemented by executing computer readable code/instructions in/on a medium/media…executed by a computing device")
a control switching section that preforms an operation (See at least ¶0065 via "If determining (S250) in the comparing/analyzing unit 130 that the predicted image and the obtained image are equal to each other, the first image is obtained again, and the above steps are repeated. Preferably, the obtained second image is similar to the following first image. If determining (S250) that the predicted image and the obtained image are different from each other, the system may sound the alarm to inform the user, or take follow-up measures such as an actuation of the security system (S260)" **Which illustrates different control for different results of the comparison. Also see ¶0068 via "exemplary embodiments can also be implemented by executing computer readable code/instructions in/on a medium/media…executed by a computing device")
Furthermore, Park discloses continuing or repeating the imaging operation when the comparison of the predicted and actual second image match each other, instead of treating the object. Park also discloses the conditional branch of when the comparison does not yield an equal match between the estimated and actual second image, but discloses sounding an alarm or informing a user based on this determination, rather than explicitly not treating an object [See Park ¶0064].
However, Ishihara--who is directed towards an articulated robot--discloses: a robot treating the object…an imaging device provided on a robot arm of the robot (See at least ¶0019 via "Processing objects 400 to be processed by the actuation device 100 are placed on a field 11 (an area or a stage). In the present embodiment, processing by the actuation device 100 is grasping the processing objects 400 with the hand 121…" and ¶0020 via "The cameras 211, 221, and 222 are examples of an image taking unit" and Figure 1 which illustrates camera 221 attached to the robot arm)
a predetermined position closer to the object…when the robot arm moves together with the imaging device from the position having the predetermined distance from the object to the predetermined position closer to the object; (See at least ¶0029 via "The second actuation control unit 313 controls the actuator 113 to move the hand 121 from an initial position P0 to an intermediate potion P10 based on the taken image of the bird's eye view camera 211. The intermediate position P10 is an example of a proximal position closer to a target object 401 than the initial position P0.")
preforms an operation to treat the object using the distance information (See at least ¶0019 via "In the present embodiment, processing by the actuation device 100 is grasping the processing objects 400 with the hand 121…" as well as Figure 3 and ¶0032 via "comparison between a deviation of the position and attitude and a threshold of the deviation based on image information is performed (S13). When the deviation is equal to or smaller than the threshold at S13 (YES at 313), control by the processing control unit 314 is executed 14")
does not perform the operation to treat the object (See at least Figure 3 and ¶0032 via "comparison between a deviation of the position and attitude and a threshold of the deviation based on image information is performed (S13). When the deviation is equal to or smaller than the threshold at S13 (YES at 313), control by the processing control unit 314 is executed 14). When the deviation is larger than the threshold at S13 (NO at S13), the control by the first actuation control unit 312 is executed again (S12)" as well as ¶0030 via "The processing control unit 314 controls the actuator 113 to cause the actuation device 100 (the hand 121) to perform predetermined processing in a state where the hand 121 has reached the target position and has the target attitude" and ¶0020 via "The actuation system 1 controls the actuator 113 to cause the hand 121 to perform predetermined processing at a time point when the hand 121 has reached the target position and has the target attitude" and ¶0028 via "The first actuation control unit 312 controls the actuator 113 locate the hand 121 at the target position and the target attitude based on the taken images of the first camera 221 and the second camera 222. The first actuation control unit 312 executes feedback control according to visual servo" **Wherein whether a deviation is within a threshold is determined based on the image information and when the deviation is within a threshold, the predetermined processing is performed, but when the deviation is not within a threshold, the predetermined processing is not performed and instead, the control by the first actuation control unit 312 is executed again, which under BRI is interpreted as not treating the object (proceeding with the predetermined processing)**).
Therefore it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the given invention to modify Park in view of applying Ishihara's robotic arm object processing based on the deviation conditions of visual servoing to the image prediction and comparison system of Park, to enable Park's system to be implemented to a robotic arm that approaches and processes/grasps a target object when image conditions are met. This modification would yield a predictable system of image prediction and comparison in a robotic arm application while determining/verifying if image(s) satisfy specific conditions for further processing to be executed. Additionally, by including Ishihara's intermediate position P10 (proximal position) closer to the target object, the actuation system is able to quickly and accurately move to the desired position/attitude: "…control according to visual servo by the first actuation control unit 312 can be started more reliably or more smoothly and thus the actuation system 1 capable of more quickly reaching desired position and attitude can be obtained" [Ishihara ¶0068] while "having higher robustness" [Ishihara ¶0061] and "having a relatively-high control accuracy" [Ishihara ¶0062].
However Modified Park does not explicitly disclose the predetermined distance from the object nevertheless, Park discloses "the camera sensor 20 is disposed in relationship to the light source 10 to maintain a constant distance d from the light source 10 and to obtain the image information." [Park ¶0033]. Additionally, Ishihara discloses initial position p0 and also a fixed camera: "…bird's eye view camera 211 is also referred to as “fixed camera”. The distance of the bird's eye view camera 211 from the processing objects 400 is longer than those of other cameras 221 and 222" [Ishihara ¶0021]. Therefore, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the given invention to implement imaging/obtaining image information at a predetermined distance from the object in combination with Modified Park's existing distance measurement, in order to have a known starting point/initial position to base measurements from, by defining more variables within the spatial relationship, which would yield predictable results in line with "having a relatively-high control accuracy" [Ishihara ¶0062].
Regarding Claim 3, Modified Park discloses the robot control system according to Claim 1.
Furthermore, Park discloses: …when a plurality of pieces of image information is obtained by imaging by the imaging device at respective positions (See at least ¶0040 via "On the basis of the distance data of the first image and the moving information (moving vector and rotational angle) of the system 100 after obtaining the first image, the distance data of a second image that is an image at the moved position may be predicted by calculation through the data converting unit 122 and the data extracting unit 124 in the image predicting unit 120. The second image may be obtained (captured) by using the light source 10 in the image obtaining unit 110. Accordingly, it may detect the moving object by comparing the distance data calculated from the second image and the distance data of the predicted second image in the comparing/analyzing unit 130.")
the object information holding section to hold the object information including the distance information of the object, (See at least ¶0039 via "The image predicting unit 120 may include a data converting unit 122 that converts the distance calculated from the image information into the distance data predicted from the moving information of the system 100 and a data extracting unit 124 that extracts the data deviated from the visual angle a of the light source 10 from the predicted distance data after the system 100 is moved." and also see ¶0068 via "exemplary embodiments can also be implemented by executing computer readable code/instructions in/on a medium/media…executed by a computing device").
However, Park does not explicitly disclose an image information update section.
Nevertheless, Ishihara discloses: further comprising an image information update section, wherein when a plurality of pieces of image information is obtained by imaging by the imaging device at respective positions that have different distances to the object and from where the distance information is acquired, (See at least ¶0023 via "The control unit 310 is, for example, a central processing unit (CPU) or a controller" as well as Figure 1 which shows the imaging device including cameras 221 and 222 that have different distances to the target object(s). Further, see ¶0060 via "Alternatively, in a case where a condition such as the condition in which the deviation becomes equal to or smaller than the threshold, the condition in which the distance between the second camera 222 and the target object 401 becomes equal to or smaller than the threshold, the condition in which the distance between the second camera 222 and the surface of the field 11 becomes equal to or smaller than the threshold, or the condition in which the size of the target object 401 in the image taken by the second camera 222 becomes equal to or larger than the threshold is met, it is possible to calculate the command value u using only the image taken by the second camera 222 closer to the target object 401…set the weighing coefficient k to 0 (k=0) in the calculation of the command value a according to the expression (8) in the command-value calculation unit 312e" and also see ¶0057 via " An upper portion of FIG. 7 represents a state where the hand 121 is located at the intermediate position P10 described above, a middle portion of FIG. 7 represents a state where the hand 121 is located at a position P11 closer to the target position than the intermediate position P10, and a lower portion of FIG. 7 represents a state where the hand 121 is located at a position P12 closer to the target position than the position P11. The position P12 is a position in a state where the hand 121 has almost reached the target position." **Wherein under BRI, the control system determines which image information is to be used for control operations, and for example, when a condition is met, the image closer to the target object is used, and further, the weight coefficient is changed which corresponds to the updating.)
the image information update section causes (See at least ¶0023 via "The control unit 310 is, for example, a central processing unit (CPU) or a controller" and also ¶0060 via "…it is possible to calculate the command value u using only the image taken by the second camera 222 closer to the target object 401…set the weighing coefficient k to 0 (k=0) in the calculation of the command value a according to the expression (8) in the command-value calculation unit 312e" **Wherein for example, when a condition is met, the image closer to the target object is used, and further, the weight coefficient is changed which corresponds to the updating).
Therefore, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the given invention to modify Modified Park in view of Ishihara's changing of the weighting coefficient based on the number of matched feature points between images taken and the target image: "The weighting coefficient k can be determined according to the number of feature points matched in each of the matching processing units 312a1 and 312a2." [Ishihara ¶0054], thus, applying the weighting from Ishihara to modified Park yields predictable results by selecting images/assigning a higher weight coefficient to the image(s) that have a higher correspondence with the target image, which improves the reliability and accuracy of the control that is based on the image(s): : "The weighting coefficient of the present embodiment is set to be larger as the number of feature points matched between taken images from which the original command values have been calculated and the target image is larger. Therefore, according to the present embodiment, for example, the actuation system 1 having a relatively-high control accuracy and being capable of more quickly reaching desired position and attitude can be obtained." [Ishihara ¶0064]. Furthermore, Ishihara discloses a distance condition and selecting the image that is closer to target object when the condition is met [Ishihara ¶0060].
Claims 2 are rejected under 35 U.S.C. 103 as being unpatentable over Park et. al. (US 20080008353 A1) and Ishihara et. al. (US 20190143525 A1) in view of Kume et. al. (US 20210229290 A1).
Regarding Claim 2, Modified Park discloses the robot control system according to Claim 1.
Furthermore, Park discloses: wherein when the control switching section (See at least ¶0065 via "If determining (S250) in the comparing/analyzing unit 130 that the predicted image and the obtained image are equal to each other, the first image is obtained again, and the above steps are repeated. Preferably, the obtained second image is similar to the following first image. If determining (S250) that the predicted image and the obtained image are different from each other, the system may sound the alarm to inform the user, or take follow-up measures such as an actuation of the security system (S260)". And also ¶0068 via "exemplary embodiments can also be implemented by executing computer readable code/instructions in/on a medium/media…executed by a computing device" **Wherein Park also discloses the conditional branch of when the comparison does not yield an equal match between the estimated and actual second image, but discloses sounding an alarm or informing a user based on this determination, rather than explicitly not treating an object [See Park ¶0064])
Furthermore, Park discloses the object information holding section and to hold the object information including the distance information in the object information holding section (See at least ¶0039 via "The image predicting unit 120 may include a data converting unit 122 that converts the distance calculated from the image information into the distance data predicted from the moving information of the system 100 and a data extracting unit 124 that extracts the data deviated from the visual angle a of the light source 10 from the predicted distance data after the system 100 is moved." and also see ¶0068 via "exemplary embodiments can also be implemented by executing computer readable code/instructions in/on a medium/media…executed by a computing device").
Furthermore, Ishihara discloses: does not perform the operation to treat the object, (See at least Figure 3 and ¶0032 via "comparison between a deviation of the position and attitude and a threshold of the deviation based on image information is performed (S13). When the deviation is equal to or smaller than the threshold at S13 (YES at 313), control by the processing control unit 314 is executed 14). When the deviation is larger than the threshold at S13 (NO at S13), the control by the first actuation control unit 312 is executed again (S12)" as well as ¶0030 via "The processing control unit 314 controls the actuator 113 to cause the actuation device 100 (the hand 121) to perform predetermined processing in a state where the hand 121 has reached the target position and has the target attitude" and ¶0020 via "The actuation system 1 controls the actuator 113 to cause the hand 121 to perform predetermined processing at a time point when the hand 121 has reached the target position and has the target attitude" and ¶0028 via "The first actuation control unit 312 controls the actuator 113 locate the hand 121 at the target position and the target attitude based on the taken images of the first camera 221 and the second camera 222. The first actuation control unit 312 executes feedback control according to visual servo" **Wherein when the deviation is not within a threshold, the predetermined processing is not performed and instead, the control by the first actuation control unit 312 is executed again, which under BRI is interpreted as not treating the object (proceeding with the predetermined processing)**).
Therefore it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the given invention to modify Park in view of applying Ishihara's robotic arm object processing based on the deviation conditions of visual servoing to the image prediction and comparison system of Park, to enable Park's system to be implemented to a robotic arm that approaches and processes/grasps a target object when image conditions are met. This modification would yield a predictable system of image prediction and comparison in a robotic arm application while determining/verifying if image(s) satisfy specific conditions for further processing to be executed. Additionally, by including Ishihara's intermediate position P10 (proximal position) closer to the target object, the actuation system is able to quickly and accurately move to the desired position/attitude: "…control according to visual servo by the first actuation control unit 312 can be started more reliably or more smoothly and thus the actuation system 1 capable of more quickly reaching desired position and attitude can be obtained" [Ishihara ¶0068] while "having higher robustness" [Ishihara ¶0061] and "having a relatively-high control accuracy" [Ishihara ¶0062].
However Modified Park does not explicitly disclose the predetermined distance from the object nevertheless, Park discloses "the camera sensor 20 is disposed in relationship to the light source 10 to maintain a constant distance d from the light source 10 and to obtain the image information." [Park ¶0033]. Additionally, Ishihara discloses initial position p0 and also a fixed camera: "…bird's eye view camera 211 is also referred to as “fixed camera”. The distance of the bird's eye view camera 211 from the processing objects 400 is longer than those of other cameras 221 and 222" [Ishihara ¶0021]. Therefore, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the given invention to implement imaging/obtaining image information at a predetermined distance from the object in combination with Modified Park's existing distance measurement, in order to have a known starting point/initial position to base measurements from, by defining more variables within the spatial relationship, which would yield predictable results in line with "having a relatively-high control accuracy" [Ishihara ¶0062].
However, Modified Park does not explicitly disclose deleting the object information when the object is not treated.
Nevertheless, Kume--who is directed towards camera position/attitude calibration for a robot--discloses: the object information held in the object information (See at least ¶0060 via "the acquired operation data D1 stored in the acquired operation database DB1 is to be formed by deleting all images and poses, initializing, and acquiring images again at various poses of the robot 200.")
imaging by the imaging device is performed at the position having the predetermined distance from the object so as to perform once again an operation (See at least ¶0060 via "…initializing, and acquiring images again at various poses of the robot 200.")
Therefore, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the given invention to modify Modified Park in view of the deletion and reacquiring of information such as in Kume in order to account for changes in the frame of reference or when the data is not/no longer matching or corresponding to the previously calculated image information. Applying Kume's concepts of deleting and reacquiring image information to Modified Park would have yielded predictable results of preventing relying on inaccurate image information when the actual and estimated images do not match: "…when there is no change in the relative pose of the camera 201 and the robot 200, the pose data when an image is acquired at the same place as that in the past should show the same value as the pose data acquired in the past. However, this relationship is lost when there is a change in the relative pose of the camera 201 and the robot 200. That is, even if the image is acquired from the same place as that in the past, the pose will be different." [Kume ¶0059], and to further avoid large errors that can lead to task failure: "However, the relative pose between the camera 201 and the robot 200 may be changed by vibrations during operation of the robot 200, or collision of the camera 201 and the robot 200 with an object existing in a surrounding environment during operation, which may cause an error with respect to an initially set value. Since the error in the relative pose between the camera 201 and the robot 200 is to be an error in the pose of the robot 200, a large error will cause a task failure." [Kume ¶0034].
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Park et. al. (US 20080008353 A1) and Ishihara et. al. (US 20190143525 A1) in view of Shalom et. al. (US 20080212840 A1).
Regarding Claim 4, Modified Park discloses the robot control system according to Claim 1.
Furthermore, Park discloses: …when a plurality of pieces of image information is obtained by imaging by the imaging device at respective positions (See at least ¶0040 via "On the basis of the distance data of the first image and the moving information (moving vector and rotational angle) of the system 100 after obtaining the first image, the distance data of a second image that is an image at the moved position may be predicted by calculation through the data converting unit 122 and the data extracting unit 124 in the image predicting unit 120. The second image may be obtained (captured) by using the light source 10 in the image obtaining unit 110. Accordingly, it may detect the moving object by comparing the distance data calculated from the second image and the distance data of the predicted second image in the comparing/analyzing unit 130.")
the object information holding section to hold the object information including the distance information of the object, (See at least ¶0039 via "The image predicting unit 120 may include a data converting unit 122 that converts the distance calculated from the image information into the distance data predicted from the moving information of the system 100 and a data extracting unit 124 that extracts the data deviated from the visual angle a of the light source 10 from the predicted distance data after the system 100 is moved." and also see ¶0068 via "exemplary embodiments can also be implemented by executing computer readable code/instructions in/on a medium/media…executed by a computing device").
However, Park does not explicitly disclose an image information update section.
Nevertheless, Ishihara discloses: further comprising an image information update section, wherein when a plurality of pieces of image information is obtained by imaging by the imaging device at respective positions that have different distances to the object and from where the distance information is acquired, (See at least ¶0023 via "The control unit 310 is, for example, a central processing unit (CPU) or a controller" as well as Figure 1 which shows the imaging device including cameras 221 and 222 that have different distances to the target object(s). Also see ¶0057 via " An upper portion of FIG. 7 represents a state where the hand 121 is located at the intermediate position P10 described above, a middle portion of FIG. 7 represents a state where the hand 121 is located at a position P11 closer to the target position than the intermediate position P10, and a lower portion of FIG. 7 represents a state where the hand 121 is located at a position P12 closer to the target position than the position P11. The position P12 is a position in a state where the hand 121 has almost reached the target position." **Wherein under BRI, the control system determines which image information is to be used for control operations, and the weight coefficient is changed which corresponds to the updating.)
the image information update section selects the (See at least ¶0023 via "The control unit 310 is, for example, a central processing unit (CPU) or a controller" and also see ¶0054 via " The weighting coefficient k can be determined according to the number of feature points matched in each of the matching processing units 312a1 and 312a2. For example, when the number of feature points matched in the matching processing unit 312a1 is m.sub.1 and the number of feature points matched in the matching processing unit 312a2 is m.sub.2, the weighting coefficient k can be determined by a following expression (9)." **Wherein the more feature points being matched to the target image corresponds to a more accurate image. Also see ¶0060 via "…set the weighing coefficient k to 0 (k=0) in the calculation of the command value a according to the expression (8) in the command-value calculation unit 312e").
Therefore, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the given invention to modify Modified Park in view of Ishihara's changing of the weighting coefficient based on the number of matched feature points between images taken and the target image: "The weighting coefficient k can be determined according to the number of feature points matched in each of the matching processing units 312a1 and 312a2." [Ishihara ¶0054], thus, applying the weighting from Ishihara to modified Park yields predictable results by selecting images/assigning a higher weight coefficient to the image(s) that have a higher correspondence with the target image, which improves the reliability and accuracy of the control that is based on the image(s): : "The weighting coefficient of the present embodiment is set to be larger as the number of feature points matched between taken images from which the original command values have been calculated and the target image is larger. Therefore, according to the present embodiment, for example, the actuation system 1 having a relatively-high control accuracy and being capable of more quickly reaching desired position and attitude can be obtained." [Ishihara ¶0064].
However, Ishihara does not explicitly disclose the "clearest" image being selected.
Nevertheless, Shalom--who is directed towards an imaging system and method--discloses: the clearest image information (See at least ¶0126 via "For each ROI, the optimum position(s) and orientation(s) of the image acquisition system 150, and of the illumination system 120, may be determined to enable the clearest image data of the ROI to be obtained").
Therefore, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the given invention to modify Modified Park in view of the enablement of obtaining the clearest image such as in Shalom, in order to minimize shadows or high contrast between different parts of the region of interest being imaged: "the illumination provided by the illumination system 120 to an ROI may be optimized such that for each image acquisition position/orientation of the image acquisition system 150 with respect to an ROI of the object O, the illumination system 120 provides maximum uniformity and intensity of illumination for every part of the ROI being imaged, minimizing shadows and/or high contrast between different parts of the ROI" [Shalom ¶0128], which improves the quality of the image data being gathered: "…providing image data from an object O being imaged, such as to enable the surface topography or other dimensional data, including for example location, size and shape of recesses or protrusions, for example bores and the like, to be determined from the image data" [Shalom ¶0109] and thus provides more accurate and higher quality data to base the control on.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Park et. al. (US 20080008353 A1) and Ishihara et. al. (US 20190143525 A1) in view of Yu et. al. (US 20210019891 A1).
Regarding Claim 5, Modified Park discloses the robot control system according to Claim 1.
Furthermore, Park discloses: …when a plurality of pieces of image information is obtained by imaging by the imaging device at respective positions (See at least ¶0040 via "On the basis of the distance data of the first image and the moving information (moving vector and rotational angle) of the system 100 after obtaining the first image, the distance data of a second image that is an image at the moved position may be predicted by calculation through the data converting unit 122 and the data extracting unit 124 in the image predicting unit 120. The second image may be obtained (captured) by using the light source 10 in the image obtaining unit 110. Accordingly, it may detect the moving object by comparing the distance data calculated from the second image and the distance data of the predicted second image in the comparing/analyzing unit 130.")
the object information holding section to hold the object information including the distance information of the object, (See at least ¶0039 via "The image predicting unit 120 may include a data converting unit 122 that converts the distance calculated from the image information into the distance data predicted from the moving information of the system 100 and a data extracting unit 124 that extracts the data deviated from the visual angle a of the light source 10 from the predicted distance data after the system 100 is moved." and also see ¶0068 via "exemplary embodiments can also be implemented by executing computer readable code/instructions in/on a medium/media…executed by a computing device").
However, Park does not explicitly disclose an image information update section.
Nevertheless, Ishihara discloses: further comprising an image information update section, wherein when a plurality of pieces of image information is obtained by imaging by the imaging device at respective positions that have different distances to the object and from where the distance information is acquired, (See at least ¶0023 via "The control unit 310 is, for example, a central processing unit (CPU) or a controller" as well as Figure 1 which shows the imaging device including cameras 221 and 222 that have different distances to the target object(s). Also see ¶0057 via " An upper portion of FIG. 7 represents a state where the hand 121 is located at the intermediate position P10 described above, a middle portion of FIG. 7 represents a state where the hand 121 is located at a position P11 closer to the target position than the intermediate position P10, and a lower portion of FIG. 7 represents a state where the hand 121 is located at a position P12 closer to the target position than the position P11. The position P12 is a position in a state where the hand 121 has almost reached the target position." **Wherein under BRI, the control system determines which image information is to be used for control operations, and the weight coefficient is changed which corresponds to the updating.)
the image information update section (See at least ¶0023 via "The control unit 310 is, for example, a central processing unit (CPU) or a controller" and also see ¶0054 via "The weighting coefficient k can be determined according to the number of feature points matched in each of the matching processing units 312a1 and 312a2. For example, when the number of feature points matched in the matching processing unit 312a1 is m.sub.1 and the number of feature points matched in the matching processing unit 312a2 is m.sub.2, the weighting coefficient k can be determined by a following expression (9)." **Wherein the more feature points being matched to the target image corresponds to a more accurate image. Also see ¶0060 via "…set the weighing coefficient k to 0 (k=0) in the calculation of the command value a according to the expression (8) in the command-value calculation unit 312e" and Figure 1 which shows the imaging device including cameras 221 and 222 that have different distances to the target object(s), and thus provide different pieces of image information).
Therefore, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the given invention to modify Modified Park in view of Ishihara's changing of the weighting coefficient based on the number of matched feature points between images taken and the target image: "The weighting coefficient k can be determined according to the number of feature points matched in each of the matching processing units 312a1 and 312a2." [Ishihara ¶0054], thus, applying the weighting from Ishihara to modified Park yields predictable results by updating the data used for control by selecting images/assigning a higher weight coefficient to the image(s) that have a higher correspondence with the target image, which improves the reliability and accuracy of the control that is based on the image(s): : "The weighting coefficient of the present embodiment is set to be larger as the number of feature points matched between taken images from which the original command values have been calculated and the target image is larger. Therefore, according to the present embodiment, for example, the actuation system 1 having a relatively-high control accuracy and being capable of more quickly reaching desired position and attitude can be obtained." [Ishihara ¶0064].
However, Modified Park does not explicitly disclose synthesizing the plurality of pieces of image information.
Nevertheless, Yu--who is directed towards object detection based on image data--discloses: synthesizes the plurality of pieces of image information (See at least ¶0089 via " By fusing image data (representing 2D image data), as illustrated by first image data representation 610, with second image data representation 615 (representing 3D depth camera data), as illustrated by first image data representation 610, the fused image data, as illustrated by image data representation 605, may be generated.")
the object information being calculated based on the synthesized image information (See at least Claim 15 via "fusing the first edge information with the second edge information to generate fused edge information; generating an object detection hypothesis based on the fused edge information;" and ¶0095 via "The edge information may also include height or depth information (e.g., information representing a distance between the image capture device and a point on a surface) and locations of regions with similar height. As an example, the depth information can include a value that represents distance from the image capture device (i.e. 3D camera)." and ¶0117 via "the robotic system 100 can generate the object detection hypothesis based segmenting the 3D point cloud information based on the fused edges of fused edge information").
Therefore, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the given invention to modify Modified Park in view of the fusion of image data disclosed by Yu in order to utilize more image data/information to increase the probability of accurately detecting object positions and contours, which would yield more accurate picking operations in the system of Modified Park: "related art systems may have difficulty detecting contrast or boundaries between two objects or boxes, between an object and the surrounding environment, or between multiple objects packaged on a single pallet due to color or optical resolution of a camera or other imaging device that is being used. Related art systems relying on a single imaging device may have an increased probability of failing to find or detect the boundaries or edges around the objects" [Yu ¶0003] and " By using two or more modes of image data, differences in material, background, case or objects, environmental lighting, or reflection may be detected, and, based on these detections, the computing system can generate fused edge information to identify edges between gaps, or point clouds in order to determine object sizes. Further, as explained below, by fusing edge information detected using different modes of image data, example implementations may see the surrounding contour of the object or box, which is the edges of the box on the map, resulting in a higher possibility to define this region and give correct detection hypothesis for later processing. By improving the probability of detecting object positions and contours, objects may be better tracked and located".
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Park et. al. (US 20080008353 A1) and Ishihara et. al. (US 20190143525 A1) in view of Takaoka (US 20210023716 A1).
Regarding Claim 6, Modified Park discloses the robot control system according to Claim 1.
Furthermore, Park discloses: further comprising (See at least Figure 7A via step S250: Is Second SL Image Equal to Predicted Second SL Image? -> Yes **Which illustrates the condition where the actual second image matches the estimated/predicted image: Also see ¶0065 via "If determining (S250) in the comparing/analyzing unit 130 that the predicted image and the obtained image are equal to each other, the first image is obtained again, and the above steps are repeated. Preferably, the obtained second image is similar to the following first image " and additionally see ¶0068 via "exemplary embodiments can also be implemented by executing computer readable code/instructions in/on a medium/media…executed by a computing device")
However, although Park teaches rotating: "After the image obtaining unit 110 obtains the first image, the system 100b each moves by T.sub.x and T.sub.y in an x direction and a y direction and rotates by .theta. angle." [Park ¶0044], Park does not explicitly disclose the imaging of the object from a different direction than that of which the actual image information is obtained.
Nevertheless, Takaoka--who is directed towards automatically annotating items by robots--discloses: a viewpoint change instruction section (See at least ¶0005 via "…instructions, when executed by one or more processors, cause the controller to capture an image with the one or more imaging devices…rotate the one or more imaging devices based on the center, and capture an image of the item at a different viewing angle")
the viewpoint change instruction section causes the imaging device to image the object from a direction different from the imaging direction of the imaging device from which the actual image information is obtained (See at least ¶0046 via "…machine readable instructions that, when executed by the one or more processors 202, capture an image, identify one or more target areas in the image in response to one or more points on the image designated by a user, obtain depth information for each of the target areas, calculate a center of each of items corresponding to target areas based on the depth information, rotate the imaging device based on the center, and capture an image of each of the items at a different viewing angle in response to rotating the view of the imaging device").
Therefore, it would have been obvious to one of ordinary skill in the art prior to the effective filing date of the given invention to modify Modified Park in view of incorporating Takaoka's rotation of the imaging device to capture images of object(s) from different viewing angles, as Park already discloses imaging again when the second image and predicted second image are a match/equal. Changing the viewing angle when taking subsequent images is an obvious consideration because by rotating the imaging device, additional visual and depth information is obtained which helps to better identify the object: "the robot may identify target areas based on semantic segmentation and learn the appearance of multiple items in various different orientations by rotating imaging devices and capturing the multiple items at different angles. The robot trains the semantic segmentation module based on the images of the multiple items at different angles, which enhances the ability of identifying items and assigning the items to right destinations." [Takaoka ¶0060] and would thus yield a more efficient and accurate picking operation in the system of Modified Park.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KAYLA RENEE DOROS whose telephone number is (703)756-1415. The examiner can normally be reached Generally: M-F (8-5) EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abby Lin can be reached on (571) 270-3976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/K.R.D./Examiner, Art Unit 3657
/ABBY LIN/ Supervisory Patent Examiner, Art Unit 3657
Prosecution Insights