DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Applicant’s submission filed on 07/07/2026 has been entered. Claims 1, 3-5, and 10-12 were amended. Claims 15-20 were added. Claims 1-20 are pending in the application.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 4, 10-12, and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tanida et al. (US 2021/0350588) in view of Ma et al. (US 2024/0288932).
Regarding claim 1, Tanida teaches/suggests: A processing device (Tanida Fig. 1: wearable device 7), configured to:
acquire an image including a plurality of markers attached to a first tool for turning a screw (Tanida [0131] “the virtual tool IT in a one-to-one correspondence with the real tool RT imaged by the camera 10” [0103] “a real tool including a plurality of marks indicating a position of a head (socket) portion of the tool”);
calculate a first position of the first tool based on a plurality of positions of the plurality of markers in the image (Tanida [0106] “the augmented reality server 12 can obtain the coordinates of the head portion RTH by referring to the information on the mark RTMn (n=1 to 3) imaged by the camera 10 when the real bolt RBn having an axis along the RZ axis is to be fastened, and obtain the coordinates of the head portion RTH by mainly referring to the information on the mark RTMn (n=2, 4) when the real bolt RBn having an axis along an RY axis is to be fastened”); and
determine whether the screw is turned at the fastening location corresponding to the virtual first object based on a first distance between the virtual first object and the first position (Tanida [0123] “The augmented reality server 12 determines that a virtual bolt IBn having the virtual perpendicular line IZL, which is a basis of a smallest calculated distance ID1, is determined as the fastening target” [0132] “The augmented reality server 12 rotates the virtual tool IT around the IZ axis in the IZ-IY plane, in a state where the vertical axis RZ of the head portion of the virtual tool IT called in D4 is superimposed onto the IZ axis of coordinates IBO (IBX1, IBY1, IBZ1) of the virtual bolt IBn acquired in D6”).
Tanida is silent regarding an image including a hand of a worker. Nor does Tanida teach/suggest:
detect a position of the hand based on the image;
determine, based on a distance between the position of the hand and a virtual first object, that the hand touches the virtual first object, the virtual first object corresponding to a fastening location of the screw;
calculate a first position of the first tool based on a plurality of positions of the plurality of markers in the image in response to determining that the hand touches the virtual first object;
Ma, however, teaches/suggests an image including a hand of a worker (Ma [0046] “Based on the image of the physical scene 110, the electronic device 150 may determine whether the finger 111 is in direct contact with the keyboard 112 ... the electronic device 150 may determine which key or keys on the keyboard are the one or more keys”). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the augmented reality of Tanida to include the hand of the operator as taught/suggested by Ma to determine the bolt to be fastened. As such, Tanida as modified by Ma teaches/suggests:
detect a position of the hand based on the image (Ma [0046] “Based on the image of the physical scene 110, the electronic device 150 may determine whether the finger 111 is in direct contact with the keyboard 112”);
determine, based on a distance between the position of the hand and a virtual first object, that the hand touches the virtual first object, the virtual first object corresponding to a fastening location of the screw (Tanida [0123] “The augmented reality server 12 determines that a virtual bolt IBn having the virtual perpendicular line IZL, which is a basis of a smallest calculated distance ID1, is determined as the fastening target” Ma [0046] “if the finger 111 contacts one or more keys on the keyboard 112, or a distance between the finger 111 and the one or more keys is less than a predetermined distance, then the electronic device 150 may determine which key or keys on the keyboard are the one or more key”);
calculate a first position of the first tool based on a plurality of positions of the plurality of markers in the image in response to determining that the hand touches the virtual first object (Tanida [0106] “the augmented reality server 12 can obtain the coordinates of the head portion RTH by referring to the information on the mark RTMn (n=1 to 3) imaged by the camera 10 when the real bolt RBn having an axis along the RZ axis is to be fastened, and obtain the coordinates of the head portion RTH by mainly referring to the information on the mark RTMn (n=2, 4) when the real bolt RBn having an axis along an RY axis is to be fastened” Ma [0046] “if the finger 111 contacts one or more keys on the keyboard 112, or a distance between the finger 111 and the one or more keys is less than a predetermined distance, then the electronic device 150 may determine which key or keys on the keyboard are the one or more key”);
Regarding claim 2, Tanida as modified by Ma teaches/suggests: The processing device according to claim 1, wherein the processing device is configured to
receive a detection value from the first tool (Tanida [0067] “when the operator starts fastening the real bolt RBn determined as the fastening target with the real tool RT, the real tool RT detects generation of the fastening torque”), and
store the detection value in association with data related to the fastening location in a case where the screw is determined to be turned at the fastening location (Tanida [0068] “The augmented reality server 12 having received the initial fastening torque TB may store the date and time information together with the initial fastening torque TB” [0086] “the fastening management information 30 (such as the operator ID, the fastening completion date, the fastening completion time point, the completion fastening torque, and the design drawing number) associated with the fastening target object are stored”).
Regarding claim 4, Tanida as modified by Ma teaches/suggests: The processing device according to claim 1, wherein the processing device is configured to display a message in a case where the screw is determined not to be turned at the fastening location (Tanida [0074] “the augmented reality server 12 displays the fastening management information 30 such as an operator ID, a fastening completion date, a fastening completion time point, and a design drawing number, on the transmission type screen 14 in the vicinity of the virtual bolt IB1 whose display color is changed in S150 and on which the fastening is finished” [The finished fastening meets the screw is determined not to be turned.]).
Regarding claim 10, Tanida as modified by Ma teaches/suggests: A processing system (Tanida Fig. 1: wearable device 7), comprising:
the processing device according to claim 1 [See the treatment of claim 1.];
a display device configured to display the virtual first object (Tanida [0059] “displaying the specified virtual bolt IBn in a flashing manner in the augmented reality space projected on the transmission type screen 14 for the operator”); and
an imaging device configured to image the hand and the plurality of markers (Tanida [0103] “a real tool including a plurality of marks indicating a position of a head (socket) portion of the tool” Ma [0046] “Based on the image of the physical scene 110, the electronic device 150 may determine whether the finger 111 is in direct contact with the keyboard 112”).
The same rationale to combine as set forth in the rejection of claim 1 is incorporated herein.
Regarding claim 11, Tanida teaches/suggests: A mixed reality device (Tanida Fig. 1: wearable device 7), configured to:
display a virtual first object indicating a position where a hand of a worker should be positioned (Tanida [0059] “displaying the specified virtual bolt IBn in a flashing manner in the augmented reality space projected on the transmission type screen 14 for the operator” Ma [0046] “Based on the image of the physical scene 110, the electronic device 150 may determine whether the finger 111 is in direct contact with the keyboard 112”);
image the hand and a plurality of markers attached to a first tool for turning a screw (Tanida [0103] “a real tool including a plurality of marks indicating a position of a head (socket) portion of the tool” Ma [0046] “Based on the image of the physical scene 110, the electronic device 150 may determine whether the finger 111 is in direct contact with the keyboard 112”);
detect a position of the hand based on an image generated by imaging the hand and the plurality of markers (Ma [0046] “Based on the image of the physical scene 110, the electronic device 150 may determine whether the finger 111 is in direct contact with the keyboard 112”);
determine, based on a distance between the position of the hand and the virtual first object, that the hand touches the virtual first object (Tanida [0123] “The augmented reality server 12 determines that a virtual bolt IBn having the virtual perpendicular line IZL, which is a basis of a smallest calculated distance ID1, is determined as the fastening target” Ma [0046] “if the finger 111 contacts one or more keys on the keyboard 112, or a distance between the finger 111 and the one or more keys is less than a predetermined distance, then the electronic device 150 may determine which key or keys on the keyboard are the one or more key”);
calculate a first position of the first tool based on a plurality of positions of the plurality of markers in the image in response to determining that the hand touches the virtual first object (Tanida [0106] “the augmented reality server 12 can obtain the coordinates of the head portion RTH by referring to the information on the mark RTMn (n=1 to 3) imaged by the camera 10 when the real bolt RBn having an axis along the RZ axis is to be fastened, and obtain the coordinates of the head portion RTH by mainly referring to the information on the mark RTMn (n=2, 4) when the real bolt RBn having an axis along an RY axis is to be fastened” Ma [0046] “if the finger 111 contacts one or more keys on the keyboard 112, or a distance between the finger 111 and the one or more keys is less than a predetermined distance, then the electronic device 150 may determine which key or keys on the keyboard are the one or more key”);
calculate a first distance between the first position and the virtual first object (Tanida [0123] “The augmented reality server 12 determines that a virtual bolt IBn having the virtual perpendicular line IZL, which is a basis of a smallest calculated distance ID1, is determined as the fastening target”);
The same rationale to combine as set forth in the rejection of claim 1 is incorporated herein.
Tanida as modified by Ma does not teach/suggest:
display a message in a case where the first distance is longer than a first threshold.
However, official notice is taken that the concept and advantages of such a message are well known and expected in the art. It would have been obvious for the augmented reality of Tanida as modified by Ma to include such a message for warning.
Regarding claim 12, Tanida teaches/suggests: A processing method, comprising:
causing a display device to display a virtual first object indicating a position where a hand of a worker should be positioned (Tanida [0059] “displaying the specified virtual bolt IBn in a flashing manner in the augmented reality space projected on the transmission type screen 14 for the operator” Ma [0046] “Based on the image of the physical scene 110, the electronic device 150 may determine whether the finger 111 is in direct contact with the keyboard 112”);
causing an imaging device to image the hand and a plurality of markers attached to a first tool for turning a screw (Tanida [0103] “a real tool including a plurality of marks indicating a position of a head (socket) portion of the tool” Ma [0046] “Based on the image of the physical scene 110, the electronic device 150 may determine whether the finger 111 is in direct contact with the keyboard 112”);
causing a processing device to detect a position of the hand based on an image generated by the imaging device (Ma [0046] “Based on the image of the physical scene 110, the electronic device 150 may determine whether the finger 111 is in direct contact with the keyboard 112”);
causing the processing device to determine, based on a distance between the position of the hand and the virtual first object, that the hand touches the virtual first object (Tanida [0123] “The augmented reality server 12 determines that a virtual bolt IBn having the virtual perpendicular line IZL, which is a basis of a smallest calculated distance ID1, is determined as the fastening target” Ma [0046] “if the finger 111 contacts one or more keys on the keyboard 112, or a distance between the finger 111 and the one or more keys is less than a predetermined distance, then the electronic device 150 may determine which key or keys on the keyboard are the one or more key”);
causing the processing device to calculate a first position of the first tool for turning the screw using the plurality of markers attached to the first tool in response to determining that the hand touches the virtual first object (Tanida [0106] “the augmented reality server 12 can obtain the coordinates of the head portion RTH by referring to the information on the mark RTMn (n=1 to 3) imaged by the camera 10 when the real bolt RBn having an axis along the RZ axis is to be fastened, and obtain the coordinates of the head portion RTH by mainly referring to the information on the mark RTMn (n=2, 4) when the real bolt RBn having an axis along an RY axis is to be fastened” Ma [0046] “if the finger 111 contacts one or more keys on the keyboard 112, or a distance between the finger 111 and the one or more keys is less than a predetermined distance, then the electronic device 150 may determine which key or keys on the keyboard are the one or more key”); and
causing the processing device to determine whether the screw is turned at a fastening location corresponding to the virtual first object based on a first distance between the first position and the virtual first object (Tanida [0123] “The augmented reality server 12 determines that a virtual bolt IBn having the virtual perpendicular line IZL, which is a basis of a smallest calculated distance ID1, is determined as the fastening target” [0132] “The augmented reality server 12 rotates the virtual tool IT around the IZ axis in the IZ-IY plane, in a state where the vertical axis RZ of the head portion of the virtual tool IT called in D4 is superimposed onto the IZ axis of coordinates IBO (IBX1, IBY1, IBZ1) of the virtual bolt IBn acquired in D6”).
The same rationale to combine as set forth in the rejection of claim 1 is incorporated herein.
Regarding claim 14, Tanida as modified by Ma teaches/suggests: A non-transitory computer-readable storage medium storing a program (Tanida [0034] “The augmented reality server 12 is a computer including a memory area that stores application software”), the program causing a computer to execute the processing method according to claim 12. (See the treatment of claim 12.)
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tanida et al. (US 2021/0350588) in view of Ma et al. (US 2024/0288932) as applied to claim 1 above, and further in view of Valeda (US 6035747).
Regarding claim 3, Tanida as modified by Ma does not teach/suggest: The processing device according to claim 1, wherein the processing device is configured to
in a case where the screw is turned by the first tool via a second tool, calculate a second position of the second tool based on a plurality of positions of a plurality of markers attached to the second tool in response to determining that the hand touches the virtual first object,
calculate a second distance between a virtual second object and the second position, the virtual second object being displayed between the fastening location and the virtual first object, and
determine based on the first distance and the second distance whether the screw is turned at the fastening location.
Valeda, however, teaches/suggests a second tool (Valeda col. 1 ll. 21-42 “Extension bars are a common accessory frequently used with socket wrenches”). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the wrench (the first tool) of Tanida as modified by Ma to include the extension bar (the second tool) of Valeda as an accessory for the fastening. As such, Tanida as modified by Ma and Valeda teaches/suggests:
in a case where the screw is turned by the first tool via a second tool, calculate a second position of the second tool based on a plurality of positions of a plurality of markers attached to the second tool in response to determining that the hand touches the virtual first object (Tanida [0106] “the augmented reality server 12 can obtain the coordinates of the head portion RTH by referring to the information on the mark RTMn (n=1 to 3) imaged by the camera 10 when the real bolt RBn having an axis along the RZ axis is to be fastened, and obtain the coordinates of the head portion RTH by mainly referring to the information on the mark RTMn (n=2, 4) when the real bolt RBn having an axis along an RY axis is to be fastened” Ma [0046] “if the finger 111 contacts one or more keys on the keyboard 112, or a distance between the finger 111 and the one or more keys is less than a predetermined distance, then the electronic device 150 may determine which key or keys on the keyboard are the one or more key” Valeda col. 1 ll. 21-42 “Extension bars are a common accessory frequently used with socket wrenches” [In view of Tanida and Valeda, the extension bar would have its own markers for positioning.]),
calculate a second distance between a virtual second object and the second position, the virtual second object being displayed between the fastening location and the virtual first object (Tanida [0123] “The augmented reality server 12 determines that a virtual bolt IBn having the virtual perpendicular line IZL, which is a basis of a smallest calculated distance ID1, is determined as the fastening target” Valeda col. 1 ll. 21-42 “Extension bars are a common accessory frequently used with socket wrenches”), and
determine based on the first distance and the second distance whether the screw is turned at the fastening location (Tanida [0132] “The augmented reality server 12 rotates the virtual tool IT around the IZ axis in the IZ-IY plane, in a state where the vertical axis RZ of the head portion of the virtual tool IT called in D4 is superimposed onto the IZ axis of coordinates IBO (IBX1, IBY1, IBZ1) of the virtual bolt IBn acquired in D6” Valeda col. 1 ll. 21-42 “Extension bars are a common accessory frequently used with socket wrenches”).
Claim(s) 5, 9, and 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tanida et al. (US 2021/0350588) in view of Ma et al. (US 2024/0288932) as applied to claims 1 and 12 above, and further in view of Heymer (US 2020/0249009).
Regarding claim 5, Tanida as modified by Ma teaches/suggests four or more markers (Tanida [0104] “The mark RTMn (n=1 to 5) has a code indicating a direction from each mark RTMn to the head portion RTH and indicating each of distances L1 to L5 from each mark RTMn to the head portion RTH”). Tanida as modified by Ma does not teach/suggest: The processing device according to claim 1, wherein the processing device is configured to
in a case where the image includes four or more markers, extract a first marker, a second marker, and a third marker from the four or more markers, and
measure a position of each of the first to third markers to calculate the first position.
Heymer, however, teaches/suggests:
in a case where the image includes four or more markers, extract a first marker, a second marker, and a third marker from the four or more markers (Heymer [0049]-[0050] “a mark 38 for the purposes of alignment … Regarding the type A marker 22, the three points A, B and C are defined as corner points of an equilateral triangle” [0062] “These values (i.e., A.sub.0, B.sub.0, C.sub.0) represent the positions of the three points captured in the plane of the sensor”), and
measure a position of each of the first to third markers to calculate the first position (Heymer [0143] “comparing the x- and y-coordinates and the alignment of the three points A, B, C in the two images and calculating the translation and rotation of the component 20 in three-dimensional space between the first and the second position of the component 20”).
Before the effective filing date of the claimed invention, the substitution of one known element (the markers of Heymer) for another (the markers of Tanida) would have been obvious to one of ordinary skill in the art because such substitutions would have yielded predictable results, namely, for positioning.
Regarding claim 9, Tanida as modified by Ma does not teach/suggest: The processing device according to claim 1, wherein the processing device is configured to
set three points with fixed relative positions based on the plurality of markers, and
calculate the first position using a position of each of the three points.
Heymer, however, teaches/suggests:
set three points with fixed relative positions based on the plurality of markers (Heymer [0050] “Regarding the type A marker 22, the three points A, B and C are defined as corner points of an equilateral triangle”), and
calculate the first position using a position of each of the three points (Heymer [0143] “comparing the x- and y-coordinates and the alignment of the three points A, B, C in the two images and calculating the translation and rotation of the component 20 in three-dimensional space between the first and the second position of the component 20”).
The same rationale to combine as set forth in the rejection of claim 5 is incorporated herein.
Regarding claim 13, Tanida as modified by Ma does not teach/suggest: The processing method according to claim 12, wherein:
the plurality of markers including a first marker, a second marker, and a third marker are provided on the plane, and
a distance between the first marker and the second marker is different from a distance between the second marker and the third marker.
Heymer, however, teaches/suggests:
the plurality of markers including a first marker, a second marker, and a third marker are provided on the plane (Heymer [0087] “Regarding the type B marker 22 ... the points A, B, C no longer rely on the corner points of an equilateral triangle, but of an isosceles right triangle”), and
a distance between the first marker and the second marker is different from a distance between the second marker and the third marker (Heymer [0087] “Regarding the type B marker 22 ... the points A, B, C no longer rely on the corner points of an equilateral triangle, but of an isosceles right triangle” [The isosceles right triangle meets the distances.]).
The same rationale to combine as set forth in the rejection of claim 5 is incorporated herein.
Tanida, Ma, and Heymer are silent regarding:
a jig having a plane is attached to the first tool,
However, official notice is taken that the concept and advantages of such a rig are well known and expected in the art. It would have been obvious for the wrench (the first tool) of Tanida as modified by Ma and Heymer to include such a jig for attaching the markers.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tanida et al. (US 2021/0350588) in view of Ma et al. (US 2024/0288932) and Heymer (US 2020/0249009) as applied to claim 5 above, and further in view of Kurashima et al. (US 2023/0249341).
Regarding claim 6, Tanida as modified by Ma and Heymer teaches/suggests calculate a rotation amount and a translation amount (Heymer [0143] “comparing the x- and y-coordinates and the alignment of the three points A, B, C in the two images and calculating the translation and rotation of the component 20 in three-dimensional space between the first and the second position of the component 20”). The same rationale to combine as set forth in the rejection of claim 5 is incorporated herein.
Tanida as modified by Ma and Heymer does not teach/suggest: The processing device according to claim 5, wherein the processing device is configured to
calculate a rotation amount and a translation amount from pre-registered positions of the first to third markers to the measured positions of the first to third markers, and
calculate the first position by rotating and translating a pre-registered position of the first tool corresponding to the first to third markers according to the rotation amount and the translation amount.
Kurashima, however, teaches/suggests a pre-registered position of the first tool (Kurashima [0034] “in order to increase the accuracy of specifying the self-position, it is more preferable to register in advance information about objects placed in the surroundings or to place markers or the like in the surroundings” [0048] “The current relative position of the workpiece 40 and the pre-registered relative position of the workpiece 40 are compared to acquire a difference”). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the wrench (the first tool) of Tanida as modified by Ma and Heymer to be pre-registered as taught/suggested by Kurashima to specify its position. As such, Tanida as modified by Ma, Heymer, and Kurashima teaches/suggests:
calculate a rotation amount and a translation amount from pre-registered positions of the first to third markers to the measured positions of the first to third markers (Heymer [0143] “comparing the x- and y-coordinates and the alignment of the three points A, B, C in the two images and calculating the translation and rotation of the component 20 in three-dimensional space between the first and the second position of the component 20” Kurashima [0048] “The current relative position of the workpiece 40 and the pre-registered relative position of the workpiece 40 are compared to acquire a difference”), and
calculate the first position by rotating and translating a pre-registered position of the first tool corresponding to the first to third markers according to the rotation amount and the translation amount (Tanida [0132] “The augmented reality server 12 rotates the virtual tool IT around the IZ axis in the IZ-IY plane, in a state where the vertical axis RZ of the head portion of the virtual tool IT called in D4 is superimposed onto the IZ axis of coordinates IBO (IBX1, IBY1, IBZ1) of the virtual bolt IBn acquired in D6” Heymer [0143] “comparing the x- and y-coordinates and the alignment of the three points A, B, C in the two images and calculating the translation and rotation of the component 20 in three-dimensional space between the first and the second position of the component 20” Kurashima [0048] “The current relative position of the workpiece 40 and the pre-registered relative position of the workpiece 40 are compared to acquire a difference”).
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tanida et al. (US 2021/0350588) in view of Ma et al. (US 2024/0288932) and Heymer (US 2020/0249009) as applied to claim 5 above, and further in view of Kurashima et al. (US 2023/0249341) and Watanabe (US 2018/0061135).
Regarding claim 7, Tanida as modified by Ma and Heymer teaches/suggests indicating rotation, and translation (Heymer [0143] “comparing the x- and y-coordinates and the alignment of the three points A, B, C in the two images and calculating the translation and rotation of the component 20 in three-dimensional space between the first and the second position of the component 20”). The same rationale to combine as set forth in the rejection of claim 5 is incorporated herein.
Tanida as modified by Ma and Heymer does not teach/suggest: The processing device according to claim 5, wherein the processing device is configured to
calculate an
calculate the first position by
Kurashima, however, teaches/suggests a pre-registered position of the first tool (Kurashima [0034] “in order to increase the accuracy of specifying the self-position, it is more preferable to register in advance information about objects placed in the surroundings or to place markers or the like in the surroundings” [0048] “The current relative position of the workpiece 40 and the pre-registered relative position of the workpiece 40 are compared to acquire a difference”). The same rationale to combine as set forth in the rejection of claim 6 is incorporated herein. As such, Tanida as modified by Ma, Heymer, and Kurashima teaches/suggests:
calculate an
calculate the first position by (conversion data) for converting environment coordinate and robot coordinate can be created”).
Tanida as modified by Ma, Heymer, and Kurashima does not teach/suggest an affine transformation matrix. Watanabe, however, teaches/suggests an affine transformation matrix (Watanabe [0042] “The image processing unit 12 may apply an affine transformation to the each pixel of the reference image so that the orientation, position, and size of the communication device in the reference image are transformed to have a predetermined orientation, position, and size”). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the transformation matrix of Tanida as modified by Ma, Heymer, and Kurashima to be an affine transformation matrix as taught/suggested by Watanabe to preserve lines and parallelism.
Tanida, Ma, Heymer, Kurashima, and Watanabe are silent regarding deformation. However, official notice is taken that the concept and advantages of deformation are well known and expected in the art. It would have been obvious for the affine transformation matrix of Tanida as modified by Ma, Heymer, Kurashima, and Watanabe to include deformation for distortion.
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tanida et al. (US 2021/0350588) in view of Ma et al. (US 2024/0288932) as applied to claim 1 above, and further in view of Rodriguez Bravo et al. (US 2023/0054236).
Regarding claim 15, Tanida as modified by Ma teaches/suggests: The processing device according to claim 1, wherein the processing device is configured to:
determine that the hand touches the virtual first object when the distance is less than a first threshold (Tanida [0123] “The augmented reality server 12 determines that a virtual bolt IBn having the virtual perpendicular line IZL, which is a basis of a smallest calculated distance ID1, is determined as the fastening target” Ma [0046] “if the finger 111 contacts one or more keys on the keyboard 112, or a distance between the finger 111 and the one or more keys is less than a predetermined distance, then the electronic device 150 may determine which key or keys on the keyboard are the one or more key”);
The same rationale to combine as set forth in the rejection of claim 1 is incorporated herein.
Tanida as modified by Ma does not teach/suggest:
determine that the screw is turned at the fastening location when the first distance is not greater than a second threshold.
Rodriguez Bravo, however, teaches/suggests a second threshold (Rodriguez Bravo [0081] “a component is deemed to be 'nearby' if the distance between the component and the current screw hole 470b does not exceed a predetermined threshold”). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the distance between the wrench and the bolt (the first distance) of Tanida as modified by Ma to not exceed a threshold as taught/suggested by Rodriguez Bravo to determine that wrench is nearby the bolt. As such, Tanida as modified by Ma and Rodriguez Bravo teaches/suggests:
determine that the screw is turned at the fastening location when the first distance is not greater than a second threshold (Tanida [0123] “The augmented reality server 12 determines that a virtual bolt IBn having the virtual perpendicular line IZL, which is a basis of a smallest calculated distance ID1, is determined as the fastening target” Rodriguez Bravo [0081] “a component is deemed to be 'nearby' if the distance between the component and the current screw hole 470b does not exceed a predetermined threshold”).
Allowable Subject Matter
Claim 8 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claims 16-20 are allowed.
The following is a statement of reasons for the indication of allowable subject matter: The limitations “extract one or more triangles with a side along a first direction from a plurality of the triangles, the first direction corresponding to an orientation of the first tool,” “extract one triangle with the smallest angle between a normal vector of the triangle and a vector connecting an observation point and the triangle,” and “extract three markers corresponding to the one triangle from the plurality of markers as the first marker, the second marker, and the third marker,” taken as a whole, render the claim patentably distinct over the prior art.
Response to Arguments
Applicant's arguments filed on 07/07/2026 have been fully considered but they are moot in view of the new ground(s) of rejection set forth in this Office action.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
US 2021/0346098 – select tool based on distance
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANH-TUAN V NGUYEN whose telephone number is 571-270-7513. The examiner can normally be reached on M-F 9AM-5PM ET. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JASON CHAN can be reached on 571-272-3022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANH-TUAN V NGUYEN/
Primary Examiner, Art Unit 2619
Prosecution Insights