DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined
under the first inventor to file provisions of the AIA .
Priority
2. Receipt is acknowledged of certified copies of papers submitted under 35 U.S.C. 119(a)-(d), which papers have been placed of record in the file.
Specification Objection
3. The disclosure is objected to because of the following informalities:
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
In line 22 of page 8: “… In order to located ..." should be changed to --... In order to locate ...--.
Appropriate correction is required.
Claim Objections
4. Claims 6, 10 and 12-20 are objected to because of the following informalities:
In line 5 of claim 6: “… the position of ..." should be changed to --... a position of ...--;
In line 3 of claim 10: “… was turned to ..." should be changed to --... is turned to ...--;
In line 4 of claim 10: “… was turned to ..." should be changed to --... is turned to ...--;
In line 7 of claim 12: “… the virtual object, ..." should be changed to --... the virtual object, and ...--;
In line 1 of claim 13: “… according to any one of claim 12 ..." should be changed to --... according to claim 12 ...--;
In line 3 of claim 17: “… was turned to ..." should be changed to --... is turned to ...--;
In line 4 of claim 17: “… was turned to ..." should be changed to --... is turned to ...--; and
In line 4 of claim 19: “… the fastening object; ..." should be changed to --... the fastening object; and ...--.
Appropriate correction is required.
Claim Rejections - 35 USC § 102
5. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
6. Claims 1-2, 4, 6-10, 12-17 and 19-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Tanida (U.S. Pub. No. US 2021/0350588 A1).
As to claim 1, Tanida (Figs. 1-19) teaches a mixed reality device (a wearable device 7; Figs. 1-2), configured to display (Fig. 18) a virtual object (a virtual fastening unit IBn) corresponding to a fastening location (a location of a real fastening unit RBn) where a screw (a screw) is turned (Fig. 3), the virtual object (the virtual fastening unit IBn) being displayed at a position (coordinates IBO (IBX1, IBY1, IBZ1)) away from the fastening location (the location of the real fastening unit RBn) (Figs. 1-3, 8-9 and 18),
the mixed reality device (the wearable device 7) being configured to change a display position (coordinates IBO (IBX1, IBY1, IBZ1)) of the virtual object (the virtual fastening unit IBn) with respect to the fastening location (the location of the real fastening unit RBn) according to a physique of a wearer (an operator) (Figs. 1-3, 8-9 and 18).
As to claim 2, Tanida teaches the mixed reality device according to claim 1,
wherein the mixed reality device (the wearable device 7) is configured to change a height (a virtual line segment ILH) where the virtual object (the virtual fastening unit IBn) is displayed using a position of the fastening location (a location of the real fastening unit RBn) and a length (e.g., distances L1 to L5) of a pre-registered tool (a real tool RT) (Figs. 16-18).
As to claim 6, Tanida teaches the mixed reality device according to claim 1, wherein
the mixed reality device (the wearable device 7) is configured to recognize a prescribed marker (e.g., a socket portion RTH), and set a three-dimensional coordinate system (coordinates RTHO (RTHX1, RTHY1, RTHZ1)) based on the marker (the socket portion RTH) (Fig. 7), and
the position (coordinates (RBX1, RBY1, RBZ1)) of the fastening location (a location of the real fastening unit RBn) is preset in the three-dimensional coordinate system (3-D coordinate system) ([0093], lines 1-5) (Fig. 7).
As to claim 7, Tanida teaches the mixed reality device according to claim 1, wherein the mixed reality device (the wearable device 7) is configured to
display a plurality of the virtual objects (the virtual fastening units, i.e., the virtual bolts, IB1 to IB4) respectively corresponding to a plurality of the fastening locations (locations of the real fastening units, i.e., the real bolts, RB1 to RB4) (Figs. 3 and 7), and
display information indicating an order (an order) in which the screw (the screw) is to be turned on each of the plurality of virtual objects (the virtual fastening units, i.e., the virtual bolts, IB1 to IB4) (In step 70 (S70), the augmented reality server 12 specified an order in which all of the virtual bolts Ibn are to be fastened, based on the virtual space information acquired in S50, in the augmented reality space, and shows a bolt to be fastened to the operator; [0059], lines 1-5) (Figs. 3 and 8).
As to claim 8, Tanida teaches the mixed reality device according to claim 1, wherein
the mixed reality device (the wearable device 7) is configured to determine whether the virtual object (e.g., the virtual fastening unit IBn) comes into contact with a prescribed physical object (a virtual base ID1) (Fig. 7), and
in a case where the virtual object (e.g., the virtual fastening unit IBn) comes into contact with the prescribed physical object (the virtual base ID1), the mixed reality device (the wearable device 7) estimates that the screw (the screw) is being turned to the fastening location (the location of the real fastening unit RBn) (Figs. 7 and 19).
As to claim 9, Tanida teaches the mixed reality device according to claim 8,
wherein the mixed reality device (the wearable device 7) is configured to associate data related to the estimated fastening location (the location of the real fastening unit RBn) with a task record (a part of fastening management information 30) ([0080], lines 8-10) (Figs 10-11 and 19).
As to claim 10, Tanida teaches the mixed reality device according to claim 9, wherein
the task record (the part of fastening management information 30) includes a torque (a specified fastening torque, i.e., 106(Nm); Fig. 13) detected when the screw (the screw) was turned to the fastening location (the location of the real fastening unit RBn) and a number of times (the number of fastening times which correspond to the number of turning times; [0080], lines 8-10) the screw (the screw) was turned to the fastening location (the location of the real fastening unit RBn) (Figs 10-11, 13-14 and 19), and
the mixed reality device (the wearable device 7) is configured to
compare the torque (the torque) and the number of times (the number of fastening times) with a required torque (a specified fastening torque, i.e., 106(Nm); Fig. 13) and a required number of times (a number of fastening times which corresponds to a fastening completion time point HH:MM:SS) preset for the fastening location (the location of the real fastening unit RBn), respectively (Figs. 11 and 13-14), and
determine that an error (a difference) is present in a case where the torque (the torque) is insufficient compared to the required torque (the specified fastening torque TS, i.e., 106(Nm); Fig. 13) or in a case where the number of times is less than the required number of times (Figs. 11 and 13-14).
As to claim 4, Tanida (Figs. 1-19) teaches a mixed reality device (a wearable device 7; Figs. 1-2), configured to display (Fig. 18) a virtual object (a virtual fastening unit IBn) corresponding to a fastening location (a location of a real fastening unit RBn) where a screw (a screw) is turned (Fig. 3), the virtual object (the virtual fastening unit IBn) being displayed at a position (coordinates IBO (IBX1, IBY1, IBZ1)) away from the fastening location (the location of the real fastening unit RBn) (Figs. 1-3, 8-9 and 18),
the mixed reality device (the wearable device 7) being configured to change a display position (coordinates IBO (IBX1, IBY1, IBZ1)) of the virtual object (the virtual fastening unit IBn) with respect to the fastening location (the location of the real fastening unit RBn) according to a positional relationship between the fastening location (the location of the real fastening unit RBn) and the mixed reality device (the wearable device 7) (Figs. 2 and 18).
As to claim 12, Tanida (Figs. 1-19) teaches a mixed reality device (a wearable device 7; Figs. 1-2), configured to display (Fig. 18) a virtual object (a virtual fastening unit IBn) corresponding to a fastening location (a location of a real fastening unit RBn) where a screw (a screw) is turned (Fig. 3), the virtual object (the virtual fastening unit IBn) being displayed at a position (coordinates IBO (IBX1, IBY1, IBZ1)) away from the fastening location (the location of the real fastening unit RBn) (Figs. 1-3, 8-9 and 18),
the mixed reality device (the wearable device 7) measuring a distance between a hand (a hand) of a worker (an operator) and the mixed reality device (the wearable device 7) in a case where a prescribed physical object (a real fastening unit RBn) comes into contact with the virtual object (the virtual fastening unit IBn) (Figs. 1-2 and 19),
the mixed reality device (the wearable device 7) then displaying an alert (‘No’) in a case where the distance is outside a first range set based on a physique of a wearer (determine fastening target RBn (S90) based on bringing tool (T) close to fastening target RBn (S80)) (Fig. 8), or in a case where a tilt of the mixed reality device exceeds a first threshold.
As to claim 13, Tanida teaches the mixed reality device according to any one of claim 12, wherein
the mixed reality device (the wearable device 7) is configured to recognize a prescribed marker (e.g., a socket portion RTH), and set a three-dimensional coordinate system (coordinates RTHO (RTHX1, RTHY1, RTHZ1)) based on the marker (the socket portion RTH) (Fig. 7), and
the position (coordinates (RBX1, RBY1, RBZ1)) of the fastening location (a location of the real fastening unit RBn) is preset in the three-dimensional coordinate system (3-D coordinate system) ([0093], lines 1-5) (Fig. 7).
As to claim 14, Tanida teaches the mixed reality device according to claim 12, wherein the mixed reality device (the wearable device 7) is configured to
display a plurality of the virtual objects (the virtual fastening units, i.e., the virtual bolts, IB1 to IB4) respectively corresponding to a plurality of the fastening locations (locations of the real fastening units, i.e., the real bolts, RB1 to RB4) (Figs. 3 and 7), and
display information indicating an order (an order) in which the screw (the screw) is to be turned on each of the plurality of virtual objects (the virtual fastening units, i.e., the virtual bolts, IB1 to IB4) (In step 70 (S70), the augmented reality server 12 specified an order in which all of the virtual bolts Ibn are to be fastened, based on the virtual space information acquired in S50, in the augmented reality space, and shows a bolt to be fastened to the operator; [0059], lines 1-5) (Figs. 3 and 8).
As to claim 15, Tanida teaches the mixed reality device according to claim 12, wherein
the mixed reality device (the wearable device 7) is configured to determine whether the virtual object (e.g., the virtual fastening unit IBn) comes into contact with a prescribed physical object (a virtual base ID1) (Fig. 7), and
in a case where the virtual object (e.g., the virtual fastening unit IBn) comes into contact with the prescribed physical object (the virtual base ID1), the mixed reality device (the wearable device 7) estimates that the screw (the screw) is being turned to the fastening location (the location of the real fastening unit RBn) (Figs. 7 and 19).
As to claim 16, Tanida teaches the mixed reality device according to claim 15,
wherein the mixed reality device (the wearable device 7) is configured to associate data related to the estimated fastening location (the location of the real fastening unit RBn) with a task record (a part of fastening management information 30) ([0080], lines 8-10) (Figs 10-11 and 19).
As to claim 17, Tanida teaches the mixed reality device according to claim 16, wherein
the task record (the part of fastening management information 30) includes a torque (a specified fastening torque, i.e., 106(Nm); Fig. 13) detected when the screw (the screw) was turned to the fastening location (the location of the real fastening unit RBn) and a number of times (the number of fastening times which correspond to the number of turning times; [0080], lines 8-10) the screw (the screw) was turned to the fastening location (the location of the real fastening unit RBn) (Figs 10-11, 13-14 and 19), and
the mixed reality device (the wearable device 7) is configured to
compare the torque (the torque) and the number of times (the number of fastening times) with a required torque (a specified fastening torque, i.e., 106(Nm); Fig. 13) and a required number of times (a number of fastening times which corresponds to a fastening completion time point HH:MM:SS) preset for the fastening location (the location of the real fastening unit RBn), respectively (Figs. 11 and 13-14), and
determine that an error (a difference) is present in a case where the torque (the torque) is insufficient compared to the required torque (the specified fastening torque TS, i.e., 106(Nm); Fig. 13) or in a case where the number of times is less than the required number of times (Figs. 11 and 13-14).
As to claim 19, Tanida (Figs. 1-19) teaches a processing method (Figs. 8-9), causing a mixed reality device (a wearable device 7; Figs. 1-2) to:
display a virtual object (a virtual fastening unit IBn) corresponding to a fastening location (a location of a real fastening unit RBn) where a screw (a screw) is turned (Fig. 3), the virtual object (the virtual fastening unit IBn) being displayed at a position (coordinates IBO (IBX1, IBY1, IBZ1)) away from the fastening location (the location of the real fastening unit RBn) (Figs. 1-3, 8-9 and 18);
change a display position (coordinates IBO (IBX1, IBY1, IBZ1)) of the virtual object (the virtual fastening unit IBn) with respect to the fastening location (the location of the real fastening unit RBn) according to a physique of a wearer or a positional relationship between the fastening location (the location of the real fastening unit RBn) and the mixed reality device (the wearable device 7) (Figs. 2 and 18).
As to claim 20, Tanida teaches a non-transitory computer-readable storage medium (an augmented reality server (comprehensive control unit) 12; Fig. 2), storing a program for causing a computer to execute the processing method (Figs. 8-9) according to claim 19.
Allowable Subject Matter
7. Claims 3, 5, 11 and 18 are objected to as being dependent upon a
rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: The prior art of record, Tanida, Marchand, and McCulloch, either individually or in combination, does not teach a limitation “wherein the mixed reality device is configured to calculate a first range within which a task can be performed to the fastening location using a position of the fastening location and a length of a pre-registered tool, calculate a second range within which the wearer can perform the task based on a position of the mixed reality device and the physique, and display the virtual object within an overlapping range of the first range and the second range” of claim 3, a limitation “wherein in a case where the mixed reality device is positioned in a first direction with respect to the fastening location, the mixed reality device changes the display position of the virtual object towards the first direction, and in a case where the mixed reality device is positioned in a second direction opposite to the first direction with respect to the fastening location, the mixed reality device changes the display position of the virtual object towards the second direction” of claim 5, and a limitation “an acquisition system configured to acquire the physique referred by the mixed reality device according to claim 1, the acquisition system configured to calculate the physique based on a distance between the mixed reality device and the hand of the wearer or from an image showing a body of the wearer” of claim 11/18 in combination with other limitations of base claims, respectively.
Conclusion
8. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Marchand (U.S. Pub. No. US 2021/0158587 A1) is cited to teach an augmented reality spatial guidance and a procedure control system by which a worker and trainee is provided in a setting computer-guided assistance through execution of a predetermined procedure.
McCulloch (U.S. Pub. No. US 2013/0328762 A1) is cited to teach a system for controlling a virtual object displayed by a near-eye, augmented reality display with a real controller device.
Inquiry
9. Any inquiry concerning this communication or earlier communications from the examiner should be directed to Kwang-Su Yang whose telephone number is (571)270-7307. The examiner can normally be reached on Mon-Fri during 9:00am-6:00pm EST.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chanh Nguyen, can be reached on (571)272-7772. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
/KWANG-SU YANG/
Primary Examiner, Art Unit 2623