DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Election/Restrictions
The applicant elects Group 1 including pending claims 1-9. Claims 10-13 are hereby withdrawn from consideration.
35 USC § 112 (f) Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are: “imaging part”, and “processing part” in claim 1. “Part” is a generic placeholder and is coupled with functional language “to acquire” and “to recognize”.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-9 are rejected under 35 U.S.C. 103 as being unpatentable over Karri (US 20230089436 A1) and in view of Tokita (US 20080059131 A1 ).
Regarding to claim 1 (Original), Karri discloses a mixed reality device ([0013]: identify if the worker has properly fastened or unfastened the machine parts as per the identified specification information; Fig. 1; [0018]: server computer 102, wearable device 104, and augmented reality (AR) device 106; [0020]: AR device 106 represents a user wearable augment reality device; assembly validation program 112 receives a video feed from camera 120 of a user performing a fastening or unfastening operation during an assembly or disassembly process; [0032]: receive a video feed from a camera on an augmented reality device associated with the user), comprising:
an imaging part configured to acquire an image ([0014]: an augmented reality (AR) device associated with the user captures a feed of the fastening or unfastening operation; [0020]: assembly validation program 112 receives a video feed from camera 120 of a user performing a fastening operation during an assembly process; [0032]: assembly validation program 112 receives a video feed from a camera on an augmented reality device associated with the user; [0038]: a video is received from a camera on the augmented reality device); and
a processing part configured to recognize, from the image, a hand of a worker performing a fastening task ([0023]: assembly validation program 112 identifies individual parts , e.g., bolts, screws, clamps, for assembly or disassembly; identify one or more tools; Fig. 2B; [002B]: a hex bolt is torqued with a wrench by a user hands with a wearable device;
PNG
media_image1.png
366
590
media_image1.png
Greyscale
; Fig. 2C; [0029]: a socket hand tool; determines machine component 200 is being improperly fastened by the user; [0032]: identify an assembly or disassembly process being performed on the product; Fig. 3B; [0037]: assembly validation program 112 identifies a current step in the assembly or disassembly process based on a pattern of hand motions being performed by the user; recognize hand motions; [0045]: detect if a hand is rotating, and a type of rotating; recognize a hand; [0047]: compare the expected versus actual speed of rotation and movement of a hand during assembly; recognize a hand) and to determine whether the fastening task is appropriate based on a first distance between a first position corresponding to a fastening location of a screw and a second position ([0023]: assembly validation program 112 validates proper fastening of components during an assembly process; [0024]: assembly validation program 112 monitors actions of the user while the user performs the fastening during the current step and determine whether corrective action is needed; [0027]: ensure a proper fastening; Fig. 2B; [0028]: assembly validation program 112 monitors the progress of the 45 degrees via wearable device 104 to ensure the user does not overlook the subsequent turn to complete the torquing procedure;
PNG
media_image2.png
326
506
media_image2.png
Greyscale
; Fig. 2C; [0029]: assembly validation program 112 determines machine component 200 is being improperly fastened by the user and a corrective action by the user is needed; [0047]: the additional embodiments compares the expected versus actual speed of rotation and movement of a hand during assembly or disassembly operations and predict if proper fastening or unfastening is being performed; speed and movement of a hand includes location changes and distance).
Karri fails to explicitly disclose a second position of the hand holding a first tool.
In same field of endeavor, Tokita teaches a second position of the hand holding a first tool (Fig. 1; [0124]: a hand grips a virtual machine tool; virtual bolt is in a first location.
PNG
media_image3.png
284
546
media_image3.png
Greyscale
; Fig. 2A; [0126]: the user grips an actual machine tool;
PNG
media_image4.png
234
508
media_image4.png
Greyscale
; Fig. 30; [0245]: when actual bolt tightening is simulated by using a virtual machine tool, the operability is evaluated and verified;
PNG
media_image5.png
506
578
media_image5.png
Greyscale
; a hand holds a tool as illustrated in Fig. 30).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Karri to include a second position of the hand holding a first tool as taught by Tokita. The motivation for doing so would have been to present a sense of force of tightening a screw; to verify whether there is sufficient space to allow bolt tightening; to determine whether the user tightens the screw with a strong or weak force; to present a force sense in tightening or undoing a bolt in the machine tool grip styles; to improve the reality of the torque virtually perceived by the user as taught by Tokita in paragraphs [0026], [0124], [0185], [0187], and [0196]
Regarding to claim 2 (Original), Karri in view of Tokita discloses the mixed reality device according to claim 1, wherein
Karri in view of Tokita further discloses determine that the fastening task is not appropriate (Karri; Fig. 2C; [0029]: assembly validation program 112 determines machine component 200 is being improperly fastened by the user and a corrective action by the user is needed);
Karri in view of Tokita further more discloses the processing part is configured to calculate a first error between a first length of the first tool registered in advance and the first distance (Tokita; [0121]: verify whether assembly or maintenance can be done by inserting a jig or tool; Fig. 1; [0124]: verify whether an operation such as assembly or maintenance can be done by inserting a jig or tool into a virtually displayed product in operation verification of an industrial product or device by virtual reality or mixed reality; accurately verify whether a machine tool can enter or whether there is sufficient space to allow bolt tightening; [0198]: verify the operability in assembly or maintenance by inserting a jig or tool in a virtually displayed product; [0215]: accurately verify whether a machine tool can enter or whether there is sufficient space to allow bolt tightening), and determine that the fastening task is not appropriate in a case where the first error is greater than a first threshold value (Tokita; [0121]: verify whether a hand can enter a narrow internal space or whether an operation such as assembly or maintenance can be done by inserting a jig or tool; Fig. 1; [0124]: accurately verify whether a machine tool can enter or whether there is sufficient space to allow bolt tightening).
Same motivation of claim 1 is applied here.
Regarding to claim 3 (Original), Karri in view of Tokita discloses the mixed reality device according to claim 1, wherein
the processing part is configured to determine whether the first position is appropriate by comparing the first position with a position of the fastening location registered in advance (Tokita; Fig. 30; [0245]: a view showing the images of virtual objects displayed by virtual reality or mixed reality and hands gripping a virtual machine tool; more accurately verify whether a machine tool can enter or whether there is sufficient space to allow bolt tightening; [0246]: the user wraps multiple fingers of one hand around the machine tool while holding a screw with the other hand; Fig. 34; [0262]).
Same motivation of claim 1 is applied here.
Regarding to claim 4 (Original), Karri in view of Tokita discloses the mixed reality device according to claim 1, wherein
in a case where the screw is tightened by a second tool via the first tool (Karri; [0013]: identify one or more tools utilized during the assembled; [0023]: identify one or more tools for assembly; [0025]: tools for fastening and unfastening each of the individual parts; Fig. 2C; [0029]: a socket hand tool and torque wrench),
Karri in view of Tokita further discloses:
the processing part recognizes two hands of the worker (Tokita; Fig. 30; [0245]: when actual bolt tightening is simulated by using a virtual machine tool, the operability is evaluated and verified;
PNG
media_image5.png
506
578
media_image5.png
Greyscale
; recognize left hand and right hand as illustrated in Fig. 30; Fig. 31; [0248]: recognize two hands as illustrated in Fig. 31;
PNG
media_image6.png
600
724
media_image6.png
Greyscale
) and
determines whether the second tool is appropriate based on a second distance between the second position of one of the two hands and a third position of the other hand of the two hands (Tokita; Fig. 30; [0245]: when actual bolt tightening is simulated by using a virtual machine tool, the operability is evaluated and verified;
PNG
media_image5.png
506
578
media_image5.png
Greyscale
; Fig. 34; [0265]: the positioning sensor 550 measures the relative position and orientation between the fixed unit 2 and the force sense presentation units 3 and sends the measured position and orientation to the input device 706 of the arithmetic processing unit 7; the positions of two hands are proper as illustrated in Fig. 34;
PNG
media_image7.png
244
528
media_image7.png
Greyscale
).
Same motivation of claim 1 is applied here.
Regarding to claim 5 (Original), Karri in view of Tokita discloses the mixed reality device according to claim 4, wherein
the processing part determines that the fastening task is not appropriate in a case where a difference between a height of the second position and a height of the third position exceeds a threshold value (Karri; Fig. 2B; [0028]: if the torquing procedures requires machine component 200 be torqued to 100 ft-lbs. and subsequently turned 45 degrees, assembly validation program 112 can monitor the progress of the 45 degrees via wearable device 104; the sharp jolt of the torque wrench is capturable by wearable device 104 associated with the user;
PNG
media_image8.png
374
572
media_image8.png
Greyscale
; the different of height is proper as illustrated in Fig. 2B, otherwise, is not proper; Fig. 2C; [0029]: Assembly validation program 112 determines machine component 200 is being improperly fastened by the user and a corrective action by the user is needed; assembly validation program 112 display the corrective action to the user, which includes the proper torquing procedures from the specification associated with machine component 200 and tool, i.e., torque wrench, required to perform the proper torquing procedure per the specification).
Karri in view of Tokita further discloses:
the processing part determines that the fastening task is not appropriate in a case where a difference between a height of the second position and a height of the third position exceeds a threshold value (Tokita; Fig. 30; [0245]: when actual bolt tightening is simulated by using a virtual machine tool, the operability is evaluated and verified; decide proper or not proper;
PNG
media_image5.png
506
578
media_image5.png
Greyscale
; Fig. 34; [0265]: the positioning sensor 550 measures the relative position and orientation between the fixed unit 2 and the force sense presentation units 3 and sends the measured position and orientation to the input device 706 of the arithmetic processing unit 7; the positions of two hands are proper as illustrated in Fig. 34, otherwise is not proper;
PNG
media_image7.png
244
528
media_image7.png
Greyscale
).
Regarding to claim 6 (Original), Karri in view of Tokita discloses the mixed reality device according to claim 1, wherein
the processing part is configured to measure the first position at a timing indicated by a hand gesture or a voice command (Karri; Fig. 2B; [0028]: if the torquing procedures requires machine component 200 be torqued to 100 ft-lbs. and subsequently turned 45 degrees, assembly validation program 112 can monitor the progress of the 45 degrees via wearable device 104; the sharp jolt of the torque wrench is capturable by wearable device 104 associated with the user;
PNG
media_image8.png
374
572
media_image8.png
Greyscale
).
Karri in view of Tokita further discloses the processing part is configured to measure the first position at a timing indicated by a hand gesture or a voice command (Tokita; [0005]: measure the position of a human hand; Fig. 30; [0245]: when actual bolt tightening is simulated by using a virtual machine tool, the operability can be evaluated and verified;
PNG
media_image5.png
506
578
media_image5.png
Greyscale
).
Regarding to claim 7 (Original), Karri in view of Tokita discloses the mixed reality device according to claim 1, further comprising a display part,
the processing part is configured to cause the display part to display a first object indicating the first position and a second object indicating the second position (Tokita; Fig. 30; [0245]: when actual bolt tightening is simulated by using a virtual machine tool, the operability can be evaluated and verified;
PNG
media_image5.png
506
578
media_image5.png
Greyscale
; Fig. 34; [0265]: the positioning sensor 550 measures the relative position and orientation between the fixed unit 2 and the force sense presentation units 3 and sends the measured position and orientation to the input device 706 of the arithmetic processing unit 7; the positions of two hands are proper as illustrated in Fig. 34;
PNG
media_image7.png
244
528
media_image7.png
Greyscale
).
Regarding to claim 8 (Original), Karri in view of Tokita discloses the mixed reality device according to claim 1, wherein
in a case where a marker is imaged by the imaging part, the processing part sets a three-dimensional coordinate system based on the marker and measures the first position and the second position in the coordinate system (Tokita; Fig. 20A; [0191]: a three-dimensional pattern for easy grip of the gripping region, or a mark serving as a guide for grip; Fig. 35; [0271]: a three-dimensional pattern for easy grip of the gripping region or a mark serving as a guide for grip may be formed;
PNG
media_image9.png
334
558
media_image9.png
Greyscale
).
Regarding to claim 9 (Original), Karri in view of Tokita discloses the mixed reality device according to claim 1, wherein
in a case where the processing part receives a detection value when the screw is tightened, the processing part stores the detection value in association with data related to the fastening location (Karri; [0025]: stores various data utilized by assembly validation program 112 including specification information for individual parts and tools for fastening and unfastening each of the individual parts; Fig. 2B; [0028]: if the torquing procedures requires machine component 200 be torqued to 100 ft-lbs. and subsequently turned 45 degrees, assembly validation program 112 can monitor the progress of the 45 degrees via wearable device 104; the sharp jolt of the torque wrench is capturable by wearable device 104 associated with the user;
PNG
media_image8.png
374
572
media_image8.png
Greyscale
; [0038]: assembly validation program 112 stores the first step of a set of activation events; compare the actions of the user to a set of actions associated with an assembly or disassembly process for each specific product stored in a library).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Hai Tao Sun whose telephone number is (571)272-5630. The examiner can normally be reached 9:00AM-6:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached at 5712727642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HAI TAO SUN/Primary Examiner, Art Unit 2616
Prosecution Insights