DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The following action is in response to the original filing of 07/11/2024.
Claims 1-20 are pending and have been considered below.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-6, 11-14 and 18-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 8 and 10-12 of U.S. Patent No. 12,061,743 (‘743) in view of Balan’139, US 2017/0358139. Although the claims at issue are not identical, they are not patentably distinct from each other as follows.
Regarding instant claim 1, claim 8 of ‘743 augmented reality (AR) system comprising:
a handheld device comprising a set of handheld fiducials (“a handheld device pose sensor configured to capture handheld device pose data including positions of a set of keypoints of a handheld device”);
a wearable device comprising:
having a field of view that at least partially includes the set of handheld fiducials and a hand of a user (“a hand pose sensor configured to capture hand pose data including positions of a set of keypoints of a hand”); and
a computing apparatus (“one or more processors in communication with the hand pose sensor and the handheld device pose sensor”) configured to:
receive hand pose data associated with the hand based on an image captured by the imaging device (“determining a pose discrepancy between the hand pose data and the handheld device pose data”);
receive handheld device pose data associated with the handheld device based on the set of handheld fiducials appearing in the image captured by the imaging device (“determining a pose discrepancy between the hand pose data and the handheld device pose data”);
determine a pose discrepancy between the hand pose data and the handheld device pose data (“determining a pose discrepancy between the hand pose data and the handheld device pose data”); and
perform an operation to fuse the hand pose data with the handheld device pose data (“and performing a fusion operation based on the pose discrepancy”).
Claim 8 of ‘743 fails to disclose wherein the wearable device comprises a display operable to display virtual content; and an imaging device mounted to the wearable device.
Balan’139 discloses methods for disambiguating tracking of handheld devices in AR systems, an analogous art (pp. 4). In particular, Balan’139 discloses a wearable device having display for virtual content and an imaging device for capturing field of view including handheld device and hands (pp. 32, pp.42, pp. 92). Therefore it would have been obvious to one having ordinary skill in the art and the teachings of claim 8 of ‘743 and Balan’139 before them before the effective filing of the claimed invention to combine the teachings of utilizing a wearable device having a display for virtual content and an imaging device for capturing images, as taught by Balan’139, with the wearable device of claim 8 of ‘743. One would have been motivated to make this combination to provide more accurate discrepancy determinations in HMD device situations, as suggested by Balan’139 (pp. 20, pp. 22).
Regarding claim 2, claim 8 of ‘743 and Balan’139 disclose the AR system of claim 1, and claim 8 of ‘743 further discloses wherein determining the pose discrepancy includes determining a position discrepancy between a position of the hand and a position of the handheld device (“wherein determining the pose discrepancy includes: determining a position discrepancy based on a set of distances, wherein each distance in the set of distances corresponds to a distance between a position of a first keypoint in the set of keypoints of the handheld device and a position of a second keypoint matched to the first keypoint in the set of keypoints of the hand”).
Regarding claim 3, claim 8 of ‘743 and Balan’139 disclose the AR system of claim 2, and claim 8 of ‘743 further discloses wherein the position discrepancy is a distance between the position of the hand and the position of the handheld device (“wherein determining the pose discrepancy includes: determining a position discrepancy based on a set of distances, wherein each distance in the set of distances corresponds to a distance between a position of a first keypoint in the set of keypoints of the handheld device and a position of a second keypoint matched to the first keypoint in the set of keypoints of the hand”).
Regarding claim 4, claim 8 of ‘743 and Balan’139 disclose the AR system of claim 1, and claim 8 of ‘743 further discloses wherein determining the pose discrepancy comprises determining an orientation discrepancy between an angle of orientation of the hand and an angle of orientation of the handheld device (“and determining an orientation discrepancy between the orientation of the hand and the orientation of the handheld device”).
Regarding claim 5, claim 8 of ‘743 and Balan’139 disclose the AR system of claim 1, and claim 8 of ‘743 further discloses wherein the image captured by the imaging device is analyzed to determine positions of a set of keypoints of the handheld device and a set of keypoints of the hand (“including positions of a set of keypoints of a hand” .. “including positions of a set of keypoints of a handheld device”), and wherein each keypoint of the set of keypoints of the handheld device corresponds to a different keypoint of the set of keypoints of the hand (“wherein each keypoint of the set of keypoints of the handheld device is matched to a different keypoint of the set of keypoints of the hand”).
Regarding claim 6, claim 8 of ‘743 and Balan’139 disclose the AR system of claim 5, and claim 8 of ‘743 further discloses wherein the computing apparatus determines the pose discrepancy using a set of distances between the positions of the set of keypoints of the handheld device and the set of keypoints of the hand (“wherein determining the pose discrepancy includes: determining a position discrepancy based on a set of distances, wherein each distance in the set of distances corresponds to a distance between a position of a first keypoint in the set of keypoints of the handheld device and a position of a second keypoint matched to the first keypoint in the set of keypoints of the hand”).
Regarding instant claims 11-14, instant claims 11-14 recite limitations similar to instant claims 1-4 and are similarly rejected.
Regarding claim 18, claim 8 of ‘743 and Balan’139 disclose the method of claim 11, and claim 10 of ‘743 further discloses further comprising:
determining whether the pose discrepancy is greater than or less than a threshold (“determining whether the pose discrepancy is greater than or less than a threshold”).
Regarding claim 19, claim 8 of ‘743 and Balan’139 disclose the method of claim 18, and claim 11 of ‘743 further discloses further wherein if it is determined that the pose discrepancy is greater than the threshold, performing the operation to fuse the hand pose data with the handheld device pose data includes determining that the handheld device is not being held by the hand (“wherein if it is determined that the pose discrepancy is greater than the threshold, performing the fusion operation includes: determining that the handheld device is not being held by the hand”).
Regarding claim 20, claim 8 of ‘743 and Balan’139 disclose the method of claim 19, and claim 12 of ‘743 further discloses further wherein if it is determined that the pose discrepancy is less than or equal to the threshold, performing the operation to fuse the hand pose data with the handheld device pose data includes determining that the handheld device is being held by the hand (“wherein if it is determined that the pose discrepancy is less than or equal to the threshold, performing the fusion operation includes: determining that the handheld device is being held by the hand”).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-6, 9-15 and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Balan’332, US 2017/0357332, in view of Balan’139, US 2017/0358139.
Regarding claim 1, Balan’332 discloses an augmented reality (AR) system (pp. 1, pp. 4) comprising:
a handheld device (pp. 4: controller with inertial measurement unit);
a wearable device (pp. 4: HMD) comprising:
a display operable to display virtual content (pp. 4: HMD, head mounted display, pp. 1: displays virtual content); and
an imaging device mounted to the wearable device and having a field of view that at least partially includes a hand of a user (pp. 4: HMD forward-facing depth camera, pp. 6: detect user’s hand within field of view, pp. 41: camera detects 3D location of the controller); and
a computing apparatus configured to:
receive hand pose data associated with the hand based on an image captured by the imaging device (pp. 69-70: HMD optical sensor track’s user hand);
receive handheld device pose data associated with the handheld device (pp. 69, pp. 72-73: receive acceleration data from IMU of the controller);
determine a pose discrepancy between the hand pose data and the handheld device pose data (pp. 69, pp. 72-73); and
perform an operation to fuse the hand pose data with the handheld device pose data (pp. 63, pp. 69, pp. 72-76: fusion operation based on pose discrepancy).
While Balan’332 discloses determining the discrepancy between positions of the hand and positions of the handheld device (pp. 69, pp. 72-75) and further discloses wherein the detection of hands may be done through segmenting (pp. 60), Balan’332 ails to explicitly disclose wherein the handheld device pose data is based on captured handheld fiducials of the handheld device captured by the imaging device.
Balan’139 discloses methods for disambiguating tracking of handheld devices in AR systems, an analogous art (pp. 4). In particular, Balan’139 discloses capturing fiducials of a handheld device (pp. 92: markers on the controller) and fiducials of the hand (pp. 92: hand centroids, palms, fingers), wherein the captured fiducials are used in determining distances used for disambiguation calculations (pp. 92-94). Therefore it would have been obvious to one having ordinary skill in the art and the teachings of Balan’332 and Balan’139 before them before the effective filing of the claimed invention to combine the teachings of utilizing captured handheld device fiducials when determining discrepancies between hand positions and handheld device positions, as taught by Balan’139, with the determining the discrepancies of Balan’332. One would have been motivated to make this combination to provide more accurate discrepancy determinations, as suggested by Balan’139 (pp. 92: accurately location and uniquely and unambiguously identify).
Regarding claim 2, Balan’332 and Balan’139 disclose the system of claim 1, and Balan’332 further discloses wherein determining the pose discrepancy includes determining a position discrepancy between a position of the hand and a position of the handheld device (pp. 69, pp. 72-75: determine discrepancy between relative hand position and relative IMU position).
Regarding claim 3, Balan’332 and Balan’139 disclose the system of claim 2, and Balan’332 further discloses wherein the position discrepancy is a distance between the position of the hand and the position of the handheld device (pp. 69, pp. 72-75).
Regarding claim 4, Balan’332 and Balan’139 disclose the system of claim 1, and Balan’332 further discloses wherein determining the pose discrepancy comprises determining an orientation discrepancy between an angle of orientation of the hand and an angle of orientation of the handheld device (pp. 46: angle, pp. 60: orientation discrepancy data from IMU and hand pose sensor).
Regarding claim 5, Balan’332 and Balan’139 disclose the system of claim 1, and Balan’139 further discloses wherein the image captured by the imaging device is analyzed to determine positions of a set of keypoints of the handheld device and a set of keypoints of the hand (pp. 92: determine positions of markers on controller and hand centroids/palms/fingers), and wherein each keypoint of the set of keypoints of the handheld device corresponds to a different keypoint of the set of keypoints of the hand (pp. 92: set of markers are measured relative to centroids of the hands and fingers, pp. 55: location of the button near to thumb or finger).
Regarding claim 6, Balan’332 and Balan’139 disclose the system of claim 5, and Balan’139 further discloses wherein the computing apparatus determines the pose discrepancy using a set of distances between the positions of the set of keypoints of the handheld device and the set of keypoints of the hand (pp. 92-94: measuring distances between corresponding keypoints to aid in disambiguation).
Regarding claim 9, Balan’332 and Balan’139 disclose the system of claim 1, and Balan’332 further discloses wherein the wearable device further comprises a wearable sensor mounted on the wearable device, and wherein the wearable sensor is operable to capture wearable device data indicative of movement of the wearable device (pp. 24).
Regarding claim 10, Balan’332 and Balan’139 disclose the AR system of claim 1, and Balan’332 further discloses a belt pack (pp. 83: ex. hand-held device/pager, PDAs), wherein the computing apparatus is disposed in the belt pack (pp. 83: hardwired data link to perform computer tasks).
Regarding claim 11-14, claims 11-14 recite limitations similar to claims 1-4, respectively, and are similarly rejected.
Regarding claim 15, claim 15 recites limitations similar to claim 9 and is similarly rejected.
Regarding claim 18, Balan’332 and Balan’139 disclose the method of claim 11, and Balan’332 further discloses:
determining whether the pose discrepancy is greater than or less than a threshold (pp. 74: compare to determine if there is a lack of correlation, based on a determined confidence level, between the relative positions used determining the pose discrepancy).
Regarding claim 19, Balan’332 and Balan’139 disclose the method of claim 18, and Balan’332 further discloses wherein if it is determined that the pose discrepancy is greater than the threshold (pp. 74: confidence level indicates a lack of correlation), performing the operation to fuse the hand pose data with the handheld device pose data includes determining that the handheld device is not being held by the hand (pp. 74: if not, do not associated with being held by hand).
Regarding claim 20, Balan’332 and Balan’139 disclose the method of claim 18, and Balan’332 further discloses wherein if it is determined that the pose discrepancy is less than or equal to the threshold (pp. 74-75: confidence level indicates sufficient confidence for correlation), performing the operation to fuse the hand pose data with the handheld device pose data includes determining that the handheld device is being held by the hand (pp. 74-75: associate with being held in user hand).
Claims 7-8 and 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Balan’332 in view of Balan’139 and in further view of Chen, US 2017/0011553.
Regarding claim 7, Balan’332 and Balan’139 disclose the system of claim 1, and but fail to disclose wherein the wearable device comprises a set of wearable fiducials affixed to the wearable device , and wherein the handheld device comprises a handheld imaging device affixed to the handheld device and operable to capture a fiducial image of the wearable device containing the set of wearable fiducials.
Chen discloses methods for tracking movement of a handheld device of a wearable device, an analogous art (pp. 4). In particular, Chen discloses the wearable device comprising a set of wearable fiducials affixed to the wearable device (pp. 22) and wherein the handheld device comprises a handheld imaging device affixed to the handheld device and operable to capture a fiducial image of the wearable device containing the set of wearable fiducials (pp. 29). Therefore it would have been obvious to one having ordinary skill in the art and the teachings of Balan’332, Balan’139 and Chen before them before the effective filing of the claimed invention to combine the wearable device having a set of fiducials that are captured by an imaging device of the handheld device, as taught by Chen, with the wearable device and handheld device of Balan’332 and Balan’139. One would have been motivated to make this combination in order to provide a method of quickly pairing a new handheld device with the wearable device, as suggested by Chen (pp. 22: quick and accurate, pp. 30: pairing).
Regarding claim 8, Balan’332, Balan’139 and Chen disclose the AR system of claim 7, and Chen further discloses wherein the computing apparatus is further configured to update the handheld device pose data by performing a localization of the handheld device with respect to the wearable device using the fiducial image (pp. 42-43).
Regarding claims 16-17, claims 16-17 recite limitations similar to claims 7-8 and are similarly rejected.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Yalniz; Ismet Zeki et al.
US 10026229 B1
AUXILIARY DEVICE AS AUGMENTED REALITY PLATFORM
Hatzilias; Karol
US 20160051134 A1
GUIDANCE OF THREE-DIMENSIONAL SCANNING DEVICE
Woods; Michael Janusz et al.
US 20180300897 A1
SYSTEMS AND METHODS FOR AUGMENTED REALITY
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDREW L TANK whose telephone number is (571)270-1692. The examiner can normally be reached Monday-Thursday 9a-6p.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Ell can be reached at 571-270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW L TANK/ Primary Examiner, Art Unit 2141
Prosecution Insights