DETAILED ACTION
*Note in the following document:
1. Texts in italic bold format are limitations quoted either directly or conceptually from claims/descriptions disclosed in the instant application.
2. Texts in regular italic format are quoted directly from cited reference or Applicant’s arguments.
3. Texts with underlining are added by the Examiner for emphasis.
4. Texts with
5. Acronym “PHOSITA” stands for “Person Having Ordinary Skill In The Art”.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 3-4, 7 and 10-11 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 3-4, 7 and 10-11 recite the limitation "each of the camera". There is insufficient antecedent basis for this limitation in the claim. For compact prosecution purpose, the Examiner interprets the cited each of the camera as “each of the cameras”.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-2, 4-7, 8-9 and 11-15 are rejected under 35 U.S.C. 103 as being unpatentable over Li et al. (CN 113538700 A) in view of Cajigas et al. (US 2013/0120224 A1).
Regarding Claim 1, Li discloses an operation method of a virtual image display system ([n0001]: This disclosure relates to the field of computer technology, and more particularly to a calibration method and apparatus for augmented reality devices, electronic devices, and storage media), comprising:
using a plurality of cameras to perform image capturing action on a target portion of acalibration object, and obtain a plurality of image information ([n0119]: In the case that the AR glasses are equipped with at least two cameras, at least two first captured images are acquired by at least two cameras, and AprilTag is detected based on at least two first captured images);
using the cameras to obtain a plurality of position information corresponding to the target portion according to the image information ([n0065]: Through steps S121 to S123, the prediction from two-dimensional image to three-dimensional space can be realized based on the first acquired image to obtain the pose corresponding to the calibration in the real scene, and the predicted image of the calibration object can be displayed in AR glasses based on the pose. In this way, by utilizing the transformation relationship between two-dimensional and three-dimensional space, the predicted image of the calibration object can be presented simply and conveniently, reducing the amount of calculation in the calibration process and improving the convenience of the entire calibration process. [n0066]: In one possible implementation, the first acquired image may include at least two acquired images, and step S121 may include: determining the pose of the calibrated object relative to the AR device by a triangulation method based on the position of the corner points of the calibrated object in the at least two acquired images);
using a head-mounted display (HMD) to receive the position information, calculating an extrinsic parameter according to the position information, and establishing a plurality of re-projection image information according to the extrinsic parameter and the position information ([n0116]: Figure 4 shows a schematic diagram of an application scenario according to the present disclosure. As shown in Figure 4, in the application example of the present disclosure, the optical see-through AR glasses can transmit the tagged object in the real scene (i.e., the AprilTag printed in the figure) into the human eye through a semi-transmissive and semireflective lens, and reflect the virtual predicted image generated based on the AprilTag into the human eye to produce an AR effect. [n0118]: Using the camera built into the AR glasses, identify the pose of the tagged object (AprilTag) in the real scene within the field of view relative to the camera. [n0007]: determining the pose of the calibrated object relative to the AR device based on the first acquired image; predicting the target display state of the calibrated object in the AR device based on the pose, and generating a predicted image based on the target display state, wherein the target display state includes the target display position and/or target display size of the predicted image; and displaying the predicted image in the AR device. [n0104]: The prediction module 22 is used to display the predicted image corresponding to the calibrated object in the AR device based on the first acquired image. Note the pose of the tagged object is the extrinsic parameter and the virtual predicted image is interpreted as the re-projection image).
Li fails to disclose the calibration object can be a user.
However Cajigas discloses instead of using a 2D image including QR code, checkerboard or AprilTag, an image of a person can be used as a calibration object ([0062]: One or more processors of the computing system 12, or the display device system 8 or both also execute the object recognition engine 192 to identify real objects in image data captured by the environment facing cameras 113. As in other image processing applications, a person can be a type of object. For example, the object recognition engine 192 may implement pattern recognition based on structure data 200 to detect particular objects (e.g., soccer balls, cars, or landmarks) as well as a skeleton indicating a human. The object recognition engine 192 may also include facial recognition software which is used to detect the face of a particular person).
Therefore it would have been obvious to a PHOSITA before the effective filing date to incorporate the teaching of Cajigas into that of Li and to include the limitation of using a plurality of cameras to perform image capturing action on a target portion of a user in order to use the user as a calibration object when there was no available calibration image with a valid QR code or special pattern.
Li modified by Cajigas does not explicitly recite using the HMD to obtain a plurality of error information according to a prediction image information and the re-projection image information; and using the HMD to display each of the error information, and calibrating the extrinsic parameter to generate a calibrated extrinsic parameter according to each of the error information.
However Li discloses aligning the predicted image with the AprilTag in the real scene seen by the user ([n0105-n0106]: The second image acquisition module 23 is used to acquire a second image of the calibration object in the real scene in response to a movement operation of the AR device, provided that the predicted image matches the calibration object. The calibration module 24 is used to calibrate the AR device based on the second acquired image. [n0124]: The user moves their head to align the drawn predicted image with the AprilTag in the real scene seen by the user). Li further discloses In some possible implementations, when the predicted image matches the calibrated object, the AR device can also output a prompt for the user to know. The form of the prompt can be flexibly selected according to the actual situation, such as emitting a prompt sound, displaying prompt text on the display interface, or displaying a prompt image box, or one or more other methods ([n0087]).
A skilled person would have recognized that the relative pose of the tagged object (which is the extrinsic parameter) changes as the user moves his head. Therefore as the user moves his head, the user is adjusting the relative pose of the tagged object. In addition, in order to align the two images, the user must know the difference between the two images. The difference it the error information. By displaying the error information, it would give a user a visual guidance during calibration. Therefore Li indirectly teaches or suggests using the HMD to obtain a plurality of error information according to a prediction image information and the re-projection image information; and using the HMD to display each of the error information, and calibrating the extrinsic parameter to generate a calibrated extrinsic parameter according to each of the error information since in order to efficiently assist the user to align the predicted mage (the re-projection image information as claimed in instant application) with the second image of the calibration object in the real scene (which is the predicted image information as claimed in instant application).
Regarding Claim 2, Li teaches or suggests receiving a user input to adjust a position or orientation of the target portion corresponding to each of the error information ([n0124]: Step S34: The user moves their head to align the drawn predicted image with the AprilTag in the real scene seen by the user, and acquires a second captured image of the AprilTag in the aligned state. [n0085]: In this process, the calibration objects in the predicted image are aligned with the calibration objects in the real scene. This can be achieved by drawing the calibration objects within the predicted image displayed on the AR glasses and aligning them with the images of the calibration objects in the real scene, resulting in a 6-DOF overlap. This matching method can achieve high accuracy and improve the precision of the calibration).
Regarding Claim 4, Li further teaches or suggests wherein the step of establishing the re-projection image information according to the extrinsic parameter and the position information comprises: triangulating the position information and the location information of each of the camera based on the extrinsic parameter to establish each of the re-projection image information ([n0008]: In one possible implementation, the first acquired image includes at least two acquired images; determining the pose of the calibrated object relative to the AR device based on the first acquired image includes: determining the pose of the calibrated object relative to the AR device by triangulation based on the position of the corner points of the calibrated object in the at least two acquired images).
Regarding Claim 5, Li modified by Cajigas further teaches or suggests using the HMD to establish the prediction image information of the user; and calculating a position deviation between the prediction image information and each of the re-projection image information to generate each of the error information (Li Fig.4 and Li [n0055]: As described in the above-disclosed embodiments, the position of the calibrated object in the real scene can be predicted based on the first acquired image. Since the position predicted based on the first acquired image is not accurate, while the second acquired image is an image of the calibrated object acquired at the accurate predicted position, the positional transformation relationship between the AR glasses and the real scene can be corrected based on the deviation between the second acquired image and the first acquired image, thereby realizing the calibration of the AR glasses. Note Li teaches more than one images need to be aligned, see Li [n0002]: For example, AR glasses can project virtual images to each of the user's eyes to achieve the fusion of virtual images and real scenes. In order to align the virtual images displayed in AR glasses with the real scene, the AR glasses need to be calibrated. Also notice Cajigas teaches a calibration object can be any part of a person, see Cajigas [0062]).
Regarding Claim 6, Li further teaches or suggests using the HMD to project the prediction image information and one of the re-projection image information; and using the HMD to guide the user to adjust the position or orientation of the target portion so that the prediction image information overlaps with each of the projected re-projection image information ([n0075]: In cases where at least one image contains multiple images, the method for determining pose based on a single acquired image, as proposed in the above-disclosed embodiments, can be used to process multiple images separately to obtain multiple pose solution results, and then perform average calculation or weighted average calculation based on these poses to obtain the pose of the calibrated object in the real scene relative to the AR glasses. A pose inherently includes rotation and translation).
Regarding Claim 7, Li teaches or suggests wherein the extrinsic parameter comprises rotation and translation information between a coordinate system of each of the camera and a coordinate system of the HMD ([n0007]: determining the pose of the calibrated object relative to the AR device based on the first acquired image; predicting the target display state of the calibrated object in the AR device based on the pose, and generating a predicted image based on the target display state, wherein the target display state includes the target display position and/or target display size of the predicted image; and displaying the predicted image in the AR device. [0008]: In one possible implementation, the first acquired image includes at least two acquired images; determining the pose of the calibrated object relative to the AR device based on the first acquired image includes: determining the pose of the calibrated object relative to the AR device by triangulation based on the position of the corner points of the calibrated object in the at least two acquired images).
Regarding Claims 8-9, 11-13 and 15, Claims 8-9, 11-13 and 15 are similar to Claims 1-2, 4-6 and 7 except in the format of system. Li also discloses a plurality cameras and a head-mounted display (HMD), coupled to the cameras ([n0119]: In the case that the AR glasses are equipped with at least two cameras, at least two first captured images are acquired by at least two cameras, and AprilTag is detected based on at least two first captured images). Therefore the same reason(s) for rejection is/are applied to 1-2, 4-6 and 7 are also applied to Claims 8-9, 11 and 15.
Regarding Claim 14, Li discloses wherein the number of the cameras is greater than or equal to 2 ([n0119]: In the case that the AR glasses are equipped with at least two cameras, at least two first captured images are acquired by at least two cameras, and AprilTag is detected based on at least two first captured images).
Claims 3 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Li et al. (CN 113538700 A) and Cajigas et al. (US 2013/0120224 A1) as applied to Claims 1 and 8 above and further in view of and Kim (US 2024/0078773 A1).
Regarding Claim 3, Li modified by Cajigas teaches or suggests using each of the camera to establish posture information of the user according to a posture prediction algorithm; using each of the camera to transmit each of the corresponding posture information to the HMD (Li [n0046]: In some possible implementations, the calibration object may include a two-dimensional calibration object. The two-dimensional calibration object can be any calibration object in two-dimensional form. The implementation form of the two-dimensional calibration object can be flexibly determined according to the actual situation and is not limited to the disclosed embodiments below. In some possible implementations, the two-dimensional calibration object may include images such as QR codes and/or checkerboard patterns for visual calibration. In one example, the two-dimensional calibration object may be an AprilTag tag for visual calibration ([n0046]. Note Cajigas teaches a calibration object can be any part of a person, see Cajigas [0062]).
But Li modified by Cajigas explicitly disclose using the HMD to display a full-body three-dimensional image of the user according to each of the posture information.
Kim discloses using each of the camera to establish posture information of the user according to a posture prediction algorithm; using each of the camera to transmit each of the corresponding posture information to the HMD; and using the HMD to display a full-body three-dimensional image of the user according to each of the posture information had been known to a PHOSITA before the effective filing date of the claimed invention (Fig.6 and Fig.12. Also see [0110]-[0113]: In operation 610, the processor may predict a whole-body 3D pose representing a pose of a person based on a coordinate system of a camera capturing an image, using a pose prediction model for predicting a pose of a person. The processor may predict positions and poses of joints of parts of a body excluding a head. The processor may predict a pose of the head. The processor may predict the whole-body 3D pose based on the coordinate system of the camera capturing the image by combining the predicted positions and poses of the joints of the parts excluding the head, and the predicted pose of the head. ... In operation 620, the processor may generate part-specific normalized images based on the whole-body 3D pose. [0160]-[0163]: In operation 1110, the processor may update a whole-body 3D pose by integrating part-specific appearance control parameters that are included in parts-specific control parameters and control an appearance of a 3D model of a person. The processor may transform each piece of part-specific pose information included in the part-specific appearance control parameters into a coordinate system of a whole-body normalization camera among part-specific normalization cameras that capture normalized images of respective parts. The processor may update a whole-body 3D pose in the coordinate system of the whole-body normalization camera by integrating pieces of pose information transformed into the coordinate system of the whole-body normalization camera. In operation 1120, the processor may update the canonical 3D model using the updated whole-body 3D pose and the part-specific control parameters).
Therefore it would have been obvious to a PHOSITA before the effective filing date to incorporate the teaching of Kim into that of Li as modified and to include the limitation of using each of the camera to establish posture information of the user according to a posture prediction algorithm; using each of the camera to transmit each of the corresponding posture information to the HMD; and using the HMD to display a full-body three-dimensional image of the user according to each of the posture information in order to apply the proven working technology as disclosed by Kim therefore reducing calibration implantation cost and time.
Regarding Claim 10, Claim 10 is similar to Claim 3 except in the format of system. Therefore the same reason(s) for rejection is applied to 3 is also applied to Claim 10.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YINGCHUN HE whose telephone number is (571)270-7218. The examiner can normally be reached M-F 8:00-5:00 MT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao M Wu can be reached at 571-272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YINGCHUN HE/Primary Examiner, Art Unit 2613