Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Zhu et al. (US 20190101758, hereinafter Zhu)
Regarding claim 1, “A method comprising: obtaining, using multiple imaging sensors of an electronic device, a left image frame and a right image frame forming a stereo pair of image frames” Zhu teaches (¶0128) a method for reconstructing the center-line perspective of the image captured by the stereo camera pair; (¶0129, Fig. 7, and ¶0084) stereo cameras (left and right camera) are used to capture raw images of the surrounding environment (act 1620).
As to “identifying, using at least one processing device of the electronic device, extrinsic parameters associated with relative positions and orientations of the imaging sensors” Zhu teaches (Figs. 6-7 and ¶0068, see also ¶0066-¶0075, ¶0129, ¶0132) To capture as much of the surrounding environment as possible, the left camera 605 and the right camera 615 are positioned apart, at a preselected distance 625 from each other, and are angled away from each other. This distance 625 may be any distance. Commonly, however, the distance 625 is usually at least 7 centimeters (cm). It will be appreciated that the distance 625 may more than 7 cm (e.g., 7.5 cm, 8.0 cm, 8.5 cm, 9.0 cm, 9.5 cm, 10.0 cm, 10.5 cm, 11.0 cm, 11.5, cm, 12.0 cm, 12.5 cm, etc.), or less than 7 cm. This distance between the stereo camera pair (i.e. left camera 605 and right camera 615) is the camera baseline (i.e. the distance 625) of at least 7 cm.
As to “performing, using the at least one processing device, an online stereo rectification of the stereo pair of image frames based on the extrinsic parameters such that epipolar lines of the left and right image frames are horizontally aligned to generate a rectified stereo pair of image frames” Zhu teaches (¶0085-¶0087, ¶0131) To correct for these distortions, the embodiments perform epipolar transforms 905 to the images captured by the left and right cameras. The epipolar transform 905 alters/re-aligns the center-line perspective of an image captured by the left camera so that it is parallel to the center-line perspective of an image captured by the right camera; (¶0130, ¶0080) camera distortion corrections are applied/performed (act 1630) to create two “corrected” images (i.e. a corrected left image and a corrected right image). This is performed by the distortion component 410 of the computer system 400 shown in FIG. 4. In some instances, this correction corrects a lens distortion. Additionally, this correction corrects other distortions associated with the cameras as described previously in relation to FIG. 8; (¶0132) a depth map is generated which maps the distances between the stereo camera pair and objects in the surrounding environment (act 1650); (¶0041, ¶0046) distributed system.
As to “and rendering, using the at least one processing device, one or more images for display based on the rectified stereo pair of image frames.” Zhu teaches (¶0133) the left and right passthrough visualizations are generated (act 1660) by reprojecting the two transformed images using a result obtained from the depth map; (¶0141) Finally, the two passthrough visualizations are displayed using the head-mounted device (act 1840), such as can be performed with the reprojection component 425 and the display devices rendering the passthrough visualizations.
Regarding claim 2, “The method of Claim 1, wherein identifying the extrinsic parameters comprises: identifying a common image feature in the left and right image frames; extracting left and right image features associated with the common image feature from the left and right image frames; determining if the left and right image features match; in response to a determination that the left and right image features match, obtaining a feature correspondence between the left and right image frames; optimizing an energy function associated with the feature correspondence; and identifying the extrinsic parameters based on the optimized energy function.” Zhu teaches (¶0115) generating the depth map is performed by identifying all of the corresponding pixels between the transformed left and right images. Once the corresponding pixels of the two images are all identified, then the embodiments calculate the displacement between the coordinates of the pixels in the transformed left image and the pixels in the transformed right image. This displacement is referred to as a “disparity” and is represented by the “d” in the third equation which is produced below. Relatedly, the “u.sub.l” and “u.sub.r” variables represent the x-coordinates (i.e. horizontal coordinates) of each pixel in the transformed left image and its corresponding pixel in the transformed right image. Accordingly, “d” is the difference in x-coordinates between corresponding pixels in the two images (i.e. the transformed left image and the transformed right image). In other words, d is the computed disparity at u.sub.l (i.e. “d” represents individualized portions of the depth map); (¶0004, ¶0038, ¶0082-¶0083) for optimizing passthrough visualizations.
Regarding claim 3, “The method of Claim 2, wherein performing the online stereo rectification of the stereo pair of image frames comprises: identifying first correspondence feature points using the extrinsic parameters; identifying second correspondence feature points using depth data associated with the stereo pair of image frames; determining that a difference between the first and second correspondence feature points is greater than a threshold; refining the extrinsic parameters by further optimizing the energy function based on the difference; and performing online stereo rectification of the rectified stereo pair of image frames based on the refined extrinsic parameters.” Zhu teaches (¶0115) generating the depth map is performed by identifying all of the corresponding pixels between the transformed left and right images. Once the corresponding pixels of the two images are all identified, then the embodiments calculate the displacement between the coordinates of the pixels in the transformed left image and the pixels in the transformed right image. This displacement is referred to as a “disparity” and is represented by the “d” in the third equation which is produced below. Relatedly, the “u.sub.l” and “u.sub.r” variables represent the x-coordinates (i.e. horizontal coordinates) of each pixel in the transformed left image and its corresponding pixel in the transformed right image. Accordingly, “d” is the difference in x-coordinates between corresponding pixels in the two images (i.e. the transformed left image and the transformed right image). In other words, d is the computed disparity at u.sub.l (i.e. “d” represents individualized portions of the depth map); (¶0098-¶0100) “Similar,” in this connotation, means that the coordinate values of the neighboring pixels satisfy a particular threshold; (¶0004, ¶0038, ¶0082-¶0083) for optimizing passthrough visualizations; (¶0101, ¶0141) After generating the depth map, the left and right images, which were previously transformed and corrected, are reprojected so that a center-line perspective of the left image aligns with the user's left pupil and a center-line perspective of the right image aligns with the user's right pupil.
Regarding claim 4, “The method of Claim 1, wherein performing the online stereo rectification comprises performing viewpoint matching by transforming the rectified stereo pair of image frames to match one or more user eye viewpoints and generate one or more viewpoint matched frames.” Zhu teaches (¶0104) Because the baseline of the stereo cameras (i.e. the distance between the left camera and the right camera) is much larger than the baseline of a human's eyes (i.e. the interpupil distance), the resulting images (after performing the five operations discussed above) will still look distorted to the user. As a result, it is necessary to “reproject” the images to reflect a perspective that matches the perspective of the user, which user perspective is influenced by the user's interpupil distance. Accordingly, the embodiments reproject the transformed left and right images.
Regarding claim 5, “The method of Claim 1, wherein the extrinsic parameters comprise a rotation matrix and a translation vector.” Zhu teaches (¶0087) To perform the epipolar transforms 905, the embodiments perform one or more rotational transforms, translation transforms, and/or scaling transforms.
Regarding claim 6, “The method of Claim 1, further comprising: applying a transformation to the rectified stereo pair of image frames in order to generate one or more transformed image frames; wherein rendering the one or more images for display comprises rendering the one or more transformed image frames.” Zhu teaches (¶0133) the left and right passthrough visualizations are generated (act 1660) by reprojecting the two transformed images using a result obtained from the depth map; (¶0141) Finally, the two passthrough visualizations are displayed using the head-mounted device (act 1840), such as can be performed with the reprojection component 425 and the display devices rendering the passthrough visualizations.
Regarding claim 7, “The method of Claim1, wherein the online stereo rectification is performed automatically based on the extrinsic parameters or based on a user request.” Zhu teaches (¶0037) The disclosed embodiments, in contrast, not only perform corrections to accommodate camera distortions but they also reconstruct/alter a perspective captured by a camera image so that the captured perspective matches the user's own unique perspective; (¶0043) implemented using software objects, routines, or methods that may be executed on the computing system 100; (¶0049) The graphics rendering engine 115 is configured, with the processor(s) 105, to render one or more virtual objects within the scene. As a result, the virtual objects accurately move in response to a movement of the user and/or in response to user input as the user interacts within the virtual scene.
Regarding claim 8, its rejection is similar to claim 1.
Regarding claim 9, its rejection is similar to claim 2.
Regarding claim 10, its rejection is similar to claim 3.
Regarding claim 11, its rejection is similar to claim 4.
Regarding claim 12, its rejection is similar to claim 5.
Regarding claim 13, its rejection is similar to claim 6.
Regarding claim 14, its rejection is similar to claim 7.
Regarding claim 15, its rejection is similar to claim 1.
Regarding claim 16, its rejection is similar to claim 2.
Regarding claim 17, its rejection is similar to claim 3.
Regarding claim 18, its rejection is similar to claim 4.
Regarding claim 19, its rejection is similar to claim 5.
Regarding claim 20, its rejection is similar to claim 7.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
You et al. (US 10650602) – (Claim 1) A three-dimensional information augmented video see-through display device, comprising: a camera interface module which obtains at least two real images from at least two camera modules; a rectification module which performs rectification on the at least two real images; a lens distortion correction module which corrects at least two composite images obtained by combining a virtual image to the at least two real images, based on a lens distortion compensation value indicating a value for compensating for a distortion of a wide angle lens for the at least two real images; and an image generation module which performs side-by-side image processing on the at least two composite images to generate a three-dimensional image for virtually reality VR or augmented reality AR, wherein the rectification module performs rectification to change the two real images which are photographed from the at least two camera modules, based on adjusting an epipolar geometry through a field programable gate array FPGA module to make an optical axis and a reference point be parallel and intersect at infinity.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to FRANK J JOHNSON whose telephone number is (571)272-9629. The examiner can normally be reached 9:00AM-5:00PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Brian T. Pendleton can be reached on 571-272-7527. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Frank Johnson/Primary Examiner, Art Unit 2425
Prosecution Insights