DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 11/14/2024, 05/15/2025 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are considered by examiner.
Claim Objections
Claim 10 is objected to because of the following informalities: The preamble recites “The device of any of claim 1,” and is interpreted as a typographical oversight such that the preamble should recite “The device of any of claim 1,”. Appropriate correction is required.
Claim Rejections - 35 USC § 112(b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 1, 11 each claim a “second first image” in the limitation “generating a second de-warped image from second first image” and it is therefore unclear if the second de-warped image is generated from the first image or the second image. The limitation also lacks antecedent basis. Thus, Applicant has failed to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
For purposes of examination, the limitation is interpreted as “generating a second de-warped image from the second image”.
Claims 2-10 are rejected as dependent on claim 1.
Claims 12-20 are rejected as dependent on claim 11.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Sadi et al (US 2016/0088280).
Regarding Claim 1, Sadi et al teach a device (client computing device 150 of system 100; Fig 1, 2, 33 and ¶ [0046], [0184]) comprising:
a first camera (left camera L of cameras 112 (784L Fig 33); Fig 1, 2, 33 and ¶ [0047], [0051]-[0052], [0184]);
a second camera (right camera R of cameras 112 (784L Fig 33); Fig 1, 2, 33 and ¶ [0047], [0051]-[0052], [0184]);
a set of displays (left display 782L and right display 782R; Fig 1, 24, 33 and ¶ [0155]);
a memory (memory of server 140; Fig 1, 18-20, 33 and ¶ [0142], [0144]-[0146]); and
one or more processors operatively coupled to the memory (processor 786 processes images from server 140 received via communication module 790 (noted front-end processor 120 and server 140 also contains processor to process video stream data); Fig 1, 24, 33 and ¶ [0144], [0150]-[0155], [0185]), wherein the one or more processors are configured to execute instructions (server 140 processor and client device 150 processor execute image processing of video stream; Fig 1, 24, 33 and ¶ [0150]-[0155]) causing the one or more processors to:
capture a first set of image pairs of a scene using the first camera and second camera (the left camera L and right camera R capture images and each corresponding left and right image set is a stereoscopic pair 200; Fig 2, 33 and ¶ [0051], [0184]), wherein each image pair of the first set of image pairs (first image pair of stereoscopic pair 200; Fig 2 and ¶ [0051], [0184]) comprises: a first image captured by the first camera (the left camera L captures a left image of the camera image stereoscopic pair 200; Fig 2, 33 and ¶ [0051], [0184]); and a second image captured by the second camera (the right camera R captures a right image of the camera image stereoscopic pair 200; Fig 2, 33 and ¶ [0051], [0184]);
generate a first set of transformed image pairs from the first set of image pairs (correcting the stereoscopic image pair 200; Fig 1, 2 and ¶ [0086]-[0088]), wherein generating the first set of transformed image pairs comprises, for each image pair of the first set of image pairs (the camera stereoscopic image pair 200 may be (corrected) during image capture or post-capture correction for each image of the pair of images (using homography transform techniques); Fig 1, 2, 12, 33, 34 and ¶ [0088], [0118]-[0120], [0184]-[0186]):
generating a first perspective-corrected image from the first image based on a difference between a point-of-view of the first camera and a point-of-view of a user (the left image L is corrected for alignment based on an orientation error or offset correction with respect to adjacent camera image, with the images taken from the perspective of the user; Fig 1, 2, 12, 33, 34 and ¶ [0088], [0118]-[0120], [0184]-[0186]); and
generating a second perspective-corrected image from the second image based on a difference between a point-of-view of the second camera and the point-of-view of the user (the right image R is corrected for alignment based on an orientation error or offset correction with respect to adjacent camera image, with the images taken from the perspective of the user; Fig 1, 2, 12, 33, 34 and ¶ [0088], [0118]-[0120], [0184]-[0186]);
display, on the set of displays, the first set of transformed image pairs (the user may view the image of the scene on the left display 782L and right display 782R; Fig 1, 24, 33, 34 and ¶ [0155], [0184], [0186]);
receive a capture request (the system 150 includes microphone 794 in which the user may provide a verbal command, such as to capture one or more images of the user’s physical environment with cameras 784L, 784R; Fig 1, 33 and ¶ [0184]-[0185]);
select, in response to receiving the capture request, a second set of image pairs from the first set of image pairs (the left camera L and right camera R capture images as a stereoscopic pair 200 and a plurality of images may be captured (thus a second set of image pairs); Fig 1, 2, 33 and ¶ [0051], [0184]);
generate a second set of transformed image pairs from the second set of image pairs (correcting the stereoscopic image pair 200, such that the images may contain images located close to camera 110 and causing warp and may be transformed with a dewarp transformation; Fig 1, 2, 9, 10 and ¶ [0080], [0105]), wherein generating the second set of transformed image pairs comprises, for each image pair of the first set of image pairs (the camera stereoscopic image pair 200 may be (corrected) during image capture or post-capture correction for each image of the pair of images (using dewarp transform techniques); Fig 1, 2, 9, 10 and ¶ [0088], [0105], [0116], [0132]):
generating a first de-warped image from the first image (images (left camera L image, I-6) may be dewarped based on a global transform and a dewarp transformation to generate a dewarped (left camera) image); ¶ [0105], [0116]);
generating a second de-warped image from second first image (images (right camera R image, I-7) may be dewarped based on a global transform and a dewarp transformation to generate a dewarped (right camera) image); ¶ [0105], [0116]); and
aligning the first and second de-warped images (the stereoscopic pair 200 of images I-6, I-7 may be aligned (to be stitched) to each other based on a homography operation; ¶ [0105], [0116]); and
generate a set of output images using the second set of transformed image pairs (the output images with a successful convergence is output such that the left-eye view image is stitched with a corresponding right-eye view (stereoscopic pair) of stitched images; ¶ [0117]-[0118]).
Regarding Claim 2, Sadi et al teach the device of claim 1 (as described above), wherein generating the set of output images comprises: generating a fused stereo output image from two or more image pairs of the second set of transformed image pairs (the dewarped imaged are stitched together to output a dewarped stitched stereoscopic image output representing a real-life scene; Fig 33, 34 and ¶ [0117]-[0118], [0185]-[0186]).
Regarding Claim 3, Sadi et al teach the device of claim 1 (as described above), wherein the set of output images comprises a stereo video formed from the second set of transformed image pairs (the output image data may be stereoscopic videos of the user’s physical environment and incorporate image processing to dewarp the image data; Fig 33, 34 and ¶ [0116], [0118], [0184]).
Regarding Claim 4, Sadi et al teach the device of claim 3 (as described above), wherein generating the set of output images comprises: performing a video stabilization operation on the second set of transformed image pairs (the vertical offset between the stereoscopic pair of stitched images is stabilized based on determining a correspondence between features of the images (of the video data); ¶ [0118]-[0119]).
Regarding Claim 5, Sadi et al teach the device of claim 1 (as described above), wherein generating the set of output images comprises generating metadata associated with the set of output images (metadata of the video stream data is generated from the sub-streams produced by the client device 150, including the image processed data; ¶ [0145], [0180]).
Regarding Claim 6, Sadi et al teach the device of claim 5 (as described above), wherein the metadata comprises field of view information of at least one of the first camera or second camera (metadata of the video stream data is generated from the sub-streams produced by the client device 150 from the left camera L and right camera R in the user’s field of view; ¶ [0145], [0180]).
Regarding Claim 7, Sadi et al teach the device of claim 5 (as described above), wherein the metadata comprises pose information for the set of output images (camera 112 data includes position and orientation (pose), which may be included in the metadata of the stereoscopic video data; ¶ [0053], [0088],[0180]).
Regarding Claim 8, Sadi et al teach the device of claim 5 (as described above), wherein: the set of output images comprises a set of stereo images (the dewarped output images with a successful convergence is output such that the left-eye view image is stitched with a corresponding right-eye view (stereoscopic pair) of stitched images; ¶ [0117]-[0118]); and the metadata comprises a set of default disparity values for the set of stereo output images (the inter-camera spacing (ICS) between cameras 112 has a default distance between 6-11 cm (manufactured with ICS of 6.5 mm), based on average distance between human pupils (thereby default), for the generation of the stereoscopic image pair, which may be included metadata in the generation of the stereoscopic video data; ¶ [0057], [0088], [0180]).
Regarding Claim 9, Sadi et al teach the device of claim 8 (as described above), wherein generating metadata associated with the set of output images comprises selecting the set of default disparity values based on the scene captured by the set of output images (the ICS value is based on the approximate/average distance between pupils and used when the images are played back to a viewer; ¶ [0057], [0180]).
Regarding Claim 10, Sadi et al teach the device of claim 1 (as described above), wherein the processor is configured to add virtual content to the first set of transformed image pairs (the client system 150 may present a user a 3D scene (stereoscopic pair) with a virtual graphical object, such as a notification 930 on the display; Fig 36 and ¶ [0188]).
Regarding Claim 11, Sadi et al teach a method (method of using client computing device 150 of system 100; Fig 1, 2, 32-34 and ¶ [0046], [0183]-[0186]) comprising: steps identical to claim 1 (as described above).
Regarding Claim 12, Sadi et al teach the method of claim 11 (as described above), with further limitations claimed in parallel to claim 2 (as described above).
Regarding Claim 13, Sadi et al teach the method of claim 11 (as described above), with further limitations claimed in parallel to claim 3 (as described above).
Regarding Claim 14, Sadi et al teach the method of claim 13 (as described above), with further limitations claimed in parallel to claim 4 (as described above).
Regarding Claim 15, Sadi et al teach the method of claim 11 (as described above), with further limitations claimed in parallel to claim 5 (as described above).
Regarding Claim 16, Sadi et al teach the method of claim 15 (as described above), with further limitations claimed in parallel to claim 6 (as described above).
Regarding Claim 17, Sadi et al teach the method of claim 15 (as described above), with further limitations claimed in parallel to claim 7 (as described above).
Regarding Claim 18, Sadi et al teach the method of claim 15 (as described above), with further limitations claimed in parallel to claim 8 (as described above).
Regarding Claim 19, Sadi et al teach the method of claim 18 (as described above), with further limitations claimed in parallel to claim 9 (as described above).
Regarding Claim 20, Sadi et al teach the method of claim 11 (as described above), with further limitations claimed in parallel to claim 10 (as described above).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Menzies et al (US 2024/0406365, application 18/661,336), from the same assignee and co-inventors, claim the use of a plurality of images from a set of cameras to capture an image stream and performing an operation to generate a set of transformed images. The current application is distinct by explicitly claiming at least two cameras with dewarping and perspective correcting the image pairs, thereby claim limitations are more specific and distinct than the prior application.
Menzies et al (US 2025/0193361, application 19/060,516), from the same assignee and co-inventors, claim the use of a plurality of images from at least one camera and using a first transformation and second transformation operation to generate a stereo image. The current application is distinct by explicitly claiming at least two cameras with dewarping and perspective correcting the image pairs, thereby claim limitations are more specific and distinct than the prior application.
Zhu et al (US 2019/0101758) teach a method and system with a head0mounted device to capture stereo camera image pairs and an ICP method to align the images with respect to a center perspective as well as epipolar transforms applied to the corrected images to reduce distortions.
Gallup et al (US 2016/0353089) teach a method and system for generating spherical perspective projection images in a head mounted display and performing stereo parallax distortion correction for display in a VR environment.
Dal Mutto et al (US 2015/0195443) teach a method and system for real-time multi-camera image processing to generate video recordings used for video conferencing or live broadcast.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KATHLEEN M BROUGHTON whose telephone number is (571)270-7380. The examiner can normally be reached Monday-Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John Villecco can be reached at (571) 272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KATHLEEN M BROUGHTON/Primary Examiner, Art Unit 2661
Prosecution Insights