DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP §§ 706.02(l)(1) - 706.02(l)(3) for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-20 of U.S. Patent No. 12148128. Although the claims at issue are not identical, they are not patentably distinct from each other because claims 1-20 of 12148128 disclose every limitation of claims 1-20 in the instant application.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-6, 8, 10-16, 18, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over BLEYER (PGPUB: 20220028094 A1) in view of Zhao (PGPUB: 20210110615), and further in view of BISAIN (PGPUB: 20220198677 A1).
Regarding claims 1, 11, and 20. BLEYER a device comprising:
a camera (see Fig. 1, paragraph 84, the visual tracking system(s) of an HMD 100 (e.g., head tracking cameras) is/are implemented as one or more dedicated cameras. In other instances, the visual tracking system(s) is/are implemented as part of a camera system that performs other functions (e.g., as part of one or more cameras of the scanning sensor(s) 105, described hereinbelow));
a processor; and a memory storing instructions that, when executed by the processor (see Fig. 44, item 4405, 4445, and 4450), configure the device to perform operations comprising:
accessing a first image generated by the camera (see Fig. 1, paragraph 80, a visual tracking system includes one or more cameras (e.g., head tracking cameras) that capture image data of an environment);
identifying operating parameters of the camera associated with the first image (see paragraph 64, the computing system also performs an act of determining an image kernel using at least one of the following as inputs for determining the image kernel: the motion attribute, a camera exposure time, a camera field of view, or a camera angular resolution);
identifying a likelihood of motion blur of a second image following the first image, the likelihood of motion blur being based on the operating parameters associated with the first image (see Fig. 2 and 41, paragraph 291, a magnitude of a motion attribute 4105 (e.g., a measure of motion during camera exposure time) is associated with an amount of motion blur that is present or expected to be present in an image captured by a reference camera 260 and/or a match camera 215), and a motion of the camera prior to the camera generating the second image (see paragraph 271, relative motion between a camera and/or objects in the captured environment may cause multiple pixels of the image sensor of the camera to capture the same resolvable 3D point in the captured environment during the camera exposure time, which may cause blurring effects).
However, BLEYER does not expressly teaches detecting that the likelihood of motion blur of the second image exceeds a motion blur threshold.
Zhao teaches that selecting a subset of the plurality of images by filtering out images comprises filtering out images in which motion blur exceeding a threshold is detected (see paragraph 28).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify BLEYER by Zhao to obtain selecting a subset of the plurality of images by filtering out images comprises filtering out images in which motion blur exceeding a threshold is detected, in order to provide detecting that the likelihood of motion blur of the second image exceeds a motion blur threshold. Therefore, combining the elements from prior arts according to known methods and technique would yield predictable results.
However, the combination does not expressly teach in response to detecting that the likelihood of motion blur level of the second image exceeds the motion blur threshold, applying a pyramid computation algorithm to the first image
BISAIN teaches that the device can generate an image pyramid by downscaling, downsampling, subscaling, and/or subsampling the image one or more times, for instance by generating a multi-scale image pyramid and obtaining one of the rescaled and/or resampled images from the image pyramid. The image pyramid may be, for example, a Gaussian pyramid, a Laplacian pyramid, a steerable pyramid, or a combination thereof (see Fig. 5, paragraph 96); the device may apply additional image processing to the portions of the image that include depictions of dynamic objects, for example to reduce motion blur of moving objects. The device may apply facial recognition or object recognition to detect who, and what types of objects, are present in the environment in the portions of the image that include depictions of dynamic objects. For augmented or mixed reality, the device may generate virtual objects and realistically have the virtual objects be partially occluded by dynamic objects in the scene (see Fig. 5, paragraph 112).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination by BISAIN to obtain the device can generate an image pyramid by downscaling, downsampling, subscaling, and/or subsampling the image one or more times, for instance by generating a multi-scale image pyramid and obtaining one of the rescaled and/or resampled images from the image pyramid. And Zhao teaches to detect motion blur. Therefore, combining Zhao and BISAIN would provide in response to detecting that the likelihood of motion blur level of the second image exceeds the motion blur threshold, applying a pyramid computation algorithm to the first image. Therefore, combining the elements from prior arts according to known methods and technique would yield predictable results.
Regarding claims 2 and 12. The combination teaches the device of claim 11, wherein the operations further comprise:
generating a downscaled version of the first image based on the pyramid computation algorithm (see BISAIN, Fig. 5, paragraph 96, the device can generate an image pyramid by downscaling, downsampling, subscaling, and/or subsampling the image one or more times, for instance by generating a multi-scale image pyramid and obtaining one of the rescaled and/or resampled images from the image pyramid. The image pyramid may be, for example, a Gaussian pyramid, a Laplacian pyramid, a steerable pyramid, or a combination thereof); and
identifying a feature in the downscaled version of the first image (see BISAIN, Fig. 2, paragraph 80, use of a downscaled and/or downsampled version of the original image for identifying the portion of the image that includes the depiction of the dynamic object can allow the image segmentation and dynamic object masking engine 290 to identify the portion of the image that includes the depiction of the dynamic object more quickly and efficiently, since the downscaled and/or downsampled version of the original image has fewer pixels that the image segmentation and dynamic object masking engine 290 must analyze).
Regarding claims 3 and 13. The combination teaches the device of claim 12, wherein the operations further comprise:
matching feature points between the downscaled version of the first image and a downscaled version of the second image (see BISAIN, paragraph 100, The first trained NN or ML model can receive, as input, either the image or a downscaled or low-resolution variant of the image. If the image is a video frame of a video, the first trained NN or ML model can in some cases receive information regarding one or more video frames that come before or after the video frame in the video, for instance regarding a dynamic object that appears before and after the video frame in the video. The first trained NN or ML model can determine whether or not the image includes a depiction of a dynamic object, or a depiction of a particular category of dynamic object (e.g., humans, vehicles, etc.) (see Zhao, paragraph 152, the AR device may construct a map from the feature points recognized in successive images in a series of image frames captured as a user moves throughout the physical world with the AR device. Though each image frame may be taken from a different pose as the user moves, the system may adjust the orientation of the features of each successive image frame to match the orientation of the initial image frame by matching features of the successive image frames to previously captured image frames. Translations of the successive image frames so that points representing the same features will match corresponding feature points from previously collected image frames, can be used to align each successive image frame to match the orientation of previously processed image frames)); and
identifying a pose of the device based on the matched feature points (see Zhao, paragraph 152, the AR device may construct a map from the feature points recognized in successive images in a series of image frames captured as a user moves throughout the physical world with the AR device. Though each image frame may be taken from a different pose as the user moves, the system may adjust the orientation of the features of each successive image frame to match the orientation of the initial image frame by matching features of the successive image frames to previously captured image frames. Translations of the successive image frames so that points representing the same features will match corresponding feature points from previously collected image frames, can be used to align each successive image frame to match the orientation of previously processed image frames. The frames in the resulting map may have a common orientation established when the first image frame was added to the map. This map, with sets of feature points in a common frame of reference, may be used to determine the user's pose within the physical world by matching features from current image frames to the map. In some embodiments, this map may be called a tracking map).
Regarding claims 4 and 14. The combination teaches the device of claim 11,
wherein identifying the likelihood of motion blur of the second image is based on an exposure time of the first image (see Bleyer, Fig. 2 and 41, paragraph 291, a magnitude of a motion attribute 4105 (e.g., a measure of motion during camera exposure time) is associated with an amount of motion blur that is present or expected to be present in an image captured by a reference camera 260 and/or a match camera 215).
Regarding claims 5 and 15. The combination teaches the device of claim 11,
wherein the likelihood of motion blur of the second image is based on an angular velocity of the camera associated with the first image (see Bleyer, Fig. 2 and 41, paragraph 291 and 308, a magnitude of a motion attribute 4105 (e.g., a measure of motion during camera exposure time) is associated with an amount of motion blur that is present or expected to be present in an image captured by a reference camera 260 and/or a match camera 215; a system utilizes various inputs to determine the downsampling resolution, such as the motion attribute (from act 4306) and/or one or more camera attributes (e.g., camera exposure time, camera field of view, camera angular resolution, etc.). In some instances, the downsampling resolution may be based on a magnitude of the motion attribute. Furthermore, in some instances, the downsampling resolution may be based on a directional component associated with the motion attribute).
Regarding claims 6 and 16. The combination does not expressly teach the device of claim 11, wherein the likelihood of motion blur of the second image is based on a linear velocity of the camera associated with the first image.
The examiner is taking "Official Notice" that the limitation about wherein the likelihood of motion blur of the second image is based on a linear velocity of the camera associated with the first image is well known in the art.
Therefore, it would have been obvious to a person having ordinary skill in the art at the time the invention was made to have modified the combination so that wherein the likelihood of motion blur of the second image is based on a linear velocity of the camera associated with the first image would be available.
Regarding claims 8 and 18. The combination teaches the device of claim 11,
wherein the operations further comprise:
determining the motion of the camera by: retrieving inertial sensor data from an inertial sensor of the device, the inertial sensor data corresponding to the first image (see Bleyer, Fig. 1, paragraph 81, visual-inertial Simultaneous Location and Mapping (SLAM) in an HMD 100 fuses (e.g., with a pose filter) visual tracking data obtained by one or more cameras (e.g., head tracking cameras) with inertial tracking data obtained by the accelerometer(s) 155, gyroscope(s) 160, and compass(es) 165 to estimate six degree of freedom (6DOF) positioning (i.e., pose) of the HMD 100 in space and in real time); and
determining an angular velocity of the device based on the inertial sensor data (see Bleyer, Fig. 1, paragraph 79, the accelerometer(s) 155, gyroscope(s) 160, and compass(es) 165 are configured to measure inertial tracking data. Specifically, the accelerometer(s) 155 is/are configured to measure acceleration, the gyroscope(s) 160 is/are configured to measure angular velocity data, and the compass(es) 165 is/are configured to measure heading data),
wherein the motion blur level is based on the operating parameters (see Bleyer, paragraph 295, an amount of motion blur present in images captured by a camera may depend on one or more camera attributes 4140. For example, when capturing the same environment under the same motion conditions, a camera with a longer camera exposure time 4145 may capture a greater degree of motion blur than a camera with a shorter camera exposure time 4145) and the angular velocity of the device without analyzing a content of the first image (see Bleyer, paragraph 295, when capturing the same environment under the same motion conditions, a camera with a higher camera angular resolution 4155 and/or a smaller camera field of view 4150 may capture a greater degree of motion blur than a camera with a lower camera angular resolution 4155 and/or a larger camera field of view 4150. Accordingly, in some instances, a system utilizes one or more camera attributes 4140 (e.g., camera exposure time 4145, camera field of view 4150, camera angular resolution 4155, and/or others indicated by the ellipsis 4160) in addition to one or more motion attributes 4105 as inputs for determining downsampling resolution(s) 4035 (or for determining other implementations of an image kernel 4020)).
Regarding claim 10. The combination teaches the method of claim 1, wherein the operating parameters comprise a combination of an exposure time of the camera (see Bleyer, paragraph 271, motion blur refers to blurring artifacts present in a frame captured by a camera when relative motion between the camera and the captured environment causes at least a portion of the captured environment to shift during the camera exposure time. Movement of objects in the captured environment and/or movement of the camera may cause motion blur. For example, relative motion between a camera and/or objects in the captured environment may cause multiple pixels of the image sensor of the camera to capture the same resolvable 3D point in the captured environment during the camera exposure time, which may cause blurring effects), a field of view of the camera, an ISO value of the camera , and an image resolution (see Leyer, paragraph 233, it should be noted that the images captured by the reference camera 260 and the match camera 215 may have the same, or different, angular resolution, depending on the combination of field of view and image sensor resolution for the different cameras) (see BISAIN, 58, Based on this exposure setting, the exposure control mechanism 125A can control a size of the aperture (e.g., aperture size or f/stop), a duration of time for which the aperture is open (e.g., exposure time or shutter speed), a sensitivity of the image sensor 130 (e.g., ISO speed or film speed), analog gain applied by the image sensor 130, or any combination thereof. The exposure setting may be referred to as an image capture setting and/or an image processing setting).
Claim(s) 9 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over BLEYER (PGPUB: 20220028094 A1) in view of Zhao (PGPUB: 20210110615), in view of BISAIN (PGPUB: 20220198677 A1), and further in view of GUPTE (PGPUB: 20220245832 A).
Regarding claim 9. The combination teaches the method of claim 1, wherein the likelihood of motion blur is based on the operating parameters (see Bleyer, paragraph 271, motion blur refers to blurring artifacts present in a frame captured by a camera when relative motion between the camera and the captured environment causes at least a portion of the captured environment to shift during the camera exposure time. Movement of objects in the captured environment and/or movement of the camera may cause motion blur. For example, relative motion between a camera and/or objects in the captured environment may cause multiple pixels of the image sensor of the camera to capture the same resolvable 3D point in the captured environment during the camera exposure time, which may cause blurring effects).
However, the VIO data of the device without analyzing a content of the first image.
GUPTE teaches that the VIO tracker 315 can detect the dynamic objects using facial detection, facial recognition, facial tracking, object detection, object recognition, object tracking, or a combination thereof. The VIO tracker 315 can detect the dynamic objects using one or more artificial intelligence algorithms, one or more trailed machine learning models, one or more trained neural networks (see Fig. 3, paragraph 94).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination by GUPTE to obtain the VIO tracker 315 can detect the dynamic objects using facial detection, facial recognition, facial tracking, object detection, object recognition, object tracking, in order to provide VIO data of the device without analyzing a content of the first image. Therefore, combining the elements from prior arts according to known methods and technique would yield predictable results.
Regarding claims 19. The combination teaches the device of claim 11,
wherein the likelihood of motion blur is based on the operating parameters and VIO data of the device without analyzing a content of the first image (see claim 9 above),
wherein the operating parameters comprise a combination of an exposure time of the camera (see Bleyer, paragraph 271, motion blur refers to blurring artifacts present in a frame captured by a camera when relative motion between the camera and the captured environment causes at least a portion of the captured environment to shift during the camera exposure time. Movement of objects in the captured environment and/or movement of the camera may cause motion blur. For example, relative motion between a camera and/or objects in the captured environment may cause multiple pixels of the image sensor of the camera to capture the same resolvable 3D point in the captured environment during the camera exposure time, which may cause blurring effects), a field of view of the camera, an ISO value of the camera, and an image resolution (see Leyer, paragraph 233, it should be noted that the images captured by the reference camera 260 and the match camera 215 may have the same, or different, angular resolution, depending on the combination of field of view and image sensor resolution for the different cameras) (see BISAIN, 58, Based on this exposure setting, the exposure control mechanism 125A can control a size of the aperture (e.g., aperture size or f/stop), a duration of time for which the aperture is open (e.g., exposure time or shutter speed), a sensitivity of the image sensor 130 (e.g., ISO speed or film speed), analog gain applied by the image sensor 130, or any combination thereof. The exposure setting may be referred to as an image capture setting and/or an image processing setting).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XIN JIA whose telephone number is (571)270-5536. The examiner can normally be reached 9:00 am-7:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at (571)272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/XIN JIA/Primary Examiner, Art Unit 2663
Prosecution Insights