DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on November 22, 2024, and March 26, 2025, are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the Examiner.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Additionally, the disclosure is objected to because it refers to labels in the Figures which do not exist. On page 3, lines 14-18, of the specification, a front-facing camera 122 and an antenna 124 are referred to, but these labels are absent from Fig. 1. Appropriate correction is required.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-11 and 13-20 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Donner et al. (US 2017/0148206 A1), hereafter Donner.
Regarding claim 1, Donner teaches an image processing method (Figs. 1 and 4 show an overview of the image processing method.) comprising:
obtaining a first velocity of one or more parts of a second image frame between a first image frame and the second image frame, wherein the first image frame and the second image frame are frames for display at a head mounted display, HMD ([0016] “As each input texture 122 is rendered, the display system 100 performs a motion analysis process 124 to determine a velocity, Vp, for each pixel in the input texture 122.” Step 124 of Fig. 1 includes determining the velocity field for pixels between an input texture (first image frame) and a display texture (second image frame). In the Background section, Donner explains that the term ‘textures’ can also refer to ‘frames.’ [0003] “HMD-based VR and AR systems display 3D imagery as a sequence of display textures (or ‘frames’),” Thus, the ‘input texture’ and the ‘display texture’ taught by Donner are equivalent to a ‘first image frame’ and a ‘second image frame’ of the claimed invention.);
receiving motion data indicative of motion of the HMD between a display time of the first image frame and a display time of the second image frame (Step 130 of Fig. 1 includes determining the HMD velocity field which describes movement of the HMD in pixel velocities. [0018] “In parallel with the motion analysis process 124, the display system 100 performs a HMD motion conversion process 130 whereby the display system 100 samples one or more motion-based sensors (e.g., gyroscope, accelerometer, magnetometer, etc.) to determine a current motion vector, denoted “MVHMD”, of the HMD device 102 relative to the world space.” [0019] “After determining the current motion vector MVHMD, the HMD motion conversion process 130 converts the current motion vector MVHMD to a per-pixel velocity representation of the motion of the HMD device 102 in the screen space, resulting in an HMD velocity field 132,”);
adjusting the first velocity based on the received motion data to determine an adjusted velocity of the one or more parts of the second image frame between the first image frame and the second image frame (Step 134 of Fig. 1 includes adjusting the pixel velocity field (first velocity) by combining it with the HMD velocity field (motion data) to generate a net velocity field (adjusted first velocity). [0020] “The display system 100 then performs a velocity combination process 134 whereby the pixel velocity field 128 and the HMD velocity field 132 are combined to generate a net velocity field 136. The net velocity field 136 represents the net relative velocity of each pixel when considering both the velocity of the pixel in the rendered imagery and the velocity of the HMD device 102.”); and
performing one or more image processing operations on the second image frame based on the adjusted velocity of the one or more parts of the second image frame (Referring to Fig. 1, see the original position 146 of an object in the input texture (first image frame) and the updated position 144 of that object in the display texture (second image frame). The net velocity field (adjusted first velocity) is used for performing advection to determine and display the updated position of the object in the display texture. See 414-422 of Fig. 4 and [0039-0043].).
Regarding claim 2, Donner teaches the method of claim 1, wherein the one or more image processing operations comprise one or more selected from the list consisting of: motion-blurring, temporal anti-aliasing, and depth of field processing (In 414-422 of Fig. 4 and [0039-0043], Donner teaches using the net velocity field (adjusted first velocity) to perform advection. In [0043], Donner teaches using the disocclusion module for applying motion blur to the display texture based on the advection process.).
Regarding claim 3, Donner teaches the method of claim 1, further comprising outputting the processed second image frame to the HMD for display (Step 140 of Fig. 1 shows the display texture, which is equivalent to the second image frame (as discussed above in the rejection to claim 1). The display texture shows the processed frame where advection has been performed to compensate for pixel velocities and the HMD movement.).
Regarding claim 4, Donner teaches the method of claim 1, wherein adjusting the first velocity based on the received motion data comprises subtracting a function of the motion data from the first velocity (Step 134 of Fig. 1 includes adjusting the pixel velocity field (first velocity) by combining it with the HMD velocity field (motion data) to generate a net velocity field (adjusted first velocity). Thus, the net velocity field is obtained by the addition of the pixel velocities and the HMD velocities. [0020] “Vnet=Vp+Vh”. This function would also encompass subtraction depending on the direction of the motion of the HMD and the pixel velocities. Furthermore, in an embodiment, Donner teaches that the velocity of the movement of the HMD can be subtracted from the velocity of a fragment to more accurately account for the depth of each fragment when converting the HMD movement into the HMD velocity field. [0037] “A more accurate (and more computationally expensive) transformation includes accounting for the depth of every fragment (essentially treating each fragment as a 3D point), and subtracting the HMD pose velocity vector from the per fragment velocity vector (assuming both are in virtual world space) to get a HMD-relative fragment velocity vector that could then be transformed into a screen space velocity vector. This approach thus adds the velocities in virtual world space and then transforms to screen space;”).
Regarding claim 5, Donner teaches the method of claim 4, wherein, for each image part of the one or more parts of the second image frame, the function of the motion data is further based on a position of the image part (Donner teaches obtaining motion data and converting it to a HMD velocity field. [0020] “The net velocity field 136 represents the net relative velocity of each pixel when considering both the velocity of the pixel in the rendered imagery and the velocity of the HMD device 102.” Here, the term “parts” is being interpreted as pixels, and the pixel velocity field (first velocity data) can be adjusted by the HMD velocity field (motion data). Additionally, Donner teaches performing the method for a virtual object in the images [0021]. In Fig. 1, the pixel velocity field for a virtual object 142 is determined, and the position of virtual object 144 is corrected in the display texture (second image frame) by using the net velocity field (first velocity adjusted by motion data).).
Regarding claim 6, Donner teaches the method of claim 1, wherein obtaining the first velocity comprises retrieving the first velocity from a memory buffer ([0035] “At block 406 the motion analysis module 302 buffers the per-pixel velocities Vp for the pixels constituting the input texture 322 as a pixel velocity field 328 (one embodiment of the pixel velocity field 128, FIG. 1) in the deep frame buffer 232 in association with the color field 323 for the input texture 322.”).
Regarding claim 7, Donner teaches the method of claim 6, wherein adjusting the first velocity comprises directly adjusting the first velocity stored in the memory buffer (Donner teaches a system memory (label 206 of Fig. 2) and a processor for performing the operations of the invention [0030]. Furthermore, in [0035] Donner mentions storing the pixel velocities (first velocity) for in input texture in a memory buffer, and operations regarding fragments are performed.).
Regarding claim 8, Donner teaches the method of claim 1, wherein obtaining the first velocity comprises determining the first velocity based on motion of the respective parts of the second image frame between the first image frame and the second image frame (See Fig. 1. [0016] “As each input texture 122 is rendered, the display system 100 performs a motion analysis process 124 to determine a velocity, Vp, for each pixel in the input texture 122…. In other embodiments, the pixel velocities for an input texture 122 may be determined through a pixel motion estimation technique, such as through application of an optical flow analysis to a sequence 126 of two or more textures 122, including the current texture 122 being processed and one or more previously rendered textures 122.” Furthermore, Fig. 1 shows the method applied to a virtual object in the image(s). See the virtual object at 124 and 144 in Fig. 1.).
Regarding claim 9, Donner teaches the method of claim 8, wherein determining the first velocity comprises: generating an image comprising the one or more parts of the second image frame, using a position of the one or more parts of the second image frame and of a virtual camera from the first image frame; and determining a change in position of the one or more parts of the second image frame between the generated image and the second image frame (Donner teaches obtaining the pixel velocity field (first velocity) by determining the movement of pixels between a the current texture (second image frame) being processed and a previously rendered texture (first image frame). [0020] “In other embodiments, the pixel velocities for an input texture 122 may be determined through a pixel motion estimation technique, such as through application of an optical flow analysis to a sequence 126 of two or more textures 122, including the current texture 122 being processed and one or more previously rendered textures 122.”).
Regarding claim 10, Donner teaches the method of claim 1, wherein each part of the second image frame comprises a pixel of the second image frame (Image frames contain objects comprised of pixels. See the object 144 in the second image frame 140 in Fig. 1.).
Regarding claim 11, Donner teaches the method of claim 1, further comprising storing the adjusted velocity of the one or more parts of the second image frame in a memory buffer (Donner teaches a system memory (label 206 of Fig. 2) and a processor for performing the operations of the invention [0030]. Furthermore, in [0038-0039] Donner mentions storing the net velocity field (first velocity adjusted by the motion data) in a memory buffer, and the advection module performs operations for adjusting the second image frame using that net velocity field.).
Regarding claim 13, Donner teaches the method of claim 1, further comprising detecting the motion data using one or more sensors, wherein the one or more sensors comprise one or more selected from the list consisting of: one or more motion sensors, one or more cameras operable to capture images of the HMD, and one or more cameras mounted on the HMD and operable to capture images of an environment around the HMD (Fig. 2 shows the hardware configuration of the HMD. The HMD comprises two cameras—212 and 214—which are mounted on the HMD and capture images of the environment around the HMD. Furthermore, the HMD includes motion sensors for determining the motion data. [0018] “…whereby the display system 100 samples one or more motion-based sensors (e.g., gyroscope, accelerometer, magnetometer, etc.) to determine a current motion vector,”).
Regarding claim 14, Donner teaches a non-transitory computer-readable medium comprising computer executable instructions adapted to cause a computer system to perform an image processing method ([0045] “The software comprises one or more sets of executable instructions stored or otherwise tangibly embodied on a non-transitory computer readable storage medium.”) comprising:
obtaining a first velocity of one or more parts of a second image frame between a first image frame and the second image frame, wherein the first image frame and the second image frame are frames for display at a head mounted display, HMD ([0016] “As each input texture 122 is rendered, the display system 100 performs a motion analysis process 124 to determine a velocity, Vp, for each pixel in the input texture 122.” Step 124 of Fig. 1 includes determining the velocity field for pixels between an input texture (first image frame) and a display texture (second image frame). In the Background section, Donner explains that the term ‘textures’ can also refer to ‘frames.’ [0003] “HMD-based VR and AR systems display 3D imagery as a sequence of display textures (or ‘frames’),” Thus, the ‘input texture’ and the ‘display texture’ taught by Donner are equivalent to a ‘first image frame’ and a ‘second image frame’ of the claimed invention.);
receiving motion data indicative of motion of the HMD between a display time of the first image frame and a display time of the second image frame (Step 130 of Fig. 1 includes determining the HMD velocity field which describes movement of the HMD in pixel velocities. [0018] “In parallel with the motion analysis process 124, the display system 100 performs a HMD motion conversion process 130 whereby the display system 100 samples one or more motion-based sensors (e.g., gyroscope, accelerometer, magnetometer, etc.) to determine a current motion vector, denoted “MVHMD”, of the HMD device 102 relative to the world space.” [0019] “After determining the current motion vector MVHMD, the HMD motion conversion process 130 converts the current motion vector MVHMD to a per-pixel velocity representation of the motion of the HMD device 102 in the screen space, resulting in an HMD velocity field 132,”);
adjusting the first velocity based on the received motion data to determine an adjusted velocity of the one or more parts of the second image frame between the first image frame and the second image frame (Step 134 of Fig. 1 includes adjusting the pixel velocity field (first velocity) by combining it with the HMD velocity field (motion data) to generate a net velocity field (adjusted first velocity). [0020] “The display system 100 then performs a velocity combination process 134 whereby the pixel velocity field 128 and the HMD velocity field 132 are combined to generate a net velocity field 136. The net velocity field 136 represents the net relative velocity of each pixel when considering both the velocity of the pixel in the rendered imagery and the velocity of the HMD device 102.”); and
performing one or more image processing operations on the second image frame based on the adjusted velocity of the one or more parts of the second image frame (Referring to Fig. 1, see the original position 146 of an object in the input texture (first image frame) and the updated position 144 of that object in the display texture (second image frame). The net velocity field (adjusted first velocity) is used for performing advection to determine and display the updated position of the object in the display texture. See 414-422 of Fig. 4 and [0039-0043].).
Regarding claim 15, Donner teaches the non-transitory computer-readable medium of claim 14, wherein the one or more image processing operations comprise one or more selected from the list consisting of: motion-blurring, temporal anti-aliasing, and depth of field processing (In 414-422 of Fig. 4 and [0039-0043], Donner teaches using the net velocity field (adjusted first velocity) to perform advection. In [0043], Donner teaches using the disocclusion module for applying motion blur to the display texture based on the advection process.).
Regarding claim 16, Donner teaches the non-transitory computer-readable medium of claim 14, wherein adjusting the first velocity based on the received motion data comprises subtracting a function of the motion data from the first velocity (Step 134 of Fig. 1 includes adjusting the pixel velocity field (first velocity) by combining it with the HMD velocity field (motion data) to generate a net velocity field (adjusted first velocity). Thus, the net velocity field is obtained by the addition of the pixel velocities and the HMD velocities. [0020] “Vnet=Vp+Vh”. This function would also encompass subtraction depending on the direction of the motion of the HMD and the pixel velocities. Furthermore, in an embodiment, Donner teaches that the velocity of the movement of the HMD can be subtracted from the velocity of a fragment to more accurately account for the depth of each fragment when converting the HMD movement into the HMD velocity field. [0037] “A more accurate (and more computationally expensive) transformation includes accounting for the depth of every fragment (essentially treating each fragment as a 3D point), and subtracting the HMD pose velocity vector from the per fragment velocity vector (assuming both are in virtual world space) to get a HMD-relative fragment velocity vector that could then be transformed into a screen space velocity vector. This approach thus adds the velocities in virtual world space and then transforms to screen space;”).
Regarding claim 17, Donner teaches the non-transitory computer-readable medium of claim 16, wherein, for each image part of the one or more parts of the second image frame, the function of the motion data is further based on a position of the image part (Donner teaches obtaining motion data and converting it to a HMD velocity field. [0020] “The net velocity field 136 represents the net relative velocity of each pixel when considering both the velocity of the pixel in the rendered imagery and the velocity of the HMD device 102.” Here, the term “parts” is being interpreted as pixels, and the pixel velocity field (first velocity data) can be adjusted by the HMD velocity field (motion data). Additionally, Donner teaches performing the method for a virtual object in the images [0021]. In Fig. 1, the pixel velocity field for a virtual object 142 is determined, and the position of virtual object 144 is corrected in the display texture (second image frame) by using the net velocity field (first velocity adjusted by motion data).).
Regarding claim 18, Donner teaches the non-transitory computer-readable medium of claim 14, wherein obtaining the first velocity comprises determining the first velocity based on motion of the respective parts of the second image frame between the first image frame and the second image frame (See Fig. 1. [0016] “As each input texture 122 is rendered, the display system 100 performs a motion analysis process 124 to determine a velocity, Vp, for each pixel in the input texture 122…. In other embodiments, the pixel velocities for an input texture 122 may be determined through a pixel motion estimation technique, such as through application of an optical flow analysis to a sequence 126 of two or more textures 122, including the current texture 122 being processed and one or more previously rendered textures 122.” Furthermore, Fig. 1 shows the method applied to a virtual object in the image(s). See the virtual object at 124 and 144 in Fig. 1.).
Regarding claim 19, Donner teaches the non-transitory computer-readable medium of claim 18, wherein determining the first velocity comprises: generating an image comprising the one or more parts of the second image frame, using a position of the one or more parts of the second image frame and of a virtual camera from the first image frame; and determining a change in position of the one or more parts of the second image frame between the generated image and the second image frame (Donner teaches obtaining the pixel velocity field (first velocity) by determining the movement of pixels between a the current texture (second image frame) being processed and a previously rendered texture (first image frame). [0020] “In other embodiments, the pixel velocities for an input texture 122 may be determined through a pixel motion estimation technique, such as through application of an optical flow analysis to a sequence 126 of two or more textures 122, including the current texture 122 being processed and one or more previously rendered textures 122.”).
Regarding claim 20, Donner teaches an image processing system ([0045] “certain aspects of the techniques described above may implemented by one or more processors of a processing system executing software.”) comprising:
a first velocity processor configured to obtain a first velocity of one or more parts of a second image frame between a first image frame and the second image frame, wherein the first image frame and the second image frame are frames for display at a head mounted display, HMD ([0016] “As each input texture 122 is rendered, the display system 100 performs a motion analysis process 124 to determine a velocity, Vp, for each pixel in the input texture 122.” Step 124 of Fig. 1 includes determining the velocity field for pixels between an input texture (first image frame) and a display texture (second image frame). In the Background section, Donner explains that the term ‘textures’ can also refer to ‘frames.’ [0003] “HMD-based VR and AR systems display 3D imagery as a sequence of display textures (or ‘frames’),” Thus, the ‘input texture’ and the ‘display texture’ taught by Donner are equivalent to a ‘first image frame’ and a ‘second image frame’ of the claimed invention.);
an input processor configured to receive motion data indicative of motion of the HMD between a display time of the first image frame and a display time of the second image frame (Step 130 of Fig. 1 includes determining the HMD velocity field which describes movement of the HMD in pixel velocities. [0018] “In parallel with the motion analysis process 124, the display system 100 performs a HMD motion conversion process 130 whereby the display system 100 samples one or more motion-based sensors (e.g., gyroscope, accelerometer, magnetometer, etc.) to determine a current motion vector, denoted “MVHMD”, of the HMD device 102 relative to the world space.” [0019] “After determining the current motion vector MVHMD, the HMD motion conversion process 130 converts the current motion vector MVHMD to a per-pixel velocity representation of the motion of the HMD device 102 in the screen space, resulting in an HMD velocity field 132,”);
an adjustment processor configured to adjust the first velocity based on the received motion data to determine an adjusted velocity of the one or more parts of the second image frame between the first image frame and the second image frame (Step 134 of Fig. 1 includes adjusting the pixel velocity field (first velocity) by combining it with the HMD velocity field (motion data) to generate a net velocity field (adjusted first velocity). [0020] “The display system 100 then performs a velocity combination process 134 whereby the pixel velocity field 128 and the HMD velocity field 132 are combined to generate a net velocity field 136. The net velocity field 136 represents the net relative velocity of each pixel when considering both the velocity of the pixel in the rendered imagery and the velocity of the HMD device 102.”); and
an image processor configured to perform one or more image processing operations on the second image frame based on the adjusted velocity of the one or more parts of the second image frame (Referring to Fig. 1, see the original position 146 of an object in the input texture (first image frame) and the updated position 144 of that object in the display texture (second image frame). The net velocity field (adjusted first velocity) is used for performing advection to determine and display the updated position of the object in the display texture. See 414-422 of Fig. 4 and [0039-0043].).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Donner (US 2017/0148206 A1) in view of Mallinson (US 2016/0189429 A1).
Regarding claim 12, Donner teaches the method of claim 1, but Donner fails to teach further comprising, upon determining that the motion of the HMD is below a predetermined threshold, performing one or more image processing operations on the second image frame based on the first velocity, without adjusting the first velocity.
However, Mallinson teaches further comprising, upon determining that the motion of the HMD is below a predetermined threshold, performing one or more image processing operations on the second image frame based on the first velocity, without adjusting the first velocity (Mallinson teaches methods for updating the display frame of a HMD when movement of the HMD is above a certain threshold [0010]. Compensating for HMD movement in subsequent display frames occurs only when the HMD moves enough to distort the display image. [0094-0095] “In one embodiment, the threshold amount of motion is the amount of motion that would cause distortion on the image presented on the display as perceived by the user wearing the HMD. In another embodiment, the threshold amount of motion is the amount of motion that would make pixel 506 (as described with reference to FIG. 5) closer to another pixel different from pixel 504, i.e., the adjusted pixel value for pixel 504 is closer to the value of a pixel different from pixel 504. In one embodiment, the pixel values are adjusted when the head rotates an amount that causes the angle traversed by the physical HMD, in the time between two successively scanned pixels, to be a significant proportion of the angular distance between pixels in the display system. The significant proportion value may be in the range from 10% to 100%, in some embodiments, although other values are also possible. If the motion is greater than the threshold motion, the method flows to operation 714 where the display data is modified based on the motion.”).
Donner and Mallinson are analogous in the art to the claimed invention, because both teach methods of updating the display of a HMD to compensate for HMD motion. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Donner’s invention by utilizing a minimum threshold requirement for HMD movement. This modification would allow the system to skip compensating for HMD movement when the amount of HMD movement is too little to provide noticeable distortion or discomfort to the user. See [0094-0095] of Mallinson describing the thresholds. Furthermore, [0069] and Fig. 3D explains that HMD movement above a threshold provides visible distortion which must be compensated for.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Chang (US 2018/0090078 A1) teaches systems and methods for reducing motion blur in the video signal of a HMD. The methods involve tracking the eyes of the user to determine a view area and generating an intermediate frame between a first and second frame to synthesize pixel values in the view area of the intermediate frame.
Leiby (US 2021/0258555 A1) teaches systems and methods for motion smoothing for a HMD. The methods involve generating motion vectors to modify pixel data of a frame.
Osman et al. (US 2023/0042920 A1) teaches systems and methods for motion blur compensation for a HMD. The methods involve tracking the user’s eye and head movements and keying the motion blur of objects on the display based on the tracking.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ERIC JAMES SHOEMAKER whose telephone number is (571)272-6605. The examiner can normally be reached Monday through Friday from 8am to 5pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner' s supervisor, JENNIFER MEHMOOD, can be reached at (571)272-2976. The fax phone number for the organization where this application or proceeding is assigned is (571)273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Eric Shoemaker/
Patent Examiner
/JENNIFER MEHMOOD/ Supervisory Patent Examiner, Art Unit 2664
Prosecution Insights