Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Response to Arguments
Applicant's arguments filed 04/06/2026 have been fully considered but they are not persuasive. Applicant argues on pg. 8 that Jung teaches subsequent displays of the two predicted object poses, while asserting claim 1 teaches them both at the display time, which applicant asserts is the exact same moment, and that the two poses occur simultaneously instead of in a sequence. However, the language of claim 1 does not clearly state that the two must be displayed at the exact same instant in time. Claim 1 could be read as simply describing the general time that display happens. Examiner recommends applicant amends claim language to specify this distinction, with language such as “exact same time” or “the same time step” or “the same frame”. Instead, claim 1 only describes that both and shown at the display time. The display time is not clarified and there is not clear support in the drawings of multiple predicted positions being displayed simultaneously.
In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., “the same display time”) are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993).
Additionally, applicant in pg. 9 argues that Jung’s prediction disclosure teaches the motion of the virtual object, not to a predicted pose. Jung [0063] teaches either the latest pose or a predicted pose, so Jung does teach a predicted position that is different from an actual position.
Applicant argues the following:
PNG
media_image1.png
281
649
media_image1.png
Greyscale
(middle of page 9 in the filed response):
In response to this argument, the Examiner respectfully maintains that the prior art reference of Jung teaches the claimed feature being argued. As mentioned below in the rejection of claim 1, the claimed “object in the physical environment” is mapped to being a physical object detected by an optical sensor to assist in the tracking algorithm. This tracking information is later used to help determine the first predicted object pose.
For example, please see Jung towards the end of [0034] where they refer to: “… The AR application generates virtual content corresponding to an identified object (e.g., physical object 104) in the image and presents the virtual content in a display of the AR/VR device 106.” Figure 1 of Jung shows the physical object 104 moving. Thus, generating the first predicted object pose (virtual content) to the object in the physical.
In particular, generating the first predicted object pose (virtual content) to the object in the physical environment because [0034] says that the virtual content corresponds to the physical object.
Also, please see Jung in [0045] “The display controller 218 receives the image data (e.g., rendered frame) from the Graphical Processing Unit 216, re-adjusts a location of the rendered virtual content in a time-warped frame by performing a late-warping transformation based on a latest pose of the AR/VR device 106 and the latest tracking information (of a tracked physical object, of a preset animation of a virtual object, of multiple physical objects having different movement)”.
According to this passage from Jung, the predicted first object pose (predicted location by a late-warping transformation) is based on the tracked physical object.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-6, 9, 12-17, and 19-20 are rejected under 35 U.S.C. 102(a)(1) or 102(a)(2) (or both) as being anticipated by Jung (Pub No. US 20220375026 A1).
As per claim 1, Jung teaches the claimed:
1. A method comprising: at a device including an image sensor, a display, one or more processors, and non- transitory memory: capturing, using the image sensor, an image of an object in a physical environment; (Jung teaches receiving data of an object in a physical environment, because it is a dynamic physical object. The optical sensor is the image sensor. Jung [0064]: “The multiple objects warping module 606 receives tracking data of one or more physical object being tracked by a computer vision algorithm of the dynamic physical object tracking system 316. The tracking data include a latest location/position of a first and second physical object depicted in an image captured by the optical sensor 212. For example, the first physical object is stationary, while the second physical object is moving.” Latest location/position of an object implies a tracking of time and the position at the current display time.).
obtaining, based on the image, a first predicted object pose of the object in the physical environment at a display time; (Jung teaches a first pose at a first location. Jung [0062]: “The face warping module 602 receives tracking data from the face tracking system 312. The tracking data include a latest location/position of a face depicted in an image captured by the optical sensor 212. For example, the user may be moving. The optical sensor 212 captures a first picture that depicts the user at a first location in the first picture.” Jung [0063]: “The animated virtual object warping module 604 applies a late-warping algorithm to a rendered image based on the latest location (or predicted location) of the animated virtual object (per settings in the animation configuration) and based on the latest IMU data from the inertial sensor module 302.” This contains the predicted location of the object in which the position is based. The latest location is for the first object.).
rendering virtual content based on the first predicted object pose; (Jung [0063]: “… The tracking data include a latest position of the animated virtual object (generated by the AR/VR application 210) within an image captured by the optical sensor 212”. Since it is animated, it is rendered.).
obtaining a second predicted object pose of the object in the physical environment at the display time;
warping the virtual content based on the second predicted object pose; (Jung teaches a second picture with a second location and warps the image based on that. Jung [0062]: “The face warping module 602 receives tracking data from the face tracking system 312. The tracking data include a latest location/position of a face depicted in an image captured by the optical sensor 212. For example, the user may be moving. The optical sensor 212 captures a first picture that depicts the user at a first location in the first picture. The 3D render engine 502 generates a rendered virtual object based on the pose of the AR/VR device 106 and the face of the user at the first location. Subsequently, the optical sensor 212 captures a second picture depicting that the user has moved to a second location in the second picture. The face warping module 602 applies a late-warping algorithm to the rendered image based on the latest IMU data from the inertial sensor module 302 and the tracked face of the user at the second location (the latest user face location as determined by the face tracking algorithm).” The warping algorithm is based on the predicted latest location of the virtual object, as described above in Jung [0063]. This includes the second location as well as the first location.).
And displaying, on the display at the display time, the warped virtual content.
(Jung fig. 10 depicts a flow chart describing applying the warping algorithm to the images based on the object locations and then displaying them.)
As per claims 14 and 20, these claims are similar in scope to limitations recited in claim 1, and thus are rejected under the same rationale. Jung teaches a non-transitory computer-readable medium for its method in its claim 20. Jung teaches a device to implement it as well.
As per claim 2, Jung teaches the claimed:
2. The method of claim 1, wherein the object is a dynamic object in the physical environment. (Jung [0029]: “In yet another example, the late-warping process considers both the latest location of a static physical object, and the latest location of a dynamic physical object.” Jung [0064] “The multiple objects warping module 606 receives tracking data of one or more physical object being tracked by a computer vision algorithm of the dynamic physical object tracking system 316.”).
As per claim 15, this claim is similar in scope to limitations recited in claim 2, and thus is rejected under the same rationale.
As per claim 3, Jung teaches the claimed:
The method of claim 2, wherein the object is a hand of a user of the device. (Jung [0028]: “Previous solutions for reducing M2P latency rely on detecting feature points on stationary physical objects. In other words, previous solutions only address the latency due to motion of the AR display device and not the physical objects. M2P of physical objects that also move independently of the AR display device (e.g., such as the user's hands or physical objects moving in the real-world environment) result in additional M2P latency.” The dynamic tracking of an object is done for the hand of the user. Jung [0054]: “The dynamic physical object tracking system 316 uses computer vision to identify a moving physical object and track its latest location (within an image). For example, the dynamic physical object tracking system 316 tracks a location of a moving physical object (e.g., a hand, a human body, a car, or any other physical object).”).
As per claim 16, this claim is similar in scope to limitations recited in claim 3, and thus is rejected under the same rationale.
As per claim 4, Jung teaches the claimed:
4. The method of claim 2, wherein the object is a face of a person (Jung [0029]: “The system applies a late-warping process that is optimized for both (movement of the AR device, and movement of the physical object). For example, the AR device tracks another person's face (using computer vision algorithm). The AR device applies the late-warping process based on the latest location of the tracked face and the latest IMU data. By tracking and considering both the AR device motion and the face motion, the AR device can generate augmentations that are more accurately placed on top of the face. In another example, the late-warping process considers the predetermined animation motion of a virtual object. In yet another example, the late-warping process considers both the latest location of a static physical object, and the latest location of a dynamic physical object.”).
As per claim 17, this claim is similar in scope to limitations recited in claim 4, and thus is rejected under the same rationale.
As per claim 5, Jung teaches the claimed:
5. The method of claim 1, wherein obtaining the second predicted object pose is based on an additional image of the object. (Jung teaches that the image sensor tracks a user, the object, to a second location with an additional image. Jung [0062]: “The 3D render engine 502 generates a rendered virtual object based on the pose of the AR/VR device 106 and the face of the user at the first location. Subsequently, the optical sensor 212 captures a second picture depicting that the user has moved to a second location in the second picture.”).
As per claim 6, Jung teaches the claimed:
6. The method of claim 1, wherein obtaining the second predicted object pose is based on data received from an inertial measurement unit. (Jung teaches applying the warping for the second image of the second location based on an inertial sensor module, which is the inertial measurement unit. Jung [0062]: “The face warping module 602 applies a late-warping algorithm to the rendered image based on the latest IMU data from the inertial sensor module 302 and the tracked face of the user at the second location (the latest user face location as determined by the face tracking algorithm”).
As per claim 9, Jung teaches the claimed:
9. The method of claim 1, wherein warping the virtual content includes shifting the virtual content. (Jung [0060]: “The display controller 218 applies the time-warping engine 402 to the rendered virtual object by performing a three-dimensional shift operation to the rendered frame (e.g., frame a) based on the latest IMU data and latest content tracking data to generate a new frame (e.g., frame b). The display controller 218 communicates frame b to the display 204 for display.” The rendered frame is the virtual content.).
As per claim 12, Jung teaches the claimed:
12. The method of claim 1, further comprising: obtaining a first predicted device pose of the device at the display time, wherein rendering the virtual content is based on a first predicted device pose, and (Jung teaches rendering the virtual object, which is the virtual content, based on the first pose and the first location. Jung [0062]: “by the optical sensor 212. For example, the user may be moving. The optical sensor 212 captures a first picture that depicts the user at a first location in the first picture. The 3D render engine 502 generates a rendered virtual object based on the pose of the AR/VR device 106 and the face of the user at the first location…”).
after rendering the virtual content, obtaining a second predicted device pose of the device at the display time, wherein warping the virtual content is based on the second predicted device pose. (Jung then teaches obtaining the second picture and applying the warping, after the rendering of the object. Jung [0062]: “…Subsequently, the optical sensor 212 captures a second picture depicting that the user has moved to a second location in the second picture. The face warping module 602 applies a late-warping algorithm to the rendered image based on the latest IMU data from the inertial sensor module 302 and the tracked face of the user at the second location (the latest user face location as determined by the face tracking algorithm).” The second location is the second pose.).
As per claim 19, this claim is similar in scope to limitations recited in claim 12, and thus is rejected under the same rationale.
As per claim 13, Jung teaches the claimed:
13. wherein obtaining the second predicted device pose is based on data received from an inertial measurement unit of the device. (The system used for tracking the objects and determining their poses, including the second pose of the second object. involves an inertial sensor module, which is the inertial measurement unit. Jung [0049]: “The visual tracking system 308 includes an inertial sensor module 302, an optical sensor module 304, and a pose estimation module 306. The inertial sensor module 302 accesses inertial sensor data from the inertial sensor 214. The optical sensor module 304 accesses optical sensor data from the optical sensor 212.” The rendering of the second location of the second object is based on the inertial sensor module. Jung [0062]: “…Subsequently, the optical sensor 212 captures a second picture depicting that the user has moved to a second location in the second picture. The face warping module 602 applies a late-warping algorithm to the rendered image based on the latest IMU data from the inertial sensor module 302 and the tracked face of the user at the second location (the latest user face location as determined by the face tracking algorithm).”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 7-8, 10-11, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Jung in view of Xue (Pub No. US 20200098186 A1).
As per claim 7, Jung alone does not explicitly teach the claimed limitations.
However, Jung in combination with Xue teaches the claimed:
7. The method of claim 1, wherein warping the virtual content is based on a difference between the first predicted object pose and the second predicted object pose. (Xue teaches performing a homography based on a difference in poses of a first and second image. Xue [0068]: “In addition, controller 50 may be configured to perform a homography based on the difference in the poses of the current frame and a previous frame. The previous frame may be the frame that is displayed or is to be displayed immediately before the current frame. In some examples, the previous frame may be the frame that is decoded immediately before the current frame. However, the techniques are not so limited, and the previous frame may be any previously decoded frame, and not necessarily the immediately preceding frame in display or decoding order.” The frames indicate the pose information of the objects. Xue [0070]: “In examples described in this disclosure, the perspective associated with the previous frame is the pose information associated with the previous frame, and the perspective associated with the current frame is the pose information associated with the current frame. …”
The homography is used for warping. “One example way in which to perform the warping is via texture mapping. In texture mapping, the GPU maps image content from a texture (e.g., the previous frame) to a frame mesh. In this example, the GPU receives the coordinates of vertices in the previous frame and coordinates for where the vertices are to be mapped for the warping based on the homography determined by controller 50. In turn, the GPU maps the image content of the vertices to points on the frame mesh determined from the homography. The result is the warped image content.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the homography-based warping based on the difference of a first and second position as taught by Xue with the system of Jung in order to apply the warping more accurately to the two different object positions taught by Jung.
As per claim 18, this claim is similar in scope to limitations recited in claim 7, and thus is rejected under the same rationale.
As per claim 8, Jung alone does not explicitly teach the claimed limitations.
However, Jung in combination with Xue teaches the claimed:
8. The method of claim 1, wherein warping the virtual content includes performing a homographic transformation of the virtual content. (Xue [0068]: “In addition, controller 50 may be configured to perform a homography based on the difference in the poses of the current frame and a previous frame. The previous frame may be the frame that is displayed or is to be displayed immediately before the current frame. In some examples, the previous frame may be the frame that is decoded immediately before the current frame…” The images in the frames are the virtual content to which the homography is applied. The homography includes a homographic transformation. Xue [0069]: “Homography is the process by which controller 50 determines where a point in the previous frame would be located in the current frame given the pose associated with the previous frame and the pose associated with the current frame. As one example, homography is a transformation where coordinates in a point in the previous frame are multiplied by a 3×3 matrix to generate the coordinates of that point in the current frame. Stated another way, homography transforms image content of an image from its perspective to the perspective of another image.”
The homography is used for warping. Xue [0073]: “As described above, in performing the homography, controller 50 may determine the coordinates of where points in the previous frame would be located in the current frame. Based on the determined coordinates and the color values of the pixels in the previous frame, controller 50 may cause a graphics processing unit (GPU) of multimedia processor 52 to warp the image content of the previous frame. For example, controller 50 may output graphics commands that causes the GPU to perform the warping.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the homographic transformation as taught by Xue with the system of Jung in order to use homography to perform the warping between two positions of two images as taught by Jung.
As per claim 10, Jung alone does not explicitly teach the claimed limitations.
However, Jung in combination with Xue teaches the claimed:
10. The method of claim 1, further comprising generating a stencil indicating portions of the virtual content to be warped based on the second predicted object pose, wherein warping the virtual content is further based on the stencil. (Applicant describes a “stencil” in [0053]-[0054] of the specification as a matrix based on the pixels of the image that contains information about whether each pixel is warped: “The reprojection module 312 further generates, for each composite image, a stencil indicating how portions of the composite image are to be warped by the late-stage warp (LSW) module 321 described further below.
[0054] In various implementations, the stencil includes a matrix of the same resolution of the composite image, each element of the matrix corresponding to a pixel of the composite image. In various implementations, each element of the matrix has a value of ‘0’ if the corresponding pixel of the composite image is not to be warped by the LSW module 321, a value of ‘1’ if the corresponding pixel of the composite image is to be warped by the LSW module 321 based only on the pose of the electronic device 300, a value of ‘2’ if the corresponding pixel of the composite image is to be warped by the LSW module 321 based on the pose of the electronic device 300 and the pose of a first dynamic object, a value of ‘3’ if the corresponding pixel of the composite image is to be warped by the LSW module 321 based on the pose of the electronic device 300 and the pose of a second dynamic object, etc.”
Xue teaches using warping to fill in areas of image frames that are different from the previous frame of the same image. It teaches basing the warping on different between a current frame and a previous frame. Xue [0064]: “In the example techniques described in this disclosure, multimedia processor 52 may use the pose information of frames to warp image content to fill in portions of a frame that could not be reconstructed. Such filling of portions of the frame may be for error concealment or as part of constructing the frame.”
Xue teaches a mask that indicates which portion of the frame are to be warped. Xue [0067]: “As an example, multimedia processor 52 may include a bitstream parser circuit, illustrated in FIGS. 3 and 4, which receives the bitstream via path 62 generated by host device 10. The bitstream parser circuit may determine portions of the current frame for which there is no image content information. For example, the bitstream parser circuit may determine for which slices of the current frame there was packet loss. Based on the determination of which slices had packet loss, controller 50 may generate a mask for the current frame. In this mask, a logic zero for a portion (e.g., slice) indicates that image content information was received, and a logic one for a portion (e.g., slice) indicates that image content information was not received. In this way, the mask indicates holes/missing macroblocks in the frame.” The mask with the logic zero or logic one is the stencil which shows which portions of the image are to be warped. While the mask is used to indicate warping based on image content that can’t be reconstructed, it would be obvious to use the mask to indicate warping areas based on differences from motion of a portion of the image.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the mask indicating a portion of an image to be warped based on a difference from another image as taught by Xue with the system of Jung in order to show which parts of a frame are to be warped based on changes in position and physical characteristics of the subject of the frame.
As per claim 11, Jung alone does not explicitly teach the claimed limitations.
However, Jung in combination with Xue teaches the claimed:
11. The method of claim 10, wherein the stencil further indicates portions of the virtual content that are not to be warped. (Xue teaches a mask indicating where to warp an image based on differences between it and a previous frame. Xue [0066]: “For instance, multimedia processor 52 may determine which portions of the current frame cannot be reconstructed (e.g., due to dropped information or errors in the bitstream). Multimedia processor 52 may also generate warped image content based on the pose information of the current frame and a previous frame. Multimedia processor 52 may then copy the warped image content into the current frame.” The mask includes a logical one where image information was not received, and thus where the warping would take place. Xue [0067]: “For example, the bitstream parser circuit may determine for which slices of the current frame there was packet loss. Based on the determination of which slices had packet loss, controller 50 may generate a mask for the current frame. In this mask, a logic zero for a portion (e.g., slice) indicates that image content information was received, and a logic one for a portion (e.g., slice) indicates that image content information was not received. In this way, the mask indicates holes/missing macroblocks in the frame.” This would indicate which portions of the frame would not be warped as well.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the mask to indicate warping with a clear indication of which areas are warped and which aren’t, as taught by Xue with the system of Jung in order to show which areas of the second image are not warped because they don’t contain the different position of the object.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
/THOMAS JOHN FOSTER/Examiner, Art Unit 2616
/DANIEL F HAJNIK/Supervisory Patent Examiner, Art Unit 2616
Prosecution Insights