Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Priority
Acknowledgment is made of applicant’s foreign priority claim, for U.S. Application No. 18/982,236, based on a foreign application filed on 12/25/2023.
Status of Claims
Claims 1–11 are pending in the application. Claims 1-4, 6-7, 9, 10, 11 are rejected.
Claims 5, 8 are objected to.
Allowable Subject Matter
Claims 5, 8 are objected to as being dependent upon a rejected base claim(s), but would be allowable if rewritten in independent form including all of the limitations of the base claim(s) and any intervening claim(s).
Overview of Grounds of Rejection
Ground of Rejection
Claim(s)
Statute(s)
Reference(s)
Ground 1
1, 2, 4, 9
§ 103
Thaut et al. (US20220066546A1) and Fortin-Deschênes et al. (US20210011289A1)
Ground 2
3
§ 103
Thaut et al. (US20220066546A1), Fortin-Deschênes et al. (US20210011289A1), and Haraden et al. (WO2018183025A1)
Ground 3
6
§ 103
Thaut et al. (US20220066546A1) and Krishnaprasad et al. (US20230039100A1)
Ground 4
7
§ 103
Thaut et al. (US20220066546A1), Krishnaprasad et al. (US20230039100A1), and Fortin-Deschênes et al. (US20210011289A1)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
(Please see the cited paragraphs, sections, pages, or surrounding text in the references for the paraphrased content.)
Ground of Rejection 1
Claims 1, 2, 4, 9, 10, 11 are rejected under 35 U.S.C. § 103 as being unpatentable over Thaut et al. (US20220066546A1) in view of Fortin-Deschênes et al. (US20210011289A1).
As per Claim 1, Thaut teaches the following portion of Claim 1, which recites:
“An image processing apparatus comprising one or more processors and/or circuitry configured to: perform first acquisition processing to acquire information about position and orientation of a display apparatus;”
Thaut et al. teaches acquiring position/orientation information of the HMD/display apparatus.
Thaut discloses that “HMD device 400 includes a near-eye display 402” and that “IMU 414 may be configured to provide position and/or orientation data of HMD device 400 to on-board computer 404,” including determining “a position and orientation (or 6DOF pose) of HMD device 400.” Thaut et al., ¶¶ [0025], [0034]-[0036].
Thaut teaches the following portion of Claim 1, which recites: “perform second acquisition processing to acquire information about relative position and orientation of a mobile object, which moves independently of the display apparatus, with respect to the display apparatus;”
Thaut et al. teaches acquiring manipulator/mobile-object pose information, where the manipulator may be a user’s hand or an input device, and the HMD may capture manipulator motion using image data or an IMU-equipped input device. Thaut et al., ¶¶ [0015], [0018], [0033]. Fortin-Deschênes et al. directly teaches the relative relationship because “User body tracking estimates the position and orientation of the user’s body (in particular, but not limited to hands and fingers) relative to the HMD.” The hands/fingers are mobile objects that move independently of the HMD/display apparatus. Fortin-Deschênes et al., ¶ [0006].
Thaut teaches the following portion of Claim 1, which recites: “perform rendering processing to render a first object arranged in real space, which serves as a field of view of a user, based on position and orientation of the display apparatus”
Thaut et al. teaches rendering first/viewer-frame content based on HMD pose. Thaut discloses that virtual imagery may be “rendered based on the head pose of a wearer” so that a virtual object appears “anchored to a fixed position in the surrounding physical environment.”
Thaut also teaches a first rendered image including content “associated with a viewer reference frame.” Thaut et al., ¶¶ [0010], [0015], [0026].
Thaut teaches the following portion of Claim 1, which recites: “and to render a second object arranged in the real space based on position and orientation of the mobile object;”
Thaut et al. teaches rendering second/manipulator-attached content based on the mobile object’s position/orientation. Thaut states that virtual imagery may be rendered “based on the position of a wearer’s hand, or the position of an input device manipulated by the wearer’s hand,” and describes this as “manipulator-attached content.” Thaut also teaches a virtual object “locked or anchored to a manipulator.” Thaut et al., ¶¶ [0012], [0026].
Thaut teaches the following portion of Claim 1, which recites: “and perform reprojection processing on the first object based on a change in position and orientation of the display apparatus during the rendering processing”
Thaut et al. teaches reprojection of the first/viewer-frame object based on updated HMD pose.
Thaut discloses rendering based on “an initial head pose” and subsequently reprojecting based on “an updated head pose.” Thaut further teaches that “First rendered image 202 is reprojected based on the head pose of a user to thereby produce a first reprojected image 206.” Thaut et al., ¶¶ [0010], [0016]-[0017], [0041].
Thaut alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Fortin-Deschênes, they collectively teach all of the limitation(s).
Thaut and Fortin-Deschênes teach the following portion of Claim 1, which recites: “and perform reprojection processing on the second object based on a change in relative position and orientation of the mobile object with respect to the display apparatus during the rendering processing.”
Thaut et al. teaches reprojection of second/manipulator-attached content based on updated head pose and updated manipulator pose. Thaut states that “Second rendered image 204 is reprojected based on the head pose of a user and a pose of a manipulator to thereby produce a second reprojected image 208.” Thaut further teaches reprojection based on “a first manipulator pose at a render time” and “a second pose of the manipulator determined later than the first pose.” Fortin-Deschênes et al. supplies the relative-position aspect by teaching hand/finger position and orientation “relative to the HMD.” Thaut et al., ¶¶ [0018]-[0020], [0041]; Fortin-Deschênes et al., ¶ [0006].
Before the effective filing date of the claimed invention, a person of ordinary skill in the art would have been motivated to combine Thaut et al.’s manipulator-based reprojection pipeline with Fortin-Deschênes et al.’s HMD-based relative hand/body tracking to improve mixed-reality alignment and reduce latency for both viewer-frame content and hand/controller-attached content. Thaut recognizes that manipulator-attached content can lag behind manipulator motion and teaches reducing that lag by reprojecting based on a later manipulator pose. Fortin-Deschênes teaches obtaining hand/finger position and orientation relative to the HMD using integrated HMD tracking. A POSITA would have found it predictable to use Fortin-Deschênes’s relative hand/HMD tracking data as the manipulator-pose input in Thaut’s reprojection process, yielding the expected improvement of more accurately aligned rendered objects during HMD and mobile-object movement.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
As per Claim 2, Thaut teaches Claim 2, which recites: “The image processing apparatus according to claim 1, wherein the one or more processors and/or the circuitry is further configured to perform display control processing to perform control so as to combine a first image obtained by performing reprojection processing on the first object and a second image obtained by performing reprojection processing on the second object to be displayed on the display apparatus.”
Thaut et al. teaches this limitation. Thaut discloses that “First rendered image 202 is reprojected based on the head pose of a user to thereby produce a first reprojected image 206,” and that “Second rendered image 204 is reprojected based on the head pose of a user and a pose of a manipulator to thereby produce a second reprojected image 208.” Thaut then directly teaches the claimed combining/display control because “After reprojection, first and second reprojected images 206 and 208 are composited to thereby produce a composited image 210,” and “Composition may include any suitable image processing steps, including but not limited to overlaying manipulator-attached content over content that is not manipulator-attached, and blending content.” Thaut further teaches display output because the HMD or other device may “composite the reprojected images and display the resulting composited image.” Thaut et al., ¶¶ [0016]-[0021], [0043].
Thus, Thaut’s first reprojected image 206 corresponds to the claimed “first image obtained by performing reprojection processing on the first object,” Thaut’s second reprojected image 208 corresponds to the claimed “second image obtained by performing reprojection processing on the second object,” and Thaut’s compositing and display of composited image 210 teaches “perform control so as to combine” those images “to be displayed on the display apparatus.”
PNG
media_image1.png
13
460
media_image1.png
Greyscale
As per Claim 4, Thaut alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Fortin-Deschênes, they collectively teach all of the limitation(s).
Thaut and Fortin-Deschênes teach Claim 4, which recites: “The image processing apparatus according to claim 1, wherein the one or more processors and/or the circuitry is further configured to perform display control processing to perform control so as to combine a first image obtained by performing reprojection processing on the first object, a second image obtained by performing reprojection processing on the second object, and a captured image of the real space captured by an image capturing apparatus after the rendering processing to be displayed on the display apparatus.”
Thaut et al. teaches display control processing for combining “a first image obtained by performing reprojection processing on the first object” and “a second image obtained by performing reprojection processing on the second object” because Thaut discloses that “First rendered image 202 is reprojected based on the head pose of a user to thereby produce a first reprojected image 206,” and “Second rendered image 204 is reprojected based on the head pose of a user and a pose of a manipulator to thereby produce a second reprojected image 208.” Thaut further teaches that, “After reprojection, first and second reprojected images 206 and 208 are composited to thereby produce a composited image 210,” and that the HMD may “composite the reprojected images and display the resulting composited image.” Thaut et al., ¶¶ [0016]-[0021].
Fortin-Deschênes et al. teaches combining the reprojected/processed virtual images with “a captured image of the real space captured by an image capturing apparatus after the rendering processing to be displayed on the display apparatus.” Fortin-Deschênes discloses an HMD having RGB camera sensors and a processing unit, wherein instructions perform the steps of “obtaining from the pair of RGB camera sensors pass-through stereo view images,” “generating rendered graphics based on the positional tracking,” “performing graphics image processing on the rendered graphics,” “mixing processed images and the processed rendered graphics resulting in the graphic content,” and “providing the graphic content to the display.”
Fortin-Deschênes also teaches that, for MR, “two forward facing cameras capture the environment,” and “the camera images are then merged in real-time with computer generated images and shown on the display system.” Fortin-Deschênes et al., ¶¶ [0018], [0021]-[0030], [0068], [0076].
Accordingly, Thaut teaches combining the first and second reprojected images for display, and Fortin-Deschênes teaches adding the real-space camera/pass-through image to the rendered/processed graphics and displaying the resulting combined graphic content on the HMD display.
Before the effective filing date, a POSITA would have been motivated to combine Thaut et al.’s compositing of separately reprojected images with Fortin-Deschênes et al.’s pass-through camera compositing to improve mixed-reality alignment. The combination would predictably display reprojected virtual objects together with captured real-space imagery.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
As per Claim 9, Thaut teaches Claim 9, which recites: “The image processing apparatus according to claim 1, wherein, in the rendering processing, an external apparatus is caused to perform the rendering processing on at least any of the first object and the second object.”
Thaut discloses that the reprojection pipeline may be partly performed on “a device that is separate from an HMD device,” and that the separate device may “render first and second rendered images 202 and 204 and transmit the rendered images” to the HMD device. The first rendered image 202 corresponds to rendering the first object, and the second rendered image 204 corresponds to rendering the second/manipulator-attached object. Thaut et al., ¶ [0021].
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Claim 10 does not include any additional limitations that would significantly distinguish it from claim 1. Therefore, it is likewise rejected under 35 U.S.C. § 103 in view of the same references and for the same reasons set forth above.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Claim 11 does not include any additional limitations that would significantly distinguish it from claim 1. Therefore, it is likewise rejected under 35 U.S.C. § 103 in view of the same references and for the same reasons set forth above.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Ground of Rejection 2
Claim 3 is rejected under 35 U.S.C. § 103 as being unpatentable over Thaut et al. (US20220066546A1) in view of Fortin-Deschênes et al. (US20210011289A1), and further in view of Haraden et al. (WO2018183025A1).
As per Claim 3, Thaut alone do not explicitly teach all of the limitation(s) of the claim. However, when combined with Haraden et al., they collectively teach all the limitation(s).
Thaut and Haraden teach the limitation(s) of Claim 3 that recites: “The image processing apparatus according to claim 2, wherein, in the display control processing, depth information about the first image and depth information about the second image are used to combine the first image and the second image.”
Thaut et al. teaches the base Claim 2 combination because “first and second reprojected images 206 and 208 are composited to thereby produce a composited image 210,” including overlaying and blending content. Thaut et al., ¶ [0020].
Haraden et al. supplies the depth-based combining aspect. Haraden teaches that LSR processing is applied to foreground and background layers, where “the difference may be due to a difference in determined depth between the FG layer 320 and the BG layer 330,” and then “after the LSR processing is applied to the various layers of the scene . . . the computer system composites the layers together to form a unified layer.” Haraden further states that the layers are visualized with “correct depth and orientation with regard to one another,” including use of “depth processing transformations.” Haraden et al., ¶¶ [0062]-[0064].
Thus, Thaut teaches combining the first reprojected image and second reprojected image, while Haraden teaches using determined depth of the respective image/layer content during LSR/compositing so that the combined image has proper depth relationship.
Before the effective filing date of the claimed invention, a POSITA would have been motivated to combine Thaut et al.’s compositing of separately reprojected viewer-frame and manipulator-frame images with Haraden et al.’s depth-aware multi-layer reprojection/compositing to improve visual alignment and occlusion handling in mixed-reality scenes. Thaut already teaches combining first and second reprojected images for display, while Haraden teaches using determined depth differences between foreground/background layers and compositing the layers with correct depth and orientation. Applying Haraden’s depth-based compositing technique to Thaut’s reprojected image combination would have been a predictable use of known MR rendering techniques to obtain the expected result of a more accurately combined displayed image.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Ground of Rejection 3
Claims 6, 7 are rejected under 35 U.S.C. § 103 as being unpatentable over Thaut et al. (US20220066546A1) in view of Fortin-Deschênes et al. (US20210011289A1), and further in view of Krishnaprasad et al. (US20230039100A1).
As per Claim 6, Thaut alone do not explicitly teach all of the limitation(s) of the claim. However, when combined with Krishnaprasad, they collectively teach all the limitation(s).
Thaut and Krishnaprasad teach the limitation(s) of Claim 6 that recites: “The image processing apparatus according to claim 1, wherein in the rendering processing, an ID that is mutually identifiable is assigned to each of the first object and the second object, and in the reprojection processing, the first object and the second object are identified based on the ID.”
Krishnaprasad et al. teaches the claimed “in the rendering processing, an ID that is mutually identifiable is assigned to each of the first object and the second object” because it discloses rendering “a plurality of virtual objects in a layer” where “each of the plurality of virtual objects is associated with at least one bounding box,” and determining metadata corresponding to reprojection instructions. Krishnaprasad further teaches that the metadata includes “at least one of anchor identifiers or pose information corresponding to the plurality of virtual objects.” Krishnaprasad et al., ¶¶ [0099]-[0102], [0126], [0142].
Krishnaprasad et al. also teaches “in the reprojection processing, the first object and the second object are identified based on the ID” because the AR runtime/warp engine receives reprojection metadata “for reprojecting each of the virtual objects,” and separately reprojects the virtual objects based on the metadata. Krishnaprasad teaches that the device may “reproject the plurality of virtual objects based on the first metadata and the second metadata,” where the second metadata includes the “anchor identifiers” corresponding to the virtual objects. Krishnaprasad et al., ¶¶ [0092]-[0094], [0108].
Thus, Krishnaprasad’s anchor identifiers correspond to the claimed mutually identifiable ID, and the use of those identifiers/metadata during reprojection teaches identifying the first and second objects based on the ID.
Before the effective filing date, a POSITA would have been motivated to combine Thaut et al.’s separate reprojection of viewer-frame and manipulator-frame objects with Krishnaprasad et al.’s use of object metadata, including anchor identifiers, for per-object reprojection. The combination would predictably allow Thaut’s first and second objects to be identified during reprojection using object IDs, improving object-specific reprojection accuracy and display alignment.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
As per Claim 7, Thaut alone do not explicitly teach all of the limitation(s) of the claim. However, when combined with Krishnaprasad and Fortin-Deschênes et al., they collectively teach all the limitation(s).
Krishnaprasad and Fortin-Deschênes teach Claim 7, which recites: “The image processing apparatus according to claim 6, wherein, in the rendering processing, each ID is stored in any of a plurality of channels included in each of the first object data and the second object data.”
Krishnaprasad et al. teaches assigning/rendering object-associated ID metadata because it discloses rendering “a plurality of virtual objects in a layer,” determining metadata for reprojection, and that the metadata includes “anchor identifiers or pose information corresponding to the plurality of virtual objects.” Krishnaprasad et al., ¶¶ [0099]-[0102], [0126], [0142].
Fortin-Deschênes et al. teaches storing/rendering graphics data using a plurality of channels, including “color and alpha channel,” “per pixel channel,” and “a total of 6 channels per pixel (namely R, G, B, Ar, Ag and Ab),” with “the three color channels” encoded and “the two alpha channels” mapped into another channel format. Fortin-Deschênes et al., ¶¶ [0076], [0087].
Thus, Krishnaprasad teaches object IDs/anchor identifiers used for reprojection, and Fortin-Deschênes teaches that rendered object/image data includes multiple channels. It would have been obvious to store Krishnaprasad’s object ID/anchor identifier in one of Fortin-Deschênes’s available object-data channels so that the object can be identified during reprojection.
Before the effective filing date, a POSITA would have been motivated to combine Krishnaprasad et al.’s object-specific anchor identifiers used for reprojection with Fortin-Deschênes et al.’s multi-channel rendered image/object data format. Storing the object ID in an available channel would have been a predictable implementation choice to allow the reprojection process to identify each object and apply object-specific reprojection accurately.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Conclusion
The prior art made of record and relied upon in this action is as follows:
Patent Literature:
Thaut et al. (US20220066546A1) — “Manipulator-based image reprojection.”
Haraden et al. (WO2018183025A1) — “Selective application of reprojection processing on layer sub-regions for optimizing late stage reprojection power.”
Fortin-Deschênes et al. (US20210011289A1) — “Head-mounted display for virtual and mixed reality with inside-out positional, user body and environment tracking.”
Krishnaprasad et al. (US20230039100A1) — “Multi-layer reprojection techniques for augmented reality.”
Non-Patent Literature (NPL):
(none)
Note: A PDF copy of each NPL reference is attached with this Office Action. URLs are included for applicant convenience. If a link becomes unavailable in the future, the citation information may be used to locate the reference or access archived versions via the Wayback Machine.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure and is listed as follows:
Patent Literature:
Williams et al. (US20150029218A1) — “Late stage reprojection.”
Non-Patent Literature (NPL):
(none)
Any inquiry concerning this communication or earlier communications from the examiner s
hould be directed to ADEEL BASHIR whose telephone number is (571) 270-0440. The examiner can normally be reached Monday-Thursday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached on (571) 276-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ADEEL BASHIR/
Examiner, Art Unit 2616
/DANIEL F HAJNIK/Supervisory Patent Examiner, Art Unit 2616
Prosecution Insights