DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Notice to Applicants
This action is in response to the Restriction Election filed on 05/29/2026.
Claims 1-15 are pending.
Priority
The present Application claims foreign priority to JP-2023-075689 with a filing date of 05/01/2023, which is acknowledged.
Information Disclosure Statement
The Information Disclosure Statement (IDS) filed on 04/29/2024 has been fully considered by the examiner.
Restriction/Election
The examiner thanks Applicant for their careful consideration of the Restriction Requirement mailed on 04/09/2026.
Applicant’s election without traverse of Group I (Claims 1-7 & 12-15) in the reply filed on 05/29/2026 is acknowledged.
Claims 8-11 are withdrawn from further consideration pursuant to 37 CFR 1.142(b) as being drawn to a nonelected invention, there being no allowable generic or linking claim. Election was made without traverse in the reply filed on 05/29/2026.
Claim Objections
Claim 6 is objected to.
Regarding claim 6, in lines 3-5, the claim lists a set of options for the correction methods, preceded by “any one of”. However, line 4 uses the word “and” to join the last two options, which reads as requiring both options together, which is not believed to be Applicant’s intention. Thus, the examiner suggests amending claim 6 to read in part “a maximum value, or a neighboring value of the combining information corresponding to the coordinate information” (emphasis added). In the interest of compact prosecution, the claim will be examined assuming the conjunctive “or” interpretation.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The examiner notes that each of the following seven units of claims 1-7 and 12-13 pass the first two prongs of the three-prong test to invoke 112(f). However, they are each preceded and modified by the structural language of the image processing apparatus and physical processor that “functions as” each of the units as claimed. Thus, the 3rd prong of the 3-prong test fails, and 112(f) is not invoked.
Acquisition unit
Motion acquisition unit
CG generation unit
CG correction unit
Combining unit
Captured image correction unit
Sensing unit
Additionally, claim 13 imparts specific structure on “sensing unit” by defining it to be an inertial measurement unit (IMU).
Furthermore, claims 1-7 and 12-13 recite an “imaging apparatus” separate from the image processing apparatus, but the term “imaging” and the subsequent “capturing” of the real world impart structure and thus this term also avoids 112(f).
Claim Rejections – 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 3-6 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 3, lines 3-4 recite that the CG correction unit “performs the correction based on transformed coordinates calculated from the homography matrix” (emphasis added). However, claim 1, which claim 3 depends on, recites two distinct corrections, namely, correcting the image channel by pixel interpolation (lines 14-16) and correcting the combining information channel by pixel replacement (lines 16-17). Thus, it is unclear if claim 3 is referring to both corrections, or only one of them.
Regarding claims 4-6, each of these claims depend on claim 3. Each of these claims offer further details on one of the two corrections, but it is still unclear if both or only one of the corrections in each claim are based on the transformed coordinates.
To overcome the above 35 U.S.C. 112(b) rejections, the examiner suggests amending claim 3 to read in part “performs each correction based on transformed coordinates calculated from the homography matrix” (emphasis added), or alternatively, “performs both corrections based on transformed coordinates calculated from the homography matrix” (emphasis added).
35 USC § 101 Analysis
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
The examiner has determined that all of claims 1-7 and 12-15 are eligible under 35 USC 101. The 101 analysis is provided below for purposes of a clear record.
Analysis for claim 1 is provided in the following. Claim 1 is reproduced in the following (annotation added):
An image processing apparatus comprising:
one or more processors; and one or more memories coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the processor to function as:
an acquisition unit configured to acquire a captured image obtained by an imaging apparatus capturing a real world;
a motion acquisition unit configured to acquire motion information on the imaging apparatus in a space of the real world;
a computer graphics (CG) generation unit configured to generate a CG image including combining information related to combining with the captured image, based on the motion information on the imaging apparatus;
a CG correction unit configured to separate the CG image including the combining information into an image channel and a combining information channel,
correct the image channel by pixel interpolation according to the motion information on the imaging apparatus,
and correct the combining information channel by pixel replacement according to the motion information on the imaging apparatus;
and a combining unit configured to combine the captured image and the CG image corrected by the CG correction unit.
Step 1: Does the claim belong to one of the statutory categories? Claim 1 is directed to a machine, which is a statutory category of invention (YES).
Step 2A Prong One: Does the claim recite a judicial exception? The claim does not appear to recite any judicial exceptions. The claim involves steps of generating CG images (step e), correcting channels of said CG image (steps f-h), and finally combining the CG image with a captured image (step i). Even if the “correcting” in steps g and h were interpreted to be mental processes practically performable in the human mind, the rest of the claim would integrate them into a practical application, such as step i combining the corrected CG image with the captured image (NO). Claim 1 is eligible.
Similar analysis is applicable to independent claims 14 and 15. Claims 14 and 15 are eligible.
Claim 2 and 7 do not recite any new judicial exceptions. Claims 2 and 7 are eligible.
Claim 3 recites calculating a homography matrix based on motion information, which is directed to mathematical calculations. However, the claim further requires the correction to be based on transformed coordinates calculated from the homography matrix, which integrates this into a practical application. Claim 3 is eligible.
Claims 4-6 each require the transformed coordinates to have accuracy after a decimal point, which is directed to mathematical calculations. However, like claim 3 which each of claims 4-6 depend on, claims 4-6 integrate this into a practical application by requiring the corrections to be based on the coordinates in different ways. Claims 4-6 are eligible.
Claims 12 and 13 recite a sensing unit that senses a motion of the imaging apparatus in the space of the real world, which can be practically performed in the human mind. However, the claims integrate this into a practical application by requiring the generation of the CG image to be based on position information derived from said sensed position. Claims 12 and 13 are eligible.
Claim Rejections – 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 7, 12, and 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over Ohashi (U.S. Publ. US-2021/0191124-A1) in view of Taylor et al. (U.S. Publ. US-2021/0142575-A1).
Regarding claim 1, Ohashi discloses an image processing apparatus (see figure 2, image generation apparatus 200 and paragraphs 0031-0033) comprising:
one or more processors; and one or more memories coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the processor to function as (paragraph 0033 specifies that the image generation apparatus 200 can be a game console or general-purpose computer, both of which standardly include one or more processors coupled to a memory; paragraph 0058 specifies that the system can include a CPU and GPU):
an acquisition unit configured to acquire a captured image obtained by an imaging apparatus capturing a real world (see figures 1, 8, and paragraphs 0027-0030, where a head-mounted display equipped with a camera is in communication with the image generation apparatus; see figure 8 and paragraphs 0086-0087, where the camera unit of the head mounted display captures an image of the outside world and transmits it to the image generation apparatus's rendering section 232);
a motion acquisition unit configured to acquire motion information on the imaging apparatus in a space of the real world (see figure 8 and paragraph 0087, where the head mounted display also transmits current orientation and depth information; later on, paragraph 0090 specifies that the subsequent reprojection steps are based on the latest viewpoint location and direction of the head-mounted display; paragraphs 0057-0058 specify that these reprojections reflect the motion of the head-mounted display);
a computer graphics (CG) generation unit configured to generate a CG image including combining information related to combining with the captured image, based on the motion information on the imaging apparatus (see figure 8 and paragraphs 0088-0089, where the rendering section 232 generates a CG image of a virtual object seen from the viewpoint of the head-mounted display, then the image-with-alpha generation section 233 incorporates an alpha/combining information channel into the CG image, based on the depth/other combining information of the real-world image);
a CG correction unit configured to separate the CG image including the combining information into an image channel and a combining information channel, correct the image channel by pixel interpolation according to the motion information on the imaging apparatus, and correct the combining information channel by pixel replacement according to the motion information on the imaging apparatus (see figure 8 and paragraph 0089, where the CG image with alpha is corrected via reprojection and distortion to fit the latest viewpoint/motion information of the head-mounted display, but not specifically by performing interpolation on the image channel and replacement on the combining/alpha channel);
and a combining unit configured to combine the captured image and the CG image corrected by the CG correction unit (see figure 8 and paragraphs 0089-0090, where the AR superimposition section combines the corrected CG image with a corrected real-world image to produce the final AR image for the user).
Ohashi fails to disclose the limitations indicated via strikethrough above. In other words, Ohashi merely fails to disclose that their correction of the CG image includes applying the separate interpolation and replacement processes to the image and correction information channels, respectively.
Pertaining to the same field of endeavor, Taylor discloses a CG correction unit configured to separate the CG image including the combining information into an image channel and a combining information channel (paragraphs 0050-0051 specify that Taylor also operates on and reprojects virtual/CG images in augmented reality space; figure 8 and paragraphs 0093-0094 provide an overview of the cited reprojection/correction method, which the subsequent citations will show implicitly separates the color/image channel from the combining/depth channel, as the steps operate on each channel separately),
correct the image channel by pixel interpolation according to the motion information on the imaging apparatus (see figure 8, step 832 and paragraph 0102, where, after the following depth replacement steps, a convolution filter can be applied to any individual channel, such as the color/image channel, to modify each pixel based on characteristics of neighboring pixels, which reads as pixel interpolation; as illustrated by the following steps, these pixels are also obtained from and thus based on motion vectors corresponding to movement of an AR device),
and correct the combining information channel by pixel replacement according to the motion information on the imaging apparatus (see figure 8, steps 804-812 and paragraphs 0095-0097, where the method obtains 3D coordinates of each pixel at a first time, then applies motion vectors to said pixels to update their positions to a new time, then reprojects the pixels to map them to the final 2D pixel coordinates to be rendered; then see figure 8, steps 816-828 and paragraphs 0098-0101, where a linked list is then generated for each 2D pixel position, where each linked list contains each updated 3D pixel coordinate that maps to said 2D pixel; each linked list is then sorted by depth/combining information, and the pixel with the lowest depth is chosen for the final 2D mapping, which reads on replacing the depth value with one from the linked list).
Ohashi and Taylor are considered analogous art, as they are both directed to virtual image reprojections in AR environments. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have integrated the teachings of Taylor into Ohashi by applying the separate correction steps to the image and combining information channels because doing so allows for the final rendering to have accurate object occlusion (see Taylor paragraph 0093).
Regarding claim 2, Ohashi in view of Taylor discloses wherein the acquisition unit further acquires depth information on the real world corresponding to the captured image (see Ohashi figure 8 and paragraph 0087, where the camera unit supplies both the real-world image and depth information therefrom),
and wherein the combining information includes at least one of depth information and transparency information (see Ohashi figure 8 and paragraphs 0088-0089, where the image-with-alpha generation section 233 incorporates an alpha/transparency/combining information channel into the CG image, based on the depth/other combining information of the real-world image).
Regarding claim 7, Ohashi discloses wherein the acquisition unit further acquires depth information on the real world corresponding to the captured image (see figure 8 and paragraph 0087, where the camera unit supplies both the real-world image and depth information therefrom),
and wherein the image processing apparatus further comprises a captured image correction unit configured to separate the captured image into an image channel and a depth information channel, corrects the image channel by pixel interpolation according to the motion information on the imaging apparatus, and correct the depth information channel by pixel replacement according to the motion information on the imaging apparatus (see figure 8 and paragraph 0090, where the reprojection section performs a similar reprojection on the real-world image to fit the latest viewpoint/motion information of the head-mounted display, but again not specifically by performing interpolation on the image channel and replacement on the combining/alpha channel),
and wherein the combining unit combines the captured image corrected by the captured image correction unit and the CG image corrected by the CG correction unit (see figure 8 and paragraphs 0089-0090, where the AR superimposition section combines the corrected CG image with a corrected real-world image to produce the final AR image for the user).
Ohashi fails to disclose the steps indicated via strikethrough above. In other words, Ohashi does apply a correction step to the real-world image as well, just not via the specific claimed method.
Pertaining to the same field of endeavor, Taylor discloses and wherein the image processing apparatus further comprises a captured image correction unit configured to separate the captured image into an image channel and a depth information channel, corrects the image channel by pixel interpolation according to the motion information on the imaging apparatus, and correct the depth information channel by pixel replacement according to the motion information on the imaging apparatus (paragraph 0051 specifies that depth values are also assigned to the real-world images and objects therein; thus, in combination with Ohashi, who already does a separate correction of the real-world image, Taylor's correction method would also apply to the real-world images with predictable results, as the type of object (real or virtual) represented by the pixels would not impact the algorithm).
Ohashi and Taylor are considered analogous art, as they are both directed to virtual image reprojections in AR environments. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have integrated the teachings of Taylor into Ohashi by applying Taylor’s correction methods to Ohashi’s real-world image correction because doing so allows for the final rendering to have accurate object occlusion (see Taylor paragraph 0093).
Regarding claim 12, Ohashi in view of Taylor discloses wherein the imaging apparatus includes a sensing unit that senses a motion of the imaging apparatus in a space of the real world (see Ohashi figure 3 and paragraph 0039, where the head-mounted display includes an orientation sensor 64, such as a gyro sensor, acceleration sensor, etc.),
and wherein the motion acquisition unit acquires motion information on the imaging apparatus, based on sensing information on the sensing unit, and wherein the image processing apparatus further comprises a position and orientation acquisition unit configured to acquire a position and orientation of the imaging apparatus in the space of the real world, based on motion information on the imaging apparatus (see Ohashi paragraph 0044, where the image generation apparatus is notified of the head-mounted display's motion/orientation via the orientation sensor's output),
wherein the CG generation unit generates the CG image based on the position and orientation of the imaging apparatus (see Ohashi figure 8 and paragraphs 0088-0089, where the rendering section 232 generates a CG image of a virtual object seen from the viewpoint and orientation of the head-mounted display).
Regarding claim 14, Ohashi discloses an image processing method (see figures 7-8).
The remainder of claim 14 recites steps identical to those of claim 1. Therefore, Ohashi in view of Taylor discloses claim 14 as applied to claim 1 above.
Regarding claim 15, Ohashi discloses a non-transitory computer-readable storage medium storing a program for causing a computer to execute an image processing method, the image processing method comprising (paragraph 0033 specifies that the image generation apparatus 200 can be a game console or general-purpose computer, both of which standardly include one or more processors coupled to a transitory or non-transitory memory).
The remainder of claim 15 recites steps identical to those of claim 1. Therefore, Ohashi in view of Taylor discloses claim 15 as applied to claim 1 above.
Claims 3-6 are rejected under 35 U.S.C. 103 as being unpatentable over Ohashi (U.S. Publ. US-2021/0191124-A1) in view of Taylor et al. (U.S. Publ. US-2021/0142575-A1), and further in view of Sinclair (U.S. Publ. US-2021/0019526-A1).
Regarding claim 3, Ohashi in view of Taylor fails to disclose the limitations of claim 3.
Pertaining to the same field of endeavor, Sinclair discloses wherein the CG correction unit calculates a homography matrix based on the motion information on the imaging apparatus (see paragraph 0098, where a homography matrix can be determined that relates poses of an object across images; paragraphs 0044 and 0167 specify that the pose change of the object correlates to motion of the imaging apparatus),
and performs the correction based on transformed coordinates calculated from the homography matrix (see paragraph 0098, where the homography matrix is used to transform the coordinates across poses; in combination with Ohashi and Taylor, this would also predictably apply before their transformation methods, both for claim 3 and claims 4-6 below).
Ohashi and Sinclair are considered analogous art, as they are both directed to virtual image reprojections in AR environments. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have integrated the teachings of Sinclair into Ohashi and Taylor by using a homography matrix to convert coordinates for the reprojections because doing so allows for positioning, orienting, and scaling the CG image properly to the camera viewpoint (see Sinclair paragraph 0142).
Regarding claim 4, Ohashi fails to disclose the limitations of claim 4.
Pertaining to the same field of endeavor, Sinclair discloses wherein the transformed coordinates are coordinate information with accuracy after a decimal point (paragraph 0068 specifies that the image coordinates can have sub-pixel precision, which reads on accuracy after a decimal point).
Ohashi and Sinclair are considered analogous art, as they are both directed to virtual image reprojections in AR environments. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have integrated the teachings of Sinclair into Ohashi by using a homography matrix that transforms coordinates with sub-pixel accuracy because doing so allows for specifying finer measurement granularity if desired (see Sinclair paragraph 0068).
Ohashi in view of Sinclair fails to further disclose the remaining limitations of claim 4.
Pertaining to the same field of endeavor, Taylor discloses the correction by pixel interpolation is a correction using a value obtained by smoothing color information corresponding to the coordinate information as a correction value for pixel interpolation (see figure 8, step 832 and paragraph 0102, where, after the depth replacement steps, a convolution filter can be applied to any individual channel, such as the color/image channel, to modify each pixel based on characteristics of neighboring pixels, which reads as pixel interpolation).
Ohashi and Taylor are considered analogous art, as they are both directed to virtual image reprojections in AR environments. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have integrated the teachings of Taylor into Ohashi and Sinclair by performing pixel interpolation on the homography-transformed pixels because doing so allows for filling gaps or artifacts from disocclusion (see Taylor paragraph 0102).
Regarding claim 5, Ohashi fails to disclose the limitations of claim 5.
Pertaining to the same field of endeavor, Sinclair discloses wherein the transformed coordinates are coordinate information with accuracy after a decimal point (paragraph 0068 specifies that the image coordinates can have sub-pixel precision, which reads on accuracy after a decimal point).
Ohashi and Sinclair are considered analogous art, as they are both directed to virtual image reprojections in AR environments. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have integrated the teachings of Sinclair into Ohashi by using a homography matrix that transforms coordinates with sub-pixel accuracy because doing so allows for specifying finer measurement granularity if desired (see Sinclair paragraph 0068).
Ohashi in view of Sinclair fails to further disclose the remaining limitations of claim 5.
Pertaining to the same field of endeavor, Taylor discloses the correction by pixel replacement is a correction using combining information corresponding to the coordinate information as a correction value for pixel replacement (see figure 8, step 824 and paragraph 0100, where the minimum depth value is chosen from the linked list for pixel replacement; each of the depth values in the linked list correspond to depth values from various locations in the transformed image).
Ohashi and Taylor are considered analogous art, as they are both directed to virtual image reprojections in AR environments. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have integrated the teachings of Taylor into Ohashi and Sinclair by using the homography-transformed pixels for replacement because doing so allows for the final rendering to have accurate object occlusion (see Taylor paragraph 0093).
Regarding claim 6, Ohashi fails to disclose the limitations of claim 6.
Pertaining to the same field of endeavor, Sinclair discloses wherein the transformed coordinates are coordinate information with accuracy after a decimal point (paragraph 0068 specifies that the image coordinates can have sub-pixel precision, which reads on accuracy after a decimal point).
Ohashi and Sinclair are considered analogous art, as they are both directed to virtual image reprojections in AR environments. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have integrated the teachings of Sinclair into Ohashi by using a homography matrix that transforms coordinates with sub-pixel accuracy because doing so allows for specifying finer measurement granularity if desired (see Sinclair paragraph 0068).
Ohashi in view of Sinclair fails to further disclose the remaining limitations of claim 6.
Pertaining to the same field of endeavor, Taylor discloses the correction by pixel replacement is a correction using any one of a minimum value, an intermediate value, a maximum value, and a neighboring value of the combining information corresponding to the coordinate information as a correction value for pixel replacement (see figure 8, step 824 and paragraph 0100, where the minimum depth value is chosen from the linked list for pixel replacement).
Ohashi and Taylor are considered analogous art, as they are both directed to virtual image reprojections in AR environments. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have integrated the teachings of Taylor into Ohashi and Sinclair by using the minimum value of the homography-transformed pixels for replacement because doing so allows for the final rendering to have accurate object occlusion (see Taylor paragraph 0093).
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Ohashi (U.S. Publ. US-2021/0191124-A1) in view of Taylor et al. (U.S. Publ. US-2021/0142575-A1), and further in view of Tu et al. (U.S. Publ. US-2019/0235622-A1).
Regarding claim 13, Ohashi in view of Taylor fails to disclose the limitations of claim 13.
Pertaining to the same field of endeavor, Tu discloses wherein the sensing unit is an inertial measurement unit (IMU) (see figure 3, head-mounted device 300, input unit 302 and paragraphs 0051, 0053, where the input unit can include an IMU).
Ohashi and Tu are considered analogous art, as they are both directed to head-mounted devices for augmented reality. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have integrated the teachings of Tu into Ohashi and Taylor by using an IMU in the head-mounted display because IMUs can be used to trace a moving path and orientation of a user in 3D space (see Tu paragraphs 0053, 0063).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICHOLAS JOHN HELCO whose telephone number is (703)756-5539. The examiner can normally be reached on Monday-Friday from 9:00 AM to 5:00 PM.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella, can be reached at telephone number 571-272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center. Status information for published applications may be obtained from Patent Center. Status information for unpublished applications is available through Patent Center for authorized users only. Should you have questions about access to Patent Center, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-automated- interview-request-air-form.
/NICHOLAS JOHN HELCO/Examiner, Art Unit 2667
/MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights