DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments/amendments filed on May 20, 2026 have been fully considered.
Applicant’s arguments with respect to claims 1, 2, and 12 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Objections
Claim 1 is objected to because of the following informalities:
The phrase “the user” in line 21 of claim 1 lacks antecedent basis.
For the purposes of this examination the term “the user” in line 21 is interpreted as “the use” consistent with corresponding method claim 2, and CRM claim 12.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 7-12 are rejected under 35 U.S.C. 103 as being unpatentable over Ohashi (US 20200125312 A1, hereinafter “Ohashi”) in view of Lee et al. (US 20200302664 A1, hereinafter “Lee”).
Regarding claim 1,
Ohashi teaches:
An image processing apparatus (Ohashi: ¶23, “. . . an image generating apparatus 200 . . .”) comprising:
at least one processor (Ohashi: ¶27, “. . . control unit 10 is a main processor that processes and outputs signals such as image signal and sensor signal and instructions and data.. . .”)
that functions to:
acquire a real image captured by an image capturing device(Ohashi: ¶40-41, “. . . An HDMI transmitting-receiving unit 280 receives video of a real space photographed by the camera unit 80. . .”; NOTE: The frames in the video captured of a real space is are real images.);
generate a first layer image including the acquired real image (Ohashi: ¶63, “. . . The camera unit 80 of the head-mounted display 100 photographs the external world and outputs a Raw image (S10). . .”; ¶3, “. . .generate and display video of augmented reality (AR) by superimposing objects of a virtual world generated based on computer graphics (CGs) on the video of the external world photographed by the camera. . .”; NOTE: The raw image of the real world is the first layer image, and is at the bottom of the synthesis because the superimposed objects are displayed on top of the real world image.)
and output the generated first layer image to each of an image synthesis unit and an arithmetic unit; (Ohashi: ¶Fig. 6, NOTE: The raw image captured by the camera, that is used as the first layer image is processed and simultaneously sent to each of:
a synthesis unit: (Which is the AR superposition S26 implemented by an AR superimposing unit (para 73). In Fig. 6, the branch of steps S10-S22-S24. . . S32, the lower branch, receives the outputted real image, which becomes processes the. This is the low-delay path as the raw image does not get extra image processing in comparison to the upper branch S10-S12. . .S32 path); and
an arithmetic unit: (Which is the components corresponding to S10-S12-S16-S18-S20-S24. . . S32 corresponding to orientation estimation and calculation of point of view and physical operation blocks in Fig. 6, and described in paragraph 63-64) where the raw real image undergoes extra processing as described in S12-S20. This path is the high-delay path due to processing time before it gets to be displayed.)
generate, by the arithmetic unit, a converted image of the acquired real image by controlling an image characteristic level of the acquired real image (Ohashi: ¶88, “. . . chroma key processing may be executed after the step S86 in the case of using chroma key synthesis in order to superimpose the CG image on the camera image. As described above, in this case, the chroma key generating unit 244 generates a chroma key image from the CG image based on the depth information of the camera image that has a low resolution and involves delay. . .”; ¶55, “chroma key generating unit 244 generates a chroma key image . . . . . . generates a chroma key image obtained by painting out, with specific one color (for example red), . . . The chroma key image is generated by using the camera image. . .”; NOTE: S86 correspond to the distortion processing block as shown in Fig. 8, S32 in Figure 6. The chroma key is based on depth information, therefore, the chroma key processing is generated by the high delay path S10-S12-S16. . .S32, which are the components constituting the arithmetic unit. The chroma key image is the converted image because it is converted by controlling an image characteristic level (by painting out portions of the real image for the chroma key, with the level being the size of the holes or the painted out portions in the chroma key image).),
wherein the converted image or the real image are output as an second intermediate layer image (NOTE: The chroma key image is the second intermediate layer because the virtual object is superimposed (being the top layer), and the chroma key image is used so that the real image is visible. The layer arrangement in Ohashi is:
first layer bottom (real image from the camera),
second intermediate layer (chroma key image because it directly situated between the virtual object and the real image,
third layer top (CG image or the virtual object superimposed);
generate, by the arithmetic unit, a virtual image including computer graphics for output as a third layer image (Ohashi: ¶49, “. . . the augmented reality image obtained by superimposing the virtual objects on the image of the real space looks natural and smooth. . .”; NOTE: As also cited above in reference to paragraph 88, the CG image, which is the virtual image of a virtual object, is superimposed on the real camera image; with an intermediate chroma key image for masking. The bottom first layer is the real camera image, the second intermediate layer is the chroma key image, and the top third layer is the CG image being superimposed.);
synthesize a display image (Ohashi: Fig. 6, Image output), combining the first layer image with the third layer image with a second intermediate layer being either the converted image or the acquired real image; (Ohashi: ¶67, “. . .The AR superimposing unit 234 generates an augmented realty image by superimposing the CG image on the camera image (S24). . .”; ¶88, “. . . using chroma key synthesis in order to superimpose the CG image on the camera image. As described above, in this case, the chroma key generating unit 244 generates a chroma key image. . .; NOTE: The AR superimposing unit synthesizes all layers, by combining the first layer which is the real camera image, the chroma key image as the second intermediate layer, and the CG image superimposed as the third layer for display.)
display the synthesized display image by a display device (Ohashi: ¶33-34, “The HDMI transmitting-receiving unit 90 receives an image generated by the image generating apparatus 200 . . . can supply an image or text data to the output interface 30 to cause the display panel 32 to display it. . .” ¶22, “. . . a generating method of an image displayed on the head-mounted display 100 . . .”;),
and changes a delay time of the real image being displayed on the display device (Ohashi: Fig. 6, ¶80, “. . . Processing for update of computer graphics, such as calculation of the point of view and physical operation of virtual objects, is executed based on the orientation estimation data (S48). . .”; ¶88, “. . . the chroma key generating unit 244 generates a chroma key image from the CG image based on the depth information of the camera image that has a low resolution and involves delay. . .”; NOTE: In reference to Figure 6, the upper branch of the raw image processing, because of the processing steps S16-S18-S20, these processes takes time to complete and would inherently cause the output to be delayed. Therefore, by processing the raw real image to generate a chroma key image, the delay of the real image being displayed on the display device is changed. Also in reference to Fig. 8, the rendering S20 processes the n-1 frame of the video feed, physically delaying the display by 1 frame. );
However, Ohashi’s fails to disclose if second intermediate layer, which is Ohashi’s Chroma Key Image is selectable by a user, and therefore fails to disclose: wherein the use of either the converted image or the acquired real image as the second intermediate layer is selectable by a user.
The analogous art Lee teaches:
wherein the use of either the converted image or the acquired real image as the second intermediate layer is selectable by a user (Lee: ¶22, “. . . The image processor is configured to select at least one of the background image layer, the object image layer, and the third image layer. . .”; “. . . FIG. 7A and FIG. 7B, three image layers A 620, B 630, and C 640 . . . The image layer B 630 may be created as a 2D object image layer. The image layer B 630 includes an object area (e.g., botted region, which is an extracted object from the captured images) and a remaining area processed to be transparent.”; ¶86, “. . . AR user device 100 may overlay one image layer over the other image layer . . . significantly improve the 3D effect and the depth of field effect without requiring high processing power and lengthy processing time, which is essential for processing a live image to produce augmented reality. . .”; ¶44, “. . . A 3D object may be overlaid and overlapped with at least one adjacent image layer in order to improve the 3D effect and depth of filed. The multiple image layers of the AR image may be separately displayed on displays in response to a user selection. . .”; NOTE: The image layer B of Lee constitutes a converted image because it is an image that is converted to have transparent areas, similar to Ohashi’s Chroma Key Image. It is a second intermediate layer because it is in between layers A and C. The multiple image layers can be separately displayed in response to a user selection, including image layer B. Therefore, image layer B, which is the second intermediate layer, is selectable. )
It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to combine Ohashi, and Lee, and include: wherein the use of either the converted image or the acquired real image as the second intermediate layer is selectable by a user.
The reason for doing so is to “significantly improve the 3D effect and the depth of field effect without requiring high processing power and lengthy processing time, which is essential for processing a live image to produce augmented reality” (Lee: ¶86).
Regarding claim 2, method claim 2 is drawn to the method corresponding to the configuration of the processor of using same as claimed in apparatus claim 1. Therefore, method claim 2 corresponds to the configuration of the processor in the apparatus of claim 1, and is rejected for the same reasons of obviousness as used above.
Regarding claim 12, CRM claim 12 is drawn to the CRM corresponding to the configuration of the processor of using same as claimed in apparatus claim 1. Therefore, CRM claim 12 corresponds to the configuration of the processor in the apparatus of claim 1, and is rejected for the same reasons of obviousness as used above.
Regarding claim 3, depending on 2,
The combination of Ohashi and Lee teaches:
The method of image processing according to Claim 2, wherein, the synthesizing includes combining the virtual image output in the generating, the real image or the converted image output in the generating, and the real image output in the acquiring in this order. (Ohashi: ¶48, “. . . The AR superimposing unit 234 generates an augmented reality image by superimposing the CG image . . . on the camera image. . .”; ¶55, “. . . The chroma key image is used in order to superimpose the CG image on the camera image to generate an augmented reality image. . .”; NOTE: The virtual image (CG image) is on top of the real image. Since the converted image (chroma key image) is used to superimpose the CG image, therefore, the CG image and the chroma key image is on top of the real image. Top is the CG image , Middle is the chroma key image, bottom is the real image (camera image).
Regarding claim 4, depending on 2
The combination of Ohashi and Lee teaches:
The method of image processing according to Claim 2, wherein the generating of the converted image includes changing a delay time of the real image in the image displayed on the display device based on whether the real image is output or not. (NOTE: As discussed in claim 1, Ohashi’s low-delay path displays the real camera image with low-delay. In the high-delay path, the real image is being processed for chroma key generation, and is not output until the chroma key is generated to be synthesized for outputting an AR image, and have a higher-delay to be displayed. Therefore, Ohashi’s low-delay path displays the real image with minimal delay, and the real image is output; and the high-delay path does not output the real image and delay increases due to the chroma key generating processing times. Therefore, depending on which path is displayed, the delay time is changed whether the real image is output (low delay path) or not (high delay path).
Regarding claim 7, depending on 2,
The combination of Ohashi and Lee teaches:
The method of image processing according to Claim 2, wherein the generating of the converted image includes outputting, as the converted image, an image in which color of part or whole of the received real image is converted (Ohashi: ¶55, “. . . generates a chroma key image obtained by painting out, with specific one color (for example red), the background of the virtual objects and part of the objects of the real space existing on the front side relative to the virtual objects in the CG image. . .”; NOTE: The generated chroma key image have a specific color for the parts where the CG is to be mapped.)
Regarding claim 8, depending on 7,
The combination of Ohashi and Lee teaches:
The method of image processing according to Claim 7, wherein the generating of the converted image includes outputting, as the converted image, an image in which the color of part or the whole of the received real image is converted to a color representing chromakey information (NOTE: As discussed in claim 7 rejection, Ohashi uses a specific color for its chroma key, for example, the color red as disclosed in paragraph 55)
Regarding claim 9, depending on 2,
The combination of Ohashi and Lee teaches:
The method of image processing according to Claim 2,
Although Ohashi teaches automatic switching (Paragraph 88 discloses that chroma key processing is conditional in a case of using chroma key synthesis, therefore, it can switch between chroma key processing which is the high-delay path, and switches to a low-delay path if there is no chroma key processing) between outputting the real image (low-delay path) or outputting the converted image (high-delay path), Ohashi further teaches an input interface that accepts setting signal from a user. Therefore, Ohashi fails to teach: wherein the generating of the converted image includes switching between outputting the real image or outputting the converted image based on user instructions.
It would have been obvious an obvious design choice among finite number of solutions (automatic switching by the system or manual switching by a user) to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to include wherein the generating includes switching between outputting the real image or outputting the converted image based on user instructions.
The reason for doing so is to provide options to allow users to select a preferred display path between a high-delay display and a low-delay path display.
Regarding claim 10, depending on 2,
The combination of Ohashi and Lee teaches:
The method of image processing according to Claim 2, further comprising controlling whether to output the real image or the converted image in the generating based on a state of the image output in the generating (Ohashi: ¶55, “A chroma key generating unit 244 generates a chroma key image from a CG image based on the depth information of a camera image. Specifically, the chroma key generating unit 244 determines the positional relationship between objects of a real space and objects of a virtual space and generates a chroma key image . . . by painting out, with specific one color (for example red), the background of the virtual objects and part of the objects of the real space existing on the front side relative to the virtual objects in the CG image.”; ¶88, “. . . chroma key processing may be executed after the step S86 in the case of using chroma key synthesis in order to superimpose the CG image on the camera image. . .”; NOTE: The chroma key processing is executed on a system logic as described in paragraph 88. The chroma key image is the converted image. The state of which the chroma key generation is executed is based on the positional information such as if a CG is supposed to be in front of a real object in the real world. A chroma key is then generated for that portion so the CG can be superimposed properly, this is the high-delay path. If occlusion is not required such as determined if a CG is supposed to be showing behind a real object in the real space, then, chroma key processing is not be executed and display via the low-delay path the real camera image. Therefore, the output is controlled where the real image is output via the low-delay path if occlusion is not required, and output the chroma key image if occlusion is required.).
Regarding claim 11, depending on 2,
The combination of Ohashi and Lee teaches:
The method of image processing according to Claim 2, wherein the acquiring includes adjusting and outputting the real image acquired from the image capturing device (Ohashi: ¶41, “The image signal processing unit 250 executes image signal processing (ISP) such as RGB conversion (demosaic processing), white balance, color correction, and noise reduction for a Raw image photographed by the camera unit 80 of the head-mounted display 100, and executes distortion correction processing of removing distortion and so forth due to the optical system of the camera unit 80. The image signal processing unit 250 supplies an RGB image for which the image signal processing and the distortion correction processing have been executed to an image generating unit 230”; NOTE: The real image, which is the raw image from the camera, is adjusted by processing the image with RGB conversion, white balance, and other image processing methods disclosed in paragraph 41. After adjusting the raw camera image, the real image converted to RGB is then output and supplied to image generating unit. Camera takes real image >> adjusted by image process i.e. color correction, noise reduction etc. >> output and supplied to image generating unit.).
Claims 5 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over Ohashi (US 20200125312 A1, hereinafter “Ohashi”) in view of Lee (US 20200302664 A1, hereinafter “Lee”) and further in view of Proksch et al. (US 2022/0215539, hereinafter “Proksch”).
Regarding claim 5, depending on 2,
The combination of Ohashi and Lee teaches:
The method of image processing according to Claim 2, wherein the generating of the converted image includes outputting, as the converted image, an image in which part or whole of the received real image is converted (Ohashi: ¶55, “. . . chroma key generating unit 244 generates a chroma key image from a CG image . . . generates a chroma key image obtained by painting out, . . . part of the objects of the real space existing on the front side relative to the virtual objects in the CG image. . .”; NOTE: The parts of the real image that are painted out are the part of the received real image in which transparency is converted. The painted out portion of the chroma key is for mapping the CG image relative to the camera image so it can be superimposed in the correct position. Therefore, by painting out the parts of the real space to generate a chroma key image, the transparency of part or whole of the received real image is converted.
Ohashi does not teach: wherein the generating of the converted image includes outputting, as the converted image, an image in which transparency of part or whole of the received real image is converted.
Proksch teahes: a converted image with a transparent region.(paragraph 66 augmentation region 502 is more transparent) of the real image (fig. 5) is converted,
Therefore, it would have been obvious to a person with ordinary skill in the art to have modifed Ohashi as modified to include: wherein the generating of the converted image includes outputting, as the converted image, an image in which transparency of part or whole of the received real image is converted.
The reason of doing so would have allowed the user to see through a layer of a combined image for understand the image combining process.
Regarding claim 6, depending on 5
The combination of Ohashi, Lee, and Proksch teaches:
The method of image processing according to Claim 5,
Although Ohashi teaches painting out parts of the real image during chroma key generation process in which part or whole of the received real image is converted, (see rejection of claim 5).
Ohashi fails to disclose wherein the generating of the converted image includes outputting, as the converted image, an image in which part or the whole of the received real image is made transparent.
Proksch teaches: a converted image in which part or the whole of the received real image (fig. 5) is made transparent (paragraph 66 augmentation region 502 is more transparent).
Therefore, it would have been obvious to a person with ordinary skill in the art to have modified Ohashi as modified to include: wherein the generating of the converted image includes outputting, as the converted image, an image in which part or the whole of the received real image is made transparent.
The reason of doing so would have allowed the user to see through a layer of a combined image for understand the image combining process.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PATRICK GALERA whose telephone number is (571)272-5070. The examiner can normally be reached Mon-Fri 0800-1700 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at 571-270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PATRICK P GALERA/Examiner, Art Unit 2617 /KING Y POON/Supervisory Patent Examiner, Art Unit 2617
Prosecution Insights