DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Compact Prosecution
With respect to Claim Interpretation, the Examiner has provided some notes regarding “[BRI on the record]” throughout the Office Action, so that the record is clear about the scope of the claimed invention, and the record is also clear about the basis for the Examiner’s analyses. A clear record of the claim interpretation could expedite the examination by creating the condition to allow the examination to focus on Applicant’s inventive concept and its comparison with related prior art.
If there are disagreements, Applicant may present an alternative interpretation based on MPEP 2111. The Examiner will adopt Applicant’s interpretation on the record, if Applicant’s interpretation is reasonable and/or arguments are persuasive.
Applicant may amend claims relying on the Examiner’s claim interpretation provided on the record.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 18 is rejected, because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because Claim 18 recites, “A program causing a computer to execute an information processing method comprising: . . ..” Under BRI, the claimed program is directed to software per se. MPEP 210-6 states, “Products that do not have a physical or tangible form, such as information (often referred to as ‘data per se’) or a computer program per se (often referred to as ‘software per se’) when claimed as a product without any structural recitations; . . ..”
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 5-6 and 11-12 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 5 recites, “a light emission start time of a pixel caused by the distortion of the display.”
However, the distortion of the display does not cause “a light emission start time of a pixel,” because there will be a start time of the light emission of a pixel if an electronic display displays. What the distortion causes actually is a change, e.g., delay, to the start time. Applicant’s Fig. 7 appears to support the Examiner’s concern. Distortion appears to cause “early” and “late” timing, not whether there is a start time. Applicant’s Fig. 7:
PNG
media_image1.png
240
660
media_image1.png
Greyscale
Due to the unclear language in the claim and apparent inconsistency with the specification, the scope of the claim is unclear.
For the purpose of art rejection, the Examiner is reading the limitation to be: “a change of a light emission start time of a pixel caused by the distortion of the display.”
Claim 11 recites “condensation start time of a pixel caused by the distortion of the lens of the camera.” It is unclear what “condensation start time of a pixel” is. The specification does not provides clear guidance. For example, the specification states:
[0117] Assuming that an ideal condensation start time t.sub.C+Δt of pixels constituting a frame image is t.sub.i, and an actual condensation start time is t.sub.a, the coefficient Coef satisfying following equation 5 is calculated according to following equation 6. v represents the change velocity v of the own position/posture that can be calculated according to equation 2 in the first embodiment.
The Examiner searched the internet and PE2E and it does not appear that “condensation start time of a pixel” is a term of art. The plain meaning of the term does not clarify the meaning or scope of the limitation.
For the purposes of art rejection, the Examiner is reading “condensation start time of a pixel caused by the distortion of the lens of the camera” as “a change to a start time of a pixel caused by the distortion of the lens of the camera.”
Claims 6 and 12 are also rejected because they depend on either Claim 5 or 11 and inherit the deficiency.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “an own position/posture estimation unit” and “an image deformation unit” in claim 1; “an image synthesization unit” in claim 14; and “an output image deformation unit” in Claim 15.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 7, and 16-18 are rejected under 35 U.S.C. 103 as being unpatentable over Kajita (US20210063732).
Regarding Claim 1, Kajita teaches An information processing device (
PNG
media_image2.png
534
446
media_image2.png
Greyscale
Kajita Figs. 1, 6) comprising:
an own position/posture estimation unit (Kajita Fig. 6 601) that estimates a position and a posture of a device (HMD 101) based on sensing information acquired by a sensor unit (
PNG
media_image3.png
520
774
media_image3.png
Greyscale
“The position and orientation sensor 601 acquires information for calculating the three-dimensional position and orientation information related to a position and direction of the HMD user's perspective.” Kajita ¶ 79.), and
outputs own position/posture information (output from Kajita Fig. 6 601 to Kajita Fig. 6 602); and
an image deformation unit that performs deformation (Kajita Fig. 7A-C) processing on an image based on the own position/posture information (e.g., active visible area 750 of the optical system) and information (e.g., “correction table” to correct distortion Kajita ¶ 62) related to distortion of an optical system included in the device (
PNG
media_image4.png
166
768
media_image4.png
Greyscale
“FIG. 7C illustrates a case where the display optical system distortion correction is performed in relation to inside the active visible area 750 of the display optical system in the procedure from when the display image data is rendered to when the HMD user observes the display image.” Kajita ¶ 60.
As Fig. 7C shows the distortion correction actually is deformation (Kajita Fig. 7C 706.) performed on an generated image (Kajita Fig. 7C 701.). “The display image data 701 in FIG. 7C is the image data that was rendered without considering the distortion of the display optical system similarly to FIGS. 7A and 7B. The image that is displayed on the display device 303 after having performed display optical system distortion correction processing in the display image processing unit 608 in relation to the active visible area 750 of the display optical system in the display image data 701 is a display image 706.” Kajita ¶ 61.
The active visible area 750 of an HMD is based on the position/posture information of the HMD.
Kajita teaches distortion correction (mapped to deformation) based on correction table, stating “Generally, the wider a target region for the distortion correction is, the larger the size of a correction table used for the correction processing gets, and a processing load also increases. . . . Here, when FIGS. 7B and 7C are compared, it can be confirmed that the target region for the distortion correction is narrower in the processing in FIG. 7C than in the processing in FIG. 7B. Furthermore, from the fact that a distortion amount DIST 1 in the target region for correction in the display image 704 is smaller than a distortion amount DIST 2 in the display image 706, it can be seen that the processing in FIG. 7C requires less memory amount for the distortion correction processing.” Kajita ¶ 62.).
Kajita does not explicitly disclose an active visible area of a HMD is based on the position/posture information of the HMD. The Examiner’s view is that the feature is inherent or implicit. However, the Examiner takes an Official Notice that it would have been well-known in the art that an active visible area of a HMD could be based on the position/posture information. The benefits of combining this well-known knowledge would have been that an immersive experience could be provided for the user who is wearing the HMD. For example, when the user’s head turns left, what the user sees would turn left as well. This responsiveness provides immersive experiences for the user.
Regarding Claim 2, Kajita further teaches The information processing device according to claim 1, wherein the image (Kajita Fig. 7C 701) is an output image generated by a drawing unit by drawing a virtual object based on the own position/posture information (“A CG image 203 is an image rendered by the CG rendering and compositing unit in the image processing apparatus 104 based on the position and orientation information of the HMD 101 and renders a CG object 204 of a virtual space. The CG rendering and compositing unit superimposes the CG image 203 on the captured image 201 to thereby generate an MR image 205, and supplies the MR image 205 to the display unit of the HMD 101 to be displayed.” Kajita ¶ 27.), and the image deformation unit is an output image deformation unit that performs the deformation processing (processing to generate Kajita Fig. 7C 706) on the output image (Kajita Fig. 7C 701) (As Fig. 7C shows the distortion correction is deformation (Kajita Fig. 7C 706.) performed on an generated image (Kajita Fig. 7C 701.). “The display image data 701 in FIG. 7C is the image data that was rendered without considering the distortion of the display optical system similarly to FIGS. 7A and 7B. The image that is displayed on the display device 303 after having performed display optical system distortion correction processing in the display image processing unit 608 in relation to the active visible area 750 of the display optical system in the display image data 701 is a display image 706.” Kajita ¶ 61.).
Regarding Claim 7, Kajita further teaches The information processing device according to claim 2, wherein the output image deformation unit performs distortion correction processing of applying, to the output image (701), distortion opposite to the distortion of the display that is the optical system (
Kajita:
PNG
media_image5.png
160
758
media_image5.png
Greyscale
“A display image 704 is generated by performing processing to distort the display image data in an opposite direction from the distortion in the display optical system 304.” Kajita ¶ 59.).
Regarding Claim 16, Kajita further teaches The information processing device according to claim 1, wherein the device is a head mount display (Kajita Fig. 1, which teaches HMD 101; Fig. 6).
Claims 17-18 are substantially similar to Claim 1. The rejection analyses for Claim 1 based on Kajita are applied to Claims 17-18. In addition, Claim 17 recites “An information processing method comprising: . . .” (Kajita Abstract); and Claim 18 recites “A program causing a computer to execute an information processing method comprising: . . .” (Kajita ¶ 127).
Claims 3-5 are rejected under 35 U.S.C. 103 as being unpatentable over Kajita as applied to Claim 2, in further view of YEUNG et al. (US 20190080505).
Regarding Claim 3, Kajita teaches The information processing device according to claim 2.
Kajita does not explicitly disclose; however, Yeung teaches wherein the output image deformation unit performs delay compensation processing on the output image, the delay compensation processing compensating for delay of display of the output image on a display that is the optical system (“At least a second portion of an image, to be displayed at the second time, is distorted based at least in part on a function of the motion and the latency to compensate for the latency. A first portion of the image is displayed at the first time by activating the first portion of the display device. The second portion of the image, as distorted, is displayed at the second time by activating the second portion of the display device.” Yeung Abstract.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Yeung’s delay correction with Kajita. One of ordinary skill in the art would be motivated to display images more accurately. “At least a second portion of an image, to be displayed at the second time, is distorted based at least in part on a function of the motion and the latency to compensate for the latency.” Yeung Abstract
Regarding Claim 4, Kajita in view of Yeung teaches The information processing device according to claim 3,
wherein the output image deformation unit performs conversion processing such that the output image (Kajita Fig. 7B 701) subjected to the delay compensation processing (Yeung Abstract) is identical to a display result of the display (Kajita Fig. 7B 705) (
Kajita:
PNG
media_image5.png
160
758
media_image5.png
Greyscale
).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Yeung’s delay correction with Kajita. One of ordinary skill in the art would be motivated to display images more accurately. “At least a second portion of an image, to be displayed at the second time, is distorted based at least in part on a function of the motion and the latency to compensate for the latency.” Yeung Abstract
Regarding Claim 5, Kajita in view of Yeung teaches The information processing device according to claim 4,
wherein the conversion processing is performed based on a light emission start time of a pixel caused by the distortion of the display (
[BRI on the record] With respect to “light emission start time of a pixel,” the Examiner is reading the limitation to mean the time when color of a pixel is displayed on a display.
[Mapping Analysis]
Yeung teaches the start time of different regions of a display could be different, stating “For example, if a first portion of the image is shown at a first time and a second portion of the image shown at a second time, where there is motion in between these two times, the second portion of the image may appear distorted from the first portion on a scan-out display in the VR device.” Yeung ¶ 3.
Yeung teaches remedying the latency with a conversion, stating “At least a second portion of an image, to be displayed at the second time, is distorted based at least in part on a function of the motion and the latency to compensate for the latency. A first portion of the image is displayed at the first time by activating the first portion of the display device. The second portion of the image, as distorted, is displayed at the second time by activating the second portion of the display device.” Yeung Abstract.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Yeung’s delay correction with Kajita. One of ordinary skill in the art would be motivated to display images more accurately. “At least a second portion of an image, to be displayed at the second time, is distorted based at least in part on a function of the motion and the latency to compensate for the latency.” Yeung Abstract
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Kajita in view of Yeung as applied to Claim 5, in further view of Aga et al. (“Latency Compensation for Optical See-Through Head-Mounted with Scanned Display”).
Regarding Claim 6, Kajita in view of Yeung teaches The information processing device according to claim 5, wherein the conversion processing (reversion of a function of the motion and the latency to compensate for the latency) is performed based on a difference (the latency = second time – first time) between an ideal value (e.g., first time, ideal for second region when first region is displayed) and a real value (e.g., second time, real/actual time when the second region is displayed) of the light emission start time per pixel (
Yeung teaches remedying the latency with a conversion, stating “At least a second portion of an image, to be displayed at the second time, is distorted based at least in part on a function of the motion and the latency to compensate for the latency. A first portion of the image is displayed at the first time by activating the first portion of the display device. The second portion of the image, as distorted, is displayed at the second time by activating the second portion of the display device.” Yeung Abstract.
The latency is determined/calculated for the disclosed function.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Yeung’s delay correction with Kajita. One of ordinary skill in the art would be motivated to display images more accurately. “At least a second portion of an image, to be displayed at the second time, is distorted based at least in part on a function of the motion and the latency to compensate for the latency.” Yeung Abstract.
Kajita in view of Yeung does not explicitly disclose; however, Aga teaches conversion processing is performed using information obtained by calculating a difference between an ideal value and a real value of displaying time of a pixel (
Aga 3.5.1:
PNG
media_image6.png
256
628
media_image6.png
Greyscale
Here, t=0 is idea value, because this is when there is no latency; and “t” is the calculated difference between idea time value (0) and actual latency (t).).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Aga’s delay correction with Kajita in view of Yeung. One of ordinary skill in the art would be motivated to display images more accurately. “Latency Compensation for Optical See-Through Head-Mounted with Scanned Display.” Aga Title.
Claims 8-15 are rejected under 35 U.S.C. 103 as being unpatentable over Kajita as applied to Claim 1, in further view of Xiong et al. (WO 2020181409 A1).
Regarding Claim 8, Kajita teaches The information processing device according to claim 1.
Kajita does not explicitly disclose; however, Xiong teaches wherein the image is an input image captured by a camera that is the optical system, and the image deformation unit is an input image deformation unit that performs the deformation processing on the input image (
“For example, when the first camera and the second camera are Rolling Shutter cameras, the same 3D point P may be in different exposure lines due to the movement of the mobile platform in the image, resulting in the same 3D point P at different times. The exposure time in the image is also different. The movement distance d in the above embodiment can be compensated according to the difference in the exposure time of the target object in the first image 31 and the third image 33. Optionally, the compensated d can be expressed as d=(t2- t1+Δt)*speed, where Δt represents the difference in the exposure time of the target object in the first image 31 and the third image 33.” Xiong p. 12.
Rolling shutter refers to a distortion effect of an input image captured by a camera.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Xiong’s input image correction with Kajita. One of ordinary skill in the art would be motivated to display images more accurately. “This embodiment compensates for the movement distance of the movable platform through the different exposure time of the target object in the images captured by the same camera at different times, reduces the influence of the rolling shutter camera on the parameter calibration of the camera, and further improves the shooting.” Xiong p. 12.
Regarding Claim 9, Kajita in view of Xiong teaches The information processing device according to claim 8, wherein the input image deformation unit performs rolling shutter distortion correction processing on the input image (
“For example, when the first camera and the second camera are Rolling Shutter cameras, . . .. The movement distance d in the above embodiment can be compensated according to the difference in the exposure time of the target object in the first image 31 and the third image 33.” Xiong p. 12.), the rolling shutter distortion correction processing correcting distortion of a lens of the camera of a rolling shutter system (The Examiner takes an Official Notice that it would have been well-known in the art that the distortion of a “Rolling Shutter camera” could be caused by lens of such a camera. The benefits of combining this well-known knowledge would have been that distortion caused by a specific type of camera system could be corrected through image processing.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Xiong’s input image correction with Kajita. One of ordinary skill in the art would be motivated to display images more accurately.
Regarding Claim10, Kajita in view of Xiong teaches The information processing device according to claim 9, wherein the input image deformation unit performs conversion processing such that the input image subjected to the rolling shutter distortion correction processing is identical to an expected correction result (“For example, when the first camera and the second camera are Rolling Shutter cameras, . . .. The movement distance d in the above embodiment can be compensated according to the difference in the exposure time of the target object in the first image 31 and the third image 33.” Xiong p. 12. After the correction, the output image achieves its expected correction result by being identical to an expected correction result.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Xiong’s input image correction with Kajita. One of ordinary skill in the art would be motivated to display images more accurately.
Regarding Claim 11, Kajita in view Xiong teaches The information processing device according to claim 10, wherein the conversion processing is performed based on a condensation start time of a pixel caused by the distortion of the lens of the camera (
[BRI] The claim has been rejected under 112(b) for being indefinite. For the purposes of art rejection, the Examiner is reading the limitation to be “time when a pixel is captured by the lens of the cameras, and the time is related to the distortion.”
[Mapping Analysis]
“For example, when the first camera and the second camera are Rolling Shutter cameras, the same 3D point P may be in different exposure lines due to the movement of the mobile platform in the image, resulting in the same 3D point P at different times. The exposure time in the image is also different. The movement distance d in the above embodiment can be compensated according to the difference in the exposure time of the target object in the first image 31 and the third image 33. Optionally, the compensated d can be expressed as d=(t2- t1+Δt)*speed, where Δt represents the difference in the exposure time of the target object in the first image 31 and the third image 33.” Xiong p. 12.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Xiong’s input image correction with Kajita. One of ordinary skill in the art would be motivated to display images more accurately.
Regarding Claim 12, Kajita in view of Xiong teaches The information processing device according to claim 11, wherein the conversion processing is performed using information obtained by calculating a difference between an ideal value and a real value of the condensation start time per pixel (
[BRI] The claim has been rejected under 112(b) for being indefinite. For the purposes of art rejection, the Examiner is reading the limitation “the condensation start time per pixel” as “the capturing time of the pixel.”
[Mapping Analysis]
“For example, when the first camera and the second camera are Rolling Shutter cameras, the same 3D point P may be in different exposure lines due to the movement of the mobile platform in the image, resulting in the same 3D point P at different times. The exposure time in the image is also different. The movement distance d in the above embodiment can be compensated according to the difference in the exposure time of the target object in the first image 31 and the third image 33. Optionally, the compensated d can be expressed as d=(t2- t1+Δt)*speed, where Δt represents the difference in the exposure time of the target object in the first image 31 and the third image 33.” Xiong p. 12.
Rolling shutter refers to a distortion effect of an input image captured by a camera.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Xiong’s input image correction with Kajita. One of ordinary skill in the art would be motivated to display images more accurately. “This embodiment compensates for the movement distance of the movable platform through the different exposure time of the target object in the images captured by the same camera at different times, reduces the influence of the rolling shutter camera on the parameter calibration of the camera, and further improves the shooting.” Xiong p. 12.
Regarding Claim 13, Kajita in view of Xiong teaches The information processing device according to claim 8, wherein the input image deformation unit performs distortion correction processing of applying, to the input image, distortion opposite to the distortion of the lens of the camera (
“For example, when the first camera and the second camera are Rolling Shutter cameras, . . .. The movement distance d in the above embodiment can be compensated according to the difference in the exposure time of the target object in the first image 31 and the third image 33.” Xiong p. 12. Here, the compensation is distortion opposite to the initial distortion to cancel the distortion in the final image.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Xiong’s input image correction with Kajita. One of ordinary skill in the art would be motivated to display images more accurately.
Regarding Claim 14, Kajita in view of Xiong teaches The information processing device according to claim 8, further comprising an image synthesization unit that synthesizes the input image deformed by the input image deformation unit (Xiong’s Roller Shutter effect correction) and an output image generated by a drawing unit by drawing a virtual object based on the own position/posture information, and generates a synthesized image (Kajita’s mixed reality (MR) image) (
“For example, when the first camera and the second camera are Rolling Shutter cameras, . . .. The movement distance d in the above embodiment can be compensated according to the difference in the exposure time of the target object in the first image 31 and the third image 33.” Xiong p. 12.
“A CG image 203 is an image rendered by the CG rendering and compositing unit in the image processing apparatus 104 based on the position and orientation information of the HMD 101 and renders a CG object 204 of a virtual space. The CG rendering and compositing unit superimposes the CG image 203 on the captured image 201 to thereby generate an MR image 205, and supplies the MR image 205 to the display unit of the HMD 101 to be displayed.” Kajita ¶ 27.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Xiong’s input image correction with Kajita. One of ordinary skill in the art would be motivated to display images more accurately.
Regarding Claim 15, Kajita in view of Xiong teaches The information processing device according to claim 14, further comprising an output image deformation unit that performs the deformation processing on the synthesized image (
“Display image data 701 is the display image data generated in the CG rendering and compositing unit 607 and should be understood as image data rendered without considering a distortion of the display optical system. The image that is displayed on the display device 303 without display optical system distortion correction processing being performed in the display image processing unit 608 in relation to the display image data 701 is a display image 702.” Kajita ¶ 57.
PNG
media_image3.png
520
774
media_image3.png
Greyscale
Fig. 6 shows a synthetic image from 607 undergoes correction within 608 based on deformation. “The display image data 701 in FIG. 7C is the image data that was rendered without considering the distortion of the display optical system similarly to FIGS. 7A and 7B. The image that is displayed on the display device 303 after having performed display optical system distortion correction processing in the display image processing unit 608 in relation to the active visible area 750 of the display optical system in the display image data 701 is a display image 706.” Kajita ¶ 61.).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Ubillos (US 20120092559 A1) provides detailed explanation about rolling shutter effect, a feature related to Claims 9-10:
“Complementary metal oxide semi-conductor (CMOS)-based camera sensors have a rolling shutter, as opposed to having a global shutter like, e.g., charge-coupled device (CCD)-based camera sensors. Consequently, pixels of a CMOS-based camera sensor are read top-to-bottom, left-to-right, such that the data read at the top of the frame is acquired at a point in time different than the time when the data at the bottom of the frame was acquired. Accordingly, when the camera happens to be moving (e.g., panning) during the roller shutter sequence the resulting image will appear to be tilted. For example, if the camera is panning left (or right) while the shutter is rolling, the resulting image will appear to be tilted to the left (or right). As another example, when panning up (or down), the image appears stretched (or compressed).” Ubillos ¶ 3.
Zhuang et al. (CN 109788189 B) teaches correcting rolling shutter effect, a feature related to Claims 9-10:
“The distortion grid is determined using intra-frame three-dimensional rotation and intra-frame residual two-dimensional translation to compensate for rolling shutter (RS) effects.” Zhuang p. 12.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ZHENGXI LIU whose telephone number is (571)270-7509. The examiner can normally be reached M-F 9 AM - 5 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at 571-272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ZHENGXI LIU/Primary Examiner, Art Unit 2611
Prosecution Insights