DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Claims 14-20 are newly added.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 6-9, 11-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kajita (US Pub 2021/0231950) in view of Takahashi (US Pub 2008/0297437).
With respect to claim 1, Kajita discloses an image display device (fig. 1; disclose mixed reality system) comprising: an image sensor configured to capture a real space to acquire a captured image (fig. 1; HMD 101; par 0031; discloses the HMD 101 includes an image capturing unit that captures an image of a physical space); an orientation sensor configured to detect an orientation of the image sensor to acquire orientation information (par 0031; discloses a sensor that measures (measurement processing) the position and orientation of the HMD 101); one or more processors and/or circuitry configured to, (fig. 1; image processing apparatus 104; fig. 5; discloses image processing apparatus 104 include one or more processors 510, 511, 513, 514) on a basis of a position and orientation of the image sensor, which are determined from a first captured image and first orientation information corresponding to the first captured image when the first captured image has been captured, perform a rendering process for rendering a first virtual image representing a virtual space as viewed from a viewpoint corresponding to the position and orientation; (par 0035; discloses the computer apparatus 103 obtains the position and orientation (position and orientation of the image capturing unit of the HMD 101) of the HMD 101 based on a captured image and a position and orientation received from the controller 102, and generates an image of a virtual space viewed from a viewpoint having the obtained position and orientation); and a display configured to display the composite image (fig. 5; discloses HMD includes a display unit 504; par 0037; discloses The computer apparatus 103 generates an image 205 of the mixed reality space as a composite image of the captured image 201 and the image 203 of the virtual space. The computer apparatus 103 transmits the generated image 205 to the HMD 101);
Kajita doesn’t expressly disclose perform a correction process for correcting the first virtual image on a basis of second orientation information corresponding to a second captured image acquired after the first captured image, to acquire a first corrected virtual image; and perform a composition process for combining the second captured image with the first corrected virtual image, to acquire a composite image; and a display configured to display the composite image;
In the same field of endeavor, Takahashi discloses head mounted display and control method (see abstract); Takahashi discloses perform a correction process for correcting the first virtual image on a basis of second orientation information corresponding to a second captured image acquired after the first captured image, to acquire a first corrected virtual image; (par 0049; discloses Images (physical space images) of frames sensed by the camera 203 are sequentially sent to subsequent camera image processing unit 204 and camera viewpoint storage unit 207 (the first memory); i.e. plurality of frame sequentially captured includes first captured image and second images image; fig. 4; point 401 indicates first captured image takes at time t1 and point 403 indicates a second captures image taken at time t3; par 0063; discloses since it takes time until the composite image is generated, the position and orientation of the viewpoint for the CG image used to generate the CG image is different from that of the camera 203 which has sensed the physical space image to be composited with the generated CG image; fig. 7; steps S1005-S1107 discloses the correction steps for adjusting the virtual object based on the new physical space image (i.e. second captured image);) and perform a composition process for combining the second captured image with the first corrected virtual image, to acquire a composite image; and a display configured to display the composite image; (par 0136; discloses step S1006, the viewpoint information calculation unit 210 calculates the shift amount P; par 0137; discloses in step S1007, the CG image position correction unit 211 instructs the composition processing unit 205 to shift the CG image stored in the CG buffer 209 by the shift amount calculated by the viewpoint information calculation unit 210. The composition processing unit 205 composites the CG image stored in the CG buffer 209 on the physical space image (i.e. second captured image) which has been sent from the camera image processing unit 204 at the time of storing the position and orientation information in the CG viewpoint storage unit 208. In this case, assume that the composition position of the CG image is a position shifted from the true composition position by the shift amount calculated by the viewpoint information calculation unit 210. A composite image generated by the composition process is rendered on the display buffer 212; see par 0059-0062 as well;);
Therefore, it would have been obvious to one having ordinary skill in the art to modify the invention disclosed by Kajita to incorporate the teachings of Takahashi to adjust the display position of virtual image based on the changed orientation of the HMD in order to accurately display the virtual image over the physical image in order to compensate for the processing delay caused by the generation of virtual image and change in the user’s head orientation;
With respect to claim 2, Kajita as modified by Takahashi don’t expressly disclose wherein, in the correction process, the first virtual image is corrected based on third orientation information acquired after the second orientation information, to acquire a second corrected virtual image, wherein in the composition process, a third captured image acquired after the second captured image and the second corrected virtual image are combined, to acquire a second composite image, and wherein the display displays the second composite image following the composite image;
Takahashi discloses wherein, in the correction process, the first virtual image is corrected based on third orientation information acquired after the second orientation information, to acquire a second corrected virtual image, wherein in the composition process, a third captured image acquired after the second captured image and the second corrected virtual image are combined, to acquire a second composite image, (par 0063; discloses since it takes time until the composite image is generated, the position and orientation of the viewpoint for the CG image used to generate the CG image is different from that of the camera 203 which has sensed the physical space image to be composited with the generated CG image; par 0060; discloses As indicated by a point 401, the position and orientation of the viewpoint for the CG image is calculated at the time t1; par 0061; discloses As indicated by a point 402, the position and orientation measured by the position and orientation sensor 105 at the time t2, that is, the position and orientation of the viewpoint for the camera is calculated at the time t2 ; par 0062; discloses As indicated by a point 403, the position and orientation measured by the position and orientation sensor 105 at the time t3, that is, the position and orientation of the current viewpoint is calculated at the time t3; par 0082; discloses The CG image position correction unit 211 shifts the CG image stored in the CG buffer 209 by the shift amount calculated by the viewpoint information calculation unit 710, and sends the shifted image to the composition processing unit 705. If the CG image position correction unit 211 receives the shift amount "0", it sends the CG image stored in the CG buffer 209 to the subsequent composition processing unit 705 without any change); and wherein the display displays the second composite image following the composite image (fig. 7; step s1008; displays the composite image on the HMD);
Therefore it would have been obvious to one having ordinary skill in the art to modify the invention disclosed by Kajita as modified by Takahashi to continuously monitor orientation of the HMD and correct the virtual based on the changed orientation as disclosed by Takahashi such that corrected composition image is generated based on the changed orientation and correctly display without any delay.
With respect to claim 3, Kajita as modified by Takahashi don’t expressly disclose wherein, in the correction process, the first corrected virtual image is corrected based on third orientation information acquired after the second orientation information, to acquire a second corrected virtual image, wherein in the composition process, a third captured image acquired after the second captured image and the second corrected virtual image are combined to acquire, a second composite image, and wherein the display displays the second composite image following the composite image;
Takahashi discloses wherein, in the correction process, the first corrected virtual image is corrected based on third orientation information acquired after the second orientation information, to acquire a second corrected virtual image, wherein in the composition process, a third captured image acquired after the second captured image and the second corrected virtual image are combined to acquire, a second composite image, (par 0063; discloses since it takes time until the composite image is generated, the position and orientation of the viewpoint for the CG image used to generate the CG image is different from that of the camera 203 which has sensed the physical space image to be composited with the generated CG image; par 0060; discloses As indicated by a point 401, the position and orientation of the viewpoint for the CG image is calculated at the time t1; par 0061; discloses As indicated by a point 402, the position and orientation measured by the position and orientation sensor 105 at the time t2, that is, the position and orientation of the viewpoint for the camera is calculated at the time t2 ; par 0062; discloses As indicated by a point 403, the position and orientation measured by the position and orientation sensor 105 at the time t3, that is, the position and orientation of the current viewpoint is calculated at the time t3; par 0082; discloses The CG image position correction unit 211 shifts the CG image stored in the CG buffer 209 by the shift amount calculated by the viewpoint information calculation unit 710, and sends the shifted image to the composition processing unit 705. If the CG image position correction unit 211 receives the shift amount "0", it sends the CG image stored in the CG buffer 209 to the subsequent composition processing unit 705 without any change) and wherein the display displays the second composite image following the composite image (fig. 7; step s1008; displays the composite image on the HMD);
Therefore it would have been obvious to one having ordinary skill in the art to modify the invention disclosed by Kajita as modified by Takahashi and Hare to continuously monitor orientation of the HMD and correct the virtual based on the changed orientation as disclosed by Takahashi such that corrected composition image is generated based on the changed orientation and correctly display without any delay.
With respect to claim 6, Kajita as modified by Takahashi and Hare don’t expressly disclose wherein the first orientation information is orientation information acquired by the orientation sensor at a timing that is the same as or closest to a timing at which the image sensor acquires the first captured image, and wherein the second orientation information is orientation information acquired by the orientation sensor at a timing that is the same as or closest to a timing at which the image sensor acquires the second captured image;
Takahashi discloses wherein the first orientation information is orientation information acquired by the orientation sensor at a timing that is the same as or closest to a timing at which the image sensor acquires the first captured image, (par 063; discloses since it takes time until the composite image is generated, the position and orientation of the viewpoint for the CG image used to generate the CG image is different from that of the camera 203 which has sensed the physical space image to be composited with the generated CG image.; par 0060; discloses A time t1 indicates a timing when the PC 102 calculates the position and orientation of the camera 203 (the viewpoint for the CG image) based on the position and orientation information received from the viewpoint information detection unit 202. As indicated by a point 401, the position and orientation of the viewpoint for the CG image is calculated at the time t1.) and the second orientation information is orientation information acquired by the orientation sensor at a timing that is the same as or closest to a timing at which the image sensor acquires the second captured image (par 0061; discloses time t2 indicates a timing when the PC 102 stores the CG image data in the CG buffer 209. The difference between the times t1 and t2 is almost equal to the time taken to create the CG image. As indicated by a point 402, the position and orientation measured by the position and orientation sensor 105 at the time t2, that is, the position and orientation of the viewpoint for the camera is calculated at the time t2);
Therefore it would have been obvious to one having ordinary skill in the art to modify the invention disclosed by Kajita as modified by Takahashi to continuously monitor orientation of the HMD and correct the virtual based on the changed orientation as disclosed by Takahashi such that corrected composition image is generated based on the changed orientation and correctly display without any delay.
With respect to claim 7, Kajita as modified by Takahashi don’t expressly disclose wherein the third orientation information is orientation information acquired by the orientation sensor at a timing that is the same as or closest to the timing at which the image sensor acquires the third captured image;
Takahashi discloses wherein the third orientation information is orientation information acquired by the orientation sensor at a timing that is the same as or closest to the timing at which the image sensor acquires the third captured image (par 0063; discloses since it takes time until the composite image is generated, the position and orientation of the viewpoint for the CG image used to generate the CG image is different from that of the camera 203 which has sensed the physical space image to be composited with the generated CG image; par 0062; discloses A time t3 indicates a display timing of the composite image. The difference between the times t2 and t3 is almost equal to the time taken for the camera image processing and the time taken to composite the physical space image and the CG image. As indicated by a point 403, the position and orientation measured by the position and orientation sensor 105 at the time t3, that is, the position and orientation of the current viewpoint is calculated at the time t3.);
Therefore, it would have been obvious to one having ordinary skill in the art to modify the invention disclosed by Kajita as modified by Takahashi to continuously monitor orientation of the HMD and correct the virtual based on the changed orientation as disclosed by Takahashi such that corrected composition image is generated based on the changed orientation and correctly display without any delay.
With respect to claim 8, Kajita as modified by Takahashi discloses further comprising: a head-mounted display device including the image sensor, the orientation sensor, and the display, (Kajita; fig. 1; HMD 101; par 0031; discloses The HMD 101 includes an image capturing unit that captures an image of a physical space, a sensor that measures (measurement processing) the position and orientation of the HMD 101, and a display unit that displays an image of a mixed reality space transmitted from an image processing apparatus 104); and a image processing device configured to perform the rendering process, wherein the head-mounted display device and the image processing device are communicably connected to each other via a wired or wireless connection (Kajita; fig. 1; image processing apparatus 104; par 0033; discloses the HMD 101 and the controller 102 are connected by wire. However, the connection form between the HMD 101 and the controller 102 is not limited to wire, and may be wireless or a combination of wireless and wire. That is, the connection form between the HMD 101 and the controller 102 is not limited to a specific one; par 0035; discloses The computer apparatus 103 obtains the position and orientation (position and orientation of the image capturing unit of the HMD 101) of the HMD 101 based on a captured image and a position and orientation received from the controller 102, and generates an image of a virtual space viewed from a viewpoint having the obtained position and orientation. The computer apparatus 103 generates a composite image (image of the mixed reality space) of the image of the virtual space and the captured image transmitted from the HMD 101 via the controller 102, and transmits the generated composite image to the controller 102);
Kajita as modified by Takahashi and expressly discloses head-mounted display device configured to perform the correction process and the composition process;
Takahashi discloses head-mounted display device configured to perform the correction process and the composition process (fig. 2; discloses HMD comprises the OG image position correction unit 211 and composition processing unit 205);
Therefore, it would have been obvious to one having ordinary skill in the art to modify the invention disclosed by Kajita as modified by Takahashi to include the correction unit and composition unit within the HMD as disclosed by Takahashi such that processing load is evenly distributed among plurality of connected devices.
With respect to claim 9, Kajita as modified by Takahashi discloses wherein the image sensor includes: a first image sensor configured to acquire a captured image used for combining the composite image (Kajita; fig. 5; image capturing unit 501; par 0044; discloses An image capturing unit 501 captures an image of the physical space that is composited with an image of the virtual space.); and a second image sensor configured to acquire a captured image used for determining the position and orientation (Kajita; fig. 5; image capturing unit 502; par 0046; discloses An image capturing unit 502 includes a plurality of image capturing portions for capturing a marker arranged in the physical space, and obtains captured images as stereo images having a parallax. Par 0036; discloses A captured image 201 includes a marker 202 (the number of markers is one in FIG. 2 for descriptive convenience, but a plurality of markers are included in practice) artificially arranged in the physical space. The computer apparatus 103 extracts the marker 202 from the captured image 201, and obtains the position and orientation of the HMD 101 based on the extracted marker 202).
With respect to claim 11, Kajita as modified by Takahashi discloses wherein the image display device includes a head-mounted display device in which at least the image sensor, the orientation sensor, and the display are provided (Kajita; fig. 5; discloses HMD 101 includes, image capturing unit 501, orientation sensor 503, display unit 504).
With respect to claim 12, Kajita discloses a method of controlling an image display device including an image sensor configured to capture a real space to acquire a captured image, (fig. 1; HMD 101; par 0031; discloses the HMD 101 includes an image capturing unit that captures an image of a physical space) an orientation sensor configured to detect an orientation of the image sensor to acquire orientation information, (par 0031; discloses a sensor that measures (measurement processing) the position and orientation of the HMD 101) and a display, (fig. 5; discloses HMD includes a display unit 504;) the method comprising: rendering a first virtual image representing a virtual space as viewed from a viewpoint corresponding to a position and orientation of the image sensor when a first captured image has been captured, the position and orientation being determined based on the first captured image and first orientation information corresponding to the first captured image; (par 0035; discloses the computer apparatus 103 obtains the position and orientation (position and orientation of the image capturing unit of the HMD 101) of the HMD 101 based on a captured image and a position and orientation received from the controller 102, and generates an image of a virtual space viewed from a viewpoint having the obtained position and orientation); and displaying the composite image on the display (fig. 5; discloses HMD includes a display unit 504; par 0037; discloses The computer apparatus 103 generates an image 205 of the mixed reality space as a composite image of the captured image 201 and the image 203 of the virtual space. The computer apparatus 103 transmits the generated image 205 to the HMD 101);
Kajita doesn’t expressly disclose acquiring a first corrected virtual image by correcting the first virtual image on a basis of second orientation information corresponding to a second captured image acquired after the first captured image; acquiring a composite image by combining the second captured image with the first corrected virtual image;
In the same field of endeavor, Takahashi discloses head mounted display and control method (see abstract); Takahashi discloses acquiring a first corrected virtual image by correcting the first virtual image on a basis of second orientation information corresponding to a second captured image acquired after the first captured image; (par 0049; discloses Images (physical space images) of frames sensed by the camera 203 are sequentially sent to subsequent camera image processing unit 204 and camera viewpoint storage unit 207 (the first memory); i.e. plurality of frame sequentially captured includes first captured image and second images image; fig. 4; point 401 indicates first captured image takes at time t1 and point 403 indicates a second captures image taken at time t3; par 0063; discloses since it takes time until the composite image is generated, the position and orientation of the viewpoint for the CG image used to generate the CG image is different from that of the camera 203 which has sensed the physical space image to be composited with the generated CG image; fig. 7; steps S1005-S1107 discloses the correction steps for adjusting the virtual object based on the new physical space image (i.e. second captured image);) acquiring a composite image by combining the second captured image with the first corrected virtual image, displaying the composite image on the display (par 0136; discloses step S1006, the viewpoint information calculation unit 210 calculates the shift amount P; par 0137; discloses in step S1007, the CG image position correction unit 211 instructs the composition processing unit 205 to shift the CG image stored in the CG buffer 209 by the shift amount calculated by the viewpoint information calculation unit 210. The composition processing unit 205 composites the CG image stored in the CG buffer 209 on the physical space image (i.e. second captured image) which has been sent from the camera image processing unit 204 at the time of storing the position and orientation information in the CG viewpoint storage unit 208. In this case, assume that the composition position of the CG image is a position shifted from the true composition position by the shift amount calculated by the viewpoint information calculation unit 210. A composite image generated by the composition process is rendered on the display buffer 212; see par 0059-0062 as well;);
Therefore, it would have been obvious to one having ordinary skill in the art to modify the invention disclosed by Kajita to incorporate the teachings of Takahashi to adjust the display position of virtual image based on the changed orientation of the HMD in order to accurately display the virtual image over the physical image in order to compensate for the processing delay caused by the generation of virtual image and change in the user’s head orientation;
With respect to claim 13, Kajita discloses a non-transitory computer-readable medium that stores a program, (par 0143; discloses Embodiment(s) of the present invention can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s),) wherein the program causes a computer to execute a method of controlling an image display device, including an image sensor configured to capture a real space to acquire a captured image, (fig. 1; HMD 101; par 0031; discloses the HMD 101 includes an image capturing unit that captures an image of a physical space) an orientation sensor configured to detect an orientation of the image sensor to acquire orientation information, (par 0031; discloses a sensor that measures (measurement processing) the position and orientation of the HMD 101) and a display, (fig. 5; discloses HMD includes a display unit 504;) the method comprising: rendering a first virtual image representing a virtual space as viewed from a viewpoint corresponding to a position and orientation of the image sensor when a first captured image has been captured, the position and orientation being determined based on the first captured image and first orientation information corresponding to the first captured image; (par 0035; discloses the computer apparatus 103 obtains the position and orientation (position and orientation of the image capturing unit of the HMD 101) of the HMD 101 based on a captured image and a position and orientation received from the controller 102, and generates an image of a virtual space viewed from a viewpoint having the obtained position and orientation); and displaying the composite image on the display (fig. 5; discloses HMD includes a display unit 504; par 0037; discloses The computer apparatus 103 generates an image 205 of the mixed reality space as a composite image of the captured image 201 and the image 203 of the virtual space. The computer apparatus 103 transmits the generated image 205 to the HMD 101);
Kajita doesn’t expressly disclose acquiring a first corrected virtual image by correcting the first virtual image on a basis of second orientation information corresponding to a second captured image acquired after the first captured image; acquiring a composite image by combining a second captured image with the first corrected virtual image;
In the same field of endeavor, Takahashi discloses head mounted display and control method (see abstract); Takahashi discloses acquiring a first corrected virtual image by correcting the first virtual image on a basis of second orientation information corresponding to a second captured image acquired after the first captured image; (par 0049; discloses Images (physical space images) of frames sensed by the camera 203 are sequentially sent to subsequent camera image processing unit 204 and camera viewpoint storage unit 207 (the first memory); i.e. plurality of frame sequentially captured includes first captured image and second images image; fig. 4; point 401 indicates first captured image takes at time t1 and point 403 indicates a second captures image taken at time t3; par 0063; discloses since it takes time until the composite image is generated, the position and orientation of the viewpoint for the CG image used to generate the CG image is different from that of the camera 203 which has sensed the physical space image to be composited with the generated CG image; fig. 7; steps S1005-S1107 discloses the correction steps for adjusting the virtual object based on the new physical space image (i.e. second captured image);) acquiring a composite image by combining a second captured image with the first corrected virtual image; displaying the composite image on the display (par 0136; discloses step S1006, the viewpoint information calculation unit 210 calculates the shift amount P; par 0137; discloses in step S1007, the CG image position correction unit 211 instructs the composition processing unit 205 to shift the CG image stored in the CG buffer 209 by the shift amount calculated by the viewpoint information calculation unit 210. The composition processing unit 205 composites the CG image stored in the CG buffer 209 on the physical space image (i.e. second captured image) which has been sent from the camera image processing unit 204 at the time of storing the position and orientation information in the CG viewpoint storage unit 208. In this case, assume that the composition position of the CG image is a position shifted from the true composition position by the shift amount calculated by the viewpoint information calculation unit 210. A composite image generated by the composition process is rendered on the display buffer 212; see par 0059-0062 as well;);
Therefore, it would have been obvious to one having ordinary skill in the art to modify the invention disclosed by Kajita to incorporate the teachings of Takahashi to adjust the display position of virtual image based on the changed orientation of the HMD in order to accurately display the virtual image over the physical image in order to compensate for the processing delay caused by the generation of virtual image and change in the user’s head orientation;
With respect to claim 14, Kajita as modified by Takahashi further discloses wherein the image sensor is configured to sequentially acquire a plurality of captured images including the first captured image and the second captured image (Takahashi; par 0049; discloses Images (physical space images) of frames sensed by the camera 203 are sequentially sent to subsequent camera image processing unit 204 and camera viewpoint storage unit 207).
With respect to claim 15, Kajita as modified by Takahashi further discloses wherein, in the correction process, the first virtual image is corrected on a basis of a change between the first orientation information and the second orientation information (Takahashi; par 0060; discloses As indicated by a point 401, the position and orientation of the viewpoint for the CG image is calculated at the time t1. Par 0062; discloses As indicated by a point 403, the position and orientation measured by the position and orientation sensor 105 at the time t3, that is, the position and orientation of the current viewpoint is calculated at the time t3; par 0136; discloses In step S1006, the viewpoint information calculation unit 210 calculates the shift amount P; par 0137; discloses In step S1007, the CG image position correction unit 211 instructs the composition processing unit 205 to shift the CG image stored in the CG buffer 209 by the shift amount calculated by the viewpoint information calculation unit 210).
With respect to claim 16, Kajita as modified by Takahashi further discloses wherein the second captured image is a latest captured image used in the composition process (Takahashi; par 0063; discloses since it takes time until the composite image is generated, the position and orientation of the viewpoint for the CG image used to generate the CG image is different from that of the camera 203 which has sensed the physical space image to be composited with the generated CG image; par 0137; discloses In step S1007, the CG image position correction unit 211 instructs the composition processing unit 205 to shift the CG image stored in the CG buffer 209 by the shift amount calculated by the viewpoint information calculation unit 210. The composition processing unit 205 composites the CG image stored in the CG buffer 209 on the physical space image which has been sent from the camera image processing unit 204 at the time of storing the position and orientation information in the CG viewpoint storage unit 208. In this case, assume that the composition position of the CG image is a position shifted from the true composition position by the shift amount calculated by the viewpoint information calculation unit 210. A composite image generated by the composition process is rendered on the display buffer 212; hence physical space image used for composition is the second captured image).
With respect to claim 17, Kajita as modified by Takahashi further discloses wherein in the correction process, a plurality of corrected virtual images respectively corresponding to a plurality of captured images acquired after the first captured image are acquired from the first virtual image (Takahashi; par 0135; discloses In step S1005, the CG image data transmitted from the PC 102 is stored in the CG buffer 209; par 0137; discloses In step S1007, the CG image position correction unit 211 instructs the composition processing unit 205 to shift the CG image stored in the CG buffer 209 by the shift amount calculated by the viewpoint information calculation unit 210. Hence the original CG image is used to generate the corrected CG images; par 0049; discloses Images (physical space images) of frames sensed by the camera 203 are sequentially sent to subsequent camera image processing unit 204 and camera viewpoint storage unit 207 (the first memory).).
Claim(s) 4-5, 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kajita (US Pub 2021/0231950) in view of Takahashi (US Pub 2008/0297437) and HER et al (US Pub 2024/0087523).
With respect to claim 4, Kajita as modified by Takahashi don’t expressly disclose wherein a frame rate of the composite image displayed by the display is higher than a frame rate of the virtual image rendered by the rendering process;
In the same field of endeavor, HER discloses a display device and control method (see abstract); HER discloses wherein a frame rate of the composite image displayed by the display is higher than a frame rate of the virtual image rendered by the rendering process (par 0100; discloses the display driving circuit 530 may determine the driving frequency of the display based on the rendering rate of the image data received from the processor 540, and may operate at a lower or higher driving frequency than the rendering rate according to the performance of the display, the user's configuration, and/or the operating mode (e.g., a low power mode). For example, the display module 510 may operate at various driving frequencies such as 30 Hz, 60 Hz, 72 Hz, 80 Hz, 90 Hz, and 120 Hz);
Therefore it would have been obvious to one having ordinary skill in the art to modify the invention disclosed by Kajita as modified by Takahashi to incorporate the teachings of HER to drive the display at higher frequency than the rendering rate such that composite image is displayed properly in accordance with the operating mode of the device.
With respect to claim 5, Kajita as modified by Takahashi don’t expressly disclose wherein a frame rate of the captured image acquired by the image sensor is higher than a frame rate of the virtual image rendered by the rendering process;
In the same field of endeavor, HER discloses a display device and control method (see abstract); HER discloses wherein a frame rate of the captured image acquired by the image sensor is higher than a frame rate of the virtual image rendered by the rendering process; (par 0100; discloses the display driving circuit 530 may determine the driving frequency of the display based on the rendering rate of the image data received from the processor 540, and may operate at a lower or higher driving frequency than the rendering rate according to the performance of the display, the user's configuration, and/or the operating mode (e.g., a low power mode). For example, the display module 510 may operate at various driving frequencies such as 30 Hz, 60 Hz, 72 Hz, 80 Hz, 90 Hz, and 120 Hz);
Therefore it would have been obvious to one having ordinary skill in the art to modify the invention disclosed by Kajita as modified by Takahashi to incorporate the teachings of HER to drive the display at higher frequency than the rendering rate such that composite image is displayed properly in accordance with the operating mode of the device.
With respect to claim 18, Kajita as modified by Takahashi don’t expressly disclose wherein the plurality of corrected virtual images are acquired at a frame rate higher than a frame rate at which virtual images are rendered in the rendering process;
In the same field of endeavor, HER discloses a display device and control method (see abstract); HER discloses wherein the plurality of corrected virtual images are acquired at a frame rate higher than a frame rate at which virtual images are rendered in the rendering process; (par 0100; discloses the display driving circuit 530 may determine the driving frequency of the display based on the rendering rate of the image data received from the processor 540, and may operate at a lower or higher driving frequency than the rendering rate according to the performance of the display, the user's configuration, and/or the operating mode (e.g., a low power mode). For example, the display module 510 may operate at various driving frequencies such as 30 Hz, 60 Hz, 72 Hz, 80 Hz, 90 Hz, and 120 Hz);
Therefore it would have been obvious to one having ordinary skill in the art to modify the invention disclosed by Kajita as modified by Takahashi to incorporate the teachings of HER to drive the display at higher frequency than the rendering rate such that composite image is displayed properly in accordance with the operating mode of the device.
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kajita (US Pub 2021/0231950) in view of Takahashi (US Pub 2008/0297437) and YOON et al (US Pub 2024/0073508).
With respect to claim 10, Kajita as modified by Takahashi don’t expressly disclose wherein a frame rate of the captured image acquired by the second image sensor is lower than a frame rate of the captured image acquired by the first image sensor;
In the same field of endeavor, YOON discloses wearable electronic device and method for controlling camera modules (see abstract); YOON discloses wherein a frame rate of the captured image acquired by the second image sensor is lower than a frame rate of the captured image acquired by the first image sensor (par 0008; discloses the at least one processor may be configured to control the at least one first camera module to obtain first image frames at a first frame rate The at least one processor may be configured to, based on identification, based on at least a part of the first image frames, that a user's hand is disposed in a designated area of the first FOV, control the at least one first camera module to obtain second image frames at a second frame rate greater than the first frame rate. The at least one processor may be configured to control the at least one second camera module to maintain, in an inactive state, the at least one second camera module, or to activate the at least one second camera module and operate the at least one second camera module at the first frame rate);
Therefore, it would have been obvious to one having ordinary skill in the art to modify the invention disclosed by Kajita as modified by Takahashi to incorporate the teachings of YOON to operate one image sensor at lower than the second image sensor in order to continue to capture images while conserving power.
Claim(s) 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kajita (US Pub 2021/0231950) in view of Takahashi (US Pub 2008/0297437) and Gould et al (US Pub 2024/0290037).
With respect to claim 19, Kajita as modified by Takahashi don’t expressly disclose wherein the composition process is performed at a frame rate corresponding to a frame rate at which the plurality of captured images are acquired;
In the same field of endeavor, Gould discloses system and method for generating visual clues (see abstract); Gould discloses wherein the composition process is performed at a frame rate corresponding to a frame rate at which the plurality of captured images are acquired (par 0051; discloses the processing units can execute the render command based on a capture rate of the data, a refresh rate of the display, a rate of change of the pose of the data capture device 108, or some combination thereof. In some embodiments, the processing units can execute the render command at the same rate as the capture rate of the data, the refresh rate of the display, or the rate of change of the pose of the data capture device 108. In some embodiments, the processing units can execute the render command at a rate that is a multiple (e.g., twice, half, etc.) of the capture rate of the data, the refresh rate of the display, or the rate of change of the pose of the data capture device 108. The capture rate of the data is the rate at which data is captured by the data capture device 108.);
Therefore it would have been obvious to one having ordinary skill in the art to modify the invention disclosed by Kajita as modified by Takahashi to incorporate the teachings of Gould to render the augmented visual data corresponding to the rate of the image capture rate such that real time feedback is feedback through the imaging system in order to improve capture of a given subject, thereby enabling enhanced use and output of a given captured image.
Claim(s) 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kajita (US Pub 2021/0231950) in view of Takahashi (US Pub 2008/0297437) and GOLDBERG et al (US Pub 2022/0019801).
With respect to claim 20, Kajita as modified by Takahashi don’t expressly disclose wherein in the correction process, the first virtual image is corrected by performing homography transformation on the first virtual image on a basis of the change between the first orientation information and the second orientation information;
In the same field of endeavor, GOLDBERG discloses system and method for augmented reality (see abstract) ; GOLDBERG discloses wherein in the correction process, the first virtual image is corrected by performing homography transformation on the first virtual image on a basis of the change between the first orientation information and the second orientation information (par 0027; discloses The two or more component transformations can include a transformation from the pose of AR content 102 into a projection 105, consistent with disclosed embodiments. Projection 105 can be a perspective view of object 101, an isometric view of object 101, or the like. As shown in FIG. 1B, projection 105 can be a perspective top-down view onto the upper surface 109 of object 101. The transformation can determine the position and orientation of AR content 102 in projection 105. Par 0029; discloses par 0047; discloses The two or more component transformations can further include a transformation from the perspective of the reference image to the perspective of image 104. In some embodiments, the transformation can be a homography (e.g., a Euclidean or projective homography). par 0037; discloses the projective homography matrix can encode information about changes in position and orientation between the reference image and image 302);
Therefore it would have been obvious to one having ordinary skill in the art to modify the invention disclosed by Kajita as modified by Takahashi to incorporate the teachings of GOLDBERG to use the homography transformation to determine the corrected virtual image such that correct pose for the virtual object is accurately determined and displayed in an augmented reality environment.
Response to Arguments
Applicant's arguments filed with respect to claim 1 have been fully considered but they are not persuasive and do not put the application in condition for allowance.
With respect to claims 1, 12 and 13 applicants representative argued that the cited reference fails to disclose perform a correction process for correcting the first virtual image on a basis of second orientation information corresponding to a second captured image acquired after the first captured image, to acquire a first corrected virtual image; and perform a composition process for combining the second captured image with the first corrected virtual image, to acquire a composite image; and a display configured to display the composite image;
However examiner respectfully disagrees and maintains that the claimed invention is obvious over the cited references. Applicant’s amendment to claim allows examiner to interpret the claim in different perspective and the current rejection for claims 1, 12 and 13 is based on Kajita in view of Takahashi.
Kajita discloses generating a virtual object based on an image and displaying the generated virtual image on the image (see fig. 2);
However Kajita doesn’t expressly disclose perform a correction process for correcting the first virtual image on a basis of second orientation information corresponding to a second captured image acquired after the first captured image, to acquire a first corrected virtual image; and perform a composition process for combining the second captured image with the first corrected virtual image, to acquire a composite image; and a display configured to display the composite image;
Takahashi discloses an head mounted display and control method where Takashi discloses capturing series of images sequentially (par 0049; discloses Images (physical space images) of frames sensed by the camera 203 are sequentially sent to subsequent camera image processing unit 204 and camera viewpoint storage unit 207 (the first memory); i.e. plurality of frame sequentially captured includes first captured image and second images image;)
Takashi discloses generating a virtual image based on the initial image and further corrects the virtual image based on the latest image and viewpoint change;( fig. 4; point 401 indicates first captured image takes at time t1 and point 403 indicates a second captures image taken at time t3; par 0063; discloses since it takes time until the composite image is generated, the position and orientation of the viewpoint for the CG image used to generate the CG image is different from that of the camera 203 which has sensed the physical space image to be composited with the generated CG image; fig. 7; steps S1005-S1107 discloses the correction steps for adjusting the virtual object based on the new physical space image (i.e. second captured image););
Takashi further discloses perform a composition process for combining the second captured image with the first corrected virtual image, to acquire a composite image; and a display configured to display the composite image; (par 0136; discloses step S1006, the viewpoint information calculation unit 210 calculates the shift amount P; par 0137; discloses in step S1007, the CG image position correction unit 211 instructs the composition processing unit 205 to shift the CG image stored in the CG buffer 209 by the shift amount calculated by the viewpoint information calculation unit 210. The composition processing unit 205 composites the CG image stored in the CG buffer 209 on the physical space image (i.e. second captured image) which has been sent from the camera image processing unit 204 at the time of storing the position and orientation information in the CG viewpoint storage unit 208. In this case, assume that the composition position of the CG image is a position shifted from the true composition position by the shift amount calculated by the viewpoint information calculation unit 210. A composite image generated by the composition process is rendered on the display buffer 212; see par 0059-0062 as well;);
Therefore it would have been obvious to one having ordinary skill in the art to modify the invention disclosed by Kajita to incorporate the teachings of Takahashi to adjust the display position of virtual image based on the changed orientation of the HMD in order to accurately display the virtual image over the physical image in order to compensate for the processing delay caused by the generation of virtual image and change in the user’s head orientation; hence the rejection is maintained.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SUJIT SHAH whose telephone number is (571)272-5303. The examiner can normally be reached Monday-Friday, 9:00 am-6:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Eason can be reached at (571)270-7230. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SUJIT SHAH/Examiner, Art Unit 2624
Prosecution Insights