DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Double Patenting
Claims 1, 6-8, 13, and 18 provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-2, 7-9, and 14 of copending Application No. 18/426,202 (reference application) in view of Abe et al. (US 20200043240 A1) (Hereinafter referred to as Abe). It should be note that the main distinction that Claim 1 in the current application has over Claim 1 of the copending application is that it additionally requires a determination to be made on whether or not to a use an intermediate image while drawing the virtual environment. However, Abe teaches a trigger (See Abe [0254]) that transitions the view image from camera perspective mode (intermediate image) to avatar perspective mode (without intermediate image). This trigger can be viewed as a determination being made regarding the usage of the intermediate image, and would thus teach the additional limitation required by current Application.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify copending Application No. 18/426,202 with Abe to include a determination made to use the intermediate image.
The motivation to combine would have been because the copending Application and Abe are within the same field of capturing an image using a camera and augmenting the captured image with virtual objects (See Abe Abstract). The benefit of including a determination of using an intermediate image is that it can increase the user’s immersion. See Abe [0235], “FIG. 36 is an explanatory diagram for describing a camera perspective mode and an avatar perspective mode. . . By introducing such an avatar perspective mode, the user's sense of immersion into the AR space is heightened.”
This is a provisional nonstatutory double patenting rejection because the patentably indistinct claims have not in fact been patented. Note that the copending Application No. 18/426,202 is published as (US 20240278113 A1).
Claim Mapping between the current application and Application No. 18/426,202
Current Application
1
6
7
8
13
18
App No. 18/426,202
1 and Abe
2
7 and Abe
8 and Abe
9
14
Below is a Limitation Mapping between Claim 1 of the current application and Claim 1 of Application No. 18/426,202
Current Application
Application No. 18/426,202
An apparatus comprising: one or more memory devices configured to store instructions; and one or more processors, that upon execution of the instructions, are configured to:
A display image generation device comprising: one or more processors; and one or more memory devices storing instructions that, upon execution by the one or more processors, configure the display image generation device to
acquire data of an image captured by from a camera;
acquire data of an image captured by a camera;
arrange a virtual object in a virtual three-dimensional space; draw an image of the virtual object;
generate an intermediate image representing a virtual object arranged in a three-dimensional space for a display object, with the camera as a viewpoint;
synthesize a display image from the image of the virtual object and the acquired image; and
generate a composite image representing the intermediate image and the captured image, with a virtual camera for display as a viewpoint;
provide data of the display image, wherein the one or more processors are configured to determine whether to use an intermediate image while drawing the image of the virtual object according to a state of a three-dimensional space to be displayed including the virtual object, the intermediate image representing the virtual object from a viewpoint of the camera.
See Abe
Claim Objections
Claims 13 and 18 objected to because of the following informalities:
Claims 13 and 18 recites the limitation, “providing data of the display image to the HMD comprising the camera.” There is a lack of antecedent basis for HMD as the base claims of 13 and 18 do not contain a mention of an HMD. Also, HMD needs to be spelled out before its abbreviation can be used. Note that Claim 6, which recites similar limitations as to Claim 13 and 18, does exactly that. Thus the limitations of Claim 13 and 18 should read, “providing data of the display image to a head-mounted display (HMD) comprising the camera.” Appropriate correction is required.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-3, 6-10, 13-15 and 18 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Abe et al. (US 20200043240 A1) (Hereinafter referred to as Abe).
Regarding Claim 1, Abe discloses An apparatus comprising: one or more memory devices configured to store instructions; and one or more processors, that upon execution of the instructions, are configured to: (See Fig. 3 showing the hardware configuration of an apparatus with memory and a processor.)
acquire data of an image captured by from a camera; (See [0103], “The camera of the information processing device 100a generates a captured image by capturing the real space 11a.”)
arrange a virtual object in a virtual three-dimensional space; (See [0105], “In the example in FIG. 1, the virtual object A11 is an avatar of the user Ua. The avatar A11 is placed within an augmented reality space (AR space) set in association with the real space 11a, and may execute various actions.”)
draw an image of the virtual object; (See [0252], “The object control unit 146 ordinarily generates an image of an AR space by taking the position and orientation of the camera 102 as a reference.”)
synthesize a display image from the image of the virtual object and the acquired image; and (See [0103], “The AR application receives a captured image depicting a real space as an input image, and outputs an output image superimposed with a virtual object to a display.” Note that a superimposed captured image with the virtual object would correspond to a synthesized “display image”.)
provide data of the display image, wherein the one or more processors are configured to determine whether to use an intermediate image while drawing the image of the virtual object according to a state of a three-dimensional space to be displayed including the virtual object, the intermediate image representing the virtual object from a viewpoint of the camera. (See [0252], “The object control unit 146 ordinarily generates an image of an AR space by taking the position and orientation of the camera 102 as a reference. Such a display mode is designated camera perspective mode in this specification. In contrast, in avatar perspective mode, the object control unit 146 generates an image of an AR space by taking the position and orientation of an avatar placed within the AR space as a reference.” Note that the limitation of using an intermediate image while drawing the image of the virtual object would correspond to the “camera perspective mode” taught by Abe as this mode would generate an image of the AR space from the viewpoint of the camera.
Also see [0254], “In FIG. 37, the trigger T3 is a trigger for transitioning from camera perspective mode to avatar perspective mode. The trigger T3 may include the occurrence of a designated event and detecting designated user input. The trigger T4 is a trigger for transitioning from avatar perspective mode to camera perspective mode. The trigger T4 may include the end of an occurring event, detecting designated user input, and the elapse of a fixed amount of time. A designated event may include, for example, communication between avatars as described using FIGS. 27 and 28, collision between an avatar and another virtual object (for example, a block tossed by another avatar), as well as attacking and defending in a competitive multiplayer game.” Note that since there are triggers for transition between the different perspective modes of camera or avatar, this can be considered as a determination being made on whether to use an intermediate image while drawing the image of the virtual object.)
Regarding Claim 2, Abe discloses The apparatus according to claim 1, wherein one or more processors are configured to draw the image of the virtual object without using the intermediate image when the virtual object enters a predetermined range from another virtual object that is configured to be drawn without using the intermediate image. (See [0254], “In FIG. 37, the trigger T3 is a trigger for transitioning from camera perspective mode to avatar perspective mode. The trigger T3 may include the occurrence of a designated event and detecting designated user input. . . . A designated event may include, for example, communication between avatars as described using FIGS. 27 and 28, collision between an avatar and another virtual object (for example, a block tossed by another avatar), as well as attacking and defending in a competitive multiplayer game.” Also see Figs. 27 and 28 showing different avatars being in a range of each other for communication.
Once again, camera perspective mode corresponds to drawing the image using the intermediate image, and subsequently the avatar perspective mode corresponds to drawing the image without using the immediate image. Here, Abe teaches that there can be a transition from a camera perspective mode to an avatar perspective mode (draw the image of the virtual object without using the intermediate image) when there is a designated event, such as communication between avatars. As seen from Figs. 27 and 28, communication between avatars can be considered as a virtual object entering a range of another virtual object, noting the avatars are considered as the virtual objects. Lastly, note when the other avatar is another user operating with the system of Abe, then other avatar (another virtual object) would similarly have a transition between a camera perspective mode to avatar perspective mode when avatars are communicating, and thus be considered as being “configured to be drawn without using the intermediate image.”)
Regarding Claim 3, Abe discloses The apparatus according to claim 1, wherein the one or more processors are configured to arrange the virtual object at a position designated by a user in the three-dimensional space to be displayed and (See [0105], “In the example in FIG. 1, the virtual object A11 is an avatar of the user Ua. The avatar A11 is placed within an augmented reality space (AR space) set in association with the real space 11a, and may execute various actions.” Also see [0301], “FIG. 44 is an explanatory diagram for describing a first example of a technique for setting an initial position of a virtual object in an AR space.”
Lastly see Fig. 44 showing that the user can designated an initial position for their avatar (virtual object) which is in the three-dimensional space to be displayed.)
the processor is configured to draw the image of the virtual object without using the intermediate image when the virtual object is configured to be drawn without using the intermediate image. (See [0254], “In FIG. 37, the trigger T3 is a trigger for transitioning from camera perspective mode to avatar perspective mode. The trigger T3 may include the occurrence of a designated event and detecting designated user input. . . . A designated event may include, for example, communication between avatars as described using FIGS. 27 and 28, collision between an avatar and another virtual object (for example, a block tossed by another avatar), as well as attacking and defending in a competitive multiplayer game.” In this case, the image can be in avatar perspective mode (drawing without using the intermediate image) when there is a designated event that triggers the transition (the virtual object is configured to be drawn without using the intermediate image).)
Regarding Claim 6, Abe discloses The apparatus according to claim 1, wherein the one or more processors are configured to provide data of the display image to a head-mounted display (HMD) comprising the camera. (See Fig. 1 showing a head-mounted display HMD. Also see [0057], “As shown in FIG. 2, an image viewing system 1 of the present embodiment includes a display terminal 10 that displays an image of a virtual space, an image processing apparatus 100 that provides the image of the virtual space to the display terminal 10, and an image transmission terminal 200 that transmits an image captured in a real space to the image processing apparatus 100.”
Also see [0103], “The camera of the information processing device 100a generates a captured image by capturing the real space 11a.” Lastly, see [0058], “The image transmission terminal 200 is configured of, for example, a computer (for example, a personal computer) having a communication function, a portable terminal (for example, a smartphone, a mobile phone, or a tablet terminal), and an imaging device (for example, a digital camera) having a communication function. These configurations themselves are known, so that detailed description thereof will be omitted.”)
Regarding Claim 7, Abe discloses A method comprising: (See [0002], “The present disclosure relates to an information processing device, a client device, an information processing method, and a program.”)
acquiring data of an image from a camera; arranging a virtual object in a virtual three-dimensional space; drawing an image of the virtual object, comprising determining whether to use an intermediate image according to a state of a three-dimensional space to be displayed including the virtual object, wherein the intermediate image comprises the virtual object from a viewpoint of the camera; synthesizing a display image from the image of the virtual object and the acquired image; and providing data of the display image. (The above limitations are similar to those of Claim 1 and are therefore rejected under a similar rationale as that of Claim 1.)
Regarding Claim 8, Abe discloses A non-transitory, computer-readable storage medium containing a computer program, which when executed by a computer, causes the computer to carry out actions, comprising: (See [0368], “As one example, programs composed of software that realizes such series of processes are stored in advance on a storage medium (non-transitory medium) provided internally in or externally to such apparatuses.”)
acquiring data of an image from a camera; arranging a virtual object in a virtual three-dimensional space; drawing an image of the virtual object, comprising determining whether to use an intermediate image according to a state of a three-dimensional space to be displayed including the virtual object, wherein the intermediate image comprises the virtual object from a viewpoint of the camera; synthesizing a display image from the image of the virtual object and the acquired image; and providing data of the display image. (The above limitations are similar to those of Claim 1 and are therefore rejected under a similar rationale as that of Claim 1.)
Regarding Claim 9, Claim 9 contains similar limitations as to Claim 2 and therefore is also rejected under a similar rationale as that of Claim 2.
Regarding Claim 10, Claim 10 contains similar limitations as to Claim 3 and therefore is also rejected under a similar rationale as that of Claim 3.
Regarding Claim 13, Claim 13 contains similar limitations as to Claim 6 and therefore is also rejected under a similar rationale as that of Claim 6.
Regarding Claim 14, Claim 14 contains similar limitations as to Claim 2 and therefore is also rejected under a similar rationale as that of Claim 2.
Regarding Claim 15, Claim 15 contains similar limitations as to Claim 3 and therefore is also rejected under a similar rationale as that of Claim 3.
Regarding Claim 18, Claim 18 contains similar limitations as to Claim 6 and therefore is also rejected under a similar rationale as that of Claim 6.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 5, 12, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Abe in view of Shimizu (US 20220337805 A1).
Regarding Claim 5, Abe discloses The apparatus according to claim 1, wherein the one or more processors are further configured to represent, on a plane of the display image, the intermediate image when the image of the virtual object is drawn using the intermediate image. (See [0103], “The AR application receives a captured image depicting a real space as an input image, and outputs an output image superimposed with a virtual object to a display. In the example in FIG. 1, a virtual object A11 is superimposed onto an output image Im11 as though the virtual object A11 were standing on top of the table R10a.”
Also see [0252], “The object control unit 146 ordinarily generates an image of an AR space by taking the position and orientation of the camera 102 as a reference. Such a display mode is designated camera perspective mode in this specification.”)
However, Abe fails to explicitly disclose wherein the one or more processors are further configured to represent on a plane of the display image, the acquired image projected onto a projection surface set in the virtual three-dimensional space and
further configured to represent, on the plane of the display image, the intermediate image represented on the projection surface when the image of the virtual object is drawn using the intermediate image.
Shimizu teaches represent, on a plane of the display image, the acquired image projected onto a projection surface set in the virtual three-dimensional space and (See [0071], “The image projection unit 121 performs processing of projecting a captured image obtained from image data on (the projection surface of) the full spherical screen Sc in a three-dimensional virtual space.”
Also see [0100], “That is, in this image processing, the image processing unit 104 of the reproduction device 10 performs processing of projecting (S12) an image (captured image) captured from a specific position (positions of cameras such as the full spherical camera 21 and the stereo imaging cameras 22-1 and 22-2) in a real space on a projection surface (full spherical screen Sc) arranged according to a reference position (position corresponding to the center Ct of the full spherical screen Sc) set in a virtual space . . .”)
further configured to represent, on the plane of the display image, the intermediate image represented on the projection surface when the image of the virtual object is drawn using the intermediate image. (See [0100] teaching a projection surface set in a virtual space. In combination with Abe [0252] already teaching to generate an image from the position and orientation of the camera (intermediate) and outputting a superimposed image of the captured image with the virtual objects, then the above limitations are taught as the superimposed image would be represented on the projection surface.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Abe with Shimizu to include having the acquired image projected onto a projection surface set in the virtual three-dimensional space and to also have the intermediate image represented on the projection surface.
The motivation to combine Abe with Shimizu would have been obvious as both arts are within the same field involving the display of virtual spaces and head-mounted displays (See Shimizu Abstract). Shimizu simply shows that the technique of having a projection surface set in virtual three-dimensional space for the acquired image to be projected onto is common and well-known, especially for image processing within head-mounted displays (See Shimizu [0100]).
Regarding Claim 12, Claim 12 contains similar limitations as to Claim 5 and therefore is also rejected under a similar rationale as that of Claim 5.
Regarding Claim 17, Claim 17 contains similar limitations as to Claim 5 and therefore is also rejected under a similar rationale as that of Claim 5.
Allowable Subject Matter
Claims 4, 11, and 16 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding Claim 4, the cited prior art does not disclose or render obvious the combination of elements cited in the claims as a whole. Specifically, the cited prior art fails to disclose or render obvious the limitations: wherein the one or more processors are configured to draw the image of the virtual object without using the intermediate image when the virtual object is based on a template provided by middleware. Thus Claim 4 contains allowable subject matter.
Regarding Claim 11, Claim 11 contains similar limitations as to Claim 4 and thus also contains similar allowable subject matter.
Regarding Claim 16, Claim 16 contains similar limitations as to Claim 4 and thus also contains similar allowable subject matter.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THANG G HUYNH whose telephone number is (571)272-5432. The examiner can normally be reached Mon-Thu 7:30am-4:30pm EST | Fri 7:30am-11:30am EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/T.G.H./Examiner, Art Unit 2611
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611
Prosecution Insights