Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4, 8, 11-12, 14, 18, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al (US20170200313) in view of Oh (KR20180116708).
Regarding Claim 1. Lee teaches A projection device comprising:
a projection unit comprising projection circuitry;
a memory storing one or more instructions; and
at least one processor, comprising processing circuitry, configured to execute the one or more instructions, wherein the one or more instructions, when executed by the at least one processor individually and/or collectively, cause the projection device to (Lee, abstract, the invention describes method for providing projection mapping based augmented reality (AR). According to an exemplary embodiment, the apparatus includes an input to acquire real space information and user information; and a processor to recognize a real environment by using the acquired real space information and the acquired user information, map the recognized real environment to a virtual environment, generate augmented content that changes corresponding to a change in space or a user's movement, and project and visualize the generated augmented content through a projector.
[0048] Referring to FIGS. 1 and 2, an apparatus 1 for providing AR includes an input 10, a processor 12, memory 14, and a communicator 16.
[0049] The input 10 acquires, from an input device 2, real space information and user information for the projection in a user's experience environment. The processor 12 generates augmented content by mapping a real environment to a virtual environment based on the acquired real space information and user information which are acquired by the input 10, and projects and visualizes the generated augmented content through a projector 30.):
Lee fails to explicitly teach, however, Oh teaches obtain a first image (Oh, abstract, the invention describes method for providing content for a layered hologram and an apparatus thereof. According to the present invention, a method for providing content of the apparatus for providing content for a layered hologram includes: a step of acquiring a plurality of layers by a user operation; a step of synthesizing a plurality of acquired layers to generate a layered type hologram content; and a step of outputting a selected function to a virtual space corresponding to the selected function when the function is selected by the user operation. According to an embodiment of the present invention, it is possible to solve the disadvantages of the conventional commercialized tool in the production and edition of the content of the layered hologram, and to provide an optimal working environment suitable for the interface environment of a user.
[0082] A content provider (100) for stacked holograms or a web server/application server (200) connected thereto can provide a user interface screen such as FIG. 4 through a terminal (300). In FIG. 4, a virtual space screen (410) for playing stacked hologram content generated by a control unit (320), a screen object (420) placed on the virtual space screen (410), and a plurality of function menus (430, 440, 450, 460, 470) that a user can select to execute a function are provided.
[0091] Referring to FIG. 7(a) and FIG. 7(b), when a user selects a function menu (430) that allows the user to execute a depth adjustment function for the effect layer among the multiple layers required for creating stacked hologram content, such as the main source layer, the effect layer, and the background layer, through an input interface, the fix point for the effect layer is adjusted in response to user operations such as clicking the mouse or dragging while holding down the mouse, thereby adjusting the depth of the stacked hologram content played on the virtual space screen (410).
Therefore, the main source layer is equivalent to the first image.),
Lee and Oh are analogous art because they both teach method of generating 3D virtual image and project to real physical environment. Oh further teaches combining background image and virtual effect image with main source image. Therefore, it would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention, to modify the 3D virtual composite image generation method (taught in Lee), to further combine a target image/video (taught in Oh), so as to generate commercial video with reusable backgrounds, CG, lighting effects and etc. (Oh, [0002-0004]).
The combination of Lee and Oh further teaches obtain spatial information of a screen and location information of a user (Lee, [0053] The input 10 acquires, from the input device 2, point cloud information, user skeleton information, and information of the video that is being played, with regard to real three-dimensional space where the augmented content will be represented. Also, the input 10 acquires information for recognizing and tracking various real objects existing in an experience space.
[0056] The projection visualizer 120 combines virtual space to real space to generate augmented content, and visualizes the generated augmented content through one or more projectors 30 and various displays. To this end, mapping parameters are acquired through a calibration step of linking the input device 2 to the projector 30, so as to calculate the correlation between the real space for projecting the augmented content and a virtual three-dimensional coordinate space. For example, an intrinsic parameter and an extrinsic parameter of the input device 2 and the projector 30 are acquired in the calibration step, and then they are combined together so that the virtual space and the real space may be mapped equally. In addition, in order to process the area overlapped by several projectors, the projection visualizer 120 may expand the space for experience through edge blending, masking, etc., on the image. The above-mentioned processes may be performed based on an association analysis based on various patterns that are used in computer vision.
[0058] When the interaction processor 122 calculates the interaction between the real space and the virtual space by using the information that is acquired by the input 10, and applies it to the augmented content, the projection visualizer 120 represents the augmented content through the projector 30 in the virtual space that is mapped to the real space. The real space may be, for example, the surface of a wall, the surface of a floor, the surface of a three-dimensional item object, and a part of a user's body.
[0061] In order to analyze space needed for the use of an augmented content scenario and recognize and use an object, image processing and machine learning may be used based on depth information acquired by a depth sensor, which is one of input devices, or an iterative closest point, etc., may be used based on point cloud information.
Therefore, in order to correctly project the virtual content in the real physical environment, the positions of the virtual content are mapped to the sensed positions in the real world. Those positions include projection surface position and user position.),
generate a three-dimensional (3D) object image and a background image, based on the spatial information of the screen and the location information of the user (Oh, [0044] In the multiple layer acquisition step (S10), a content creation application for a stacked hologram installed on a user terminal is executed by user operation. Specifically, the user can create a 3D holographic image from rotoscoped and digitally corrected sequence images by downloading the stacked hologram content creation application from the application server via a network, installing it on a terminal, and running it.
[0091] Referring to FIG. 7(a) and FIG. 7(b), when a user selects a function menu (430) that allows the user to execute a depth adjustment function for the effect layer among the multiple layers required for creating stacked hologram content, such as the main source layer, the effect layer, and the background layer, through an input interface, the fix point for the effect layer is adjusted in response to user operations such as clicking the mouse or dragging while holding down the mouse, thereby adjusting the depth of the stacked hologram content played on the virtual space screen (410).), and
control the projection unit to project a second image obtained by synthesizing the first image, the 3D object image, and the background image (Lee, [0056] The projection visualizer 120 combines virtual space to real space to generate augmented content, and visualizes the generated augmented content through one.).
Regarding Claim 2. The combination of Lee and Oh further teaches The projection device of claim 1, wherein the one or more instructions, when executed by the at least one processor individually and/or collectively, further cause the projection device to:
set the 3D object image and the background image, based on a user input (Oh, [0056], a content providing device (100) for stacked holograms can generate stacked hologram content according to a user request as described below and provide the generated stacked hologram content. For example, multiple image layers, such as a target image layer, an effect image layer, and a background image layer, which are not sequence images, can be input through the user terminal (300). The input layer is transmitted to a content provider for stacked holograms (100), and the content provider for stacked holograms (100) can generate and provide stacked hologram content by executing a function according to the user's request.), and
adjust the set 3D object image and background image, based on the location information of the user and the spatial information of the screen (Lee, the invention describes method for providing projection mapping based augmented reality (AR). According to an exemplary embodiment, the apparatus includes an input to acquire real space information and user information; and a processor to recognize a real environment by using the acquired real space information and the acquired user information, map the recognized real environment to a virtual environment, generate augmented content that changes corresponding to a change in space or a user's movement, and project and visualize the generated augmented content through a projector.).
The reasoning for combination of Lee and Oh is the same as described in Claim 1.
Regarding Claim 4. The combination of Lee and Oh further teaches The projection device of claim 1, wherein the one or more instructions, when executed by the at least one processor individually and/or collectively, further cause the projection device to determine depth information about at least one object included in the 3D object image, based on the location information of the user and the spatial information of the screen (Oh, [0094] Referring to FIG. 8(a) and FIG. 8(b), when a user selects a function menu (440) that allows the user to execute a depth adjustment function for the effect layer among multiple layers, such as the main source layer, the effect layer, and the background layer, through an input interface, the depth adjustment (Transform) value corresponding to the movement of at least one axis among the x-axis, y-axis, and z-axis of the screen object is adjusted in response to user operations such as clicking the mouse or dragging while holding down the mouse, and the depth of the stacked hologram content played on the virtual space screen (410) is adjusted based on the adjusted depth adjustment (Transform) value.).
The reasoning for combination of Lee and Oh is the same as described in Claim 1.
Regarding Claim 8. The combination of Lee and Oh further teaches The projection device of claim 1, wherein the one or more instructions, when executed by the at least one processor individually and/or collectively, further cause the projection device to synthesize the first image and at least one object included in the 3D object image, based on a shape and a size of the at least one object (Lee, [0058] When the interaction processor 122 calculates the interaction between the real space and the virtual space by using the information that is acquired by the input 10, and applies it to the augmented content, the projection visualizer 120 represents the augmented content through the projector 30 in the virtual space that is mapped to the real space. The real space may be, for example, the surface of a wall, the surface of a floor, the surface of a three-dimensional item object, and a part of a user's body. In a case of the three-dimensional item object, a three-dimensional background model, which is made in advance and simplified, is trained and then registered; an object location is searched for on space, where augmented content will be represented, by using the data acquired by the input 10; and then the searched object location is replaced with a virtual object mesh that is made in advance and simplified, so the augmented content is presented. Since the location information on space may have a different relative coordinate system depending on each input device 2, the information regarding all the input device 2 is relatively adjusted, calculated, and processed based on the registered three-dimensional background model. As described above, FIG. 4 illustrates an example, in which the interaction processor 122 calculates an interaction according to the progression of an augmented content scenario of the content logic processor 126 based on the information acquired by the input 10, so the augmented content is visualized by the projection visualizer 120.
Therefore, the size and the shape of the projected surface such as a user’s body, a wall, or a 3D object, is obtained and registered, so as to correctly map the coordinates for projecting virtual content.).
Claim 11 is similar in scope as Claim 1, and thus is rejected under same rationale.
Claim 12 is similar in scope as Claim 2, and thus is rejected under same rationale.
Claim 14 is similar in scope as Claim 4, and thus is rejected under same rationale.
Claim 18 is similar in scope as Claim 8, and thus is rejected under same rationale.
Regarding Claim 20. The combination of Lee and Oh further teaches A non-transitory computer-readable recording medium having stored therein a program for performing the method of claim 11 (Lee, [0048] Referring to FIGS. 1 and 2, an apparatus 1 for providing AR includes an input 10, a processor 12, memory 14, and a communicator 16.).
Claims 5, 15 are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al (US20170200313) in view of Oh (KR20180116708) further in view of Wilson et al (US20160098862).
Regarding Claim 5. The combination of Lee and Oh fails to explicitly teach, however, Wilson teaches The projection device of claim 1, wherein the one or more instructions, when executed by the at least one processor individually and/or collectively, further cause the projection device to adjust at least one of the 3D object image or the background image according to a gaze direction of the user, based on a change in the location information of the user (Wilson, abstract, the invention describes method for shared spatial augmented reality (SSAR). Sensor units in a room output sensor signals that are indicative of positions of two or more users in the room and gaze directions of the two or more users. Views of at least one virtual object are computed separately for each of the two or more users, and projectors project such views in the room. The projected views cause the two or more users to simultaneously perceive the virtual object in space.
[0038] Further, since the first and second users 110 and 112 may themselves be projection surfaces, the render component 212 can compute the first view for the first user 110 based upon the position of the second user 112 in the room 100. Likewise, the render component 212 can compute the second view for the second user 112 based upon the location of the first user 110 in the room, wherein the locations of the users 110 and 112 in the room 100 are computed by the tracker component 208. As indicated previously, the tracker component 208 and the render component 212 can compute the positional data 210 and the frames 214, respectively, in real-time, such that imagery projected by the projectors 114-118 is updated as the user 110 and/or the user 112 changes location and/or gaze direction in the room 100.).
Lee, Oh and Wilson are analogous art because they all teach method of generating 3D virtual image and project to real physical environment. Wilson further teaches update the virtual image based on user position and/or gaze direction changes. Therefore, it would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention, to modify the 3D virtual composite image generation method (taught in Lee and Oh), to further update the composite virtual image based on user position and/or gaze direction changes (taught in Wilson), so as to create AR/VR environment shared among plurality of users (Wilson, [0002-0003]).
Claim 15 is similar in scope as Claim 5, and thus is rejected under same rationale.
Claims 6, 16 are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al (US20170200313) in view of Oh (KR20180116708) further in view of Choi et al (KR100630082).
Regarding Claim 6. The combination of Lee and Oh fails to explicitly teach, however, Choi teaches The projection device of claim 1, wherein the one or more instructions, when executed by the at least one processor individually and/or collectively, further cause the projection device to:
obtain at least one of time information or weather information, and
adjust at least one of the 3D object image or the background image, based on at least one of the time information or the weather information (Choi, page 13, par 2, the invention describes methods for displaying background screen in wireless terminal.
Page 18, par 1, In addition, a method for displaying a background image of a mobile terminal to achieve the above objective is characterized by comprising a process of displaying a current background image in standby mode and, upon receiving weather information, changing and displaying a background image corresponding to the weather information as the current background image.).
Lee, Oh and Choi are analogous art because they all teach method of generating virtual background image. The combination of Lee and Oh further teaches composite 3D virtual image and project to real physical environment. Choi further teaches displaying weather information in background image. Therefore, it would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention, to modify the 3D virtual composite image generation method (taught in Lee and Oh), to further update the composite virtual image based on weather information (taught in Choi), so as to display a background image of a mobile terminal that can change and display the background image over time (Choi, page 17, par 1).
Claim 16 is similar in scope as Claim 6, and thus is rejected under same rationale.
Claims 9, 19 are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al (US20170200313) in view of Oh (KR20180116708) further in view of Wu et al (CN110392251).
Regarding Claim 9. The combination of Lee and Oh fails to explicitly teach, however, Wu teaches The projection device of claim 1, wherein the one or more instructions, when executed by the at least one processor individually and/or collectively, further cause the projection device to determine at least one of a projection direction, a projection location, or a projection size of the second image, based on the spatial information of the screen (Wu, abstract, the invention describes a dynamic projection method and system based on virtual reality. The dynamic projection method based on virtual reality comprises the steps of collecting a real space image, and constructing a space three-dimensional model; determining a virtual object to be projected and a real object to be projected and combined by the virtual object, wherein the real object is located in the real space; acquiring characteristic parameters of the real object, and searching a virtual object picture matched with the characteristic parameters according to the characteristic parameters of the real object; and adjusting the projection direction of the projector according to the position relationship between the projector and the real object, and projecting the virtual object picture to the real object in a fixed size. According to the method, the size of the projection image of the virtual object picture can be fixed, virtual reality fusion is coordinated and vivid, and then the user experience is improved.
[0089] The image processing specifically involves compressing, enhancing, restoring, cropping, and matting the matched virtual object image using image processing techniques. This further improves the size, color, pixels, and clarity of the matched virtual object image, resulting in a realistic visual experience when the vertical projection of the virtual object image onto the projection plane merges with the real object. It is understood that after image processing, a virtual object image of optimal size can be obtained. At this time, the projected image corresponding to the virtual object image also reaches the optimal size.
[0090] S40: Adjust the projection direction of the projector according to the positional relationship between the projector and the real object, and project the virtual object image onto the real object at a fixed size.).
Lee, Oh and Wu are analogous art because they all teach method of generating virtual background image and project to real physical environment. Wu further teaches adjusting projection size and/or direction based on positional relationship between the projector and the projected surface. Therefore, it would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention, to modify the 3D virtual composite image generation method (taught in Lee and Oh), to further adjust the projection size and/or direction based on spatial information of the projection surface (taught in Wu), so as to provide a virtual reality-based dynamic projection method that fixes the size of the projected image of a virtual object, achieving harmonious and realistic virtual reality integration, and thus enhances the user experience (Wu, [0008]).
Claim 19 is similar in scope as Claim 9, and thus is rejected under same rationale.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Lee et al (US20170200313) in view of Oh (KR20180116708) further in view of Kim et al (AU2015354933).
Regarding Claim 10. The combination of Lee and Oh fails to explicitly teach, however, Kim teaches The projection device of claim 1, further comprising: at least one camera,
wherein the projection device is configured to obtain the spatial information of the screen and the location information of the user, based on an image captured by the at least one camera (Kim, the invention describes method for providing three-dimensional (3D) map processing and a 3D map service. The electronic device includes a memory configured to store an image set and a map platform module which is functionally connected with the memory and is implemented with a processor. The map platform module is configured to obtain an image set comprising a plurality of images for a path on an external space surrounding the electronic device, to determine an area corresponding to an object included in the external space from at least one of the plurality of images, to obtain information about the object based on whether the object is configured to communicatively connect with the electronic device, and to display the information in association with the area through a display functionally connected with the electronic device.
Page 36, line 23-35, According to an embodiment of the present disclosure, the map generation module 510 may perform image capturing based on a guide direction and motion of the electronic device. For example, the map generation module 510 may determine in which direction from the current virtual point an image has been obtained by the electronic device. The map generation module 510 determines in which direction a camera's focus has to be rotated so that the electronic device may obtain an image in a direction, which has not yet been obtained. The map generation module 510 displays a capturing guide for the direction to the user. If determining that the camera's focus is
oriented in a proper direction according to the guide, the map generation module 510 may automatically capture an image corresponding to the direction. For example, the map generation module 510 may automatically capture the image if the user moves the camera and thus a capturing focus matches a capturing guide. The map generation module 510 also captures an image based on at least one input, if the user moves the camera and thus the capturing focus matches the capturing guide. For example, if the capturing focus matches the capturing guide, the map generation module 510 may highlight the capturing guide to guide the user to make an input with respect to capturing.
Therefore, the projected virtual content is positioned based on the image captured by a camera and user’s movement which changes the direction of camera‘s image capturing.).
Lee, Oh and Kim are analogous art because they all teach method of generating virtual background image and project to real physical environment. Kim further teaches capturing image in the direction of the user’s movement. Therefore, it would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention, to modify the 3D virtual composite image generation method (taught in Lee and Oh), to further capturing image in the direction of the user’s movement and thus determining the position and direction of the virtual content (taught in Kim), so as to provide a map service for a surrounding space (Kim, page 4, line 7-31).
Allowable Subject Matter
Claims 3, 7, 13, 17 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding Claim 3, it recites “The projection device of claim 1, wherein the one or more instructions, when executed by the at least one processor individually and/or collectively, further cause the projection device to: obtain category information of the first image, and determine the background image based on the category information” in the context of Claim 3.
The prior arts of record either alone or in combination fails to teach or suggest the above quoted limitation of Claim 3. Therefore, Claim 3 is allowable over prior art.
Regarding Claim 7, it recites “The projection device of claim 6, wherein the one or more instructions, when executed by the at least one processor individually and/or collectively, further cause the projection device to:
obtain a projection direction, and
adjust a shadow effect appearing on at least one of the 3D object image or the background image, based on the time information and the projection direction” in the context of Claim 7.
The prior arts of record either alone or in combination fails to teach or suggest the above quoted limitation of Claim 7. Therefore, Claim 7 is allowable over prior art.
Claim 13 recites similar limitations as discussed above with regard to claim 3. Therefore, claim 13 is allowable over prior art.
Claim 17 recites similar limitations as discussed above with regard to claim 7. Therefore, claim 17 is allowable over prior art.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Sugiyama et al (US20220076803), abstract, the invention describes a rehabilitation system for performing rehabilitation of higher brain dysfunction, and includes: an image processing apparatus that executes an app for presenting a patient a problem for rehab based on an image using virtual reality, augmented reality, or mixed reality and stores the patient's problem solution record as rehab record information; a practitioner-side terminal that receives the rehab record information from the image processing apparatus; a server that saves the rehab record information transmitted from the practitioner-side terminal ; and a doctor side terminal that receives the rehab record information from the server and displays the state of rehabilitation performed for the patient on the basis of the rehab record information.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XIN SHENG whose telephone number is (571)272-5734. The examiner can normally be reached M-F 9:30AM-3:30PM 6:00PM-8:30PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jason Chan can be reached at 5712723022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Xin Sheng/ Primary Examiner, Art Unit 2619
Prosecution Insights