DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1 and 5 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Koperwas et al. (US 2024/0420430).
Regarding independent claim 1, Koperwas teaches a method of generating an augmented reality (AR) image (Figures 2 – 4), comprising:
generating, by a mobile device, based on a camera image and sensor data collected from a camera and a sensor mounted on the mobile device (paragraph 28: As used herein, location data (sometimes referred to as position data) refers to data that locates a location or position, such as can be indicated by GPS coordinates, of the mobile device, while orientation data refers to data that indicates the direction the camera and display of the mobile device are facing at a given location; paragraph 33: When app 528 is executing on the mobile device, telemetry information and other data related to frames in video stream captured by cameras 522 can be sent to the immersive content server at a known frequency rate which can render virtual content that is sent back to the mobile device), pose information of the mobile device (paragraph 39: Once collected, the telemetry information can be wirelessly communicated to the immersive content server from the mobile device) and geometric information of the camera image (paragraph 46: The immersive content server can also receive depth information about objects in the environment collected by depth sensors embedded in the mobile device (e.g., a Lidar sensor));
generating, by a rendering server, a color image (paragraph 47: The rendering module 552 can generate images for an item of content to be displayed by mobile device 520 based on the sensor input, positional information, calibration information, visual information, and other information from the other modules) and a depth image (paragraph 34: Depth information for the first and second virtual objects generated by the immersive content system 540 enables app 528 to place the virtual objects within the composited image at specific distances from the participant; paragraph 49: The image data can include red, green, blue, alpha (R,G,B,A) data, depth data, and/or shadow data) of a virtual object based on the pose information (paragraph 47: Rendering module 552 can render images from the virtual environment in a way that matches the view of a participant holding mobile device 520 based, in part, on the location and orientation information provided by location module 550) and 3D scene data of the virtual object (paragraph 47: Rendering module can then position a virtual camera at the identified position, oriented in the same lines as the camera of mobile device 520 to render the virtual content); and
generating, by the mobile device, a mixed reality image based on the color image (paragraph 71: When mobile device 620 receives the rendered image from server 640, the mobile device can composite together the rendered image with a view of the real world on the display of the device), the depth image (paragraph 34: Depth information for the first and second virtual objects generated by the immersive content system 540 enables app 528 to place the virtual objects within the composited image at specific distances from the participant), the camera image (paragraph 71: the virtual content from the virtual environment that is in the field of view of the participant, is displayed on the mobile device display), and the geometric information (paragraph 71: Some portions of the rendered image can be hidden or partially behind objects of the background camera image through the depth channel and other portions of the rendered image can be made transparent or partially transparent through the alpha channel data).
Regarding independent claim 5, Koperwas teaches a system for generating an augmented reality (AR) image (Figures 1 and 4), comprising:
a mobile device (Figure 4: AR Viewer Client 102); and a rendering server (Figure 4: AR Content Server 108),
the mobile device, based on a camera image and sensor data collected from a camera and a sensor mounted on the mobile device (paragraph 28: As used herein, location data (sometimes referred to as position data) refers to data that locates a location or position, such as can be indicated by GPS coordinates, of the mobile device, while orientation data refers to data that indicates the direction the camera and display of the mobile device are facing at a given location; paragraph 33: When app 528 is executing on the mobile device, telemetry information and other data related to frames in video stream captured by cameras 522 can be sent to the immersive content server at a known frequency rate which can render virtual content that is sent back to the mobile device), configured to generate pose information of the mobile device (paragraph 39: Once collected, the telemetry information can be wirelessly communicated to the immersive content server from the mobile device) and geometric information of the camera image (paragraph 46: The immersive content server can also receive depth information about objects in the environment collected by depth sensors embedded in the mobile device (e.g., a Lidar sensor)),
the rendering server configured to generate a color image (paragraph 47: The rendering module 552 can generate images for an item of content to be displayed by mobile device 520 based on the sensor input, positional information, calibration information, visual information, and other information from the other modules) and a depth image (paragraph 34: Depth information for the first and second virtual objects generated by the immersive content system 540 enables app 528 to place the virtual objects within the composited image at specific distances from the participant; paragraph 49: The image data can include red, green, blue, alpha (R,G,B,A) data, depth data, and/or shadow data) of a virtual object based on the pose information and (paragraph 47: Rendering module 552 can render images from the virtual environment in a way that matches the view of a participant holding mobile device 520 based, in part, on the location and orientation information provided by location module 550) 3D scene data of the virtual object (paragraph 47: Rendering module can then position a virtual camera at the identified position, oriented in the same lines as the camera of mobile device 520 to render the virtual content); and
the mobile device configured to generate a mixed reality image based on the color image (paragraph 71: When mobile device 620 receives the rendered image from server 640, the mobile device can composite together the rendered image with a view of the real world on the display of the device), the depth image (paragraph 34: Depth information for the first and second virtual objects generated by the immersive content system 540 enables app 528 to place the virtual objects within the composited image at specific distances from the participant), the camera image (paragraph 71: the virtual content from the virtual environment that is in the field of view of the participant, is displayed on the mobile device display), and the geometric information (paragraph 71: Some portions of the rendered image can be hidden or partially behind objects of the background camera image through the depth channel and other portions of the rendered image can be made transparent or partially transparent through the alpha channel data).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 2 and 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Koperwas et al. (US 2024/0420430) in view of Ha et al. (US 2018/0293745).
Regarding dependent claim 2, Koperwas teaches <<does not disclose>> wherein generating of the color image and the depth image includes:
performing, by the rendering server, a render pass based on the pose information (paragraph 47: Rendering module 552 can render images from the virtual environment in a way that matches the view of a participant holding mobile device 520 based, in part, on the location and orientation information provided by location module 550) and the 3D scene data of the virtual object (paragraph 47: Rendering module can then position a virtual camera at the identified position, oriented in the same lines as the camera of mobile device 520 to render the virtual content) to generate the color image (paragraph 47: The rendering module 552 can generate images for an item of content to be displayed by mobile device 520 based on the sensor input, positional information, calibration information, visual information, and other information from the other modules), a scene depth (paragraph 46: The immersive content server can also receive depth information about objects in the environment collected by depth sensors embedded in the mobile device (e.g., a Lidar sensor)), and <<a scene normal vector>>; and
performing an AR render pass based on the color image (paragraph 33: App 528 can then composite the virtual content with the video stream as discussed below. In this manner, the user (participant) can view the virtual content within the real-world physical environment visible within the field of view of the mobile device camera), the scene depth (paragraph 46: The immersive content server can also receive depth information about objects in the environment collected by depth sensors embedded in the mobile device (e.g., a Lidar sensor)), and <<the scene normal vector>> to generate the depth image (paragraph 34: Depth information for the first and second virtual objects generated by the immersive content system 540 enables app 528 to place the virtual objects within the composited image at specific distances from the participant). Koperwas does not expressly disclose generating the scene normal vector from a render pass and generating the depth image from the scene normal vector. Ha discloses The depth image refining apparatus 130 refines depth values of the depth image 125 to generate the refined depth image (paragraph 51), where the depth image refining apparatus 130 distinguishes between a noise region and an edge region in the depth image 125 based on a characteristic of a surface normal distribution represented in the color image 115, and effectively reduces a noise component while maintaining an edge characteristic substantially unchanged by applying filters having different characteristics to the noise region and the edge region in the process of refining the depth image 125 (paragraph 52). It would have been obvious for one of ordinary skill in the art at the time of the invention (pre-AIA ) or at the time of the effective filing date of the application (AIA ) to modify Koperwas's system to obtaining surface normal from the image values to refine/produce depth values. One would be motivated to do so because this help reduce noise from the sensor images and input images.
Regarding dependent claim 6, Koperwas teaches <<does not disclose>> wherein the rendering server is configured to: perform a render pass based on the pose information (paragraph 47: Rendering module 552 can render images from the virtual environment in a way that matches the view of a participant holding mobile device 520 based, in part, on the location and orientation information provided by location module 550) and the 3D scene data of the virtual object (paragraph 47: Rendering module can then position a virtual camera at the identified position, oriented in the same lines as the camera of mobile device 520 to render the virtual content) to generate the color image (paragraph 47: The rendering module 552 can generate images for an item of content to be displayed by mobile device 520 based on the sensor input, positional information, calibration information, visual information, and other information from the other modules), a scene depth (paragraph 46: The immersive content server can also receive depth information about objects in the environment collected by depth sensors embedded in the mobile device (e.g., a Lidar sensor)), and <<a scene normal vector>>; and
perform an AR render pass based on the color image (paragraph 33: App 528 can then composite the virtual content with the video stream as discussed below. In this manner, the user (participant) can view the virtual content within the real-world physical environment visible within the field of view of the mobile device camera), the scene depth (paragraph 46: The immersive content server can also receive depth information about objects in the environment collected by depth sensors embedded in the mobile device (e.g., a Lidar sensor)), and <<the scene normal vector>> to generate the depth image (paragraph 34: Depth information for the first and second virtual objects generated by the immersive content system 540 enables app 528 to place the virtual objects within the composited image at specific distances from the participant). Koperwas does not expressly disclose generating the scene normal vector from a render pass and generating the depth image from the scene normal vector. Ha discloses The depth image refining apparatus 130 refines depth values of the depth image 125 to generate the refined depth image (paragraph 51), where the depth image refining apparatus 130 distinguishes between a noise region and an edge region in the depth image 125 based on a characteristic of a surface normal distribution represented in the color image 115, and effectively reduces a noise component while maintaining an edge characteristic substantially unchanged by applying filters having different characteristics to the noise region and the edge region in the process of refining the depth image 125 (paragraph 52). It would have been obvious for one of ordinary skill in the art at the time of the invention (pre-AIA ) or at the time of the effective filing date of the application (AIA ) to modify Koperwas's system to obtaining surface normal from the image values to refine/produce depth values. One would be motivated to do so because this help reduce noise from the sensor images and input images.
Claim(s) 3 and 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Koperwas et al. (US 2024/0420430) in view of Ponto et al. (US 2020/0126208).
Regarding dependent claim 3, Koperwas does not expressly disclose wherein the depth image of the virtual object is a colorized depth image. Ponto discloses a head mounted display set (paragraph 60) and the color of each point is based on the (Euclidean) distance from the point having the previous index value, with only points being shown where the distance is greater than a threshold (paragraph 77). It would have been obvious for one of ordinary skill in the art at the time of the invention (pre-AIA ) or at the time of the effective filing date of the application (AIA ) to modify Koperwas's system to display a virtual object in color coded depth pixels from scanning data. One would be motivated to do so because this would help recognize irregularities in the depth data (paragraph 87).
Regarding dependent claim 7, Koperwas does not expressly disclose wherein the depth image of the virtual object is a colorized depth image. Ponto discloses a head mounted display set (paragraph 60) and the color of each point is based on the (Euclidean) distance from the point having the previous index value, with only points being shown where the distance is greater than a threshold (paragraph 77). It would have been obvious for one of ordinary skill in the art at the time of the invention (pre-AIA ) or at the time of the effective filing date of the application (AIA ) to modify Koperwas's system to display a virtual object in color coded depth pixels from scanning data. One would be motivated to do so because this would help recognize irregularities in the depth data (paragraph 87).
Claim(s) 4 and 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Koperwas et al. (US 2024/0420430) in view of Ponto et al. (US 2020/0126208) and Kaser et al. (US 2012/0284214).
Regarding dependent claim 4, Koperwas does not expressly disclose wherein the depth image of the virtual object is represented on a log scale when depth information of a specific point included in the depth image of the virtual object is greater than a predetermined reference value. Ponto discloses a head mounted display set (paragraph 60) and the color of each point is based on the (Euclidean) distance from the point having the previous index value, with only points being shown where the distance is greater than a threshold (paragraph 77). Kaser discloses the 2D number line in 2D is presented as a 2D view of a 3D space comprising a z-axis component configured such that the spacing of intervals on the number line correctly represents their numerical value and corresponds to a predetermined z depth allowing the line to delineate logarithmic increments (paragraph 63 and Figures 5A – 5C). It would have been obvious for one of ordinary skill in the art at the time of the invention (pre-AIA ) or at the time of the effective filing date of the application (AIA ) to modify Koperwas's system to represent depth image of the virtual object in color based on Ponto’s system that displays color coded depth pixels of a scanned object after a threshold depth range and further modify the color coded depth to be represented in a logarithmic scale based on the concept of Kaser that color codes depth in an image in a logarithmic scale. One would be motivated to do so because this would help recognize irregularities in the depth data (Ponto, paragraph 87) and allow the user to perceive delineations not perceptible using a 2D model (Kaser, paragraph 63).
Regarding dependent claim 8, Koperwas does not expressly disclose wherein the depth image of the virtual object is represented on a log scale when depth information of a specific point included in the depth image of the virtual object is greater than a predetermined reference value. Ponto discloses a head mounted display set (paragraph 60) and the color of each point is based on the (Euclidean) distance from the point having the previous index value, with only points being shown where the distance is greater than a threshold (paragraph 77). Kaser discloses the 2D number line in 2D is presented as a 2D view of a 3D space comprising a z-axis component configured such that the spacing of intervals on the number line correctly represents their numerical value and corresponds to a predetermined z depth allowing the line to delineate logarithmic increments (paragraph 63 and Figures 5A – 5C). It would have been obvious for one of ordinary skill in the art at the time of the invention (pre-AIA ) or at the time of the effective filing date of the application (AIA ) to modify Koperwas's system to represent depth image of the virtual object in color based on Ponto’s system that displays color coded depth pixels of a scanned object after a threshold depth range and further modify the color coded depth to be represented in a logarithmic scale based on the concept of Kaser that color codes depth in an image in a logarithmic scale. One would be motivated to do so because this would help recognize irregularities in the depth data (Ponto, paragraph 87) and allow the user to perceive delineations not perceptible using a 2D model (Kaser, paragraph 63).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JEFFREY J CHOW whose telephone number is (571)272-8078. The examiner can normally be reached 11AM-7PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JEFFREY J CHOW/Primary Examiner, Art Unit 2618
Prosecution Insights