DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 10/31/2024 was filed in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 9 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 9 recites the limitation " the second image " There is insufficient antecedent basis for this limitation in the claim or independent claim 1.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 1-6, 8-11 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al (Wang hereinafter US 20240193846 A1) in view Lindner et al (Lindner hereinafter WO 2017087088 A1 “SYSTEMS AND METHODS FOR CORRECTING ERRONEOUS DEPTH INFORMATION”)
As per claim 1
Wang teaches An information processing device comprising one or more processors and / or circuitry configured to (Figure 12) acquiring first depth information (Figure 6) which is depth information of a first image in which a plurality of virtual objects including a translucent material are disposed, (Figure 3, Figure 4. Figure 7. Paragraph [0013] “ the copying the first rendering result onto the second color texture comprises copying color information and depth information in the first rendering result onto the second color texture” Paragraph [0042] “copy color information and depth information in the first rendering result onto the second color texture” Paragraph [0089] “the electronic device 100 further comprises an AR (Augmented Reality) device, a VR (Virtual Reality) device, a MR (Mixed Reality) device, and the like” Figure 7. ) Paragraph [0097] “ The transparent object comprises a semi-transparent object and a fully transparent object.” The “first depth information” is the depth information in the first rendering result. The “first image is the actual“ rendering result) generating the first image on a basis of the first depth information (Paragraph [0086] “ rendering the transparent object based on the second color texture copied with the first rendering result, to obtain a second rendering result;” Paragraph [0102] “ rendering the opaque object A to obtain a first rendering result, provided in some embodiments of the present disclosure, where the opaque object may be ranked according to an ascending order of coordinate values of the opaque object A in a z-axis direction in a visual coordinate system, and the opaque object is sequentially rendered according to the ranking, and a pixel point corresponding to the opaque object with a different z-coordinate value is rendered only once, wherein the coordinate value of the opaque object A in the z-axis direction in the visual coordinate system represents depth information of the opaque object A.” Paragraph [103] rendering a pixel only once can be realized through depth detection, the depth detection is used to judge the rendering of the same position pixels) execute acquisition processing of acquiring second depth information indicating a depth of the translucent material in the first image (Figure 7, paragraph [0013] “rendering the transparent object based on the second color texture copied with the first rendering result and the depth information of the transparent object.” Paragraph [0015] “determining first target depth information of the transparent object and second target depth information in the first rendering result” Paragraph [0043] render the transparent object based on the second color texture copied with the first rendering result and the depth information of the transparent object.” Paragraph [0120] “determining first target depth information of the transparent object and second target depth information in the first rendering result, which correspond to target pixel points respectively;”)
Although Wang teaches comparison of the first target depth information with the second target depth information (Paragraph [0121-0122] which shows comparison, occlusion determination and a render or not thresholding scenario) Wang does not teach execute correction processing of correcting the first depth information on a basis of the second depth information.
Lindner teaches execute correction processing of correcting the first depth information on a basis of the second depth information. (Paragraph [0060] “Correcting the erroneous depth information may generate a corrected depth map. For instance, the depth information corrector 118 may correct erroneous depth information in one depth map of a scene based on another depth map of the scene and the displacement information.” Paragraph [0088] “ the electronic device 102 may replace the erroneous depth information with one or more interpolated depths (based on the same depth map and/or one or more transformed depth maps, for example” Lindner broadly teaches the use of multiple depth datasets to identify and replace erroneous depth information and generate a corrected depth map”).
Accordingly, a person of ordinary skill in the art, at the time this invention was effectively filed would have found it obvious to modify Wang’s pipeline with Lindner’s depth map correction. Wang recognizes that transparent objects may result in erroneous rendering and states in paragraph [0123] that “ a situation that the transparent object B blocked by the opaque object A is erroneously rendered can be avoided, and the rendering reliability is improved. “ This expressly identifies the problem of erroneous depth representation and rendering reliability in scenes containing transparent objects. Lindner teaches the general principle that when multiple sources of depth information are available, erroneous depth information can be corrected using another available depth dataset to generate a corrected depth map. A person of ordinary skill in the art would recognize that the identity of the depth map being corrected depends on which of the available depth datasets is determined to contain this erroneous information. Wang already obtains depth information in the first rendering result and depth information of the transparent object. Wang also compares those depth values to determine the proper render. A person of ordinary skill in the art would have found it obvious to use the transparent object depth information to correct erroneous portions of the depth information in the first rendering result as taught win Lindner’s depth correction techniques in order to improve rendering fidelity, reduce erroneous rendering of transparent/translucent objects, improve depth accuracy of the rendered scene and generate a more accurate depth representation of transparent or translucent objects.
As per claim 2
Wang and Lindner teach all claim limitations previously rejected in claim 1’s 103 rejection. See claim 1’s 103 rejection.
Wang teaches wherein in the rendering processing; the first depth information is acquired by ignoring presence of the translucent material in the plurality of virtual objects. (Figure 4. Paragraph [0102] “ refer to FIG. 4, which is a schematic diagram of rendering the opaque object A to obtain a first rendering result, provided in some embodiments of the present disclosure, where the opaque object may be ranked according to an ascending order of coordinate values of the opaque object A in a z-axis direction in a visual coordinate system, and the opaque object is sequentially rendered according to the ranking, and a pixel point corresponding to the opaque object with a different z-coordinate value is rendered only once, wherein the coordinate value of the opaque object A in the z-axis direction in the visual coordinate system represents depth information of the opaque object A.” Figure 4 shows that the transparent object is ignored as opposed to figure 7 where both objects are addressed.)
As per claim 3
Wang and Lindner teach all claim limitations previously rejected in claim 1’s 103 rejection. See claim 1’s 103 rejection.
Wang teaches each of the first depth information and the second depth information is information indicating depths of a plurality of positions of the first image (Figure 7, paragraph [0013] “ color information and depth information in the first rendering result”. Paragraph (0013) “Depth information of the transparent object”. Pixels are represented by coordinates. Therefore, each pixel is a position within the first image. The entire rendering of the objects in the image are a plurality of pixel/positions within the first image)
In regards to “ the correction processing, for each of the plurality of positions of the first image, smaller one of a depth indicated by the first depth information and a depth indicated by the second depth information is adopted as a depth indicated by the corrected first depth information.”
Within the Wang/Lindner modified workflow, Wang shows in paragraphs [0120]- [0122] determining first target depth information of the transparent object, determination of second target depth information in the first rendering result and comparison of the two depths. Then when one depth relationship exists to not render the target pixel point. This is a per-pixel depth comparison operation.
Lindner then teaches generating a corrected depth map by correcting erroneous depth information based on first depth information and second depth information as well as replacing erroneous depth information with corresponding depth information. This correction occurs pixel by pixel when replacing erroneous depth values. When it comes to “smaller one of a depth” Wang already teaches comparing the two depth values at corresponding pixels. A person of ordinary skill in the art would have recognized that when the need is determining visibility and avoiding erroneous rendering, the nearer depth value (smaller depth) corresponds to the visible surface and should be adopted for the corrected map. The smaller depth is the closest surface which in turn is the visible surface. Wang expressly is concerned with avoiding the incorrect rendering caused by translucent objects. Adopting the smaller depth value at each position would have been obvious implementation of the depth correction that Lindner supports and a person of ordinary skill in the art recognizes this.
As per claim 4
Wang and Lindner teach all claim limitations previously rejected in claim 1’s 103 rejection. See claim 1’s 103 rejection.
Wang teaches , wherein each of the first depth information and the second depth information is information indicating depths of the plurality of virtual objects in the first image (Figure 7) in the correction processing, for each of the plurality of virtual objects, in a case where a difference between a depth indicated by the first depth information and a depth indicated by the second depth information is larger than a threshold value (Paragraph [0047] “in a case where the first target depth information is greater than the second target depth information, not render the target pixel points.” The threshold here is the second target depth information itself.) the depth indicated by the second depth information is adopted as a depth indicated by the corrected first depth information. (Paragraph [0047] “in a case where the first target depth information is greater than the second target depth information, not render the target pixel points.” By not rendering the first target depth information’s target pixel points, the second depth information’s pixel points are instead the ones rendered. The first was “greater” meaning it was at a further distance from the observer and should not be rendered. The second depth information is adopted as the corrected depth by not rendering the first.)
As per claim 5
Wang and Lindner teach all claim limitations previously rejected in claim 1’s 103 rejection. See claim 1’s 103 rejection.
Wang teaches wherein in the acquisition processing; the second depth information is acquired on a basis of the first image generated by the rendering processing. ( Paragraph [0086] “ rendering the transparent object based on the second color texture copied with the first rendering result, “)
As per claim 6
Wang and Lindner teach all claim limitations previously rejected in claim 1’s 103 rejection. See claim 1’s 103 rejection.
Lindner teaches wherein in the acquisition processing; the second depth information is acquired on a basis of a plurality of images viewing the plurality of virtual objects from a plurality of different viewpoints. (Paragraph [0030] “Examples of the electronic device 102 include… virtual reality devices (e.g., headsets), augmented reality devices (e.g., headsets), mixed reality devices (e.g., headsets),” Paragraph [0052] “ In some configurations, the depth information obtained 116 may determine the depth information based on moving cameras…depth may be estimated based on two or more image frames due to camera motion…the depth information obtainer 116 may determine a distance between the image sensor (e.g., image sensor(s) 104 and/or remote image sensor(s)) and the object. The object points from two views may be matched and the relative camera motion may be estimated. Then, the depth information (e.g., distances) of the object may be estimated (e.g., generated) by triangulation. A moving video is a plurality of images that captures information from a plurality of different viewpoints.
As far as “viewing the plurality of virtual objects” Lindner should not be seen in a vacuum. The Wang/Lindner modified methodology supplies the plurality of virtual objects through Wang’s virtual device 100 which “comprises an AR (Augmented Reality) device, a VR (Virtual Reality) device, a MR (Mixed Reality) device, and the like.” (Wang, Paragraph [0089])
As per claim 8
Wang and Lindner teach all claim limitations previously rejected in claim 1’s 103 rejection. See claim 1’s 103 rejection.
Lindner also teaches wherein on a basis of the first depth information corrected in the correction processing, related processing, which is processing related to the first image, is executed ([Paragraph [0105] ‘the visualization element may generate a 3D model based on the corrected depth map 660. Image data (e.g., from the images 648) may be rendered on the corrected depth map in some configurations.” After the correction is made, a rendering process occurs. A rendering process of the image is a process related to the first image as it is the same scene)
As per claim 9
Wang and Lindner teach all claim limitations previously rejected in claim 8’s 103 rejection. See claim 8’s 103 rejection.
Lindner wherein in the related processing, the first image and the second image are synthesized on a basis of the first depth information. ( Paragraph [0058] “a depth sensor may sample a first portion of a scene at a first sampling and may sample a second portion of the scene at a second sampling “ Paragraph [0105] “Additionally or alternatively, the visualization element may generate a 3D model based on the corrected depth map 660. Image data (e.g., from the images 648) may be rendered on the corrected depth map in some configurations.” Paragraph [0060] “Correcting the erroneous depth information may generate a corrected depth map. For instance, the depth information corrector 118 may correct erroneous depth information in one depth map of a scene based on another depth map of the scene and the displacement information.” Paragraph [0088] “ the electronic device 102 may replace the erroneous depth information with one or more interpolated depths (based on the same depth map and/or one or more transformed depth maps, for example” Lindner broadly teaches the use of multiple depth datasets to identify and replace erroneous depth information and generate a corrected depth map” this shows either a first image and or a second image can be synthesized based on the correction being done. This correction has a basis in the first depth information as this depth information is needed to evaluate the correction.)
As per claim 10
Claim 10 is the is the parallel method claim of claim 1’s system claim and will be rejected under the same premise.
As per claim 11
Claim 11 is the parallel non-transitory computer readable medium claim to claim 1’s system claim and will be rejected under the same premise.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Wang et al (Wang hereinafter US 20240193846 A1) in view Lindner et al (Lindner hereinafter WO 2017087088 A1 “SYSTEMS AND METHODS FOR CORRECTING ERRONEOUS DEPTH INFORMATION”) in further view of Smolyanskiy et al (Smolyanskiy hereinafter WO 2019182974 A2 “STEREO DEPTH ESTIMATION USING DEEP NEURAL NETWORKS”)
As per claim 7
Wang and Lindner teach all claim limitations previously rejected in claim 1’s 103 rejection. See claim 1’s 103 rejection.
Wang nor Lindner teaches wherein in the acquisition processing; the second depth information is acquired by using a deep learning model learned in advance
Smolyanskiy teaches wherein in the acquisition processing; the second depth information is acquired by using a deep learning model learned in advance (Figure 2, Description: “The outputs of a DNN in accordance with the present disclosure may include disparity maps corresponding to each of the input images, and the disparity maps may be used to calculate depth in the field of view of the sensors (e.g., the stereo cameras). The depth information may be useful for a robot, an autonomous vehicle, a drone, a virtual reality system, an augmented reality system…The input data may include data representative of the images 102 generated from a stereoscopic camera (e.g., one or more stereo cameras 568) of the vehicle 500, or another object (e.g., a robot, a drone, a VR system, an AR system, etc.)…The DNN 100 may include a first stream or tower 118A (e.g., a left image stream) corresponding to the first image 102A and a second stream or tower 118B (e.g., a right image stream) corresponding to the second image 102B. For example, the first stream 118A and the second stream 118B may be executed in parallel to generate or compute the disparity maps 116 from the images 102” Smolyanskiy exhibits a stereoscopic deep neural network that receives images, generates disparity maps from the images and converts disparity maps into depth values. Therefore, Smolyanskiy teaches acquiring depth information using a deep learning model that has been trained.)
Accordingly, a person of ordinary skill in the art, at the time this invention was effectively filed would have found it obvious to further modify the Wang/Lindner system with Smolyanskiy’s concept of using a stereoscopic deep learning model to acquire the second depth information. A person of ordinary skill in the art would have been motivated to do so because Smolyanskiy teaches generating depth information from image data using a DNN and further teaches that the depth information is useful for virtual reality systems and augmented reality systems. A person of ordinary skill in the art would have recognized that substituting or supplementing the conventional depth acquisition techniques taught by the Wang/Lindner system with the DNN based depth estimation of Smolyanskiy would have predictable improvements in the strength and accuracy of obtaining depth information for the translucent object correction while remaining within the same realm of endeavor which is namely image based depth acquisition for AR/VR rendering.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHANE WRENSFORD CODRINGTON whose telephone number is (571)272-8130. The examiner can normally be reached 8:00am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571) 272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHANE WRENSFORD CODRINGTON/Examiner, Art Unit 2667 /MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights