DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they do not include the following reference sign(s) mentioned in the description: “012” from paragraph 132. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The disclosure is objected to because of the following informalities:
Paragraph 82, “and if so” should be “and if so,”
Paragraph 115, the character “96” is used to designate the read only memory device. The character from Figure 7 for a read only memory device is “98”.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The term “quasi-simultaneously” in claims 1, 12, and 17 is a relative term which renders the claim indefinite. The term “quasi-simultaneously” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. The term, “quasi-simultaneously” is defined in the specification in paragraphs 34-35. The definitions for simultaneously or taken at times t1 and t2 such that the scene is unchanged are definite. However, the definition further includes “essentially unchanged”. “Essentially unchanged” is defined in paragraph 35 as being the when the spatial resolution of the digital images and depth sensor is insensitive to any change in the scene. As spatial resolution of an image represents the change in individual pixels, any change would cause the spatial resolution to be changed and to be sensitive to such change. As such, what does “essentially unchanged” actually entail? It is not clear what the term fully includes, and as such it is indefinite. Claims 2-11, 13-16, and 18-20 inherit this issue, and they are similarly rejected.
The term “sufficiently spatially refined” in claim 9 is a relative term which renders the claim indefinite. The term “sufficiently spatially refined” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. What does “sufficiently spatially refined” mean? How does one judge the sufficiency of the refinement? The specification, in paragraph 82, does not give a clear definition of this or what it entails. As people can have differing opinions on what that really entails. Whether one must mask and segment every object in an image, simply cover most of the image, or some other metric to have sufficiently refined the space. As such, the claim is indefinite.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-8 and 10-20 as best understood are rejected under 35 U.S.C. 102(a)(1) and (a)(2) as being anticipated by Sheu et al. (US 20220027675 A1), hereinafter referred to as Sheu.
In regards to claim 1, Sheu discloses a method for segmenting digital images obtained by a digital camera and analyzing the segments by a machine learning model (MLM), said method comprising: receiving, by one or more processors of a computer system, a first digital image of a scene obtained by a digital camera, said first digital image characterized by a first pixel configuration (Paragraph 8, abstract, and Figure 1, All of these disclose the capturing of an image via a camera with figure 1 showing the camera system used to capture images); receiving, by the one or more processors, a second digital image of the scene obtained by a depth sensor, said second digital image characterized by a second pixel configuration (Paragraph 8, abstract, and Figure 1, These paragraphs disclose the use of a LiDAR system which would correspond to the depth sensor); said first and second digital images being obtained quasi-simultaneously (Paragraph 8, Discloses that both images are taken on and correspond to the same frame, so they must be simultaneous which would read upon the term “quasi-simultaneous”); generating, by the one or more processors using the second digital image, a binary mask characterized by the second pixel configuration, wherein said generating the binary mask comprises selectively digitizing each pixel of the binary mask to 1 or 0 to identify one or more regions of the scene to be subsequently segmented from the first digital image (Paragraph 42, Discloses that the second image or the depth map is given a corresponding binary mask with binary on and binary off which corresponds to 1 and 0); and generating, by the one or more processors by applying the binary mask to the first digital image, one or more segments of the first digital image, each generated segment corresponding to a subset of the pixels of the binary mask that are digitized to 1 (Paragraph 16, Discloses the use of a segmentation model which applies the masks to the 2D image).
In regards to claim 2, Sheu discloses wherein said generating the binary mask comprises for each pixel of the second pixel configuration: specifying one or more designated regions in the scene (Paragraph 42, Discloses that the second image or the depth map is given a corresponding binary mask with binary on and binary off which corresponds to 1 and 0. A binary mask would separate each pixel into either the on or off region); ascertaining that a spatial position in the scene corresponding to the pixel is within, or is not within, the one or more designated regions (Paragraph 41, Discloses that depth values, intensity values, and height values can be used to segment the image); and digitizing the pixel to 1 or 0 in response to having ascertained that the spatial position in the scene corresponding to the pixel is within, or is not within, respectively, the one or more designated regions (Paragraph 42, Discloses that the second image or the depth map is given a corresponding binary mask with binary on and binary off which corresponds to 1 and 0. A binary mask would separate each pixel into either the on or off region).
In regards to claim 3, Sheu discloses wherein said ascertaining comprises ascertaining that the spatial position in the scene corresponding to the pixel is within or not within a location constraint defining each designated region of the one or more designated regions in the second digital image (Paragraph 41, Discloses that depth values, intensity values, and height values can be used to segment the image which would cover options 1 and 3 which merely require the depth constraint and the elevation constraint), and wherein the location constraint is selected from the group consisting of: (i) a depth constraint on the spatial position, (ii) the depth constraint and an azimuthal angle constraint on the spatial position, (iii) the depth constraint and an elevation constraint on the spatial position, and (iv) the depth constraint, the azimuthal angle constraint, and the elevation constraint on the spatial position (Paragraph 41, Discloses that depth values, intensity values, and height values can be used to segment the image which would cover options 1 and 3 which merely require the depth constraint and the elevation constraint).
In regards to claim 4, Sheu discloses wherein the second pixel configuration differs from the first pixel configuration, and wherein said generating the one or more segments of the first digital image comprises: transforming the second pixel configuration of the binary mask to the first pixel configuration, resulting in the binary mask being characterized by the first pixel configuration (Paragraphs 40-43, These paragraphs disclose that the 2D sparse segmentation masks at first correspond to the 3D data points from the LiDAR system which are then adapted to correspond with the 2D images made from the cameras).
In regards to claim 5, Sheu discloses wherein said generating the one or more segments of the first digital image comprises: defining each segment as corresponding to at least one designated region selected from the one or more designated regions, wherein the subset of the pixels of the binary mask digitized to 1 are the pixels digitized to 1 that correspond to the at least one designated region (Paragraph 42, Discloses that the second image or the depth map is given a corresponding binary mask with binary on and binary off which corresponds to 1 and 0 with the pixels marked as 1 or on are in a region of the image or make a region of the image, so this is covered).
In regards to claim 6, Sheu discloses wherein the method further comprises: providing, by the one or more processors, at least one segment of the one or more segments as input to the machine learning model (MLM) (Paragraphs 14 and 17, Discloses inputting the data into the segmentation model which would include segmented masks); and executing, by the one or more processors, the MLM to determine scene information pertaining to the scene (Paragraph 14, Identifying objects in the image even as object 1 would constitute identifying scene information).
In regards to claim 7, Sheu discloses wherein the scene information determined from said executing the MLM is a classification of an object in each segment of the at least one segment (Paragraph 42, Discloses that the second image or the depth map is given a corresponding binary mask with binary on and binary off which corresponds to 1 and 0. The binary on or off as disclosed functionally acts as a classification as it is classified into two binary categories).
In regards to claim 8, Sheu discloses wherein the at least one segment is a plurality of segments, and wherein the scene information determined from said executing the MLM is a geometric relationship between a first segment and a second, different segment of the plurality of segments, wherein the geometric relationship is between respective entities in the first and second segments, and wherein each entity of the respective entities is independently selected from the group consisting of a spatial location, a line, an area, and a boundary of an area, a volume, or an object (Paragraph 45, The LiDAR system can determine distances between objects in the sensed environment which would represent a geometric relationship as it relates to distance).
In regards to claim 10, Sheu discloses wherein the depth sensor is Light Detection and Ranging (LiDAR) (Paragraph 35, Discloses the usage of LiDAR systems).
In regards to claim 11, Sheu discloses wherein the depth sensor is Radio Detection and Ranging (Radar) (Paragraph 35, Discloses the usage of Radar systems).
In regards to claim 12, it is similar to claim 1, and it is similarly rejected.
In regards to claim 13, it is similar to claim 2, and it is similarly rejected.
In regards to claim 14, it is similar to claim 3, and it is similarly rejected.
In regards to claim 15, it is similar to claim 5, and it is similarly rejected.
In regards to claim 16, it is similar to claim 6, and it is similarly rejected.
In regards to claim 17, it is similar to claims 1 and 12, and it is similarly rejected.
In regards to claim 18, it is similar to claims 2 and 13, and it is similarly rejected.
In regards to claim 19, it is similar to claims 5 and 15, and it is similarly rejected.
In regards to claim 20, it is similar to claims 6 and 16, and it is similarly rejected.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 9 is rejected under 35 U.S.C. 103 as being unpatentable over Sheu et al. (US 20220027675 A1), hereinafter referred to as Sheu, in view of Ponjou Tasse et al. (US 20200372709 A1), hereinafter referred to as Ponjou Tasse.
In regards to claim 9, Sheu discloses and repeating said receiving the second digital image of the scene, said generating the binary mask, said generating the one or more segments of the first digital image, said providing the at least one segment as input to the MLM, and said executing the MLM (Paragraph 64, discloses that the operation including all of the steps can be repeated multiple times).
Sheu does not explicitly disclose wherein the method further comprises: determining, by the one or more processors, that the scene information that was determined from said executing the MLM is not sufficiently spatially refined and in response, increasing, by the one or more processors, a scanning period of the depth sensor.
However, Ponjou Tasse does disclose wherein the method further comprises: determining, by the one or more processors, that the scene information that was determined from said executing the MLM is not sufficiently spatially refined and in response, increasing, by the one or more processors, a scanning period of the depth sensor (Paragraph 39, Discloses that the process can be repeated as many time as necessary for the depth sensor to full map the total space from all of the images which acts as a way of increasing the scanning time to fully spatially refine the results.)
It would be prima facie obvious to combine the teachings of Sheu and Ponjou Tasse. Sheu discloses in paragraph 64 that the process can be repeated multiple times to generate a mask for every object instance in an image. Ponjou Tasse discloses repeating the process until the entire space is fully captured. Incorporating the teaching of Ponjou Tasse would allow for a predictable increase in accuracy as the full space will be fully mapped out. This would allow for more accurate binary masks to be generated by Sheu. As such, it would be prima facie obvious to combine.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Kadambi et al. (US 20160261844 A1) covers similar concepts as the cited arts and involves the improvement of depth maps via polarization cues. It also uses azimuth angles in a similar manner as the current application.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CONOR AIDAN O'MALLEY whose telephone number is (571)272-0226. The examiner can normally be reached Monday - Friday 9:00 am. - 5:00 pm. EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at 5712723838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CONOR A O'MALLEY/ Examiner, Art Unit 2675
/GREGORY A MORSE/ Supervisory Patent Examiner, Art Unit 2698
Prosecution Insights