DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation 112(f)
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
As to claims 1-10, the “processing unit” is considered to read on electronic hardware (Specification as filed: page 18, lines 11-12; PGPUB: [0078]).
As to claims 1-10, the “acquisition module” is considered to read on a hardware processor for operating the acquisition process (Specification as filed: p. 18, lines 1-23; PGPUB: [0076-0078]).
As to claims 1-10, the “preprocessing module” is considered to read on a hardware processor for operating the preprocessing process (Specification as filed: p. 18, lines 1-23; PGPUB: [0076-0078]).
As to claims 1-10, the “creation module” is considered to read on a hardware processor for operating the creating process (Specification as filed: p. 18, lines 1-23; PGPUB: [0076-0078]).
As to claims 1-10, the “assignment module” is considered to read on a hardware processor for operating the assigning process (Specification as filed: p. 18, lines 1-23; PGPUB: [0076-0078]).
As to claims 1-10, the “decision-making module” is considered to read on a hardware processor for operating the decision-making process (Specification as filed: p. 18, lines 1-23; PGPUB: [0076-0078]).
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-10 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. The “visibility map” and “visibility probabilities of the grid cells” is unclear. For example, is the visibility probability when a person sees the grid cell, the person can visually recognize that the grid cell is actually occupied by some object; or the probability that an object exists between the grid cell and the sensor, and the grid cell is behind the object when viewed from the sensor; or the probability defined as something else. A potential correction could be to include that “wherein the visibility probabilities of each of the grid cells represent the visibility of each grid cell relative to the sensor”.
Claims 2 and 3 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Both claims 2 and 3 start with “The processing unit according to claim 1, wherein if the acquisition module acquires sensor data from two or more sensors…”. The claims become indefinite because if there is only one sensor then the limitations within the claim are moot. One way to correct this would be to state ““The processing unit according to claim 1, wherein the acquisition module acquires sensor data from two or more sensors…”. Appropriate correction is required.
Claims 8 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. It is unclear based on claim 8 being a system claim and depending off of independent claim 1 (which is a processing unit claim) if the processing unit is the only limitation to include in claim 8, or all the limitations within claim 1, which would then be redundant with the rest of the limitations within claim 8. Normally this would be based on a system claim depending off a method claim and including the method thereof. One way to clarify the claim language would be to make claim 8 a solo independent system claim, with the limitations from claim 1 (as system limitations).
Claims 9 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. It is unclear based on claim 9 being a method claim and depending off of independent claim 1 (which is a processing unit claim) if the processing unit is the only limitation to include in claim 9, or all the limitations within claim 1, which would then be redundant with the rest of the limitations within claim 9. Normally this would be based on a system claim depending off a method claim and including the method thereof. One way to clarify the claim language would be to make claim 9 a solo independent method claim, with the limitations from claim 1 (as method limitations).
Claim 1 (and thus dependent claims 2-10) recites the limitation "the position information". There is insufficient antecedent basis for this limitation in the claim. The first time it is used it should read as “…and
Claim 1 (and thus dependent claims 2-10) recites the limitation "the visibility probabilities". There is insufficient antecedent basis for this limitation in the claim. The first time it is used it should read as “…ambient environment and
Claim 1 (and thus dependent claims 2-10) recites the limitation "the confidences". There is insufficient antecedent basis for this limitation in the claim. The first time it is used it should read as “…configured to assign
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 10 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because the claim states a “machine readable storage medium” which can read on a signal. One correction would be to add “non-transitory” to the machine readable storage medium. Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-3 and 5-10 are rejected under 35 U.S.C. 103 as being unpatentable over Ye et al., US 2022/0148328 A1 (Ye) and further in view of Das et al., US 2021/0181758 A1 (Das).
Regarding claim 1, Ye teaches a processing unit (an apparatus including a processor) ([0072]) for processing object detection results (pedestrian detection result) ([0059]) based on sensor visibilities (based on object visibility from a camera) (Abstract and [0217]), comprising:
an acquisition module (obtaining unit 3001) (Fig. 25; [0374]) configured to acquire sensor data output when one or more sensors (wherein the image is obtained through photographing by using a camera) ([0217]) detect an ambient environment (wherein the image is taken in an outside environment) ([0215]);
a preprocessing module (processing unit 3002) (Fig. 25; [0374]) configured to calculate a set of detection results (wherein each feature map reflects a different feature; such as the pedestrian head, hand, and a feature of a background object, etc.) ([0271]) based on the acquired sensor data (generating a basic feature map based on the image taken) (Fig. 10, step 1002; [0218-0221]), wherein each detection result contains a detection object and the position information thereof (wherein each feature map reflects a different feature; such as the pedestrian head, hand, and a feature of a background object, etc.) ([0271]);
a creation module (processing unit 3002) (Fig. 25; [0374]) configured to create a visibility map based on the acquired sensor data and the calculated detection results (generating a visibility map based on the acquired image and the basic feature map of the image) (Fig. 10, steps 1004 and 1005; [0245-0246] and [0271-0275]), wherein the visibility map comprises a plurality of the ambient environment and the visibility probabilities (wherein the visibility map of the image has different response degrees to different objects; i.e. strong response to a pedestrian visible part and a weak response to the pedestrian invisible part) ([0246]);
an assignment module (processing unit 3002) (Fig. 25; [0374]) configured to assign the confidences of the detection results (confidence level of the detection results) ([0301-0302]) based on the visibility probabilities associated with the detection objects of the detection results in terms of position (when the object visibility map of the image is obtained by performing weighted summation on the plurality of first feature maps by using a weighting coefficient the same as that used by the RCNN module to perform weighted summation processing, a feature that makes a great contribution to pedestrian detection can be highlighted, and a feature that makes a small contribution can be suppressed) ([0276-0278]) (because the object visibility map of the image has a stronger response to a human body visible part, the location of the proposal can be more accurately determined by using the object visibility map) ([0298-0300]); and
a decision-making module (processing unit 3002) (Fig. 25; [0374]) configured to determine whether to remove detection results from the set of detection results based on the confidences of the detection results (non-maximum suppression (non-maximize suppression, NMS) processing may be further performed on all output proposals, to combine highly overlapped proposals and filter out a proposal with an excessively low confidence level) ([0368]).
Although Ye does not explicitly state visibility “probabilities” of the detection results, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention that the “response degrees” in Ye are obviously a type of probability; i.e. strong or weak (Ye; [0246]).
However, Ye does not explicitly teach “grid cells” or “a threshold predetermined for the set of detection results”.
Das teaches a perception system for determining an estimated object detection from one or more object detections (Fig. 3; [0058]); wherein using a plurality of grid cells (a grid having a discrete portion, such as a pixel, that indicates whether a corresponding location in the environment is occupied or unoccupied according to the perception pipeline associated with the respective sensor data type) ([0014]); and a threshold predetermined for the set of detection results (these positive object detections may be portions of the environment representation associated with a likelihood that meets or exceeds a threshold confidence) ([0096]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Ye to include using the techniques of Das since the techniques can increase the accuracy of object detection (Das; [0022]).
Regarding claim 2, Ye teaches wherein if the acquisition module (obtaining unit 3001) (Fig. 25; [0374]) acquires sensor data from a sensor (wherein the image is obtained through photographing by using a camera) ([0217]):
the preprocessing module (processing unit 3002) (Fig. 25; [0374]) is configured to calculate detection results (wherein each feature map reflects a different feature; such as the pedestrian head, hand, and a feature of a background object, etc.) ([0271]) based on sensor data (generating a basic feature map based on the image taken) (Fig. 10, step 1002; [0218-0221]) output from the sensor (the image output from the camera) ([0217]);
the creation module (processing unit 3002) (Fig. 25; [0374]) is configured to create a visibility map based on sensor data output from the sensor (generating a visibility map based on the acquired image and the basic feature map of the image) (Fig. 10, steps 1004 and 1005; [0245-0246] and [0271-0275]);
the assignment module (processing unit 3002) (Fig. 25; [0374]) is configured to assign confidences to the detection results (confidence level of the detection results) ([0301-0302]) in the corresponding based on the visibility map (when the object visibility map of the image is obtained by performing weighted summation on the plurality of first feature maps by using a weighting coefficient the same as that used by the RCNN module to perform weighted summation processing, a feature that makes a great contribution to pedestrian detection can be highlighted, and a feature that makes a small contribution can be suppressed) ([0276-0278]); and
the decision-making module (processing unit 3002) (Fig. 25; [0374]) is configured to determine to remove detection results when the confidences of detection results in the subset are lower (non-maximum suppression (non-maximize suppression, NMS) processing may be further performed on all output proposals, to combine highly overlapped proposals and filter out a proposal with an excessively low confidence level) ([0368]).
However, Ye does not explicitly teach using “two or more sensors”, to calculate a “subset” or a “threshold”.
Das teaches a perception system for determining an estimated object detection from one or more object detections (Fig. 3; [0058]); wherein the system uses various environment representations produced by the different pipelines may be aggregated into a multi-channel data structure (Fig. 3; [0083]); wherein two or more sensors are used (such as vision, radar, and LiDAR) (Fig. 3; [0082-0083]) to generate a subset that is combined to detect the occupancy/unoccupancy ([0083]); and wherein a threshold is used in the detection (positive object detections may be portions of the environment representation associated with a likelihood that meets or exceeds a threshold confidence) ([0096]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Ye to include using the techniques of Das since the techniques can increase the accuracy of object detection (Das; [0022]).
Regarding claim 3, Ye teaches the decision-making module (processing unit 3002) (Fig. 25; [0374]) is configured to determine to remove detection results when the confidences of detection results are lower (non-maximum suppression (non-maximize suppression, NMS) processing may be further performed on all output proposals, to combine highly overlapped proposals and filter out a proposal with an excessively low confidence level) ([0368]).
However, Ye does not explicitly teach using “two or more sensors”, to calculate a “fusion” or a “threshold”
Das teaches wherein if the acquisition module acquires sensor data from two or more sensors (two or more sensors such as vision, radar, and LiDAR) (Fig. 3; [0082-0083]): the preprocessing module is configured to calculate a fusion set of detection results based on sensor data output from the sensors (various environment representations produced by the different pipelines may be aggregated into a multi-channel data structure) (Fig. 3; [0083]); the creation module is configured to create a fusion visibility map based on sensor data output from the sensors and the fusion set (generating an the occupancy/unoccupancy grid based on the aggregated output) ([0083]); the assignment module is configured to assign confidences to the detection results in the fusion set based on the fusion visibility map (positive object detections may be portions of the environment representation associated with a likelihood that meets or exceeds a threshold confidence; wherein each and any of the object detection components discussed above may be associated with a regressed confidence score) ([0096]); and wherein a threshold is used in the detection (positive object detections may be portions of the environment representation associated with a likelihood that meets or exceeds a threshold confidence) ([0096]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Ye to include using the techniques of Das since the techniques can increase the accuracy of object detection (Das; [0022]).
Regarding claim 5, Das teaches wherein the preprocessing module (computing device) ([0025]) is further configured to: receive a sensor state signal indicating the state of a sensor (determining if the sensor is occluded) ([0108]); and forbid the detection results detected by one sensor to participate in the creation of the visibility map when receiving a sensor state signal indicating a degraded state of the sensor (wherein the occupancy map is not created using the sensor when it is fully occluded by retiring the detection track) ([0108]).
Regarding claim 6, Das teaches wherein the creation module (computing device) ([0025]) is configured to: provide an initial value of the visibility probability for each grid cell (a grid having a discrete portion, such as a pixel, that indicates whether a corresponding location in the environment is occupied or unoccupied according to the perception pipeline associated with the respective sensor data type) ([0014]) and regulate the initial value to a higher or lower visibility probability according to an update of sensor data and/or detection results (wherein the detection result can be updated and the visibility can be lowered (i.e. detected to be occluded and retired) based on the update) ([0108]).
Regarding claim 7, Das teaches wherein the creation module (computing device) ([0025]) is configured to: create an occupancy grid map (creating an occupancy grid) ([0014]), which contains the occupancy state of each grid cell (occupancy grid) ([0014]) and occlusion relationships (occlusion relationships) ([0035] and [0083]) between detection objects (a grid having a discrete portion, such as a pixel, that indicates whether a corresponding location in the environment is occupied or unoccupied according to the perception pipeline associated with the respective sensor data type) ([0014]), based on sensor data or a set of detection results (the system uses various environment representations produced by the different pipelines may be aggregated into a multi-channel data structure (Fig. 3; [0083]) (wherein two or more sensors are used) (such as vision, radar, and LiDAR) (Fig. 3; [0082-0083]), and determine the visibility probability of each grid cell according to the occupancy grid map (a probability that a portion of the environment is occupied) ([0014]).
Regarding claim 8, Ye teaches a processing system for processing object detection results (pedestrian detection result) ([0059]) based on sensor visibilities (based on object visibility from a camera) (Abstract and [0217]), comprising: a sensing unit comprising one or more sensors (wherein the image is obtained through photographing by using a camera) ([0217]) and configured to detect an ambient environment (wherein the image is taken in an outside environment) ([0215]) and output sensor data (output image data) ([0213-0217]); and the processing unit according to claim 1 (see the rejection made to claim 1 above) which is configured to create a visibility map containing sensor visibilities (generating a visibility map based on the acquired image and the basic feature map of the image) (Fig. 10, steps 1004 and 1005; [0245-0246] and [0271-0275]), assign a confidence to each detection result (confidence level of the detection results) ([0301-0302]) based on the visibility map (based on the object visibility map) ([0300-0302]), and determine whether to remove a detection result based on the confidence of the detection result (non-maximum suppression (non-maximize suppression, NMS) processing may be further performed on all output proposals, to combine highly overlapped proposals and filter out a proposal with an excessively low confidence level) ([0368]).
Regarding claim 9, Ye teaches a processing method for processing object detection results (pedestrian detection result) ([0059]) based on sensor visibilities (based on object visibility from a camera) (Abstract and [0217]), the processing method being executed by the processing unit (an apparatus including a processor) ([0072]) according to claim 1 (see the rejection made to claim 1 above), and
the processing method comprising:
acquiring sensor data (obtaining unit 3001) (Fig. 25; [0374]) output when one or more sensors (wherein the image is obtained through photographing by using a camera) ([0217]) detect an ambient environment (wherein the image is taken in an outside environment) ([0215]);
figuring out a set of detection results (wherein each feature map reflects a different feature; such as the pedestrian head, hand, and a feature of a background object, etc.) ([0271]) based on the acquired sensor data (generating a basic feature map based on the image taken) (Fig. 10, step 1002; [0218-0221]), wherein each detection result contains a detection object and the position information thereof (wherein each feature map reflects a different feature; such as the pedestrian head, hand, and a feature of a background object, etc.) ([0271]);
creating a visibility map based on the acquired sensor data and the calculated detection results (generating a visibility map based on the acquired image and the basic feature map of the image) (Fig. 10, steps 1004 and 1005; [0245-0246] and [0271-0275]), wherein the visibility map comprises the ambient environment and the visibility probabilities (wherein the visibility map of the image has different response degrees to different objects; i.e. strong response to a pedestrian visible part and a weak response to the pedestrian invisible part) ([0246]);
assigning the confidences of the detection results (confidence level of the detection results) ([0301-0302]) based on the visibility probabilities associated with the detection objects of the detection results in terms of position (when the object visibility map of the image is obtained by performing weighted summation on the plurality of first feature maps by using a weighting coefficient the same as that used by the RCNN module to perform weighted summation processing, a feature that makes a great contribution to pedestrian detection can be highlighted, and a feature that makes a small contribution can be suppressed) ([0276-0278]) (because the object visibility map of the image has a stronger response to a human body visible part, the location of the proposal can be more accurately determined by using the object visibility map) ([0298-0300]); and
determining whether to remove detection results from the set of detection results based on the confidences of the detection results (non-maximum suppression (non-maximize suppression, NMS) processing may be further performed on all output proposals, to combine highly overlapped proposals and filter out a proposal with an excessively low confidence level) ([0368]).
Although Ye does not explicitly state visibility “probabilities” of the detection results, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention that the “response degrees” in Ye are obviously a type of probability; i.e. strong or weak (Ye; [0246]).
However, Ye does not explicitly teach “grid cells” or “a threshold predetermined for the set of detection results”.
Das teaches a perception system for determining an estimated object detection from one or more object detections (Fig. 3; [0058]); wherein using a plurality of grid cells (a grid having a discrete portion, such as a pixel, that indicates whether a corresponding location in the environment is occupied or unoccupied according to the perception pipeline associated with the respective sensor data type) ([0014]); and a threshold predetermined for the set of detection results (these positive object detections may be portions of the environment representation associated with a likelihood that meets or exceeds a threshold confidence) ([0096]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Ye to include using the techniques of Das since the techniques can increase the accuracy of object detection (Das; [0022]).
Regarding claim 10, Ye teaches a machine readable storage medium (memory 4001) (Fig. 26; [0377]), storing executable instructions (the memory can store a program) (Fig. 26; [0377]), wherein when the instructions are executed, one or more processors (wherein the processor is configured to execute a program) (Fig. 26; [0378]) will be allowed to execute the method according to claim 9 (see the rejection made to claim 9 above).
Allowable Subject Matter
Claim 4 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims (including overcoming any 35 USC 112 issues addressed above).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. CN 112307826 A: is also by prior art Ye with an earlier publication date, however, it teaches the same concepts.
Contact
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL J VANCHY JR whose telephone number is (571)270-1193. The examiner can normally be reached Monday - Friday 9am - 5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL J VANCHY JR/Primary Examiner, Art Unit 2666 Michael.Vanchy@uspto.gov
Prosecution Insights