DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This is a Final Office Action on the merits. Claims 1-3, 5-7, and 9-12 are currently pending and are addressed below.
Response to Amendment
The specification was objected to due to minor informalities. Applicant amended the specification accordingly; therefore, the specification objection is withdrawn.
Claims 4-5, 7, and 10-11 were objected to due to minor informalities. Applicant amended the claims accordingly and canceled claim 4; therefore, the claims objection is withdrawn.
Claims 2-3 and 5-6 were rejected under 35 U.S.C. 112 as being indefinite. Applicant amended the claims accordingly; therefore, the rejection is withdrawn.
Response to Arguments
Applicant’s arguments on pages 11-13 of the response, with respect to the rejection(s) of claim(s) 1-11 under 35 U.S.C. 101 have been fully considered but they are not persuasive. Applicant argues the claims recite specific technical steps that cannot be practically performed in the human mind (e.g., applying a machine learning model to evaluate image data and determine occlusion labels). Examiner respectfully disagrees. The claims recite the abstract ideas of performing an evaluation of image data to determine an occlusion label and performing the detection of an obstacle based on the occlusion label are abstract ideas, which encompass mentally determining an occlusion label and mentally detecting (i.e., recognizing) an obstacle, respectively, as well as the additional element of “providing image data specific to a recording of an environment of the driving system”, which is a form of insignificant extra-solution activity (i.e., data gathering). The additional elements “an automated and/or at least semi-autonomous driving system” and “wherein the evaluation takes placed based on an application of a machine learning model” do not amount to significantly more than the judicial exception since they are recited at a high level of generality such that they merely integrate the judicial exception into a technological environment. The amended limitations “wherein the occlusion label is specific to the at least one occlusion and is designed as an occlusion map which identifies at least one or multiple areas in the image data that are occluded by at least one object in the environment” further describes the occlusion label and does not amount to significantly more than the judicial exception. Lastly, the functions of performing an evaluation of an image to determine an occlusion label and performing the detection of an obstacle based on the occlusion label are recited at a high level of generality such that they can be performed by any generic computer.
Applicant further argues that “even if the claims were found to be directed to an abstract idea, Applicant submits that they recite an inventive concept”. Examiner respectfully disagrees. There is no inventive concept because the additional elements discussed above do not amount to significantly more than the judicial exception.
Applicant’s arguments on pages 13-14 of the response, with respect to the rejection(s) of claim(s) 1 under 35 U.S.C. 103 have been fully considered but they are not persuasive. Applicant argues “there is a fundamental technical distinction between Yu’s “occlusion status” and the claimed “occlusion label””, where “this label serves as the technical basis for the detection of the obstacle, rather than being a status update for an obstacle already detected”. Examiner respectfully disagrees. The arguments are not directed to the claims as written since the claims do not exclude a “status update” as an occlusion label nor do they require that the occlusion label be determined “as an intermediate step”.
Applicant’s arguments on pages 15-16 of the response, with respect to the rejection(s) of claim(s) 2 under 35 U.S.C. 103 have been fully considered but they are not persuasive. Applicant argues that neither Tsokgas nor Lin suggest training a machine learning model based on movement in a camera recording to determine an occlusion label. Examiner respectfully disagrees. As discussed in the rejection, [0103] & [0112] of Tsokgas recite creating a dataset of image bursts or controlled image sequences, which are generated from video, to train an occlusion-aware optical flow model.
Applicant’s arguments on page 16 of the response, with respect to the rejection(s) of claim(s) 5 under 35 U.S.C. 103 have been fully considered but they are not persuasive. Applicant argues that Wu’s feature vector similarity comparison is fundamentally different from evaluating the occlusion label by performing classification of a hazardous object. However, the arguments are not directed to the rejection as written since Wu’s method of “comparing feature vector similarities in stereo pairs”, which is the subject of Applicant’s argument, was not cited in the rejection. Furthermore, Wu was used to teach the hazardous object being cargo that has fallen from a truck, and was not relied upon to teach evaluating the occlusion label.
Applicant’s arguments on page 17 of the response, with respect to the rejection(s) of claim(s) 7 and 11 under 35 U.S.C. 103 have been fully considered but they are not persuasive. Applicant argues that “Yu does not teach training a model to predict an occlusion label by taking optical flow into account”. However, the arguments are not directed to the rejection as written since Yu was not relied upon to teach training an optical flow.
Furthermore, Applicant argues that “while Tsokgas mentions occlusion-aware optical flow for image reconstruction (de- fencing), it does not suggest this for the purpose of training a model to detect obstacles in a driving system”. Examiner respectfully disagrees. As noted in the rejection, the limitation “in order to predict the occlusion label” is considered intended use since it merely recites a desired result and the “predicting” is not positively recited. Additionally, as discussed in the rejection, [0103] & [0112] of Tsokgas was used to teach training a machine learning model based on an optical flow of image bursts, and was not relied upon to teach predicting occlusion labels. Furthermore, the argument is not directed to the claims as written since claims 7 and 11 do not recite “training a model to detect obstacles”.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 12 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 12 recites “…the estimated optical flow…” and “…the optical flow”. It is unclear whether these limitations are referring to the same optical flow.
Claim 12 recites the limitation "the relative transformations between two images of the image sequence" in line 6. There is insufficient antecedent basis for this limitation in the claim.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-3, 7, and 9-12 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Independent Claim 1:
Step 1: Claim 1 is directed to a method (i.e., a process). Therefore, claim 1 is within at least one of the four statutory categories.
Step 2A Prong 1: Regarding Prong 1 of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether they recite subject matter that falls within one of the following groups of abstract ideas: a) mathematical concepts, b) certain methods of organizing human activity and/or c) mental processes.
Independent claim 1 includes limitations that recite an abstract idea (emphasized below) and will be used as a representative claim for the remainder of the 101 rejection. Claim 1 recites:
A method for detecting at least one obstacle in an automated and/or at least semi-autonomous driving system, said method comprising the following steps:
providing image data, wherein the image data are specific to a recording of an environment of the driving system,
performing an evaluation of the image data provided, wherein the evaluation takes place based on an application of a machine learning model, by means of which an occlusion label is determined for at least one occlusion of the environment, wherein the occlusion label is specific to the at least one occlusion and is designed as an occlusion map which identifies at least one or multiple areas in the image data that are occluded by at least one object in the environment,
performing the detection of the at least one obstacle on the basis of the occlusion label determined.
The examiner submits that the foregoing bolded limitations constitute a “mental process” because under its broadest reasonable interpretation, the claim covers performance of the limitations in the human mind. For example, “performing an evaluation of the image data provided…by means of which an occlusion label is determined for at least one occlusion of the environment, wherein the occlusion label is specific to the at least one occlusion and is designed as an occlusion map which identifies at least one or multiple areas in the image data that are occluded by at least one object in the environment” in the context of this claim encompasses mentally evaluating an image and mentally determining an occlusion label. The limitation “performing the detection of the at least one obstacle on the basis of the occlusion label determined” in the context of this claim encompasses mentally recognizing an obstacle based on an occlusion label.
Step 2A Prong 2: Regarding Prong II of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether the claim, as a whole, integrates the abstract into a practical application. As noted in the 2019 PEG, it must be determined whether any additional elements in the claim beyond the abstract idea integrate the exception into a practical application in a manner that imposes a meaningful limit on the judicial exception. The courts have indicated that additional elements merely using a computer to implement an abstract idea, adding insignificant extra solution activity, or generally linking use of a judicial exception to a particular technological environment or field of use do not integrate a judicial exception into a “practical application.”
In the present case, the additional limitations beyond the above-noted abstract idea are as follows (where the underlined portions are the “additional limitations” while the bolded portions continue to represent the “abstract idea”):
A method for detecting at least one obstacle in an automated and/or at least semi-autonomous driving system, said method comprising the following steps:
providing image data, wherein the image data are specific to a recording of an environment of the driving system,
performing an evaluation of the image data provided, wherein the evaluation takes place based on an application of a machine learning model, by means of which an occlusion label is determined for at least one occlusion of the environment, wherein the occlusion label is specific to the at least one occlusion and is designed as an occlusion map which identifies at least one or multiple areas in the image data that are occluded by at least one object in the environment,
performing the detection of the at least one obstacle on the basis of the occlusion label determined.
For the following reason(s), the examiner submits that the above identified additional limitations do not integrate the above-noted abstract idea into a practical application.
The additional limitation “providing image data” is recited at a high level of generality and amounts to no more than extra-solution activity (i.e., data gathering). The additional limitations “an automated and/or at least semi-autonomous driving system”, “wherein the image data are specific to a recording of an environment of the driving system”, and “wherein the evaluation takes place based on an application of a machine learning model” are also recited at a high level of generality such that they merely integrate the judicial exception into a technological environment.
Thus, taken alone, the additional elements do not integrate the abstract idea into a practical application. Further, looking at the additional limitation(s) as an ordered combination or as a whole, the limitation(s) add nothing that is not already present when looking at the elements taken individually. For instance, there is no indication that the additional elements, when considered as a whole, reflect an improvement in the functioning of a computer or an improvement to another technology or technical field, apply or use the above-noted judicial exception to effect a particular treatment or prophylaxis for a disease or medical condition, implement/use the above-noted judicial exception with a particular machine or manufacture that is integral to the claim, effect a transformation or reduction of a particular article to a different state or thing, or apply or use the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is not more than a drafting effort designed to monopolize the exception (MPEP § 2106.05). Accordingly, the additional limitation(s) do/does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
Step 2B: Regarding Step 2B of the 2019 PEG, representative independent claim 1 does not include additional elements (considered both individually and as an ordered combination) that are sufficient to amount to significantly more than the judicial exception for the same reasons to those discussed above with respect to determining that the claim does not integrate the abstract idea into a practical application. As discussed above with respect to the integration of the abstract idea into a practical application, the examiner submits that the additional limitation “providing image data” is recited at a high-level of generality (i.e. as a generic means for data gathering) and does not amount to significantly more than the judicial exception. The additional limitations “an automated and/or at least semi-autonomous driving system”, “wherein the image data are specific to a recording of an environment of the driving system”, and “wherein the evaluation takes place based on an application of a machine learning model” amount to nothing more than applying the exception using generic computer components.
Therefore, claim 1 is ineligible under 35 U.S.C §101.
Regarding Independent Claim 7:
Step 1: Claim 7 is directed to a method (i.e., a process). Therefore, claim 7 is within at least one of the four statutory categories.
Step 2A Prong 1: Regarding Prong 1 of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether they recite subject matter that falls within one of the following groups of abstract ideas: a) mathematical concepts, b) certain methods of organizing human activity and/or c) mental processes.
Independent claim 7 includes limitations that recite an abstract idea (emphasized below) and will be used as a representative claim for the remainder of the 101 rejection. Claim 7 recites:
A training method for training a machine learning model, said method comprising:
providing training data, wherein the training data comprise at least one sequence of images representing an environment of a driving system during a trip, wherein the training data further comprise annotation data which indicate an occlusion label representing at least one occlusion of the environment during the trip,
performing training of the machine learning model on the basis of the training data, during which training an optical flow in the sequence of images is taken into account in order to predict the occlusion label.
The examiner submits that the foregoing bolded limitations constitute a “mental process” because under its broadest reasonable interpretation, the claim covers performance of the limitations in the human mind. For example, “performing training of the machine learning model on the basis of the training data, during which training an optical flow in the sequence of images is taken into account in order to predict the occlusion label” in the context of this claim encompasses mentally predicting an occlusion label using image data.
Step 2A Prong 2: Regarding Prong II of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether the claim, as a whole, integrates the abstract into a practical application. As noted in the 2019 PEG, it must be determined whether any additional elements in the claim beyond the abstract idea integrate the exception into a practical application in a manner that imposes a meaningful limit on the judicial exception. The courts have indicated that additional elements merely using a computer to implement an abstract idea, adding insignificant extra solution activity, or generally linking use of a judicial exception to a particular technological environment or field of use do not integrate a judicial exception into a “practical application.”
In the present case, the additional limitations beyond the above-noted abstract idea are as follows (where the underlined portions are the “additional limitations” while the bolded portions continue to represent the “abstract idea”):
A training method for training a machine learning model, said method comprising:
providing training data, wherein the training data comprise at least one sequence of images representing an environment of a driving system during a trip, wherein the training data further comprise annotation data which indicate an occlusion label representing at least one occlusion of the environment during the trip,
performing training of the machine learning model on the basis of the training data, during which training an optical flow in the sequence of images is taken into account in order to predict the occlusion label.
For the following reason(s), the examiner submits that the above identified additional limitations do not integrate the above-noted abstract idea into a practical application.
The additional limitation “providing training data, wherein the training data comprise at least one sequence of images representing an environment of a driving system during a trip, wherein the training data further comprise annotation data which indicate an occlusion label representing at least one occlusion of the environment during the trip” is recited at a high level of generality and amounts to no more than extra-solution activity (i.e., data gathering). The additional limitation “a machine learning model” is also recited at a high level of generality such that it merely integrates the judicial exception into a technological environment.
Thus, taken alone, the additional elements do not integrate the abstract idea into a practical application. Further, looking at the additional limitation(s) as an ordered combination or as a whole, the limitation(s) add nothing that is not already present when looking at the elements taken individually. For instance, there is no indication that the additional elements, when considered as a whole, reflect an improvement in the functioning of a computer or an improvement to another technology or technical field, apply or use the above-noted judicial exception to effect a particular treatment or prophylaxis for a disease or medical condition, implement/use the above-noted judicial exception with a particular machine or manufacture that is integral to the claim, effect a transformation or reduction of a particular article to a different state or thing, or apply or use the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is not more than a drafting effort designed to monopolize the exception (MPEP § 2106.05). Accordingly, the additional limitation(s) do/does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
Step 2B: Regarding Step 2B of the 2019 PEG, representative independent claim 7 does not include additional elements (considered both individually and as an ordered combination) that are sufficient to amount to significantly more than the judicial exception for the same reasons to those discussed above with respect to determining that the claim does not integrate the abstract idea into a practical application. As discussed above with respect to the integration of the abstract idea into a practical application, the examiner submits that the additional limitation “providing training data…” is recited at a high-level of generality (i.e. as a generic means for data gathering) and does not amount to significantly more than the judicial exception. The additional limitation “a machine learning model” amounts to nothing more than applying the exception in a technological environment.
Therefore, claim 7 is ineligible under 35 U.S.C §101.
Regarding Independent Claim 9:
Step 1: Claim 9 is directed to a device (i.e., a machine). Therefore, claim 9 is within at least one of the four statutory categories.
Step 2A Prong 1: Regarding Prong 1 of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether they recite subject matter that falls within one of the following groups of abstract ideas: a) mathematical concepts, b) certain methods of organizing human activity and/or c) mental processes.
Independent claim 9 includes limitations that recite an abstract idea (emphasized below) and will be used as a representative claim for the remainder of the 101 rejection. Claim 9 recites:
A device for data processing, which is configured to: provide image data, wherein the image data are specific to a recording of an environment of the driving system,
perform an evaluation of the image data provided, wherein the evaluation takes place based on an application of a machine learning model, by means of which an occlusion label is determined for at least one occlusion of the environment, and
perform the detection of the at least one obstacle on the basis of the occlusion label determined.
The examiner submits that the foregoing bolded limitations constitute a “mental process” because under its broadest reasonable interpretation, the claim covers performance of the limitations in the human mind. For example, “performing an evaluation of the image data provided…by means of which an occlusion label is determined for at least one occlusion of the environment” in the context of this claim encompasses mentally evaluating an image and mentally determining an occlusion label. The limitation “performing the detection of the at least one obstacle on the basis of the occlusion label determined” in the context of this claim encompasses mentally recognizing an obstacle based on an occlusion label.
Step 2A Prong 2: Regarding Prong II of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether the claim, as a whole, integrates the abstract into a practical application. As noted in the 2019 PEG, it must be determined whether any additional elements in the claim beyond the abstract idea integrate the exception into a practical application in a manner that imposes a meaningful limit on the judicial exception. The courts have indicated that additional elements merely using a computer to implement an abstract idea, adding insignificant extra solution activity, or generally linking use of a judicial exception to a particular technological environment or field of use do not integrate a judicial exception into a “practical application.”
In the present case, the additional limitations beyond the above-noted abstract idea are as follows (where the underlined portions are the “additional limitations” while the bolded portions continue to represent the “abstract idea”):
A device for data processing, which is configured to: provide image data, wherein the image data are specific to a recording of an environment of the driving system,
perform an evaluation of the image data provided, wherein the evaluation takes place based on an application of a machine learning model, by means of which an occlusion label is determined for at least one occlusion of the environment, and
perform the detection of the at least one obstacle on the basis of the occlusion label determined.
For the following reason(s), the examiner submits that the above identified additional limitations do not integrate the above-noted abstract idea into a practical application.
The additional limitation “providing image data” is recited at a high level of generality and amounts to no more than extra-solution activity (i.e., data gathering). The additional limitations “wherein the image data are specific to a recording of an environment of the driving system” and “wherein the evaluation takes place based on an application of a machine learning model” are also recited at a high level of generality such that they merely integrate the judicial exception into a technological environment.
Thus, taken alone, the additional elements do not integrate the abstract idea into a practical application. Further, looking at the additional limitation(s) as an ordered combination or as a whole, the limitation(s) add nothing that is not already present when looking at the elements taken individually. For instance, there is no indication that the additional elements, when considered as a whole, reflect an improvement in the functioning of a computer or an improvement to another technology or technical field, apply or use the above-noted judicial exception to effect a particular treatment or prophylaxis for a disease or medical condition, implement/use the above-noted judicial exception with a particular machine or manufacture that is integral to the claim, effect a transformation or reduction of a particular article to a different state or thing, or apply or use the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is not more than a drafting effort designed to monopolize the exception (MPEP § 2106.05). Accordingly, the additional limitation(s) do/does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
Step 2B: Regarding Step 2B of the 2019 PEG, representative independent claim 9 does not include additional elements (considered both individually and as an ordered combination) that are sufficient to amount to significantly more than the judicial exception for the same reasons to those discussed above with respect to determining that the claim does not integrate the abstract idea into a practical application. As discussed above with respect to the integration of the abstract idea into a practical application, the examiner submits that the additional limitation “providing image data” is recited at a high-level of generality (i.e. as a generic means for data gathering) and does not amount to significantly more than the judicial exception. The additional limitations “an automated and/or at least semi-autonomous driving system”, “wherein the image data are specific to a recording of an environment of the driving system”, and “wherein the evaluation takes place based on an application of a machine learning model” amount to nothing more than applying the exception using generic computer components.
Therefore, claim 9 is ineligible under 35 U.S.C §101.
Regarding Independent Claim 10:
Step 1: Claim 10 is directed to a non-transitory computer-readable storage medium (i.e., a manufacture). Therefore, claim 10 is within at least one of the four statutory categories.
Step 2A Prong 1: Regarding Prong 1 of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether they recite subject matter that falls within one of the following groups of abstract ideas: a) mathematical concepts, b) certain methods of organizing human activity and/or c) mental processes.
Independent claim 10 includes limitations that recite an abstract idea (emphasized below) and will be used as a representative claim for the remainder of the 101 rejection. Claim 10 recites:
A non-transitory computer-readable storage medium comprising instructions which, when executed by a computer, prompt the latter to perform the following steps:
provide image data, wherein the image data are specific to a recording of an environment of the driving system,
perform an evaluation of the image data provided, wherein the evaluation takes place based on an application of a machine learning model, by means of which an occlusion label is determined for at least one occlusion of the environment, and
perform the detection of the at least one obstacle on the basis of the occlusion label determined.
The examiner submits that the foregoing bolded limitations constitute a “mental process” because under its broadest reasonable interpretation, the claim covers performance of the limitations in the human mind. For example, “performing an evaluation of the image data provided…by means of which an occlusion label is determined for at least one occlusion of the environment” in the context of this claim encompasses mentally evaluating an image and mentally determining an occlusion label. The limitation “performing the detection of the at least one obstacle on the basis of the occlusion label determined” in the context of this claim encompasses mentally recognizing an obstacle based on an occlusion label.
Step 2A Prong 2: Regarding Prong II of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether the claim, as a whole, integrates the abstract into a practical application. As noted in the 2019 PEG, it must be determined whether any additional elements in the claim beyond the abstract idea integrate the exception into a practical application in a manner that imposes a meaningful limit on the judicial exception. The courts have indicated that additional elements merely using a computer to implement an abstract idea, adding insignificant extra solution activity, or generally linking use of a judicial exception to a particular technological environment or field of use do not integrate a judicial exception into a “practical application.”
In the present case, the additional limitations beyond the above-noted abstract idea are as follows (where the underlined portions are the “additional limitations” while the bolded portions continue to represent the “abstract idea”):
A non-transitory computer-readable storage medium comprising instructions which, when executed by a computer, prompt the latter to perform the following steps:
provide image data, wherein the image data are specific to a recording of an environment of the driving system,
perform an evaluation of the image data provided, wherein the evaluation takes place based on an application of a machine learning model, by means of which an occlusion label is determined for at least one occlusion of the environment, and
perform the detection of the at least one obstacle on the basis of the occlusion label determined.
For the following reason(s), the examiner submits that the above identified additional limitations do not integrate the above-noted abstract idea into a practical application.
The additional limitation “providing image data” is recited at a high level of generality and amounts to no more than extra-solution activity (i.e., data gathering). The additional limitations “computer”, “wherein the image data are specific to a recording of an environment of the driving system”, and “wherein the evaluation takes place based on an application of a machine learning model” are also recited at a high level of generality such that they merely integrate the judicial exception into a technological environment.
Thus, taken alone, the additional elements do not integrate the abstract idea into a practical application. Further, looking at the additional limitation(s) as an ordered combination or as a whole, the limitation(s) add nothing that is not already present when looking at the elements taken individually. For instance, there is no indication that the additional elements, when considered as a whole, reflect an improvement in the functioning of a computer or an improvement to another technology or technical field, apply or use the above-noted judicial exception to effect a particular treatment or prophylaxis for a disease or medical condition, implement/use the above-noted judicial exception with a particular machine or manufacture that is integral to the claim, effect a transformation or reduction of a particular article to a different state or thing, or apply or use the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is not more than a drafting effort designed to monopolize the exception (MPEP § 2106.05). Accordingly, the additional limitation(s) do/does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
Step 2B: Regarding Step 2B of the 2019 PEG, representative independent claim 10 does not include additional elements (considered both individually and as an ordered combination) that are sufficient to amount to significantly more than the judicial exception for the same reasons to those discussed above with respect to determining that the claim does not integrate the abstract idea into a practical application. As discussed above with respect to the integration of the abstract idea into a practical application, the examiner submits that the additional limitation “providing image data” is recited at a high-level of generality (i.e. as a generic means for data gathering) and does not amount to significantly more than the judicial exception. The additional limitations “an automated and/or at least semi-autonomous driving system”, “wherein the image data are specific to a recording of an environment of the driving system”, and “wherein the evaluation takes place based on an application of a machine learning model” amount to nothing more than applying the exception using generic computer components.
Therefore, claim 10 is ineligible under 35 U.S.C §101.
Regarding Independent Claim 11:
Step 1: Claim 11 is directed to a non-transitory computer-readable storage medium (i.e., a manufacture). Therefore, claim 11 is within at least one of the four statutory categories.
Step 2A Prong 1: Regarding Prong 1 of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether they recite subject matter that falls within one of the following groups of abstract ideas: a) mathematical concepts, b) certain methods of organizing human activity and/or c) mental processes.
Independent claim 11 includes limitations that recite an abstract idea (emphasized below) and will be used as a representative claim for the remainder of the 101 rejection. Claim 11 recites:
A non-transitory computer-readable storage medium comprising instructions which, when executed by a computer, prompt the latter to:
provide training data, wherein the training data comprise at least one sequence of images representing an environment of a driving system during a trip, wherein the training data further comprise annotation data which indicate an occlusion label representing at least one occlusion of the environment during the trip,
performing training of the machine learning model on the basis of the training data, during which training an optical flow in the sequence of images is taken into account in order to predict the occlusion label.
The examiner submits that the foregoing bolded limitations constitute a “mental process” because under its broadest reasonable interpretation, the claim covers performance of the limitations in the human mind. For example, “performing training of the machine learning model on the basis of the training data, during which training an optical flow in the sequence of images is taken into account in order to predict the occlusion label” in the context of this claim encompasses mentally predicting an occlusion label using image data.
Step 2A Prong 2: Regarding Prong II of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether the claim, as a whole, integrates the abstract into a practical application. As noted in the 2019 PEG, it must be determined whether any additional elements in the claim beyond the abstract idea integrate the exception into a practical application in a manner that imposes a meaningful limit on the judicial exception. The courts have indicated that additional elements merely using a computer to implement an abstract idea, adding insignificant extra solution activity, or generally linking use of a judicial exception to a particular technological environment or field of use do not integrate a judicial exception into a “practical application.”
In the present case, the additional limitations beyond the above-noted abstract idea are as follows (where the underlined portions are the “additional limitations” while the bolded portions continue to represent the “abstract idea”):
A non-transitory computer-readable storage medium comprising instructions which, when executed by a computer, prompt the latter to:
provide training data, wherein the training data comprise at least one sequence of images representing an environment of a driving system during a trip, wherein the training data further comprise annotation data which indicate an occlusion label representing at least one occlusion of the environment during the trip,
performing training of the machine learning model on the basis of the training data, during which training an optical flow in the sequence of images is taken into account in order to predict the occlusion label.
For the following reason(s), the examiner submits that the above identified additional limitations do not integrate the above-noted abstract idea into a practical application.
The additional limitation “providing training data, wherein the training data comprise at least one sequence of images representing an environment of a driving system during a trip, wherein the training data further comprise annotation data which indicate an occlusion label representing at least one occlusion of the environment during the trip” is recited at a high level of generality and amounts to no more than extra-solution activity (i.e., data gathering). The additional limitation “computer” is also recited at a high level of generality such that it merely integrates the judicial exception into a technological environment.
Thus, taken alone, the additional elements do not integrate the abstract idea into a practical application. Further, looking at the additional limitation(s) as an ordered combination or as a whole, the limitation(s) add nothing that is not already present when looking at the elements taken individually. For instance, there is no indication that the additional elements, when considered as a whole, reflect an improvement in the functioning of a computer or an improvement to another technology or technical field, apply or use the above-noted judicial exception to effect a particular treatment or prophylaxis for a disease or medical condition, implement/use the above-noted judicial exception with a particular machine or manufacture that is integral to the claim, effect a transformation or reduction of a particular article to a different state or thing, or apply or use the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is not more than a drafting effort designed to monopolize the exception (MPEP § 2106.05). Accordingly, the additional limitation(s) do/does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
Step 2B: Regarding Step 2B of the 2019 PEG, representative independent claim 11 does not include additional elements (considered both individually and as an ordered combination) that are sufficient to amount to significantly more than the judicial exception for the same reasons to those discussed above with respect to determining that the claim does not integrate the abstract idea into a practical application. As discussed above with respect to the integration of the abstract idea into a practical application, the examiner submits that the additional limitation “providing training data…” is recited at a high-level of generality (i.e. as a generic means for data gathering) and does not amount to significantly more than the judicial exception. The additional limitation “a machine learning model” amounts to nothing more than applying the exception in a technological environment.
Therefore, claim 11 is ineligible under 35 U.S.C §101.
Dependent Claims
Dependent claims 2-3, 5, and 12 do not recite any further limitations that cause the claims to be patent eligible. Rather, the limitations of dependent claims are directed toward additional aspects of the judicial exception. Dependent claim 2 is further directed to the abstract ideas of training the machine learning and estimating an optical flow. Dependent claim 3 further describes the image data and is also directed to the abstract idea of limiting image data used for determining the occlusion label to the individual image. Dependent claim 5 is further directed to the abstract idea of evaluating an occlusion label. Dependent claim 12 is further directed to the abstract idea of calculating an essential matrix and performing 3D point triangulation or depth estimation. Therefore, claims 2-3, 5, and 12 are not patent eligible under the same rationale as provided in the rejections of independent claims 1, 7, and 9-11.
As such, claims 1-3, 5, 7, and 9-12 are rejected under 35 USC §101 as being drawn to an abstract idea without significant more, and thus are ineligible.
Eligible Claims
Dependent claim 6 is found to have a practical application of performing at least partially autonomous control of an ego vehicle or robot and is thus considered eligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 6, and 9-10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Park of US 20220250624 A1, filed 11/29/2021, hereinafter “Park”, in view of Yu of US 20190272433 A1, filed 05/19/2019, hereinafter “Yu”.
Regarding claim 1, Park teaches:
A method for detecting at least one obstacle in an automated and/or at least semi-autonomous driving system, said method comprising the following steps: (See at least [0004]: “Embodiments of the present disclosure relate to multi-view geometry-based hazard detection for autonomous and semi-autonomous machine applications…”)
providing image data, wherein the image data are specific to a recording of an environment of the driving system, (See at least [0029]: “…two synchronized stereo cameras may be mounted on the vehicle 600…” & [0059]: “…One or more of the camera(s) (e.g., all of the cameras) may record and provide image data (e.g., video) simultaneously.”)
performing an evaluation of the image data provided, area which identifies at least one or multiple areas in the image data that are occluded by at least one object in the environment, (See at least [0036]: “…the height of object 312 may cause an occlusion of the roadway 310 behind the object 312 from a perspective of the camera 302. As such, when the object 312 is present, there may be a discontinuity in disparity values indicative of a distance jump between the pixel 304 corresponding to a top of the object 312 and the pixel 306 immediately above the pixel 304—e.g., because the distance from the camera 302 to the pixel 304 is different from the distance to the pixel 306, and the camera 302 cannot capture the portions of the roadway 310 occluded by the object 312. In some embodiments, the system may analyze the pixel 304 and the pixel 306 and, when the system determines that a disparity between the pixel 304 and the pixel 306 satisfies a disparity threshold, the system may identify the pixel 304 and label the pixel as a hazard pixel…”)
performing the detection of the at least one obstacle on the basis of the occlusion label determined. (See at least [0036]: “…the system may further identify pixel pairs neighboring the pixel 304 (e.g., pixel identified as a hazard pixel) and determine whether those pixel pairs also show a disparity. When the system identifies one or more hazard pixels, the system may determine that a hazard exists at the location of the pixel 304 and one or more neighboring hazard pixels.”)
However, Park does not explicitly teach evaluating the image data using a machine learning model or an occlusion “map” that identifies areas of occlusion in an image.
Yu, however, teaches using machine learning classifiers to detect the occlusion status of vehicles in a set of input images (See [0049]) and “a visual representation of the vehicle object occlusion data 220 showing an occlusion status for each vehicle object,” where different vehicles in an image are outlined and assigned an occlusion status value (See at least Fig. 12, [0047] & [0070]).
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Park’s method with Yu’s technique of using a machine learning model to evaluate image data and an occlusion map that identifies areas of occlusion in an image. Doing so would be obvious to “effectively and efficiently detect the occlusion status of each vehicle from a set of input images” (See [0049] of Yu).
Regarding claim 6, Park and Yu in combination teach all the limitations of claim 1 as discussed above.
Park additionally teaches:
characterized in that, based on the evaluation of the image data and/or the detection of the at least one obstacle, at least partially autonomous control of an ego vehicle and/or a robot is performed by the driving system, preferably by a motion planning system. (See at least [0047]: “The method 500, at block B508, includes performing one or more operations based at least in part on the hazard. For example, the location of the hazard may then be mapped—e.g., using intrinsic and/or extrinsic parameters of the camera(s)—to a world space location and provided to the drive stack 160—e.g., including one or more planning, control, or obstacle avoidance systems of the ego-machine…”)
NOTE: The limitation(s) “preferably by a motion planning system” is/are contingent due to the term “preferably”, which does not further limit the claim. Therefore, the BRI of claim 6 only requires the limitations “characterized in that, based on the evaluation and/or detection, at least partially autonomous control of an ego vehicle and/or a robot is performed by the driving system”.
Regarding claim 9, Park teaches:
A device for data processing, which is configured to: provide image data, wherein the image data are specific to a recording of an environment of the driving system, (See at least [0029]: “…two synchronized stereo cameras may be mounted on the vehicle 600…” & [0059]: “…One or more of the camera(s) (e.g., all of the cameras) may record and provide image data (e.g., video) simultaneously.” See also [0039] regarding the computing system.)
perform an evaluation of the image data provided, (See at least [0036]: “…the height of object 312 may cause an occlusion of the roadway 310 behind the object 312 from a perspective of the camera 302. As such, when the object 312 is present, there may be a discontinuity in disparity values indicative of a distance jump between the pixel 304 corresponding to a top of the object 312 and the pixel 306 immediately above the pixel 304—e.g., because the distance from the camera 302 to the pixel 304 is different from the distance to the pixel 306, and the camera 302 cannot capture the portions of the roadway 310 occluded by the object 312. In some embodiments, the system may analyze the pixel 304 and the pixel 306 and, when the system determines that a disparity between the pixel 304 and the pixel 306 satisfies a disparity threshold, the system may identify the pixel 304 and label the pixel as a hazard pixel…”)
perform the detection of the at least one obstacle on the basis of the occlusion label determined. (See at least [0036]: “…the system may further identify pixel pairs neighboring the pixel 304 (e.g., pixel identified as a hazard pixel) and determine whether those pixel pairs also show a disparity. When the system identifies one or more hazard pixels, the system may determine that a hazard exists at the location of the pixel 304 and one or more neighboring hazard pixels.”)
However, Park does not explicitly teach evaluating the image data using a machine learning model.
Yu, however, teaches using machine learning classifiers to detect the occlusion status of vehicles in a set of input images (See [0049]).
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Park’s device with Yu’s technique of using a machine learning model to evaluate image data. Doing so would be obvious to “effectively and efficiently detect the occlusion status of each vehicle from a set of input images” (See [0049] of Yu).
Regarding claim 10, Park teaches:
A non-transitory computer-readable storage medium comprising instructions which, when executed by a computer, prompt the latter to perform the following steps: provide image data, wherein the image data are specific to a recording of an environment of the driving system, (See at least [0029]: “…two synchronized stereo cameras may be mounted on the vehicle 600…” & [0059]: “…One or more of the camera(s) (e.g., all of the cameras) may record and provide image data (e.g., video) simultaneously.” See also [0039] regarding the data storage device.)
perform an evaluation of the image data provided, (See at least [0036]: “…the height of object 312 may cause an occlusion of the roadway 310 behind the object 312 from a perspective of the camera 302. As such, when the object 312 is present, there may be a discontinuity in disparity values indicative of a distance jump between the pixel 304 corresponding to a top of the object 312 and the pixel 306 immediately above the pixel 304—e.g., because the distance from the camera 302 to the pixel 304 is different from the distance to the pixel 306, and the camera 302 cannot capture the portions of the roadway 310 occluded by the object 312. In some embodiments, the system may analyze the pixel 304 and the pixel 306 and, when the system determines that a disparity between the pixel 304 and the pixel 306 satisfies a disparity threshold, the system may identify the pixel 304 and label the pixel as a hazard pixel…”)
perform the detection of the at least one obstacle on the basis of the occlusion label determined. (See at least [0069]: “The autonomous vehicle occlusion detection system 210 can process the input image data with the plurality of trained classifiers to produce vehicle object occlusion data 220…” & [0070]: “…FIG. 12 illustrates a visual representation of the vehicle object occlusion data 220 showing an occlusion status for each vehicle object as generated by the autonomous vehicle occlusion detection system 210 of an example embodiment. In the example shown in FIG. 12, red numbers (outlined with squares) are the output of our system and method…”)
However, Park does not explicitly teach evaluating the image data using a machine learning model.
Yu, however, teaches using machine learning classifiers to detect the occlusion status of vehicles in a set of input images (See [0049]).
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Park’s storage medium with Yu’s technique of using a machine learning model to evaluate image data. Doing so would be obvious to “effectively and efficiently detect the occlusion status of each vehicle from a set of input images” (See [0049] of Yu).
Claim(s) 2 is/are rejected under 35 U.S.C. 103 as being unpatentable over Park in view of Yu and further in view of Tsokgas of US 20240071035 A1, filed 02/22/2023, hereinafter “Tsokgas”, and Lin of US 20220156946 A1, filed 10/26/2021, hereinafter “Lin”.
Regarding claim 2, Park and Yu in combination teach all the limitations of claim 1 as discussed above.
Park and Yu in combination do not explicitly teach:
characterized in that training of the machine learning model is based on an occlusion area being determined on the basis of a movement in the recording,
wherein an optical flow is preferably estimated for determining the occlusion area in a sequence of images resulting from the recording, and
the machine learning model is trained in reference to the estimated optical flow in order to determine the occlusion label,
Tsokgas teaches:
characterized in that training of the machine learning model is based on an occlusion area being determined on the basis of a movement in the recording, (See at least [0103]: “At block 510 of FIG. 5, the method 500 may include creating a dataset of synthetic fence bursts by overlaying fences on clean backgrounds. That is, controlled image sequences (e.g., image bursts) may be used to create the dataset, which come with a clean background scene (without an obstruction) that may be used as ground truth during the training process…” & [0104]: “…the synthetic sequences may be generated from videos depicting every day activities in realistic settings, that may include people and/or other objects in the background…”)
wherein an optical flow is preferably estimated for determining the occlusion area in a sequence of images resulting from the recording, and (See at least [0087]: “Continuing to refer to FIG. 3, the multi-frame de-fencing model 320 may apply each image of the input image burst {I.sub.i} 310 to a fence segmentation model 322. The fence segmentation model 322 may be configured and/or trained to output fence segmentation predictions of the foreground occlusion {F.sub.i}. That is, the fence segmentation model 322 may output K fence masks {S.sub.i} 325, as described in detail with reference to FIG. 6. The fence segmentation predictions (e.g., fence masks 325) may be used by the multi-frame de-fencing model 320 to mark the occluded areas in the input image burst 310 that may need to be reconstructed and/or recovered. Alternatively or additionally, the multi-frame de-fencing model 320 may use the fence masks 325 to condition a segmentation-aware optical flow model 326 that may be configured and/or trained to compute K−1 optical flows 327 corresponding to the background scene of interest {B.sub.i} only. The segmentation-aware optical flow model 326 may compute the optical flows 327 using the input image burst 310 and the fence masks 325. As described in detail with reference to FIGS. 7 and 8, the segmentation-aware optical flow model 326 may extract K−1 optical flows {f.sub.kj} 327 between the keyframe I.sub.k and each remaining frame (e.g., image) I.sub.j of the input image burst {I.sub.i} 310.”)
the machine learning model is trained in reference to the estimated optical flow (See at least [0103]: “At block 510 of FIG. 5, the method 500 may include creating a dataset of synthetic fence bursts by overlaying fences on clean backgrounds. That is, controlled image sequences (e.g., image bursts) may be used to create the dataset, which come with a clean background scene (without an obstruction) that may be used as ground truth during the training process…” & [0112]: “…the occlusion-aware optical flow model 326 may be trained and/or configured by applying an optical flow model to corresponding images of the background scenes, which correspond to at least one portion of the dataset, to compute background optical flow maps…”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Park and Yu’s method and Tsokgas’s technique of training the machine learning model based on an occlusion area determined based on a movement in a camera recording, estimating the optical flow from a sequence of images and training the machine learning model in reference to the estimated optical flow in order to determine the occlusion label. Doing so would be obvious since “the occlusions may be preferably thin enough such that there is a high probability that the motion of the user motion as described above is sufficient to allow for a significant portion of the background area to be visible in at least one image of the image burst” (See [0075] of Tsokgas).
Park, Yu, and Tsokgas in combination do not explicitly teach:
wherein the training is preferably performed in the form of a self-supervised training process.
Lin teaches:
wherein the training is preferably performed in the form of a self-supervised training process. (See at least [0084]: “FIG. 11 illustrates an example process for self-supervised learning. In this example, the model 1110 can process an image 1100 (denoted in FIG. 11 as image I.sub.t), an occluded image 1102 (denoted in FIG. 11 as occluded image O(I.sub.t)), and a randomly-shifted occluded image 1104 (denoted in FIG. 11 as randomly-shifted occluded image RS(O(I.sub.t)). The model 1110 can include stages 1112, 1114, 1116, and 1118. The stages 1112, 1114, 1116, and 1118 can generate an optical flow map (V) and a matching map (m(O) using the occluded image 1102 and the random shifted occluded image 1104 as input.”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Park, Yu, and Tsokgas’s method with Lin’s technique of performing a self-supervised training process. Doing so would be obvious “for solving occlusion problems” (See [0043] of Lin).
NOTE: The limitation(s) “wherein an optical flow is preferably estimated for determining the occlusion area in a sequence of images resulting from the recording” and “wherein the training is preferably performed in the form of a self-supervised training process” is/are contingent due to the term “preferably”, which does not further limit the claim. Furthermore, the limitation “in order to determine the occlusion label” is written as intended use since it/they merely recite(s) a desired result. Therefore, the BRI of claim 2 only requires the limitation “the machine learning model is trained in reference to the estimated optical flow”.
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Park in view of Yu and further in view of Tsokgas.
Regarding claim 3, Park and Yu in combination teach all the limitations of claim 1 as discussed above.
Park and Yu in combination do not explicitly teach:
characterized in that the image data, in particular in inference mode, comprise at least one or exactly one individual image, which results from a recording by means of a monocular or stereo camera, wherein, in the alternative where exactly one individual image is used, the image data used for the machine learning model as input for determining the occlusion label are preferably limited to the individual image.
Tsokgas teaches:
characterized in that the image data, in particular in inference mode, comprise at least one or exactly one individual image, (See at least [0099]: “…block 450 may include providing the input image burst 310, the fence masks 325, and the optical flows 327 to the image fusion and inpainting model 328 to reconstruct and/or recover the portions of the keyframe that may be occluded to yield the single keyframe background image {tilde over (B)}.sub.k 330…” & [0046]: “…Because of the opaque nature of the obstruction, the occluded parts of the background scene may need to be hallucinated by the inpainting algorithm.”)
Although Tsokgas does not explicitly teach that the images are obtained by a monocular or stereo camera, Park teaches stereo cameras used to capture images, as discussed above (See [0029] of Park). Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to perform the teachings of Tsokgas using any type of camera, such as the stereo cameras taught by Park, as an obvious design choice.
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Park and Yu’s method with Tsokgas’s technique of the image data comprising at least one image in inference mode and the image data used as input for the machine learning model being limited to the image. Doing so would be obvious since “the occlusions may be preferably thin enough such that there is a high probability that the motion of the user motion as described above is sufficient to allow for a significant portion of the background area to be visible in at least one image of the image burst” (See [0075] of Tsokgas).
NOTE: The limitation(s) “wherein the image data used for the machine learning model as input for determining the occlusion label are preferably limited to the individual image” is/are contingent due to the term “preferably”, which does not further limit the claim. Furthermore, the limitation “for determining the occlusion label” is recited as intended use since it/they merely recite(s) a desired result. Therefore, the BRI of claim 3 only requires the limitation “characterized in that the image data, in particular in inference mode, comprise at least one or exactly one individual image, which results from a recording by means of a monocular or stereo camera”.
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Park in view of Yu and further in view of Wu of US 20230351769 A1, filed 04/29/2022, hereinafter “Wu”.
Regarding claim 5, Park and Yu in combination teach all the limitations of claim 1 as discussed above.
Park additionally teaches:
characterized in that the detection of the at least one obstacle comprises an evaluation of the occlusion label, (See at least Fig. 4, [0088]: “…An original image captured by the sensor pair is shown at 710 and the results of clustering shown at 712. Pixels that were classified as hazard pixels by the method at 524, but that do not form clusters, are shown at 720. Pixels that do form a cluster are shown at 722. Such hazard element clusters are considered to represent actual hazard objects on the path of the ego-machine. In some embodiments, the method 500 may identify one or more hazards as being represented by the sensor data based at least in part on correlating the hazard element cluster 722 with the original image from sensor data…” & [0040]: “…A hazard on the path may be revealed by a hazard pixel, which would be a pixel having a larger disparity than the disparity computed from the path disparity model…”. See also [0064] regarding a discontinuity in disparity values between images from a pair of stereo cameras resulting from the roadway being occluded by an object.)
Although Park does not explicitly teach evaluating the occlusion label using a classifier, Yu teaches using classifiers for detecting the occlusion status of vehicles in an image, as discussed above (See [0049] of Yu). Therefore, one having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to use classifiers, such as those taught by Yu, to perform the evaluating step taught by Park as an obvious design choice, since machine learning classifiers can “effectively and efficiently detect the occlusion status of each vehicle from a set of input images” (See [0049] of Yu).
Park and Yu in combination do not explicitly teach:
wherein the hazardous object in particular comprises cargo that has fallen from a truck.
Wu, however, teaches that hazards may be “foreign material on the roadway” (See at least [0027]). Therefore, one having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it an obvious design choice to include “cargo that has fallen from a truck” as a type of hazard to “assist an ego-machine in detecting hazards within its path of travel” and since “failure to avoid these hazards can result in damage or injury” (See [0027] of Wu).
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Park and Yu’s method with Wu’s technique of evaluating the occlusion label and classification of a detected hazardous object. Doing so would be obvious so to obtain “reliable and precise information about the disparity on the road so that hazard pixels can be robustly recognized from the detected feature offsets” (See [0028] of Wu).
NOTE: The limitation(s) “preferably by means of a classifier” is contingent due to the term “preferably”, which does not further limit the claim. Therefore, the BRI of claim 5 only requires the limitations “characterized in that the detection of the at least one obstacle comprises an evaluation of the occlusion label…during which evaluation a classification of one of the objects, which is in the form of a hazardous object associated with the respective occlusion and detected in the image data, is performed in reference to the occlusion label, wherein the hazardous object in particular comprises cargo that has fallen from a truck”.
Claim(s) 7 and 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yu in view of Tsokgas.
Regarding claim 7, Yu teaches:
A training method for training a machine learning model, said method comprising: (See at least [0049]: “By training a first machine learning classifier for static images and a second machine learning classifier for image sequences (e.g., multiple images or dynamic images), the example embodiments disclosed herein can effectively and efficiently detect the occlusion status of each vehicle from a set of input images…”)
providing training data, wherein the training data comprise at least one sequence of images representing an environment of a driving system during a trip, (See at least Fig. 4 & [0061]: “…The image sequences 260 can be a plurality of the training images provided to the autonomous vehicle occlusion detection system 210 by the training image data collection system 201 as described above…”)
wherein the training data further comprise annotation data which indicate an occlusion label representing at least one occlusion of the environment during the trip, (See at least [0065]: “…the occlusion status for each vehicle object detected in the input image sequences by the autonomous vehicle occlusion detection system 210 can be compared with ground truth data corresponding to the manually generated labeling generated by the manual annotation data collection system 203 in operation block 526. As a result, the second classifier 212 can be trained to generate occlusion status for vehicle objects in image sequences that closely correlates to ground truth data (operation block 528).”)
Yu does not explicitly teach:
performing training of the machine learning model on the basis of the training data, during which training an optical flow in the sequence of images is taken into account in order to predict the occlusion label.
Tsokgas teaches:
performing training of the machine learning model on the basis of the training data, during which training an optical flow in the sequence of images is taken into account (See at least [0103]: “At block 510 of FIG. 5, the method 500 may include creating a dataset of synthetic fence bursts by overlaying fences on clean backgrounds. That is, controlled image sequences (e.g., image bursts) may be used to create the dataset, which come with a clean background scene (without an obstruction) that may be used as ground truth during the training process…” & [0112]: “…the occlusion-aware optical flow model 326 may be trained and/or configured by applying an optical flow model to corresponding images of the background scenes, which correspond to at least one portion of the dataset, to compute background optical flow maps…”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Yu’s method with Tsokgas’s technique of performing training of the machine learning model on the basis of the training data, during which an optical flow in the sequence of images is taken into account. Doing so would be obvious to “effectively and efficiently detect the occlusion status of each vehicle from a set of input images” (See [0049] of Tsokgas).
NOTE: The limitation “in order to predict the occlusion label” is recited as intended use since it merely recites a desired result. Therefore, the BRI of claim 7 does not require predicting the occlusion label.
Regarding claim 11, Yu teaches:
A non-transitory computer-readable storage medium comprising instructions which, when executed by a computer, prompt the latter to: provide training data, wherein the training data comprise at least one sequence of images representing an environment of a driving system during a trip, (See at least Fig. 4 & [0061]: “…The image sequences 260 can be a plurality of the training images provided to the autonomous vehicle occlusion detection system 210 by the training image data collection system 201 as described above…”)
wherein the training data further comprise annotation data which indicate an occlusion label representing at least one occlusion of the environment during the trip, (See at least [0065]: “…the occlusion status for each vehicle object detected in the input image sequences by the autonomous vehicle occlusion detection system 210 can be compared with ground truth data corresponding to the manually generated labeling generated by the manual annotation data collection system 203 in operation block 526. As a result, the second classifier 212 can be trained to generate occlusion status for vehicle objects in image sequences that closely correlates to ground truth data (operation block 528).”)
Yu does not explicitly teach:
performing training of a machine learning model on the basis of the training data, during which training an optical flow in the sequence of images is taken into account in order to predict the occlusion label.
Tsokgas teaches:
performing training of a machine learning model on the basis of the training data, during which training an optical flow in the sequence of images is taken into account in order to predict the occlusion label. (See at least [0103]: “At block 510 of FIG. 5, the method 500 may include creating a dataset of synthetic fence bursts by overlaying fences on clean backgrounds. That is, controlled image sequences (e.g., image bursts) may be used to create the dataset, which come with a clean background scene (without an obstruction) that may be used as ground truth during the training process…” & [0112]: “…the occlusion-aware optical flow model 326 may be trained and/or configured by applying an optical flow model to corresponding images of the background scenes, which correspond to at least one portion of the dataset, to compute background optical flow maps…”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Yu’s storage medium with Tsokgas’s technique of performing training of the machine learning model on the basis of the training data, during which an optical flow in the sequence of images is taken into account. Doing so would be obvious to “effectively and efficiently detect the occlusion status of each vehicle from a set of input images” (See [0049] of Tsokgas).
NOTE: The limitation(s) “in order to predict the occlusion label” is/are recited as intended use since it/they merely recite(s) a desired result. Therefore, the BRI of claim 11 does not require predicting the occlusion label.
Claim(s) 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yu in view of Tsokgas and further in view of Guizilini of US 20220392089 A1, filed 09/29/2021, hereinafter “Guizilini”.
Regarding claim 12, Yu and Tsokgas in combination teach all the limitations of claim 7 as discussed above.
Yu and Tsokgas in combination do not explicitly teach:
characterized in that the training of the machine learning model comprises at least one of the following steps: calculating an essential matrix based on the estimated optical flow and the occlusion label in the form of an occlusion map that indicates the at least one occlusion; performing a 3D point triangulation and/or depth estimation; or applying triangulation in reference to the relative transformations between two images of the image sequence in order to obtain 3D points for each point correspondence from the optical flow.
Guizilini teaches:
characterized in that the training of the machine learning model comprises at least one of the following steps: calculating an essential matrix based on the estimated optical flow and the occlusion label in the form of an occlusion map that indicates the at least one occlusion; performing a 3D point triangulation and/or depth estimation; or applying triangulation in reference to the relative transformations between two images of the image sequence in order to obtain 3D points for each point correspondence from the optical flow. (See at least [0066-0067]: “Triangulation module 620 generally includes instructions that when executed by the one or more processors 305 cause the one or more processors 305 to triangulate the optical flow estimate 140 to generate a triangulated depth map 150, as discussed above in connection with the Combined Embodiment (see Eq. 1 above). Training module 625 generally includes instructions that when executed by the one or more processors 305 cause the one or more processors 305 to extract a set of encoded depth context features 160 from the triangulated depth map 150 using a depth context encoder 155. Training module 625 also includes instructions to combine the set of encoded image context features 125 and the set of encoded depth context features 160 to improve the performance of the second neural network structure in estimating depth 175 and scene flow 170.”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Yu and Tsokgas’s method with Guizilini’s technique of training the machine learning model using a step of performing a 3D point triangulation and/or depth estimation. Doing so would be obvious “to improve the performance of the second neural network structure in estimating depth 175 and scene flow 170” (See [0067] of Guizilini).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NIKKI MARIE M MOLINA whose telephone number is (571)272-5180. The examiner can normally be reached M-F, 9am-6pm PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Aniss Chad can be reached at 571-270-3832. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NIKKI MARIE M MOLINA/Examiner, Art Unit 3662
/ANISS CHAD/Supervisory Patent Examiner, Art Unit 3662
Prosecution Insights