DETAILED ACTION
This Action is responsive to Claims filed 02/06/2024.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 02/06/2024 was filed before the mailing date of the first Action. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Drawings
Receipt of Drawings filed 02/06/2024 is acknowledged. These Drawings are acceptable.
Status of the Claims
Claims 1-20 are currently pending.
Claim Objections
Claims 2 and 17 objected to because of the following informalities:
Claim 2: should be “wherein determining…”
Claim 17: “labeling data for a machine learning (ML) models” is not grammatically correct
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-16 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 1 and 9 recite “scenes” associated with “nodes,” which is presumably the nodes of the knowledge graph as per subsequent independent claim 17; however, nowhere in Claims 1-16 such a knowledge graph claimed, save for Claim 11, which does not address this association. The paragraphs of the Specification pertinent to these embodiments also do not address a knowledge graph or the associated between “scenes” and “nodes” clearly. This creates ambiguity as to what a “node” refers to in the claim. The dependent claims 2-8 and 10-16 do not rectify this ambiguity, and are similarly rejected. The Examiner notes amending or adding similar limitations or claims to claims 17-20 would address the issue for Claims 1-8, but also duplicate much of the embodiment.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more; and because the claims as a whole, considering all claim elements both individually and in combination, do not amount to significantly more than the abstract idea, see Alice Corporation Pty. Ltd. v. CLS Bank International, et al, 573 U.S. (2014). In determining whether the claims are subject matter eligible, the Examiner applies the 2019 USPTO Patent Eligibility Guidelines. (2019 Revised Patent Subject Matter Eligibility Guidance, 84 Fed. Reg. 50, Jan. 7, 2019.)
Step 1:
Claims 1-8 recite a method for labeling data, which falls under the statutory category of a process. Claims 9-16 recite a system, which falls under the statutory category of a machine. Claims 17-20 recite a method for labeling data, which falls under the statutory category of a process.
Step 2A – Prong 1:
Claim 1:
Claim 1 recites an abstract idea, law of nature, or natural phenomenon. The limitations of “identifying a first set of scenes associated with a first set of nodes and relations in a first sequence utilizing the one or more datasets and labels to create a first window;”, “identifying a second set of scenes associated with a second set of nodes and relations in a second sequence utilizing the one or more datasets and labels to create a second window, wherein the second window is in a future position compared to the first window;”, “extracting a set of observed entities within the first window and the second window in response to inspecting one or more scenes across at least the first window and second window;”, “determining the one or more unobserved entities associated within a target scene utilizing at least the set of observed entities, wherein the target scene is sequentially between the first window and the second window;”, and “augmenting the dataset with additional scene labels associated with the unobserved entities.” under the broadest reasonable interpretation, cover a mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper.
Identifying first and second generic “scenes” from a dataset is practically performed within the human mind or with the aid of pen and paper. Extracting unobserved generic “scenes” from the first and second window is practically performed within the human mind or with the aid of pen and paper. Determining unobserved generic “entities” from a target scene is practically performed within the human mind or with the aid of pen and paper. Augmenting the generic dataset with additional labels is practically performed within the human mind or with the aid of pen and paper.
Step 2A – Prong 2:
The additional elements of claim 1 do not integrate the abstract idea into a judicial exception. The claim recites the additional elements “data”, “dataset”, “scene”, and “window” are recognized as generic computer components recited at a high level of generality. Although they have and execute instructions to perform the abstract idea itself, this also does not serve to integrate the abstract idea into a practical application as it merely amounts to instructions to "apply it." (See MPEP 2106.04(d)(2) indicating mere instructions to apply an abstract idea does not amount to integrating the abstract idea into a practical application).
The additional elements of “machine learning models” are recognized as non-generic computer components, but are recited at a high level of generality and are found to generally link the abstract idea to a particular technological environment or field of use (See MPEP 2106.05(h)).
The additional elements recited in the limitation “receiving one or more datasets that includes one or more labels associated with objects within the dataset;” is found to be pre- or post-extra-solution activity steps (See MPEP 2106.05(g)(iii) first list).
Step 2B:
The only limitation on the performance of the described method is a limitation reciting “data”, “dataset”, “scene”, and “window” These elements are insufficient to transform a judicial exception to a patentable invention because the recited elements are considered insignificant extra-solution activity (generic computer system, processing resources, links the judicial exception to a particular, respective, technological environment). The claim thus recites computing components only at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components; mere instructions to apply an exception using a generic computer component cannot provide an inventive concept (see MPEP 2106.05(f)).
The additional elements of “machine learning models” are recognized as non-generic computer components, but are recited at a high level of generality and are found to generally link the abstract idea to a particular technological environment or field of use (See MPEP 2106.05(h)).
The additional elements recited in the limitation “receiving one or more datasets that includes one or more labels associated with objects within the dataset;” is found to be well-understood, routine, or conventional activity (See MPEP 2106.05(d)(II)(iv) third list).
Taken alone or in ordered combination, these additional elements do not amount to significantly more than the above-identified abstract idea. There is no indication that the combination of elements improves the functioning of a computer or improves any other technology. Their collective functions merely provide conventional computer implementation.
For the reasons above, claim 1 is rejected as being directed to non-patentable subject matter under §101.
Claim 9:
Claim 9 recites an abstract idea, law of nature, or natural phenomenon. The limitations of “identify a first set of scenes associated with a first set of nodes and relations in a first sequence utilizing the one or more datasets and labels to create a first window; identify a second set of scenes associated with a second set of nodes and relations in a second sequence utilizing the one or more datasets and labels to create a second window, wherein the second window is in a future position compared to the first window; extract a set of observed entities within the first window and the second window in response to inspecting one or more scenes across at least the first window and second window; determine the one or more unobserved entities associated within a target scene utilizing at least the set of observed entities, wherein the target scene is sequentially between the first window and the second window; and augment the dataset with additional scene labels associated with the unobserved entities.” under the broadest reasonable interpretation, cover a mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper.
Identifying first and second generic “scenes” from a dataset is practically performed within the human mind or with the aid of pen and paper. Extracting unobserved generic “scenes” from the first and second window is practically performed within the human mind or with the aid of pen and paper. Determining unobserved generic “entities” from a target scene is practically performed within the human mind or with the aid of pen and paper. Augmenting the generic dataset with additional labels is practically performed within the human mind or with the aid of pen and paper.
Step 2A – Prong 2:
The additional elements of claim 9 do not integrate the abstract idea into a judicial exception. The claim recites the additional elements “one or more sensors configured to retrieve image data; and a controller”, “data”, “dataset”, “scene”, and “window” are recognized as generic computer components recited at a high level of generality. Although they have and execute instructions to perform the abstract idea itself, this also does not serve to integrate the abstract idea into a practical application as it merely amounts to instructions to "apply it." (See MPEP 2106.04(d)(2) indicating mere instructions to apply an abstract idea does not amount to integrating the abstract idea into a practical application).
The additional elements recited in the limitation “receive one or more datasets including the image data, wherein the one or more datasets further includes one or more labels associated with objects within the dataset;” is found to be pre- or post-extra-solution activity steps (See MPEP 2106.05(g)(iii) first list).
Step 2B:
The only limitation on the performance of the described method is a limitation reciting “one or more sensors configured to retrieve image data; and a controller”, “data”, “dataset”, “scene”, and “window” These elements are insufficient to transform a judicial exception to a patentable invention because the recited elements are considered insignificant extra-solution activity (generic computer system, processing resources, links the judicial exception to a particular, respective, technological environment). The claim thus recites computing components only at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components; mere instructions to apply an exception using a generic computer component cannot provide an inventive concept (see MPEP 2106.05(f)).
The additional elements recited in the limitation “receive one or more datasets including the image data, wherein the one or more datasets further includes one or more labels associated with objects within the dataset;” is found to be well-understood, routine, or conventional activity (See MPEP 2106.05(d)(II)(iv) third list).
Taken alone or in ordered combination, these additional elements do not amount to significantly more than the above-identified abstract idea. There is no indication that the combination of elements improves the functioning of a computer or improves any other technology. Their collective functions merely provide conventional computer implementation.
For the reasons above, claim 9 is rejected as being directed to non-patentable subject matter under §101.
Claim 17:
Claim 17 recites an abstract idea, law of nature, or natural phenomenon. The limitations of “generating one or more datasets utilizing at least the one or more images and the machine learning model;”, “identifying a first set of scenes associated with a first set of nodes and relations in a first sequence utilizing the one or more datasets and labels to create a first window;”, “identifying a second set of scenes associated with a second set of nodes and relations in a second sequence utilizing the one or more datasets and labels to create a second window, wherein the second window is in a future position compared to the first window;”, “extracting a set of observed entities within the first window and the second window in response to inspecting one or more scenes across at least the first window and second window; determining the one or more unobserved entities associated within a target scene utilizing at least the set of observed entities, wherein the target scene is sequentially between the first window and the second window;”, and “and augmenting the dataset with additional scene labels associated with the unobserved entities.” under the broadest reasonable interpretation, cover a mental process including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper.
Generating a dataset of images is practically performed within the human mind or with the aid of pen and paper. Identifying first and second generic “scenes” from a dataset is practically performed within the human mind or with the aid of pen and paper. Extracting unobserved generic “scenes” from the first and second window is practically performed within the human mind or with the aid of pen and paper. Determining unobserved generic “entities” from a target scene is practically performed within the human mind or with the aid of pen and paper. Augmenting the generic dataset with additional labels is practically performed within the human mind or with the aid of pen and paper.
Step 2A – Prong 2:
The additional elements of claim 17 do not integrate the abstract idea into a judicial exception. The claim recites the additional elements “a computer-implemented method”, “images”, “sensors”, “data”, “dataset”, “scene”, and “window” are recognized as generic computer components recited at a high level of generality. Although they have and execute instructions to perform the abstract idea itself, this also does not serve to integrate the abstract idea into a practical application as it merely amounts to instructions to "apply it." (See MPEP 2106.04(d)(2) indicating mere instructions to apply an abstract idea does not amount to integrating the abstract idea into a practical application).
The additional elements of “machine learning models” are recognized as non-generic computer components, but are recited at a high level of generality and are found to generally link the abstract idea to a particular technological environment or field of use (See MPEP 2106.05(h)).
The additional elements recited in the limitations “receiving one or more images from one or more sensors;” and “receiving the one or more datasets that includes one or more labels associated with objects within the dataset;” are found to be pre- or post-extra-solution activity steps (See MPEP 2106.05(g)(iii) first list).
Step 2B:
The only limitation on the performance of the described method is a limitation reciting “a computer-implemented method”, “images”, “sensors”, “data”, “dataset”, “scene”, and “window” These elements are insufficient to transform a judicial exception to a patentable invention because the recited elements are considered insignificant extra-solution activity (generic computer system, processing resources, links the judicial exception to a particular, respective, technological environment). The claim thus recites computing components only at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components; mere instructions to apply an exception using a generic computer component cannot provide an inventive concept (see MPEP 2106.05(f)).
The additional elements of “machine learning models” are recognized as non-generic computer components, but are recited at a high level of generality and are found to generally link the abstract idea to a particular technological environment or field of use (See MPEP 2106.05(h)).
The additional elements recited in the limitation “receiving one or more images from one or more sensors;” and “receiving the one or more datasets that includes one or more labels associated with objects within the dataset;” is found to be well-understood, routine, or conventional activity (See MPEP 2106.05(d)(II)(iv) third list).
Taken alone or in ordered combination, these additional elements do not amount to significantly more than the above-identified abstract idea. There is no indication that the combination of elements improves the functioning of a computer or improves any other technology. Their collective functions merely provide conventional computer implementation.
For the reasons above, claim 17 is rejected as being directed to non-patentable subject matter under §101.
Dependent Claims:
Claim 2 (claim 10) recites an abstract idea mental process step “determining the one or more unobserved entities includes determining a confidence score associated with the one or more unobserved entities and exceeding a threshold associated with the confidence score.”
Claim 3 recites refinements to the data manipulated.
Claim 4 (claim 12) recites refinements to the data manipulated.
Claim 5 recites refinements to the data manipulated.
Claim 6 (claim 14) recites refinements to the data manipulated.
Claim 7 recites refinements to the data manipulated. Additional elements recited in Claim 7 are recognized as non-generic computer components, but are recited at a high level of generality and are found to generally link the abstract idea to a particular technological environment or field of use (See MPEP 2106.05(h)).
Claim 8 (claim 16) recites an abstract idea mental process step “removing duplicates associated with a class of objects.”
Claim 11 recites an abstract idea mental process step “generate a knowledge graph utilizing at least the dataset.”
Claim 13 recites refinements to the data manipulated.
Claim 15 recites refinements to the data manipulated. Additional elements recited in Claim 7 are recognized as non-generic computer components, but are recited at a high level of generality and are found to generally link the abstract idea to a particular technological environment or field of use (See MPEP 2106.05(h)).
Claim 18 recites an abstract idea mental process step “creating a knowledge graph utilizing at least the first set of scenes, the first set of nodes and relations in the first sequence, and the one or more labels.”
Claim 19 recites an abstract idea mental process step “creating a knowledge graph utilizing at least the first set of scenes, the first set of nodes and relations in the first sequence, the one or more labels, and the additional scene labels.”
Claim 20 recites refinements to the data manipulated.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-7, 9-15, and 17-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tokmakov et al. (Learning to Track with Object Permanence, 2021), hereinafter Tokmakov1, and Tokmakov et al. (Object Permanence Emerges in a Random Walk along Memory, 2022), hereinafter Tokmakov2.
In regards to claim 1: The present invention claims: “A method of labeling data for machine learning (ML) models, the method comprising: receiving one or more datasets that includes one or more labels associated with objects within the dataset;” Tokmakov1 teaches generating and/or receiving a set of synthetic videos with accurate labels for training their model “Using the Parallel Domain (PD) [1] simulation platform, we generate a dataset of synthetic videos that automatically provides accurate labels for all objects, irrespective of their visibility (see Figure 2). We then use this dataset to analyze various approaches for supervising tracking behind occlusions with both ground-truth and pseudo-ground-truth labels for occluded instances.” (Page 10861, left column).
“identifying a first set of scenes…in a first sequence utilizing the one or more datasets and labels to create a first window;” Tokmakov1 teaches “Our model, shown in Figure 3, takes a sequence of frames…as input. Each frame is passed through the backbone f individually to obtain feature maps…which, per CenterTrack formalism, encode the locations of visible objects in that frame - an instantaneous representation.” (Page 10862, right column, mapping a set of frames into a window of scenes).
“identifying a second set of scenes…in a second sequence utilizing the one or more datasets and labels to create a second window, wherein the second window is in a future position compared to the first window;” Tokmakov1’s method is trained on all past data within the video sequence of frames (Page 10862), therefore making new frames or synthesized frames future to previously learned frames. Tokmkov1 also teaches synthesizing a video sequence with labeled entities, including those occluded by other entities “Concretely, we introduce two thresholds Tvis and Toccl. Then, starting from the first frame in a sequence O1, for every object o1i, if vis1i < Tvis the object is treated as a negative, if Tvis < vis1i < Toccl it is ignored (the model is not penalized for predicting it), and finally, if vis1i > Toccl its marked as visible and used to produce the labels. The same procedure is repeated for every frame in a sequence, with the only difference that, starting from frame 3, objects that were marked as visible for two consecutive frames in the past are treated as positives regardless of their visibility status in the current frame.” (Page 10863, right column, mapping to a second sequence of scenes in a window).
“extracting a set of observed entities within the first window and the second window in response to inspecting one or more scenes across at least the first window and second window;” Section 3.2 and 3.3.1 of Tokmakov1 details how objects within the real or synthetic video frames are extracted into features sets and their positions detected (Figure 3, as well, shows the process).
“determining the one or more unobserved entities associated within a target scene utilizing at least the set of observed entities, wherein the target scene is sequentially between the first window and the second window;” Section 3.3.1 of Tokmakov1 details how the real and synthetic data is used to detect and label entities previously detected in the video frames, but that are occluded by other entities in subsequent frames (Page 10863).
“and augmenting the dataset with additional scene labels associated with the unobserved entities.” Tokmakov1 teaches “Instead, we propose to generate pseudoground-truth labels for supervising our model by propagating the occluded object locations with their last observed velocity in 3D, and projecting the resulting centers to the camera frame, which is made possible by the availability of the full ground truth information in our synthetic dataset.” (Page 10864, left column).
Tokmakov1 fails to explicitly teach the association of objects within the frames to nodes/edges of a graph as is assumed (See 112(b) Rejection above) to be claimed in “…associated with a first set of nodes and relations…” and “…associated with a second set of nodes and relations…” within the context of their system. However, Tokmakov2, which builds on Tokmakov1, teaches “Rather than directly supervising the locations of invisible objects, in this work we propose a self-supervised objective that encourages object permanence to naturally emerge from data (see Figure 2). To this end, we leverage the recent Contrastive Random Walk objective of Jabri et al. (2020), which models space-time correspondence as a Markov walk on a spatio-temporal graph of patches (i.e. from a video).” (Page 1) and “Concretely, we consider the spatio-temporal graph of memory states, where nodes correspond to potential object locations (Figure 3). For visible instances, transition probability is supervised directly (i.e. we assume labels for visible objects during training). During occlusions, we employ the objective of Jabri et al. (2020), supervising the walk with ground truth object locations before and after the occlusion (see Figure 4). By optimizing for correspondence on the resulting graph of memories, we learn a representation that stores object-centric information in a spatially-grounded manner even for unlabeled, invisible objects.” (Page 2).
Tokmakov uses their method in Tokmakov2 on the method of Tokmakov1 “We then evaluate our method on a synthetic multi-object tracking benchmark introduced in (Tokmakov et al., 2021), and demonstrate that it is effective at handling occlusions despite requiring less supervision.” (Page 2). A person of ordinary skill in the art at the time of the Applicant’s filing would have been aware of Tokmkov1’s method and Tokmakov’s subsequent extended research into Tokmakov2 and the benefits to the system realized representing the video data as a spatio-temporal graph.
In regards to claim 2: The present invention claims: “determining the one or more unobserved entities includes determining a confidence score associated with the one or more unobserved entities and exceeding a threshold associated with the confidence score.” Tokmakov1 teaches “Concretely, we introduce two thresholds Tvis and Toccl. Then, starting from the first frame in a sequence O1, for every object o1i, if vis1i < Tvis the object is treated as a negative, if Tvis < vis1i < Toccl it is ignored (the model is not penalized for predicting it), and finally, if vis1i > Toccl its marked as visible and used to produce the labels.” In determining if an object is occluded by another object in a frame. Tokmakov2 also makes use of a confidence and threshold value in detecting or predicting occluded objects in a frame (Section 3.5, Algorithm 1).
In regards to claim 3: The present invention claims: “wherein the multiple positions includes one or more window of future scenes or one or more windows of past scenes.” Tokmakov1 utilizes past and real time data (Page 10861).
In regards to claim 4: The present invention claims: “wherein the multiple positions includes one or more window of future scenes and one or more windows of past scenes.” While Tokmakov1 explicitly teaches “On the other hand, offline approaches [5, 6, 10, 34] first
build a spatio-temporal graph spanning the whole video, with object detections as nodes [5]. Edge costs are defined based on overlap between detections [30, 42, 65], their appearance similarity [10, 39, 47, 62], or motion-based models [2, 13, 16, 34, 44]. The association can then be formulated as maximum flow [6] or, minimum cost problem [30, 34]. While these methods can handle complex scenarios, they are not practical due to their non-casual nature
and computational complexity. In contrast, our approach does not require future frames and runs in real time.” (Page 10861), and therefore does not use specifically future frames, both Tokmakov1 and Tokmakov2 make inferences for occluded objects in real time based on past and current video frame data, which inherently may include frame data in the future of a target object.
In regards to claim 5: The present invention claims: “wherein the dataset includes time-series data.” Both Tokmakov1 and Tokmakov2 operate on videos/video frame data of varying lengths (Tokmakov2, page 6).
In regards to claim 6: The present invention claims: “wherein the unobserved entities includes two or more objects associated with the one or more labels.” Tokmakov1 uses their method on any occluded entity in the video frames, either a pedestrian (Figure 1, for example) and other vehicles (Figure 2, for example).
In regards to claim 7: The present invention claims: “wherein the dataset is associated with autonomous driving, natural language processing, or audio information.” Both Tokmakov1 and Tokmakov2 are geared towards computer vision as used in autonomous driving (Tokmakov1 Figure 2 and 3, at least).
In regards to claims 9-10, 12, and 14: Claims 9-10, 12, and 14 recite similar limitations to claims 1-2, 4, and 6, with the exception of “A system, comprising: one or more sensors configured to retrieve image data; and a controller configured to:” of Claim 9. The Examiner submits a system in an autonomous vehicle implementing the algorithms of Tokmakov1 and Tokmakov2 would necessarily contain sensors and a controller to manipulate the sensed image data; therefore, both sets of claims are similarly rejected.
In regards to claim 11: The present invention claims: “wherein the controller is further configured to generate a knowledge graph utilizing at least the dataset.” The Examiner submits in a processor implementing a combination of Tokmakov1 and Tokmakov2 would inherently need to generate the knowledge graph of Tokmakov2.
In regards to claim 13: The present invention claims: “wherein the dataset is PandaSet” While neither of Tokmakov1 and Tokmakov2 use PandaSet in their systems, a cursory search shows PandaSet was used for more specifically autonomous vehicle systems prior to the Applicant’s filing date. It would have been obvious to one having ordinary skill in the art at the time the invention was made to use PandaSet for an autonomous vehicle system based on a combination of Tokmakov1 and Tokmakov2, since the operation on video data would be the same (See references not cited below).
In regards to claim 15: The present invention claims: “wherein the dataset is associated with an industrial application.” Based on Applicant’s Specification [0021], the Examiner submits Tokmakov1 and Tokmakov2’s use of the KITTI dataset (Figures 1 and 6, respectively, at least) reads on using a dataset relevant to autonomous vehicle operation.
In regards to claims 17-19: Claims 17-19 recite similar limitations to claims 1-10, 12, 14, and 16, with the exception of “A computer-implemented method of labeling data for a machine learning (ML) models, the method comprising:” of Claim 17, therefore, both sets of claims are similarly rejected.
Claim(s) 8 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tokmakov1 and Tokmakov2 as applied to claim 1 and 9 above, and further in view of Lobato et al. (Redundancy Mitigation Mechanism for Collective Perception in Connected and Autonomous Vehicles, 2022), hereinafter Lobato.
In regards to claim 8: The combination of Tokmakov1 and Tokmakov2 does not explicitly teach removing redundant features of repeatedly observed entities as recited in “wherein extracting the set of observed entities includes removing duplicates associated with a class of objects.” However, Lobato, in a similar field of endeavor of autonomous vehicle vision, teaches “Therefore, we propose a reliable redundancy mitigation mechanism for collective perception services to reduce the transmission of inefficient messages, which is called VILE. Knowledge, selection, and perception are the three phases of the cooperative perception process developed in VILE. The results have shown that VILE is able to reduce it the absolute number of redundant objects of 75% and generated packets by up to 55%. Finally, we discuss possible research challenges and trends.” (Abstract) and “Evaluation results demonstrate the efficiency of VILE mechanism to reduce the amount of redundant data by up to 85%, while maintaining the number of detected objects compared to the basic transmission approach defined by European Telecommunications Standards Institute (ETSI) [32].”(Page 2).
Lobato highlights the need to efficiently communicate data for autonomous vehicle operation (Abstract, Introduction), and addresses several state of the art methods and techniques for removing redundant sensed entities in their disclosure (Introduction). It would have been obvious to one of ordinary skill in the art at the time of the Applicant’s filing to reduce the size of training datasets or sensed data in an autonomous vehicle system to remove redundant observed entities or feature sets.
In regards to claim 16: Claim 16 recite similar limitations to claim 8, with the exception of “A system, comprising: one or more sensors configured to retrieve image data; and a controller configured to:” of Claim 9. The Examiner submits a system in an autonomous vehicle implementing the algorithms of Tokmakov1 and Tokmakov2 would necessarily contain sensors and a controller to manipulate the sensed image data; therefore, both sets of claims are similarly rejected.
Claim(s) 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tokmakov1 and Tokmakov2 as applied to claim 17 above, and further in view of Luettin et al. (A Survey on Knowledge Graph-based Methods for Automated Driving, 2022), hereinafter Luettin.
In regards to claim 20: While Tokmakov1 and Tokmakov2 includes both a graph representation of sensed data as well as labels for the sensed entities, the combination fails to explicitly teach the features recited in “wherein the knowledge graph includes a link between an entity graph and label graph.” However, Luettin, in a similar field of endeavor of autonomous vehicle vision, teaches “However, there are still unsolved problems to guarantee reli ability and safety of automated systems, especially to effectively incorpo rate all available information and knowledge in the driving task. Knowl edge graphs (KG) have recently gained significant attention from both industry and academia for applications that benefit by exploiting struc tured, dynamic, and relational data.” (Abstract) and several versions of a knowledge graph composed of nodes, edges, and associated labels in Section 2.2.
Luettin highlights the benefits to data representation granted by using knowledge graphs for driving data (Abstract). It would have been obvious to one of ordinary skill in the art at the time of the Applicant’s filing to represent the data used in a combination of Tokmakov1 and Tokmakov2 in one or more of the known methods indicated in Luettin in order to better represent the driving data.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Xiao, Pengchuan, et al. "Pandaset: Advanced sensor suite dataset for autonomous driving." 2021 IEEE international intelligent transportation systems conference (ITSC). IEEE, 2021. (Demonstrates use of PandaSet would have been known in the art before the Applicant’s filing)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GRIFFIN T BEAN whose telephone number is (703)756-1473. The examiner can normally be reached M - F 7:30 - 4:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li Zhen can be reached at (571) 272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GRIFFIN TANNER BEAN/Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights