DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submissions, filed on April 28th, 2026, have been entered.
Status of Claims
Claims 1-9 and 11-21 are pending, claims 1, 6-9, 11-12 and 17-20 have has been amended, claim 21 has been added. Claims 1-9 and 11-21 remains rejected.
Response to Argument(s)
101 rejection:
Regarding the amendment to the independent claims 1, 12 and 20, which has reflected the suggestion the examiner provided during the interview hence, overcome the previously stated 101 rejection. The Applicants’ arguments found to be persuasive, the 101 rejection has been withdrawn.
112(b):
The amendment has corrected the terms to no longer be indefinite for the corresponding claims 6-7 and 17-28. Therefore, the 112(b) rejection has been withdrawn.
103 rejection:
Although the amendment to the independent claims 1, 12 and 20, has narrowed down the scope of the claims which had introduced new scope to the claims and bring prosecution forward, previously stated 103 rejection has been overcome. However, the examiner finds the Applicants’ argument to be some non-persuasive.
In pages 8-11 of the remarks, the Applicants argue that the proposed Kalogeiton and other cited portions of the cited references, considered alone or in combination, does not teach or suggest the features of the claims (independent claim 1 and its analogous independent claims 12 and 20):
“…the object embedding containing data elements for differentiating objects within a single object category;
after receiving the output data,…”
In support of the above argument, the Applicants assert that Kalogeiton discloses P--_o and P_a are the outputs of the two branches that predict the object and action labels, hence, does not teach or suggest “the object embedding containing data elements for differentiating objects within a single object category”. Specifically, Kalogeiton’s “action labels” are not the same as and does not suggest an “object embedding…for differentiating objects within a signal object category.”
The Applicants further state that Kalogeiton discloses “an end-to-end multi-task objective that jointly learns object-action relationships” where P--_o and P_a are the outputs of the two branches that predict the object and action labels therefore, does not teach or suggest “after….” The action appears to equate the claimed “automated action” either to Kalogeiton’s process by which an action label is generated, or “for tracking of the object over frames”. Therefore, Kalogeiton’s “object and action labels” are not the same as and do not suggest “determining whether to perform an automated action [after receiving the output data]”. The Applicants further assert that Kalogeiton’s generation of an action label is not the same as the recited, analogously, “subsequent determination” of a system action based on that data, that, the automated action is a separate functional step that occurs after the model has already produced the detection and embedding results, therefore, by having Kalogeiton’s “action detection” being the output itself, it cannot be simultaneously serving as the subsequent automated action triggered by that output.
Secondly, the Applicants finds that the recited “automated action” cannot be Kalogeiton’s “tracking” since, such tracking is not using the object detection result, and the object embedding for the target object. Kalogeiton’s mere mention that detections are linked or tracked overtime is not the same as and does not suggest any determination, much less a determination to track an object using the object embedding which contains data elements for differentiating objects within a single object category.
Examiner’s reply:
The examiner respectfully finds the Applicants’ arguments to be non-persuasive. The Applicants are reminded that the claims are construed based on BRI (broadest reasonable interpretation) scope in light of the specification which, the examiner finds the claims to fall within the scope of the proposed prior art Kalogeiton.
Specifically, the examiner finds the argued limitations as mentioned above, to be taught in Kalogeiton such as,
For the limitation of “…the object embedding containing data elements for differentiating objects within a single object category”, Kalogeiton’s section 2, last paragraph, presented in page 4165, which suggests and/or teaches that existing approaches have already define that “categories rely on attributes. Attributes have been used for human actions…each action class has an intra-class variability…attributes are relevant for each class” which indicates that actions are categorized into classes and each class has intra-class variability, indicating that the class attributes within the same class can be variable [different] therefore, as the recited “object embedding” being mapped to be based on Kalogeiton’s attributes/action label data, the object embedding would be understood to contain attributes [or data elements] that carry intra-class variability [these elements can be variable in the same class/category], the action being associated with an object hence, being data/information of an object, in other words, is analogous to same class of object’s action, there are elements that differentiate it from other object elements within the same class/same category.
For the limitation of “after receiving the output data,…”, which merely indicate that the limitation following this feature is a after step of the previous step, therefore, the limitation of “determine whether to perform….the object embedding for the target object” is the step that happen after receiving the output data, wherein Kalogeiton also teaches this following limitation to happen after the output data has been received of “determining whether to perform an automated action using the output data” since Kalogeiton teaches obtain an output data of the end-to-end multitask network architecture of Fig. 2 (the output of this network is analogous to the recited output data), wherein the output of the network of Fig. 2 includes an output of the object detection branch and an output of an action detection branch, the output of the object detection branch is an object detected in the image (therefore indicating that the object is whether detected in the image through the use of candidate bounding boxes, see section 3.1, 1st Par.) and the output of the action detection branch is the object’s detection of its associated action (is analogous to the recited object embedding, which as discussed above to have data elements for differentiating objects within a single object category). Importantly, the output of the network include object and its action, which, as described in section 1, 1st Par., which discloses “a detector to localize human actions in individual frames, and then either link them or track them over time to create spatio-temporal detections” indicating the output of the network is used to track the object over frames overtime. Importantly, the recited “automated action” in the claim merely indicate, based on BRI, an action that can be automatic/automated, therefore, if a network such as the network of Fig 2 produce output for a detector that can track an object through frames is an approach to track object automatically using neural network, what else but this approach is an automatic/automated action? The claimed feature of “automated action” cannot be imported direct support from the specification to be its instant scope, but based solely on the claim’s language alone, the feature falls within the scope of the proposed prior arts.
Therefore, the 103 rejection remain.
Claim Objections
Claim 21 is objected to because of the following informalities:
The reference “The computer storage media of claim 20” should be read as “The one or more non-transitory computer storage media of claim 20” to follow proper antecedent basis issue. Appropriate correction is required to avoid indefiniteness antecedent basis 112(b) issues.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-9 and 12-21 are rejected under 35 U.S.C. 103 as being unpatentable over Vicky Kalogeiton et. al. (“Joint Learning of Object and Action Detectors, 2017, Proceedings of the IEEE International Conference on Computer Vision, pp. 4163-4172” hereinafter as “Kalogeiton”) in view of Nishitkumar Ashokkumar Desai et. al. (“US 11,263,795 B1” hereinafter as “Desai”).
Regarding claim 1, Kalogeiton explicitly teaches a system comprising one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising (abstract discloses the use of machine learning which indicates the use of a computer to have computer components such as storage storing instructions to be executed by a processor for the operations of the invention; FIG. 2 shows that the input into the machine learning model being an image and being maintained for the whole processing of the image data, by BRI [broadest reasonable interpretation] cover the scope of the limitation); providing, by the system and to a machine learning model, data that represents an image (FIG. 2 shows the image is being input into the machine learning model of its data, hence being part of the system of the invention); receiving, from the machine learning model, output data that includes (FIG. 2 shows that the output from the model includes two branches) i) an object detection result that indicates whether a target object is detected in the image (FIG. 2 shows that one branch is for object detection which indicates whether an object is detected in the image, by BRI, covers the scope of the limitation) and ii) an object embedding for the target object (another branch of FIG. 2 shows action detection which include action label [by BRI, can be understood to be an object embedding for the target object as claimed]), the object embedding containing data elements for differentiating objects within a single object category (Kalogeiton’s section 2, last paragraph, presented in page 4165, which suggests and/or teaches that existing approaches have already define that “categories rely on attributes. Attributes have been used for human actions…each action class has an intra-class variability…attributes are relevant for each class” which indicates that actions are categorized into classes and each class has intra-class variability, indicating that the class attributes within the same class can be variable [different] therefore, as the recited “object embedding” being mapped to be based on Kalogeiton’s attributes/action label data, the object embedding would be understood to contain attributes [or data elements] that carry intra-class variability [these elements can be variable in the same class/category], the action being associated with an object hence, being data/information of an object, in other words, is analogous to same class of object’s action, there are elements that differentiate it from other object elements within the same class/same category); after receiving the output data (Kalogeiton teaches obtain an output data of the end-to-end multitask network architecture of Fig. 2 (the output of this network is analogous to the recited output data), wherein the output of the network of Fig. 2 includes an output of the object detection branch and an output of an action detection branch, the output of the object detection branch is an object detected in the image (therefore indicating that the object is whether detected in the image through the use of candidate bounding boxes, see section 3.1, 1st Par.) and the output of the action detection branch is the object’s detection of its associated action (is analogous to the recited object embedding, which as discussed above to have data elements for differentiating objects within a single object category). Importantly, the output of the network include object and its action, which, as described in section 1, 1st Par., which discloses “a detector to localize human actions in individual frames, and then either link them or track them over time to create spatio-temporal detections” indicating the output of the network is used to track the object over frames overtime. Importantly, the recited “automated action” in the claim merely indicate, based on BRI, an action that can be automatic/automated, therefore, if a network such as the network of Fig 2 produce output for a detector that can track an object through frames is an approach to track object automatically using neural network), determining whether to perform an automated action using the output data (FIG. 2 of the action detection is understood to indicate an action is determined for the object using the output data of the model of FIG. 2, the action detection is an automated process hence, by BRI, can be understood to be an automated action detection of the result is being an automated action determined, by BRI, covers the scope of the claim; moreover, section 1, 1st par., and FIG 1 and section 4.1, 3rd par., discloses the detected object in the image frames and the paired action detected is for tracking of the object over frames to be used for the system developed, therefore, it can be understood that the invention is used for tracking of object automatically when an object is detected with an action paired with it, by BRI, covers the scope of the claim).
However, Kalogeiton does not explicitly teach and is for generation of an object embedding for provision to another system that runs on other hardware; in response to determining to perform the automated action using the output data, providing, to the other system that runs on the other hardware, the object embedding for the target object to cause the other system to process the object embedding and perform an action using a result of the processing of the object embedding.
In the same field of action detection (abstract, Desai) Desai explicitly teaches and is for generation of an object embedding for provision to another system that runs on other hardware (column 24, 3rd par., discloses an aggregated image can be created by using a plurality of images that are merged together obtained from a plurality of cameras/imaging sensors, moreover, column 15, , lines 15-27, discloses that there is the use of multiple computers to access one or more functions associated with the facility, such as providing the processed result to a system administrator [to another system that runs on other hardware]); in response to determining to perform the automated action using the output data (as discussed above to Kalogeiton’s teaching, wherein Desai teaches object tracking being analogous to Kalogeiton’s tracking as the action in response to the output data of the network), providing, to the other system that runs on the other hardware, the object embedding for the target object to cause the other system to process the object embedding and perform an action using a result of the processing of the object embedding (column 3, 1st par., discloses when the result is processed and presented to the administrator system of the analyst [other system that runs on the other hardware], the visualization information here being the object embedding for the target object, and the administrator system analyze the visualization information. and determine actions to take intended to improve the operation of the facility [perform an action using a result of the processing of the object embedding] by changing the data processing parameters [to process the object embedding information]; Therefore, it would be obvious for one person of ordinary skill in the art at the time the invention was made to perform tracking of an object, wherein the tracking is performed based on output data indicating an object-action detection to be provided this detection result from one system to another to perform the tracking automatically. Thus in order to transmit data/information for sensors to perform location tracking of object more effectively, see Desai’s Col. 1, lines 13-35 and perform visualization processing correctly and efficiently, see abstract and column 3, 1st par., Desai).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing data of the claimed invention was made to combine the teachings of Kalogeiton of a system comprising one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising; providing, by the system and to a machine learning model, data that represents an image; receiving, from the machine learning model, output data that includes i) an object detection result that indicates whether a target object is detected in the image and ii) an object embedding for the target object, the object embedding containing data elements for differentiating objects within a single object category; after receiving the output data, determining whether to perform an automated action using the output data. Moreover, Kalogeiton’s output and automated action carrying out can be modified to be for generation of an object embedding for provision to another system that runs on other hardware; in response to determining to perform the automated action using the output data, providing, to the other system that runs on the other hardware, the object embedding for the target object to cause the other system to process the object embedding and perform an action using a result of the processing of the object embedding as taught in Desai.
Such a modification is the result of combing prior art elements. Kalogeiton and Desai share the same field of endeavor of object tracking. The motivation for the proposed modification would have been to have a system comprising one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising; providing, by the system and to a machine learning model, data that represents an image; receiving, from the machine learning model, output data that includes i) an object detection result that indicates whether a target object is detected in the image and ii) an object embedding for the target object, the object embedding containing data elements for differentiating objects within a single object category; after receiving the output data, determining whether to perform an automated action using the output data, generation of an object embedding for provision to another system that runs on other hardware; in response to determining to perform the automated action using the output data, providing, to the other system that runs on the other hardware, the object embedding for the target object to cause the other system to process the object embedding and perform an action using a result of the processing of the object embedding. Thus in order to transmit data/information for sensors to perform location tracking of object more effectively, see Desai’s Col. 1, lines 13-35 and perform visualization processing correctly and efficiently, see abstract and column 3, 1st par., Desai.
Regarding claim 2, Kalogeiton in view of Desai teaches the system of claim 1, wherein Kalogeiton explicitly teaches receiving the output data comprises receiving the output data from the machine learning model that comprises i) a visual recognition branch that generates the object detection result and ii) an embedding branch that generates the object embedding (FIG. 2 shows that the model has two branches an object detection branch and an action detection branch, the object detection branch can be understood to be analogous to the visual recognition branch that generate the object detection result, and the action detection branch can be understood to be the embedding branch as claimed, by BRI, which generate the action label which is understood to be the object embedding, by BRI).
Regarding claim 3, Kalogeiton in view of Desai teaches the system of claim 2, wherein Kalogeiton explicitly teaches receiving the output data comprises receiving the output data from the machine learning model (as discussed above in claim 2) that includes the embedding branch (the action detection branch as discussed above in claim 2) that includes a first proper subset of one or more training layers (FIG. 2 of the action detection branch include several layers for an end-to-end training process, as disclosed in FIG. 2, which, by BRI, is analogous to the recited first proper subset of one or more training layers as claimed, since any set of layers within a branch is a subset of layers to be proper [completed for the model] used for training to be training layers), the one or more training layers having included a) the first proper subset (any portion of the layers of the action detection branch can be understood to be the first proper subset as claimed, by BRI) and b) a second proper subset (and the remaining portion is the second proper subset, by BRI) that was not included in the machine learning model for inference (section 4.2, last paragraph, discloses zero shot learning table 5 shows that the network is able to infer information about actions that were not seen at training time for a given object, therefore, in this instance, the action detection branch would have layers that were not learnt these new information in other words, was not included in the machine learning model for inferring such new information, by BRI, covers the scope of the claim, and the default layers that have been learnt the information during the training can be understood to be the first proper subset as claimed, by BRI).
Regarding claim 4, Kalogeiton in view of Desai teaches the system of claim 2, wherein Kalogeiton explicitly teaches receiving the output data comprises receiving the output data from the machine learning model that includes one or more shared initial layers that generate data used by both the visual recognition branch and the embedding branch (FIG. 2 shows that the middle portion provide information to be used by both the action and object detection branches, hence, can be understood to be analogous to the one or more shared initial layers as claimed, by BRI).
Regarding claim 5, Kalogeiton in view of Desai teaches the system of claim 4, wherein Kalogeiton explicitly teaches receiving the output data comprises receiving the output data from the machine learning model (as discussed above in claim 1) that was trained using i) a first loss value for the one or more shared initial layers and the visual recognition branch and ii) a second loss value for the one or more shared initial layers and the embedding branch (equation 2 of section 3.2, shows that a multi-task loss is computed for the training of the model, per branch, each branch is calculated the loss for the training, as shown in the equation 2, therefore, is analogous to the claimed limitation wherein a second loss is a value for the initial layer and the embedding branch and the first loss is for the initial layers and the object detection or visual recognition branch, by BRI, covers the scope of the claimed limitation).
Regarding claim 6, Kalogeiton in view of Desai teaches the system of claim 1, wherein Kalogeiton explicitly teaches receiving the output data comprises receiving the output data (as discussed above in claim 1) that includes the object embedding for the target object that was extracted from an image object embedding for the image (the action detection branch, as discussed above in claim 1, in FIG. 2 to determine the action label [object embedding] from the image for the object extracted from the image) using location data that indicates a location of the target object detected in the image (using bounding box data [according to section 2., 2nd to the last par.] which is the location data indicates the likely location of the object detected in the image, by BRI, covers the scope of the claimed limitation).
Regarding claim 7, Kalogeiton in view of Desai teaches the system of claim 1, wherein Kalogeiton explicitly teaches receiving, from the machine learning model, the output data (as discussed above in claim 1) that includes i) an object detection result that indicates whether a target object is detected in the image (FIG. 2 shows that one branch is for object detection which indicates whether an object is detected in the image, by BRI, covers the scope of the limitation) and ii) an object embedding for the target object (another branch of FIG. 2 shows action detection which include action label [by BRI, can be understood to be an object embedding for the target object as claimed]) comprises: receiving, from the machine learning model, the output data that includes i) an object detection result that indicates that a target object is detected in the image (FIG. 2 shows the output of the object detection branch is the data indicates an object is detected in the image) and location data that indicates a location of the target object detected in the image (using bounding box data [according to section 2., 2nd to the last par.] which is the location data indicates the likely location of the object detected in the image, by BRI, covers the scope of the claimed limitation), and ii) an object embedding for the target object (another branch of FIG. 2 shows action detection which include action label [by BRI, can be understood to be an object embedding for the target object as claimed]).
Regarding claim 8, Kalogeiton in view of Desai teaches the system of claim 7, wherein Kalogeiton explicitly teaches the location data comprises a bounding box for the detected target object (another branch of FIG. 2 shows action detection which include action label [by BRI, can be understood to be an object embedding for the target object as claimed]).
Regarding claim 9, Kalogeiton in view of Desai teaches the system of claim 1, wherein Kalogeiton explicitly teaches receiving, from the machine learning model, output data that includes an object detection result that indicates whether a target object is detected in the image comprises (as discussed above in claim 1): receiving output data that includes, for the object detection result, an object category (the output of the object detection branch of FIG. 2 is the object label such as shown and disclosed in FIG. 4 which, by BRI, is analogous to the object category as claimed); and receiving, for the object detection result, a likelihood that the detected target object belongs to the object category (based on a probability calculation that the box to be the object-action instance as disclosed in section 3.2, “Multitask” section, by BRI, the probability is analogous to the likelihood as claimed that the detected target belongs to the object category, by BRI).
Regarding claim 12, Kalogeiton explicitly teaches a computer-implemented method comprising (abstract discloses the use of machine learning which indicates the use of a computer to have computer components such as storage storing instructions to be executed by a processor for the operations of the invention; FIG. 2 shows that the input into the machine learning model being an image and being maintained for the whole processing of the image data, by BRI [broadest reasonable interpretation] cover the scope of the limitation); providing, by a system and to a machine learning model, data that represents an image (FIG. 2 shows the image is being input into the machine learning model of its data; hence being part of the system of the invention); receiving, from the machine learning model, output data that includes (FIG. 2 shows that the output from the model includes two branches) i) an object detection result that indicates whether a target object is detected in the image (FIG. 2 shows that one branch is for object detection which indicates whether an object is detected in the image, by BRI, covers the scope of the limitation) and ii) an object embedding for the target object (another branch of FIG. 2 shows action detection which include action label [by BRI, can be understood to be an object embedding for the target object as claimed]); and determining whether to perform an automated action using the output data (FIG. 2 of the action detection is understood to indicate an action is determined for the object using the output data of the model of FIG. 2, the action detection is an automated process hence, by BRI, can be understood to be an automated action detection of the result is being an automated action determined, by BRI, covers the scope of the claim; moreover, section 1, 1st par., and FIG 1 and section 4.1, 3rd par., discloses the detected object in the image frames and the paired action detected is for tracking of the object over frames to be used for the system developed, therefore, it can be understood that the invention is used for tracking of object automatically when an object is detected with an action paired with it, by BRI, covers the scope of the claim) that includes (FIG. 2 shows that the output from the model includes two branches) i) an object detection result that indicates whether a target object is detected in the image (FIG. 2 shows that one branch is for object detection which indicates whether an object is detected in the image, by BRI, covers the scope of the limitation) and ii) an object embedding for the target object (another branch of FIG. 2 shows action detection which include action label [by BRI, can be understood to be an object embedding for the target object as claimed]) , the object embedding containing data elements for differentiating objects within a single object category (Kalogeiton’s section 2, last paragraph, presented in page 4165, which suggests and/or teaches that existing approaches have already define that “categories rely on attributes. Attributes have been used for human actions…each action class has an intra-class variability…attributes are relevant for each class” which indicates that actions are categorized into classes and each class has intra-class variability, indicating that the class attributes within the same class can be variable [different] therefore, as the recited “object embedding” being mapped to be based on Kalogeiton’s attributes/action label data, the object embedding would be understood to contain attributes [or data elements] that carry intra-class variability [these elements can be variable in the same class/category], the action being associated with an object hence, being data/information of an object, in other words, is analogous to same class of object’s action, there are elements that differentiate it from other object elements within the same class/same category); after receiving the output data (Kalogeiton teaches obtain an output data of the end-to-end multitask network architecture of Fig. 2 (the output of this network is analogous to the recited output data), wherein the output of the network of Fig. 2 includes an output of the object detection branch and an output of an action detection branch, the output of the object detection branch is an object detected in the image (therefore indicating that the object is whether detected in the image through the use of candidate bounding boxes, see section 3.1, 1st Par.) and the output of the action detection branch is the object’s detection of its associated action (is analogous to the recited object embedding, which as discussed above to have data elements for differentiating objects within a single object category). Importantly, the output of the network include object and its action, which, as described in section 1, 1st Par., which discloses “a detector to localize human actions in individual frames, and then either link them or track them over time to create spatio-temporal detections” indicating the output of the network is used to track the object over frames overtime. Importantly, the recited “automated action” in the claim merely indicate, based on BRI, an action that can be automatic/automated, therefore, if a network such as the network of Fig 2 produce output for a detector that can track an object through frames is an approach to track object automatically using neural network), determining whether to perform an automated action using the output data (FIG. 2 of the action detection is understood to indicate an action is determined for the object using the output data of the model of FIG. 2, the action detection is an automated process hence, by BRI, can be understood to be an automated action detection of the result is being an automated action determined, by BRI, covers the scope of the claim; moreover, section 1, 1st par., and FIG 1 and section 4.1, 3rd par., discloses the detected object in the image frames and the paired action detected is for tracking of the object over frames to be used for the system developed, therefore, it can be understood that the invention is used for tracking of object automatically when an object is detected with an action paired with it, by BRI, covers the scope of the claim).
However, Kalogeiton does not explicitly teach and is for generation of an object embedding for provision to another system that runs on other hardware; in response to determining to perform the automated action using the output data, providing, to the other system that runs on the other hardware, the object embedding for the target object to cause the other system to process the object embedding and perform an action using a result of the processing of the object embedding.
In the same field of action detection (abstract, Desai) Desai explicitly teaches and is for generation of an object embedding for provision to another system that runs on other hardware (column 24, 3rd par., discloses an aggregated image can be created by using a plurality of images that are merged together obtained from a plurality of cameras/imaging sensors, moreover, column 15, , lines 15-27, discloses that there is the use of multiple computers to access one or more functions associated with the facility, such as providing the processed result to a system administrator [to another system that runs on other hardware]); in response to determining to perform the automated action using the output data (as discussed above to Kalogeiton’s teaching, wherein Desai teaches object tracking being analogous to Kalogeiton’s tracking as the action in response to the output data of the network), providing, to the other system that runs on the other hardware, the object embedding for the target object to cause the other system to process the object embedding and perform an action using a result of the processing of the object embedding (column 3, 1st par., discloses when the result is processed and presented to the administrator system of the analyst [other system that runs on the other hardware], the visualization information here being the object embedding for the target object, and the administrator system analyze the visualization information. and determine actions to take intended to improve the operation of the facility [perform an action using a result of the processing of the object embedding] by changing the data processing parameters [to process the object embedding information]; Therefore, it would be obvious for one person of ordinary skill in the art at the time the invention was made to perform tracking of an object, wherein the tracking is performed based on output data indicating an object-action detection to be provided this detection result from one system to another to perform the tracking automatically. Thus in order to transmit data/information for sensors to perform location tracking of object more effectively, see Desai’s Col. 1, lines 13-35 and perform visualization processing correctly and efficiently, see abstract and column 3, 1st par., Desai).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing data of the claimed invention was made to combine the teachings of Kalogeiton of a method of providing, by the system and to a machine learning model, data that represents an image; receiving, from the machine learning model, output data that includes i) an object detection result that indicates whether a target object is detected in the image and ii) an object embedding for the target object, the object embedding containing data elements for differentiating objects within a single object category; after receiving the output data, determining whether to perform an automated action using the output data. Moreover, Kalogeiton’s output and automated action carrying out can be modified to be for generation of an object embedding for provision to another system that runs on other hardware; in response to determining to perform the automated action using the output data, providing, to the other system that runs on the other hardware, the object embedding for the target object to cause the other system to process the object embedding and perform an action using a result of the processing of the object embedding as taught in Desai.
Such a modification is the result of combing prior art elements. Kalogeiton and Desai share the same field of endeavor object tracking. The motivation for the proposed modification would have been to have a method of providing, by the system and to a machine learning model, data that represents an image; receiving, from the machine learning model, output data that includes i) an object detection result that indicates whether a target object is detected in the image and ii) an object embedding for the target object, the object embedding containing data elements for differentiating objects within a single object category; after receiving the output data, determining whether to perform an automated action using the output data, generation of an object embedding for provision to another system that runs on other hardware; in response to determining to perform the automated action using the output data, providing, to the other system that runs on the other hardware, the object embedding for the target object to cause the other system to process the object embedding and perform an action using a result of the processing of the object embedding. Thus in order to transmit data/information for sensors to perform location tracking of object more effectively, see Desai’s Col. 1, lines 13-35 and perform visualization processing correctly and efficiently, see abstract and column 3, 1st par., Desai.
Regarding claim 13, Kalogeiton in view of Desai teaches the method of claim 12, wherein Kalogeiton explicitly teaches receiving the output data comprises receiving the output data from the machine learning model that comprises i) a visual recognition branch that generates the object detection result and ii) an embedding branch that generates the object embedding (FIG. 2 shows that the model has two branches an object detection branch and an action detection branch, the object detection branch can be understood to be analogous to the visual recognition branch that generate the object detection result, and the action detection branch can be understood to be the embedding branch as claimed, by BRI, which generate the action label which is understood to be the object embedding, by BRI).
Regarding claim 14, Kalogeiton in view of Desai teaches the method of claim 13, wherein Kalogeiton explicitly teaches receiving the output data comprises receiving the output data from the machine learning model (as discussed above in claim 13) that includes the embedding branch (the action detection branch as discussed above in claim 2) that includes a first proper subset of one or more training layers (FIG. 2 of the action detection branch include several layers for an end-to-end training process, as disclosed in FIG. 2, which, by BRI, is analogous to the recited first proper subset of one or more training layers as claimed, since any set of layers within a branch is a subset of layers to be proper [completed for the model] used for training to be training layers), the one or more training layers having included a) the first proper subset (any portion of the layers of the action detection branch can be understood to be the first proper subset as claimed, by BRI) and b) a second proper subset (and the remaining portion is the second proper subset, by BRI) that was not included in the machine learning model for inference (section 4.2, last paragraph, discloses zero shot learning table 5 shows that the network is able to infer information about actions that were not seen at training time for a given object, therefore, in this instance, the action detection branch would have layers that were not learnt these new information in other words, was not included in the machine learning model for inferring such new information, by BRI, covers the scope of the claim, and the default layers that have been learnt the information during the training can be understood to be the first proper subset as claimed, by BRI).
Regarding claim 15, Kalogeiton in view of Desai teaches the method of claim 13, wherein Kalogeiton explicitly teaches receiving the output data comprises receiving the output data from the machine learning model that includes one or more shared initial layers that generate data used by both the visual recognition branch and the embedding branch (FIG. 2 shows that the middle portion provide information to be used by both the action and object detection branches, hence, can be understood to be analogous to the one or more shared initial layers as claimed, by BRI).
Regarding claim 16, Kalogeiton in view of Desai teaches the method of claim 14, wherein Kalogeiton explicitly teaches receiving the output data comprises receiving the output data from the machine learning model (as discussed above in claim 14) that was trained using i) a first loss value for the one or more shared initial layers and the visual recognition branch and ii) a second loss value for the one or more shared initial layers and the embedding branch (equation 2 of section 3.2, shows that a multi-task loss is computed for the training of the model, per branch, each branch is calculated the loss for the training, as shown in the equation 2, therefore, is analogous to the claimed limitation wherein a second loss is a value for the initial layer and the embedding branch and the first loss is for the initial layers and the object detection or visual recognition branch, by BRI, covers the scope of the claimed limitation).
Regarding claim 17, Kalogeiton in view of Desai teaches the method of claim 12, wherein Kalogeiton explicitly teaches receiving the output data comprises receiving the output data (as discussed above in claim 12) that includes the object embedding for the target object that was extracted from an image object embedding for the image (the action detection branch, as discussed above in claim 12, in FIG. 2 to determine the action label [object embedding] from the image for the object extracted from the image) using location data that indicates a location of the target object detected in the image (using bounding box data [according to section 2., 2nd to the last par.] which is the location data indicates the likely location of the object detected in the image, by BRI, covers the scope of the claimed limitation).
Regarding claim 18, Kalogeiton in view of Desai teaches the method of claim 12, wherein Kalogeiton explicitly teaches receiving, from the machine learning model, the output data (as discussed above in claim 12) that includes i) an object detection result that indicates whether a target object is detected in the image (FIG. 2 shows that one branch is for object detection which indicates whether an object is detected in the image, by BRI, covers the scope of the limitation) and ii) an object embedding for the target object (another branch of FIG. 2 shows action detection which include action label [by BRI, can be understood to be an object embedding for the target object as claimed]) comprises: receiving, from the machine learning model, the output data that includes i) an object detection result that indicates that a target object is detected in the image (FIG. 2 shows the output of the object detection branch is the data indicates an object is detected in the image) and location data that indicates a location of the target object detected in the image (using bounding box data [according to section 2., 2nd to the last par.] which is the location data indicates the likely location of the object detected in the image, by BRI, covers the scope of the claimed limitation), and ii) an object embedding for the target object (another branch of FIG. 2 shows action detection which include action label [by BRI, can be understood to be an object embedding for the target object as claimed]).
Regarding claim 19, Kalogeiton in view of Desai teaches the method of claim 18, wherein Kalogeiton explicitly teaches the location data comprises a bounding box for the detected target object (another branch of FIG. 2 shows action detection which include action label [by BRI, can be understood to be an object embedding for the target object as claimed]).
Regarding claim 20, Kalogeiton discloses one or more non-transitory computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising (abstract discloses the use of machine learning which indicates the use of a computer to have computer components such as non-transitory storage storing instructions to be executed by a processor for the operations of the invention; FIG. 2 shows that the input into the machine learning model being an image and being maintained for the whole processing of the image data, by BRI [broadest reasonable interpretation] cover the scope of the limitation); providing, by a system and to a machine learning model, data that represents an image (FIG. 2 shows the image is being input into the machine learning model of its data, hence being part of the system of the invention); receiving, from the machine learning model, output data that includes (FIG. 2 shows that the output from the model includes two branches) i) an object detection result that indicates whether a target object is detected in the image (FIG. 2 shows that one branch is for object detection which indicates whether an object is detected in the image, by BRI, covers the scope of the limitation) and ii) an object embedding for the target object (another branch of FIG. 2 shows action detection which include action label [by BRI, can be understood to be an object embedding for the target object as claimed]); and determining whether to perform an automated action using the output data (FIG. 2 of the action detection is understood to indicate an action is determined for the object using the output data of the model of FIG. 2, the action detection is an automated process hence, by BRI, can be understood to be an automated action detection of the result is being an automated action determined, by BRI, covers the scope of the claim; moreover, section 1, 1st par., and FIG 1 and section 4.1, 3rd par., discloses the detected object in the image frames and the paired action detected is for tracking of the object over frames to be used for the system developed, therefore, it can be understood that the invention is used for tracking of object automatically when an object is detected with an action paired with it, by BRI, covers the scope of the claim) that includes (FIG. 2 shows that the output from the model includes two branches) i) an object detection result that indicates whether a target object is detected in the image (FIG. 2 shows that one branch is for object detection which indicates whether an object is detected in the image, by BRI, covers the scope of the limitation) and ii) an object embedding for the target object (another branch of FIG. 2 shows action detection which include action label [by BRI, can be understood to be an object embedding for the target object as claimed]) , the object embedding containing data elements for differentiating objects within a single object category (Kalogeiton’s section 2, last paragraph, presented in page 4165, which suggests and/or teaches that existing approaches have already define that “categories rely on attributes. Attributes have been used for human actions…each action class has an intra-class variability…attributes are relevant for each class” which indicates that actions are categorized into classes and each class has intra-class variability, indicating that the class attributes within the same class can be variable [different] therefore, as the recited “object embedding” being mapped to be based on Kalogeiton’s attributes/action label data, the object embedding would be understood to contain attributes [or data elements] that carry intra-class variability [these elements can be variable in the same class/category], the action being associated with an object hence, being data/information of an object, in other words, is analogous to same class of object’s action, there are elements that differentiate it from other object elements within the same class/same category); after receiving the output data (Kalogeiton teaches obtain an output data of the end-to-end multitask network architecture of Fig. 2 (the output of this network is analogous to the recited output data), wherein the output of the network of Fig. 2 includes an output of the object detection branch and an output of an action detection branch, the output of the object detection branch is an object detected in the image (therefore indicating that the object is whether detected in the image through the use of candidate bounding boxes, see section 3.1, 1st Par.) and the output of the action detection branch is the object’s detection of its associated action (is analogous to the recited object embedding, which as discussed above to have data elements for differentiating objects within a single object category). Importantly, the output of the network include object and its action, which, as described in section 1, 1st Par., which discloses “a detector to localize human actions in individual frames, and then either link them or track them over time to create spatio-temporal detections” indicating the output of the network is used to track the object over frames overtime. Importantly, the recited “automated action” in the claim merely indicate, based on BRI, an action that can be automatic/automated, therefore, if a network such as the network of Fig 2 produce output for a detector that can track an object through frames is an approach to track object automatically using neural network), determining whether to perform an automated action using the output data (FIG. 2 of the action detection is understood to indicate an action is determined for the object using the output data of the model of FIG. 2, the action detection is an automated process hence, by BRI, can be understood to be an automated action detection of the result is being an automated action determined, by BRI, covers the scope of the claim; moreover, section 1, 1st par., and FIG 1 and section 4.1, 3rd par., discloses the detected object in the image frames and the paired action detected is for tracking of the object over frames to be used for the system developed, therefore, it can be understood that the invention is used for tracking of object automatically when an object is detected with an action paired with it, by BRI, covers the scope of the claim).
However, Kalogeiton does not explicitly teach and is for generation of an object embedding for provision to another system that runs on other hardware; in response to determining to perform the automated action using the output data, providing, to the other system that runs on the other hardware, the object embedding for the target object to cause the other system to process the object embedding and perform an action using a result of the processing of the object embedding.
In the same field of action detection (abstract, Desai) Desai explicitly teaches and is for generation of an object embedding for provision to another system that runs on other hardware (column 24, 3rd par., discloses an aggregated image can be created by using a plurality of images that are merged together obtained from a plurality of cameras/imaging sensors, moreover, column 15, , lines 15-27, discloses that there is the use of multiple computers to access one or more functions associated with the facility, such as providing the processed result to a system administrator [to another system that runs on other hardware]); in response to determining to perform the automated action using the output data (as discussed above to Kalogeiton’s teaching, wherein Desai teaches object tracking being analogous to Kalogeiton’s tracking as the action in response to the output data of the network), providing, to the other system that runs on the other hardware, the object embedding for the target object to cause the other system to process the object embedding and perform an action using a result of the processing of the object embedding (column 3, 1st par., discloses when the result is processed and presented to the administrator system of the analyst [other system that runs on the other hardware], the visualization information here being the object embedding for the target object, and the administrator system analyze the visualization information. and determine actions to take intended to improve the operation of the facility [perform an action using a result of the processing of the object embedding] by changing the data processing parameters [to process the object embedding information]; Therefore, it would be obvious for one person of ordinary skill in the art at the time the invention was made to perform tracking of an object, wherein the tracking is performed based on output data indicating an object-action detection to be provided this detection result from one system to another to perform the tracking automatically. Thus in order to transmit data/information for sensors to perform location tracking of object more effectively, see Desai’s Col. 1, lines 13-35 and perform visualization processing correctly and efficiently, see abstract and column 3, 1st par., Desai).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing data of the claimed invention was made to combine the teachings of Kalogeiton of a storage media of providing, by the system and to a machine learning model, data that represents an image; receiving, from the machine learning model, output data that includes i) an object detection result that indicates whether a target object is detected in the image and ii) an object embedding for the target object, the object embedding containing data elements for differentiating objects within a single object category; after receiving the output data, determining whether to perform an automated action using the output data. Moreover, Kalogeiton’s output and automated action carrying out can be modified to be for generation of an object embedding for provision to another system that runs on other hardware; in response to determining to perform the automated action using the output data, providing, to the other system that runs on the other hardware, the object embedding for the target object to cause the other system to process the object embedding and perform an action using a result of the processing of the object embedding as taught in Desai.
Such a modification is the result of combing prior art elements. Kalogeiton and Desai share the same field of endeavor object tracking. The motivation for the proposed modification would have been to have a storage media of providing, by the system and to a machine learning model, data that represents an image; receiving, from the machine learning model, output data that includes i) an object detection result that indicates whether a target object is detected in the image and ii) an object embedding for the target object, the object embedding containing data elements for differentiating objects within a single object category; after receiving the output data, determining whether to perform an automated action using the output data, generation of an object embedding for provision to another system that runs on other hardware; in response to determining to perform the automated action using the output data, providing, to the other system that runs on the other hardware, the object embedding for the target object to cause the other system to process the object embedding and perform an action using a result of the processing of the object embedding. Thus in order to transmit data/information for sensors to perform location tracking of object more effectively, see Desai’s Col. 1, lines 13-35 and perform visualization processing correctly and efficiently, see abstract and column 3, 1st par., Desai.
Regarding claim 21, Kalogeiton in view of Desai teaches the computer storage media of claim 20, wherein Kalogeiton explicitly teaches receiving, from the machine learning model, the output data that includes (FIG. 2 shows that the output from the model includes two branches) i) the object detection result that indicates whether the target object is detected in the image (FIG. 2 shows that one branch is for object detection which indicates whether an object is detected in the image, by BRI, covers the scope of the limitation) and ii) the object embedding for the target object (another branch of FIG. 2 shows action detection which include action label [by BRI, can be understood to be an object embedding for the target object as claimed]) comprises: receiving, from the machine learning model (FIG. 2 shows that the output from the model includes two branches being a neural network), the output data that includes i) the object detection result that indicates that the target object is detected in the image (the output of the object detection branch is an object detected in the image (therefore indicating that the object is whether detected in the image through the use of candidate bounding boxes, see section 3.1, 1st Par.) and the output of the action detection branch is the object’s detection of its associated action) and location data that indicates a location of the target object detected in the image (the bounding boxes indicate location of the target object in the image), and ii) the object embedding for the target object (another branch of FIG. 2 shows action detection which include action label [by BRI, can be understood to be an object embedding for the target object as claimed]).
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Vicky Kalogeiton et. al. (“Joint Learning of Object and Action Detectors, 2017, Proceedings of the IEEE International Conference on Computer Vision, pp. 4163-4172” hereinafter as “Kalogeiton”) in view of Nishitkumar Ashokkumar Desai et. al. (“US 11,263,795 B1” hereinafter as “Desai”) and further in view of Philippe Weinzaepfel et. al. (“Learning to track for spatio-temporal action localization, 2015, Proceedings of the IEEE International Conference on Computer Vision, pp. 3164-3172” hereinafter as “Weinzaepfel”).
Regarding claim 11, Kalogeiton in view of Desai teaches the system of claim 1 wherein Kalogeiton explicitly teaches determining whether to perform an automated action using the output data (as discussed above in claim 1) comprising: providing the i) object detection result that indicates whether a target object is detected in the image (FIG. 2 shows that one branch is for object detection which indicates whether an object is detected in the image, by BRI, covers the scope of the limitation) and ii) object embedding for the target object (another branch of FIG. 2 shows action detection which include action label [by BRI, can be understood to be an object embedding for the target object as claimed]).
However, Kalogeiton in view of Desai does not explicitly teaches providing, to an object matching engine, the data and receiving, from the object matching engine, data that includes an object matching result indicating whether the detected target object is the same as another object detected in another image from a sequence of images that includes the image as part of an object tracking process.
In the same field of object action localization (title, Weinzaepfel), Weinzaepfel explicitly teaches providing, to an object matching engine, the data (section 3, 3rd par. of “tracking best candidates” section, discloses providing the extracted regions to the processing for finding best candidates [analogous to providing to an object matching engine as claimed, since this processing is performed by a processor]) and receiving, from the object matching engine, data that includes an object matching result indicating whether the detected target object is the same as another object detected in another image from a sequence of images (section 3, “Tracking best candidates” and “Scoring tracks” discloses performing best candidates for the tracking based on scoring to determine the action and the object match through the images of the tracking, hence, based on, BRI, is analogous to the claimed limitation wherein the scoring is analogous to object matching result indicating the likelihood that the same object is being tracked among images, by BRI covers the scope of the claimed limitation) that includes the image as part of an object tracking process (for the object tracking as discussed previously; Therefore, it would be obvious for one person of ordinary skill in the art at the time of the invention was made to perform a method of system that have object-action detection, wherein the object detection include a matching process to match the detected object to a target object for tracking processing. Thus in order to detect object and perform tracking of the object based on scoring or matching of the tracking objects more robustly (abstract, Weinzaepfel).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing data of the claimed invention was made to combine the teachings of Kalogeiton in view of Desai of a system of determining whether to perform an automated action using the output data comprising: providing the i) object detection result that indicates whether a target object is detected in the image and ii) object embedding for the target object. Moreover, Kalogeiton’s system can be modified to be perform providing, to an object matching engine, the data and receiving, from the object matching engine, data that includes an object matching result indicating whether the detected target object is the same as another object detected in another image from a sequence of images that includes the image as part of an object tracking process as taught in Weinzaepfel.
Such a modification is the result of combing prior art elements. Kalogeiton, Desai and Weinzaepfel share the same field of endeavor of object tracking. The motivation for the proposed modification would have been to have a system of determining whether to perform the automated action using the output data comprises :providing, to an object matching engine, i) the object detection result that indicates whether a target object is detected in the image and ii) the object embedding for the target object; and receiving, from the object matching engine, data that includes an object matching result indicating whether the detected target object is likely the same as another object detected in another image from a sequence of images that includes the image as part of an object tracking process. Thus in order to detect object and perform tracking of the object based on scoring or matching of the tracking objects more robustly (abstract, Weinzaepfel).
Pertinent Prior Art(s)
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Hutz; David James et. al., “US 20200404190 A1”, discloses preserving privacy in surveillance. The methods, systems, and apparatus include actions of obtaining images of a scene captured by a camera, identifying an object in the images through object recognition, determining that the object that is identified in the images is of a particular type that has a privacy restriction, and in response to determining that the object in the images is of the particular type that has the privacy restriction, obfuscating an appearance of the object in the images.
Richardson; James, “US 20150325002 A1”, discloses a motion capture system includes motion capture cameras positioned in various locations and orientations with respect to a motion capture volume. The motion capture system includes a host computing device that is operatively coupled with the motion capture cameras. The host computing device remotely controls operation of the motion capture cameras to record movement within the motion capture volume. At least one of the motion capture cameras includes a user-interface that is operable by a user to remotely initiate a control operation of the host computing device.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PHUONG HAU CAI whose telephone number is (571)272-9424. The examiner can normally be reached M-F 8:30 am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PHUONG HAU CAI/Examiner, Art Unit 2673
/CHINEYERE WILLS-BURNS/Supervisory Patent Examiner, Art Unit 2673
Prosecution Insights