DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Reissue Applications
This application seeks to reissue US Patent No. 11,755,984 (“the ‘984 patent”). In response to the 01/02/2026 non-final Office action (NFOA), the applicant amended claims 1, 2, 4, 6, 7, 9, 10, 12, 13, 28, and 33. Claims 1-39 are pending.
For reissue applications filed before September 16, 2012, all references to 35 U.S.C. 251 and 37 CFR 1.172, 1.175, and 3.73 are to the law and rules in effect on September 15, 2012. Where specifically designated, these are “pre-AIA ” provisions.
For reissue applications filed on or after September 16, 2012, all references to 35 U.S.C. 251 and 37 CFR 1.172, 1.175, and 3.73 are to the current provisions.
Applicant is reminded of the continuing obligation under 37 CFR 1.178(b), to timely apprise the Office of any prior or concurrent proceeding in which Patent No. 11,755,984 is or was involved. These proceedings would include any trial before the Patent Trial and Appeal Board, interferences, reissues, reexaminations, supplemental examinations, and litigation.
Applicant is further reminded of the continuing obligation under 37 CFR 1.56, to timely apprise the Office of any information which is material to patentability of the claims under consideration in this reissue application.
These obligations rest with each individual associated with the filing and prosecution of this application for reissue. See also MPEP §§ 1404, 1442.01 and 1442.04.
Applicant’s Response
Claims 13-39 were rejected in the NFOA under §251 for violating the recapture rule. NFOA at 5-7. The applicant submits that the recent amendment of claims 13 and 28 has materially narrowed the claims relative to the surrendered subject matter. Remarks at 19. This is persuasive. These claims omit the surrendered limitations described in the NFOA at 5-7. However, amended claims 13 and 28 now recite that the second and third probability scores are determined based on the first and second probability scores, respectively. This was not taught in the prior art that was identified in the original application, thus the claims have been materially narrowed relative to the original claims prior to the addition of the surrender generating limitations. See MPEP 1412.02(II)(C). This rejection under §251 is withdrawn.
Claims 1-12 were rejected under §§ 112(a) and 251 for new matter. NFOA at 7-9. The claims have been amended to remove the limitations that were the focus of these rejections. These rejections are withdrawn. However, these limitations were surrendered during prosecution of US 16/933,016, and their removal violates the recapture rule. See the §251 rejection below.
Claims 1-12 were rejected under §112(b) for various reasons. See NFOA at 10-12. The recent amendment overcomes certain of these issues, but introduces new uncertainty in claim scope. See the rejections below.
Claims 1 and 12 were rejected under §103 based on a combination of JP538, JP129, and Chan. NFOA at 12-17. The applicant traverses this rejection. Remarks at 21-23. These arguments are not persuasive.
Prior to the introduction of the Chan reference, the rejection of claim 1 proposed a combination of JP538 and JP129. NFOA at 12-16. The combination of these two references results in a drone that captures an image of a person and determines if the image identifies the person with sufficient confidence. Id. If a confidence threshold is not reached, the drone moves to a new location and captures another image. Id. These iterations continue until an image is captured having a confidence that exceeds the threshold. Id. This JP538-JP129 combination does not arrive at the claimed invention because it does not teach or suggest aggregating confidence scores across iterations. Id.
Chan describes capturing multiple images of a person from different angles, and generating a confidence score indicating that the images identify an individual. Id. at 16 and Chan ¶¶ 28-34. Rather than relying on a single confidence score for each image, Chan shows that confidence scores associated with the individual images can be aggregated. Id. and Chan ¶¶ 28-34. In particular, Chan describes a scenario in which two low-confidence images of a person are captured, but none of these two “sub-optimal images will generate a high enough matching confidence level of matching. However, when user identification program 106 considers both sub-optimal images together, the confidence level determined is higher.” Chan ¶34. The NFOA stated that this disclosure in Chan would have motivated the POSITA to modify the JP538-JP129 combination to aggregate its individual probability scores, rather than continue to iterate until a single probability score exceeds a threshold, because this would enable the drone to identify the person in fewer iterations. NFOA at 16-17.
The applicant submits that “Chan does not describe an iterative process” but instead captures, and calculates probability based on, a “fixed set of images.” Remarks at 22. The applicant submits that any combination with JP129 would result in the combined confidence score being calculated “after all the iterations” of JP129 are completed. Id. This is not persuasive.
The test for obviousness is not limited to what Chan teaches; it is what Chan, together with the JP538-JP129 combination, would have suggested to the POSITA. MPEP 2145 IV. While Chan does not describe an iterative process, the JP538-JP129 combination does, as described above. The disclosure in Chan that plural confidence scores can be aggregated would have shown to the person of ordinary skill that the JP538-JP129 combination could be improved by aggregating its probability scores across iterations, for reasons given above and in the NFOA at 16-17.
The POSITA would have understood that to achieve this advantage, an aggregation of confidence scores at each iteration would be necessary. Only in this manner would the drone be able to identify its target in fewer iterations. See id. This is clearly suggested in Chan ¶¶ 34 and 37, which shows that aggregating two images with low confidence scores would result in a positive identification. In the same scenario, an unmodified JP538-JP129 system would have required further iterations. Based on teachings found in Chan, the POSITA would have realized that these further iterations would not actually be necessary. This would naturally have suggested a modification to aggregate the confidence scores, which would enable the drone to more quickly identify the person.
"A person of ordinary skill in the art is also a person of ordinary creativity, not an automaton." KSR Int’l Co. v. Teleflex Inc., 550 U.S. 398, 421 (2007). A determination of obviousness must take into account “the inferences and creative steps that a person of ordinary skill in the art would employ." Id. at 418. Based on the guidance in KSR, the examiner maintains that a person of ordinary skill and creativity would not have concluded that capturing a “fixed set of images” would be necessary when combining JP538-JP129 with Chan. While an automaton may have combined the references in this manner, a person possessing common sense and ordinary creativity would not. The POSITA would recognize that capturing a “fixed set of images” would likely negate any advantage that would be achieved by aggregating the scores, as this would have required the drone to always perform multiple iterations. In the JP538-JP129 combination, it remains possible that a single iteration alone may be sufficient to identify the person. The POSITA would understand that a better solution would be to perform only as many iterations as necessary, by taking advantage of the statistical fact that multiple low-confidence images may be sufficient to identify the person. See Chan ¶34. Therefore the examiner disagrees that a person of skill in the art would have found reason to combine the references in the manner described by the applicant, or would have been constrained to doing so.
It is maintained, for reasons given above, that the JP538, JP129, and Chan reference would have suggested the invention that is recited in these claims. The rejections under §103 are maintained.
The applicant’s arguments regarding the §103 rejection of amended claims 13-39 are moot in view of the new grounds of rejection below.
Claim Interpretation – Contingent Language
Method claims 1-11 and 13-27 recite steps do not limit the scope of the claims, thus are not given patentable weight. The broadest reasonable interpretation (BRI) of a method claim includes only steps that must be performed, and does not include steps that are only performed when a preceding condition is met. MPEP 2111.04 II. These claims recite steps that are only performed when certain non-required conditions are encountered.
Claim 1 recites “initiating another iteration in case the aggregated probability score does not exceed a certain threshold.” Claim 1 does not require that the probability score does not exceed the threshold, and therefore the claim scope includes the scenario in which the probability score exceeds the threshold. Accordingly, it is within the scope of the claim that the “another iteration” is never initiated. Because this language describes a step that is only performed when a non-required condition is met, this step is not included in the claim’s BRI. Claims 2-11 depend on claim 1 and inherit this language and interpretation. See also the §112(b) rejection of these claims below.
Claim 13 recites “determining that the first probability score is no greater than a threshold.” This positive determination that the score does not exceed the threshold ensures that the command to capture a second image is transmitted by the remote server. The claim later recites “in response to determining that the second probability score is no greater than the threshold,” a third image is captured and is scored. In light of the previous language describing “determining that” the first probability score is no greater than the threshold, this later language is understood to describe what occurs if the second probability score is not greater than the threshold. The claim therefore does not require the second probability score to be less than the threshold, which means it is within claim scope that the third iteration is not performed. Therefore the claim includes steps that are not required to be performed and thus are not included in claim scope. Claims 14-27 depend from claim 13 and inherit this interpretation.
In the interest of compact prosecution, all language in these claims is addressed in the prior art rejections below. Claims 12 and 28-39 are not drawn to methods and are exempt from this interpretation. See MPEP 2111.04 II.
Claim Interpretation – Preamble
Claims 1-12 include language in their preamble that does not limit the scope of the claims. When a “preamble merely states, for example, the purpose or intended use of the invention, rather than any distinct definition of any of the claimed invention’s limitations, then the preamble is not considered a limitation and is of no significance to claim construction.” MPEP 2111.02 II. Claim 1 is drawn to a method “of increasing reliability of face recognition in analysis of images captured by drone mounted imaging sensors.” Claim 12 includes similar language. These preambles only describe an intended result of the claimed methods, but do not provide any “manipulative difference” in the method claims. Id. As such, the language in the preambles of these claim is not given patentable weight.
Objection, 37 CFR 1.173 – Improper Amendment
This application is objected to because the claim amendments do not conform to 37 CFR 1.173(d), which requires changes to be shown “relative to the patent being issued.” New claims 13, 28, and 32-33 show changes relative to the previous versions of these claims, and not relative to the patent, which included none of these claims. Claim 13 also shows limitations being simultaneously added (underlined) and omitted (bracketed), raising the question of whether these terms are included in the claim. Please refer to MPEP 1453(V)(D) for examples of how to amend new claims in a reissue application.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-12 and 28-39 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 1 and 12 recite “recognizing the target person in at least one iteration.” Based on the “at least one” language, the scope of the claim includes that the person is recognized in a single iteration. This single iteration appears to correlate to the new limitations1 that have been added to these claims. The specification suggests that these new limitations describe a first iteration, because “another iteration” of these steps is initiated if the probability score threshold is not satisfied:
In case the probability score exceeds the threshold, the face recognition app 220 may determine that the target person 204 is recognized with high confidence and the process 100 may branch to 116. However, in case the probability score does not exceed the threshold, the face recognition app 220 may determine that the target person 204 is not recognized with sufficient confidence and the process 100 may branch to 102 to initiate another iteration …
See ‘984 patent at 14:59-66 (emphasis added). Amended claims 1 and 12 do not recite that the first “probability score” (see line 10 of claim 1) is not sufficient to recognize the person, and based on this, the claim scope appears to allow for the person to be recognized based on the first iteration alone.
The claim goes on to recite what occurs in “each iteration.” It is not clear if “each iteration” described here is limited to the steps that follow this phrase, or if “each iteration” also includes the first iteration. The claim recites that “each iteration” includes identifying additional positioning properties based on additional images, adjusting the drone to an additional position, receiving additional facial images of the person, etc. It is not clear how the first iteration could comprise these “additional” steps when the first iteration is the first time these steps are performed. It is also not clear if the scope of the claim could include recognizing the person in only a single iteration, or if the “additional” steps are required to be performed.
The applicant’s intent appears to be that a first iteration is not sufficient to recognize the person, and at least one additional iteration is required. The claims will be examined based on this understanding. To overcome this rejection, it is recommended that claim 1 be amended to recite “determining that the probability score does not exceed a threshold, and in response, recognizing [a] the target person in at least one additional iteration, each additional iteration comprising: …” A similar amendment of claim 12 would likewise overcome the rejection of that claim.
Claims 1-12 further recite “calculating an updated aggregated probability score based on the additional probability score received in a current iteration of the at least one iteration to an aggregated score calculated before said current iteration.” This language is also indefinite.
This limitation requires that in each iteration, an updated aggregated probability score is determined based on two pieces of information: 1) an additional probability score received in the current iteration, and 2) an aggregated score that was calculated before the current iteration. It is not clear how the “aggregated score calculated before said current iteration” can be available prior to a third iteration. In a first iteration, there are no previous scores to aggregate. During a second iteration, a probability score would be calculated based on an additional score received in that iteration and the (non-aggregated) first score received in the first iteration. Only beginning at the third iteration would the score from previous iterations be an aggregate score (e.g., in the third iteration, an “aggregated score calculated before said current iteration” would be available based on iterations one and two). Therefore, during iterations one and two, it is not clear how the “updated aggregated probability score” is to be calculated. In addition, it is not clear what is required by reciting that this score is calculated “based on the additional probability score received in a current iteration of the at least one iteration to an aggregated score calculated before said current iteration.”
Claim 9 is additionally rejected under §112(b). Claim 9 recites the phrase “the at least two iterations.” There is insufficient antecedent basis for this in the claim.
Claims 28-39 are drawn to a drone. Limitations in an apparatus claim must limit the apparatus (in this case, the drone) to a particular structure. However, claims 28-39 recite various steps that a server, which is remote from the drone, is configured to perform. Because the claim is drawn to the drone, and not to a system comprising the drone and the server, this language does not appear to limit the claim: “Claim scope is not limited … by claim language that does not limit a claim to a particular structure.” MPEP 2111.04 I. However, given the extensive limitation placed on the functionality of the server, it is not clear what language does and does not limit these claims. As such, the claim scope is indefinite.
In addition, claim 28 recites “transmitting the second image to the remote server for” performing various steps at the server. Because these steps are performed at the server, and not at the drone to which claim 28 is drawn, this appears to describe the intended result of the “transmitting” step. It is not clear what structural limitations this places on the drone. See MPEP 2173.05(g) (a claimed intended result “does not provide a clear cut indication of scope because it imposed no structural limits” on the claim).
To overcome this issue, it is recommended that the claim be amended to recite a system comprising a drone and a remote server, rather than just the drone.
Claim Rejection, 35 USC 251 – Recapture
Claims 1-12 are rejected under 35 U.S.C. 251 as being an impermissible recapture of broadened claimed subject matter surrendered in the application for the patent upon which the present reissue is based. In re McDonald, 43 F.4th 1340, 1345, 2022 USPQ2d 745 (Fed. Cir. 2022); Greenliant Systems, Inc. et al v. Xicor LLC, 692 F.3d 1261, 103 USPQ2d 1951 (Fed. Cir. 2012); In re Youman, 679 F.3d 1335, 102 USPQ2d 1862 (Fed. Cir. 2012); In re Shahram Mostafazadeh and Joseph O. Smith, 643 F.3d 1353, 98 USPQ2d 1639 (Fed. Cir. 2011); North American Container, Inc. v. Plastipak Packaging, Inc., 415 F.3d 1335, 75 USPQ2d 1545 (Fed. Cir. 2005); Pannu v. Storz Instruments Inc., 258 F.3d 1366, 59 USPQ2d 1597 (Fed. Cir. 2001); Hester Industries, Inc. v. Stein, Inc., 142 F.3d 1472, 46 USPQ2d 1641 (Fed. Cir. 1998); In re Clement, 131 F.3d 1464, 45 USPQ2d 1161 (Fed. Cir. 1997); Ball Corp. v. United States, 729 F.2d 1429, 1436, 221 USPQ 289, 295 (Fed. Cir. 1984). The reissue application contains claim(s) that are broader than the issued patent claims. The record of the application for the patent family shows that the broadening aspect (in the reissue) relates to claimed subject matter that applicant previously surrendered during the prosecution of the application. Accordingly, the narrow scope of the claims in the patent was not an error within the meaning of 35 U.S.C. 251, and the broader scope of claim subject matter surrendered in the application for the patent cannot be recaptured by the filing of the present reissue application.
The test for recapture involves a three-step process:
(1) first, we determine whether, and in what respect, the reissue claims are broader in scope than the original patent claims;
(2) next, we determine whether the broader aspects of the reissue claims relate to subject matter surrendered in the original prosecution; and
(3) finally, we determine whether the reissue claims were materially narrowed in other respects, so that the claims may not have been enlarged, and hence avoid the recapture rule.
MPEP 1412.02 II.
Step (1): claims 1-12 are broader in scope than the claims of the ‘984 patent because they have been amended to remove language relating to “accumulating” probability scores. Analysis therefore proceeds to step 2.
Step (2): subject matter omitted in claims 1-12 was surrendered during prosecution of application 16/933,016 (“the ‘016 application”) from which the ‘984 patent issued.
In claim amendments made during prosecution of the ‘016 application, the following limitation was added to claims 1 and 13: “in each iteration said aggregated probability score is increased by an amount of said probability score received in the current iteration” was added in a 08/15/2022 amendment.
This amendment was made in direct reply to rejections in the Office action that preceded the response. In addition, the limitation was identified by the applicant as being made in order to render the claims allowable over the prior art of record. See e.g. Remarks filed 08/15/2022 at 8-9. The limitation was surrendered because it was “originally relied upon by applicant in the original application to make the claims allowable.” MPEP 1412.02 (II)(B)(1). Because the omitted claim amendments relate to surrendered subject matter, analysis proceeds to step 3.
Step (3): there has been no material narrowing relative to the surrendered subject matter, as the surrendered limitations have been entirely omitted from claims 1-12. MPEP 1412.02(II)(C).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-6 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over JP 6340538 (“JP ‘538”), in view of Nishio, US 11417088, in view of JP 3580129 (“JP ‘129”), and further in view of Chan, US 20190163962.
Note: for the Japanese documents below, reference is made to the English translations that have been provided previously.
Claim 1: JP ‘538 discloses a computer implemented method of increasing reliability of face recognition in analysis of images captured by drone mounted imaging sensors, comprising:
identifying a positioning property of a target person based on analysis of a image captured by at least one imaging sensor mounted on a drone operated to approach the target person (After determining that the image contains a person, it is determined if the person’s face is observable. If not observable, the drone flies to a position on the “front side” of the person’s face. Pg. 5 ¶7. This shows that the drone determines the direction that the person is facing, i.e. a positioning property of the person.), the at least one imaging sensor locally controlled at the drone (The camera 350 is operated locally by the drone. Fig. 2 and pg. 2 ¶5.)
instructing the drone to adjust its position to an optimal facial image capturing position selected based on the positioning property (The drone flies to a position on the “front side” of the person’s face in order to capture another image and detect the face of the person. Pg. 5 ¶7. The location to which the drone is moved meets the BRI of the claimed “optimal … position” when that term is understood in light of the ‘984 patent at 4:44-62),
receiving a facial image of the target person captured by the at least one imaging sensor while the drone is located at the optimal facial image capturing position, receiving a face classification associated with an image recognition procedure wherein the image recognition procedure is executed by at least one remote system connected to the drone via at least one network, and outputting the face classification for use by at least one face recognition based system (The image is analyzed to identify the person, and an identification (i.e. a face classification) is output for recognition. Pg. 5 ¶9. The image recognition is performed on server 50, which is connected to the drone via a network. See Fig. 1 and 2),
recognizing the target person in at least one iteration (A person is recognized in the person recognition step S160. As shown in Fig. 3, if the person is not recognized in this step, the process returns to the moving step S120, and another iteration of the steps S120-S160 is performed.), and
initiating another iteration in case the probability score does not exceed a certain threshold (If a match rate is not reached (e.g. a probability of identification does not exceed a threshold), another iteration of the steps of Fig. 3 is performed. See Fig. 3 and its description.).
wherein the operation of the drone is controlled locally at the drone (Various drone operations are controlled on-board the drone. Fig. 2 and pg. 3 ¶¶1-3.).
In case it is argued that JP ‘538 does not disclose identifying at least one positioning property of the target person and instructing the drone to adjust its position to an optimal facial image capturing position selected based on the at least one positioning property, this is rendered obvious by a combination of JP ‘538 and Nishio.
Nishio discloses a drone that is configured for identifying at least one positioning property of a target person and instructing the drone to adjust its position to a position selected based on the at least one positioning property (Abstract and 5:15-41).
It would have been obvious to a skilled artisan before the effective filing date of the claimed invention to modify JP ‘538 with teachings found in Nishio, the rationale being to enable the drone to quickly and efficiently identify and move to a location from which an image of the face of the person can be captured.
JP ‘538 fails to disclose receiving a face classification associated with a probability score from at least one machine learning model trained to recognize the target person which is applied to the facial image, wherein the at least one machine learning model is executed by at least one remote system connected to the drone via at least one network.
JP ‘538 also fails to disclose that each iteration comprises:
identifying at least one additional positioning property of the target person based on analysis of at least one additional image captured by the at least one imaging sensor;
instructing the drone to adjust its position to an additional optimal facial image capturing position selected based on the at least one additional positioning property;
receiving at least one additional facial image of the target person captured by the at least one imaging sensor while the drone is located at the additional optimal facial image capturing position,
receiving an additional face classification associated with an additional probability score from the at least one machine learning model, and
calculating an updated aggregated probability score based on the additional probability score received in a current iteration of the at least one iteration to an aggregated score calculated before said current iteration such that, in each iteration of the at least one iteration, said aggregated probability score is determined based on the probability score and respective additional probability scores of the at least one iteration, and
initiating another iteration in case the aggregated probability score does not exceed a certain threshold.
JP ‘129 discloses receiving a face classification associated with a probability score from at least one machine learning model trained to recognize a person which is applied to at least one facial image (A machine learning (neural network – see ¶37) routine processes an image of the face of a person and provides a classification (likeness) that is associated with a probability, or likeness degree. ¶¶38-39. This implicitly describes that the ML routine is trained to recognize the person),
identifying at least one additional positioning property of the target person based on analysis of at least one additional image captured by at least one imaging sensor (Based on captured images and the direction that the person is facing, a condition is estimated for acquiring another image that will effectively identify the person. ¶¶ 7 and 33-35. This estimated condition includes changing the camera position. ¶¶ 29 and 33-35. Therefore an additional positioning property of the person is identified based on imagery),
instructing the camera to adjust its position to an additional optimal facial image capturing position selected based on the at least one additional positioning property (The camera is moved to a position based on the estimated condition, or the additional positioning property. ¶¶ 29 and 34.),
receiving at least one additional facial image of the target person captured by the at least one imaging sensor while the camera is located at the additional optimal facial image capturing position, receiving an additional face classification associated with an additional probability score from the at least one machine learning model (¶¶ 29-35 and 38-39.),
and initiating another iteration in case the probability score does not exceed a certain threshold; and outputting the face classification (If the probability does not exceed the threshold, a new iteration of imaging and identification is performed. ¶42. The new iteration involves moving the imaging unit to capture an image of the person’s face from a new location. ¶¶ 17 and 29. When the identification probability exceeds the threshold, the iterations cease and the person is identified, i.e. the face classification is output. ¶39-41.).
It would have been obvious to a skilled artisan before the effective filing date of the claimed invention to modify JP ‘538 with teachings found in JP ‘129 by enabling the face of the person to be re-captured from a different location, the motivation being to increase the likelihood of positively identifying the person. See JP ‘129 ¶¶ 5 and 29. When modifying JP ‘538 to include this feature, the POSITA would have found that the logical and common-sense manner to achieve this advantage would have been to adjust the drone’s location, similar to camera location adjustment in JP ‘129, in order to re-capture the face of the person from a different angle in order to generate a more confident identification.
The JP ‘538 – JP ‘129 combination fails to disclose calculating an updated aggregated probability score based on the additional probability score received in a current iteration of the at least one iteration to an aggregated score calculated before said current iteration such that, in each iteration of the at least one iteration, said aggregated probability score is determined based on the probability score and respective additional probability scores of the at least one iteration.
However, this is suggested in Chan. Chan suggests calculating an updated aggregated probability score based on an additional probability score and an aggregated score calculated before the current calculation such that said aggregated probability score is determined based on the probability score and respective additional probability scores of the at least one iteration. (Multiple images of an unknown person are analyzed in an attempt to identify the person. ¶¶27-28. The confidence of a match of a first image is combined with the confidence of a match in a second image to generate an aggregated probability score. ¶¶28-34.).
It would have been obvious to a skilled artisan before the effective filing date of the claimed invention to modify the system of JP ‘539 and JP ‘129 with these teachings in Chan. The POSITA would have been motivated by Chan to calculate the probability score by updating a combined probability in each iteration, rather than simply determining if a probability score of the current iteration exceeds a threshold, as JP ‘129 discloses (see above). The POSITA would understand that calculating probability in this way would reflect the statistical fact that when multiple images are captured of a person, multiple lower-confidence identifications provide a higher confidence than is reflected by the individual scores. See Chan ¶34. Therefore calculating probability in this manner would enable higher-confidence results and would reduce the number of iterations that the JP ‘539 - JP ‘129 system would have to perform before the person is identified.
Claim 2: the JP ‘538 – Nishio – JP ‘129 – Chan combination discloses that the at least one positioning property is a member of a group consisting of: a head pose of the target person (JP ‘538 pg. 5 ¶7 and Nishio 5:15-41), a property of at least one potentially blocking object with respect to the target person and at least one environmental parameter affecting image capturing of the target person (Nishio Fig. 9: 1024).
Claim 3: the JP ‘538 – Nishio – JP ‘129 – Chan combination discloses that the optimal facial image capturing position is defined by at least one position parameter of the drone, the at least one position parameter is a member of a group consisting of: a location of the drone with respect to the target person (JP ‘538 pg. 5 ¶7 and Nishio 5:15-41).
Claim 4 the JP ‘538 – Nishio – JP ‘129 – Chan combination does not disclose adjusting at least one operational parameter of the at least one imaging sensor based on the at least one positioning property, the at least one operational parameter is a member of a group consisting of: a resolution, a zoom, a color, a field of view, an aperture, a shutter speed, a sensitivity (ISO), a white balance and an auto exposure. However, official notice is taken that this was well known in the art before the effective filing date of the claimed invention. For example, it was well known to adjust an imaging property such as a zoom level based on a distance, or positioning property, from the object being photographed. Therefore it would have been obvious to the skilled artisan to modify the JP ‘538 – Nishio – JP ‘129 – Chan combination to include this, the rationale being to generate an improved image of the person’s face.
Claim 5: the JP ‘538 – Nishio – JP ‘129 – Chan combination discloses that the at least one machine learning model is based on a neural network (JP ‘129 ¶37).
Claim 6: the JP ‘538 – Nishio – JP ‘129 – Chan combination discloses that at least part of the face classification is done by the at least one remote system connected to the drone via at least one network to receive facial image captured by the at least one imaging sensor mounted on the drone (The image recognition is performed on server 50, which is connected to the drone via a network, based on images captured by the drone-mounted camera. See JP ‘538 pg. 5 ¶9 and Fig. 1 and 2. When modified to perform the face classification using the JP ‘129, the face classification is done by the remote system).
Claim 12: See rejection of claim 1. The JP ‘538 – Nishio – JP ‘129 – Chan combination also discloses a system for increasing reliability of face recognition in analysis of images captured by drone mounted imaging sensors, comprising: at least one processor executing a code, the code comprising: code instructions for performing the method recited in claim 1. See e.g. JP ‘538 Figures 1-2 and their description.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over the JP ‘538 – Nishio – JP ‘129 – Chan combination in view of Okubo, JP 2004362079.
Claim 7: the JP ‘538 – Nishio – JP ‘129 – Chan combination fails to disclose initiating additional iterations to capture a plurality of facial images depicting the face of the target person from a plurality of view angles to form an at least partial three dimensional (3D) representation of at least part of a head of the target person, the plurality of facial images different from the at least one additional facial image of the target person.
However, Okubo discloses initiating a plurality of iterations to capture a plurality of facial images depicting the face of the target person from a plurality of view angles to form an at least partial three dimensional (3D) representation of at least part of a head of the target person, the plurality of facial images different from the at least one additional facial image of the target person (Images of the face of a person is captured from a plurality of angles to generate a 3D model of the person. ¶12).
It would have been obvious to a skilled artisan before the effective filing date of the claimed invention to modify the JP ‘538 – Nishio – JP ‘129 – Chan combination with teachings found in Okubo in order to ensure proper identification of the person.
Claims 8-11 are rejected under 35 U.S.C. 103 as being unpatentable over the JP ‘538 – Nishio – JP ‘129 – Chan combination in view of Kelly, US 20180253805.
Claim 8: the JP ‘538 – Nishio – JP ‘129 – Chan combination fails to disclose that the face recognition based system is an automated delivery system using the face classification for authenticating an identity of the target person for delivery of goods to the target person.
However, this is suggested in Kelly, which discloses a face recognition based system, where an automated delivery system uses a face classification for authenticating an identity of a target person for delivery of goods to the target person (A drone-based delivery system identifies a recipient of a delivered good using a facial scan. ¶297).
It would have been obvious to a skilled artisan before the effective filing date of the claimed invention to modify the JP ‘538 – Nishio – JP ‘129 – Chan combination with teachings in Kelly, the rationale being to diversity the capabilities of the system, thereby providing additional opportunities for revenue.
Claim 9: Kelly discloses initiating the at least two iteration for recognizing the target person in correlation with the goods at a time of delivery (¶¶ 291 and 297).
Claim 10: the JP ‘538 – Nishio – JP ‘129 – Chan – Kelly combination discloses that a location of the target person where the at least one image is captured for recognizing the target person is different form the location of the target person during the time of delivery (This is implicit – the drone location from where the image is captured is different from the location of the person being photographed. See also the §112(b) rejection above).
Claim 11: Kelly discloses instructing the drone to initiate at least one additional authentication sequence which is a member of a group consisting of: manual signature of the target person, a biometric authentication of the target person and a voice authentication of the target person (¶297).
Claims 13-15, 21, 27-33, and 39 are rejected under 35 U.S.C. 103 as being unpatentable over JP ‘538, JP ‘129, Applicant Admitted Prior Art (AAPA), and Chan.
Claim 13: the combination of JP ‘538 and JP ‘129, described in the rejection of claim 1 above, discloses a computer-implemented method, comprising:
receiving, by a remote server, a first facial image of a person captured by a drone, the drone including at least one imaging sensor configured to capture images and one or more processors configured to control the at least one imaging sensor (An image is captured by a drone and delivered to a server to be analyzed to identify the person. JP ‘538 Pg. 5 ¶9. The image recognition is performed on server 50, which is connected to the drone via a network. JP ‘538 Fig. 1 and 2. The drone contains image sensors and processors. See JP ‘538 Fig. 1 and its description);
applying, by the remote server, at least one machine learning model to the first facial image; determining a first probability score indicating a likelihood that the person corresponds to an identity of a target person based on the first facial image (A machine learning (neural network – see JP ‘129 ¶37) routine processes the image of the face of a person and provides a classification (likeness) that is associated with a probability of identification. JP ‘129 ¶¶38-39.);
determining that the first probability score is no greater than a threshold;
transmitting, by the remote server, a command to the drone to capture a second facial image of the person; receiving, by the remote server, the second facial image of the person captured by the drone; and determining, by the remote server, a second probability score indicating a likelihood that the person corresponds to the identity of the target person (If the probability does not exceed the threshold, a new iteration of imaging and identification is performed. JP ‘129 ¶42. The new iteration involves moving the imaging unit to capture an image of the person’s face from a new location. JP ‘129 ¶¶ 17 and 29. In the JP ‘538 – JP ‘129 combination, capturing the image from a second location entails commanding the drone to relocate to a second position. See rejection of claim 1.); and
in response to determining that the second probability score is no greater than the threshold, determining, by the remote server, whether a third probability score is greater than the threshold, in response to determining that the third probability score is greater than the threshold, outputting the identity of the target person for use by at least one face recognition-based system (When the identification probability exceeds the threshold, the iterations cease and the person is identified. JP ‘129 ¶41. The scope of this disclosure includes that the user is identified in a third iteration.).
The JP ’538 – JP ‘129 system fails to disclose: extracting a first feature vector from the first facial image; determining the probability score based on a distance between the first feature vector and a second feature vector extracted from training facial images of a target person; that the second probability score is determined based on the first probability score and a distance between a third feature vector extracted from the second facial image and a fourth feature vector extracted from training facial images of the target person; the third probability score determined based on the first probability score and the second probability score, the third probability score determined using a third facial image of the person captured by the drone.
However, AAPA admits that it was known in the art to extract a feature vector from an image and determine a probability score based on a distance between this feature vector and a feature vector extracted from training images. For example, the ‘984 patent describes:
As known in the art, the ML model(s) are configured and trained to extract feature vectors from the images and classify the extracted feature vectors to respective classes (labels), for example, an identifier (identity) of the target person according to matching of the feature vectors with feature vectors extracted from training images of the target person 204 processed and learned by the ML model(s) during their training phase.
The ML model(s) may typically compute a probability score (confidence score) indicating a probability (confidence level) of correct classification of the face of the target person 204 and hence the probability and confidence of correctly recognizing the target person 204.
The probability score expressing confidence of the correct recognition may be interpreted as the distance between the feature vector(s) extracted from the facial image(s) of the target person 204 captured by the imaging sensor(s) 216 from the optimal image capturing position and feature vectors which are extracted from training facial images of the target person 204 which were used to train the ML model(s). For example, as known in the art, cosine similarity may be measured between feature vectors A∙B extracted from facial image(s) of the target person 204 captured by the imaging sensor(s) 216 and ∥A∥ ∥B˜ extracted from the training facial image(s) of the target person.
‘984 patent at 14:15-45 (emphasis added). It would have been obvious to a skilled artisan to modify JP538-JP129 combination with this by determining the probability of identification based on the distance of the claimed feature vectors, the rationale being to provide an improved confidence score.
The JP538-JP129-AAPA combination would calculate a first, second, and third probability score based on distances between feature vectors extracted from images in its respective iteration and feature vectors extracted from training images. However, this system would not determine the second probability score based on the first probability score and a distance between feature vectors; or that the third probability score determined based on the first probability score, the second probability score, and a third facial image of the person captured by the drone.
However, this is suggested in Chan. For reasons given above, Chan would have provided suggestion to modify the JP538-JP129 combination to aggregate probability scores at each iteration. See the rejection of claim 1. The POSITA would have found the same motivation to modify the JP538-JP129-AAPA combination in the same manner. Namely, when probability scores are computed in the manner taught by AAPA, the system would still benefit from aggregating probability scores of individual iterations, because doing so would enable the drone to identify the person in fewer iterations. Stated in another way, regardless of how the scores are calculated, the fact remains that multiple low-confidence probability scores may be sufficient to identify the person, as taught in Chan ¶34.
Based on this, the POSITA would have found it obvious to modify the JP538-JP129-AAPA combination to determine the second probability score based on the first probability score and a distance between a third feature vector extracted from a second facial image and a fourth feature vector extracted from training facial images of the target person; and determine the third probability score based on the first probability score, the second probability score, and a third facial image of the person captured by the drone. The rationale for this modification is that aggregating individual probability scores in this manner would have enabled the drone to identify the user in fewer iterations.
Claim 14: the JP ‘538 – JP ‘129 – AAPA - Chan system discloses that determining, by the remote server, a second probability score indicating a likelihood that the person corresponds to the identity of the target person based on both the first facial image and the second facial image comprises: applying, by the remote server, the at least one machine learning model to the second facial image to determine a third probability score indicating a likelihood that the person corresponds to the identity of the target person based on the second facial image; and aggregating, by the remote server, the first probability score and the third probability score to determine the second probability score (Multiple images of an unknown person are analyzed in an attempt to identify the person. Chan ¶¶27-28. The confidence of a match of a first image is combined with the confidence of a match in a second image, to generate an aggregated probability score. Chan ¶¶28-34. When the JP ‘538 – JP ‘129 system is modified with Chan, the aggregated probability scores are determined at the server based on the drone-captured images).
Claim 15: the JP ’538 – JP ‘129 system discloses that the second facial image is captured from a different view angle relative to the first facial image (JP ‘129 ¶¶17 and 29).
Claim 21: JP ‘538 discloses that drone comprises one or more imaging sensors selected from a visible-light camera, an infrared camera, or a thermal camera (Fig. 2 and its description – camera 350 is a visible light or IR camera).
Claim 27: Chan discloses capturing one or more additional facial images after the second facial image, and updating the second probability score based on the one or more additional facial images (see ¶¶ 27-34).
Claim 28: The rejection of claim 28 adopts the combination that is described in the rejection of claims 1 and 13. See above. The JP ‘538–JP ‘129-AAPA-Chan combination described above discloses a drone comprising :at least one imaging sensor; at least one processor; and a non-transitory storage medium, storing instructions, which when executed by the at least one processor, cause the at least one processor to performs steps (see JP ‘538 and its description) including:
capturing a first facial image of a person; transmitting the first facial image to a remote server configured to apply at least one machine learning model to extract a first feature vector (An image is captured by a drone and delivered to a server to be analyzed to identify the person. JP ‘538 Pg. 5 ¶9. The image recognition is performed on server 50, which is connected to the drone via a network. JP ‘538 Fig. 1 and 2. A machine learning (neural network – see JP ‘129 ¶37) routine on the remove server processes the image of the face of a person and provides a likelihood of identification. JP ‘129 ¶¶38-39. The ML model extracts feature vectors from the image. AAPA, ‘984 patent at 14:15-45);
determine a first probability score indicating a likelihood that the person corresponds to an identity of a target person, the first probability score determined based on a distance between the first feature vector and a second feature vector extracted from training facial images of the target person (AAPA, ‘984 patent at 14:15-45);
receiving, from the remote server, a command to capture a second facial image of the person in response the first probability score not exceed a threshold; capturing, by the at least one imaging sensor, the second facial image of the person according to the command; and transmitting the second facial image to the remote server for determining a second probability score based on the probability score and a distance between a third feature vector extracted from the second facial image and a fourth feature vector extracted from training facial images of the target person (AAPA, ‘984 patent at 14:15-45 and Chan ¶34),
outputting the identity of the target person in response to the second probability score exceeding the threshold (If the probability of identification does not exceed the threshold, a new iteration of imaging and identification is performed. JP ‘129 ¶42. The new iteration involves moving the imaging unit to capture an image of the person’s face from a new location. JP ‘129 ¶¶ 17 and 29. In the JP ‘538 – JP ‘129 combination, capturing the image from a second location entails commanding the drone to relocate to a second position. See rejection of claim 1. When the probability exceeds the threshold, an identity of the person is output. JP ‘129 ¶¶ 17 and 29), and
transmitting a third facial image to the remote server in response to the second probability score not exceeding the threshold, the remote server determining whether a third probability score is greater than the threshold, the third probability score determined based on the first probability score and the second probability score, the third probability score determined using the third facial image of the person captured by the drone (When the identification probability exceeds the threshold, the iterations cease and the person is identified. JP ‘129 ¶41. The scope of this disclosure includes that the user is identified in a third iteration. The third probability score is calculated based on the feature vectors described in AAPA and the aggregation described in Chan ¶¶ 28-34.); and
in response to determining that the third probability score is greater than the threshold, outputting the identity of the target person for use by at least one face recognition-based system (When the probability exceeds the threshold, an identity of the person is output. JP ‘129 ¶¶ 17 and 29).
Claim 29 JP ‘538 discloses that the command received from the remote server comprises at least one of: a location of the drone with respect to the person (pg. 5 ¶¶ 7-9).
Claim 30-31: see rejection of claim 4.
Claim 32: JP ‘538 – JP ‘129 discloses receiving, from the remote server, one or more additional commands to capture one or more additional facial image of the person (If the probability of identification does not exceed the threshold, a new iteration of imaging and identification is performed. JP ‘129 ¶42. The new iteration involves moving the imaging unit to capture an image of the person’s face from a new location. JP ‘129 ¶¶ 17 and 29.).
Claim 33: JP ‘538 – JP ‘129 discloses that the second facial image is captured from a different view angle relative to the first facial image (The new iteration involves moving the imaging unit to capture an image of the person’s face from a new location. JP ‘129 ¶¶ 17 and 29.)
Claim 39: see rejection of claim 21.
Claims 16 and 34 are rejected under 35 U.S.C. 103 as being unpatentable over JP ‘538, JP ‘129, AAPA, and Chan in view of Okubo.
Claim 16: the JP ‘538, JP ‘129, AAPA, and Chan system does not disclose constructing at least a partial three-dimensional representation of a head of the person based on the first facial image and the second facial image, wherein the second probability score is determined based in part on the partial three-dimensional representation of the head of the person.
Okubo discloses initiating a plurality of iterations to capture a plurality of facial images depicting the face of the target person from a plurality of view angles, and constructing at least a partial three-dimensional representation of a head of the person based on the first facial image and the second facial image (Images of the face of a person is captured from a plurality of angles to generate a 3D model of the person. ¶12).
It would have been obvious to a skilled artisan before the effective filing date of the claimed invention to modify the JP ‘538 – JP ‘129 combination with teachings found in Okubo in order to ensure proper identification of the person. When motivated to make this modification, the skilled artisan would have found it obvious to calculate the second probability score based in part on the partial three-dimensional representation of the head of the person, as this would produce a more reliable probability of identification. Therefore the combined teachings of JP ‘538, JP ‘129, and Okubo suggest the invention that is recited in claim 16.
Claim 34: see rejection of claim 16.
Claims 17-20 and 35-38 are rejected under 35 U.S.C. 103 as being unpatentable over JP ‘538, JP ‘129, APAA, Chan, and Kelly.
Claims 17 and 18: see rejection of claims 8-11.
Claim 19: Kelly discloses that the outputted identity of the target person is used by a security system (¶291 describes a system to securely deliver goods, which is a security system).
Claim 20: Kelly discloses instantiating an additional authentication operation, the additional authentication operation including at least one of: manual signature capture, biometric authentication, or voice authentication (¶297).
Claims 35-38: see rejection of claims 17-20.
Claims 22-26 are rejected under 35 U.S.C. 103 as being unpatentable over JP ‘538, JP ‘129, AAPA, Chan, and Nishio.
Claim 22: the JP ‘538 – JP ‘129 – Nishio combination that is described in the rejection of claim 1 discloses determining one or more positioning properties of the person based on the first facial image (After determining that the image contains a person, it is determined if the person’s face is observable. If not observable, the drone flies to a position on the “front side” of the person’s face. JP ‘538 Pg. 5 ¶7. This shows that the drone determines the direction that the person is facing, i.e. a positioning property of the person. See also Nishio Abstract and 5:15-41); and determining an updated position of the drone relative to the person based on the one or more positioning properties of the person, wherein the command transmitted to the drone to capture the second facial image of the person includes the updated position of the drone, instructing the drone to capture the second facial image at the updated position (The drone flies to a position on the “front side” of the person’s face in order to capture a second image and detect the face of the person. JP ‘538 Pg. 5 ¶7.).
Claim 23: the JP ‘538 – JP ‘129 – Nishio combination discloses that the one or more positioning properties of the person comprise at least one of: a head pose of the person, a blocking object that blocks a portion of a face of the person, or one or more environmental parameters (JP ‘538 pg. 5 ¶7 and Nishio 5:15-41; and Fig. 9: 1024).
Claim 24 the JP ‘538 – JP ‘129 – Nishio combination discloses that the command transmitted to the drone to capture the second facial image comprises at least one of: a location of the drone with respect to the person (JP ‘538 pg. 5 ¶7 and Nishio 5:15-41)
Claims 25-26: see rejection of claim 4.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT J HANCE whose telephone number is (571)270-5319. The examiner can normally be reached M-F 11:00am-7:00pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael Fuelling can be reached at (571) 270-1367. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ROBERT J HANCE/Primary Examiner, Art Unit 3992
Conferees:
/CHARLES R CRAVER/Reexamination Specialist, Art Unit 3992 /M.F/Supervisory Patent Examiner, Art Unit 3992
1 For example, the “identifying,” “instructing,” “receiving a facial image,” and “receiving a face classification” steps that have been added to claim 1
Prosecution Insights