DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The Amendment filed October 27, 2025 has been entered. Claims 1-14, 16 and 18 remain pending in the application. Applicant’s amendments to the Claims have overcome each and every objection, 112(a) rejection and 112(b) rejection previously set forth in the Non-Final Office Action mailed July 2, 2025.
Priority
Acknowledgment is made of Applicant's claim for foreign priority based on an application filed in Japan on 01/01/2022. It is noted, however, that Applicant has not filed a certified copy of the JP2022-000011 application as required by 37 CFR 1.55.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 6-8, 11-14, 16 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Sengupta (US 2018/0101733) in view of Yoneyama (US 2021/0303846).
Regarding claim 1, Sengupta discloses an image processing apparatus (paragraph 0014: “analyzing one or more of the images to determine whether a face is (or faces are) present within a three-dimensional detection zone”) comprising: a detection circuit that applies processing for detecting a subject of a first type (paragraph 0039: “the face-detection algorithm”) and a human subject as a subject of a second type being different from the first type to an image (FIG. 3, paragraph 0044: “an algorithm for detecting the operator's head-shoulder portion may be executed”; the subject of the second type may also be the operator’s torso detected from the “full-body detector”); and a control circuit that executes tracking processing of a subject based on a detection result of the detection circuit (FIG. 3, paragraph 0014: “thereafter tracking the operator between successive images”), wherein the control circuit, if a subject of the first type and a subject of the second type that are detected by the detection circuit constitute a same subject (paragraph 0044: the face and the head-shoulder portion belong to the same operator indicated by “A match is declared when a certain fraction of key points or patches (e.g., at least half) fall within the detected head-shoulder region”; similarly, the face and the torso belong to the same operator indicated by having “a depth consistent with the most recent location of the key point cluster”), selects either the detection result regarding the subject of the first type (paragraphs 0039-0040: “The key points [of the face and head] are tracked through a stream of successive image frames”) or the detection result regarding the subject of the second type is to be used to perform tracking processing of the subject of the first type (paragraph 0044: in cases when “Face re-detection may be unsuccessful in some instances…From the detected head-shoulder portion, the face may then be estimated and used for stabilization”), and after starting to perform the tracking processing of the subject of the first type using the detection result regarding the subject of the second type (FIG. 3, paragraph 0044: “From the detected head-shoulder portion, the face may then be estimated and used for stabilization and/or re-initiation” where re-initiation of key points (step 308) occurs after tracking the face using detection of head-shoulder region and/or torso (steps 316, 318)), if a frequency at which none of the subject of the first type and the subject of the second type is detected exceeds a threshold (paragraphs 0033, 0042, 0048: “If, on the other hand, a predetermined time period for re-entry expires before a re-entry has been detected (as determined in step 224), the operator's session may be closed, and the system may go back into its start mode, in which it waits for a new (or the same) authorized user to log on” to trigger the image acquisition and the detection and tracking processing; since the predetermined time period corresponds to a number of frames, failing to detect a subject within a predetermined time period is equivalent to failing to detect a subject within a number of frames, which is equivalent to a frequency of detection failure being above a threshold) or the subject of the first type is not detected for a predetermined time period (paragraph 0044: “In such cases, when no match between the current key point cluster and a face in the detection zone can be found”) while the subject of the second type is continuously detected (paragraph 0044: “a full-body detector may check whether a foreground object is still present in front of the terminal at a depth consistent with the most recent location of the key point cluster (step 318)”), initializes or redetermines a target of the tracking processing (paragraphs 0044-0045: “If so, and if the operator's head re-appears in the images within a specified number of frames, tracking continues. Otherwise, the operator is deemed to have left the scene…Detecting walk-away conditions enables the secure system to automatically close the authenticated session in a timely manner” where as previously discussed, a specified number of frames corresponds to a predetermined time period and when the operator's session is closed, the system goes back into its start mode, in which it waits for a new (or the same) authorized user to log on to trigger the image acquisition and the detection and tracking processing). However, Sengupta fails to disclose detecting a transportation subject as a subject of a first type. In related art, Yoneyama discloses detecting a transportation subject as a subject of a first type (Yoneyama paragraphs 0029-0030: “a target subject is detected from within taken images…in a case where a target subject is a formula car [a racing car, open-wheel car]”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Sengupta to incorporate the teachings of Yoneyama to detect the position of target subjects without error when shooting a moving target subject (Yoneyama paragraph 0176).
Regarding claim 6, Sengupta, modified by Yoneyama, discloses the image processing apparatus claimed in claim 1, wherein the detection circuit detects the entirety and a passenger's head with respect to the transportation subject (Yoneyama FIG. 12, paragraph 0147: “The input image INpic is an image of an entire vehicle PICtot, and further, a helmet of the driver exists inside the entire vehicle PICtot as a parts image PICpar. The feature detection processing section 1305A sets the entire image and parts image as respective feature images”), and detects a head with respect to the human subject (Sengupta paragraph 0039: “the initial head position and size within the image is determined based on the output of the face-detection algorithm”).
Regarding claim 7, Sengupta, modified by Yoneyama, discloses the image processing apparatus claimed in claim 1, wherein the subject of the second type is a part (Sengupta FIG. 3, paragraph 0044: “an algorithm for detecting the operator's head-shoulder portion may be executed”; the subject of the second type may also be the operator’s torso detected from the “full-body detector”) that is not detected as a subject of the first type (Sengupta paragraph 0039: “the face-detection algorithm”).
Regarding claim 8, Sengupta, modified by Yoneyama, discloses the image processing apparatus claimed in claim 7, wherein the detection circuit detects the entirety and a passenger's head with respect to the transportation subject (Yoneyama FIG. 12, paragraph 0147: “The input image INpic is an image of an entire vehicle PICtot, and further, a helmet of the driver exists inside the entire vehicle PICtot as a parts image PICpar. The feature detection processing section 1305A sets the entire image and parts image as respective feature images”), and detects a face (Sengupta paragraph 0039: “the face-detection algorithm”) and a pupil with respect to the human subject (Sengupta paragraph 0039: interest point detection of “visually significant points such as, e.g., the corners of the eyes”).
Regarding claim 11, Sengupta, modified by Yoneyama, discloses an image capturing apparatus comprising: an image sensor (Sengupta paragraph 0010: “Depth-sensing cameras based on various technologies”); an image processing apparatus in which an image obtained by using the image sensor is used (Sengupta paragraph 0014: “acquiring images with a depth-sensing camera system”); and an adjustment circuit for adjusting the focal point of an imaging optical system (Yoneyama paragraph 0036: “A photographing lens 102 is an optical system for forming an image of a subject F in the vicinity of an image sensor 114, and includes a focus lens etc. A focus adjustment mechanism 104 has an actuator (for example, a motor) for moving the focus lens in an optical axis direction”) based on a result of tracking processing performed by the image processing apparatus (Yoneyama paragraph 0081: “CPU 1301 executes AF processing so as to focus on a subject that is the tracking target for which position was obtained in the tracking processing”), wherein the image processing apparatus is as claimed in claim 1. Therefore, Sengupta, modified by Yoneyama, discloses the limitations of claim 11 as it does the limitations of claim 1.
Regarding claim 12, it is the corresponding method executed by the apparatus claimed in claim 1. Therefore, Sengupta, modified by Yoneyama, discloses the limitations of claim 12 as it does the limitations of claim 1.
Regarding claim 13, it is the corresponding non-transitory computer-readable medium storing a program configured to function as the apparatus claimed in claim 1. Therefore, Sengupta, modified by Yoneyama, discloses the limitations of claim 13 as it does the limitations of claim 1.
Regarding claim 14, Sengupta, modified by Yoneyama, discloses the image processing apparatus claimed in claim 1, wherein the control circuit, if a subject of the first type and a subject of the second type that are detected by the detection circuit constitute a same subject (as claimed in claim 1), after starting to perform the tracking processing of the subject of the first type using the detection result regarding the subject of the first type (Sengupta paragraphs 0039-0040: “The key points [of the face and head] are tracked through a stream of successive image frames”), if the subject of the second type is not detected but the subject of the first type is detected (Sengupta paragraph 0044: head-shoulder detection is not performed when face re-detection is successful), redetermines that the subject of the first type is a target of the tracking processing (Sengupta paragraph 0042: “To further improve overall tracking robustness, the tracking algorithm may be periodically re-initiated by selecting a new collection of key points based on the re-detection of the operator's face”).
Regarding claim 16, Sengupta, modified by Yoneyama, discloses the image processing apparatus claimed in claim 14, wherein if a frequency at which none of the subject of the first type and the subject of the second type is detected is not less than a predetermined frequency, the control circuit initializes or redetermines a target of the tracking processing (Sengupta paragraphs 0033, 0048: “If, on the other hand, a predetermined time period for re-entry expires before a re-entry has been detected (as determined in step 224), the operator's session may be closed, and the system may go back into its start mode, in which it waits for a new (or the same) authorized user to log on” to trigger the image acquisition and the detection and tracking processing).
Regarding claim 18, Sengupta, modified by Yoneyama, discloses the image processing apparatus according to claim 1, wherein the control circuit, after starting to perform the tracking processing of the subject of the first type using the detection result regarding the subject of the second type, if a frequency at which none of the subject of the first type and the subject of the second type is detected exceeds a threshold (Sengupta paragraphs 0042, 0048: “If, on the other hand, a predetermined time period for re-entry expires before a re-entry has been detected”; since the predetermined time period corresponds to a number of frames, failing to detect a subject within a predetermined time period is equivalent to failing to detect a subject within a number of frames, which is equivalent to a frequency of detection failure being above a threshold) or a tracking reliability is lower than a predetermined reliability continuous for a predetermined time period while the subject of the second type is continuously detected (this limitation is disclosed in an alternative clause and thus, read only on the first limitation), further initializes or redetermines a target of the tracking processing (Sengupta paragraphs 0033, 0048: “the operator's session may be closed, and the system may go back into its start mode, in which it waits for a new (or the same) authorized user to log on” to trigger the image acquisition and the detection and tracking processing).
Claim(s) 2-5 are rejected under 35 U.S.C. 103 as being unpatentable over Sengupta and Yoneyama in view of Midorikawa et al. (US 2020/0036895) and in further view of Tsuji (US 2013/0286217).
Regarding claim 2, Sengupta, modified by Yoneyama, discloses the image processing apparatus claimed in claim 1, wherein the control circuit, if a subject of the first type and a subject of the second type that are detected by the detection circuit constitute a same subject, selects either the detection result regarding the subject of the first type or the detection result regarding the subject of the second type is to be used to perform tracking processing of the subject of the first type (as claimed in claim 1). However, Sengupta fails to disclose selecting a detection result, based on at least one of detection reliability, a detected position in an image, and a motion of the image processing apparatus.
In related art, Midorikawa discloses selecting a detection result (Midorikawa paragraph 0130: “the main subject recognizing unit 802 recognizes the main subject based on the motion vectors”), based on at least one of a detected position in an image (Midorikawa paragraph 0135: “motion vectors belonging to the main subject candidate cluster are motion vectors in a direction approaching the image center”), and a motion of the image processing apparatus (Midorikawa FIG. 11, paragraph 0131: “the main subject recognizing unit 802 calculates the Euclidean distance between a representative vector [mean vector, for example] of motion vectors belonging to the main subject cluster and a representative vector [background vector b, for example] of motion vectors belonging to the background cluster. Also, the main subject recognizing unit 802 determines whether or not the calculated Euclidean distance is less than a predetermined threshold value…the fact that the distance is less than the threshold value indicates a state in which the motion of the main subject in the real space is small”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Sengupta to incorporate the teachings of Midorikawa to improve the likelihood that a main subject intended by a user can be recognized (Midorikawa paragraph 0140). However, Sengupta, modified by Yoneyama and Midorikawa, still fails to disclose selecting a detection result, based on at least one of detection reliability.
In related art, Tsuji discloses selecting a detection result, based on at least one of detection reliability (Tsuji paragraph 0066: “for a subject in which there are both a face detection result and a human body detection result, the result selection unit 205 may output one with a higher degree of reliability”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Sengupta to incorporate the teachings of Tsuji to improve the rate of detection for a subject area in an image and detect the subject area with ease (Tsuji paragraph 0010).
Regarding claim 3, Sengupta, modified by Yoneyama, Midorikawa and Tsuji, discloses the image processing apparatus claimed in claim 2, wherein the control circuit performs the tracking processing of the subject of the first type using, out of the detection result regarding the subject of the first type and the detection result regarding the subject of the second type (as claimed in claim 1), a detection result having higher detection reliability (Tsuji paragraph 0066: “for a subject in which there are both a face detection result and a human body detection result, the result selection unit 205 may output one with a higher degree of reliability”) given by the detection circuit (Tsuji paragraph 0066: “the face detection result obtainment unit 201 and the human body detection result obtainment unit 202 obtain degrees of reliability indicative of certainties of a face detection result and a human body detection result, respectively”).
Regarding claim 4, Sengupta, modified by Yoneyama, Midorikawa and Tsuji, discloses the image processing apparatus claimed in claim 1, wherein, if the subject of the first type is detected in a peripheral portion of an image (Midorikawa FIG. 11: S1102; paragraph 0135: “setting a condition that motion vectors belonging to the main subject candidate cluster are motion vectors in a direction approaching the image center as the predetermined condition”), and a motion is a threshold value or more (Midorikawa FIG. 11: S1101; paragraph 0131: “the main subject recognizing unit 802 calculates the Euclidean distance between a representative vector [mean vector, for example] of motion vectors belonging to the main subject cluster and a representative vector [background vector b, for example] of motion vectors belonging to the background cluster. Also, the main subject recognizing unit 802 determines whether or not the calculated Euclidean distance is less than a predetermined threshold value…the fact that the distance is less than the threshold value indicates a state in which the motion of the main subject in the real space is small”), the control circuit performs the tracking processing of the subject of the first type using the detection result regarding the subject of the second type (Midorikawa FIG. 11: S1103; paragraph 0133: “the main subject recognizing unit 802 determines to change the main subject”), even if a detection reliability of the subject of the first type is higher than a detection reliability of the subject of the second type.
Regarding claim 5, Sengupta, modified by Yoneyama, Midorikawa and Tsuji, discloses the image processing apparatus claimed in claim 2, wherein the control circuit performs the tracking processing of the subject of the first type using, out of the detection result regarding the subject of the first type and the detection result regarding the subject of the second type (as claimed in claim 1), a detection result that is detected at a position closer to the image center (Midorikawa paragraph 0135: “motion vectors belonging to the main subject candidate cluster are motion vectors in a direction approaching the image center”).
Claim(s) 9-10 are rejected under 35 U.S.C. 103 as being unpatentable over Sengupta and Yoneyama in view of Kunishige (US 8890993).
Regarding claim 9, Sengupta, modified by Yoneyama, discloses the image processing apparatus claimed in claim 8, wherein the control circuit, if a same subject is detected as the transportation subject, and also as the human subject (Yoneyama paragraphs 0046, 0095: “the whole of a vehicle and the driver of a vehicle have been detected simultaneously as subjects by the subject detection circuit…next, detection result association processing is performed…if they are a body region and a parts region of the same subject they belong to the same subject, and the CPU 1301 performs association of the two”), and a pupil is detected as the human subject (Sengupta paragraph 0039: interest point detection of “visually significant points such as, e.g., the corners of the eyes”), determines that a detection result of the pupil is to be used in tracking processing of the transportation subject (Yoneyama paragraphs 0046, 0140: “when a specified subject is a wild bird, it is possible to simultaneously detect the whole body, head, and eyes of a wild bird…In this case when it is possible to detect the eyes of the wild bird interpolative tracking processing is performed based on position of the eyes”; This was provided as an example of the “priority order setting circuit that sets a priority order for subjects that will be targets of interpolative tracking”. It is reasonable in view of the combination of Sengupta and Yoneyama to apply the same priority in the case where the subject is a vehicle instead of a wild bird and the whole vehicle, driver, face, and pupil are detected as the “plurality of portions simultaneously”. Thus, similar to how the eyes of the wild bird can be used to track the bird, the pupil of the human subject can be used to track the vehicle). However, Sengupta fails to disclose a size of an area detected as the entirety of the transportation subject is a threshold value or more. In related art, Kunishige discloses a size of an area detected as the entirety of the transportation subject is a threshold value or more (Kunishige col 12 lines 11-39: “If the result of determination in step S21 is that the size of the face is larger than the specified value, organ detection is carried out [S23]. Here, the image processing section 111 detects organs such as eyes”). Kunishige recognizes the fact that accuracy of eye AF may be lowered in accordance with size (Kunishige col 13 lines 14-23). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Sengupta to incorporate the teachings of Kunishige to increase the accuracy of eye detection as well as to ensure the eye AF frame does not become too small (Kunishige col 9 lines 1-8).
Regarding claim 10, Sengupta, modified by Yoneyama and Kunishige, discloses the image processing apparatus claimed in claim 9, wherein the control circuit, if it is determined that the detection result of the pupil is to be used in tracking processing of the transportation subject, saves, out of the detection result regarding the subject of the first type and the detection result regarding the subject of the second type, one of detection results that are not regarding the pupil as a candidate to be used in the tracking processing (Yoneyama paragraphs 0046, 0115: “the CPU 1301 stores tracking information, for example, a face, body, brightness, or color, which are a tracking target, or feature point information etc. in memory” where “the feature detection circuit 1305 can detect a plurality of portions simultaneously, from a single image”).
Response to Arguments
Applicant's arguments with respect to independent claims 1 and 11-13 have been fully considered but they are not persuasive.
Regarding the argument that “although Sengupta may teach determining whether the operator's head re-appears within a specified number of frames, such determination in Sengupta is merely used to decide whether to continue or stop tracking the same target, rather than to initialize or redetermine a tracking target”, the decision to stop tracking the same target results in initializing or redetermining a tracking target. Sengupta teaches if a predetermined time period for re-entry expires before a re-entry has been detected (as determined in step 224), the operator's session may be closed, and the system may go back into its start mode, in which it waits for a new or the same authorized user to log on (Sengupta paragraph 0048). The user logging into the system triggers the image acquisition and the detection and tracking processing (Sengupta paragraph 0033). Thus, the decision to stop tracking the operator results in the system preparing to initialize or redetermine a tracking target. This is further depicted in FIG. 2 of Sengupta, in which if it has been determined that time has expired in step 224, the method goes back to step 202 where face detection is used to initialize the location of the operator’s face and head (Sengupta paragraph 0038).
Regarding the argument that “Sengupta does not teach or suggest a situation in which one type of subject is treated as another type of subject, for example, where a human subject is treated as a transportation subject, and under such conditions, performing initialization or re-determination of a tracking target based on detection frequency or elapsed-time conditions involving two different subject types”, Sengupta teaches using the head-shoulder or full-body to track the operator’s face (Sengupta paragraph 0044). Under such conditions, if the operator's head re-appears in the images within a specified number of frames, tracking continues. Otherwise, the operator is deemed to have left the scene (Sengupta paragraph 0044). As argued above, the decision that the operator has left the scene and to stop tracking the operator results in initializing or redetermining a tracking target.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHRISTINE ZHAO whose telephone number is (703)756-5986. The examiner can normally be reached Monday - Friday 9:00am - 5:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached on (571)270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/C.Z./Examiner, Art Unit 2677
/ANDREW W BEE/Supervisory Patent Examiner, Art Unit 2677