DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to claim 1 and 12 submitted 29 May 2026 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. The arguments focus on the amended features including “preparatory phase” which are met by the newly applied Chen (US 20220230438 A1) reference below.
Claim Objection
Claim 5 is objected to because of the following informalities:
Claim 5 recites “wherein the reliability corresponds to a degree of possibility of being a main object based on the likelihood that the objects is performing the predetermined action”. The highlighted portion indicates an apparent grammatical error.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claim 1, 2, 5, 7, 10, 12, and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Kosiakova (US 20210097291 A1), Shimauchi (US 20230124466 A1), and Chen (US 20220230438 A1)
Claim 1
In regards to claim 1, Kosiakova discloses a main object determination apparatus comprising:
one or more processors; and a memory storing instructions which, when executed by the one or more processors {Figs. 1A, 11 including server 108, [0043] processors 1114, instructions 1124 stored in memory 1112 executed by processor [0096]-[00104]}, cause the main object determination apparatus to function as:
an acquisition unit configured to acquire images captured at different timings {Figs. 1 including video camera; , [0001], [0022], [0033] acquiring successive frames of video images (images captured at different timings), Fig. 3 step 302 receive video [0051]};
an object detecting unit configured to detect a plurality of objects from the images {See Fig. 2 main object component 214, [0042]-[0046], [0052]-[0055], Fig. 4, determine main object in frame 406 [0062]-[0068] including real-time object detection (e.g. YOLO or SDD) of plural objects};
a selection unit configured to select at least one candidate of a main object from among the plurality of objects
{See Fig. 2 main object component 214, [0042]-[0046], Fig. 4, determine main object in frame 406 [0062]-[0068] including automatic (main) object detection from video clips using main object features which include colors, materials, and/or object associated with a main object}; and
a determination unit configured to determine whether at least one candidate of the main object selected by the selection unit is the same in the images captured at different timings,
{see [0044], [0051]-[0053], [0056]-[0057??], [0063], [0091]-[0093] in which the appearance time of objects frame-to-frame in a scene can be used by the main object component 214 to identify and distinguish main objects. In other words, per [0042]-[0046], main object component 214 identifies a main object in a frame of the video using automatic identification including using feature analysis (information about a feature point) while also determining whether this main object candidate persists or is otherwise the same from frame-to-frame in the video.},
wherein the main object is determined in a case where the determination unit determines that that least one candidate of the main object selected by the selection unit in the image of interest and in the least one image captured within a predetermined period of time before the image of interest is captured is the same,
{Kosiakova analyzes each of these time-sequential frames to determine whether the candidate object is the same from once frame to the next and such that the main object component 214 determines that at least one candidate of the main object selected by the selection unit in an image of interest and in at least one image captured within a predetermined period of time before the image of interest is captured are the same
See in particular [0044], [0052], [0091] in which the appearance of an object frame-to-frame in a scene can be used to distinguish the main object from background objects (other candidate objects that are not the main object) as per the determination
Further as to “image captured within a predetermined period of time before the image of interest is captured” note that this phrase merely describes a conventional video captured at a framerate (e.g. 30fps or 60 fps are common) such that each frame is captured within a predetermined period of time (e.g. 1/30sec or 1/60 sec) before the image of interest is captured. See also [0051], [0056], [0060]}
Shimauchi is analogous art from the same field of main object (main subject) determination including video cameras 100 and control unit with posture estimation 331, tracking 332 and action recognition 333, Figs. 1, 2, 6, 11 and cites below.
Shimauchi teaches
an acquisition unit configured to acquire images captured at different timings {see Fig 2. Image capturing apparatus 100, [0075]-[0076] capturing video (images at different timings):
an object detecting unit configured to detect a plurality of objects from the images; an obtaining unit configured to obtain posture information about each of the plurality of objects by obtaining positions of a plurality of joints of each of the plurality of objects
{Fig. 2, image processing apparatus 300 detects objects (subjects 10, 30) and posture estimation unit 331 estimates the postures of the objects. In more detail, see Figs. 2-4, [0089] in which the captured image includes the objects (main subject 10 and secondary subjects 30) both of which are main object candidates and may select the main object based on a facial feature match as per [0096]. Moreover, posture estimation 331 estimates positions of the joints of the object as per [0090]-[0096] and tracking unit 332 tracks the main subject based on the posture information (including joint position)};
a selection unit configured to select at least one candidate of a main object from among the plurality of objects on the basis of the posture information about the plurality objects for each image acquired by the acquisition unit
{See the tracking unit 332 tracking the objects 10, 30 across multiple frames of the captured video and such tracking of the objects across frames (images captured at different timings) is also used to individually identify/select the main object from among the other objects, [0090]-[0092], [0095]. Shimauchi also teaches that posture-based and color-based tracking are interchangeable equivalents for distinguishing between main and secondary objects as per [0095]-[0097]. Moreover, Shimauchi establishes that object posture and/or color features may be recognized and used to distinguish between main object and secondary object.
In addition, once the main object is selected Shimauchi adds an attribute/metadata item as per [0096] that selects the person determined to be the lecturer (main object). Moreover, once the main object (lecturer) is selected then that main object’s action is recognized by action recognition unit 333 to determine downstream actions (e.g. cropping and/or display screen layout based on main object’s actions as per [0097]-[0104], [0108]-[0115], Figs. 7A, 7B, 13, 16 and their corresponding disclosure sections) including [0177]-[0188]},
wherein the determination unit determines whether the at least one candidate of the main object is the same or not between the image of interest and the at least one image captured within the predetermined period of time before the image of interest is captured, based on a skeletal structure represented by a positional relationship among the joints
{see Fig. 4, [0090]-[0092], [0095] including tracking unit 332 that determines/tracks the main object 10 on the basis of color (e.g. of clothes) or by using only the posture estimation information as the main object moves from frame to frame (between the image of interest and the at least one image captured within the predetermined period of time before the image of interest is captured)},
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Kosiakova main object selection system which already discloses an object detecting unit configured to detect a plurality of objects from the images and a selection unit configured to select at least one candidate of a main object from among the plurality of objects such that the Kosiakova’s main object selection method includes an obtaining unit configured to obtain posture information about each of the plurality of objects by obtaining positions of a plurality of joints of each of the plurality of objects and wherein Kosiakova’s selection unit selects at least one candidate of a main object from among the plurality of objects on the basis of the posture information about the plurality objects for each image acquired by the acquisition unit as also taught by Shimauchi and wherein the determination unit determines whether the at least one candidate of the main object is the same or not between the image of interest and the at least one image captured within the predetermined period of time before the image of interest is captured, based on a skeletal structure represented by a positional relationship among the joints as further taught by Shimauchi because joint position enables more accurate determination of the main subject and permits enhanced layout decisions based on a determination that the main subject is, for example, walking as motivated by Shimauchi in [0108]-[0115], [0097]-[0104], Figs. 7A, 7B, 13, 16 and their corresponding disclosure sections) including [0177]-[0188]}, because Shimauchi teaches the equivalence of using color or posture to track/determine main object in [0095] such that Kosiaka’s color-based main object determination in [0053]-[0055] may be equivalently replaced by Shimauchi’s posture-based main object determination, because there is a reasonable expectation of success and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Chen analogous art from the same field of main object (main subject) determination including video cameras and control unit with posture estimation. See Fig. 3A, 3C, 4C, 5C, 6H, [0002]-[0013].
Chen also teaches
control unit configured to perform, in response to determining the main object. at least one of autofocus control, exposure control, and recording control with respect to the determined main object {see [0162]-[0170] in which slow-motion video is triggered in response to the determining the main object exhibiting a target posture in order to capture important events such as basketball shooting}
wherein the selection unit calculates, for each of the plurality of objects, a reliability indicating a likelihood that the object is performing a preparatory state of a predetermined action based on the posture information, and selects the at least one candidate of the main object based on the reliability {see [0162]-[0171] in which uses a posture classification network to determine a likelihood that the object is performing a preparatory state (target posture such as prepare to shoot a basketball) based on the posture information and selects the candidate as the main object based on this target posture determination}, and
where the reliability satisfies a predetermined preparatory action status constraint indicative of the preparatory state of the predetermined action in the at least one image captured within the predetermined period of time before the image of interest is captured to enable the control unit to execute the at least one of the autofocus control, the exposure control, and the recording control a predetermined delay time prior to an occurrence of the predetermined action
{see [0162]-[0171]. Chen does not mention reliability per se but the claims broadly recite “reliability” such that a 100% or other substantial determination of the preparatory state satisfies these claim limitations related to “reliability” which are broadly recited as “reliability satisfies a predetermined preparatory action status constraint indicative of the preparatory state” which corresponds to Chen’s determination of a target posture using a posture classification network in which the constraint is satisfaction of plural video frames having key skeletal posture points that satisfy the posture classification network as corresponding to the target posture.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Kosiakova main object selection system which already discloses an object detecting unit configured to detect a plurality of objects from the images and a selection unit configured to select at least one candidate of a main object from among the plurality of objects such that the Kosiakova’s main object selection method/apparatus includes a control unit configured to perform, in response to determining the main object. at least one of autofocus control, exposure control, and recording control with respect to the determined main object, such that wherein Kosiakova’s selection unit also calculates, for each of the plurality of objects, a reliability indicating a likelihood that the object is performing a preparatory state of a predetermined action based on the posture information, and selects the at least one candidate of the main object based on the reliability, where the reliability satisfies a predetermined preparatory action status constraint indicative of the preparatory state of the predetermined action in the at least one image captured within the predetermined period of time before the image of interest is captured to enable the control unit to execute the at least one of the autofocus control, the exposure control, and the recording control a predetermined delay time prior to an occurrence of the predetermined action as taught by Chen because doing so advantageously provides for detecting when a main object makes a (reliable) target posture such as leaping up, basketball shooting, skateboard jump, etc. to thereby trigger slow-motion video recording of the most interesting and important sport scenes (e.g. for a sports highlight broadcast or replay) as motivated by Chen in [0162]-[0171]; because there is a reasonable expectation of success and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claim 2
In regards to claim 2, Kosiakova discloses wherein the object refers to a person or an animal {main object component 214 identifies the main person (human) in the video frame as per [0044]-[0046], [0052]-[0054]}.
Claim 5
In regards to claim 5, Kosiakova discloses wherein the reliability corresponds to a degree of possibility of being a main object
Chen also teaches wherein the reliability corresponds to a degree of possibility of being a main object based on the likelihood that the objects is performing the predetermined action {see [0162]-[0171]. Chen does not mention reliability per se but the claims broadly cite “reliability” such that a 100% or other substantial determination of the preparatory state satisfies these claim limitations related to “reliability” which are broadly recited as “reliability corresponds to a degree of possibility of being a main object based on the likelihood that the objects is performing the predetermined action” which corresponds to Chen’s determination of a target posture using a posture classification network in which the constraint is satisfaction of plural video frames having key skeletal posture points that satisfy the posture classification network as corresponding to the target posture of the main object
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Kosiakova main object selection system which already discloses an object detecting unit configured to detect a plurality of objects from the images and a selection unit configured to select at least one candidate of a main object from among the plurality of objects such that the Kosiakova’s main object selection method/apparatus includes a control unit configured to perform, in response to determining the main object. at least one of autofocus control, exposure control, and recording control with respect to the determined main object, such that wherein Kosiakova’s selection unit also calculates, for each of the plurality of objects, a reliability indicating a likelihood that the object is performing a preparatory state of a predetermined action based on the posture information, and selects the at least one candidate of the main object based on the reliability, where the reliability satisfies a predetermined preparatory action status constraint indicative of the preparatory state of the predetermined action in the at least one image captured within the predetermined period of time before the image of interest is captured to enable the control unit to execute the at least one of the autofocus control, the exposure control, and the recording control a predetermined delay time prior to an occurrence of the predetermined action as taught by Chen and such that wherein the reliability corresponds to a degree of possibility of being a main object based on the likelihood that the objects is performing the predetermined action as also taught by Chen because doing so advantageously provides for detecting when a main object makes a (reliable) target posture such as leaping up, basketball shooting, skateboard jump, etc. to thereby trigger slow-motion video recording of the most interesting and important sport scenes (e.g. for a sports highlight broadcast or replay) as motivated by Chen in [0162]-[0171]; because there is a reasonable expectation of success and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claim 7
In regards to claim 7, Kosiakova discloses wherein the selection unit selects an object having a maximum value of the reliability as at least one candidate of the main object {the automatic identification used by main object component 214 includes associated confidence levels (reliability) indicating probability of correct identification and which the best match/maximum reliability is determined to be the main object as per [0044]-[0045], [0052]-[0053], [0066]-[0067], [0079], [0091]-[0092]}.
Claim 10
In regards to claim 10, Kosiakova discloses wherein the selection unit does not select the candidate of the main object from an image that is not captured within the predetermined period of time before the image of interest is captured {see [0056] in which the video clips that are analyzed for main object determination are limited to one or three frames from each of the first and second video clips. As such frames not captured within a predetermined period of time before the image of interest is captured (e.g. the fourth prior frame) are not used for main object determination and thus the selection unit does not select the candidate from such old frames}.
Claims 12 and 13
The rejection of apparatus claims 1 above applies mutatis mutandis to the corresponding limitations of method claim 12 and computer readable medium claim 13 while noting that the rejection above cites to device, method, and computer readable medium disclosures.
Claims 4, 6, 8, 9, and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Kosiakova, Shimauchi, and Chen as applied to claim 1 above, and further in view of Sugawara (JP 2018066889 A). A marked-up machine translation of Sugawara has been previously provided, all cross-references are with respect to this translation and the mark-ups are hereby incorporated by reference to further demonstrate claim mapping.
Claim 4
In regards to claim 4, Kosiakova discloses wherein object detecting unit detects
a center of gravity of the corresponding object of the plurality of objects, position information indicating a part of a body of a corresponding object of the plurality of objects, or a position or motion vector of a corresponding object of the plurality of objects
{see [0054]-[0055] in which the feature amount is position indicating an associated part of the body of the object (e.g. red plaid shirt, baseball hat, glasses, brown jacket, white shoes. See also block 308 determine continuity of main object. See also [0062] in which the main object region of interest is propagated across additional frames for identifying the main object which likely includes position information of the object but position information is not explicitly disclosed}.
Sugawara is a highly related and analogous reference from the same field of main object determination and solves a similar problem of how to reliably determine the main object. See abstract, field, pgs. 3 in which recognition unit 111 of the camera’s control device 110 tracks a main object (specific target) from a video sequence of plural time-sequential images.
Sugawara also teaches wherein object detecting unit detects a center of gravity of the corresponding object of the plurality of objects, position information indicating a part of a body of a corresponding object of the plurality of objects, or a position or motion vector of a corresponding object of the plurality of objects
{pgs. 9-10 Modification of first embodiment not only uses distance to main object (soccer ball) but also the moving direction (motion vector) of the candidate to determine main object (e.g. closest detected face moving in the same direction as the ball 151 is the main object. See also Modification 2 which also considers position (depth) relative to the important object to determine main object in pgs. 10-11}.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Kosiakova which already discloses a related concept of determining feature weight using a variety of factors including distance to the main object such that the wherein the information about the feature amount is a center of gravity of the corresponding object of the plurality of objects, position information indicating a part of a body of a corresponding object of the plurality of objects, or a position or motion vector of a corresponding object of the plurality of objects as taught by Sugawara because doing so advantageously focusses the camera on the main object (soccer player) based on which player is closest to the ball and moving in the same direction thus permitting viewers to most reliably watch the most relevant player in control of the ball (closest thereto and moving in same direction) as the main object of the camera’s focus as motivated by Sugawara.
Claim 6
In regards to claim 6, Kosiakova discloses wherein each of the plurality of objects refers to a person, and wherein the selection unit uses a distance between each person
Sugawara also teaches discloses wherein each of the plurality of objects refers to a person, and wherein the selection unit uses a distance between each person
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Kosiakova which already discloses a related concept of determining feature weight using a variety of factors including distance to the main object such that wherein each of the plurality of objects refers to a person, and wherein the selection unit uses a distance between each person and a ball to calculate the reliability as taught by Sugawara because doing so advantageously focusses the camera on the main object (soccer player) based on which player is closest to the ball thus permitting viewers to most reliably watch the player in control of the ball (closest thereto) as the main object of the camera’s focus as motivated by Sugawara.
Claim 8
In regards to claim 8, Kosiakova is not relied upon to disclose but Sugiwara teaches wherein the selection unit also selects an object having a value of the reliability different from the maximum value of the reliability by less than a predetermined value as at least one candidate of the main object {pgs. 9-10 Modification of first embodiment not only uses distance to main object (soccer ball) but also the moving direction (motion vector) of the candidate to determine main object (e.g. closest detected face that is moving in the same direction as the ball 151 is the main object. See also Modification 2 which also considers position (depth) relative to the important object to determine main object in pgs. 10-11. In both cases, the method select an object having a value of reliability different from the maximum (not the closest player) by less than a predetermined value (not closest but within the detection range) when the player object is moving in the same direction as the important object or closest in depth direction}
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Kosiakova which already discloses a related concept of determining feature weight weight using a variety of factors including distance to the main object such that wherein the selection unit also selects an object having a value of the reliability different from the maximum value of the reliability by less than a predetermined value as the candidate of the main object as taught by Sugawara because doing so advantageously focusses the camera on the main object (soccer player) based on which player is closest to the ball and moving in the same direction thus permitting viewers to most reliably watch the most relevant player in control of the ball (closest thereto and moving in same direction or closest in depth direction) as the main object of the camera’s focus as motivated by Sugawara.
Claim 9
In regards to claim 9, Kosiakova is not relied upon to disclose but Sugiwara teaches a tracking unit configured to track the objects, wherein, in a case where the determination unit determines that at least one candidate of the main object are the same, the tracking unit changes a tracking target in the image of interest to the main object {see pgs. 1-6 in which control device detects and tracks a tracking target which may initially be the ball but which is changes to the main subject when the main subject detection unit 112 detects which player is closest to the ball within a detection range. Also Focus adjustment unit 114 control the focus state to focus on the main subject detected by detection unit 112 and tracked by control unit}.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Kosiakova to include a tracking unit configured to track the objects, wherein, in a case where the determination unit determines that the candidates of the main object are the same, the tracking unit changes a tracking target in the image of interest to the main object as taught by Sugawara because doing so advantageously tracks and focusses the camera on the main object (soccer player) based on which player is closest to the ball thus permitting viewers to most reliably watch the player in control of the ball (closest thereto) as the main object of the camera’s focus as motivated by Sugawara.
Claim 11
In regards to claim 8, Kosiakova
the main object determination apparatus according to claim 1 {see above mapping of claim 1}.
Sugawara teaches an image capturing unit configured to capture an object image formed via an imaging optical system {Fig. 1 camera 100, pg. 2}
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Kosiakova to include an image capturing unit configured to capture an object image formed via an imaging optical system as taught by Sugawara because doing so enables adaptive tracking of the main object as motivated by Sugawara.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20220230438 A1 discloses main object detection and employs posture and action classification (e.g. basketball shooting) of main object to trigger camera operations such as slow motion video start as per [0162]-[0171].
Wagg (US 2009/0059007 A1 a main object determination apparatus comprising:
one or more processors; and a memory storing instructions which, when executed by the one or more processors {Fig. 1 content processing workstation 10 including a Cell processor per [0033], , cause the main object determination apparatus to function as:
an acquisition unit configured to acquire images captured at different timings {Fig. 1 including video camera(s) 20, 22.1, 22.2; , [0031], [0036] acquiring successive frames of video images (images captured at different timings)};
a selection unit configured to select a candidate of a main object from among objects using information about a feature point of each object in the images {See [0042] applying mean human shape to extract/select a candidate of a main object}
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michael R Cammarata whose telephone number is (571)272-0113. The examiner can normally be reached M-Th 7am-5pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at 571-272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL ROBERT CAMMARATA/Primary Examiner, Art Unit 2667
Prosecution Insights