Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: a registration unit, an acquisition unit. a generation unit and a detection unit in claims 1, 2, 6 and 8.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
The claimed invention is directed to non-statutory subject matter. Claim 8 is directed to a program, and programs per se are not patent eligible subject matter. See MPEP 2106 I: “Non-limiting examples of claims that are not directed to one of the statutory categories:
vi. a computer program per se, Gottschalk v. Benson, 409 U.S. at 72.”
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-3 and 6-8 is/are rejected under 35 U.S.C. 103 as being unpatentable over JP 2017-069748 A to Sakaguchi et al., hereinafter, “Sakaguchi” in view of First Order Motion Model for Image Animation to Siarohin et al., hereinafter, “Siarohin”.
Claim 1. An information processing device comprising: a registration unit that registers initial data indicating an initial state of a monitoring target; Sakaguchi [0043] When the motion of the human skeleton reproduced using the skeleton dataset 123 is overlaid on the still image 122a of the monitoring area (overlaid understood to be the equivalent of registration) including the money handling device 1 (see FIG. 8), the positional relationship between the motion of the human skeleton including the joints 51 and the connection lines 52 and the money handling device 1 can be confirmed.
Sakaguchi [0044] If it is attempted to record all the movements of the person in the monitoring area as the moving image, the capacity of the storage unit 120 required for storing the moving image data increases, but in the monitoring camera system, since the movement of the person is stored as the skeleton data 123 in the text format, the data capacity can be reduced. In addition, when the movement of a person is recorded as a human skeleton, it is not possible to record the face or clothes, but the image data 122 includes a still image obtained by imaging the face or the like in order to specify the person, and thus it is also possible to specify the person who performs the movement indicated by the human skeleton by using the still image.
Sakaguchi [0065] When controller 110 detects a person who has entered the monitoring area in the image of the time t1 (understood to be an initial state of the target) output from monitoring camera 40, controller 130 stores background image t0, which is a still image obtained by imaging the monitoring area at the time 122d before the detection (understood to be an initial state of the area), in storage 120 as image 122.
an acquisition unit that acquires feature point information indicating a feature point of the monitoring target detected on a time series basis; Sakaguchi [0039-0042] teaches the feature point information (coordinates of joints)
Sakaguchi [0043] When the motion of the human skeleton reproduced using the skeleton dataset 123 is overlaid on the still image 122a of the monitoring area including the money handling device 1 (see FIG. 8), the positional relationship between the motion of the human skeleton including the joints 51 and the connection lines (understood to be feature point information) 52 and the money handling device 1 can be confirmed.
Sakaguchi [0082] For example, in the example of FIG. 7, when the control unit 110 refers to the skeleton database 123 and finds the database 123a in which "T12" (time series basis) is recorded in the item of the trigger occurrence status, the control unit 130 recognizes that the skeleton database 124f (123b), the person image 124d (122e), and the moving image () are associated as the specific event detection database 124. 124e 123gThen, as shown in FIG. 8, the control unit 110 displays the skeleton image 124f of the specific event detection image 124 in a frame 404 on the right side of the screen as a retrieval result, and displays the person image 124d in a frame 405.
Sakaguchi [0106] Further, when the stored data is confirmed, the positions of the respective joints stored in the skeleton data are displayed in time series on the background image obtained by imaging the monitoring area, so that the movement of the person in the monitoring area can be reproduced like a moving image by the human skeleton. The movement of the person is displayed by the skeleton, and the face and the appearance of the person who has moved can be specified by the still image stored together with the skeleton data.
and a generation unit that, on the basis of the initial data and the feature point information, generates image information of the monitoring target that is in agreement with a predetermined condition. Sakaguchi [0047] Furthermore, the control unit 110 manages a plurality of pieces of data acquired when the trigger setting 121 is satisfied, that is, when the specific event is detected, in association with each other as specific event detection data 124. To be specific, as shown in FIG. 3, a moving image 124c obtained by imaging a specific event, a still image 124c obtained by imaging the face or the like of a person captured in the moving image 124a, and a skeleton 124c obtained by recording the motion of the person captured in the moving image 124b as the motion of the human skeleton are associated with each other to form the specific event detection date 124…The skeleton data-item 124b included in the specific event detection data-item 124 is obtained by extracting a data-item corresponding to a motion recorded in the moving image 124c from the skeleton data-item 123, and details thereof will be described later. Per specification [0051] Further, the information processing device 20 acquires key point data detected on a time series basis in the sensor device 10 (step S2-2). Then, on the basis of the acquired key point data, the information processing device 20 executes processing of detecting an abnormal state of the monitoring target (an example of a “predetermined condition”) (step S2-3). Any method can be employed as a method for detecting an abnormal state of the monitoring target. For example, the information processing device 20 may save reference key point data indicating a normal state for each monitoring target in advance, and may detect an abnormal state from a result of comparison between the reference key point data and key point data acquired from the sensor device 10.
Sakaguchi [0082] For example, in the example of FIG. 7, when the control unit 110 refers to the skeleton database 123 and finds the database 123a in which "T12" is recorded in the item of the trigger occurrence status, the control unit 130 recognizes that the skeleton database 124f (123b), the person image 124d (122e), and the moving image () are associated as the specific event detection database 124 (in view of specification [0051], the specific event detection is interpreted to be either a normal state and/or abnormal state – predetermined condition). 124e 123gThen, as shown in FIG. 8, the control unit 110 displays the skeleton image 124f of the specific event detection image 124 in a frame 404 on the right side of the screen as a retrieval result, and displays the person image 124d in a frame 405.
Sakaguchi [0106] Further, when the stored data is confirmed, the positions of the respective joints stored in the skeleton data are displayed in time series on the background image obtained by imaging the monitoring area, so that the movement of the person in the monitoring area can be reproduced like a moving image by the human skeleton. The movement of the person is displayed (output – generate) by the skeleton, and the face and the appearance of the person who has moved can be specified by the still image stored together with the skeleton data
Sakaguchi fails to explicitly teach a generation unit that, on the basis of the initial data and the feature point information Siarohin, in the same field of generating a video sequence (image data) of an object in motion, teaches [page 3] Figure 2: Overview of our approach. Our method assumes a source image S and a frame of a driving video frame D as inputs. The unsupervised keypoint detector extracts first order motion representation consisting of sparse keypoints(understood to be feature points) and local affine transformations with respect to the reference frame R (understood to be basis of the estimated amount of movement and a predetermined number of pieces of image data included in initial data). The dense motion network uses the motion representation to generate dense optical flow Tˆ S←D from D to S and occlusion map Oˆ S←D. The source image and the outputs of the dense motion network are used by the generator to render the target image.
Siarohin [Abstract] A generator network models occlusions arising during target motions and combines the appearance extracted from the source image and the motion derived from the driving video.
Thus, before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of Sakaguchi with the teachings of Siarohin [Introduction] for keypoints detection about a monitored person to improve the estimation of local affine transformations.
Claim 2. Sakaguchi and Siarohin further teaches further comprising: a detection unit that detects an abnormal state of the monitoring target on the basis of the feature point information, Sakaguchi [0047] Furthermore, the control unit 110 manages a plurality of pieces of data acquired when the trigger setting 121 is satisfied, that is, when the specific event is detected, in association with each other as specific event detection data 124. To be specific, as shown in FIG. 3, a moving image 124c obtained by imaging a specific event, a still image 124c obtained by imaging the face or the like of a person captured in the moving image 124a, and a skeleton 124c obtained by recording the motion of the person captured in the moving image 124b as the motion of the human skeleton are associated with each other to form the specific event detection date 124…The skeleton data-item 124b included in the specific event detection data-item 124 is obtained by extracting a data-item corresponding to a motion recorded in the moving image 124c from the skeleton data-item 123, and details thereof will be described later. Per specification [0051] Further, the information processing device 20 acquires key point data detected on a time series basis in the sensor device 10 (step S2-2). Then, on the basis of the acquired key point data, the information processing device 20 executes processing of detecting an abnormal state of the monitoring target (an example of a “predetermined condition”) (step S2-3). Any method can be employed as a method for detecting an abnormal state of the monitoring target. For example, the information processing device 20 may save reference key point data indicating a normal state for each monitoring target in advance, and may detect an abnormal state from a result of comparison between the reference key point data and key point data acquired from the sensor device 10.
The skeleton data (joints) is interprets to be feature point information.
Sakaguchi [0105] according to the monitoring camera system of the present embodiment, the inside of the monitoring area is monitored while being imaged by the monitoring camera, and in a case where a person is detected in the monitoring area, the movement of the person (understood to be an abnormal state) is regarded as the movement of the humanoid skeleton including a plurality of joints (understood to be feature points), and the change in the three dimensional coordinates of each joint (understood to be feature point information) is represented by the time series
wherein the generation unit uses the feature point information to estimate an amount of movement of the feature point from an initial state to a time of detection of an abnormal state of the monitoring target, Sakaguchi [0105] according to the monitoring camera system of the present embodiment, the inside of the monitoring area is monitored while being imaged by the monitoring camera, and in a case where a person is detected in the monitoring area, the movement of the person is regarded as the movement of the humanoid skeleton including a plurality of joints(understood to be feature points), and the change in the three dimensional coordinates of each joint is represented by the time series (understood to be image information corresponding to the time of detection)
Sakaguchi [0065] When controller 110 detects a person who has entered the monitoring area in the image of the time t1 (understood to be an initial state of the target) output from monitoring camera 40, controller 130 stores background image t0, which is a still image obtained by imaging the monitoring area at the time 122d before the detection (understood to be an initial state of the area), in storage 120 as image 122.
and generates the image information corresponding to the time of detection of the abnormal state on the basis of the estimated amount of movement and a predetermined number of pieces of image data included in initial data initially registered for the monitoring target. Sakaguchi [0105] according to the monitoring camera system of the present embodiment, the inside of the monitoring area is monitored while being imaged by the monitoring camera, and in a case where a person is detected in the monitoring area, the movement of the person is regarded as the movement of the humanoid skeleton including a plurality of joints, and the change in the three dimensional coordinates of each joint is represented by the time series (the change and time series data is understood to be the estimated amount of movement)
Sakaguchi [0065] When controller 110 detects a person who has entered the monitoring area in the image of the time t1 (understood to be an initial state of the target) output from monitoring camera 40, controller 130 stores background image t0, which is a still image obtained by imaging the monitoring area at the time 122d before the detection (understood to be an initial state of the area), in storage 120 as image 122.
Siarohin [page 3] Figure 2: Overview of our approach. Our method assumes a source image S and a frame of a driving video frame D as inputs. The unsupervised keypoint detector extracts first order motion representation consisting of sparse keypoints(understood to be feature points) and local affine transformations with respect to the reference frame R (understood to be basis of the estimated amount of movement and a predetermined number of pieces of image data included in initial data). The dense motion network uses the motion representation to generate dense optical flow Tˆ S←D from D to S and occlusion map Oˆ S←D. The source image and the outputs of the dense motion network are used by the generator to render the target image.
Claim 3. Sakaguchi further teaches wherein the feature point information includes skeleton information for specifying an attitude of the monitoring target. Sakaguchi [0008] Further, in the present invention according to the above-mentioned invention, the human body information includes data in a text format in which a motion of a person is regarded as a motion of a humanoid skeleton including a plurality of joints, and three dimensional coordinates of each joint are stored in time series.
Per specification [0037] The sensor device 10 executes key point detection (also referred to as “attitude estimation”) of detecting, from image data acquired for a monitoring target, a key point (an example of a coordinate point or “feature point information”) that can be a feature point of the monitoring target.
Claim 6. Siarohin further teaches wherein the registration unit executes processing for concealing at least part of the initial data. Siarohin [page 2] we introduce an occlusion-aware generator, which adopts an occlusion mask (understood to be concealing) automatically estimated to indicate object parts that are not visible in the source image and that should be inferred from the context… However, we model object motion in the neighbourhood of each predicted keypoint by a local affine transformation. Additionally, we explicitly model occlusions in order to indicate to the generator network the image regions that can be generated by warping the source image and the occluded areas that need to be inpainted.
Claim 7. Reviewed and analyzed in the same way as claim 1. See the above analysis and rationale.
Claim 8. Reviewed and analyzed in the same way as claim 1. See the above analysis and rationale.
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over JP 2017-069748 A to Sakaguchi et al., hereinafter, “Sakaguchi” in view of First Order Motion Model for Image Animation to Siarohin et al., hereinafter, “Siarohin” and in further view of JP09270014 A to Araki.
Claim 4. Sakaguchi and Siarohin are silent on claim 4, however, Araki, is in the same field of detecting a moving body in image data, teaches wherein the feature point information includes information based on a temperature heat map of the monitoring target. Araki [0073] Next, a third embodiment of the contour extracting apparatus of the present invention will be described. In the first and second embodiments, it has been described that a plurality of moving objects can be extracted and tracked using one type of image. Furthermore, the third embodiment integrates different types of image information such as a thermal image or a visible image when accurate extraction / tracking cannot be performed with one type of image due to information loss, noise, etc. It considers the extraction and tracking of moving objects
Araki [0074] An infrared camera 18, a half mirror 19, and a mirror 20 for picking up a thermal image having the same field of view as the visible image are added, and further, the image storage unit 2 in the first embodiment is provided with different image information of the thermal image and the visible image. Is replaced with a heterogeneous image storage unit 21 for storing the same, and the same motion detection unit 3 is replaced with a heterogeneous image information integrated motion detection unit 22 that detects motion information from a thermal image of an infrared camera and a visible image of a CCD camera, and the same feature extraction is performed…
Thus, before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of Sakaguchi with the teachings of Araki [Problem to be Solved] to correctly extract moving bodies to acquire accurate movement information.
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over JP 2017-069748 A to Sakaguchi et al., hereinafter, “Sakaguchi” in view of First Order Motion Model for Image Animation to Siarohin et al., hereinafter, “Siarohin” and in further view of US 2002/0075958 A1 to Shiro.
Claim 5. Sakaguchi and Siarohin are silent on claim 5, however, Shiro, is in the same field of detecting a moving body in image data, teaches wherein the feature point information includes traffic line information for specifying a traffic line of the monitoring target. Shiro [0021] This is a technique of tracking, as in broadcasting a ball game where only the motions of players (target objects) change against a constant background, the feature vectors (positions, sizes, shapes, textures, or the like) of the players to code the tracks of the target objects.
Thus, before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of Sakaguchi with the teachings of Shiro [0058] to optimize image quality and improve compression efficiency of a moving image.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DELOMIA L GILLIARD whose telephone number is (571)272-1681. The examiner can normally be reached 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John Villecco can be reached at (571) 272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DELOMIA L GILLIARD/Primary Examiner, Art Unit 2661