DETAILED ACTION
This communication is in response to the Application filed on 11/19/2024. Claims 1-20 are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 13, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 01/08/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. JP 2022092808, filed on 06/08/2022.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-5 and 7-20 rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claims 1 and 15 recite An information creation method comprising: a first acquisition step of acquiring sound data including a plurality of sounds emitted from a plurality of sound sources; a setting step of setting accuracy for the sound source or the sound; and a creation step of creating information on characteristics of the sound as accessory information on video data corresponding to the sound data, based on the accuracy.
The limitations in these claims, as drafted, are a process that, under broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. A human is capable of hearing sound, setting an accuracy (Ex: how close it is to the desired sound), and writing down characteristics about the sound. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claims recite an abstract idea.
This judicial exception is not integrated into a practical application. Claim 15 specifically lists the additional component of a processor. The processor is merely used to apply the method via a computing device. The processor is described in paragraph 34 of the specification with a generic description of the component. Accordingly, the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claims are not patent eligible.
Claim 2 recites wherein, in the setting step, an importance level is set for the sound or the sound source, and the accuracy is set according to the importance level.
The limitations in this claim, as drafted, are a process that, under broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. A human is capable of assigning importance to a sound (Ex: fire alarm is more important than music playing). If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
This judicial exception is not integrated into a practical application. The claims do not recite additional components to integrate the judicial exception. Accordingly, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Claim 3 recites wherein the sound is a non-verbal sound.
The limitations in this claim, as drafted, are a process that, under broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. A human is capable understanding nonverbal sounds. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
This judicial exception is not integrated into a practical application. The claims do not recite additional components to integrate the judicial exception. Accordingly, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Claim 4 recites further comprising: a second acquisition step of acquiring video data including a plurality of image frames, wherein, in the setting step, the accuracy for the sound source is set according to whether or not the sound source is present within an angle of view of a corresponding image frame among the plurality of image frames.
The limitations in this claim, as drafted, are a process that, under broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. A human is capable of viewing images and identifying if a sound they heard came from something existing in the image. They could then place greater importance on sounds that were in the image. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
This judicial exception is not integrated into a practical application. The claims do not recite additional components to integrate the judicial exception. Accordingly, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Claim 5 recites further comprising: a determination step of determining whether or not the sound satisfies a predetermined criterion in a case in which the sound source is not present within an angle of view of a corresponding image frame, wherein, in the setting step, the accuracy for the sound is set to be higher in a case in which the sound satisfies the predetermined criterion than in a case in which the sound does not satisfy the predetermined criterion.
The limitations in this claim, as drafted, are a process that, under broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. A human is capable of identifying if a sound source is in an image. They could then place greater importance on sounds that were in the image. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
This judicial exception is not integrated into a practical application. The claims do not recite additional components to integrate the judicial exception. Accordingly, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Claim 7 recites wherein, in the setting step, the accuracy for the sound source is set based on a result of image recognition on the sound source in a corresponding image frame or apparatus information associated with the image frame for an imaging apparatus that captures the image frame.
The limitations in this claim, as drafted, are a process that, under broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. A human is capable of identifying if a sound source is in an image by recognizing objects within the image. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
This judicial exception is not integrated into a practical application. The claims do not recite additional components to integrate the judicial exception. Accordingly, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Claim 8 recites wherein, in the setting step, the accuracy for the sound source is set based on the apparatus information, and the apparatus information is information on a focal position of the imaging apparatus in the image frame or a gaze position of a user of the imaging apparatus in the image frame.
The limitations in this claim, as drafted, are a process that, under broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. A human is capable of identifying if a sound source is in view of a camera or lens and considering the sound of anything in that view more important. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
This judicial exception is not integrated into a practical application. The claims do not recite additional components to integrate the judicial exception. Accordingly, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Claim 9 recites wherein, in the creation step, information on whether or not the sound source is present within an angle of view of a corresponding image frame is created as the accessory information
The limitations in this claim, as drafted, are a process that, under broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. A human is capable of identifying if a sound source is in an image. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
This judicial exception is not integrated into a practical application. The claims do not recite additional components to integrate the judicial exception. Accordingly, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Claim 10 recites further comprising: an inspection step of inspecting whether or not the sound satisfies an inspection criterion in a case in which the accuracy according to the importance level satisfies a predetermined condition, wherein, in the creation step, information on an inspection result obtained in the inspection step is created as the accessory information.
The limitations in this claim, as drafted, are a process that, under broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. A human can hear a sound and understand if it’s worth inspecting based on their prior knowledge and can record the inspection that was done. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
This judicial exception is not integrated into a practical application. The claims do not recite additional components to integrate the judicial exception. Accordingly, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Claim 11 recites wherein, in the creation step, reliability information on reliability of the inspection result is further created as the accessory information.
The limitations in this claim, as drafted, are a process that, under broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. A human is capable of making estimation on the reliability of an inspection. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
This judicial exception is not integrated into a practical application. The claims do not recite additional components to integrate the judicial exception. Accordingly, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Claim 12 recites wherein, in the creation step, importance level information on the importance level is created as the accessory information.
The limitations in this claim, as drafted, are a process that, under broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. A human is capable of assigning documenting how important a detected sound is. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
This judicial exception is not integrated into a practical application. The claims do not recite additional components to integrate the judicial exception. Accordingly, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Claim 13 recites wherein, in a case in which the accuracy according to the importance level satisfies a first condition, in the creation step, onomatopoeic word information in which the sound is converted into text as an onomatopoeic word is created as the accessory information.
The limitations in this claim, as drafted, are a process that, under broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. A human is capable of recognizing an important sound and associating it with an onomatopoeic word. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
This judicial exception is not integrated into a practical application. The claims do not recite additional components to integrate the judicial exception. Accordingly, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Claim 14 recites wherein, in a case in which the accuracy according to the importance level satisfies a second condition, in the creation step, mimetic word information in which a state of the sound source in a corresponding image frame is converted into text as a mimetic word is further created as the accessory information.
The limitations in this claim, as drafted, are a process that, under broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. A human is capable of recognizing an important sound and associating it with an mimetic word. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
This judicial exception is not integrated into a practical application. The claims do not recite additional components to integrate the judicial exception. Accordingly, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Claim 16 recites wherein the sound is a non-verbal sound.
The limitations in this claim, as drafted, are a process that, under broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. A human is capable understanding nonverbal sounds. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
This judicial exception is not integrated into a practical application. The claims do not recite additional components to integrate the judicial exception. Accordingly, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Claim 17 recites further comprising: a second acquisition step of acquiring video data including a plurality of image frames, wherein, in the setting step, the accuracy for the sound source is set according to whether or not the sound source is present within an angle of view of a corresponding image frame among the plurality of image frames.
The limitations in this claim, as drafted, are a process that, under broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. A human is capable of viewing images and identifying if a sound they heard came from something existing in the image. They could then place greater importance on sounds that were in the image. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
This judicial exception is not integrated into a practical application. The claims do not recite additional components to integrate the judicial exception. Accordingly, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Claim 18 recites further comprising: a determination step of determining whether or not the sound satisfies a predetermined criterion in a case in which the sound source is not present within an angle of view of a corresponding image frame, wherein, in the setting step, the accuracy for the sound is set to be higher in a case in which the sound satisfies the predetermined criterion than in a case in which the sound does not satisfy the predetermined criterion.
The limitations in this claim, as drafted, are a process that, under broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. A human is capable of identifying if a sound source is in an image. They could then place greater importance on sounds that were in the image. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
This judicial exception is not integrated into a practical application. The claims do not recite additional components to integrate the judicial exception. Accordingly, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Claim 19 recites wherein, in the setting step, the accuracy for the sound source is set based on a result of image recognition on the sound source in a corresponding image frame or apparatus information associated with the image frame for an imaging apparatus that captures the image frame.
The limitations in this claim, as drafted, are a process that, under broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. A human is capable of identifying if a sound source is in an image by recognizing objects within the image. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
This judicial exception is not integrated into a practical application. The claims do not recite additional components to integrate the judicial exception. Accordingly, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Claim 20 recites wherein, in the creation step, information on whether or not the sound source is present within an angle of view of a corresponding image frame is created as the accessory information
The limitations in this claim, as drafted, are a process that, under broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. A human is capable of identifying if a sound source is in an image. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
This judicial exception is not integrated into a practical application. The claims do not recite additional components to integrate the judicial exception. Accordingly, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-3, 7-8, 10-12, 15-16, and 19 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by US Patent Publication US 20150092052 A1 (Shin et al.).
Regarding Claims 1 and 15, Shin et al. teaches An information creation method comprising:
(One or more embodiments of the inventive concept include an image monitoring system that analyzes an image and/or sound source to generate, store, and display metadata about an abnormal situation so as to increase the monitoring efficiency, a surveillance camera included in the image monitoring system, and a method of operating the surveillance camera.) (Paragraph 7).
Claim 15 presents the alternative limitation of An information creation device comprising: a processor, wherein the processor is configured to:
(One or more embodiments of the inventive concept include an image monitoring system that analyzes an image and/or sound source to generate, store, and display metadata about an abnormal situation so as to increase the monitoring efficiency, a surveillance camera included in the image monitoring system, and a method of operating the surveillance camera.) (Paragraph 7).
(The image analyzer 120 may also functionally perform color processing, blur processing, edge enhancement processing, image analysis processing, image recognition processing, image effect processing, or the like on the image signal output from the image sensor 110. The image analyzer 120 may perform face recognition, scene recognition, or the like through the image recognition processing. For example, the image analyzer 120 may perform luminance level adjusting, color correction, contrast adjusting, contour emphasis adjusting, screen division processing, character image generating, image synthesis process, or the like.) (Paragraph 40).
a first acquisition step of acquiring sound data including a plurality of sounds emitted from a plurality of sound sources;
(The sound source may be included in or obtained from an image captured by the surveillance camera 100 or may be directly input to the surveillance camera 100 through the microphone 200. The surveillance camera 100 will be described in detail with reference to FIG. 2.) (Paragraph 34).
(The types of the abnormal sound source and the abnormal image each may include a plurality of candidates and accuracy information about the candidates, and the determiner may determine whether the abnormal image and the abnormal sound source indicate the same event with reference to the plurality of candidates corresponding to the types of the abnormal image and the abnormal sound source and the accuracy information.) (Paragraph 14).
Plurality of sounds are recorded.
a setting step of setting accuracy for the sound source or the sound;
(The types of the abnormal sound source and the abnormal image each may include a plurality of candidates and accuracy information about the candidates, and the determiner may determine whether the abnormal image and the abnormal sound source indicate the same event with reference to the plurality of candidates corresponding to the types of the abnormal image and the abnormal sound source and the accuracy information.) (Paragraph 14).
(The determiner 140 may determine whether an abnormal image exists, an abnormal sound exists, or an abnormal image and an abnormal sound source simultaneously exist according to an image analysis result and/or a sound source analysis result. In particular, if it is determined that an abnormal image and an abnormal sound source simultaneously exist, the determiner 140 determines whether the abnormal sound source is obtained from the abnormal image or an outside of the abnormal image. Also, if the abnormal sound source is obtained from the abnormal image, the determiner 140 determines whether the abnormal image and the abnormal sound source indicate a same event. The determiner 140 performs multiple determinations as described above because a result of the generated metadata varies according to a determination result of the determiner 140.) (Paragraph 44).
An accuracy value is assigned to a determined sound source.
and a creation step of creating information on characteristics of the sound as accessory information on video data corresponding to the sound data, based on the accuracy.
(The metadata generator 150 generates metadata about the abnormal image, metadata about the abnormal sound source, and metadata about the abnormal image and the abnormal sound source with reference to the determination result of the determiner 140.) (Paragraph 45).
Metadata related to the sound corresponding to an image is created
Regarding Claim 2, Shin et al. teaches the method of claim 1.
Shin et al. further teaches wherein, in the setting step, an importance level is set for the sound or the sound source, and the accuracy is set according to the importance level.
(If a plurality of abnormal sound sources are simultaneously generated, tracking may be performed according to priorities with reference to metadata of the respective abnormal sound sources. For example, if a plurality of events simultaneously occur, tracking may be performed according to types and priorities of the events.) (Paragraph 56).
(The metadata refers to data that provides information about original data (an image, a sound source, or the image and the sound source) and describes another data. In the present exemplary embodiment, the metadata generator 150 generate metadata including a number of times of generating the abnormal image and/or the abnormal sound source, analysis results and types of the abnormal image and/or the abnormal sound source, a number of generated events indicated by the abnormal image and/or the abnormal sound source, whether the abnormal sound source is obtained from the abnormal image, and whether the abnormal image and the abnormal sound source indicate a same event with reference to the determination result of the determiner 140.) (Paragraph 45).
Priorities (importance level) are assigned based on various metadata factors for the sound source.
Regarding Claim 3, Shin et al. teaches the method of claim 1.
Shin et al. further teaches wherein the sound is a non-verbal sound.
(The event generation condition used in the sound source analysis corresponds to generation of an abnormal sound (a friction sound (skid) of a car tire, a glass breaking sound, an alarm sound, a collision sound, or the like), generation of a sound specified by the user (for example, a man's screaming, woman's screaming, baby's crying sound, or the like), generation of a sound with an amplitude higher than or equal to a threshold value, or the like, and may be preset.) (Paragraph 43).
The recorded sounds are listed to be non-verbal
Regarding Claim 7, Shin et al. teaches the method of claim 1.
Shin et al. further teaches wherein, in the setting step, the accuracy for the sound source is set based on a result of image recognition on the sound source in a corresponding image frame or apparatus information associated with the image frame for an imaging apparatus that captures the image frame.
(The image analyzer 120 may also functionally perform color processing, blur processing, edge enhancement processing, image analysis processing, image recognition processing, image effect processing, or the like on the image signal output from the image sensor 110.) (Paragraph 40).
(For example, with reference to the first metadata shown in FIG. 3B, a probability of the abnormal sound source being a collision and a probability of the abnormal image being an explosion are the highest, and thus, the types of the abnormal sound source and the abnormal image do not correspond to each other. However, if a sum of the probability (40%) of the abnormal sound source being the explosion and the probability (95%) of the abnormal image being the explosion exceeds a preset threshold value (for example, 130%), the result of the cross check may be determined as an explosion. For example, with reference to second metadata shown in FIG. 3B, if a sum of a probability (70%) of the abnormal sound source being man's screaming, a probability (30%) of the abnormal sound source being woman's screaming, and a probability (80%) of the abnormal sound source being a violent event exceeds a preset threshold value, the result of the cross check may be determined as being a violent event. In this case, a man's screaming and a woman's screaming may be regarded as a same type of events in order to sum the accuracies, acquire a maximum value, or acquire an average value in order to use values of the accuracies.) (Paragraph 49).
Image recognition is used to analyze the object in the image and then it is associated with the detected sound source to verify they are the same source.
Regarding Claim 8, Shin et al. teaches the method of claim 7.
Shin et al. further teaches wherein, in the setting step, the accuracy for the sound source is set based on the apparatus information,
(The image analyzer 120 may also functionally perform color processing, blur processing, edge enhancement processing, image analysis processing, image recognition processing, image effect processing, or the like on the image signal output from the image sensor 110.) (Paragraph 40).
(However, if a sum of the probability (40%) of the abnormal sound source being the explosion and the probability (95%) of the abnormal image being the explosion exceeds a preset threshold value (for example, 130%), the result of the cross check may be determined as an explosion.) (Paragraph 49).
Accuracy is set based on apparatus information (image recognition and sound source).
and the apparatus information is information on a focal position of the imaging apparatus in the image frame or a gaze position of a user of the imaging apparatus in the image frame.
(The line-of-sight image generation unit 230 includes an image sensor 150, shoots the left eye of a user wearing the audio information display device 100 on the head, and calculates a line-of-sight image for calculating the direction of the line of sight of the left eye. Generate. The field-of-view specifying unit 240 analyzes the line-of-sight image generated by the line-of-sight image generation unit 230, calculates the direction of the line of sight of the user's left eye, and uses the user's field of view, that is, the left eye rim in the eyeglass frame 180 The relative position of the person's viewing range is specified.) (Fig. 2 Description, Paragraphs 12-13).
Line-of-sight and field of view information is generated for image frame.
Regarding Claim 10, Shin et al. teaches the method of claim 2.
Shin et al. further teaches an inspection step of inspecting whether or not the sound satisfies an inspection criterion in a case in which the accuracy according to the importance level satisfies a predetermined condition,
(The metadata generator 150 generates metadata about the abnormal image, metadata about the abnormal sound source, and metadata about the abnormal image and the abnormal sound source with reference to the determination result of the determiner 140.) (Paragraph 45).
(For example, with reference to the first metadata shown in FIG. 3B, a probability of the abnormal sound source being a collision and a probability of the abnormal image being an explosion are the highest, and thus, the types of the abnormal sound source and the abnormal image do not correspond to each other. However, if a sum of the probability (40%) of the abnormal sound source being the explosion and the probability (95%) of the abnormal image being the explosion exceeds a preset threshold value (for example, 130%), the result of the cross check may be determined as an explosion. For example, with reference to second metadata shown in FIG. 3B, if a sum of a probability (70%) of the abnormal sound source being man's screaming, a probability (30%) of the abnormal sound source being woman's screaming, and a probability (80%) of the abnormal sound source being a violent event exceeds a preset threshold value, the result of the cross check may be determined as being a violent event. In this case, a man's screaming and a woman's screaming may be regarded as a same type of events in order to sum the accuracies, acquire a maximum value, or acquire an average value in order to use values of the accuracies.) (Paragraph 49).
An inspection step could be considering if an object detected in the an image is the same as a sound source recorded by the microphone.
wherein, in the creation step, information on an inspection result obtained in the inspection step is created as the accessory information.
(The metadata refers to data that provides information about original data (an image, a sound source, or the image and the sound source) and describes another data. In the present exemplary embodiment, the metadata generator 150 generate metadata including a number of times of generating the abnormal image and/or the abnormal sound source, analysis results and types of the abnormal image and/or the abnormal sound source, a number of generated events indicated by the abnormal image and/or the abnormal sound source, whether the abnormal sound source is obtained from the abnormal image, and whether the abnormal image and the abnormal sound source indicate a same event with reference to the determination result of the determiner 140.) (Paragraph 45).
This inspection step is recorded as metadata in the creation step.
Regarding Claim 11, Shin et al. teaches the method of claim 10.
Shin et al. further teaches wherein, in the creation step, reliability information on reliability of the inspection result is further created as the accessory information.
(For example, with reference to the first metadata shown in FIG. 3B, a probability of the abnormal sound source being a collision and a probability of the abnormal image being an explosion are the highest, and thus, the types of the abnormal sound source and the abnormal image do not correspond to each other. However, if a sum of the probability (40%) of the abnormal sound source being the explosion and the probability (95%) of the abnormal image being the explosion exceeds a preset threshold value (for example, 130%), the result of the cross check may be determined as an explosion. For example, with reference to second metadata shown in FIG. 3B, if a sum of a probability (70%) of the abnormal sound source being man's screaming, a probability (30%) of the abnormal sound source being woman's screaming, and a probability (80%) of the abnormal sound source being a violent event exceeds a preset threshold value, the result of the cross check may be determined as being a violent event. In this case, a man's screaming and a woman's screaming may be regarded as a same type of events in order to sum the accuracies, acquire a maximum value, or acquire an average value in order to use values of the accuracies.) (Paragraph 49).
Reliability information the form of different actions based on different probabilities during an inspection step is recorded as metadata.
Regarding Claim 12, Shin et al. teaches the method of claim 2.
Shin et al. further teaches wherein, in the creation step, importance level information on the importance level is created as the accessory information.
(If a plurality of abnormal sound sources are simultaneously generated, tracking may be performed according to priorities with reference to metadata of the respective abnormal sound sources. For example, if a plurality of events simultaneously occur, tracking may be performed according to types and priorities of the events.) (Paragraph 56).
(The metadata refers to data that provides information about original data (an image, a sound source, or the image and the sound source) and describes another data. In the present exemplary embodiment, the metadata generator 150 generate metadata including a number of times of generating the abnormal image and/or the abnormal sound source, analysis results and types of the abnormal image and/or the abnormal sound source, a number of generated events indicated by the abnormal image and/or the abnormal sound source, whether the abnormal sound source is obtained from the abnormal image, and whether the abnormal image and the abnormal sound source indicate a same event with reference to the determination result of the determiner 140.) (Paragraph 45).
Priority information (importance level) is included in the metadata.
Regarding Claim 16, Shin et al. teaches the method of claim 2.
Shin et al. further teaches wherein the sound is a non-verbal sound.
(The event generation condition used in the sound source analysis corresponds to generation of an abnormal sound (a friction sound (skid) of a car tire, a glass breaking sound, an alarm sound, a collision sound, or the like), generation of a sound specified by the user (for example, a man's screaming, woman's screaming, baby's crying sound, or the like), generation of a sound with an amplitude higher than or equal to a threshold value, or the like, and may be preset.) (Paragraph 43).
The recorded sounds are listed to be non-verbal
Regarding Claim 19, Shin et al. teaches the method of claim 2.
Shin et al. further teaches wherein, in the setting step, the accuracy for the sound source is set based on a result of image recognition on the sound source in a corresponding image frame or apparatus information associated with the image frame for an imaging apparatus that captures the image frame.
(The image analyzer 120 may also functionally perform color processing, blur processing, edge enhancement processing, image analysis processing, image recognition processing, image effect processing, or the like on the image signal output from the image sensor 110.) (Paragraph 40).
(For example, with reference to the first metadata shown in FIG. 3B, a probability of the abnormal sound source being a collision and a probability of the abnormal image being an explosion are the highest, and thus, the types of the abnormal sound source and the abnormal image do not correspond to each other. However, if a sum of the probability (40%) of the abnormal sound source being the explosion and the probability (95%) of the abnormal image being the explosion exceeds a preset threshold value (for example, 130%), the result of the cross check may be determined as an explosion. For example, with reference to second metadata shown in FIG. 3B, if a sum of a probability (70%) of the abnormal sound source being man's screaming, a probability (30%) of the abnormal sound source being woman's screaming, and a probability (80%) of the abnormal sound source being a violent event exceeds a preset threshold value, the result of the cross check may be determined as being a violent event. In this case, a man's screaming and a woman's screaming may be regarded as a same type of events in order to sum the accuracies, acquire a maximum value, or acquire an average value in order to use values of the accuracies.) (Paragraph 49).
Image recognition is used to analyze the object in the image and then it is associated with the detected sound source to verify they are the same source.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 4-6, 9, 17-18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over US Patent Application Publication US 20150092052 A1 (Shin et al.) in view of Japan Patent Publication JP 2016126188 A (Yoshimura et al.).
Regarding Claim 4, Shin et al. teaches the system of claim 1.
Shin et al. further teaches a second acquisition step of acquiring video data including a plurality of image frames,
(The surveillance camera 100 is connected to the image recording apparatus 300 that may be a DVR. If the surveillance camera 100 is a network camera, the surveillance camera 100 may be connected to the image recording apparatus 300 through a network. In this case, the image recording apparatus 300 may be an NVR.) (Paragraph 33).
Shin et al. does not explicitly teach: wherein, in the setting step, the accuracy for the sound source is set according to whether or not the sound source is present within an angle of view of a corresponding image frame among the plurality of image frames.
However, Yoshimura et al. teaches wherein, in the setting step, the accuracy for the sound source is set according to whether or not the sound source is present within an angle of view of a corresponding image frame among the plurality of image frames.
(Referring to the movement direction calculated by movement determination unit 220, the position of the sound source specified by sound source position specifying unit 225, and the determination result determined by determination unit 250, the position of the sound source is within the field of view. The image display method is determined so that the image size and brightness in the case of the sound source are suppressed more than the image size and brightness in the case where the position of the sound source is outside the field of view. The image generated by the generation unit 260 is displayed on the translucent display 140.) (Fig. 6 Description, Paragraph 6).
A determination is made on if the camera has the sound source in its field of view.
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the image and sound source monitoring system as taught by Shin et al. to Check the image capture device’s angle of view for the sound source as taught by Yoshimura et al. This would have been an obvious improvement to enable the camera reposition the camera to keep the sound source in view (Yoshimura et al. Fig. 10 Description, Paragraph 10).
Regarding Claim 5, Shin et al. teaches the system of claim 1.
Furthermore, Yoshimura et al. teaches a determination step of determining whether or not the sound satisfies a predetermined criterion in a case in which the sound source is not present within an angle of view of a corresponding image frame,
(Referring to the movement direction calculated by movement determination unit 220, the position of the sound source specified by sound source position specifying unit 225, and the determination result determined by determination unit 250, the position of the sound source is within the field of view. The image display method is determined so that the image size and brightness in the case of the sound source are suppressed more than the image size and brightness in the case where the position of the sound source is outside the field of view. The image generated by the generation unit 260 is displayed on the translucent display 140.) (Fig. 6 Description, Paragraph 6).
A determination is made on if the camera has the sound source in its field of view.
Furthermore, Shin et al. teaches wherein, in the setting step, the accuracy for the sound is set to be higher in a case in which the sound satisfies the predetermined criterion than in a case in which the sound does not satisfy the predetermined criterion.
(For example, with reference to the first metadata shown in FIG. 3B, a probability of the abnormal sound source being a collision and a probability of the abnormal image being an explosion are the highest, and thus, the types of the abnormal sound source and the abnormal image do not correspond to each other. However, if a sum of the probability (40%) of the abnormal sound source being the explosion and the probability (95%) of the abnormal image being the explosion exceeds a preset threshold value (for example, 130%), the result of the cross check may be determined as an explosion. For example, with reference to second metadata shown in FIG. 3B, if a sum of a probability (70%) of the abnormal sound source being man's screaming, a probability (30%) of the abnormal sound source being woman's screaming, and a probability (80%) of the abnormal sound source being a violent event exceeds a preset threshold value, the result of the cross check may be determined as being a violent event. In this case, a man's screaming and a woman's screaming may be regarded as a same type of events in order to sum the accuracies, acquire a maximum value, or acquire an average value in order to use values of the accuracies.) (Paragraph 49).
Shin et al. adjust accuracies based on the determination that the object in the image is the sound source.
Regarding Claim 6, Shin et al. in view of Yoshimura et al. teaches the system of claim 5.
Furthermore, Yoshimura et al. teaches a change step of, in a case in which the sound satisfies the predetermined criterion, changing an orientation of an imaging lens of an imaging apparatus such that the imaging lens approaches a direction of the sound source or reducing zoom magnification of the imaging apparatus such that the sound source is included in the angle of view of the image frame.
(When an image is generated by the image generation unit 260 in the process of step S1010, the determination unit 250 determines whether the position of the sound source is within the user's field of view (step S1015). … When it is determined in step S1015 that it is within the user's field of view (step S1015: Yes), the display unit 270 moves if the movement determination unit 220 determines that the sound source is moving. Based on the movement direction calculated by the determination unit 220, sound source movement information indicating the movement direction is generated (step S1020).) (Fig 10. Description, Paragraphs 9-10).
The display unit is the current image being created by the camera. The display unit repositions to follow the sound source and keep it in the field of view.
Regarding Claim 9, Shin et al. teaches the system of claim 1.
Furthermore, Yoshimura et al. teaches wherein, in the creation step, information on whether or not the sound source is present within an angle of view of a corresponding image frame is created as the accessory information.
(Referring to the movement direction calculated by movement determination unit 220, the position of the sound source specified by sound source position specifying unit 225, and the determination result determined by determination unit 250, the position of the sound source is within the field of view. The image display method is determined so that the image size and brightness in the case of the sound source are suppressed more than the image size and brightness in the case where the position of the sound source is outside the field of view. The image generated by the generation unit 260 is displayed on the translucent display 140.) (Fig. 6 Description, Paragraph 6).
A determination is made on if the camera has the sound source in its field of view.
Regarding Claim 17, Shin et al. teaches the system of claim 2.
Furthermore, Shin et al. teaches a second acquisition step of acquiring video data including a plurality of image frames,
(The surveillance camera 100 is connected to the image recording apparatus 300 that may be a DVR. If the surveillance camera 100 is a network camera, the surveillance camera 100 may be connected to the image recording apparatus 300 through a network. In this case, the image recording apparatus 300 may be an NVR.) (Paragraph 33).
Furthermore, Yoshimura et al. teaches wherein, in the setting step, the accuracy for the sound source is set according to whether or not the sound source is present within an angle of view of a corresponding image frame among the plurality of image frames.
(Referring to the movement direction calculated by movement determination unit 220, the position of the sound source specified by sound source position specifying unit 225, and the determination result determined by determination unit 250, the position of the sound source is within the field of view. The image display method is determined so that the image size and brightness in the case of the sound source are suppressed more than the image size and brightness in the case where the position of the sound source is outside the field of view. The image generated by the generation unit 260 is displayed on the translucent display 140.) (Fig. 6 Description, Paragraph 6).
A determination is made on if the camera has the sound source in its field of view.
Regarding Claim 18, Shin et al. teaches the system of claim 2.
Furthermore, Yoshimura et al. teaches a determination step of determining whether or not the sound satisfies a predetermined criterion in a case in which the sound source is not present within an angle of view of a corresponding image frame,
(Referring to the movement direction calculated by movement determination unit 220, the position of the sound source specified by sound source position specifying unit 225, and the determination result determined by determination unit 250, the position of the sound source is within the field of view. The image display method is determined so that the image size and brightness in the case of the sound source are suppressed more than the image size and brightness in the case where the position of the sound source is outside the field of view. The image generated by the generation unit 260 is displayed on the translucent display 140.) (Fig. 6 Description, Paragraph 6).
A determination is made on if the camera has the sound source in its field of view.
Furthermore, Shin et al. teaches wherein, in the setting step, the accuracy for the sound is set to be higher in a case in which the sound satisfies the predetermined criterion than in a case in which the sound does not satisfy the predetermined criterion.
(For example, with reference to the first metadata shown in FIG. 3B, a probability of the abnormal sound source being a collision and a probability of the abnormal image being an explosion are the highest, and thus, the types of the abnormal sound source and the abnormal image do not correspond to each other. However, if a sum of the probability (40%) of the abnormal sound source being the explosion and the probability (95%) of the abnormal image being the explosion exceeds a preset threshold value (for example, 130%), the result of the cross check may be determined as an explosion. For example, with reference to second metadata shown in FIG. 3B, if a sum of a probability (70%) of the abnormal sound source being man's screaming, a probability (30%) of the abnormal sound source being woman's screaming, and a probability (80%) of the abnormal sound source being a violent event exceeds a preset threshold value, the result of the cross check may be determined as being a violent event. In this case, a man's screaming and a woman's screaming may be regarded as a same type of events in order to sum the accuracies, acquire a maximum value, or acquire an average value in order to use values of the accuracies.) (Paragraph 49).
Shin et al. adjust accuracies based on the determination that the object in the image is the sound source.
Regarding Claim 20, Shin et al. teaches the system of claim 2.
Furthermore, Yoshimura et al. teaches wherein, in the creation step, information on whether or not the sound source is present within an angle of view of a corresponding image frame is created as the accessory information.
(Referring to the movement direction calculated by movement determination unit 220, the position of the sound source specified by sound source position specifying unit 225, and the determination result determined by determination unit 250, the position of the sound source is within the field of view. The image display method is determined so that the image size and brightness in the case of the sound source are suppressed more than the image size and brightness in the case where the position of the sound source is outside the field of view. The image generated by the generation unit 260 is displayed on the translucent display 140.) (Fig. 6 Description, Paragraph 6).
A determination is made on if the camera has the sound source in its field of view.
Claims 13 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over US Patent Application Publication US 20150092052 A1 (Shin et al.) in view of US Patent Application Publication US 20130024192 A1 (Nomura et al.).
Regarding Claim 13, Shin et al. teaches the system of claim 2.
Shin et al. does not explicitly teach: wherein, in a case in which the accuracy according to the importance level satisfies a first condition, in the creation step, onomatopoeic word information in which the sound is converted into text as an onomatopoeic word is created as the accessory information.
However, Nomura et al. teaches wherein, in a case in which the accuracy according to the importance level satisfies a first condition, in the creation step, onomatopoeic word information in which the sound is converted into text as an onomatopoeic word is created as the accessory information.
(Herein, the so-called atmospheric sound information is magnitude of the sound pressure of the audio signals, the frequency of the audio signals, the type of the audio signals (for example, a classification of the voice and the environmental sounds except the voice such as the sound of rain and the sound of an automobile) or the like. The atmosphere expression word selecting unit 2 selects the atmosphere expression word corresponding to the atmospheric sound that is being generated in the field in which the audio signals have been acquired based on the atmospheric sound information prepared by the input signal analyzing unit 1. … As a representative word of the atmosphere expression word, there exist an onomatopoeic word and a mimetic word.) (Paragraphs 34-35).
The system uses a magnitude, frequency, and type of the audio to accurately determine the source. Once determined, an onomatopoeic word is generated for the sound.
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the image and sound source monitoring system as taught by Shin et al. to create onomatopoeic words for the sound as taught by Nomura et al. This would have been an obvious improvement to create additional metadata that describes the sound itself rather than using a word that represents it through meaning. (Nomura et al. Paragraph 9).
Regarding Claim 14, Shin et al. in view of Nomura et al. teaches the system of claim 13.
Furthermore, Nomura et al. teaches a determination step of determining whether or not the sound satisfies a predetermined criterion in a case in which the sound source is not present within an angle of view of a corresponding image frame,
(As a representative word of the atmosphere expression word, there exist an onomatopoeic word and a mimetic word.) (Paragraph 35).
Nomura et al. also generates mimetic words for the detected sound.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICHOLAS DANIEL LOWEN whose telephone number is (571)272-5828. The examiner can normally be reached Mon-Fri 8:00am - 4:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Paras D Shah can be reached at (571) 270-1650. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NICHOLAS D LOWEN/Examiner, Art Unit 2653
/Paras D Shah/Supervisory Patent Examiner, Art Unit 2653
06/22/2026
Prosecution Insights