Detailed Action
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 6 is objected to because of the following informalities:
“…frames that are spaced at a predetermined interval from each other from the first selected frame that is followed by and the closest to the second selected frame to the second selected frame as frames that include images to be used to estimate a lying posture...” should read “…frames that are spaced at a predetermined interval from each other from the first selected frame that is followed by and the closest to the second selected frame .
Appropriate correction is required.
Claim Interpretation
Note that according to the Federal Circuit’s 2004 Superguide v. DirecTV decision, “at least one of
… and … “ requires at least one instance of each and every item listed. Claims 1 and 15 contain such limitations, and the specifications supports a conjunctive “and” interpretation (see paragraphs 52 and 53). For examination purposes, the limitation of claims 1 and 15 will be interpreted to require at least one instance of each and every item listed. Alternatively, claims 10 and 12 contain such limitations, however, the specification supports a disjunctive “or” interpretation (see paragraphs 34, 54, and 55). For examination purposes, the limitation of claims 10 and 12 will be interpreted to require at least one instance of the items listed.
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that use the word “step” and are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. Such claim limitations are: “lying posture estimation step”, “lying posture classification step”, “body movement measurement step”, “display step”, “selection step”, “video-shortening step”, “rotation angle determination step”, and “estimated result determination step” in claim 1-12.
Because these claim limitations are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, they are being interpreted to cover the corresponding acts described in the specification (see paragraph 0014) as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitations recites sufficient structure to perform the claimed function so as to avoid them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 5 and 6 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 5 recites “a step of selecting a frame that has a difference between images of this frame and the first selected frame that is followed by and the closest to this frame not smaller than a second selection threshold in the sleeping video as a second selected frame.”, which is indefinite. It is unclear whether the limitation “this frame” is meant to refer to the frame being selected in this limitation, a frame selected in the previous step, or another frame in the video sequence. As such, the claim lacks antecedent basis for “this frame” and one of ordinary skill in the art would not be able to ascertain the scope of the claim. For examination purposes, the limitation “this frame” will be interpreted to mean the frame currently being processed in the step in which the limitation appears.
Claim 6 recites “selecting, in addition to the first and second selected frames, frames that are spaced at a predetermined interval from each other from the first selected frame that is followed by and the closest to the second selected frame to the second selected frame as frames that include images to be used to estimate a lying posture in the lying posture estimation step.”, which is indefinite. It is unclear whether the limitation “that is followed by and the closest to the second selected frame” is meant to refer to the frames being selected or the first selected frame. For example, the limitation could be interpreted to mean that the frames being selected are spaced at a predetermined interval from each other from the first selected frame and are followed by and the closest to the second selected frame. Alternatively, it could be interpreted to mean that the frames being selected could be limited as spaced at a predetermined interval from each other from the first selected frame that is followed by and closest to the second selected frame. Thus, one of ordinary skill in the art would not be able to ascertain the scope of the claim and the claim is indefinite. For examination purposes, the claim will be interpreted to mean that frames are selected, in addition to the first and second selected frames, which are spaced at a predetermined interval from each other from the first selected frame, where the first selected frame is defined as being followed by and the closest to the second selected frame.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-8, 11, and 13-16 are rejected under 35 U.S.C. 101.
Claim 1 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract
idea of visualizing data on an image, without significantly more.
The claim recites: “A sleeping video analysis method comprising: a lying posture estimation step of estimating a lying posture of a subject while sleeping based on a sleeping video of the subject while sleeping; a lying posture classification step of classifying the lying posture of the subject while sleeping based on a lying posture estimation result(s) of the lying posture estimation step; a body movement measurement step of measuring body movement of the subject while sleeping based on the lying posture estimation result(s) of the lying posture estimation step; and a display step of displaying at least one of a lying posture classification result(s) of the lying posture classification step and a body movement measurement result(s) of the body movement measurement step.”
The limitations, as drafted, are processes that, under their broadest reasonable interpretation,
cover performance of the limitation in the human mind. A person can mentally evaluate a video to estimate and classify the posture of a sleeping subject. The person can further mentally observe the estimated posture over a period of time to measure body movement.
The judicial exception is not integrated into a practical application. For example, the claim
recites the additional element, “a display step of displaying at least one of a lying posture classification result(s) of the lying posture classification step and a body movement measurement result(s) of the body movement measurement step.”. This additional element can reasonably be interpreted as merely a data outputting step of the method. The obtained posture classification and a body movement measurement result is displayed for visualization; therefore, the additional element does not add a meaningful limitation to the method as it is an insignificant extra-solution activity.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into
a practical application, the additional elements can be reasonably interpreted as well-understood,
routine, and conventional in the field. Therefore, this limitation remains insignificant extra-solution
activity even upon reconsideration and does not amount to significantly more. This claim is not patent
eligible.
Claim 2 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea of selecting frames based on thresholding, without significantly more. For example, a person can mentally observe a sequence of frames and select particular frames which fall within a difference threshold. This claim is not patent eligible.
Claim 3 is rejected under 35 U.S.C. 101 because the claimed invention is directed to a further limitation of the same abstract idea identified in the analysis of claims 1 and 2. For example, the person can perform estimation of the posture of the sleeping subject based on the selected frames.
Claim 4 is rejected under 35 U.S.C. 101 because the claimed invention is directed to a further limitation of the same abstract idea identified in the analysis of claims 1 and 2. For example, the person can additionally select frames which are adjacent to the particular frame selected previously, to estimate posture of the sleeping subject.
Claim 5 is rejected under 35 U.S.C. 101 because the claimed invention is directed to a further limitation of the same abstract idea identified in the analysis of claims 1 and 2. For example, the person can mentally evaluate frames by applying difference thresholding to select a first frame and then identify a second frame by applying an additional difference thresholding which uses the first frame for comparison.
Claim 6 is rejected under 35 U.S.C. 101 because the claimed invention is directed to a further limitation of the same abstract idea identified in the analysis of claims 1 and 2. For example, the person can additionally select frames which are in a predetermined interval from a previously selected frame for posture estimation of the sleeping subject.
Claim 7 is rejected under 35 U.S.C. 101 because the claim recites additional elements which can reasonably be interpreted as merely a data outputting step. The data is displayed as a time series; therefore, the additional element does not add a meaningful limitation to the method as it is an insignificant extra-solution activity. This claim is not patent eligible.
Claim 8 is rejected under 35 U.S.C. 101 because the claim recites additional elements recited at a high
level of generality such that they amount to merely combining frames. Accordingly, these
additional elements do not integrate the abstract idea into a practical application because they do not
impose any meaningful limits on practicing the abstract idea. This claim is not patent eligible.
Claim 11 is rejected under 35 U.S.C. 101 because the claim recites additional elements recited at a high
level of generality such that they amount to merely using machine learning to implement the abstract idea. Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. This claim is not patent eligible.
Claim 13 is rejected under 35 U.S.C. 101 because the claim recites additional elements recited at a high
level of generality such that they amount to merely using a computer as a tool to implement the abstract idea. Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. This claim is not patent eligible.
Furthermore, claim 13 recites “A program which executes a computer to perform the sleeping video analysis method according to claim 1”. The broadest reasonable interpretation of the claim is a computer program per se. Computer programs are not considered patent-eligible subject matter because they do not fall within any of the four statutory categories of appropriate subject matter for a patent: process, machines, manufactures and composition of matter. Therefore, claim 13 is rejected under 35 U.S.C. 101.
Claim 14 is rejected under 35 U.S.C. 101 because the claim recites additional elements recited at a high
level of generality such that they amount to merely using a computer as a tool to implement the abstract idea. Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. This claim is not patent eligible.
Furthermore, claim 14 recites “A computer-readable storage medium having the program according to claim 13.”. The specification does not provide a definition for the claimed “medium”, which excludes signals/waveforms. Thus, the broadest reasonable interpretation of computer-readable storage medium is a transitory signal. Transitory signals are not considered patent-eligible subject matter because they do not fall within any of the four statutory categories of appropriate subject matter for a patent: process, machines, manufactures and composition of matter. Therefore, claim 14 is rejected under 35 U.S.C. 101.
Claims 15 and 16 contain elements found analogous to that of claims 1 and 2, respectively, with the addition of “A sleeping video analysis apparatus comprising: a controller… and a display”. The additional elements can be reasonably interpreted as merely using a generic computer as a tool to implement the abstract idea. Implementing an abstract idea on a generic computer does not integrate a judicial exception into a practical application. Thus, claims 15 and 16 are similarly rejected under 35 U.S.C. 101.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 13, 14, and 15 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Min et al. (KR 20170054673 A), (hereinafter Min).
Regarding claim 1, Min teaches A sleeping video analysis method comprising:
a lying posture estimation step of estimating a lying posture of a subject while sleeping based on a sleeping video of the subject while sleeping (Min, “First, a sleeping person's position information and a human body depth information are measured in a sleeping position determining apparatus 100 provided with a Kinect sensor 110 installed at a sleeping place and including an infrared sensor 111 and a depth sensor 112 Step S101… The Kinect sensor 110 of the present invention uses the Kinect Nui API to detect skeleton information of a person and detect depth information of a sleeping person in a dark place and depth information of a sleeping person in a sleeping position It becomes possible to measure in three dimensions.”, pg. 5, lines 16-28, A human skeleton pose estimation is performed for sleeping subjects.);
a lying posture classification step of classifying the lying posture of the subject while sleeping based on a lying posture estimation result(s) of the lying posture estimation step (Min, “Then, if the joint position is confirmed in the image of the sleeping person, the sleeping posture which is compared with the previously registered sleeping posture is confirmed (step S104). As shown in FIG. 3, the registered sleeping posture can be classified into three types: Foetus, Log, Yearner, Soldier, Freefaller, Starfish) And the like.”, pg. 6, lines 6-9, The pose estimation includes classification of sleeping posture.);
a body movement measurement step of measuring body movement of the subject while sleeping based on the lying posture estimation result(s) of the lying posture estimation step (Min, “Then, in order to confirm the sleeping person's confrontation during the sleeping, the position of the joint at the time of sleeping is confirmed and accumulated (step S105). Then, the converted sleep posture is confirmed by using the joint movement value (step S106). Thus, it is possible to continuously check whether the sleeping person has moved during sleep. At this time, the motion values of all major joints are calculated every 0.5 seconds, and the calculation of joint motion values is calculated by Equation (1).”, pg. 6, lines 20-25, The skeleton data is tracked over time to measure body movement based on joint position differences of consecutive frames.); and
a display step of displaying at least one of a lying posture classification result(s) of the lying posture classification step and a body movement measurement result(s) of the body movement measurement step (Min, “The wireless terminal 200 receives and stores information about the sleeping position provided from the sleeping position determining apparatus 100 and displays it on the screen so that the sleeping person can confirm the sleeping position as shown in FIG. (7). At this time, the sleeping person will be able to identify the type corresponding to the sleeping posture and the sleeping posture which change in a time-wise manner.”, pg. 7, lines 9-13, see Fig. 7, Posture classifications are displayed to a user. The system tracks body movements to detect when a posture change occurs. When body movement changes are detected in the time-series, the system would display an updated posture classification, reflecting the results of the body movement measurement.).
Claim 13 corresponds to claim 1, with the addition of a program which executes a computer to perform the sleeping video analysis method according to claim 1. Min teaches the addition of a program which executes a computer to perform the sleeping video analysis method according to claim 1 (Min, see Fig. 8). As indicated in the analysis of claim 1, Min teaches all the limitation according to claim 1. Therefore, claim 13 is rejected for the same reason as claim 1.
Claim 14 corresponds to claim 13, with the addition of a computer-readable storage medium having the program according to claim 13. Min teaches the addition of a computer-readable storage medium having the program according to claim 13 (Min, see Fig. 8, sleep information storage unit 210). As indicated in the analysis of claim 13, Min teaches all the limitation according to claim 13. Therefore, claim 14 is rejected for the same reason as claim 13.
Claim 15 corresponds to claim 1, with the addition of a sleeping video analysis apparatus comprising a controller and a display. Min teaches the addition of a sleeping video analysis apparatus comprising a controller and a display (Min, see Fig. 8, wireless terminal 200 and sleeping position display 220). As indicated in the analysis of claim 13, Min teaches all the limitation according to claim 13. Therefore, claim 14 is rejected for the same reason as claim 13.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2, 3, 5, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Min et al. (KR 20170054673 A) in view of Zhang (US 20050228849 A1).
Regarding claim 2, Min teaches the sleeping video analysis method according to claim 1 further comprising a selection step of sequentially reading images of frames in the sleeping video, and selecting a frame(s)
Min does not teach selecting a frame(s) that has/have a difference between images of frames not smaller than a selection threshold in the sleeping videos a selected frame(s) corresponding to a lying posture change of the subject in the sleeping video.
However, Zhang teaches selecting a frame(s) that has/have a difference between images of frames not smaller than a selection threshold in the sleeping videos a selected frame(s) corresponding to a lying posture change of the subject in the sleeping video (Zhang, “FIG. 4 shows a series of example video frames 50-52 in the video 12 that include an object 54. The object 54 changes position within each subsequent video frame 50-52. The changing position of the object 54 is indicated by
changes in the color layouts for the video frames 50-52… The color layout analyzer selects a video frame as a candidate key-frame if a relatively large change in its color layout is detected in comparison to previous video frames in the video 12. Initially, the color layout analyzer selects the first video frame in the video 12 as a candidate key-frame and as a reference frame. The color layout analyzer then compares a color layout for the reference frame with a color layout for each subsequent video frame in the video 12 until a difference is higher than a predetermined threshold. The color layout analyzer selects a video frame having a difference in its color layout that exceeds the predetermined threshold as a new candidate key-frame and as a new reference frame and then repeats the process for the remaining video frames in the video 12.”, pg. 3, paragraphs 0035 and 0036, Candidate key-frames are selected corresponding to a change in position of an object. This includes selecting subsequent candidate key-frames which have a color layout higher than a difference threshold as compared to a reference frame.).
Min teaches continuously selecting frames according to a fixed time interval for posture estimation (Min, “The joint position confirmation unit 130 preferably checks and stores the joint position every 0.5 seconds so as to detect continuous movement of the sleeping person.”, pg. 8, lines 11-13). Zhang teaches selecting key-frames corresponding to object movement by applying difference thresholding with respect to a reference frame (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the frame selection of Min to include the thresholding for key-frame selection as taught by Zhang (Zhang, pg. 3, paragraphs 0035 and 0036, see Fig. 4), thereby selecting a reduced set of frames corresponding to object movement for posture estimation. The motivation for doing so would have been to reduce the computational load on the system by performing posture estimation only on frames which are likely to contain meaningful changes (as suggested by Zhang, “The above methods for key-frame extraction yield key-frames that may indicate highlights in a video clip and depict content in a video clip that may be meaningful to a viewer.”, pg. 1, paragraph 0005, lines 1-3). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Min with Zhang to obtain the invention as specified in claim 2.
Regarding claim 3, Min in view of Zhang teaches the sleeping video analysis method according to claim 2, wherein the lying posture estimation step includes a step of estimating a lying posture of the subject while sleeping by using an image of the selected frame, which is selected in the selection step prior to the lying posture estimation step (Zhang, “The color layout analyzer selects a video frame having a difference in its color layout that exceeds the predetermined threshold as a new candidate key-frame and as a new reference frame and then repeats the process for the remaining video frames in the video 12.”, pg. 3, paragraphs 0036, The key-frame selection provides a collection of frames corresponding to meaningful changes in object movements for posture estimation.).
Regarding claim 5, Min in view of Zhang teaches the sleeping video analysis method according to claim 2, wherein the selection step includes a step of sequentially reading images of frames in the sleeping video and selecting a frame(s) that has/have a difference between images of frames adjacent to each other not smaller than a first selection threshold in the sleeping video as a first selected frame(s), and a step of selecting a frame that has a difference between images of this frame and the first selected frame that is followed by and the closest to this frame not smaller than a second selection threshold in the sleeping video as a second selected frame (Zhang, “The color layout analyzer selects a video frame having a difference in its color layout that exceeds the predetermined threshold as a new candidate key-frame and as a new reference frame and then repeats the process for the remaining video frames in the video 12.”, pg. 3, paragraphs 0036, Thresholding with respect to a reference frame is applied to select key-frames which has a difference that exceeds a predetermined threshold. The selected frame then becomes the new reference frame, and this process is iteratively applied subsequent frames to identify successive key-frames.).
Claim 16 corresponds to claim 2, with the addition of a sleeping video analysis apparatus comprising a controller and a display. Min in view of Zhang teaches the addition of a sleeping video analysis apparatus comprising a controller and a display (Min, see Fig. 8, wireless terminal 200 and sleeping position display 220). As indicated in the analysis of claim 2, Min in view of Zhang teaches all the limitation according to claim 2. Therefore, claim 16 is rejected for the same reasons of obviousness as claim 2.
Claims 4 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over Min et al. (KR 20170054673 A) in view of Zhang (US 20050228849 A1) and further in view of Yang et al. (US 20210335028 A1), (hereinafter, Yang).
Regarding claim 4, Min in view of Zhang teaches the sleeping video analysis method according to claim 3. Min in view of Zhang does not teach wherein the selection step includes a step of additionally selecting, in addition to the selected frame, frames previous and subsequent to the selected frame as frames that include images to be used to estimate a lying posture in the lying posture estimation step.
However, Yang teaches wherein the selection step includes a step of additionally selecting, in addition to the selected frame, frames previous and subsequent to the selected frame as frames that include images to be used to estimate a lying posture in the lying posture estimation step (Yang, “At block 502 , the process 500 involves generating input data vector for a target video frame to input to the contact estimation model 118. In some examples, the contact estimation model 118 is configured to accept an input data vector containing K lower body joint points of the character in N1 video frames surrounding the target video frame. The lower body joint points can be determined from the 2D pose joint points 404 (or the joint points 126) extracted from each frame of the video sequence. For example, if N1 is set to 9, the video frames used to generate the input vector can include the target frame, 4 frames before the target frame, and 4 frames after the target frame. In the example skeleton shown in FIG. 3, nine joint points (points 8-13, 18-20) are lower body joint points of the character and are used to generate the input data vector…”, pg. 6, paragraph 0047, Frames previous and subsequent to a target frame are used to define an input vector for an estimation model.).
Min in view of Zhang teaches estimating posture of a sleeping subject by selecting key-frames corresponding to object movement (Zhang, “FIG. 4 shows a series of example video frames 50-52 in the video 12 that include an object 54. The object 54 changes position within each subsequent video frame 50-52. The changing position of the object 54 is indicated by changes in the color layouts for the video frames 50-52… The color layout analyzer selects a video frame as a candidate key-frame if a relatively large change in its color layout is detected in comparison to previous video frames in the video 12.”, pg. 3, paragraph 0035). Yang teaches generating an input vector for an estimation model based on a target frame and a set of surrounding frames (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the key-frame posture estimation of Min in view of Zhang to include the input vector generation as taught by Yang (Yang, pg. 6, paragraph 0047), thereby providing key-frames and their surrounding frames as the input vector for a pose estimation model. The motivation for doing so would have been to improve the accuracy of the posture estimation by incorporating neighboring frames for additional context. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Min in view of Zhang with Yang to obtain the invention as specified in claim 4.
Regarding claim 6, Min in view of Zhang teaches the sleeping video analysis method according to claim 5. Min in view of Zhang does not teach wherein the selection step further includes a step of selecting, in addition to the first and second selected frames, frames that are spaced at a predetermined interval from each other from the first selected frame that is followed by and the closest to the second selected frame to the second selected frame as frames that include images to be used to estimate a lying posture in the lying posture estimation step.
However, Yang teaches wherein the selection step further includes a step of selecting, in addition to the first and second selected frames, frames that are spaced at a predetermined interval from each other from the first selected frame that is followed by and the closest to the second selected frame to the second selected frame as frames that include images to be used to estimate a lying posture in the lying posture estimation step (Yang, “At block 502 , the process 500 involves generating input data vector for a target video frame to input to the contact estimation model 118. In some examples, the contact estimation model 118 is configured to accept an input data vector containing K lower body joint points of the character in N1 video frames surrounding the target video frame. The lower body joint points can be determined from the 2D pose joint points 404 (or the joint points 126) extracted from each frame of the video sequence. For example, if N1 is set to 9, the video frames used to generate the input vector can include the target frame, 4 frames before the target frame, and 4 frames after the target frame. In the example skeleton shown in FIG. 3, nine joint points (points 8-13, 18-20) are lower body joint points of the character and are used to generate the input data vector…”, pg. 6, paragraph 0047, A predetermined interval of frames surrounding a target frame are used to define an input vector for an estimation model.).
Min in view of Zhang teaches estimating posture of a sleeping subject by selecting key-frames corresponding to object movement (Zhang, “FIG. 4 shows a series of example video frames 50-52 in the video 12 that include an object 54. The object 54 changes position within each subsequent video frame 50-52. The changing position of the object 54 is indicated by changes in the color layouts for the video frames 50-52… The color layout analyzer selects a video frame as a candidate key-frame if a relatively large change in its color layout is detected in comparison to previous video frames in the video 12.”, pg. 3, paragraph 0035). Yang teaches generating an input vector for an estimation model by setting a predetermined interval of frames surrounding a target frame (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the key-frame posture estimation of Min in view of Zhang to include the input vector generation as taught by Yang (Yang, pg. 6, paragraph 0047), thereby providing key-frames and their surrounding frames as the input vector for a pose estimation model. The motivation for doing so would have been to improve the accuracy of the posture estimation by incorporating neighboring frames for additional context. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Min in view of Zhang with Yang to obtain the invention as specified in claim 6.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Min et al. (KR 20170054673 A) in view of Wada et al. (US 20210142049 A1), (hereinafter, Wada).
Regarding claim 7, Min teaches the sleeping video analysis method according to claim 1. Min does not teach wherein the display step includes a step of displaying the lying posture classification results of the lying posture classification step and the body movement measurement results of the body movement measurement step in a time series.
However, Wada teaches wherein the display step includes a step of displaying the lying posture classification results of the lying posture classification step and the body movement measurement results of the body movement measurement step in a time series. (Wada, “The display unit 13 displays the motion recognition result of each body part in a chronologically aligned manner on the basis of the motion recognition information 19c. FIG. 5 illustrates, as an example, a graph in which the motion recognition result of each body part is displayed in a chronologically aligned manner.”, pg. 5, paragraph 0068, see Fig. 5, A motion recognition result is displayed as a time series representation for each body parts, including when a motion changes and a classification of the motion for each change.)
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Min to display lying posture classifications and body movement measurements in a time series as taught by Wada (Wada, pg. 5, paragraph 0068, see Fig. 5). The motivation for doing so would have been to provide a chronological representation of changes in body posture over time, thereby improving the visualization for posture analysis by a user. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Min with Wada to obtain the invention as specified in claim 7.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Min et al. (KR 20170054673 A) in view of Zhang (US 20050228849 A1) and further in view of Divakaran et al. (US 7110458 B2), (hereinafter Divakaran).
Regarding claim 8, Min in view of Zhang teaches the sleeping video analysis method according to claim 2. Min in view of Zhang does not teach further comprising a video-shortening step of combining the selected frames selected from the sleeping video in the selection step so as to generate a shortened video.
However, Divakaran teaches further comprising a video-shortening step of combining the selected frames selected from the sleeping video in the selection step so as to generate a shortened video (Divakaran, “Video Summarization generates a compact representation of a video that conveys the semantic essence of the video. The compact representation can include "key-frames' or "key-segments,” or a combination of key-frames and key-segments.”, column 2, lines 15-18, “The invention also provides a method to combine motion and color based key-frame extraction by using the motion
based method for the easier to Summarize segments, and the color based methods for the harder to Summarize segments. Easier to Summarize segments are represented by a rapidly extracted Summary consisting of one or more key-frames, while a color based Summarization process extracts sequences of frames from each difficult to Summarize segment. The single frames and extracted sequences of frames are concatenated in temporal order to form the Summary of the video.”, column 4, lines 37-47, Key-frames corresponding to motion are combined in a video summarization process.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Min in view of Zhang to include the video summarization method for selected key-frames as taught by Divakaran (Divakaran, column 4, lines 37-47). The motivation for doing so would have been to provide a temporal representation of the selected keyframes, thereby improving visualization of posture-related events for a user. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Min in view of Zhangwith Divakaran to obtain the invention as specified in claim 8.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Min et al. (KR 20170054673 A) in view of Zhang (US 20050228849 A1) and further in view of Divakaran et al. (US 7110459 B2) and Matsumoto et al. (JP 2019219989 A), (hereinafter, Matsumoto).
Regarding claim 9, Min in view of Zhang and further in view of Divakaran teaches the sleeping video analysis method according to claim 8. Min in view of Zhang and further in view of Divakaran does not teach further comprising a rotation angle determination step of rotating images of frames in the shortened video by a plurality of angles prior to the lying posture estimation step to estimate a lying posture of the subject while sleeping for each of the images rotated to a plurality of angles, and determining a rotation angle of an image to be used to estimate the lying posture in the lying posture estimation step.
However, Matsumoto teaches further comprising a rotation angle determination step of rotating images of frames in the shortened video by a plurality of angles prior to the lying posture estimation step to estimate a lying posture of the subject while sleeping for each of the images rotated to a plurality of angles, and determining a rotation angle of an image to be used to estimate the lying posture in the lying posture estimation step (Matsumoto, “The rotation unit 13 of the present embodiment receives the target image from the image acquisition unit 11, rotates the target image 360 degrees at intervals of n degrees, and generates 360 / n rotated images. The rotation unit 13 outputs to the posture estimation unit 14 all the obtained rotated images having a different angle from the target image. The posture estimating unit 14 of the present embodiment receives a plurality of rotated images from the rotating unit 13 and performs posture estimation on each of the rotated images. The posture estimating unit 14 outputs all the posture estimation results obtained for the rotated image to the reliability calculating unit 31. The reliability calculation unit 31 receives a plurality of posture estimation results from the posture estimation unit 14 and calculates reliability for each posture estimation result”, pg. 9, lines 23-34, “The estimation result selection unit 32 receives the reliability of the plurality of posture estimation results from the reliability calculation unit 31, and selects an appropriate posture estimation result based on the distribution of the reliability.”, pg. 10, lines 13-15, An optimal rotation angle is determined for each frames by rotating the frame at various angles and performing posture estimation at each of the angles. The rotation angle which provides the most reliable posture estimation is then selected as the final result.).
Min in view of Zhang and further in view of Divakaran teaches selecting key-frames for posture estimation (Zhang, “The color layout analyzer selects a video frame having a difference in its color layout that exceeds the predetermined threshold as a new candidate key-frame and as a new reference frame and then repeats the process for the remaining video frames in the video 12.”, pg. 3, paragraph 0036). Matsumoto teaches determining an optimal rotation angle for each frames for pose estimation (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Min in view of Zhang and further in view of Divakaran to include determining an optimal rotation angle for the key-frames as taught by Matsumoto (Matsumoto, pg. 9, lines 23-34 and pg. 10, lines 13-15). The motivation for doing so would have been to rotate the key-frames to an orientation that maximizes posture estimation reliability, thereby improving the accuracy of posture estimation. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Min in view of Zhang and further in view of Divakaran with Matsumoto to obtain the invention as specified in claim 9.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Min et al. (KR 20170054673 A) in view of Lui (US 20190130720 A1).
Regarding claim 11, Min teaches the sleeping video analysis method according to claim 1. Min does not teach wherein the lying posture estimation step includes a step of estimating the lying posture of the subject while sleeping based on a learned model produced by machine learning; and the method further includes an estimated result determination step of determining whether the lying posture estimation result(s) estimated by the learned model is/are valid based on a certainty factor of the lying posture estimation acquired by the learned model in the lying posture estimation of the subject while sleeping and a predetermined validity threshold.
However, Lui teaches wherein the lying posture estimation step includes a step of estimating the lying posture of the subject while sleeping based on a learned model produced by machine learning; and the method further includes an estimated result determination step of determining whether the lying posture estimation result(s) estimated by the learned model is/are valid based on a certainty factor of the lying posture estimation acquired by the learned model in the lying posture estimation of the subject while sleeping and a predetermined validity threshold (Lui, “Currently, sudden Infant Death Syndrome (SIDS) is a scary prospect that has no known root cause. SIDS is one of the leading causes of death in babies less than one year old. The risk of SIDS decreases when infants are placed on their backs to sleep, instead of face-down on their stomachs. According to one embodiment, the systems and methods for a machine learning baby monitor described herein use a convolutional neural network (CNN) that is trained on thousands of baby images in various positions and conditions.”, pg. 3, paragraph 0030, lines 7-17, “An " unknown ” state is also included in some embodiments. Instead of showing classifications with low confidence, the system would show the “unknown” state instead. The threshold for the level of confidence below which the “unknown” state is shown may be selectable by the user. This would show more clearly that the baby monitor 118 needs more training and doesn't recognize anything that it sees... This feature reduces the noise of inaccurate classifications, which some users may prefer.”, pg. 6, paragraph 0054, see Fig. 3, A machine learning model is trained to classify images of babies under various positions. Confidence thresholding may be applied to filter low confidence predictions.).
Min teaches estimating the lying posture of a subject while sleeping by determining joint positions from 3D data and performing joint matching for posture classification (Min, “Based on the information measured by the Kinect sensor 110, the joint position of the sleeping person related to the motion is three-dimensionally identified and matched to the 35 sleeping image (step S103).”, pg. 5, lines 34-36). Lui teaches training a machine learning model to classify images of babies in various positions and applying confidence thresholding for low-confidence predictions (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the posture estimation and classification of Min to use