Detailed Action
1. Claims 1-20 are pending in this Application.
Notice of Pre-AIA or AIA Status
2. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless -
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
3. Claims 1-4, 6, 8-9, 12-14 and 17-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by IGO TAKAO et al., ( hereafter IGO), JP2022179218 A, pub. 12/02/2022
As to claim 1, IGO teaches An image processing device (Abstract) comprising:
a control processor, wherein the control processor is configured to: acquire an examination video capturing an observation target during a video endoscopic examination of swallowing ([0014], determine that the swallowing video acquired during the swallowing continuous period immediately preceding the non-swallowing continuous period exceeding the certain period represents one swallowing action and count the number of swallows);
perform recognition processing on the examination video for each frame image, to assign stage information indicating that an image shows which stage of the observation target among a plurality of predetermined swallowing stages to each of the frame images([0037]-[0038], During swallowing, the movement of the soft palate and epiglottis causes the endoscope tip to move violently, which blurs the entire image. This movement creates a large amount of change between consecutive video frames. If the image change in a specific target area exceeds a predefined threshold, the system flags the subsequent frame as occurring during swallowing); and
calculate a feature value indicating a feature related to swallowing of a subject person having the observation target based on the stage information assigned to each of a plurality of the frame images included in the examination video ([0039], An algorithm detects swallowing by measuring endoscope movement through the following steps: first, an edge-detection filter is applied to the inspection images to calculate the Variance of Laplacian; next, a feature extraction unit isolates specific image features from this variance value; finally, a thresholding unit identifies swallowing if these features exceed a predetermined second threshold. This algorithm operates on the principle that the violent movement of the endoscope tip during swallowing significantly increases edge sizes.)
As to claim 2, IGO teaches the stage information includes a swallowing-in-progress stage indicating that the observation target is swallowing and a swallowing-not-in-progress stage indicating that the observation target is not swallowing, and the swallowing-in-progress stage includes an initial swallowing-in-progress stage indicating that the observation target is in an initial stage of the swallowing that is in progress and a late swallowing-in-progress stage indicating that the observation target is in a late stage of the swallowing that is in progress([0042], if the number of image feature points is less than or equal to the third threshold, a swallowing determination value may be calculated by multiplying the pixel value by -1, and if the swallowing determination value is less than the threshold, it may be determined that swallowing is in progress. In this case, if the number of image feature points exceeds the third threshold, it is determined that the patient is not swallowing).
As to claim 3, IGO teaches the control processor is configured to detect a swallowing block that is a group of the frame images capturing a single swallowing motion of the observation target, and the swallowing block includes a plurality of consecutive frame images in which the stage information is the swallowing-in-progress stage([0059]-[0061], The swallowing count unit 47 determines how many times the swallowing action was performed in a series of sequentially continuous examination images (videos) that have been determined to be "swallowing in progress" or "not swallowing in progress" by one or more of the first determination unit 41, second determination unit 42, third determination unit 43, fourth determination unit 44, or fifth determination unit 45.)
As to claim 4, IGO teaches the feature value is the number of the swallowing blocks ([0059]-[0060], the swallowing count unit 47 counts the number of swallows when a non-swallowing video appears after a swallowing video, and the duration of the non-swallowing video exceeds a certain period of time.)
As to claim 6, IGO teaches the feature value is the number of the frame images in which the stage information is the initial swallowing-in-progress stage ([0059]-[0061]) The swallowing count unit 47 counts the number of swallows when a non-swallowing video appears after a swallowing video, and the duration of the non-swallowing video exceeds a
certain period of time. Figure 13 shows a series of examination images 47a that are transmitted to the swallowing count unit 47 and are determined to be either "swallowing in progress" or "not swallowing in progress". The series of examination images 47a are images acquired in chronological order, and each is determined to be either "swallowing" (indicated as "A" in Figure 13) or "non swallowing"(indicated as "B" in Figure 13).).
As to claim 8, IGO teaches the swallowing-not-in-progress stage includes a post-swallowing stage in which the stage information of an immediately preceding frame image is the late swallowing- in-progress stage, and an open stage other than the post-swallowing stage([0061], the swallowing count unit 47 counts the number of swallows when a non-swallowing video appears after a swallowing video, and the duration of the non-swallowing video exceeds a certain period of time. Figure 13 shows a series of examination images 47a that are transmitted to the swallowing count unit 47 and are determined to be either "swallowing in progress" or "not swallowing in progress". ) .
As to claim 9, IGO teaches the feature value is the number of the frame images in which the stage information is the post-swallowing stage ([0061], the swallowing count unit 47 counts the number of swallows when a non-swallowing video appears after a swallowing video, and the duration of the non-swallowing video exceeds a certain period of time. ).
As to claim 12, IGO teaches the control processor is configured to assign imaging time point information, that is a time point at which the frame image is captured, to the frame image, and the feature value is calculated based on a lightness value of each of a plurality of consecutive frame images captured in a specific period, among a plurality of time-series frame images included in the swallowing block or among a plurality of time-series frame images not included in the swallowing block ([0061] the swallowing count unit 47 counts the number of swallows when a non-swallowing video appears after a swallowing video, and the duration of the non-swallowing video exceeds a certain period of time. In other words, if the period of non-swallowing video footage exceeds a certain duration after the period of continuous swallowing video footage (the time during which swallowing video footage was acquired), the swallowing video footage immediately preceding the period of non-swallowing video footage exceeding that duration is determined to represent a single swallowing action).
As to claim 13, IGO teaches the examination video is obtained by imaging the observation target in a case in which a certain amount of water is swallowed in the video endoscopic examination of swallowing([0032], Swallowing refers to the series of actions involved in putting food or drink into the mouth, chewing it, swallowing it, and sending it down the esophagus.).
As to claim 14, IGO teaches the examination video is obtained by imaging the observation target in a case in which a certain amount of swallowing food is swallowed in the video endoscopic examination of swallowing ([0032]) Swallowing refers to the series of actions involved in putting food or drink into the mouth, chewing it, swallowing it, and sending it down the esophagus Figure 4 illustrates normal swallowing, and Figure 5 illustrates abnormal swallowing (aspiration). As shown in Figure 4, swallowing can be divided into three phases: the "oral phase," in which food F is mainly transported from the oral cavity to the pharynx by the movement of the tongue (To); the "pharyngeal phase," in which food F is transported from the pharynx to the esophagus (Es) by the swallowing reflex).
As to claim 17, IGO teaches the control processor is configured to perform control of displaying the feature value on a display([0022]The processor device 15 performs system control of the endoscope system 10 and image processing on image signals transmitted from the endoscope 12. The display 18 is a display unit that displays images captured by the endoscope 12.)
Claim 18 is rejected the same as claim 1 except claim 18 is directed to a method claim. The rejection of claim 1 includes all the limitations of claim 18. Thus, argument analogous to that presented above for claim 1 is applicable to claim 18
As to claim 19, IGO teaches A non-transitory computer readable medium for storing a computer-executable program, the computer-executable program causing a computer to implement ([0070] Various types of processors include CPUs (Central Processing Units), which are general purpose processors that execute software (programs) and function as various processing units);
regarding the remaining limitation of claim 19, all the remaining limitation are rejected the same as claim 1 except claim 19 is directed to a computer program claim. The rejection of claim 1 includes all the remaining limitations of claim 19. Thus, argument analogous to that presented above for claim 1 is applicable to the remaining limitations of claim 19
Claim 20 is rejected the same as claim 1 except claim 20 is directed to a diagnosis support apparatus claim. The rejection of claim 1 includes all the limitations of claim 20. Thus, argument analogous to that presented above for claim 1 is applicable to claim 20
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
4. Claims 5 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over IGO, JP2022179218 A, pub in view of LI, Hui et al., (hereafter LI), CN 102262079 A, pub. 11/13/2011
As to claim 5, IGO teaches the feature value 0059]-[0061]) The swallowing count unit 47 counts the number of swallows when a non-swallowing video appears after a swallowing video, and the duration of the non-swallowing video exceeds a certain period of time).
However, it is noted that IGO does not specifically teach the underline section of the limitation “the feature value is a basic statistic based on the number of the frame images.
On the other hand LI teaches the feature value is a basic statistic based on the number of the frame images included in each of the swallowing blocks (Claim 6. finding out the position coordinate of the light point in each frame picture and recording, b) drawing the cell outline, recording the coordinate and set cell nucleus center is point 0 c) calculated light point in each frame image relative to the average value S of the point distance of 0, cell contour line of each coordinate point 0 mean value of distance L, calculating the value of each frame S/L, it is defined as the opposite in swallowing distance, d) converting the frame number is time, relative distance and time to obtain relative in throughput curve.)
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the swallowing frame-counting teachings of IGO with the relative light point and cell contour distance calculations (S/L) of LI. The motivation for this combination would be to allow a user of IGO to use LI’s calculations as a precise, quantitative trigger or threshold to accurately identify, filter, and count the relevant swallowing frames in IGO’s system.
Regarding claim 7, the combination of IGO and LI teaches the feature value is a basic statistic based on the number of the frame images (this limitation is discussed in claim 5 above)in which the stage information is the initial swallowing-in-progress stage in the frame images included in the swallowing block (this limitation is discussed in claim 6 above)
5. Claims 10-11and 16 rejected under 35 U.S.C. 103 as being unpatentable over IGO, JP2022179218 A, in view of Kagaya et al., ( hereafter Kagaya), US 9801530 B2, pub. 01/31/2017.
Regarding claim 10, while IGO teaches frame image in which the stage information is the initial swallowing-in-progress stage ( this limitation discussed in claim 6 above) but files to teach the limitation “control processor is configured to calculate an area of a halation region in the frame image”
On the other hand , in the same field of endeavor the endoscope apparatus configured to controls the amount of illuminating light in accordance with the brightness of a frame disclosed by Kagaya teaches the control processor is configured to calculate an area of a halation region in the frame image, and the feature value is the area of the halation region (col.30 line 50- col.30 line 5, an area of the region of halation may be taken as the sum total of all regions of halation that have arisen on a frame image, or may be taken as a maximum value among the areas of respective halation regions that arise at separate places. Further, in a case where the area of a halation region is equal to or greater than the predetermined threshold value, the edging phenomenon detection processing in step S10 is not performed. Further regions of halation (high luminance regions larger than a predetermined luminance value) can be excluded from a target region for detection of an edge or a droplet image).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate Kagaya's method of calculating the area of a halation region into IGO's system for counting swallowing image frames. The suggestion and motivation to combine these teachings allows IGO's system to automatically identify and filter out corrupted or unreadable frames, ensuring that the final swallowing count is based only on high-quality, diagnostically viable data.
Regarding claim 11, while IGO teaches the frame image in which the stage information is the post-swallowing stage ( this limitation discussed in claim 6 above) but files to teach the limitation “calculate a lightness value in the frame image; and the feature value is the lightness value”
On the other hand Kagaya teaches the control processor is configured to calculate a lightness value in the frame image and the feature value is the lightness value(col.30 line 50- col.30 line 5, in a case where the image pickup unit 54 is brought close to an object of a site to be observed and is performing imaging, or in a case where the zoom magnification ratio of the optical zoom in the image pickup unit 54 is large, as shown in FIG. 32, halation (a high luminance region) 210 of a large area arises in a frame image that is imaged by the image pickup unit 54).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate Kagaya's method of calculating the area of a halation region into IGO's system for counting swallowing image frames. The suggestion and motivation to combine these teachings allows IGO's system to automatically identify and filter out corrupted or unreadable frames, ensuring that the final swallowing count is based only on high-quality, diagnostically viable data.
As to claim 16, Kagaya teaches receive designation of a start point and an end point for calculating the feature value; and calculate the feature value for a section defined based on the start point and the end point in the examination video (claims 1 and 7, An endoscope apparatus, comprising: an image pickup device in which a plurality of pixel sensors are arranged in a matrix shape, and which starts sequential exposure by at least one of pixel rows with respect to the plurality of pixel sensors to generate image data of a frame image, and outputs image data for each of the pixel rows in an order of starting exposure. The high luminance region extraction device that extracts a high luminance region having a luminance value that is greater than a predetermined luminance value from within the frame image; and a high luminance region determination device that determines whether or not the area of the high luminance region is equal to or greater than a predetermined area ).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate Kagaya's method of calculating the area of a halation region into IGO's system for counting swallowing image frames. The suggestion and motivation to combine these teachings allows IGO's system to automatically identify and filter out corrupted or unreadable frames, ensuring that the final swallowing count is based only on high-quality, diagnostically viable data.
6. Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over IGO, JP2022179218 A, pub in view TERAMURA, Yuichi ( hereafter ERAMURA), EP4093025A1, pub. 11/23/2022.
Regarding claim 15, while IGO teaches swallowing food to be swallowed( this limitation discussed in claim above) but files to teach the limitation “calculate a different feature value depending on a type of the swallowing food to be swallowed.”
On the other hand TERAMURA teaches calculate a different feature value depending on a type of the swallowing food to be swallowed( Page 10 4th par., the types of swallowing examination foods are classified into saliva (sa), milk (mi), colored water (cw), pudding (pu), and unknown (un). In addition, the results of swallowing are classified into normal swallowing (S), aspiration (A), and unknown swallowing (U). Further, (1) is added to the end of the endoscopic index moving image 44, and (2) is added to the end of the ultrasound index moving image 54, depending on the type of moving image).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the swallowing classification method taught by TERAMURA (distinguishing normal versus unknown swallowing) into the image-data-based swallowing method of IGO.
The motivation for this combination would be to allow clinicians using IGO’s method to instantly filter out healthy swallowing patterns, thereby permitting them to focus their attention strictly on anomalous or "unknown" cases that require immediate medical intervention.
Prior arts are not used in rejections but pertinent to the claims or disclosure.
a.. “Detection of aspiration from images of a video fluoroscopic swallowing study adopting deep learning” Oral Radiology (2023) 39:553–562, Published online: 8 February 2023, by Yukihiro Iida et al., disclosed:
A videofluoroscopic swallowing study (VFSS) is conducted to detect aspiration. However, aspiration occurs within a short time and is difficult to detect. If deep learning can detect aspirations with high accuracy, clinicians can focus on the diagnosis of the detected ( see Abstract).
PNG
media_image1.png
362
710
media_image1.png
Greyscale
b. “Can We Reduce Frame Rate to 15 Images per Second in Pediatric Videofluoroscopic Swallow Studies?”, Dysphagia (2020) 35:296–300, to Julie Layly disclosed:
Videofluoroscopic Swallow studies (VFSS) are useful radiological examinations to explore swallowing disorders but which require ionizing radiation. The aim of our study was to evaluate the comparability of pediatric VFSS at 15 frames per second (fps) with 30 fps. Fifty-five loops including 190 swallowings of VFSS at 30 fps performed on 32 consecutive pediatric patients in a University Hospital Center were retrospectively modified by a software to delete one image out of two to obtain secondary loops with a frame rate of 15 fps. An otorhinolaryngologist-phonatrician and a radiologist reviewed all swallowings blindly and randomly using the penetration and aspiration scale (PAS). In case of discordance, they concluded a consensual interpretation. Fifteen girls and seventeen boys were included. The median age was 4 years and 8 months (range = 4 months–16 yr.). 144 swallowings were normal. Swallowing disorder was confirmed in 46 swallowings, (23 supra glottic penetrations and 23 aspirations). See Abstract,
Contact Information
Any inquiry concerning this communication or earlier communication from the examiner should be directed to Mekonen Bekele whose telephone number is (469) 295-9077.The examiner can normally be reached on Monday -Friday from 9:00AM to 6:50 PM Eastern Time.
If attempt to reach the examiner by telephone are unsuccessful, the examiner’s supervisor Eng, George can be reached on (571) 272-7495.The fax phone number for the organization where the application or proceeding is assigned is 571-237-8300. Information regarding the status of an application may be obtained from the patent Application Information Retrieval (PAIR) system. Status information for published application may be obtained from either Private PAIR or Public PAIR.
Status information for unpublished application is available through Privet PAIR only.
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have question on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866.217-919 (tool-free)
/MEKONEN T BEKELE/Primary Examiner, Art Unit 2699
Prosecution Insights