DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-18 is/are rejected under 35 U.S.C. 103 (in a single reference art rejection) as being unpatentable over de Groen (Using artificial intelligence to improve adequacy of inspection in gastrointestinal endoscopy, 2020, Techniques and Innovations in Gastrointestinal Endoscopy, Vol. 22, Pages 71-79).
Claim 1 is met by de Groen, wherein
de Groen discloses:
An inspection assistance device comprising a processor (See the Abstract.), wherein the processor is configured to:
…
acquire the inspection requirement information from the storage medium (See the Abstract: “Real-time feedback provides endoscopists the option, when measurements suggest inadequate or incomplete inspection, to change technique or repeat and expand inspection.” Also see page 74, section 3: “(1) Clean mucosa; (2) Look Everywhere; and (3) Abnormality Removal [19]. Adequate mucosal inspection involves the first 2 components of CLEAR: cleaning the mucosa and then looking everywhere. AI applications that will improve adequacy of inspection will have to address these 2 components of CLEAR.” Finally see page 78, section 5: “Real-time analysis provides the opportunity for real-time feedback: to inform the endoscopist about the quality of the ongoing endoscopy, and in particular whether inspection of the mucosa is adequate.” To make the determination that an inspection is adequate/complete, acquiring of an inspection protocol (“inspection requirement information”) from the workstations of Fig. 5 (“from the storage medium”) is implied.);
acquire inspection state information indicating an inspection state by confirming a state of an inspection device used to inspect the inspection target (See page 74, section 2.4: “one should have a basic understanding of the complexity of a computer-based system that analyzes a video stream within milli seconds”. Then see page 73, section 2.2: “Before features of interest, such as adequacy of inspection, are extracted from a streaming video file…”.);
compare the inspection requirement information with the inspection state information (See page 74, section 2.4: “feeds the results from one algorithm into another algorithm, and eventually combines all the results in a decision module, where decisions are made to yes/no provide feedback, and if feedback will be provided about which feature and in what format.”);
generate comparison result information indicating a result of comparing the inspection requirement information with the inspection state information (See the Abstract: “Many endoscopic features, representing skillset and effort, such as clarity of image, absence of stool, looking sideways, performing retroflexion and obtaining all possible mucosal views, can be classified or counted”. Then see the caption of Fig. 2, describing one example of comparing video stream data to an inspection protocol: “An algorithm counts the number of times the tip of the endoscope passes through all four quadrants at a specific deflexion from the center.”); and
output the comparison result information (See the overlay in Fig. 5.).
de Groen does not explicitly disclose the following:
connect to a storage medium that stores inspection requirement information indicating an inspection requirement determined in advance for each inspection target;
However, on page 72, de Groen states: “As mentioned, it is unlikely that an AI system can be created that will work in every endoscopy unit without some adjustments for type of endoscopic instrument, light source, imaging method, and endoscopy protocol used [19].” The examiner asserts that with this passage, de Groen gives a suggestion (and motivation) that the described inspection/feedback AI can be adjusted for specific endoscopy protocols to be applied broadly to endoscopy systems. This concept, in the workstation of Fig. 5, is implied to read/write to a memory and meets “connect to a storage medium that stores inspection requirement information indicating an inspection requirement determined in advance for each inspection target”. For these reasons, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify de Groen in this way to reach the claimed invention.
Claim 2 is met by de Groen, wherein
de Groen further discloses:
The inspection assistance device according to claim 1, wherein the inspection target includes an inspection portion, and wherein the processor is configured to compare the inspection requirement information with the inspection state information when inspection portion information indicating the inspection portion is accepted (See Fig. 2, quadrants and circumferential inspections meet the claimed “inspection portion” and “inspection portion information”. Then see Fig. 5.).
Claim 3 is met by de Groen, wherein
de Groen further discloses:
The inspection assistance device according to claim 1, wherein the inspection target includes an inspection portion (See Fig. 2, quadrants and circumferential inspections meet the claimed “inspection portion”.) and
de Groen does not explicitly disclose the following:
wherein the processor is configured to compare the inspection requirement information with the inspection state information when a folder associated with the inspection portion is set.
However, on page 74, section 2.4, de Groen states: “After one has captured the video, performed the basic analysis, selected the desired features and created a large corpus of annotated video files with adequate representation of the desired features, the deep learning process can start.” Since a workstation (see Fig. 5) is used, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to add user input of a folder to select those desired features to reach the claimed “compare the inspection requirement information with the inspection state information when a folder associated with the inspection portion is set”.
Claim 4 is met by de Groen, wherein
de Groen further discloses:
The inspection assistance device according to claim 1, wherein the inspection target includes a first inspection portion and a second inspection portion (See Fig. 2, quadrants and circumferential inspections meet the claimed “inspection portion”.), and
de Groen does not explicitly disclose the following:
wherein the processor is configured to compare the inspection requirement information with the inspection state information when a first folder associated with the first inspection portion is set and then a second folder associated with the second inspection portion is set instead of the first folder.
However, on page 74, section 2.4, de Groen states: “After one has captured the video, performed the basic analysis, selected the desired features and created a large corpus of annotated video files with adequate representation of the desired features, the deep learning process can start.” Since a workstation (see Fig. 5) is used, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to add user input of a first folder and a second folder to select individual desired features to reach the claimed “compare the inspection requirement information with the inspection state information when a first folder associated with the first inspection portion is set and then a second folder associated with the second inspection portion is set instead of the first folder”.
Claim 5 is met by de Groen, wherein
de Groen further discloses:
The inspection assistance device according to claim 1, wherein,
de Groen does not explicitly disclose the following:
when a user designates the inspection requirement information, the inspection requirement information designated by the user is stored on the storage medium.
However, on page 72, de Groen states: “As mentioned, it is unlikely that an AI system can be created that will work in every endoscopy unit without some adjustments for type of endoscopic instrument, light source, imaging method, and endoscopy protocol used [19].” The examiner asserts that with this passage, de Groen gives a suggestion (and motivation) that the described inspection/feedback AI can be adjusted for specific endoscopy protocols to be applied broadly to endoscopy systems. This concept, in the workstation of Fig. 5, would accept user input of a protocol and meets “when a user designates the inspection requirement information, the inspection requirement information designated by the user is stored on the storage medium”. For these reasons, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify de Groen in this way to reach the claimed invention.
Claim 6 is met by de Groen, wherein
de Groen further discloses:
The inspection assistance device according to claim 1, wherein the inspection requirement information indicates that it is necessary to acquire position information indicating a position at which the inspection is to be performed, and the processor is configured to acquire information indicating whether the inspection assistance device is capable of acquiring the position information as the inspection state information (See Table 2 on page 73, e.g., “1 min/cm in circumferential Barrett’s epithelium”.).
Claim 7 is met by de Groen, wherein
de Groen further discloses:
The inspection assistance device according to claim 1, wherein the inspection requirement information indicates a state of an instrument used in the inspection device, and the processor is configured to acquire device information indicating the state of the instrument as the inspection state information (See the paragraph bridging the left and right columns on page 75: “Therefore, the distance between the center of the (endoscope) image and the location of the center of the lumen in the image is directly related to the amount of tip deflection away from the central colon axis. By measuring this distance away from the center of the lumen over the entire 360 degrees or circumference in 2D images, one measures whether the tip of the endoscope was circumferentially deflected in the actual 3D colon.”).
Claim 8 is met by de Groen, wherein
de Groen further discloses:
The inspection assistance device according to claim 7, wherein the inspection device includes an endoscope device having an insertion unit with an image sensor disposed at a distal end of the insertion unit (See Fig. 6 and the paragraph bridging the left and right columns on page 75: “Therefore, the distance between the center of the (endoscope) image and the location of the center of the lumen in the image is directly related to the amount of tip deflection away from the central colon axis. By measuring this distance away from the center of the lumen over the entire 360 degrees or circumference in 2D images, one measures whether the tip of the endoscope was circumferentially deflected in the actual 3D colon.”),
de Groen does not explicitly disclose the following:
wherein the inspection requirement information indicates a type of optical adaptor attached to the distal end, and wherein the processor acquires the device information indicating the type of optical adaptor.
However, on page 72, de Groen states: “As mentioned, it is unlikely that an AI system can be created that will work in every endoscopy unit without some adjustments for type of endoscopic instrument, light source, imaging method, and endoscopy protocol used [19].” The examiner asserts that with this passage, de Groen gives a suggestion (and motivation) that the described inspection/feedback AI can be adjusted for types of endoscopic instruments/components and imaging methods/devices to be applied broadly to endoscopy systems. Some indication in the workstation of Fig. 5 of these types would fall within this concept, which meets “wherein the inspection requirement information indicates a type of optical adaptor attached to the distal end, and wherein the processor acquires the device information indicating the type of optical adaptor”. For these reasons, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify de Groen in this way to reach the claimed invention.
Claim 9 is met by de Groen, wherein
Modified de Groen further discloses:
The inspection assistance device according to claim 8, wherein the type of optical adaptor corresponds to the optical adaptor used when measurement is executed (See the treatment of claim 8. Initialization of the modified de Groen system would indicate the type of optical adaptor, which would be used in the subsequent inspection.).
See the motivation to combine in the treatment of claim 8.
Claim 10 is met by de Groen, wherein
de Groen further discloses:
The inspection assistance device according to claim 7, wherein the inspection device includes an endoscope device having an insertion unit with an image sensor disposed at a distal end of the insertion unit (See Fig. 6 and the paragraph bridging the left and right columns on page 75: “Therefore, the distance between the center of the (endoscope) image and the location of the center of the lumen in the image is directly related to the amount of tip deflection away from the central colon axis. By measuring this distance away from the center of the lumen over the entire 360 degrees or circumference in 2D images, one measures whether the tip of the endoscope was circumferentially deflected in the actual 3D colon.”),
de Groen does not explicitly disclose the following:
wherein the inspection requirement information indicates a type of insertion unit, and wherein the processor is configured to acquire the device information indicating the type of insertion unit.
However, on page 72, de Groen states: “As mentioned, it is unlikely that an AI system can be created that will work in every endoscopy unit without some adjustments for type of endoscopic instrument, light source, imaging method, and endoscopy protocol used [19].” The examiner asserts that with this passage, de Groen gives a suggestion (and motivation) that the described inspection/feedback AI can be adjusted for types of endoscopic instruments/components to be applied broadly to endoscopy systems. Some indication in the workstation of Fig. 5 of these types would fall within this concept, which meets “wherein the inspection requirement information indicates a type of insertion unit, and wherein the processor is configured to acquire the device information indicating the type of insertion unit”. For these reasons, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify de Groen in this way to reach the claimed invention.
Claim 11 is met by de Groen, wherein
Modified de Groen further discloses:
The inspection assistance device according to claim 10, wherein, when the device information is acquired, the processor is configured to compare the inspection requirement information with the inspection state information (See the treatment of claim 10. The comparison described in Fig. 5 and page 74, section 2.4 (“feeds the results from one algorithm into another algorithm, and eventually combines all the results in a decision module, where decisions are made to yes/no provide feedback, and if feedback will be provided about which feature and in what format.) would be performed after device initialization.).
See the motivation to combine in the treatment of claim 10.
Claim 12 is met by de Groen, wherein
de Groen further discloses:
The inspection assistance device according to claim 1, wherein the inspection device includes an endoscope device having an insertion unit with an image sensor disposed at a distal end of the insertion unit (See Fig. 6 and the paragraph bridging the left and right columns on page 75: “Therefore, the distance between the center of the (endoscope) image and the location of the center of the lumen in the image is directly related to the amount of tip deflection away from the central colon axis. By measuring this distance away from the center of the lumen over the entire 360 degrees or circumference in 2D images, one measures whether the tip of the endoscope was circumferentially deflected in the actual 3D colon.”).
Claim 13 is met by de Groen, wherein
de Groen further discloses:
The inspection assistance device according to claim 12, wherein the inspection device includes an auxiliary device connected to the endoscope device or the inspection target (See the workstation in Fig. 5.).
Claim 14 is met by de Groen, wherein
de Groen further discloses:
The inspection assistance device according to claim 1, wherein the storage medium is configured to store two or more pieces of the inspection requirement information (See Table 2 and page 74, section 3: “(1) Clean mucosa; (2) Look Everywhere; and (3) Abnormality Removal [19]. Adequate mucosal inspection involves the first 2 components of CLEAR: cleaning the mucosa and then looking everywhere. AI applications that will improve adequacy of inspection will have to address these 2 components of CLEAR.”), and wherein the processor is configured to: acquire the two or more pieces of the inspection requirement information from the storage medium (See the Abstract: “Real-time feedback provides endoscopists the option, when measurements suggest inadequate or incomplete inspection, to change technique or repeat and expand inspection.” Also see page 74, section 3: “(1) Clean mucosa; (2) Look Everywhere; and (3) Abnormality Removal [19]. Adequate mucosal inspection involves the first 2 components of CLEAR: cleaning the mucosa and then looking everywhere. AI applications that will improve adequacy of inspection will have to address these 2 components of CLEAR.” Finally see page 78, section 5: “Real-time analysis provides the opportunity for real-time feedback: to inform the endoscopist about the quality of the ongoing endoscopy, and in particular whether inspection of the mucosa is adequate.” To make the determination that an inspection is adequate/complete, acquiring of an inspection protocol (“inspection requirement information”) from the workstations of Fig. 5 (“from the storage medium”) is implied.); acquire two or more pieces of the inspection state information (See page 74, section 2.4: “one should have a basic understanding of the complexity of a computer-based system that analyzes a video stream within milli seconds”. Then see page 73, section 2.2: “Before features of interest, such as adequacy of inspection, are extracted from a streaming video file…”.); and compare each of the two or more pieces of the inspection requirement information with each of the two or more pieces of the inspection state information (See page 74, section 2.4: “feeds the results from one algorithm into another algorithm, and eventually combines all the results in a decision module, where decisions are made to yes/no provide feedback, and if feedback will be provided about which feature and in what format.”).
Claim 15 is met by de Groen, wherein
de Groen further discloses:
The inspection assistance device according to claim 1, wherein the processor is configured to output the comparison result information to a display (See the overlay in Fig. 5.).
Claim 16 is met by de Groen, wherein
de Groen further discloses:
The inspection assistance device according to claim 15, wherein the comparison result information includes the inspection requirement information or the inspection state information (See the four green corners and spiral score in Fig. 5.).
Claim 17 is met by de Groen for the reasons given in the treatment of claim 1.
Claim 18 is met by de Groen for the reasons given in the treatment of claim 1.
Contact
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JONATHAN S LEE whose telephone number is (571)272-1981. The examiner can normally be reached 11:30 AM - 7:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached at (571)270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Jonathan S Lee/Primary Examiner, Art Unit 2677
