Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1 – 4, 7 – 14, 16 and 18 – 20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Amthor et al. (US Patent Application Publication 2022/0076395, IDS), hereinafter referred as Amthor.
Regarding claim 1, Amthor discloses a method of updating training of a sample characterization algorithm of a diagnostic laboratory system, the method comprising (Fig. 1, [0018] "For example, the raw images. can respectively show a sample carrier with two or more microscopic samples. A classification comprises the classes, e.g., sample carrier, sample and background, whereby the two microscopic samples belong to the same class."):
providing an imaging device in the diagnostic laboratory system ([0018] "A semantic segmentation corresponds to a pixel-by-pixel classification of structures or objects in the raw image. Each image pixel of a raw image is thus assigned to one of a plurality of predetermined classes of objects. In the case of an instance segmentation machine learning model, a pixel-by-pixel classification also occurs and, in addition, a plurality of objects of the same class are differentiated."), wherein the imaging device is controllably movable within the diagnostic laboratory system (Fig. 1, [0061]);
capturing a first image within the diagnostic laboratory system using the imaging device, the first image captured with an imaging condition ([0022] "For example, a coefficient table (lookup table or in general a database) can be provided, which contains coefficients for different image capture parameters and different semantic labels. Upon receipt of information regarding Image capture parameters of the raw images, it is possible to read out the coefficients set for the semantic regions in the raw images from the coefficient table.");
performing an annotation of the first image using an annotation generator of the diagnostic laboratory system to generate a first annotated image ([0016] "Coefficients can then be derived for each of the raw images depending on at least the semantic labels of included semantic regions. A semantic label is understood to denote a meaning of the image content, i.e. of one or more depicted objects, assigned to the corresponding region. The semantic labels make it possible to differentiate between a plurality of predefined classes ..."); and
updating training of the annotation generator using the first annotated image (Fig. 3, [0078] "The model M2 is trained to perform an evaluation of input image data depending on a semantic label. For this purpose, the model M2 can have been trained with training data which comprises respective image data with associated labelling of an image quality for different semantic labels. For example, the training data for the semantic label "sample" can comprise a plurality of images or image sections of samples, a user having specified a quality evaluation manually beforehand. As described above, different evaluation criteria can be preferred for the different semantic areas. The model M2 can accordingly be trained with training data in which image data relating to different semantic labels has been evaluated in different ways. The model M2 can also exploit contextual data i in the process. The contextual data i can indicate, for example, a sample type, it being known for different sample types whether the best Image quality Is more likely to be achieved in brighter or darker images").
Regarding claim 2 (depends on claim 1), Amthor discloses the method further comprising:
altering the imaging condition to an altered imaging condition ([0050] "An evaluation can be carried out regarding whether a further image should be captured for one of the sections with modified Image capture parameters. For example, a semantic segmentation and subsequent evaluation of the segmented regions can occur according to the image quality criteria mentioned above; if this evaluation reveals an inadequate quality (e.g. an underdrive in a sample area), then this area can be captured again with modified image capture parameters, e.g., with a higher illumination Intensity.");
capturing a second image within the diagnostic laboratory system using the imaging device with the altered imaging condition (Figs. 3 - 4; [0075 - 0085]);
performing an annotation of the second image using the annotation generator to generate a second annotated image (Figs. 3 - 4; [0075 - 0085]); and
updating the training of the annotation generator using the second annotated image (Fig. 3, [0078] "The model M2 is trained to perform an evaluation of input image data depending on a semantic label. For this purpose, the model M2 can have been trained with training data which comprises respective image data with associated labelling of an image quality for different semantic labels. For example, the training data for the semantic label "sample" can comprise a plurality of images or image sections of samples, a user having specified a quality evaluation manually beforehand. As described above, different evaluation criteria can be preferred for the different semantic areas. The model M2 can accordingly be trained with training data in which image data relating to different semantic labels has been evaluated in different ways. The model M2 can also exploit contextual data i in the process. The contextual data i can indicate, for example, a sample type, it being known for different sample types whether the best Image quality Is more likely to be achieved in brighter or darker images").
Regarding claim 3 (depends on claim 2), Amthor discloses the method wherein the first image and the second image include a holding location for a sample container (Fig. 1, [0061]).
Regarding claim 4 (depends on claim 2), Amthor discloses the method wherein the first image and the second image include a sample container (Figs.1 - 2; [0065]).
Regarding claim 7 (depends on claim 4), Amthor discloses the method wherein the imaging condition is pose of the imaging device relative to the sample container ([0038, 0061 - 0063, 0087]).
Regarding claim 8 (depends on claim 4), Amthor discloses the method wherein the imaging condition is a position of the imaging device relative to the sample container ([0038, 0061 - 0063, 0087]).
Regarding claim 9 (depends on claim 1), Amthor discloses the method wherein the imaging condition is intensity of illumination within the diagnostic laboratory system ([0038]).
Regarding claim 10 (depends on claim 1), Amthor discloses the method wherein the annotation is a bounding box or a pixelwise mask of an object in the first image ([0040]).
Regarding claim 11 (depends on claim 1), Amthor discloses the method wherein the annotation is one or more properties of a sample container in an image ([0022] "For example, the image capture parameters can differ with respect to an illumination intensity of a light source of the microscope. For example, it can be known that sample areas are best rendered with a high illumination intensity, while image areas of (cover) glass edges oversaturate quickly and are thus more clearly visible with a lower illumination intensity; for sample carrier areas characterized by the absence of a sample, on the other hand, a medium or low illumination intensity can be preferred in order to suppress a depiction of scratches, dirt or other undesired or irrelevant details.").
Regarding claim 12 (depends on claim 11), Amthor discloses the method wherein the one or more properties includes sample container orientation with respect to a holding location for a sample handler (Fig. 1, [0061 – 0063]).
Regarding claim 13 (depends on claim 11), Amthor discloses the method wherein the one or more properties include at least one of geometry of at least one portion of the sample container, sample container height, sample container diameter, characteristics of a liquid in the sample container, and sample container identification indicia ([0040]).
Regarding claim 14, Amthor discloses a method of training a sample characterization algorithm of a diagnostic laboratory system (Fig. 1, [0018] "For example, the raw Images can respectively show a sample carrier with two or more microscopic samples. A classification comprises the classes, e.g., sample carrier, sample and background, whereby the two microscopic samples belong to the same class."), the method comprising:
providing an imaging device in the diagnostic laboratory system ([0018] "A semantic segmentation corresponds to a pixel-by-pixel classification of structures or objects In the raw Image. Each Image pixel of a raw Image is thus assigned to one of a plurality of predetermined classes of objects. In the case of an Instance segmentation machine learning model, a pixel-by-pixel classification also occurs and, in addition, a plurality of objects of the same class are differentiated."), wherein the imaging device is controllably movable within the diagnostic laboratory system (Fig. 1, [0061]);
capturing a first image of a sample container using the imaging device, the first image captured with a first imaging condition ([0022] "For example, a coefficient table (lookup table or in general a database) can be provided, which contains coefficients for different image capture parameters and different semantic labels. Upon receipt of information regarding Image capture parameters of the raw images, it is possible to read out the coefficients set for the semantic regions in the raw images from the coefficient table.");
performing an annotation of the first image to generate a first annotated image ([0016] "Coefficients can then be derived for each of the raw images depending on at least the semantic labels of included semantic regions. A semantic label is understood to denote a meaning of the image content, i.e. of one or more depicted objects, assigned to the corresponding region. The semantic labels make it possible to differentiate between a plurality of predefined classes ...");
altering the first imaging condition to a second imaging condition ([0050] "An evaluation can be carried out regarding whether a further image should be captured for one of the sections with modified Image capture parameters. For example, a semantic segmentation and subsequent evaluation of the segmented regions can occur according to the image quality criteria mentioned above; if this evaluation reveals an inadequate quality (e.g. an underdrive in a sample area), then this area can be captured again with modified image capture parameters, e.g., with a higher illumination Intensity.");
capturing a second image of the sample container using the imaging device with the second imaging condition (Figs. 3-4; [0075 - 0085]);
performing the annotation of the second image to generate a second annotated image (Figs. 3-4; [0075 - 0085]);
training an annotation generator of the diagnostic laboratory system using at least the first annotated image and the second annotated image (Fig. 3, [0078] "The model M2 is trained to perform an evaluation of input image data depending on a semantic label. For this purpose, the model M2 can have been trained with training data which comprises respective image data with associated labelling of an image quality for different semantic labels. For example, the training data for the semantic label "sample" can comprise a plurality of images or image sections of samples, a user having specified a quality evaluation manually beforehand. As described above, different evaluation criteria can be preferred for the different semantic areas. The model M2 can accordingly be trained with training data in which image data relating to different semantic labels has been evaluated in different ways. The model M2 can also exploit contextual data i in the process. The contextual data i can indicate, for example, a sample type, it being known for different sample types whether the best Image quality Is more likely to be achieved in brighter or darker images");
altering the second imaging condition to a third imaging condition ([0050] "An evaluation can be carried out regarding whether a further image should be captured for one of the sections with modified Image capture parameters. For example, a semantic segmentation and subsequent evaluation of the segmented regions can occur according to the image quality criteria mentioned above; if this evaluation reveals an inadequate quality (e.g. an underdrive in a sample area), then this area can be captured again with modified image capture parameters, e.g., with a higher illumination Intensity.");
capturing a third image of the sample container using the imaging device with the third imaging condition (Figs. 3 - 4; [0075 - 0085]);
performing the annotation of the third image using the annotation generator to generate a third annotated image (Figs. 3 - 4; [0075 - 0085]); and
further training the annotation generator using at least the third annotated image (Fig. 3, [0078] "The model M2 is trained to perform an evaluation of input image data depending on a semantic label. For this purpose, the model M2 can have been trained with training data which comprises respective image data with associated labelling of an image quality for different semantic labels. For example, the training data for the semantic label "sample" can comprise a plurality of images or image sections of samples, a user having specified a quality evaluation manually beforehand. As described above, different evaluation criteria can be preferred for the different semantic areas. The model M2 can accordingly be trained with training data in which image data relating to different semantic labels has been evaluated in different ways. The model M2 can also exploit contextual data i in the process. The contextual data i can indicate, for example, a sample type, it being known for different sample types whether the best Image quality Is more likely to be achieved in brighter or darker images").
Regarding claim 16 (depends on claim 14), Amthor discloses the method wherein the first imaging condition is a first intensity of illumination illuminating the sample container during capturing the first image, the second imaging condition is a second intensity of illumination illuminating the sample container during capturing the second image, and the third imaging condition is a third intensity of illumination illuminating the sample container during capturing the third image (Figs. 3-4; [0075 - 0085]).
Regarding claim 18 (depends on claim 14), Amthor discloses the method wherein the first imaging condition is a first pose of the imaging device relative to the sample container during capturing the first image, the second imaging condition is a second pose of the imaging device relative to the sample container during capturing the second image, and the third imaging condition is a third pose of the imaging device relative to the sample container during capturing the third image ([0038, 0061 - 0063, 0087]).
Regarding claim 19 (depends on claim 14), Amthor discloses the method wherein the annotation is a bounding box or a pixelwise mask of the sample container ([0040]).
Regarding claim 20, Amthor discloses a diagnostic laboratory system (Fig. 1) comprising:
an imaging device controllably movable within the diagnostic laboratory system (Fig. 1, [0061]), wherein the imaging device is configured to capture images within the diagnostic laboratory system under different imaging conditions (Figs. 3-4; [0085]);
a processor coupled to the imaging device ([0052]);
a memory coupled to the processor, wherein the memory includes an annotation generator trained to annotate images captured by the imaging device, the processor further including computer program code that, when executed by the processor ([0052-0053]), causes the processor to:
receive first image data of a first image captured by the imaging device using at least one imaging condition ([0022] "For example, a coefficient table (lookup table or in general a database) can be provided, which contains coefficients for different image capture parameters and different semantic labels. Upon receipt of information regarding Image capture parameters of the raw images, it is possible to read out the coefficients set for the semantic regions in the raw images from the coefficient table.");
cause the annotation generator to perform an annotation of the first image to generate a first annotated image ([0016] "Coefficients can then be derived for each of the raw images depending on at least the semantic labels of included semantic regions. A semantic label is understood to denote a meaning of the image content, i.e. of one or more depicted objects, assigned to the corresponding region. The semantic labels make it possible to differentiate between a plurality of predefined classes ... "); and
update training of the annotation generator using the first annotated mage (Fig. 3, [0078] "The model M2 is trained to perform an evaluation of input image data depending on a semantic label. For this purpose, the model M2 can have been trained with training data which comprises respective image data with associated labelling of an image quality for different semantic labels. For example, the training data for the semantic label "sample" can comprise a plurality of images or image sections of samples, a user having specified a quality evaluation manually beforehand. As described above, different evaluation criteria can be preferred for the different semantic areas. The model M2 can accordingly be trained with training data in which image data relating to different semantic labels has been evaluated in different ways. The model M2 can also exploit contextual data i in the process. The contextual data i can indicate, for example, a sample type, it being known for different sample types whether the best Image quality Is more likely to be achieved in brighter or darker images").
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 5, 6, 15 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Amthor in view of Finkbeiner et al. (US Patent Application Publication 2015/0278625, IDS), hereinafter referred as Finkbeiner.
Regarding claim 5 (depends on claim 4), Amthor fails to explicitly disclose the method further comprising providing a robot comprising a gripper, wherein providing the imaging device comprises affixing the imaging device to the robot, and further comprising gripping the sample container during the capturing of the first image.
However, in a similar field of endeavor Finkbeiner discloses an automated robotic microscopy systems (abstract). In addition, Finkbeiner discloses wherein providing a robot comprising a gripper, wherein providing the imaging device comprises affixing the imaging device to the robot, and further comprising gripping the sample container during the capturing of the first image (Fig. 3, [0102 - 0103]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Amthor, and providing a robot comprising a gripper, wherein providing the imaging device comprises affixing the imaging device to the robot, and further comprising gripping the sample container during the capturing of the first image. The motivation for doing this is to have modified the systems and methods of Zeiss to affix the imaging device to a robotic gripper, such that the system can use the robotic gripper as a plate holder while taking microscopy images, as suggested by Finkbeiner ([0102 – 0103]).
Regarding claim 6 (depends on claim 4), Amthor fails to explicitly disclose the method wherein the imaging condition is velocity of the imaging device relative to the sample container during imaging.
However, in a similar field of endeavor Finkbeiner discloses an automated robotic microscopy systems (abstract). In addition, Finkbeiner discloses wherein the imaging condition is velocity of the imaging device relative to the sample container during imaging ([0211]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Amthor, and the imaging condition is velocity of the imaging device relative to the sample container during imaging. The motivation for doing this is to have modified the systems and methods of Zeiss to capture the velocity of an imaging device relative to a container during imaging, such that the system can track displacement for a cell in consecutive images using a maximum velocity value, as suggested by Finkbeiner ([0211]).
Regarding claim 15 (depends on claim 14), Amthor fails to explicitly disclose the method further comprising providing a robot comprising a gripper, wherein providing the imaging device comprises providing the imaging device affixed to the robot, and further comprising gripping the sample container during the capturing of the first image, the second image, or the third image.
However, in a similar field of endeavor Finkbeiner discloses an automated robotic microscopy systems (abstract). In addition, Finkbeiner discloses wherein providing a robot comprising a gripper, wherein providing the imaging device comprises providing the imaging device affixed to the robot, and further comprising gripping the sample container during the capturing of the first image, the second image, or the third image (Fig. 3, [0102 - 0103]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Amthor, and providing a robot comprising a gripper, wherein providing the imaging device comprises providing the imaging device affixed to the robot, and further comprising gripping the sample container during the capturing of the first image, the second image, or the third image. The motivation for doing this is to have modified the systems and methods of Zeiss to affix the imaging device to a robotic gripper, such that the system can use the robotic gripper as a plate holder while taking microscopy images, as suggested by Finkbeiner ([0102 – 0103]).
Regarding claim 17 (depends on claim 14), Amthor fails to explicitly disclose the method wherein the first imaging condition is a first velocity of the imaging device relative to the sample container during capturing the first image, the second imaging condition is a second velocity of the imaging device relative to the sample container during capturing the second image, and the third imaging condition is a third velocity of the imaging device relative to the sample container during capturing the third image.
However, in a similar field of endeavor Finkbeiner discloses an automated robotic microscopy systems (abstract). In addition, Finkbeiner discloses wherein the first imaging condition is a first velocity of the imaging device relative to the sample container during capturing the first image, the second imaging condition is a second velocity of the imaging device relative to the sample container during capturing the second image, and the third imaging condition is a third velocity of the imaging device relative to the sample container during capturing the third image ([0211]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Amthor, and the first imaging condition is a first velocity of the imaging device relative to the sample container during capturing the first image, the second imaging condition is a second velocity of the imaging device relative to the sample container during capturing the second image, and the third imaging condition is a third velocity of the imaging device relative to the sample container during capturing the third image. The motivation for doing this is to have modified the systems and methods of Zeiss to capture the velocity of an imaging device relative to a container during imaging, such that the system can track displacement for a cell in consecutive images using a maximum velocity value, as suggested by Finkbeiner ([0211]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to QIAN YANG whose telephone number is (571)270-7239. The examiner can normally be reached on Monday-Thursday 8am-6pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached on 571-270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/QIAN YANG/
Primary Examiner, Art Unit 2677
Prosecution Insights