DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claim(s) 12-17 is/are rejected under 35 U.S.C. 102a1/a2 as being anticipated by Wu et al. (US 2017/0124704).
Regarding claim 12, Wu et al. teach:
12. A characterization apparatus, comprising:
a track (e.g., automation track; the automation track may be configured as an integral conveyance system ¶ 0032+);
a calibration object (e.g., calibration target 600, ¶ 0064) moveable on the track (see i.e., Planes 602 may be mounted on top of a conventional tube tray, allowing calibration target 600 to be handled by a DVS during training in the same manner in which a tube tray would be handled at runtime. This allows motion mechanisms to move markers 604 relative to the cameras in the DVS. ¶ 0065; see also the automation track may be configured as an integral conveyance system ¶ 0032; and The function of a sample handling module may include managing sample containers/vessels for the purposes of inventory management, sorting, moving them onto or off of an automation track (which may include an integral conveyance system, moving sample containers/vessels onto or off of a separate laboratory automation track, and moving sample containers/vessels into or out of trays, racks, carriers, pucks, and/or storage locations. ¶ 0038);
a calibrated image capture device (e.g., image capture system 140, one or more cameras 542, 544; see i.e., The image capture system 140, according to an embodiment, includes two cameras, a left camera 542 and a right camera 544. ¶ 0052; FIG. 11 shows an exemplary method 700 for calibrating cameras utilizing a calibration target, such as that shown in FIG. 5. Using a previously calibrated high-resolution camera, multiple images 772 of the calibration target are acquired from each in the DVS. ¶ 0104) located adjacent to the track (see i.e., Planes 602 may be mounted on top of a conventional tube tray, allowing calibration target 600 to be handled by a DVS during training in the same manner in which a tube tray would be handled at runtime. This allows motion mechanisms to move markers 604 relative to the cameras in the DVS. This further allows each marker to be placed at different locations in the image plane of each camera, in much the same way that salient features of a sample tube would be positioned in the image plane of each camera during runtime operation of the DVS. By utilizing an array of markers on a 3-D planar surface, multiple 3-D locations can be easily tested for calibration by sliding target 600 through the drawer system and capturing multiple images. This allows calibration images to simulate different tube top heights, for example. In some embodiments, the DVS includes multiple cameras aligned in a row in the drawer. The direction of that row is parallel to the peak in this calibration object. This calibration object works with one camera, as well as multiple cameras, as long as the camera can see the calibration object. ¶ 0065; see also Fig. 2D showing cameras on either side of drawer tracks); and
a computer (e.g., controller 520, image processor 524, image capture controller 548 Figs. 3-4) coupled to the calibrated image capture device (see i.e., According to an embodiment, a method for detecting properties of sample tubes, includes steps of receiving a series of images of a tray from at least one camera, extracting a plurality of image patches from each image, and automatically determining, using a processor, from a first subset of the plurality image patches, each patch corresponding to one of a plurality of slots in the tray, whether each of a plurality of slots contains a sample tube. ¶ 0010; The image capture system 140 also includes one or more processors to perform the image capture algorithms, as further described below. ¶ 0043; One or more internal or external memory devices may be associated with the image capture controller 548, such as memory 540. In one embodiment, one of the one or more memory devices comprises random access memory (RAM) in which a table is stored, the table containing the images taken by the cameras 542, 544. ¶ 0054; A controller 520 is provided for managing the image analysis of the images taken by the cameras 542, 544. ¶ 0057; The one or more memory devices 540 are associated with the controller 520. The one or more memory devices 540 may be internal or external to the controller 520. ¶ 0058), the computer configured and operable to cause:
the calibration object to move to at least two different longitudinal positions along the track including a first longitudinal position and a second longitudinal position (i.e., two different longitudinal positions in Figs. 2A, 2B), wherein the second longitudinal position is different from the first longitudinal position (see i.e., Each marker 604 is a unique coded block that correlates to a unique location on each plane 602. By providing multiple planes of these markers, target 600 presents a plurality of known 3-D space points that can then be correlated to 2-D points in the image plane of each camera. Because each marker is unique, a camera can identify a grid location on each plane when only a partial view of the plane is available. Conventionally, chessboard patterns or Siemens Hoffmann markers are used in a single 2-D plane, often placed in front of the camera at various poses and distances. Multiple planes ensure that the markers on the planes present a three-dimensional target for camera calibration, where the pose of each plane varies relative to each camera as the calibration target slides past cameras in a DVS. In some embodiments, planes may be arranged in a pyramid or other shape to present a 3-D target for camera calibration. ¶ 0064; Planes 602 may be mounted on top of a conventional tube tray, allowing calibration target 600 to be handled by a DVS during training in the same manner in which a tube tray would be handled at runtime. This allows motion mechanisms to move markers 604 relative to the cameras in the DVS. This further allows each marker to be placed at different locations in the image plane of each camera, in much the same way that salient features of a sample tube would be positioned in the image plane of each camera during runtime operation of the DVS. By utilizing an array of markers on a 3-D planar surface, multiple 3-D locations can be easily tested for calibration by sliding target 600 through the drawer system [along the track in a first and second longitudinal position, see Fig. 2B) and capturing multiple images. This allows calibration images to simulate different tube top heights, for example. In some embodiments, the DVS includes multiple cameras aligned in a row in the drawer. The direction of that row is parallel to the peak in this calibration object. This calibration object works with one camera, as well as multiple cameras, as long as the camera can see the calibration object. ¶ 0065),
capture a first image with the calibrated image capture device with the calibrated object located at the first longitudinal position (see i.e., With further reference to FIG. 3, an encoder 510, such as a quadrature encoder, is used to determine when a row of the tube tray 120 is moved into a centered or substantially centered position beneath the one or more cameras 542, 544. The encoder 510 transmits a signal (i.e., a pulse) to the image capture controller 548 upon detection of movement of the tube tray 120 corresponding to a new row of the tube tray 120 moving into a centered or substantially centered position beneath the one or more cameras 542, 544. The detection is based upon the encoder 510 incrementing upon a notch that indicates that the drawer 110 and/or the tube tray 120 has been moved one row. The signal serves as an instruction for the image capture controller 548 to instruct the cameras 542, 544 to take an image upon receipt of the signal. As described above, in some embodiments, the encoding scheme may correspond to other movements, such as, for example, the drawer 110/tube tray 120 moving two rows or the drawer 110/tube tray 120 moving into a position centered between two rows. The image capture controller 548 manages the storage of the images taken by the cameras 542, 544 during a time period in which the drawer 110/tube tray 120 is being moved into the work envelope 105. This time period may also include the drawer 110/tube tray 120 being moved out of the work envelope 105 (e.g., the drawer 110/tube tray 120 may be pushed into the work envelope 105, partially pulled out of the work envelope 105, then pushed back into the work envelope 105). One or more internal or external memory devices may be associated with the image capture controller 548, such as memory 540. In one embodiment, one of the one or more memory devices comprises random access memory (RAM) in which a table is stored, the table containing the images taken by the cameras 542, 544. The image capture system 140 may capture additional rows of images at the beginning and end of each drawer 110/tube tray 120 in order to ensure that all rows in the tray are seen from the same number of perspectives (otherwise the rows at the end will not be captured from one side). Additionally, the image capture system 140 may capture extra rows of images for all rows in order to generate additional perspectives on each tube and to aid in the determination of certain features. The image capture system 140 may also capture extra rows of images in order to detect features in the sample handler work envelope 105 in order to localize the trays 120 within the work envelope 105 and auto-calibrate the trays 120 to the sample handler's coordinate system. The image capture system 140 captures a fixed number of rows of images, at predetermined locations that have a fixed relationship to features of the trays 120 and sample handler work envelope 105. ¶ 0054; The image capture system 140 may capture and store a single image corresponding to each imaging of a predetermined imaging position for the tray. For example, if a tray has 10 rows, and each row should appear in three adjacent images to provide two oblique perspectives and one substantially central perspective of each row, twelve images of the tray taken at twelve sequential imaging positions can be stored. When a new image of a particular perspective for a given row is captured, the previously stored image corresponding to that imagining position is overwritten. For example, consider the following scenario: an image is captured when row 10 is pushed into the drawer 110 and is centered or substantially centered beneath the cameras 542, 544 of the image capture system 140. If, subsequently, the drawer 110 is pulled out and then pushed in so that row 10 is again centered or substantially centered beneath the image capture system 140, a second image of this perspective is taken. ¶ 0055; and ¶ 0064-0065 as noted above),
capture a second image with the calibrated image capture device with the calibrated object located at the second longitudinal position (see ¶ 0054-0055 and ¶ 0064-0065 as noted above), and
determine a three-dimensional path trajectory of a center location along a segment of the track (i.e., a segment of the automation track which may include an integral conveyance system, sample containers/vessels of a separate laboratory automation track, and sample containers/vessels of trays, racks, carriers, pucks, and/or storage locations. ¶ 0038) extending between the first longitudinal position and the second longitudinal position based at least upon the first image and the second image (see i.e., It is desirable to ascertain various pieces of information relating to a tray, the tubes, and the tubes' location within the tray, such as, for example, the tray slots containing a tube; a tube's center point, diameter, and height; the tray's orientation within a drawer [along the track], ¶ 0008; The series of FIGS. 2A-2F illustrate the depth of information that is obtained from the images, enabling the determination of the following characteristics: a center point of each tube in set 130 (e.g., the x-y location determined by correlating image features corresponding to a tube holder) ¶ 0046; see also ¶ 0054-0055 and ¶ 0064-0065 as noted above), wherein the three dimensional path trajectory corresponds to a plurality of center locations (e.g., central perspective of each row ¶ 0055; the optical center with respect to the image ¶ 0066; a center/central row of the image center of the camera ¶ 0088) of the calibration object along the track segment (see i.e., By utilizing a calibration target, such as target 600, multiple cameras can undergo intrinsic property calibration at the same time. ¶ 0067; multiple observations of the grid placed at different locations with respect to the camera also greatly help improve the accuracy of calibration. With the cameras fixed, the drawer and target move [along the track segment], allowing the camera to take multiple images, which further allows a 3-D target with known coordinates to move around in the image plane. This results in multiple poses of the target relative to the camera, allowing sufficient data to calibrate each camera. Because a calibration target is large enough to be visible across multiple cameras, calibration software can derive the relative pose between each pair of cameras, in addition to the intrinsic properties of each camera. ¶ 0068; Images taken from a single camera with various placement of the whole calibration target in the field of view allows calibration of these 3D coordinates. The rigid transform between these two plates of the target can then be derived via a non-linear least squares optimization to infer the 3D coordinates of all calibration points on the target. ¶ 0074).
Regarding claims 13-17, Wu et al. teach:
13. The characterization apparatus of claim 12 further comprising one or more analyzers, a loading station, a centrifuging station, a quality check module, and an aliquoter station (see ¶ 0041-0042+ for example).
14. The characterization apparatus of claim 12, further comprising a specimen container and one or more light sources (e.g., 546 ¶ 0052) capable of causing front-lighting of the specimen container during imaging (see i.e., In an embodiment, in order to accurately capture the image and taking into account that the drawer 110/tube tray 120 is moving, the cameras 542, 544 use a shutter speed fast enough to essentially produce stop motion photography for capturing the images. In some embodiments, the light source 546 may be synchronized with the triggering of the cameras 542, 544 to aid in strobe or stop motion photography. In other embodiments, the light source 546 may be on continuously or may be triggered on upon a first detection of movement of a drawer 110/tube tray 120. In some embodiments, cameras that are capable of a 250 microsecond exposure time are used. In other embodiments, cameras with other capabilities may be used depending on, for example, lighting, the speed of the drawer 110/tube tray 120, and the desired quality of the images. ¶ 0053).
15. The characterization apparatus of claim 12, wherein the calibration object comprises a three-dimensional tool with known geometry and one or more calibrated patterns provided thereon (see i.e., Calibration target 600 includes a plurality of planes 602. Each of planes 602 includes an array of unique markers forming a larger optical pattern. Exemplary individual markers 604 include Siemens Hoffmann markers. Planes 602 allow calibration target 600 to present a 3-D arrangement of unique markers 604. Siemens Hoffmann markers are useful for calibrating cameras by providing known 2-D space points relative to the image plane. Each marker 604 is a unique coded block that correlates to a unique location on each plane 602. ¶ 0064 & Fig. 5).
16. The characterization apparatus of claim 15, wherein the calibration object comprises a V-shaped marker tool including at least two planar surfaces (see Fig. 5 & ¶ 0064 for example).
17. The characterization apparatus of claim 16, wherein the V-shaped marker tool includes Hoffman markers thereon (¶ 0064).
Claim Rejections - 35 USC § 103
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claim(s) 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wu et al. (US 2017/0124704) in view of Kluckner et al. (US 2018/0365530).
Regarding claim 18, Wu et al. teach: 18. The characterization apparatus of claim 12, further comprising a quality check module, wherein the calibrated image capture device is a camera positioned within the quality check module (e.g., one or more cameras 542, 544; see i.e., In some embodiments, when the sample tube lacks a cap, the images captured via image capture system 140 can be processed to determine information about the quality of the sample and any defects or anomalous characteristics of a sample tube or tray at an early stage in the process. ¶ 0049). However, Wu et al. do not explicitly teach an RGB camera.
Kluckner et al. teach determining characteristics (Claim 1) of a specimen container (e.g., 102) with an RGB camera (In optional embodiments, as best shown in FIGS. 4C and 4D, the specimen container 102 may be front lit in the quality check module 130A, such as by including light sources 444D, 444E, and 444F arranged adjacent to the cameras 440A, 440B, 440C, i.e., above, below, to the side, or combinations, but on the same side of the specimen container 102 as the respective cameras 440A-440C. In this embodiment, the cameras 440A-440C may be digital color cameras having RGB peaks of approximately 634 nm, 537 nm, and 455 nm, respectively, ¶ 0065).
It would have been obvious to one of ordinary skill in the art at the time the invention was made to modify Wu et al. to include the RGB camera as taught in Kluckner et al. for the purpose of allowing for the separation of RGB spectral components to generate the multi-spectral, multi-time exposure images (Kluckner et al. ¶ 0065).
Response to Arguments
Applicant's arguments filed 05/04/2026 have been fully considered but they are not persuasive.
The objection to the Specification has been withdrawn.
The 35 USC § 112 rejections have been withdrawn.
In response to the Applicant's argument that cited prior art fail to teach “determine a three-dimensional path trajectory of a center location along a segment of the track extending between the first longitudinal position and the second longitudinal position based at least upon the first image and the second image, wherein the three dimensional path trajectory corresponds to a plurality of center locations of the calibration object along the track segment.”, Examiner disagrees.
Wu et al. teach, among other things, determine a three-dimensional path trajectory of a center location along a segment of the track (i.e., a segment of the automation track which may include an integral conveyance system, sample containers/vessels of a separate laboratory automation track, and sample containers/vessels of trays, racks, carriers, pucks, and/or storage locations. ¶ 0038) extending between the first longitudinal position and the second longitudinal position based at least upon the first image and the second image (see i.e., It is desirable to ascertain various pieces of information relating to a tray, the tubes, and the tubes' location within the tray, such as, for example, the tray slots containing a tube; a tube's center point, diameter, and height; the tray's orientation within a drawer [along the track], ¶ 0008; The series of FIGS. 2A-2F illustrate the depth of information that is obtained from the images, enabling the determination of the following characteristics: a center point of each tube in set 130 (e.g., the x-y location determined by correlating image features corresponding to a tube holder) ¶ 0046; see also ¶ 0054-0055 and ¶ 0064-0065 as noted above), wherein the three dimensional path trajectory corresponds to a plurality of center locations (e.g., central perspective of each row ¶ 0055; the optical center with respect to the image ¶ 0066; a center/central row of the image center of the camera ¶ 0088) of the calibration object along the track segment (see i.e., By utilizing a calibration target, such as target 600, multiple cameras can undergo intrinsic property calibration at the same time. ¶ 0067; multiple observations of the grid placed at different locations with respect to the camera also greatly help improve the accuracy of calibration. With the cameras fixed, the drawer and target move [along the track segment], allowing the camera to take multiple images, which further allows a 3-D target with known coordinates to move around in the image plane. This results in multiple poses of the target relative to the camera, allowing sufficient data to calibrate each camera. Because a calibration target is large enough to be visible across multiple cameras, calibration software can derive the relative pose between each pair of cameras, in addition to the intrinsic properties of each camera. ¶ 0068; Images taken from a single camera with various placement of the whole calibration target in the field of view allows calibration of these 3D coordinates. The rigid transform between these two plates of the target can then be derived via a non-linear least squares optimization to infer the 3D coordinates of all calibration points on the target. ¶ 0074).
Applicant is encouraged to amend the claims to include additional structural elements of the apparatus.
Applicant is thanked for their thoughtful amendments to the claims.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DEAN KWAK whose telephone number is (571)270-7072. The examiner can normally be reached M-TH, 4:30 am - 2:30 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, CHARLES CAPOZZI can be reached at (571)270-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DEAN KWAK/Primary Examiner, Art Unit 1798
DEAN KWAK
Primary Examiner
Art Unit 1798
Prosecution Insights