DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-24 are pending.
Claim Rejections - 35 USC § 103
The following is a quotation of pre-AIA 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claim(s) 1-24 is/are rejected under 35 U.S.C. 103 as being unpatentable over Marrion et al (US2015/0036876) in view of Olmstead et al (US20130020391A1).
(Note: prior art reference Olmstead et al (US20130020391A1) is provided in the IDS filed on 2/25/2026)
Regarding claims 1 and 12, Marrion teaches a method for assigning a symbol to an object in an image, the method comprising:
(Marrion, "associating codes with objects.", [0024]; "The machine vision system can use data from the dimensioner (e.g., dimensioning data including object dimensions and object pose information) to determine to which objects the identified codes are affixed and associate codes with objects.", [0024]; Marrion teaches a method for associating codes with objects)
receiving the image captured by an imaging device, the symbol located within the image;
(Marrion, "Camera 115 can generate two-dimensional images of field of view 150, including objects within field of view 150. Camera 115 can provide the two-dimensional images to machine vision processor 125", [0028]; "Camera 115 can capture images of one or more of codes 140 on objects 140.", [0034]; receiving an image captured by an imaging device where the code/symbol is located within the image)
receiving, in a three-dimensional (3D) coordinate space, a 3D location of one or more points that corresponds to pose information indicative of a 3D pose of the object in the image;
(Marrion, "dimensioner 120 can generate dimensioning data (e.g., a point cloud or heights of points on objects 140 above conveyor belt 107, along with pose information) for objects 140.", [0030]; "dimensioner 120 (or machine vision processor 125) can determine the pose of object 140 b and/or the pose of one or more of the surfaces of object 140 b in the coordinate space of dimensioner 120", [0033]; receiving dimensioning data, such as a 3D point cloud, that provides 3D location and pose information of the object in a 3D coordinate space)
determining a two-dimensional (2D) location of the one or more points by mapping the 3D location of the one or more points of the object in the 3D coordinate space to a 2D location within the image in a 2D image coordinate space; and
(Marrion, "For the inverse mapping, three dimensional points in ReaderCamera3D coordinate space 420 are mapped to two dimensional points in ReaderCamera2D coordinate space 415.", [0050]; Olmstead, "The annotated image includes an outline 3215 surrounding object 3210 that corresponds to a three-dimensional model of object 3210 generated by object measurement system 115. In other words, the three-dimensional model of the object is projected onto the image of the scene", [0125]; Marrion teaches the capability of inverse mapping 3D points to 2D image coordinates. However, Marrion's core process assigns symbols via a 3D back-projection ray. Olmstead teaches mapping/projecting the 3D model of the object to a 2D image coordinate space to form an outline over the image. Together Marrion and Olmstead teach mapping the 3D location of points of the object to a 2D location within the image)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the teachings of Olmstead into the system or method of Marrion in order to provide a direct 2D spatial context for where the object's boundaries exist in the image. The combination of Marrion and Olmstead also teaches other enhanced capabilities.
The combination of Marrion and Olmstead further teaches:
assigning the symbol to the object based on a relationship between a 2D location of the symbol in the image in the 2D image coordinate space and the 2D location of the one or more points of the object in the image in the 2D image coordinate space.
(Marrion, "machine vision processor 125) can determine the locations of the one or more codes 140 in the image coordinate space of camera 115.", [0034]; "determine to which objects the identified codes are affixed and associate codes with objects.", [0024]; Olmstead, "By knowing where objects are located in the image and which objects have an exception, object annotation system 145 is able to generate annotated image data", [0124]; Marrion teaches determining the 2D location of the code in the image and assigning it to the object. Olmstead teaches locating the objects in the 2D image space based on the projected 3D model. It would have been obvious to modify Marrion with Olmstead's 2D projection mapping so that the assignment of the symbol to the object is determined by assessing the relationship between the 2D code location and the mapped 2D boundary of the object directly in the image coordinate space, serving as an efficient alternative to Marrion's 3D ray intersection method)
Regarding claims 2 and 17, the combination of Marrion and Olmstead teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, further comprising: determining a surface of the object based on the 2D location of the one or more points of the object within the image in the 2D image coordinate space; and assigning the symbol to the surface of the object based on a relationship between the 2D location of the symbol in the image and the surface of the object.
(Marrion, "identify the surface of an object to which the code is attached", [0024]; " identify an object of the one or more objects that is associated with the first surface; and associate the code with the object.", [0005]; Olmstead, "The annotated image includes an outline 3215 surrounding object 3210 that corresponds to a three-dimensional model... projected onto the image", [0125]; Marrion teaches assigning the symbol to an identified surface of the object. Olmstead teaches defining object surfaces natively in the 2D image coordinate space. Together Marrion and Olmstead teach determining the surface in 2D space based on mapped points and assigning the symbol based on its relationship to the surface in the 2D image. The combination would directly resolve spatial relationships in the image plane)
Regarding claims 3 and 18, the combination of Marrion and Olmstead teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, further comprising: determining that the symbol is associated with a plurality of images; aggregating the assignments of the symbol for each image of the plurality of images; and determining if at least one of the assignments of the symbol differs from the remaining assignments of the symbol.
(Olmstead, "If multiple back projection rays are generated (from the same image capture device or from multiple image capture devices), the back projection rays should intersect the three-dimensional model at or near the same point if they correspond to the same optical code.", [0088]; "only one optical code is associated with an object, but back projection rays of the optical code intersect the three-dimensional model at different locations. Accordingly, exception identification system 130 may generate a confidence level", [0106]; tracking a code across a plurality of images, aggregating the assignment locations on the object model, and determining if the assignments differ to trigger an exception. Integrating Olmstead's robust exception handling into Marrion would enable identifying mapping discrepancies)
Regarding claim 4, the combination of Marrion and Olmstead teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, further comprising determining an edge of the object in the image based on imaging data of the image.
(Olmstead, "lateral object sensor system 705 includes multiple sensors 710 a, 710 b... to provide a forward directed view and a rearward directed view of objects.", [0069]; "sensors 3400, 3405 capture line-scan or area images”, [0130]; “binarized row data from each sensor 3400, 3405 represents the shadow of the object ... A logical AND of the raster images produced by sensors 3400, 3405 may be computed to yield a close approximation of the footprint of the object.", [0133]; capturing image data (via line-scan/area cameras) and using the binarized images to determine a footprint/shadow, which constitutes an edge of the object. leveraging Olmstead's imaging-based edge detection in Marrion to verify object footprint bounds)
Regarding claims 5 and 24, the combination of Marrion and Olmstead teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, further comprising determining a confidence score for the symbol assignment.
(Olmstead, "exception identification system 130 may generate a confidence level that is indicative of how confident exception identification system 130 is in its decision that an exception does or does not exist.", [0106]; generating a confidence score/level regarding the association between an optical code and an object. Integrating Olmstead's confidence scoring into Marrion's symbol assignment process would allow quantifying the reliability of the mapping)
Regarding claims 6 and 14, the combination of Marrion and Olmstead teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, wherein the 3D location of one or more points is received from aa 3D sensor.
(Marrion, "In some embodiments, the dimensioner is a time-of-flight dimensioner.", [0006]; "dimensioner 120 can generate dimensioning data (e.g., a point cloud or heights of points on objects 140 above conveyor belt 107, along with pose information) for objects 140.", [0030]; the dimensioner acts as a 3D sensor, utilizing time-of-flight technology to generate 3D point cloud locations)
Regarding claims 7 and 19, the combination of Marrion and Olmstead teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, wherein the image includes a plurality of objects, the method further comprising: determining whether the plurality of objects overlap in the image.
(Olmstead, "Exception identification system 130 can also recognize when multiple objects are in view volume 445 and a back projection ray intersects more than one three-dimensional model of an object.", [0108]; "An optical code 2820 of object 2800 would also be in field of view 2815, but object 2805 blocks field of view 2815.", [0108]; evaluating scenes with multiple objects and determining whether objects block (overlap) each other within the image's field of view. Adding Olmstead's occlusion detection to Marrion would prevent incorrect symbol assignments caused by overlapping items)
Regarding claims 8 and 20, the combination of Marrion and Olmstead teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, wherein the image includes the object having a first boundary with a margin and a second object having a second boundary with a second margin, and the method further comprising: determining whether the first boundary and the second boundary overlap in the image.
(Marrion, "the selection criteria... can be a function of known errors, known object pose uncertainty, and possible confusion when the code is near a surface edge", [0045]; Olmstead, "the minimal distance between non-intersecting back projection rays... may be calculated... to determine whether the minimum distance is at or below a given tolerance.", [0094]; Marrion and Olmstead teach using tolerances and uncertainty margins when evaluating object boundaries and intersections. Determining if these boundaries (with their associated tolerances/margins) overlap is a necessary step in the intersection systems described)
Regarding claim 9, the combination of Marrion and Olmstead teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, wherein the 3D location of the one or more points is acquired at a first time and the image is acquired at a second time, and wherein the mapping of the 3D location of the one or more points in the 3D coordinate space to the 2D location within the image in the 2D coordinate space comprises mapping the 3D location of the one or more points from the first time to the second time.
(Marrion, "The transformation of coordinates... account for spatial and/or temporal differences between when and/or where the dimensioner acquires dimensioning data and when and/or where the camera acquires an image.", [0036]; "dimensioner 120 and camera 115, as illustrated, being spatially separated and acquiring dimensioning data of an object... and images of the same object... at different locations and/or times.", [0036]; mapping 3D locations by specifically accounting for the temporal difference (from a first time to a second time) between dimensioning and image capture)
Regarding claims 10 and 15, the combination of Marrion and Olmstead teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, wherein the pose information comprises a corner of the object in the 3D coordinate space.
(Marrion, "the machine vision processor can determine the coordinates in the image coordinate space of one or more corners of the code.", [0043]; "dimensioners can produce dimensioning data (or object pose data) including but not limited to the 3D pose of cuboids", [0030]; determining object poses, which for cuboids (as specifically mentioned) is defined by their corners in 3D space)
Regarding claims 11 and 16, the combination of Marrion and Olmstead teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, wherein the pose information comprises point cloud data.
(Marrion, "dimensioner 120 can generate dimensioning data (e.g., a point cloud or heights of points on objects 140 above conveyor belt 107, along with pose information) for objects 140.", [0030]; generating dimensioning and pose information that explicitly comprises 3D point cloud data)
Regarding claim 13, the combination of Marrion and Olmstead teaches its/their respective base claim(s).
The combination further teaches the system according to claim 12, further comprising: a conveyor configured to support and transport the object; and a motion measurement device coupled to the conveyor and configured to measure movement of the conveyor.
(Marrion, "System 100 includes conveyor system 105, including conveyor belt 107 and position encoder 110... Conveyor system 105 is configured to convey parts along the length of conveyor system 105", [0026]; "Position encoder 110 can provide the encoder pulse count to machine vision processor 125... The encoder pulse count can be used to identify and track the positions of objects on conveyor belt 107", [0027]; a conveyor is configured to transport objects and a motion measurement device (encoder) to measure its movement)
Regarding claim 21, the combination of Marrion and Olmstead teaches its/their respective base claim(s).
The combination further teaches the system according to claim 12, wherein assigning the symbol to the object comprises assigning the symbol to a surface.
(Marrion, "identify the surface of an object to which the code is attached ... Based on the identified surface, the machine vision system can associate the code with the object.", [0024]; assigning the code/symbol directly to an identified surface of the object)
Regarding claim 22, the combination of Marrion and Olmstead teaches a method for assigning a symbol to an object in an image, the method comprising:
(Marrion, Olmstead, see comments on claim 1)
receiving the image captured by an imaging device, the symbol located within the image(;)
(Marrion, Olmstead, see comments on claim 1)
receiving, in a three-dimensional (3D) coordinate space, a 3D location of one or more points that corresponds to pose information indicative of a 3D pose of one or more objects;
(Marrion, Olmstead, see comments on claim 1)
determining a two-dimensional (2D) location of the one or more points by mapping the 3D location of the one or more points of the object in the 3D coordinate space to a 2D location within the image in a 2D image coordinate space;
(Marrion, Olmstead, see comments on claim 1)
The combination further teaches:
determining a surface of the object based on the 2D location of the one or more points of the object within the image in the 2D image coordinate space; and
(Marrion, "determine one or more surfaces of objects 140 a, 140 b, and 140 c in shared coordinate space 305", [0041]; Olmstead, "The annotated image includes an outline 3215 surrounding object 3210 that corresponds to a three-dimensional model of object 3210", [0125]; Marrion teaches determining the surface of the object using 3D dimensioning data. Olmstead teaches defining the object boundaries (outlines) natively in the 2D image via 3D-to-2D projection. Together Marrion and Olmstead teach determining the surface of the object based on the mapped 2D locations of the object's points in the image. The combination would allow the system to identify the surfaces bounded by the mapped 2D points)
assigning the symbol to the surface based on a relationship between a 2D location of the symbol in the image in the 2D image coordinate space and the 2D location of the one or more points of the object in the image in the 2D coordinate space.
(Marrion, "identify the surface of an object to which the code is attached ... Based on the identified surface, the machine vision system can associate the code with the object.", [0024]; Marrion teaches assigning the code to the surface of the object. When modified by Olmstead's 2D projection mapping (see comments on claim 1), this assignment is executed based on the spatial relationship between the 2D code location and the mapped 2D location of the object's points comprising the surface in the 2D image space)
Regarding claim 23, the combination of Marrion and Olmstead teaches its/their respective base claim(s).
The combination further teaches the method according to claim 22, wherein assigning the symbol to the surface comprises determining an intersection between the surface and the image in the 2D coordinate space.
(Olmstead, "The annotated image includes an outline 3215 surrounding object 3210 that corresponds to a three-dimensional model of object 3210 generated by object measurement system 115. In other words, the three-dimensional model of the object is projected onto the image", [0125]; projecting the 3D surface model onto the 2D image, which effectively calculates the outline or intersection of that surface within the 2D image coordinate space. A person of ordinary skill would recognize this projection inherently determines the surface's intersection onto the 2D plane)
Response to Arguments
Applicant's arguments filed on 6/22/2026 with respect to one or more of the pending claims have been fully considered but are moot in view of the new ground(s) of rejection.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIANXUN YANG whose telephone number is (571)272-9874. The examiner can normally be reached on MON-FRI: 8AM-5PM Pacific Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on (571)272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center. for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272- 1000.
/JIANXUN YANG/
Primary Examiner, Art Unit 2662 6/29/2026
Prosecution Insights