Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
The drawings were received on 12/03/2024. These drawings are accepted.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-19 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a mental process and data gathering without significantly more. The claims recite merely a mental process because the “machine learning model” is not defined in the specifications with further details. Further, given the broadest reasonable interpretation, the claims recite merely data gathering step used in the mental process because the receiving an image, determining the type of product and comparing products with an order acquires data and conducts a process similar to how the human mind can recognize a product and compare it to a purchase order. The processes, given the broadest reasonable interpretation, could potentially be applied using a pen and paper as a mental process. The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the claimed limitations do not integrate the procedure into a practical application.
To distinguish ineligible claims that merely recite a judicial exception from eligible claims that require an implementation of judicial exception, the Supreme Court uses a two-step framework: Step One (Step 2A), determine whether the claims at issue are directed to one of those patent-ineligible concepts; and Step Two (Step 28), if so, ask "what else is there in the claims?" to determine whether the additional elements transform the nature of the claim into a patent eligible application.
The first step I Prong One of Step One (Step 2A) to determine patent eligibility requires the determination of whether the claims at issue are directed to an enumerated patent ineligible concept.
Prong (1) requires the determination of (a) the specific limitations in the claim under examination (individually or in combination) that the examiner believes recites an abstract idea and (b) determining whether the identified limitations falls within the subject matter groupings of abstract ideas enumerated.
The enumerated patent ineligible concepts comprising:
(a) Mathematical Concepts - mathematical relationships, mathematical formulas or equations, mathematical calculations;
(b) Certain methods of organizing human activity - fundamental economic
principles / practices (including hedging, insurance, mitigating risk); commercial or legal interactions (including agreements in the form of contracts; legal obligations; advertising, marketing or sales activities or behaviors; business relations); managing personal behavior or relationships or interactions between people (including social activities, teaching, and following rules/ instructions) and
(c) Mental processes - concepts performed in the human mind (including an observation, evaluation, judgment, opinion).
Claims 1-19 recite a series of steps for determining what type of product is captured in an image and comparing the type to an order received. This judicial exception is not integrated into a practical application because the data gathering step, do not add a meaningful limitation to the system, method or computer readable medium as it is insignificant extra-solution data gathering activity and is nothing more than generally linking the product to a particular technological environment.
Accordingly, this judicial exception does not integrate the abstract idea into a practical application. Specifically, independent claims 1 and 12 only recite limitations directed at a mental process and data gathering. It is not integrated into any practical application. Dependent claims 2-11 and 13-19 are only additional data manipulation and data gathering.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1 and 12-17 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Chavez et al (US20200269429, hereinafter “Chavez”)
Claim 1. Chavez teaches A computer system for validating trays loaded with products ([0044] “pick bread items from source trays to fill destination trays according to each order” ) comprising:
at least one processor; and
at least one non-transitory computer-readable medium storing:
instructions that, when executed by the at least one processor, cause the computer system to perform the following operations: ([0025] “a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor.”)
a) receiving at least one image of products in at least one tray of a plurality of trays in an imaging area; ([0056] “At 404, image and/or other sensor data is received and processed to generate a three-dimensional view of the workspace or other scene or environment. At 406, the location(s) and content(s) of source receptacles, such as trays of bread in the bakery example, is/are determined.” And fig. 1)
PNG
media_image1.png
483
529
media_image1.png
Greyscale
PNG
media_image2.png
546
762
media_image2.png
Greyscale
b) determining a type of the products in the at least one tray ([0041] “identify, and determine one or more attributes of items to be loaded into and/or unloaded from trays 120 to trays 118.”) based upon the at least one image; ([0041] “identify an item and/or its attributes, e.g., based on image”) and
c) comparing the type of products in the at least one tray to at least one order. ([0045] “In the bread example, as new trays or stacks of trays arrive in the workspace, e.g., from the bakery, storage, etc., the items are identified and a plan generated, updated, and/or confirmed to make progress toward accomplishing the high level objective (e.g., efficiently fulfill and transport all orders), such as by picking available inventory from current source locations and moving/placing the items in trays being filled to fulfill respective individual orders.”)
Claim 12. The method herein has been executed and performed by the system of claim 1 and is likewise rejected.
Claim 13. Chavez teaches The method of claim 12 wherein the plurality of trays in the at least one image of products in trays are in a plurality of stacks. ([0045] “image or other sensor data may be received and processed to generate an updated three-dimensional view of the work space scene and/or to update information as to which items of inventory are located in which source locations within the work space. In the bread example, as new trays or stacks of trays arrive in the workspace,”)
Claim 14. Chavez teaches The method of claim 13 wherein the at least one image is a front or rear view of the plurality of stacks. ([0045] “image or other sensor data may be received and processed to generate an updated three-dimensional view of the work space scene … In the bread example, as new trays or stacks of trays arrive in the workspace,” Three dimensional view of the workspace includes all views of the stacks including the front and rear view)
Claim 15. Chavez teaches The method of claim 13 wherein the at least one image is an overhead view of the plurality of stacks. ([0037] “cameras 114 and 116, mounted on a wall in this example, provide additional image data usable to construct a 3D view of the scene in which system 100 is located and configured to operate.” Fig.1 shows 116 is an overhead view of the plurality of stacks )
Claim 16. Chavez teaches The method of claim 15 including taking the at least one image with at least one camera mounted overhead the imaging area. ([0037] “the camera 112 may be located more centrally, such as on the downward-facing face of the body of end effector 108 (in the position and orientation shown in FIG. 1).” )
Claim 17. Chavez teaches The method of claim 16 including detecting a new tray in the plurality of trays in the at least one image. ([0045] “image or other sensor data may be received and processed to generate an updated three-dimensional view of the work space scene as new trays or stacks of trays arrive in the workspace, e.g., from the bakery, storage, etc., the items are identified”)
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 2, 7-8 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Chavez et al (US20200269429, hereinafter “Chavez”) and in view of Parkinson et al (US20240104947, hereinafter “Parkinson”)
Claim 2. Chavez teaches The computer system of claim 1 wherein the at least one non-transitory computer-readable medium further stores at least one machine learning model trained on a image ([0029] "The robotic system may be configured to watch (e.g., using sensors) and learn (e.g., updating a library and/or model)" and [0040] “The 3D image data may be used to identify items to be picked/placed, such as by color, shape, or other attributes.”) of bakery products of different types. ([0030] “robotic system as disclosed herein may engage in a process to gather and store (e.g., add to library) attributes and/or strategies to identify and pick/place an item of unknown or newly-discovered type. For example, the system may hold the item at various angles and/or locations to enable 3D cameras and/or other sensors to generate sensor data to augment and/or create a library entry that characterizes the item type and stores a model of how to identify and pick/place items of that type.”[0056] “For example, if one full tray of wheat bread and one full tray of white bread are identified”)
Chavez does not explicitly teach machine learning model trained on a plurality of images of bakery products of different types.
Parkinson teaches machine learning model trained on a plurality of images ([0061] “The training data can comprise thousands of images and corresponding ground truths. During training, the parameters of one or more machine learning models can be modified to be able to accurately predict a classification and identify one or more characteristics of food products such as a confectionary product,”) of bakery products of different types. ([0038] “Confectionery products can include, but are not limited to fat-based and non-fat based confectionery, snacks, breads, baked goods, crackers, cakes, cookies, pies,” is understood to be the same as the claimed bakery products in light of instant specifications [0020])
It would have been obvious to persons of ordinary skill in the art before the effective filing date of the claimed invention to modify Chavez to have a machine learning model trained on images of bakery products of different types as taught by Parkinson to arrive at the claimed invention discussed above. The motivation for the proposed modification would have been so that (Parkinson [0061]“machine learning models can be modified to be able to accurately predict a classification and identify one or more characteristics of food products such as a confectionary product,”)
Claim 7. Chavez and Parkinson teach The computer system of claim 2 wherein the plurality of images on which the at least one machine learning model is trained
Chavez teaches are overhead views of the products in trays. ([0037] “the camera 112 may be located more centrally, such as on the downward-facing face of the body of end effector 108 (in the position and orientation shown in FIG. 1).” )
Claim 8. Chavez and Parkinson teach The computer system of claim 7 further including
Chavez teaches at least one camera positioned over the imaging area. ([0037] “the camera 112 may be located more centrally, such as on the downward-facing face of the body of end effector 108 (in the position and orientation shown in FIG. 1).” )
Claim 11. Chavez and Parkinson teach The computer system of claim 7 wherein the operations further include:
Chavez teaches d) receiving an image of the imaging area; ([0045] “image or other sensor data may be received and processed to generate an updated three-dimensional view of the work space scene”)
e) based upon the image of the imaging area from operation d), determining whether a new tray is present in the imaging area; ([0045] “as new trays or stacks of trays arrive in the workspace, e.g., from the bakery, storage, etc., the items are identified and a plan generated, updated, and/or confirmed to make progress toward accomplishing the high level objective (e.g., efficiently fulfill and transport all orders),”) and
f) based upon a determination in operation e) that the new tray is present in the imaging area, proceeding to operation a) wherein the at least one tray includes the new tray. ([0056] “At 404, image and/or other sensor data is received and processed to generate a three-dimensional view of the workspace or other scene or environment. At 406, the location(s) and content(s) of source receptacles, such as trays of bread in the bakery example, is/are determined.”
Claims 3, 5-6 and 9-10 are rejected under 35 U.S.C. 103 as being unpatentable over Chavez et al (US20200269429, hereinafter “Chavez”) and in view of Parkinson et al (US20240104947, hereinafter “Parkinson”) and in view of Tsutsumi et al (US20200151511, hereinafter “Tsutsumi”)
Claim 3. Chavez and Parkinson teach The computer system of claim 2
Chavez teaches wherein the trays are bakery trays ([0056] “At 406, the location(s) and content(s) of source receptacles, such as trays of bread in the bakery example, is/are determined. For example, one or more of image data, optical code scan, recognizing logos or text on packaging, etc., may be used to determine which items are where.”) and wherein the image on which the at least one machine learning model is trained ([0029] "The robotic system may be configured to watch (e.g., using sensors) and learn (e.g., updating a library and/or model)" and [0040] “The 3D image data may be used to identify items to be picked/placed, such as by color, shape, or other attributes.”) are of the bakery products in bakery trays. ([0056] “content(s) of source receptacles, such as trays of bread in the bakery”)
Chavez and Parkinson do not explicitly teach the plurality of images on which the at least one machine learning model is trained are of the bakery products in bakery trays.
Tsutsumi teaches and wherein the plurality of images ([0052] “each learning group image 41 comprises a combination of individual images 43a to 43c shown in FIG. 4.”)on which the at least one machine learning model is trained ([0050] “The machine learning typically is performed as deep learning,”) are of the bakery products ([0052] “image 43a is an image of a croissant (product G1), …image 43b is an image of a cornbread square (product G2),… image of a bread roll (product G3).”) in bakery trays. ([0052] “The learning group images 41 shown in FIG. 3 depict one or more products G1 to G3 placed on the tray T.”)
It would have been obvious to persons of ordinary skill in the art before the effective filing date of the claimed invention to modify the proposed combination of Chavez and Parkinson to have a machine learning model trained on images of bakery products in bakery trays as taught by Tsutsumi to arrive at the claimed invention discussed above. The motivation for the proposed modification would have been so that (Tsutsumi [0028]“recognition accuracy of the neural network is improved.”)
Claim 5. Chavez, Parkinson and Tsutsumi teach The computer system of claim 3 wherein plurality of images on which the at least one machine learning model is trained
Chavez teaches are of the bakery products in packages, ([0115] “soft or otherwise fragile products in non-rigid packaging, such as loaves of bread packaged in plastic bags or wrap, to be handled by a robotic system… bread may be identified” and [0056] “For example, one or more of image data, optical code scan, recognizing logos or text on packaging, etc., may be used to determine which items are where.”) wherein the packages of the bakery products have different appearances [0040] “The 3D image data may be used to identify items to be picked/placed, such as by color, shape, or other attributes.”) that correspond to different types of bakery products. ([0056] “For example, if one full tray of wheat bread and one full tray of white bread are identified and located in the workspace”)
Claim 6. Chavez, Parkinson and Tsutsumi teach The computer system of claim 3 wherein plurality of images on which the at least one machine learning model is trained are of the bakery products in the bakery trays,
Chavez teaches wherein the bakery trays have different appearances that correspond to different types of bakery products. [0040] “The 3D image data may be used to identify items to be picked/placed, such as by color, shape, or other attributes.” And [0056] “For example, if one full tray of wheat bread and one full tray of white bread are identified and located in the workspace”)
Claim 9. Chavez and Parkinson teach The computer system of claim 7
Chavez teaches wherein the trays are bakery trays ([0056] “At 406, the location(s) and content(s) of source receptacles, such as trays of bread in the bakery example, is/are determined. For example, one or more of image data, optical code scan, recognizing logos or text on packaging, etc., may be used to determine which items are where.”) and wherein the image on which the at least one machine learning model is trained ([0029] "The robotic system may be configured to watch (e.g., using sensors) and learn (e.g., updating a library and/or model)" and [0040] “The 3D image data may be used to identify items to be picked/placed, such as by color, shape, or other attributes.”) are of the bakery products in bakery trays. [0040] “The 3D image data may be used to identify items to be picked/placed, such as by color, shape, or other attributes.” And ([0056] “content(s) of source receptacles, such as trays of bread in the bakery”)
Chavez does not explicitly teach the plurality of images on which the at least one machine learning model is trained are of the bakery products in bakery trays.
Tsutsumi teaches and wherein the plurality of images ([0052] “each learning group image 41 comprises a combination of individual images 43a to 43c shown in FIG. 4.”)on which the at least one machine learning model is trained ([0050] “The machine learning typically is performed as deep learning,”) are of the bakery products ([0052] “image 43a is an image of a croissant (product G1), …image 43b is an image of a cornbread square (product G2),… image of a bread roll (product G3).”) in bakery trays. ([0052] “The learning group images 41 shown in FIG. 3 depict one or more products G1 to G3 placed on the tray T.”)
It would have been obvious to persons of ordinary skill in the art before the effective filing date of the claimed invention to modify the proposed combination of Chavez and Parkinson to have a machine learning model trained on images of bakery products in bakery trays as taught by Tsutsumi to arrive at the claimed invention discussed above. The motivation for the proposed modification would have been so that (Tsutsumi [0028]“recognition accuracy of the neural network is improved.”)
Claim 10. Chavez, Parkinson and Tsutsumi teach The computer system of claim 9 wherein plurality of images on which the at least one machine learning model is trained
Chavez teaches are of the bakery products in packages, ([0115] “soft or otherwise fragile products in non-rigid packaging, such as loaves of bread packaged in plastic bags or wrap, to be handled by a robotic system… bread may be identified” and [0056] “For example, one or more of image data, optical code scan, recognizing logos or text on packaging, etc., may be used to determine which items are where.”) wherein the packages of the bakery products have different appearances [0040] “The 3D image data may be used to identify items to be picked/placed, such as by color, shape, or other attributes.”) that correspond to different types of bakery products. ([0056] “For example, if one full tray of wheat bread and one full tray of white bread are identified and located in the workspace”)
Claims 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Chavez et al (US20200269429, hereinafter “Chavez”) and in view of Tsutsumi et al (US20200151511, hereinafter “Tsutsumi”)
Claim 18. Chavez teaches The method of claim 12 wherein step b) includes analyzing the at least one image using a machine learning model trained with a image ([0029] "The robotic system may be configured to watch (e.g., using sensors) and learn (e.g., updating a library and/or model)" and [0040] “The 3D image data may be used to identify items to be picked/placed, such as by color, shape, or other attributes.”) of bakery products in bakery trays. ([0056] “content(s) of source receptacles, such as trays of bread in the bakery”)
Chavez does not explicitly teach machine learning model trained with a plurality of images of bakery products in bakery trays.
Tsutsumi teaches machine learning model trained ([0050] “The machine learning typically is performed as deep learning,”) with a plurality of images ([0052] “each learning group image 41 comprises a combination of individual images 43a to 43c shown in FIG. 4.”)of bakery products ([0052] “image 43a is an image of a croissant (product G1), …image 43b is an image of a cornbread square (product G2),… image of a bread roll (product G3).”) in bakery trays. ([0052] “The learning group images 41 shown in FIG. 3 depict one or more products G1 to G3 placed on the tray T.”)
It would have been obvious to persons of ordinary skill in the art before the effective filing date of the claimed invention to modify Chavez to have a machine learning model trained on images of bakery products in bakery trays as taught by Tsutsumi to arrive at the claimed invention discussed above. The motivation for the proposed modification would have been so that (Tsutsumi [0028]“recognition accuracy of the neural network is improved.”)
Claim 19. Chavez and Tsutsumi teach The method of claim 18 further including:
Chavez teaches d) indicating a confirmation ([0106] “If no error is detected (1306), the item is released from the grasp (1308) and the arm and end effector are moved away to perform the next pick/place task.”)
or an error based upon step c). ([0106] “three-dimensional image data may indicate the item height is higher than expected for a successful placement. …If an error is detected (1306), at 1310 the item is returned to the start location, or in some embodiments another staging or buffer area, and the pick/place operation is attempted again. In some embodiments, repeated attempts to pick/place an item that result in an error being detected may trigger an alert and/or other action to initiate human intervention.”)
Allowable Subject Matter
Claim 4 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims and if rewritten to overcome the 35 USC 101 rejection.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure:
Eger et al US20220076228 teaches a system to confirm a tray of food products matches an order utilizing image analysis
Ebert et al US20250313407 teaches a system for restocking pharmacy products which are in stacked trays utilizing image analysis
Any inquiry concerning this communication or earlier communications from the examiner should be directed to OWAIS MEMON whose telephone number is (571)272-2168. The examiner can normally be reached M-F (7:00am - 4:00pm) CST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at (571) 272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/OWAIS I MEMON/Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698
Prosecution Insights