DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Claims 1-2 and 4-8 have been amended. Claim 3 has been canceled. Claims 1-2 and 4-8 are pending in this action.
Applicant’s arguments, see pg. 1 section "Specification", filed 6 October 2025, with respect to the objections of the specification have been fully considered and are persuasive. The objections to the specification have been withdrawn.
Applicant’s arguments, see pg. 2 section "Claim Objections" filed 6 October 2025, and pg. 4 section "Remarks" filed 9 March 2026, with respect to the objection of claim 5 have been fully considered and are persuasive. The objection to claim 5 has been withdrawn.
Applicant’s arguments, see pg. 2-10 section "Claim Rejections – 35 U.S.C. 101", filed 6 October 2025, with respect to the rejection of claims 1-8 under 35 U.S.C. 101 have been fully considered and are not persuasive.
Specifically, the applicant argues with regards to Step 2A Prong One that tray validation may not be reasonably performed by the human mind because it is a pixel-level analysis involving thousands of pixels which a person cannot reasonably perform mentally. The examiner disagrees. The claim states "validating the tray as valid . . . comprises marking the tray as a valid tray if a number of pixels occupied by the tray in the image of the tray is greater than a predetermined threshold value;" The applicant argues that an image of a tray comprises thousands of pixels and a person may be not reasonably count and consider those pixels mentally. However, the claim does not limit the size of the image so by the broadest reasonable interpretation it may be a number of pixels which a person may count and consider mentally. Further, the claim does not recite counting the pixels. Rather, it recites comparing "a number of pixels occupied by the tray" to "a predetermined threshold value". Therefore, the action of the claim, marking the tray as valid, does not involve counting thousands of pixels but rather involves comparing one number, the number of pixels which may be acquired as a data input or by other means, to a second number, the predetermined threshold. This process may be performed mentally and the applicant's argument is not persuasive.
With regard to Step 2A Prong Two the applicant argues that the tray validation is a practical application. Specifically, on pg. 7 of their remarks filed 6 October 2026 the applicant argues "If the number of pixels exceeds the threshold, the tray is marked as valid and passed on for further processing. Otherwise, it is filtered out. This ensures that only trays of sufficient size and visibility are analyzed, thereby improving the accuracy and efficiency of the downstream deep learning models used for gap detection and produce estimation." (emphasis added) The examiner agrees that as described if validation were not a mental process (if for instance the claim were amended such that counting a large number of pixels were part of the validation process) that the described validation would be a practical application. However, the described validation is not captured by the claim language. The critical point which provides the practical application as described by the applicant is filtering out trays which are not valid (as shown by the emphasis above). The removing of invalid trays is what provides for "improving the accuracy and efficiency". This is not reflected in the claim. There is no indication in the claim that invalid trays are filtered out from consideration. Instead, by the broadest reasonable interpretation of the claim, valid trays are simply "marked" which may be understood as a recording or labeling process which does not expressly filter out invalid trays. Therefore, the applicant's arguments are not persuasive and the claim is not integrated into a practical application. The examiner notes that line 11 of claim 1 recites "a valid tray". A simple amendment which would communicate the filtering concept would be to replace each instance of "the tray" in claim 1 line 13 through to the end of the claim with "the valid tray". This would expressly indicate that only valid trays are used for further processing by referring directly to the "valid tray" obtained by performing the validation. Note, the dependent claims would need to be carefully reviewed and likewise amended to avoid any 35 U.S.C. 112(b) issues.
With regard to Step 2B the applicant argues that the claim, specifically through claim validation, amounts to significantly more than the judicial exception. The examiner disagrees. Similar to Step 2A Prong Two, the applicant's argument depends on invalid trays being removed from consideration. As shown above, this is not reflected in the claim language. Therefore, the applicant's argument is not persuasive and the claim does not amount to more than the judicial exception. Therefore the rejection of claims 1-2 and 4-8 (claim 3 being canceled) under 35 U.S.C. 101 is maintained.
Applicant’s arguments, see pg. 10-16, filed 6 October 2025, with respect to the rejection of claims 1-8 under 103 have been fully considered and are persuasive. Specifically, the applicant argues that Curlander et al. (US 9460524 B1; hereafter, Curlander) discloses a system which estimates available volume and applies a threshold to small individual volumes but does not disclose a system which determines if a tray is valid as a whole. The examiner finds this persuasive. The applicant also argues that Curlander does not disclose validation logic which filters out invalid trays. While that may be true, as shown in the response to the arguments against the rejection under 35 U.S.C. 101, the claim language does not expressly recite a filtering operation. Therefore, this argument by the applicant is not persuasive. Regardless, the examiner is persuaded by the argument that Curlander does not disclose expressly a system which determines if a tray is valid as a whole. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Chen et al. (CN 113033545 A; hereafter, Chen).
Chen discloses:
validating the tray as valid ([n0036] pallets, i.e. trays, are marked as valid when they are determined to be empty. See also [n0147]), by a tray validation module ([n0154] embodiments of the disclosure may be performed by a processor programed for the methods. A processor programmed to perform the method, including marking tray images as valid, is understood as a module, i.e. a tray validation module), using the image of the tray and a first pixel determination technique, wherein validating the tray using the image of the tray and the first pixel determination technique comprises marking the tray as a valid tray if a number of pixels occupied by the tray in the image of the tray is greater than a predetermined threshold value ([n0115] the tray image is binarized meaning that areas of interest in the image, such as the tray or objects on the tray, are given a value of one and other pixels are given a value of zero. [n0117] the sum of the pixels is calculated which is understood as the number of pixels with the value of one. [n0118] the number of pixels is compared to a threshold. [n0119]-[n0120] when the number of pixels exceed a threshold the tray is marked as not empty and when below a threshold the tray is marked as empty. A person of ordinary skill in the art would understand that if the pixels exceeding a threshold indicates that the tray is not empty that the pixels with a value of 1 must indicate objects on the tray, therefore making a large count means the tray is not empty. Conversely, a person of ordinary skill would be equipped to instead determine the number of pixels of the tray, i.e. the zero pixels. If the number of zero pixels were compared to the threshold instead then when they exceed the threshold it would indicate an empty tray. A person of ordinary skill in the art would therefore understand that they may determine the number of pixels of a tray and if they exceed a threshold the tray is empty. [n0036] if the tray is empty it is a valid reference tray, i.e. marked as valid. Therefore, a tray is marked as valid if the number of pixels occupied by the tray exceed a threshold);
The full rejection including motivations to combine is included below in the section titled "Claim Rejections - 35 USC § 103". Claim 8 is similarly amended to claim 1 and is similarly rejected.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-8 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. MPEP 2106 details the analysis for determining the eligibility of an invention as completed below.
The independent claims: Claims 1 and 8
Claim 1
Claim 1 elements:
Receiving an image of a tray.
Identifying the tray in the image.
A first deep learning model trained using a plurality of images of trays containing no produce.
Marking a tray as valid by determining if a number of pixels of the tray in the image exceeds a threshold
Estimating a total area of the tray.
Identifying one or more areas in which the surface of the tray is exposed.
A gap detection module using a second deep learning model trained by a plurality of images of trays with produce and gaps.
Estimating an area of the exposed portions of the tray
Subtracting the estimated area of the exposed portions from the area of the tray to determine the covered portion of the tray
Estimating a ratio of the area of the covered portion of the tray to the total area of the tray
A gap percentage calculation module
Step 1:
The claim is directed to a process (a method).
Step 2A Prong 1:
The claim recites a judicial exception in:
B) Identifying the tray in the image may be performed mentally by a person of ordinary skill in the art.
C) a. Determining if a number of pixels of the tray exceeds a threshold is comparison of a number to a threshold which may be performed mentally.
D) Estimating an area may be performed mentally. As estimating is given a broad interpretation and does not indicate a level of accuracy, this may be performed by a person simply stating an informed guess as to the area of the tray in the image.
E) A mental process as a person of ordinary skill in the art may identify the regions in the image in which the surface of the tray is exposed.
G) Estimating an area of the exposed portions may be performed mentally. As estimating is given a broad interpretation and does not indicate a level of accuracy, this may be performed by a person simply stating an informed guess as to the area of the exposed portions of the tray in the image.
H) A mathematical formula. This is a simple subtraction and qualifies as a mathematical formula abstract idea.
I) An estimating of a ratio may be performed mentally. As estimating is given a broad interpretation and does not indicate a level of accuracy, this may be performed by a person simply stating an informed guess as to the ratio between the areas. Alternatively, this may be interpreted as the mathematical formula of the covered area divided by the total area.
Per MPEP 2106.04(a)(2)I., “It is important to note that a mathematical concept need not be expressed in mathematical symbols, because ‘[w]ords used in a claim operating on data to solve a problem can serve the same purpose as a formula.’”
Step 2A Prong 2:
The claim recites the following additional elements:
A) Receiving an image that is then analyzed or acted on by the abstract idea is a data handling activity and is considered an extra solution activity. As such, it does not integrate the judicial exception into a practical application.
C) The first deep learning module is recited in terms of high generality. While the claim states some of the data that it was trained on, there is no indication of the structure of the machine learning model. As such, the machine learning model is interpreted as being performed by a general purpose computer. There is no indication of technical aspects which improve the functioning of a computer. Therefore, the first machine learning model amounts to instructions to apply the judicial exception on a general purpose computer and does not integrate the judicial exception into a practical application.
F) The gap detection module uses a second deep learning model. The second deep learning module is recited in terms of high generality. While the claim states some of the data that it was trained on, there is no indication of the structure of the machine learning model. As such, the machine learning model is interpreted as being performed by a general purpose computer. There is no indication of technical aspects which improve the functioning of a computer. Therefore, the second machine learning model amounts to instructions to apply the judicial exception on a general purpose computer and does not integrate the judicial exception into a practical application.
J) A gap percentage calculation module is understood to be associated with a processor because of the reference to the drawing (item 250 is a component of 105 which per [0027] may include processors). As such, given the broadest reasonable interpretation in light of the specification it is understood as a general purpose computer or a component of a general purpose computer and amounts to instructions to apply the judicial exception on a general purpose computer. Therefore it does not integrate the judicial exception into a practical application.
Step 2B:
In considering whether the claim as a whole amounts to significantly more it is important to determine whether the claim as a whole improves upon the functioning of a technology or technical field. The specification identifies that the invention improves the ability to track irregularly shaped objects in trays. If the additional elements cause the improvement to the field of endeavor, then the claim as a whole may amount to significantly more than the judicial exception, see MPEP 2106.05(a).
When considered individually and as a whole, the additional elements A), C), F), and J) amount to data handling operations and general purpose computers which are well understood, conventional, and routine in the art. They do not present an improvement to the technical field. Therefore, the claim does not amount to significantly more than the judicial exception.
Claim 8
Claim 8 recites a camera and a management server comprising a processor and a memory which perform steps significantly equivalent to the steps of claim 1. It recites the judicial exceptions and additional elements as recited by the steps of claim 1 and is analyzed similarly to claim 1.
Claim 8 additionally recites the camera. The camera collects data for analysis and is well understood, routine, and conventional in the art. The camera does not integrate the judicial exception into a practical application or amount to significantly more than the judicial exception.
Claim 8 additionally recites the management server. The management server is recited in terms of high generality and does not present an improvement to the field of endeavor. It does not integrate the judicial exception into a practical application or amount to significantly more than the judicial exception.
Therefore, claim 8 is directed to a judicial exception.
Dependent claims: Claims 2-7
Claim 2
Removing noise and obstructions prepares the data for the analysis by the judicial exception and does not integrate the judicial exception into a practical application or amount to significantly more than the exception.
Claim 4
Claim 4 details a method for validating the tray image including determining and comparing which may be performed mentally and does not integrate the judicial exception into a practical application or amount to significantly more than the exception. For instance, by the broadest reasonable interpretation, determining the number of pixels may be a number small enough to be counted by a person mentally and comparing that number to a threshold is a process which may be performed mentally.
Claim 5
Claim 5 refines the estimating of the exposed portions of the tray and the total area of the tray and does not integrate the judicial exception into a practical application or amount to significantly more than the exception.
Claim 6
Claim 6 presents a method of estimating the area of the tray which may be performed mentally and does not integrate the judicial exception into a practical application or amount to significantly more than the exception.
Claim 7
Claim 7 presents a method of estimating the area of the exposed portion of the tray which may be performed mentally and does not integrate the judicial exception into a practical application or amount to significantly more than the exception.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 and 4-8 are rejected under 35 U.S.C. 103 as being unpatentable over Gregory et al. (U.S. Publ. No. 20230081303; hereafter, Gregory) in view of Curlander et al. (U.S. Publ. No. 9460524 B1; hereafter, Curlander) in further view of Chen et al. (CN 113033545 A; hereafter, Chen) and Kim et al. (U.S. Publ. No. 20230177458; hereafter, Kim) and Herz et al. (U.S. Publ. No. 20200364501).
Regarding claim 1, Gregory discloses:
receiving, by a processor ([0082] the computing device includes a processor), an image from a camera, the image having an image of the tray ([0032] each device may comprise an imaging device such as a camera. [0026] the containers may be monitored by an imaging device. The containers are understood as trays as they contain flat bottoms, raised sides, and lack lids, see Fig. 2);
identifying the image of the tray in the received image ([0044] the classification model determines information for each container 204A-204D. As shown in fig. 2, each container is identified with a bounding box around it, understood as identifying the container), by a tray image identification module ([0044] and fig. 2, the classification model performs the identification), using a first deep learning model ([0043] the classification model may comprise a neural network, which is understood as a deep learning model);
wherein the first deep learning model is trained using different sizes and shapes of trays, wherein the plurality of images of the trays are of trays not containing produce ([0059] the classification model is trained on a plurality of images showing the containers. Fig. 2, the containers are shown to contain various shapes and sizes. [0059] the classification model is trained with at least two images, one showing a reference quantity of pixels and one showing the current capacity of the containers. [0055] The reference quantity is an image with a known object. The current capacity may display objects occupying pixels less than or equal to the reference quantity. Less than is understood to include an empty container, i.e. trays not containing produce);
estimating, by the processor, a total area of the tray ([0047] the area of the container is determined as "a total quantity of pixels associated with the container". This is performed by the image processing module 110B. Fig. 1, 110B is a component of server 110. [0082] servers may include processors);
and estimating, by the gap percentage calculation module, the quantity of the produce in the tray as a percentage of the area of the tray covered by the produce and the total area of the tray ([0049] the current capacity may be expressed as a ratio or a percentages of pixels classified as containing objects, i.e. covered by produce, and the total pixels. [0048] and [0050] the classification module may perform this function, understood as the gap percentage calculation module).
Gregory does not disclose expressly identifying, by a gap detection module, one or more areas in which the top surface of the bottom of the tray is exposed and estimating an area of the identified top surface of the bottom of the tray.
Curlander in a first embodiment discloses:
identifying, by a gap detection module (Col. 12 line 55-58 and Fig. 2B, a processor 254 is associated with the imaging device 255. It is understood that the imaging device 460 of Fig. 4A may be, in an embodiment, the imaging device 255 and associated with a processor. The combination of software and hardware that performs the gap detection below may be understood as the gap detection module), one or more areas in the image of the tray in which the top surface of the bottom of the tray is exposed (Col. 16 line 57-61, "an image of an interior of the bin 430 may be captured and evaluated in order to identify portions of the image corresponding . . . portions of the image corresponding to a net visible area of a rear face of the bin 430. " The rear face of the bin is understood as equivalent to the top surface of the bottom of the tray as it is dependent on the orientation of the bin or tray. If the tray is arranged vertically as in Fig. 4A of Curlander and as in Fig. 4A of the instant application, then the bottom of the tray may be understood as the rear of the tray. A bin is understood as a tray as it is comprised of a base and sides and an open top or face);
estimating, by a gap percentage calculation module (Col. 12 line 55-58 and Fig. 2B, a processor 254 is associated with the imaging device 255. It is understood that the imaging device 460 of Fig. 4A may be, in an embodiment, the imaging device 255 and associated with a processor. The combination of software and hardware that performs the area identification below may be understood as the gap percentage calculation module), the quantity of produce in the tray, wherein estimating the quantity of produce in the tray comprises: estimating an area of the identified top surface of the bottom of the tray exposed (Col. 17 line 11-15, "a number of the pixels of the image corresponding to the rear face of the bin 430 may be determined, e.g., by counting, and a net area of the rear face of the bin 430 may be determined based on the combined area of each of such pixels");
Curlander is combinable with Gregory because it is from the same field of endeavor of determining available space in a container (Gregory, abstract; Curlander, abstract).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify Gregory with the gap detection of Curlander.
The motivation for doing so would have been that “by estimating the dimensions of one or more available volumes of a bin based on imaging data regarding the visible portions of the bin, and comparing dimensions of the estimated available volumes of the bin to dimensions of one or more items, the eligibility of the items to be accommodated within the various estimated available volumes may be determined" (Col. 16 line 39-45 and Fig. 4B). Therefore, it may be determined what other items may be stored in the bins or trays.
Therefore, it would have been obvious to combine the first embodiment of Curlander with Gregory.
The first embodiment of Curlander does not disclose expressly subtracting the estimated area of the identified top surface of the bottom of the tray exposed from the total area of the tray to obtain an area of the tray covered by produce.
Curlander in another embodiment discloses:
subtracting, by the gap percentage calculation module, the estimated area of the identified top surface of the bottom of the tray that is exposed from the total area of the tray to obtain an area of the tray covered by the produce (Col. 17 line 15-20 " Further, the net area of the rear face of the bin 430 may be determined by identifying an area corresponding to each of the items 40, e.g., a cross-sectional area A40 of each of the baseballs, and subtracting the area from a product of the height h of the bin 430 and the width w of the bin 430." The language here indicates subtracting the area of the objects, i.e. produce, from the total area to obtain the area of the exposed rear. A person of ordinary skill in the art would understand that by performing simple algebra, the formula may be rearranged to subtract the area of the rear from the total area to obtain the area of the objects, i.e. produce. This is further discussed in the motivation statement below);
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the invention of Gregory in view of the first embodiment of Curlander with another embodiment of Curlander to incorporate using subtraction to find certain areas.
The motivation for doing so would have been that it would be obvious to try. For it to be obvious to try, there must by a finite number of identified, predictable solutions, with a reasonable expectation of success. In the described scenario of determining areas of the back of a tray (Ab), the produce (Ap), and the total area (Atot) there are three possible formulas of determining a missing area from two known areas: Atot - Ab = Ap (the claimed invention); Atot – Ap = Ab (the invention disclosed by Curlander); Ab + Ap = Atot (a third option not disclosed by the claimed invention or Curlander). A person of ordinary skill in the art would have the knowledge necessary to perform the required algebra to rearrange any one of the above formulas to determine the other two. Using any of the above formula provides the predictable solution, with a reasonable expectation of success, of determining the unknown area. As Curlander already discloses a method of determining the area of the back of the tray (see Col. 17 line 11-15 as taught above) it would have been obvious to subtract that area of the back of the tray from the total area of the tray to obtain the area of the produce, i.e. objects. This would have been obvious to try because there are a finite number of options (three as shown above) to determine a missing area from two other areas and because Curlander discloses one of the options (see Col. 17 line 15-20).
Therefore, it would have been obvious to combine another embodiment of Curlander with Gregory in view of a first embodiment of Curlander.
Gregory in view of Curlander does not disclose marking trays as valid based on comparing a number of pixels of the tray to a predetermined threshold.
Chen discloses:
validating the tray as valid ([n0036] pallets, i.e. trays, are marked as valid when they are determined to be empty. See also [n0147]), by a tray validation module ([n0154] embodiments of the disclosure may be performed by a processor programed for the methods. A processor programmed to perform the method, including marking tray images as valid, is understood as a module, i.e. a tray validation module), using the image of the tray and a first pixel determination technique, wherein validating the tray using the image of the tray and the first pixel determination technique comprises marking the tray as a valid tray if a number of pixels occupied by the tray in the image of the tray is greater than a predetermined threshold value ([n0115] the tray image is binarized meaning that areas of interest in the image, such as the tray or objects on the tray, are given a value of one and other pixels are given a value of zero. [n0117] the sum of the pixels is calculated which is understood as the number of pixels with the value of one. [n0118] the number of pixels is compared to a threshold. [n0119]-[n0120] when the number of pixels exceed a threshold the tray is marked as not empty and when below a threshold the tray is marked as empty. A person of ordinary skill in the art would understand that if the pixels exceeding a threshold indicates that the tray is not empty that the pixels with a value of 1 must indicate objects on the tray, therefore making a large count mean the tray is not empty. Conversely, a person of ordinary skill would be equipped to instead determine the number of pixels of the tray, the zero pixels. If the number of zero pixels were compared to the threshold instead then when they exceed the threshold it would indicate an empty tray. A person of ordinary skill in the art would therefore understand that they may determine the number of pixels of a tray and if they exceed a threshold the tray is empty. [n0036] if the tray is empty it is a valid reference tray, i.e. marked as valid. Therefore, a tray is marked as valid if the number of pixels occupied by the tray exceed a threshold);
Chen is combinable with Gregory in view of Curlander because it is from the related field of endeavor of detecting empty pallets or trays (Chen, [n0001]).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to combine the marking a tray as valid of Chen with the invention of Gregory in view of Curlander.
The motivation for doing so would have been that marking an image as valid allows for use of the image in future identification "for more efficiently detecting whether a pallet is empty" (Chen, [n0036]).
Therefore, it would have been obvious to combine Chen with Gregory in view of Curlander.
Gregory in view of Curlander in further view of Chen does not disclose expressly that identifying the top surface of the bottom of the tray is by using a second deep learning model.
Kim discloses:
wherein the identifying the top surface of the bottom of the tray is by using a second deep learning model ([0050] detect gap or empty space by gap detection neural network).
Kim is combinable with Gregory in view of Curlander in further view of Chen because it is in the related field of endeavor of detecting the levels that a good is stocked (Kim, abstract).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify Gregory in view of Curlander in further view of Chen with the deep learning model of Kim.
The motivation for doing so would have been the deep learning model may detect an out of stock event by determining the area of the top surface of the bottom of the tray and may alert an associate of a replenishment task (Kim, [0054]).
Therefore, it would have been obvious to combine Kim with Gregory in view of Curlander in further view of Chen.
Gregory in view of Curlander in further view of Chen and Kim does not disclose expressly that the method is for estimating produce in a tray and that a first and second deep learning model may be trained from a plurality of images with different colors and textures.
Herz discloses:
A method for estimating a quantity of a produce in a tray ([0002] the invention identifies fresh produce by machine learning at a retail checkout terminal. A checkout terminal is understood to comprise a flat surface which typically houses a scanner and/or scale and may be interpreted as a tray);
wherein the first deep learning model is trained using a plurality of images of trays of different colours, textures (Claim 24, the neural network is trained based on colour and texture features from a plurality of images);
wherein the second deep learning model is trained using a plurality of images of areas exposed in trays having different colours, and textures (Claim 24, the neural network is trained based on colour and texture features from a plurality of images);
Herz is combinable with Gregory in view of Curlander in further view of Chen and Kim because it is in the related field of endeavor of identifying products (Herz, [0002]).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify Gregory in view of Curlander in further view of Chen and Kim to include the training of models with color and texture.
The motivation for doing so would have been that "in combination with more general features such as colour and texture an exceptional level of performance and generalisation can be achieved" (Herz, [0143]). Herz does not disclose training with images of trays but Gregory does (see Gregory [0059] as taught above). Therefore, when considered in combination, the technique of Herz to train with images containing different texture and color is understood to apply to the training of Gregory with images of trays. The motivation above applies by granting a high level of performance and generalization.
Therefore, it would have been obvious to combine Herz with Gregory in view of Curlander in further view of Chen and Kim to obtain the invention as specified in claim 1.
Regarding claim 4, Gregory in view of Curlander in further view of Chen and Kim and Herz discloses the subject matter of claim 1.
Gregory in view of does not disclose expressly that the pixel determination technique comprises determining a number of pixels occupied by the tray, comparing the number of pixels with a predetermined threshold, and marking the tray as valid if the number of pixels is greater than the threshold.
Chen discloses:
wherein validating the tray using the image of the tray and the first pixel determination technique further comprises: determining the number of pixels occupied by the tray in the image of the tray ([n0117] the sum of the pixels is calculated which is understood as determining the number of pixels);
comparing the number of pixels with a predetermined threshold value ([n0118] the number of pixels is compared to a threshold).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify Gregory in view of Curlander with the first pixel determination technique of Chen.
The motivation would have been to determine empty trays for use in future tray identification (Chen, [n0036]).
Therefore it would have been obvious to combine Chen with Gregory in view of Curlander to obtain the invention as specified in claim 4.
Regarding claim 5, Gregory in view of Curlander in further view of Chen and Kim and Herz discloses the subject matter of claim 1.
Gregory further discloses:
estimating the total area of the tray based on a second pixel determination technique ([0047] the area of the container is determined as "a total quantity of pixels associated with the container").
Gregory does not disclose expressly estimating the area of the top surface of the bottom of the tray based on a second pixel determination technique.
Curlander discloses:
estimating the area of the identified top surface of the bottom of the tray that is exposed is based on a second pixel determination technique (Col. 17 line 11-15, "a number of the pixels of the image corresponding to the rear face of the bin 430 may be determined, e.g., by counting, and a net area of the rear face of the bin 430 may be determined based on the combined area of each of such pixels").
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify Gregory with the estimating of the area of the bottom of the tray of Curlander.
The motivation for doing so would have been that “by estimating the dimensions of one or more available volumes of a bin based on imaging data regarding the visible portions of the bin, and comparing dimensions of the estimated available volumes of the bin to dimensions of one or more items, the eligibility of the items to be accommodated within the various estimated available volumes may be determined" (Col. 16 line 39-45 and Fig. 4B). Therefore, it may be determined what other items may be stored in the bins or trays.
Therefore, it would have been obvious to combine the first embodiment of Curlander with Gregory to obtain the invention as specified in claim 5.
Regarding claim 6, Gregory in view of Curlander in further view of Chen and Kim and Herz discloses the subject matter of claim 5.
Gregory further discloses:
wherein estimating the total area of the tray based on the second pixel determination technique comprises: computing the number of pixels of the tray in the image of the tray ([0047] "the image processing module 110B (and/or the classification model) may use image segmentation methods—or similar to identify pixels within the images 300,301 that correspond to edges of the container 204C." Performing segmentation on the edges is understood as computing a number of pixels);
and determining the area based on the number of pixels ([0047] "Each pixel bounded by the edges of the container 204C may be classified as being associated with one of more of the objects 302A-302C or with the container 204C itself." Together, the pixels classified as the object or the container itself are the total area, "total pixels for the container 204C minus those pixels that depict the object(s) present within the container 204C").
Regarding claim 7, Gregory in view of Curlander in further view of Chen and Kim and Herz discloses the subject matter of claim 5. Gregory does not disclose expressly estimating the area of the bottom of the tray by the second pixel determination technique which comprises computing a number of pixels and determining the area based on the number of pixels.
Curlander discloses:
wherein estimating the area of the identified top surface of the bottom of the tray exposed based on the second pixel determination technique comprises: computing a number of pixels occupied by the top surface of the bottom of the tray that is exposed (Col. 15 line 46-48, "a number of pixels corresponding to the visible portions of the rear face of the bin may be determined, such as by counting,");
and determining the area of the identified top surface of the bottom of the tray exposed based on the number of pixels (Col. 15 line 48-50, "and a size of each of the pixels may be geometrically determined according to one or more transformation processes").
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify Gregory with the estimating of the area of the bottom of the tray of Curlander.
The motivation for doing so would have been that “by estimating the dimensions of one or more available volumes of a bin based on imaging data regarding the visible portions of the bin, and comparing dimensions of the estimated available volumes of the bin to dimensions of one or more items, the eligibility of the items to be accommodated within the various estimated available volumes may be determined" (Col. 16 line 39-45 and Fig. 4B). Therefore, it may be determined what other items may be stored in the bins or trays.
Therefore, it would have been obvious to combine the first embodiment of Curlander with Gregory to obtain the invention as specified in claim 7.
Regarding claim 8, claim 8 recites a system with elements corresponding to the steps recited in claim 1. Therefore, the recited elements of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim, claim 1. Additionally, the rationale and motivation to combine Gregory in view of Curlander in further view of Chen and Kim and Herz, presented in rejection of claim 1, apply to this claim. Finally, Gregory discloses:
A system (100) for estimating a quantity of a produce in a tray, the system (100) comprising: a camera (115) configured for capturing an image, the image having an image of the tray ([0032] a camera 106 for capturing images is a component of computing device 102. See also Fig. 2, 102 is represented as a camera disposed to capture images of the trays);
and a management server (105) ([0033] and Fig. 1, a server 110) comprising a processor (210) and a memory module (215) storing instructions to be executed by the processor (210) ([0033] and Fig. 1, a server 110 comprises computing devices comprising a storage module, i.e. a memory module. [0083] and Fig. 8, a server 802 may also comprise a processor 808 configured to execute software, i.e. instructions, from memory 810),
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Gregory et al. (U.S. Publ. No. 20230081303; hereafter, Gregory) in view of Curlander et al. (U.S. Publ. No. 9460524 B1; hereafter, Curlander) in further view of Chen et al. (CN 113033545 A; hereafter, Chen) and Kim et al. (U.S. Publ. No. 20230177458; hereafter, Kim) and Herz et al. (U.S. Publ. No. 20200364501) and Vepakomma et al. (U.S. Publ. No. 20180150788; hereafter, Vepakomma).
Regarding claim 2, Gregory in view of Curlander in further view of Chen and Kim and Herz discloses the subject matter of claim 1.
Gregory in view of Curlander in further view of Chen does not disclose expressly detecting obstructions in the received image.
Kim discloses:
obstructions in the received image ([0038] the system is programmed to detect obstructions and ignore if the obstruction does not prevent object detection).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify Gregory in view of Curlander in further view of Chen with the obstruction detection of Kim.
The motivation for doing so would have been that “in one aspect, when the obstruction detection neural network model detects an unacceptable number (e.g., 1, 2, 3, etc.) of obstructions in the obtained image captured by the image capture device 120, the control circuit 210 is programmed to discard the obtained image without performing an out of stock analysis with respect to the portion of the product display shelf 110 depicted the obtained image” (Kim, [0038]). This gives the benefit of not producing an out of stock message due to an obstruction.
Therefore, it would have been obvious to combine Kim with Gregory in view of Curlander in further view of Chen.
Gregory in view of Curlander in further view of Chen and Kim and Herz does not disclose expressly processing received images to remove noise.
Vepakomma discloses:
processing, by the processor ([0006] the inventory control system comprises a processor), the received image to remove noise ([0041] noise is removed from the image).
Vepakomma is combinable with Gregory in view of Curlander in further view of Chen and Kim and Herz because it is in the related field of endeavor of determining stock level from an image (Vepakomma, abstract).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify Gregory in view of Curlander in further view of Chen and Kim and Herz with the noise removal taught by Vepakomma.
The motivation for doing so would have been to remove blur and unwanted signals (Vepakomma, [0041]).
Therefore, it would have been obvious to combine Vepakomma with Gregory in view of Curlander in further view of Chen and Kim and Herz to obtain the invention as specified in claim 2.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOSHUA B CROCKETT whose telephone number is (571)270-7989. The examiner can normally be reached Monday-Thursday 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John M Villecco can be reached on (571) 272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOSHUA B. CROCKETT/Examiner, Art Unit 2661
/JOHN VILLECCO/Supervisory Patent Examiner, Art Unit 2661
Prosecution Insights