DETAILED ACTION
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 04/10/2026 has been entered.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
For the purpose of prior art consideration, the effective filing date of the instant application is based on the application filed in Japan on December 23rd, 2022. However, priority to this date is not perfected until an English translation of the certified copy of the instant application is filed.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 3-8 and 10-15 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Whether a Claim is to a Statutory Category
In the instant case, claims 1 and 3-7 recite a non-transitory computer readable storage medium/machine; claims 8 and 10-14 recite a method/process and claim 15 recites an apparatus/machine that are performing a series of functions. Therefore, these claims fall within the four statutory categories of invention of a machine. Step 1 is satisfied.
Step2A – Prong 1: Does the Claim Recite a Judicial Exception
Exemplary claim 1 recites the following abstract concepts that are found to include an enumerated “abstract idea”:
A non-transitory computer-readable storage medium storing an alert generation program that causes at least one computer to execute a process, the process comprising:
acquiring a video of a person who holds a merchandise to be registered in a checkout machine;
specifying merchandise candidates corresponding to merchandises included in the video and a number of the merchandise candidates by inputting the acquired video to a trained machine learning model including a pair of an image encoder and a text encoder, the image encoder being a multi-layer neural network configured to output, in response to an input image, a vector for embedding the input video into a multimodal feature space, the text encoder being a multi-layer neural network configured to output, in response to an input text, a vector for embedding the input text into the multimodal feature space;
acquiring, through scanning or manually inputting, by the person, of merchandise codes printed or attached to the merchandises, items of merchandises registered by the person and a number of the items of the merchandises; and
generating an alert indicating an abnormality of merchandises registered in the checkout machine based on the acquired items of the merchandises and the number of the items of the merchandises not coinciding with the specified merchandise candidates and the number of the merchandise candidates, wherein the specifying includes:
inputting the video to the image encoder included in the machine learning model to obtain, from the image encoder, a first vector being a vector for embedding the video into the multimodal feature space,
inputting, to the text encoder included in the machine learning model, a plurality of texts in accordance with a hierarchical structure including a plurality of hierarchies corresponding to attributes of merchandises and a number of merchandises to obtain a plurality of second vectors from the text encoder by obtaining, for each text of the plurality of texts, a second vector from the text encoder, the second vector beinq a vector for embedding the each text into the multimodal feature space, and
specifying the merchandise candidates corresponding to the merchandises included in the video and the number of the merchandise candidates based on maximizing, for each of the plurality of hierarchies, a similarity between the first vector obtained from the image encoder and each second vector of the plurality of second vectors obtained from the text encoder.
[Emphasis added to show the bolded abstract idea being executed by unbolded additional elements that do not meaningfully limit the abstract idea]
This non-transitory computer-readable storage medium claim is grouped within the "certain methods of organizing human activity” grouping of abstract ideas in prong one of step 2A of the Alice/Mayo test because the claims involve a series of steps for sales activities of acquiring items of merchandises registered by the person, which is a process that is encompassed by the abstract idea of commercial or legal interactions. See e.g., MPEP 2106.04(a)(2) and Subject Matter Eligibility Example 47. Accordingly, claim 1 (and similarly claims 8 and 15) are found to recite abstract idea(s).
Step2A – Prong 2: Does the Claim Recite Additional Elements that Integrate the Judicial Exception into a Practical Application
This judicial exception is not integrated into a practical application because, when analyzed under prong two of step 2A of the Alice/Mayo test, the additional elements of the claims such as non-transitory computer-readable storage medium, computer, machine learning model, image encoder, text encoder, multi-layer neural network and checkout machine merely use a computer as a tool to perform an abstract idea and/or generally link the use of a judicial exception to a particular technological environment. Specifically, the non-transitory computer-readable storage medium, computer, machine learning model, image encoder, text encoder, multi-layer neural network and checkout machine performs the steps or functions of sales activities of acquiring items of merchandises registered by the person. The use of a processor/computer as a tool to implement the abstract idea and/or generally linking the use of the abstract idea to a particular technological environment does not integrate the abstract idea into a practical application because it requires no more than a computer (or technical elements disclosed at a high level of generality such as non-transitory computer-readable storage medium, computer, machine learning model, image encoder, text encoder, multi-layer neural network and checkout machine) performing functions of storing, acquiring, specifying, inputting, outputting, embedding, scanning, generating, obtaining and maximizing that correspond to acts required to carry out the abstract idea (MPEP 2106.05(f) and (h)). Accordingly, the additional elements do not impose any meaningful limits on practicing the abstract idea, and the claims are directed to an abstract idea.
Step2B: Does the Claim Amount to Significantly More
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. The additional element analysis of Step 2A Prong 2 is equally applied to Step 2B. “Another consideration when determining whether a claim recites significantly more than a judicial exception is whether the additional element(s) are well-understood, routine, conventional activities previously known to the industry. This consideration is only evaluated in Step 2B of the eligibility analysis.” MPEP 2106.05(d). The courts have recognized the following computer functions as well‐understood, routine, and conventional (“WURC”) functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity. Exemplary claim 1 (and similarly claims 8 and 15) recites the following limitations that the courts have found to be WURC:
Claim 1 includes several limitations relating to receiving or transmitting data over a network (acquiring a video of a person who holds a merchandise …, acquiring, through scanning or manually inputting, by the person, of merchandise codes printed or attached to the merchandises, items of merchandises registered by the person … as claimed) data. See MPEP 2106.05(d)(II) where courts found to be WURC - i. Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network); but see DDR Holdings, LLC v. Hotels.com, L.P., 773 F.3d 1245, 1258, 113 USPQ2d 1097, 1106 (Fed. Cir. 2014) ("Unlike the claims in Ultramercial, the claims at issue here specify how interactions with the Internet are manipulated to yield a desired result‐‐a result that overrides the routine and conventional sequence of events ordinarily triggered by the click of a hyperlink." (emphasis added));
Claim 1 includes several limitations relating to performing repetitive calculations (specifying merchandise candidates … by inputting the acquired video to a trained machine learning model including a pair of an image encoder and a text encoder; inputting the video to the image encoder included in the machine learning model to obtain, from the image encoder, a first vector…; inputting, to the text encoder included in the machine learning model, a plurality of texts in accordance with a hierarchical structure … to obtain a plurality of second vectors from the text encoder; and specifying the merchandise candidates … based on maximizing … a similarity between the first vector obtained from the image encoder and each second vector of the plurality of second vectors obtained from the text encoder as claimed) data. See MPEP 2106.05(d)(II) where courts found to be WURC - ii. Performing repetitive calculations, Flook, 437 U.S. at 594, 198 USPQ2d at 199 (recomputing or readjusting alarm limit values); Bancorp Services v. Sun Life, 687 F.3d 1266, 1278, 103 USPQ2d 1425, 1433 (Fed. Cir. 2012) ("The computer required by some of Bancorp’s claims is employed only for its most basic function, the performance of repetitive calculations, and as such does not impose meaningful limits on the scope of those claims.");
Claim 1 includes several limitations relating to storing and retrieving information in memory (generating an alert … based on the acquired items of the merchandises and the number of the items of the merchandises not coinciding with the specified merchandise candidates and the number of the merchandise candidates as claimed) data. See MPEP 2106.05(d)(II) where courts found to be WURC - iv. Storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93;
Accordingly, when viewed alone and in ordered combination, these additional elements are not found to recite significantly more than the underlying abstract idea.
Independent claim 8 describes a method to perform functions of storing, acquiring, specifying, inputting, outputting, embedding, scanning, generating, obtaining and maximizing relating to sales activities of acquiring items of merchandises registered by the person without additional elements beyond technical elements disclosed at a high level of generality such as an computer, machine learning model, image encoder, text encoder, multi-layer neural network and checkout machine that provide significantly more than the abstract idea of commercial or legal interactions through sales activities of acquiring items of merchandises registered by the person as noted above regarding claim 1. Therefore, this independent claim is also not patent eligible.
Independent claim 15 describes an apparatus to perform functions of storing, acquiring, specifying, inputting, outputting, embedding, scanning, generating, obtaining and maximizing relating to sales activities of acquiring items of merchandises registered by the person without additional elements beyond technical elements disclosed at a high level of generality such as a memories, processors, machine learning model, image encoder, text encoder, multi-layer neural network and checkout machine that provide significantly more than the abstract idea of commercial or legal interactions through sales activities of acquiring items of merchandises registered by the person as noted above regarding claim 1. Therefore, this independent claim is also not patent eligible.
Dependent claims 3-7 and 10-14 further describes the abstract idea of commercial or legal interactions. Dependent claims 3-7 and 10-14 add inputting, specifying, outputting, referring, narrowing down, registering, acquiring and generating steps that are executed by a non-transitory computer readable medium, image encoder, machine learning model, text encoder, checkout machine, display and as disclosed in independent claims 1 and 8, however these additional steps remain disclosed at a high level of generality and do not amount to more than mere computer implementation of the abstract idea, which does not integrate the abstract idea into a practical application or provide significantly more than the abstract idea. Therefore, dependent claims 3-7 and 10-14 are also not patent eligible. Further, the dependency of these claims on ineligible independent claims 1 and 8 also renders dependent claims 3-7 and 10-14 as not patent eligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-6, 8, 10-13 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Krishnamurthy et al. (US 2022/0414374 A1) in view of Arroyo et al. (US 2023/0230408 A1).
Regarding Claims 1, 8 and 15, modified Krishnamurthy teaches:
A non-transitory computer-readable storage medium storing an alert generation program that causes at least one computer to execute a process/ An alert generation method for a computer to execute a process/ An information processing apparatus comprising: one or more memories; and one or more processors coupled to the one or more memories (See Krishnamurthy ¶ [0084] – outputting an alert based on item identification and [0092] – memory storing instructions to implement the functions of the system when executed by a processor, wherein said memory comprises one or more disks, tape drives, or solid-state drives, and may be used as an over-flow data storage device [non-transitory computer-readable storage medium by example]), the process comprising/ the one or more processors configured to:
acquiring a video of a person who holds a merchandise to be registered in a checkout machine (See Krishnamurthy ¶ [0054] – capturing images with video cameras, [0087] - the item tracking device may identify a digital cart that is associated with the user. In this example, the digital cart comprises information about items that the user has placed on the platform to purchase and [0095] – The item tracking system may employ process to detect a triggering event that corresponds with when a user puts their hand above the platform to place an item on the platform);
specifying merchandise candidates corresponding to merchandises included in the video and a number of the merchandise candidates by inputting the acquired video to a trained machine learning model (See Krishnamurthy ¶ [0078-0079] – identifying items with a machine learning model based on captured image data of said items, wherein the item tracking device may then compare the identified features of the item to a set of features that correspond with different items [merchandise candidates by example]) including a pair of an image encoder (See Krishnamurthy ¶ [0078-0079] – identifying items with a machine learning model based on captured image data of said items, wherein the item tracking device may then compare the identified features of the item to a set of features that correspond with different items [merchandise candidates by example]) …, the image encoder being a multi-layer neural network configured to output, in response to an input image, a vector for embedding the input video into a multimodal feature space (See Krishnamurthy ¶ [0054] – video cameras are one of many types of cameras used, [0059] – machine learning models include, but are not limited to, a multi-layer perceptron, a recurrent neural network (RNN), an RNN long short-term memory (LSTM), a convolution neural network (CNN), a transformer, or any other suitable type of neural network model …the machine learning model is generally configured to receive an image as an input and to output an item identifier based on the provided image and [0146-0149] – The item tracking system may employ process to filter the entries, comprising text and barcodes [image/ video output] to identify items [multimodal feature space by example], in the encoded vector library to reduce the amount of items that are considered when attempting to identify an item that is placed on the platform… the item tracking device obtains feature descriptors for an item … each entry may comprise an encoded vector that is linked with an item identifier and a plurality of feature descriptors), … being a multi-layer neural network configured to output, in response to an input text, a vector for embedding the input text into the multimodal feature space (See Krishnamurthy ¶ [0059] and [0146-0149] as noted above – machine learning models processing both text and image data);
acquiring, through scanning or manually inputting, by the person, of merchandise codes printed or attached to the merchandises, items of merchandises registered by the person and a number of the items of the merchandises (See Krishnamurthy ¶ [0195] – the item tracking device may provide instructions for the user to scan a barcode of an item using a barcode scanner. In this case, the item tracking device may use the graphical user interface to display a combination of items that were detected on the platform as well as items that were manually scanned by the user [number of items by example]); and
generating an alert indicating an abnormality of merchandises registered in the checkout machine based on the acquired items of the merchandises and the number of the items of the merchandises not coinciding with the specified merchandise candidates and the number of the merchandise candidates (See Krishnamurthy ¶ [0081-0082] – the item tracking device may compare the number of identified items from the captured images to the number of items on the platform that was determined in operation… the item tracking device determines that at least one of the items has not been identified when the number of items identified items from the captured images does not match the determined number of items on the platform… the item tracking device may output a message [alert by example] on a graphical user interface that is located at the imaging device with instructions for the user to rearrange the position of the items on the platform) wherein the specifying includes:
inputting the video to the image encoder included in the machine learning model (See Krishnamurthy ¶ [0078-0079] – identifying items with a machine learning model based on captured image data of said items and [0146] – The item tracking system may employ process to filter the entries in the encoded vector library to reduce the amount of items that are considered when attempting to identify an item that is placed on the platform) to obtain, from the image encoder, a first vector being a vector for embedding the video into the multimodal feature space (See Krishnamurthy ¶ [0054] – video cameras are one of many types of cameras used, [0059] – the machine learning model is generally configured to receive an image as an input and to output an item identifier based on the provided image, [0146-0149] – The item tracking system may employ process to filter the entries, comprising text and barcodes [image/ video output] to identify items [multimodal feature space by example], in the encoded vector library to reduce the amount of items that are considered when attempting to identify an item that is placed on the platform… the item tracking device obtains feature descriptors for an item … an encoded vector comprises an array of numerical values. Each numerical value corresponds with and describes an attribute (e.g. item type, size, shape, color, etc.) of an item []),
inputting, …, a plurality of texts in accordance with a hierarchical structure including a plurality of hierarchies corresponding to attributes of merchandises and a number of merchandises to …, the … vector being a vector for embedding the each text into the multimodal feature space, (See Krishnamurthy ¶ [0078-0079] – identifying items with a machine learning model based on captured image data of said items, wherein the item tracking device may then compare the identified features of the item to a set of features that correspond with different items [merchandise by example], [0146-0149] – The item tracking system may employ process to filter the entries, comprising text and barcodes [image/ video output] to identify items [multimodal feature space by example], in the encoded vector library to reduce the amount of items that are considered when attempting to identify an item that is placed on the platform… the item tracking device obtains feature descriptors for an item … describing the physical characteristics or attributes of an item, including an item type… an item type identifies a classification for the item, wherein said item classification is understood herein to define the limitation hierarchies based on the specification of the instant application … each entry may comprise an encoded vector that is linked with an item identifier and a plurality of feature descriptors … examples of item identifiers include, a product name, an SKU number, an alphanumeric code [text by example]), and
specifying the merchandise candidates corresponding to the merchandises included in the video and the number of the merchandise candidates based on maximizing, for each of the plurality of hierarchies, a similarity between the first vector obtained from the image encoder and each … vector of the plurality of … vectors obtained … (See Krishnamurthy ¶ [0078-0079] – identifying items with a machine learning model based on captured image data of said items, wherein, the item tracking device may then compare the identified features of the item to a set of features that correspond with different items [merchandise candidates by example], [0146-0149] – The item tracking system may employ process to filter the entries, comprising text and barcodes [image/ video output] to identify items, in the encoded vector library [reference source data] to reduce the amount of items that are considered when attempting to identify an item that is placed on the platform… the item tracking device obtains feature descriptors for an item … describing the physical characteristics or attributes of an item, including an item type… an item type identifies a classification for the item [hierarchies] and [0158-0159] - the item tracking device may generate a similarity vector for a received encoded vector. A similarity vector comprises an array of numerical values where each numerical value indicates how similar the values in the received encoded vector are to the values in an encoded vector in the encoded vector library [plurality of vectors by example]… After generating the similarity vector, the item tracking device can identify which entry, or entries, in the encoded vector library most closely matches the encoded vector for the identified item. In one embodiment, the entry that is associated with the highest numerical value [maximizing] in the similarity vector corresponds is the entry that closest matches the encoded vector for the item) ...
While Krishnamurthy teaches a machine learning based monitored checkout process that captures and analyzes image and text data to identify items for purchase (Krishnamurthy ¶ [0078-0082]), Krishnamurthy does not explicitly teach that said text data is input and output by a text encoder that is included in the machine learning model to process said text data. This is taught by Arroyo (See Arroyo ¶ [0109-0110] – the classification circuitry applies an example NLP-based model to classify the columns into the different types of facts using the detected column headers. For example, the columns can be classified into different product facts, such as code, description, quantity and price … the classification circuitry includes an example column classification model(s), which is a trained AI model that classifies text data output by the OCR circuitry. In some examples, the column classification model is based on a simple neural network that contains one layer, such as fastText, fastText is an open-source library for learning of word embeddings and text classification that can be used to train large datasets quickly and efficiently).
Arroyo further teaches obtaining a plurality of second vectors from the text encoder by obtaining, for each text of the plurality of texts, a second vector from the text encoder (See Arroyo ¶ [0120] – The encoder circuitry encodes text segments using a transformer to generate text embeddings. The transformer processes an input text sequence to generate rich embeddings for each token. For example, the transformer can convert text data into tokens, which are part of a fixed vocabulary. The tokens can be converted into embedding vectors using a fixed representation. The transformer also adds a positional encoding vector to each token embedding, obtaining a special embedding with positional information). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include in the machine learning based monitored checkout process that analyzes image and text data of Krishnamurthy the use of a text encoder model to analyze text data as taught by Arroyo to provide improvements in the efficiency and productivity of the decoding process by applying the ground truth annotations to the ML models (Arroyo ¶ [0050]), thereby increasing the accuracy of the machine learning based checkout monitoring process of Krishnamurthy.
Regarding Claim 3 and 10, modified Krishnamurthy teaches:
The non-transitory computer-readable storage medium/ alert generation method according to claim 1 and 8, wherein
the machine learning model refers to reference source data in which an attribute of a merchandise is associated with each of a plurality of hierarchies (See Krishnamurthy ¶ [0078-0079] – identifying items with a machine learning model based on captured image data of said items and [0146-0148] – The item tracking system may employ process to filter the entries in the encoded vector library [reference source data] to reduce the amount of items that are considered when attempting to identify an item that is placed on the platform… the item tracking device obtains feature descriptors for an item … describing the physical characteristics or attributes of an item, including an item type… an item type identifies a classification for the item, wherein said item classification is understood herein to define the limitation hierarchies based on the specification of the instant application), and
the specifying includes:
specifying the merchandise candidates and the number of the merchandise candidates by inputting the video to the image encoder (See Krishnamurthy ¶ [0147] – using text and barcodes [image/ video output] to identify items and [0158] - the item tracking device may generate a similarity vector for a received encoded vector. A similarity vector comprises an array of numerical values where each numerical value indicates how similar the values in the received encoded vector are to the values in an encoded vector in the encoded vector library),
inputting a text … for each of attributes of merchandises in a first hierarchy (See Krishnamurthy ¶ [0147] – using text and barcodes [image/ video output] to identify items and [0158] - the item tracking device may generate a similarity vector for a received encoded vector. A similarity vector comprises an array of numerical values where each numerical value indicates how similar the values in the received encoded vector are to the values in an encoded vector in the encoded vector library),
narrowing down attributes corresponding to the merchandises included in the video among the attributes of the merchandises in the first hierarchy based on a similarity between a vector of the video output by the image encoder and a vector of the text … (See Krishnamurthy ¶ [0146-0148] – The item tracking system may employ process to filter the entries, comprising text and barcodes [image/ video output] to identify items, in the encoded vector library [reference source data] to reduce the amount of items that are considered when attempting to identify an item that is placed on the platform… the item tracking device obtains feature descriptors for an item … describing the physical characteristics or attributes of an item, including an item type… an item type identifies a classification for the item [hierarchies] and [0158] - the item tracking device may generate a similarity vector for a received encoded vector. A similarity vector comprises an array of numerical values where each numerical value indicates how similar the values in the received encoded vector are to the values in an encoded vector in the encoded vector library),
inputting the video to the image encoder (See Krishnamurthy ¶ [0147] and [0158] as noted above),
inputting a text … for each merchandise in a second hierarchy narrowed down from among the attributes of the merchandises in the first hierarchy (See Krishnamurthy ¶ [0149-0150] - the item tracking device filters the encoded vector library based on the item type… the encoded vector library comprises a plurality of entries, each entry corresponds with a different item that can be identified by the item tracking device, each entry may comprise an encoded vector that is linked with an item identifier and a plurality of feature descriptors … Examples of item identifiers include, a product name [text], an SKU number [text], an alphanumeric code [text], a graphical code (e.g. a barcode) [video/ image], or any other suitable type of identifier… the item tracking device uses the item type to filter out or remove any entries in the encoded vector library that do not contain the same item type. This process reduces the number of entries in the encoded vector library that will be considered when identifying the item), and
specifying a number corresponding to the merchandises included in the video among the numbers of the merchandises in the second hierarchy based on a similarity between a vector of the video output by the image encoder and a vector of the text … (See Krishnamurthy ¶ [0147-0150] & [0158] as noted above and [0193] - the item tracking device determines whether the number of identified items matches the number of items that were detected on the platform in operation. The item tracking device determines that all of the items have been identified when the number of identified items matches the number of items that were detected on the platform).
While Krishnamurthy teaches a machine learning based monitored checkout process that captures and analyzes image and text data to identify items for purchase (Krishnamurthy ¶ [0078-0082]), Krishnamurthy does not explicitly teach that said text data is input and output by a text encoder to process said text data. This is taught by Arroyo (See Arroyo ¶ [0109-0110] – the classification circuitry applies an example NLP-based model to classify the columns into the different types of facts using the detected column headers. For example, the columns can be classified into different product facts, such as code, description, quantity and price … the classification circuitry includes an example column classification model(s), which is a trained AI model that classifies text data output by the OCR circuitry. In some examples, the column classification model is based on a simple neural network that contains one layer, such as fastText, fastText is an open-source library for learning of word embeddings and text classification that can be used to train large datasets quickly and efficiently). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include in the machine learning based monitored checkout process that analyzes image and text data of Krishnamurthy the use of a text encoder model to analyze text data as taught by Arroyo to provide improvements in the efficiency and productivity of the decoding process by applying the ground truth annotations to the ML models (Arroyo ¶ [0050]), thereby increasing the accuracy of the machine learning based checkout monitoring process of Krishnamurthy.
Regarding Claim 4 and 11, modified Krishnamurthy teaches:
The non-transitory computer-readable storage medium/ alert generation method according to claim 1 and 8, wherein
the machine learning model refers to reference source data in which an attribute of a merchandise is associated with each of a plurality of hierarchies (See Krishnamurthy ¶ [0078-0079] – identifying items with a machine learning model based on captured image data of said items and [0146-0148] – The item tracking system may employ process to filter the entries in the encoded vector library [reference source data] to reduce the amount of items that are considered when attempting to identify an item that is placed on the platform… the item tracking device obtains feature descriptors for an item … describing the physical characteristics or attributes of an item, including an item type… an item type identifies a classification for the item, wherein said item classification is understood herein to define the limitation hierarchies based on the specification of the instant application), and
the specifying includes:
specifying the merchandise candidates and the number of the merchandise candidates by inputting the video to the image encoder (See Krishnamurthy ¶ [0147] – using text and barcodes [image/ video output] to identify items and [0158] - the item tracking device may generate a similarity vector for a received encoded vector. A similarity vector comprises an array of numerical values where each numerical value indicates how similar the values in the received encoded vector are to the values in an encoded vector in the encoded vector library),
inputting a text … for each merchandise in a first hierarchy (See Krishnamurthy ¶ [0147] – using text and barcodes [image/ video output] to identify items and [0158] - the item tracking device may generate a similarity vector for a received encoded vector. A similarity vector comprises an array of numerical values where each numerical value indicates how similar the values in the received encoded vector are to the values in an encoded vector in the encoded vector library),
narrowing down numbers corresponding to the merchandises included in the video among the numbers of the merchandises in the first hierarchy based on a similarity between a vector of the video output by the image encoder and a vector of the text … (See Krishnamurthy ¶ [0146-0150] – The item tracking system may employ process to filter the entries, comprising text, SKU numbers and barcodes [image/ video output] to identify items, in the encoded vector library [reference source data] to reduce the amount of items that are considered when attempting to identify an item that is placed on the platform… the item tracking device obtains feature descriptors for an item … describing the physical characteristics or attributes of an item, including an item type… an item type identifies a classification for the item [hierarchies] and [0158] - the item tracking device may generate a similarity vector for a received encoded vector. A similarity vector comprises an array of numerical values where each numerical value indicates how similar the values in the received encoded vector are to the values in an encoded vector in the encoded vector library),
inputting the video to the image encoder (See Krishnamurthy ¶ [0147] and [0158] as noted above), inputting a text … for each of attributes of merchandises in a second hierarchy narrowed down from among the numbers of the merchandises in the first hierarchy (See Krishnamurthy ¶ [0149-0150] - the item tracking device filters the encoded vector library based on the item type… the encoded vector library comprises a plurality of entries, each entry corresponds with a different item that can be identified by the item tracking device, each entry may comprise an encoded vector that is linked with an item identifier and a plurality of feature descriptors … Examples of item identifiers include, a product name [text], an SKU number [text], an alphanumeric code [text], a graphical code (e.g. a barcode) [video/ image], or any other suitable type of identifier… the item tracking device uses the item type to filter out or remove any entries in the encoded vector library that do not contain the same item type. This process reduces the number of entries in the encoded vector library that will be considered when identifying the item), and
specifying attributes corresponding to the merchandises included in the video among the attributes of the merchandises in the second hierarchy based on a similarity between a vector of the video output by the image encoder and a vector of the text … (See Krishnamurthy ¶ [0149-0150] - the item tracking device filters the encoded vector library based on the item type… the encoded vector library comprises a plurality of entries, each entry corresponds with a different item that can be identified by the item tracking device, each entry may comprise an encoded vector that is linked with an item identifier and a plurality of feature descriptors … Examples of item identifiers include, a product name [text], an SKU number [text], an alphanumeric code [text], a graphical code (e.g. a barcode) [video/ image], or any other suitable type of identifier… the item tracking device uses the item type to filter out or remove any entries in the encoded vector library that do not contain the same item type. This process reduces the number of entries in the encoded vector library that will be considered when identifying the item and [0158] - the item tracking device may generate a similarity vector for a received encoded vector. A similarity vector comprises an array of numerical values where each numerical value indicates how similar the values in the received encoded vector are to the values in an encoded vector in the encoded vector library).
While Krishnamurthy teaches a machine learning based monitored checkout process that captures and analyzes image and text data to identify items for purchase (Krishnamurthy ¶ [0078-0082]), Krishnamurthy does not explicitly teach that said text data is input and output by a text encoder to process said text data. This is taught by Arroyo (See Arroyo ¶ [0109-0110] – the classification circuitry applies an example NLP-based model to classify the columns into the different types of facts using the detected column headers. For example, the columns can be classified into different product facts, such as code, description, quantity and price … the classification circuitry includes an example column classification model(s), which is a trained AI model that classifies text data output by the OCR circuitry. In some examples, the column classification model is based on a simple neural network that contains one layer, such as fastText, fastText is an open-source library for learning of word embeddings and text classification that can be used to train large datasets quickly and efficiently). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include in the machine learning based monitored checkout process that analyzes image and text data of Krishnamurthy the use of a text encoder model to analyze text data as taught by Arroyo to provide improvements in the efficiency and productivity of the decoding process by applying the ground truth annotations to the ML models (Arroyo ¶ [0050]), thereby increasing the accuracy of the machine learning based checkout monitoring process of Krishnamurthy.
Regarding Claim 5 and 18, modified Schuessler teaches:
The non-transitory computer-readable storage medium/ alert generation method according to claim 1 and 8, wherein
the checkout machine is configured to register items of merchandises selected by the person and numbers of the items of the merchandises from among a list of merchandises output in a display of the checkout machine (See Krishnamurthy ¶ [0087] - the item tracking device may identify a digital cart that is associated with the user. In this example, the digital cart comprises information about items that the user has placed on the platform to purchase [registers items] and [0192] – the item tracking device may output the identified item identifier for the identified item by adding the item identifier to a list of identified items that is on a graphical user interface), and
the acquiring includes acquiring the items of the merchandises and the numbers of the items of the merchandises from the checkout machine (See Krishnamurthy ¶ [0081-0082] – the item tracking device may compare the number of identified items from the captured images to the number of items on the platform that was determined in operation…).
Regarding Claim 6 and 13, modified Schuessler teaches:
The non-transitory computer-readable storage medium/ alert generation method according to claim 1 and 8, wherein
the generating includes generating an alert for warning that the specified number of the merchandise candidates and the number of the items of the merchandises acquired from the checkout machine do not coincide (See Krishnamurthy ¶ [0081-0082] – the item tracking device may compare the number of identified items from the captured images to the number of items on the platform that was determined in operation… the item tracking device determines that at least one of the items has not been identified when the number of items identified items from the captured images does not match the determined number of items on the platform… the item tracking device may output a message [alert by example] on a graphical user interface that is located at the imaging device with instructions for the user to rearrange the position of the items on the platform).
Claims 7 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Krishnamurthy et al. (US 2022/0414374 A1) in view of Arroyo et al. (US 2023/0230408 A1) and Guo et al. (US 2020/0357045 A1).
Regarding Claim 7 and 14, modified Krishnamurthy teaches:
The non-transitory computer-readable storage medium according to claim 1 (See claim 1 and 8 as noted above), wherein
While Krishnamurthy teaches a monitored checkout process for registering items for purchase and generating an alert when an abnormality is detected (Krishnamurthy ¶ [0078-0082]), Krishnamurthy does not explicitly teach that said generating includes generating an alert including one selected from a difference of a purchase amount based on the specified number of the merchandise candidates and the number of the items of the merchandises acquired from the checkout machine, and identification information of the checkout machine. This is taught by Guo (See Guo ¶ [0036] – a computing device is a checkout machine or a POS terminal device [identification information by example] and [0089-0090] – the price difference between the and the selected product candidate and the top ranked product candidate will trigger the verification process. For example, if the price difference is greater than a threshold, e.g., 20%, the verification process may be triggered, and meanwhile, an alert may be generated. If a potential mismatch is detected, the system may temporally suspend the ongoing transaction and generate an alert for the potential mismatch). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include in the monitored checkout process of Krishnamurthy the use of purchase amount comparison between registered items and items expected to be registered through monitoring of the item registration process as taught by Guo to improve a computing device's ability to detect unpackaged, unlabeled, or mislabeled products (Guo ¶ [0006]), thereby increasing the accuracy of the checkout monitoring process of Krishnamurthy.
Response to Arguments
Applicant's arguments filed 04/10/2026 have been fully considered but they are not persuasive.
Rejection under 35 U.S.C. § 101:
The amendments to independent claims 1, 8 and 15 do not improve patent eligibility of the claimed invention of the instant application and the previous rejection under 35 U.S.C. § 101 in maintained.
Contrary to the applicant’s assertion that amended independent claims 1, 8 and 15 show architectural specificity, involving multi-layer neural networks for both image and text encoders and their joint embedding into a multi-modal feature space, constitutes a clear improvement in AI technology or a technical field of AI itself, said amendments leave the methods of said claims as executed by technical elements disclosed at a high level of generality such that said methods are not more than merely applying a computer to perform the functions required by said methods, which does not show integration into a practical application nor does it show significantly more than the abstract ideas discussed above in the current rejection under 35 U.S.C. § 101. The claims as they are currently amended, as a whole, show results being generated by technical elements, as outputs, without showing how said results are generated. This does not amount to more than merely “applying” said technical elements to achieve said results and does not clearly reflect improvement to the underlying technology. This is unlike the decision on Desjardins since Desjardins was determined to be eligible based a technological improvement being reflected in the claims. Any improvement of a claimed invention must be clearly reflected by said claims. The specification of an instant application is not read into the claims during examination.
Rejection under 35 U.S.C. § 103:
The amendments to independent claims 1, 8 and 15 do not overcome the prior art combination of record. Therefore, the previous rejection under 35 U.S.C. § 103 is maintained.
Contrary to the applicant’s assertion that Krishnamurthy lacks any disclosure of inputting text into a text encoder to obtain text vectors or, more importantly, embedding both image and text into a multi-modal feature space for unified similarity assessment, these features are taught by the combination of Krishnamurthy and Arroyo as noted above in the current rejection under 35 U.S.C. § 103. The applicant fails to consider the prior art references in their entirety and argues against the references individually when it is that combination of references that is the basis of the rejection. Krishnamurthy, as newly cited, teaches using a machine learning model comprising a multi-layered neural network to encode text and image data in a feature space comprising both text and image data. Arroyo, as newly cited, explicitly teaches using a text encoder to embed vectors for text in a sequence of text, thereby embedding a plurality of text vectors by example.
The examiner agrees with the applicant in that Krishnamurthy does not teach generating multiple text vectors or their use directly in similarity comparison with an image vector in a multi-modal space. These features are taught by the combination of Krishnamurthy and Arroyo as noted above in the current rejection under 35 U.S.C. § 103.
No new arguments are presented for dependent claims 3-7 and 10-14 and these claims remain rejected for their dependency on their respective independent base claims as well as the reasons noted above in the current rejection under 35 U.S.C. § 103.
The applicant is generally reminded that prior art must be considered in its entirety (MPEP 2141.02 (VI)). Moreover, in response to applicant's arguments against the references individually, one cannot show non-obviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW S WERONSKI whose telephone number is (571)272-5802. The examiner can normally be reached M-F 8 am - 5 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Fahd A. Obeid can be reached at 5712703324. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW S WERONSKI/Examiner, Art Unit 3627
/PETER LUDWIG/Primary Examiner, Art Unit 3627
Prosecution Insights