DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (“IDS”) filed on 12/19/2024 has been reviewed and the listed references have been considered.
Drawings
The 15-page drawings have been considered and placed on record in the file.
Status of Claims
Claims 1-20 are pending.
Claim Objections
Claims 2, 10-11, and 15-16 are objected to because of the following informalities:
Claim 2 (similarly claim 11 and 16) recites "the main image and the reference" should be "the main image and the reference image"
Claim 10 (similarly claim 15) recites "the submitting, the image mask" should be "the submitting, an image mask
Appropriate corrections are required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 3, 5-6, 10, 12, 14-15, 17, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Goncalves et al. (US 2020/0320324 A1) in view of USDA ("Comprehensive List of Reasons for Label Modifications and Returns" - 2024).
Regarding claim 15, Goncalves teaches “A system for modifying an image comprising: a processor (Goncalves paragraph [0052] "As depicted, computing hardware 200 includes one or more processors 206, a system memory 208 and a system bus 210 that couples various system components including the system memory 208 to the processor 206");
a non-transitory computer readable memory storing instructions programmed to cooperate with the processor to perform operations for modifying an image (Goncalves paragraph [0048] "the nontransitory processor- or computer-readable media may include a database or other data structure which stores one or more of: image data, model data, training data, test data, parameter data, character detection or recognition algorithms, and/or other data"), comprising:
receiving a main image (Goncalves paragraph [0036] "Image capture engine 152 produces, as its output, image 153, which contains at least a portion of target object 106, including one or more ROIs 108") and a reference image (Goncalves paragraph [0040] "one or more ROI templates 157");
determining one or more location points of content within the main image that is a candidate for a match with the reference image (Goncalves paragraph [0037] "ROI preprocessing engine 154 receives image 153 and works to detect and locate the one or more text containing ROIs 108 from the image on which OCR operations are to be performed");
submitting the main image and the determined one or more location points to an image segmentation foundation model (Goncalves paragraph [0037] "ROI preprocessing engine 154 may extract ( e.g., crop around) the ROI to produce an image containing substantially only the ROI (i.e., with any additional content outside the boundary of the ROI being immaterial to the subsequent OCR operation). In related implementations, ROI preprocessing engine 154 produces images of extracted ROIs 155 for subsequent OCR processing, which omits non-ROI text and other extraneous content");
receiving, in response to the submitting, the image mask (Goncalves paragraph [0037] "ROI preprocessing engine 154 produces images of extracted ROIs 155 for subsequent OCR processing, which omits non-ROI text and other extraneous content");
first determining a degree of similarity between the received image mask and the reference image (Goncalves paragraph [0040] "ROI processing engine 154 compares one or more ROI templates 157 to parts of the image 153 and determines a similarity score representing the likelihood of the presence of corresponding ROI 108 in image 153");
second determining, based on a result of the first determining, whether the reference image is present in the main image at the one or more location points (Goncalves paragraph [0067] "locator engine 308 applies a detection threshold to the output of comparison engine 306 that, if exceeded by a peak of the comparison result, indicates a presence of the corresponding ROI"); and
(Goncalves paragraph [0037] "ROI preprocessing engine 154 receives image 153 and works to detect and locate the one or more text containing ROIs 108 from the image on which OCR operations are to be performed") (Goncalves paragraph [0037] "ROI preprocessing engine 154 receives image 153 and works to detect and locate the one or more text containing ROIs 108 from the image on which OCR operations are to be performed").”
However, Goncalves is not relied on to teach “generating, in response to at least the second determining, a modified image from the main image, by at least removing the reference image […] when the reference image is present in the main image at an allowable location, relocating a location of the reference image in the main image when the reference image is present in the main image at an unallowable location, or adding the reference image to the main image.”
USDA teaches “generating, in response to at least the second determining, a modified image from the main image (USDA page 1 paragraph 1 "Each reason corresponds to a number that will be placed on a label or label application that is either modified or rejected" - the label application that is modified is interpretated to be equivalent to generating a modified image), by at least:
removing the reference image from the main image when the reference image is present in the main image at an allowable location (USDA page 2 Standards and Labeling Policy Book Number 1 "Delete the term "Marinated" or "Basted" from the product name. The amount of solution in products labeled as "marinated" or "basted" are limited to 10% for red meat products, 8% boneless poultry products and 3% bone-in poultry products. Refer to the entry for "marinated" in the FSIS Standards and Labeling Policy Book" - where the terms "Marinated" and "Basted" are the reference image), relocating a location of the reference image in the main image when the reference image is present in the main image at an unallowable location (USDA page 3 Standards and Labeling Policy Book Number 13 "Correct the placement of the ingredient statement on the label. Nonspecific product like "loaves" needs a descriptive name or need to be followed immediately by the ingredient statement. A product labeled "Olive Loaf" is a non-specific product. Therefore, it must be descriptively labeled or the name must be followed by the ingredient statement. See 9 CFR 317.2{e) and the entry Nonspecific Meat Food Products!!!! in the FSIS Standards and Labeling Policy Book" - where the reference image is an image with the correct placement of the ingredient statement), or adding the reference image to the main image when the reference image is absent from the main image (USDA page 4 Standards and Labeling Policy Book Number 20 "Correct the product name to add Textured Vegetable Protein (TVP) because the ratio of meat or poultry meat to TVP is less than 7:1. See the entry Textured Vegetable Protein (TVP) Products - Fresh Meat or Poultry Meat Ratios in the FSIS Food Standards and Labeling Policy Book" - where the reference image is an image containing "Textured Vegetable Protein").“
It would have been obvious to a person having ordinary skill in the art before
effective filing date of the claimed invention of the instant application to combine a system for comparing 2 images to determine similarity as taught by Goncalves to include updating the image based on guidelines as taught by USDA.
The suggestion/motivation for doing so would have been that there is a need in
the field to update images that do not adhere to rules and guidelines, for example certain labels and text can only be displayed for specific situations "The amount of solution in products labeled as "marinated" or "basted" are limited to 10% for red meat products, 8% boneless poultry products and 3% bone-in poultry products. Refer to the entry for "marinated" in the FSIS Standards and Labeling Policy Book." as noted by the USDA disclosure in page 2 paragraph 3.
Therefore, it would have been obvious to combine the disclosure of Goncalves with the USDA disclosure to obtain the invention as specified in claim 15 as there is a
reasonable expectation of success and/or because doing so merely combines prior art
elements according to known methods to yield predictable results.
Claim 1 recites a method with steps corresponding to the system with elements
recited in claim 15. Therefore, the recited steps of this claim are mapped to the
proposed combination in the same manner as the corresponding elements of system
claim 15. Additionally, the rationale and motivation to combine the Goncalves and USDA references, presented in rejection of claim 15 apply to this claim.
Claim 10 recites a computer readable medium including computer executable instructions corresponding to the elements of the system recited in claim 15. Therefore, the recited instructions of the computer readable medium of claim 10 are mapped to the proposed combination in the same manner as the corresponding elements of the apparatus claim 15. Additionally, the rationale and motivation to combine Goncalves and USDA presented in rejection of claim 15, apply to this claim.
Regarding claim 3 (similarly claim 12 and claim 17), the combination of Goncalves and USDA teaches “The method of claim 1, wherein determining the one or more location points further comprises:
first scanning the main image using an Optical Character Recognition (OCR) technique for main image text (Goncalves paragraph [0046] "OCR engine 158 receives, as its input, one or more extracted ROI images 155, and performs OCR operations to recognize the text in those ROI images 155");
second scanning the reference image using the OCR technique for reference image text (Goncalves Figure 1B and paragraph [0040] "Each ROI template 157 represents a known, or familiar, representative ROI layout. ROI template 157 is in a format containing feature descriptors of the layout of the corresponding ROI"); and
determining one or more location points for a region (Goncalves paragraph [0037] "ROI preprocessing engine 154 receives image 153 and works to detect and locate the one or more text containing ROIs 108 from the image on which OCR operations are to be performed") in the main image that encloses text within the main image text matching the reference image text (Goncalves paragraph [0040] "ROI processing engine 154 compares one or more ROI templates 157 to parts of the image 153 and determines a similarity score representing the likelihood of the presence of corresponding ROI 108 in image 153").“
Regarding claim 5 (similarly claim 14 and claim 19), the combination of Goncalves and USDA teaches “The method of claim 1, wherein first determining a degree of similarity between the received image mask and the reference image further comprises:
applying the image mask to the main image (Goncalves paragraph [0037] "ROI preprocessing engine 154 may extract ( e.g., crop around) the ROI to produce an image containing substantially only the ROI (i.e., with any additional content outside the boundary of the ROI being immaterial to the subsequent OCR operation). In related implementations, ROI preprocessing engine 154 produces images of extracted ROIs 155 for subsequent OCR processing, which omits non-ROI text and other extraneous content"); and
comparing the reference image with content of the main image exposed through the applied image mask (Goncalves paragraph [0040] "ROI processing engine 154 compares one or more ROI templates 157 to parts of the image 153 and determines a similarity score representing the likelihood of the presence of corresponding ROI 108 in image 153").“
Regarding claim 6 (similarly claim 20), the combination of Goncalves and USDA teaches “The method of claim 1, further comprising: maintaining a plurality of rules that govern allowable and/or required content in an image (USDA page 1 paragraph 1 "This document is provided to further explain reasons why a label was modified or returned after evaluation by a staff member of the Labeling and Program Delivery Staff (LPDS). Each reason corresponds to a number that will be placed on a label or label application that is either modified or rejected. While LPDS has endeavored to be comprehensive, this list is not all inclusive. There are certain reasons for modifications or rejection, reasons that are very specific and cannot be generalized. In these situations, the reason will be noted on the label application or in a return letter"), wherein:
the removing is in response to at least the reference image in the main image violating any of the plurality of rules that prohibit presence of the reference image (USDA page 2 Standards and Labeling Policy Book Number 1 "Delete the term "Marinated" or "Basted" from the product name. The amount of solution in products labeled as "marinated" or "basted" are limited to 10% for red meat products, 8% boneless poultry products and 3% bone-in poultry products. Refer to the entry for "marinated" in the FSIS Standards and Labeling Policy Book" - where the terms "Marinated" and "Basted" are the reference image);
the relocating is in response to at least the reference image being at a location in the main image in violation any of the plurality of rules that limit allowable locations of the reference image (USDA page 3 Standards and Labeling Policy Book Number 13 "Correct the placement of the ingredient statement on the label. Nonspecific product like "loaves" needs a descriptive name or need to be followed immediately by the ingredient statement. A product labeled "Olive Loaf" is a non-specific product. Therefore, it must be descriptively labeled or the name must be followed by the ingredient statement. See 9 CFR 317.2{e) and the entry Nonspecific Meat Food Products!!!! in the FSIS Standards and Labeling Policy Book" - where the reference image is an image with the correct placement of the ingredient statement); and
the adding is in response to absence of the reference image in the main image violating any of the plurality of rules that require the presence of the reference image or content similar to the reference image (USDA page 4 Standards and Labeling Policy Book Number 20 "Correct the product name to add Textured Vegetable Protein (TVP) because the ratio of meat or poultry meat to TVP is less than 7:1. See the entry Textured Vegetable Protein (TVP) Products - Fresh Meat or Poultry Meat Ratios in the FSIS Food Standards and Labeling Policy Book" - where the reference image is an image containing "Textured Vegetable Protein").“
The proposed combination as well as the motivation for combining Goncalves and USDA references presented in the rejection of claim 15, applies to claim 6. Finally the method recited in claim 6 is met by Goncalves and USDA.
Claims 2, 11, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Goncalves and USDA in view of Horng et al. (US 2012/0057763 A1).
Regarding claim 2 (similarly claim 11 and claim 16), the combination of Goncalves and USDA teaches “The method of claim 1, wherein determining the one or more location points further comprises:
receiving the one or more location points, in response to processing of the main image and the reference (Goncalves paragraph [0037] "ROI preprocessing engine 154 receives image 153 and works to detect and locate the one or more text containing ROIs 108 from the image on which OCR operations are to be performed").“
However, the combination of Goncalves and USDA is not relied on to teach “processing the main image and the reference image using a scale invariant transformation function”.
In an analogous field of endeavor, Horng teaches “processing the main image and the reference image using a scale invariant transformation function (Horng paragraph [0049] "The method applies a convolution process to the initial image, utilizes Scale Invariant Feature Transformation (SIFT) to transform the captured image to a set of feature points and calculates the similarity through the set of feature points")”.
It would have been obvious to a person having ordinary skill in the art before
effective filing date of the claimed invention of the instant application to combine a system for comparing 2 images to determine adherence to guidelines as taught by Goncalves and USDA to include using scale invariant transformation as taught by Horng.
The suggestion/motivation for doing so would have been that “More particularly, the feature points transformed by SIFT have a considerable resistance to the scale variation and rotation respectively for resisting part of the illuminance variation of image and the interference of noise. And the method of the present invention can have a considerably good recognition" as noted by the Horng disclosure in paragraph 49.
Therefore, it would have been obvious to combine the disclosure of Goncalves and USDA with the Horng disclosure to obtain the invention as specified in claim 2 as there is a reasonable expectation of success and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claims 4, 13, 18 are rejected under 35 U.S.C. 103 as being unpatentable over Goncalves, USDA, and Horng in view of Huo et al. (US 2023/0009564 A1).
Regarding claim 4 (similarly claim 13 and claim 18), the combination of Goncalves, USDA, and Horng teaches “The method of claim 1, wherein determining the one or more location points further comprises:
processing the black and white version of the main image and the reference image using a scale invariant transformation function (Horng paragraph [0049] "The method applies a convolution process to the initial image, utilizes Scale Invariant Feature Transformation (SIFT) to transform the captured image to a set of feature points and calculates the similarity through the set of feature points"); and
determining one or more location points in response to processing the black and white version (Goncalves paragraph [0037] "ROI preprocessing engine 154 receives image 153 and works to detect and locate the one or more text containing ROIs 108 from the image on which OCR operations are to be performed").“
However, the combination of Goncalves, USDA, and Horng is not relied on to teach “generating, using an edge detector program, a black and white version of the main image and the reference image”.
In an analogous field of endeavor, Huo teaches “generating, using an edge detector program, a black and white version of the main image and the reference image (Huo paragraph [0022] "acquiring a character region image and converting the character region image into a grayscale image, where the character region image includes at least one character frame; converting the grayscale image into an edge binary image by utilizing an edge detection algorithm; acquiring from the edge binary image at least one segment for the at least one character frame by utilizing a projection approach; and determining a target character region from each of the at least one segment by utilizing a contour detection algorithm, and performing character segmentation on the character region image based on the target character region”.
It would have been obvious to a person having ordinary skill in the art before
effective filing date of the claimed invention of the instant application to combine a system for comparing 2 images to determine adherence to guidelines as taught by Goncalves, USDA, and Horng to include converting to a grayscale image as taught by Huo.
The suggestion/motivation for doing so would have been that “compared to the aforesaid embodiment, Gaussian filtering and/or median filtering is performed on the grayscale image in the technical solution according to this embodiment, which can reduce the noise influence caused by stains or reflections on the meter, and thereby improve the accuracy of the subsequent edge detection" as noted by the Huo disclosure in paragraph 85.
Therefore, it would have been obvious to combine the disclosure of Goncalves, USDA, and Horng with the Hou disclosure to obtain the invention as specified in claim 4 as there is a reasonable expectation of success and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Goncalves and USDA in view of Zhang ("Using NPL-based Machine Learning to Automate Compliance and Risk Governance" - IDS, and in further view of Gurung ("Leveraging Metadata in RAG Customization - 2024).
Regarding claim 7, the combination of Goncalves and USDA teaches “The method of claim 6, wherein maintaining a plurality of rules further comprises:
storing the plurality of rules (Goncalves paragraph [0044] "RP training engine 156 applies an adaptive training algorithm that automatically determines one or more image-processing parameters to optimize discriminator performance. RP training engine 156 produces ROI templates 157 based on user-provided examples of ROI layouts" ) into a guidelines library (USDA page 1 paragraph 1 "This document is provided to further explain reasons why a label was modified or returned after evaluation by a staff member of the Labeling and Program Delivery Staff (LPDS). Each reason corresponds to a number that will be placed on a label or label application that is either modified or rejected. While LPDS has endeavored to be comprehensive, this list is not all inclusive" - where a full list of modification and returns is equivalent to a guideline library).
However, the combination of Goncalves and USDA is not relied on to teach “identifying one or more pages including guidelines within verbose documents, using a guideline filter; extracting and categorizing content from the identified one or more pages by applying topic modeling to the identified one or more pages; collecting metadata related to the extracted and categorized content; generating a prompt input using the verbose documents, the metadata, and one or more prompts selected from a prompts library, wherein the prompt input has a structured machine interpretable format; processing the prompt input using a large language model (LLM) to generate the plurality of rules”.
In an analogous field of endeavor, Zhang teaches “identifying one or more pages including guidelines within verbose documents (Zhang page 2 paragraph 5 "Collect text-heavy documents related to cybersecurity, including security findings, policy exceptions, audit issues, and incident reports"), using a guideline filter (Zhang page 2 paragraph 5 "Conduct data preprocessing, including stop-words removal, lemmatization, and stemming");
extracting and categorizing content from the identified one or more pages by applying topic modeling to the identified one or more pages (Zhang page 2 paragraph 5 "Proceed to topic modeling (LDA/LSA)")”.
It would have been obvious to a person having ordinary skill in the art before
effective filing date of the claimed invention of the instant application to combine a system for comparing 2 images to determine adherence to guidelines as taught by Goncalves and USDA to include extracting guidelines and regulations from documents as taught by Zhang.
The suggestion/motivation for doing so would have been that “Besides, Bugra Kara bey pointed out that machine learning is already ubiquitous in cybersecurity for log, telemetry, and traffic pattern analysis, anomaly detection, and behavior analytics. However, for the identification and assessment of dormant/latent risk themes, patterns, and relationships, additional machine learning tools will be useful for the following reasons: (1) The insights reside within text-heavy datasets (security policy exceptions, internal and external incidents, and Governance Risk Compliance (GRC) security findings). (2) Natural language-driven machine learning is required for risk assessment. (3) Artifacts may be at the board room discussion level, rather than technically focused. He also described how to apply machine learning to cybersecurity risk management in three scenarios" as noted by the Zhang disclosure in page 2 paragraph 3.
However, the combination of Goncalves, USDA, and Zhang is not relied on to teach “collecting metadata related to the extracted and categorized content; generating a prompt input using the verbose documents, the metadata, and one or more prompts selected from a prompts library, wherein the prompt input has a structured machine interpretable format; processing the prompt input using a large language model (LLM) to generate the plurality of rules”.
Gurung teaches “collecting metadata related to the extracted and categorized content (Gurung page 4 paragraph 1 "The retrieval step is where relevant data is pulled from the database to augment the query to the LLM. Metadata can be used in a number of ways to enhance this process. First, it can act as a filter to narrow the search results. Such a filter can be hard-coded into the system (for example, a news-oriented system could automatically exclude articles older than a certain date), or user controlled (allowing users to specify metadata-based filters or extract them from the query itself, providing more precise and relevant results)"); generating a prompt input using the verbose documents, the metadata, and one or more prompts selected from a prompts library, wherein the prompt input has a structured machine interpretable format (Gurung page 5 paragraph 4 "The prompt is the instruction that LLM receives and uses to generate output. The more information our prompt contains, the better. So naturally, we want to use metadata at this point in the RAG pipeline as well"); processing the prompt input using a large language model (LLM) to generate the plurality of rules (Gurung page 5 paragraph 4 "The prompt is the instruction that LLM receives and uses to generate output. The more information our prompt contains, the better. So naturally, we want to use metadata at this point in the RAG pipeline as well")”.
It would have been obvious to a person having ordinary skill in the art before effective filing date of the claimed invention of the instant application to combine a system for comparing 2 images to determine adherence to guidelines as taught by Goncalves, USDA, and Zhang to include using metadata and LLM prompting as taught by Gurung.
The suggestion/motivation for doing so would have been that “Retrieval augmented generation (RAG) systems are a powerful approach to extending the capabilities of large language models (LLMs). They make LLM output more reliable by providing a knowledge base of fact-checked data in response to a user query" as noted by the Gurung disclosure in page 1 paragraph 1.
Therefore, it would have been obvious to combine the disclosure of Goncalves, USDA, and Zhang with the Gurung disclosure to obtain the invention as specified in claim 7 as there is a reasonable expectation of success and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Goncalves and USDA in view of Rigotti et al. (US 2026/0148543 A1), and in further view of FDA ("A Food Labeling Guide" - 2013).
Regarding claim 8, the combination of Goncalves and USDA teaches “The method of claim 6, further comprising: extracting text from character regions within an image using the OCR, wherein the character regions are identified using a character segmentation model (Goncalves paragraph [0037] "ROI preprocessing engine 154 receives image 153 and works to detect and locate the one or more text Containing ROIs 108 from the image on which OCR operations are to be performed");
obtaining, using the vision foundation model and based on the coarse description, a granular description of the objects (Goncalves paragraph [0041] "The layout of a ROI in the present context describes the basic appearance, or arrangement, of textual or graphical features contained within the ROI. For example, a given layout may have a defined quantity of lines of text of relative length, with each line containing a specified number of characters of relative size and spaces in particular locations. The relative positioning and sizing of the lines of characters and the characters themselves, within the ROI may constitute features of the layout, as may any other symbols, graphics, lettering type or style (e.g., block vs. script lettering, lettering in Latin vs. Chinese vs. Arabic, or various typefaces), barcodes, and other types of visual indicia") by cropping the object regions (Goncalves paragraph [0043] "ROI preprocessing engine 154 crops around each detected and located ROI 108 in scaled and rotated image 153 to extract that ROI 108");
analyzing the extracted text and the granular description against the plurality of rules (USDA page 1 paragraph 1 "This document is provided to further explain reasons why a label was modified or returned after evaluation by a staff member of the Labeling and Program Delivery Staff (LPDS). Each reason corresponds to a number that will be placed on a label or label application that is either modified or rejected. While LPDS has endeavored to be comprehensive, this list is not all inclusive. There are certain reasons for modifications or rejection, reasons that are very specific and cannot be generalized. In these situations, the reason will be noted on the label application or in a return letter");
identifying at least one of one or more non-compliant character regions (USDA page 1 paragraph 1 "This document is provided to further explain reasons why a label was modified or returned after evaluation by a staff member of the Labeling and Program Delivery Staff (LPDS). Each reason corresponds to a number that will be placed on a label or label application that is either modified or rejected. While LPDS has endeavored to be comprehensive, this list is not all inclusive. There are certain reasons for modifications or rejection, reasons that are very specific and cannot be generalized. In these situations, the reason will be noted on the label application or in a return letter") and
restricting a usage of the image upon identification of at least one of the one or more noncompliant character regions (USDA page 1 paragraph 1 "This document is provided to further explain reasons why a label was modified or returned after evaluation by a staff member of the Labeling and Program Delivery Staff (LPDS). Each reason corresponds to a number that will be placed on a label or label application that is either modified or rejected. While LPDS has endeavored to be comprehensive, this list is not all inclusive. There are certain reasons for modifications or rejection, reasons that are very specific and cannot be generalized. In these situations, the reason will be noted on the label application or in a return letter") and
However, the combination of Goncalves and USDA is not relied on to teach “generating, based on the extracted text, a coarse description of objects from object regions within the image, using a vision foundation model”, “non-compliant object regions violating any of the plurality of rules”, and “restricting a usage of the image upon identification of at least one of the one or more […] non-compliant object regions”.
In an analogous field of endeavor, Rigotti teaches “generating, based on the extracted text, a coarse description of objects from object regions within the image, using a vision foundation model (Rigotti paragraph [0023] "The computing device may allow for user input to be classified into different categories, enabling creation of segmentation masks for multiple classes. A process of repeatedly adding and adjusting user inputs may provide users with a deeper understanding of how the model operates. By demonstrating the capabilities and limitations of the models, users may be taught to collaborate with the models more effectively, leading to better outcomes")”.
It would have been obvious to a person having ordinary skill in the art before
effective filing date of the claimed invention of the instant application to combine a system for comparing 2 images to determine adherence to guidelines as taught by Goncalves and USDA to include determining regions of an object as taught by Rigotti.
The suggestion/motivation for doing so would have been that “Foundation models have improved machine learning model development and application, pivoting from a paradigm centered on training use case-tailored models on task-specific data to a paradigm where single generalist models are pretrained on diverse large-scale data, then fine-tuned for a wide range of tasks. Specifically in computer vision, models such as segment anything model (SAM), contrastive language-image pre-training (CLIP), and selfsupervised backbones such as self-distillation, no labels (DINO), and DINOv2 have unlocked powerful and versatile visual functionalities like object detection, semantic segmentation and expressive embeddings that are at the core of a multitude of diverse applications" as noted by the Rigotti disclosure in paragraph 11.
However, the combination of Goncalves, USDA, and Rigotti is not relied on to teach ““non-compliant object regions violating any of the plurality of rules”, and “restricting a usage of the image upon identification of at least one of the one or more […] non-compliant object regions”.
FDA teaches “(FDA page 6 Bullet 7 "What is the prohibition against intervening material? Answer: Information that is not required by FDA is considered intervening material and is not permitted to be placed between the required labeling on the information panel (e.g., the UPC bar code is not FDA required labeling). 21 CPR 101.2(e)")”,
PNG
media_image1.png
365
396
media_image1.png
Greyscale
FDA Page 6 bottom left figure
non-compliant object regions violating any of the plurality of rules (FDA page 6 Bullet 7 "What is the prohibition against intervening material? Answer: Information that is not required by FDA is considered intervening material and is not permitted to be placed between the required labeling on the information panel (e.g., the UPC bar code is not FDA required labeling). 21 CPR 101.2(e)")”, and
“restricting a usage of the image upon identification of at least one of the one or more […] non-compliant object regions (FDA page 6 Bullet 7 "What is the prohibition against intervening material? Answer: Information that is not required by FDA is considered intervening material and is not permitted to be placed between the required labeling on the information panel (e.g., the UPC bar code is not FDA required labeling). 21 CPR 101.2(e)")”.
It would have been obvious to a person having ordinary skill in the art before effective filing date of the claimed invention of the instant application to combine a system for comparing 2 images to determine adherence to guidelines as taught by Goncalves, USDA, and Rigotti to include identifying and restricting use of objects as taught by FDA.
The suggestion/motivation for doing so would have been that “The Food and Drug Administration (FDA) is responsible for assuring that foods dols in the Unites States are safe, wholesome and properly labeled. This applies to foods produced domestically, as well as foods from foreign countries" as noted by the FDA disclosure in 4 paragraph 2.
Therefore, it would have been obvious to combine the disclosure of Goncalves, USDA, and Rigotti with the FDA disclosure to obtain the invention as specified in claim 8 as there is a reasonable expectation of success and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Goncalves, USDA, Zhang, and Gurung in view of Juban ("Automatic Topic Modeling + Metadata Extraction" - 2024).
Regarding claim 9, the combination of Goncalves, USDA, Zhang, and Gurung teaches “The method of claim 7, further comprising:
incorporating geo-location data to refine the plurality of rules selection for specific regions (Gurung page 2 paragraph 3 "Metadata is data that provides additional information about a data point itself. It is often a byproduct of the process that created the data in the first place. For example, a digital photo has a timestamp and location, while an online article has the date and time it was published, the author, and the department that published it. New metadata can also be created to enrich the dataset, for example, product reviews can be classified as negative or positive").“
However, the combination of Goncalves, USDA, Zhang, and Gurung is not relied on to teach “implementing a continuous feedback learning mechanism for topic modeling to automatically select the plurality of rules based on various factors; segregating the selected plurality of rules at a finer level across different industry verticals”.
Juban teaches “implementing a continuous feedback learning mechanism for topic modeling to automatically select the plurality of rules based on various factors;
segregating the selected plurality of rules at a finer level across different industry verticals (Juban page 2 paragraph 2 "topic labeling: Assigning labels to identified topics based on extracted key terms. The goal of this step, shown in Figure 2, is to identify the shared document-specific themes across the corpus and cluster them together to define Categorized, themes of important metadata that will improve corpus understanding")”.
It would have been obvious to a person having ordinary skill in the art before
effective filing date of the claimed invention of the instant application to combine a system for comparing two images to determine adherence to guidelines as taught by Goncalves, USDA, Zhang, and Gurung to include topic modeling that further clusters based on various factors and themes as taught by Juban.
The suggestion/motivation for doing so would have been that “By employing NLP & clustering techniques including names entity recognition and topic modeling, ATMME can curate hierarchical keyword representation of large corpuses of documents, demonstrated in Figure 1. This enables explicit filtering for specific content and provides for stronger retrieval performance and higher end-to-end quality" as noted by the Juban disclosure on page 1 paragraph 2.
Therefore, it would have been obvious to combine the disclosure of Goncalves, USDA, Zhang, and Gurung with the Juban disclosure to obtain the invention as specified in claim 9 as there is a reasonable expectation of success and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Reference Cited
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure.
“Launch Faster with AI-Powered Artwork Management” by Esko discloses automatic proofing of artwork including labels that use data from various departments, companies and regions to identify issues with the visual content.
“Packaging Artwork Management Simplified” by Manage Artworks discloses an automatic compliance detection and error checking to ensure packaging visual content adheres to guidelines.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JASPREET KAUR whose telephone number is (571)272-5534. The examiner can normally be reached Monday - Friday 7:30 am - 4:00 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached at (571)272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JASPREET KAUR/Examiner, Art Unit 2662
/AMANDEEP SAINI/Supervisory Patent Examiner, Art Unit 2662
Prosecution Insights