DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Claims 1, 9 and 16 have been amended changing the scope and contents of the claim.
Claim 15 has been amended for clarification and the amendment does not change the scope or contents of the claim.
Claims 5-6, 13-14 and 19 have been cancelled.
Applicant’s amendment filed October 1, 2026 overcomes the following objection/rejection(s) from the last Office Action of July 2, 2026:
Rejection of the claims under 35 USC § 102
Response to Arguments
Applicant’s arguments with respect to claim(s) 1, 9 and 16 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4, 9-12 and 16-18 are rejected under 35 U.S.C. 103 as being unpatentable over P. -D. Tudosiu et al., "MULAN: A Multi Layer Annotated Dataset for Controllable Text-to-Image Generation," 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2024, pp. 22413-22422, doi: 10.1109/CVPR52733.2024.02115. (hereinafter Tudosiu), and further in view of H. Rasheed et al., "GLaMM: Pixel Grounding Large Multimodal Model," 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2024, pp. 13009-13018, doi: 10.1109/CVPR52733.2024.01236. (hereinafter Rasheed) and Cai, Chen, et al. "Top-down framework for weakly-supervised grounded image captioning." Knowledge-Based Systems 287 (2024): 111433. (hereinafter Cai).
Regarding independent claim 1, Tudosiu discloses A method of implementing automatic layer decomposition (page 22415, right column, “We build a pipeline that comprises a sequence of three main modules. First, the Decomposition module encompasses instance discovery and extraction. It focuses on scene understanding, comprising a sequence of object detection, segmentation and depth estimation models.”), comprising:
receiving an image comprising a plurality of objects (page 22415, right column, “This module decomposes I into a background image and a series of extracted instances with associated segmentation masks;” I is read as the received image);
generating object detection results based on detecting the plurality of objects in the image in response to receiving the image, the object detection results comprising an object detection result corresponding to each of the plurality of objects (page 22416, left column, “Accurately detecting all relevant in stances in an image is the first step of our pipeline. In order to achieve good quality decomposition, it is essential that we are able to detect and separate all instances present in the scene”… “This step of the pipeline outputs a series of bounding boxes with corresponding category names.”), the object detection results including object bounding boxes (page 22416, “This step of the pipeline outputs a series of bounding boxes with corresponding category names.”);
generating textual descriptions of the image, the textual descriptions comprising a textual description corresponding to each of the plurality of objects (page 22417, right column, “For instance inpainting, we leverage automatically generated instance captions”);
associating the object detection result with the textual description corresponding to each of the plurality of objects in response to generating the textual descriptions (page 22416, left column, “Accurately detecting all relevant in stances in an image is the first step of our pipeline. In order to achieve good quality decomposition, it is essential that we are able to detect and separate all instances present in the scene”… “This step of the pipeline outputs a series of bounding boxes with corresponding category names.” page 22417, right column, “For instance inpainting, we leverage automatically generated instance captions”);
generating depth estimation results by predicting a depth map of the image, the depth estimation results comprising a depth estimation result corresponding to each of the plurality of objects (page 22416, right column, “Understanding the relative position ing of instances in an image is crucial towards achieving our RGBA decomposition goal. Depth estimation provides essential information, indicating distance to the camera at capture time. We use the MiDaS model [29], chosen for its robustness: it was trained on 12 different datasets allowing it to be reliable across different type of scenes and image qualities. Once computed, we quantise the depth map into multiple bins of width 250 relative depth units to facilitate cross instance comparisons.” Page 22416, right column, “We further compute depth and occlusion instance graphs for a more comprehensive scene understanding. Specifically we are using the InstaDepthNeto,d model [17], which is capable of jointly predicting occlusion and depth relationships between instances.”); and
Tudosiu fails to explicitly disclose as further recited. However, Rasheed discloses generating textual descriptions of the image, the textual descriptions comprising a textual description corresponding to each of the plurality of objects by inputting the image and a prompt (abstract, " In this work, we present Grounding LMM (GLaMM), the first model that can generate natural language responses seamlessly in-tertwined with corresponding object segmentation masks. GLaMM not only grounds objects appearing in the con-versations but is flexible enough to accept both textual and optional visual prompts (region of interest) as input. " Figure 1, " Our multimodal conversational model can provide natural language responses grounded at the pixel-level in the input image." the prompt is within the image reading ""describe the image in detail. Please output interleaved segmentation mask") into a multi-modal large language model (abstract, " In this work, we present Grounding LMM (GLaMM)…GLaMM not only grounds objects appearing in the con-versations but is flexible enough to accept both textual and optional visual prompts (region of interest) as input. " being that the LLM can receive text and visual prompts it is read as an LLM ) in response to generating the object detection results, the textual descriptions comprising a textual description corresponding to each of the plurality of objects (page 13011, right column, " The LLM then integrates both the projected image features and the text instruction to generate output yt" page 13012, right column, "A sample prompt for querying the model in this task is: “Could you please give me a detailed description of the image? Please respond with interleaved segmentation masks for the corresponding parts of the answer.” The model generates a detailed caption along with interleaved segmentation masks");
merging the plurality of objects into layers based on the grounded caption and the depth estimation result corresponding to each of the plurality of objects in response to forming the grounded caption and generating the depth estimation results (page 13014, left column, "To provide additional context, depth values and bounding box coordinates are used to assign each object to specific spatial layers within the scene, such as immediate foreground, foreground, midground, or background. ").
Tudosiu is directed toward, “we introduce MuLAn: a novel dataset comprising over 44K MUlti-Layer ANnotations of RGB images as multi layer, instance-wise RGBA decompositions, and over 100K instance images. To build MuLAn, we developed a training free pipeline which decomposes a monocular RGB image into a stack of RGBA layers comprising of background and isolated instances. We achieve this through the use of pre trained general-purpose models, and by developing three modules: image decomposition for instance discovery and extraction, instance completion to reconstruct occluded areas, and image re-assembly (abstract).” Rasheed is directed toward, “Grounding LMM (GLaMM), the first model that can generate natural language responses seamlessly intertwined with corresponding object segmentation masks. (abstract).” As can be easily seen by one of ordinary skill in the art before the effective filing date of the claimed invention, Tudosiu and Rasheed are directed toward similar methods of endeavor of image analysis and annotation. Further, Rasheed allows for user interaction with the system (abstract, “GLaMM not only grounds objects appearing in the con versations but is flexible enough to accept both textual and optional visual prompts (region of interest) as input. This empowers users to interact with the model at various lev els of granularity, both in textual and visual domains.”). One of ordinary skill in the art would easily understand a user being able to interact with the system allows for customization of a task, so that the output is closest to what a user is interested in. Thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Rasheed in order to ensure the output is as aligned with the user input (page 13010, left column, “GLaMM can work with both textual and visual prompts and can generate visually grounded out puts, thus offering a versatile user experience.”).
Tudosiu and Rasheed fail to explicitly disclose as further recited. However, Cai discloses forming a grounded caption in which each textual description is associated with a corresponding object bounding box in response to associating the object detection result with the textual description corresponding to each of the plurality of objects (see Figure 1, "ours" the output associated with the one-stage pipeline is the bounding box associated with the caption (i.e. girl, shirt, telescope); see also Figure 2);
As noted above, Tudosiu and Rasheed are directed toward similar methods of endeavor of image analysis and annotation. Further, Cai is directed toward, “a one-stage weakly supervised grounded captioner that directly takes the RGB image as input to perform captioning and grounding at the top-down image level (abstract).” As can be easily seen by one of ordinary skill in the art before the effective filing date of the claimed invention, Tudosiu, Rasheed and Cai are directed toward similar methods of endeavor of image analysis and annotation. Further, as noted in Cai, the method provided aids in correcting for overfitting of the captioner and not determining a relationship between objects (abstract). One of ordinary skill in the art before the effective filing date of the claimed invention would easily understand knowing relitonships between objects can provide additional context to a user; for example, knowing an image contains a person and a telescope is not as useful as knowing how that person is interacting with the telescope. Thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Cai in order to ensure the outputs contain important relational detail as related to the objects in an image.
Regarding dependent claim 2, the rejection of claim 1 is incorporated herein. Additionally, Tudosiu further discloses wherein at least one of the plurality of objects in the image is occluded by one or more other objects among the plurality of objects (page 22416, right column, “Image instances are represented by graph nodes, and a directed edge from instance A to in stance B (A → B) indicates that A occludes B.”), and wherein the method further comprises:
generating occlusion relation graphs associated with the image (page 22416, right column, “We further compute depth and occlusion instance graphs for a more comprehensive scene understanding. Specifically we are using the InstaDepthNetod model [17], which is capable of jointly predicting occlusion and depth relationships between instances.”); and
sorting the plurality of objects based on the depth estimation results and the occlusion relation graphs in a foreground-to-background order (page 22417, left column, “First, instances are ordered based on their depth information, from furthest away to closest (according to instance mean depth value).”).
Regarding dependent claim 3, the rejection of claim 2 is incorporated herein. Additionally, Tudosiu further discloses further comprising:
generating inpainting masks corresponding to the layers (page 22417, right column, “we first estimate an inpainting mask using occlusion ordering information and segmentation masks from occluding instances.”); and
generating completed layer images comprising a completed layer image corresponding to each of the layers and a completed background layer image by utilizing an image inpainting model, wherein the completed layer images depict an entirety of the at least one of the plurality of objects as if it was not occluded by the one or more other objects (page 22415, right column, “We build a pipeline that comprises a sequence of three main modules. First, the Decomposition module encompasses instance discovery and extraction. It focuses on scene understanding, comprising a sequence of object detection, segmentation and depth estimation models.;” page 22417, right column, “Third, the instance is inpainted using a state-of-the-art inpainting generative model and automatic captions as prompts. Fourth, we re-extract the completed instance using our segmentation model and the occluded segmentation mask as guidance, effectively obtaining the complete instance image which will be part of our multi-layer representation.” The inpainting is read as being performed on each instance (including the background) using a generative model, to depict the entire object without an occlusion).
Regarding dependent claim 4, the rejection of claim 3 is incorporated herein. Additionally, Tudosiu further discloses further comprising:
generating refined layer masks based on the completed layer images (page 22417, right column, “Third, the instance is inpainted using a state-of-the-art inpainting generative model and automatic captions as prompts. Fourth, we re-extract the completed instance using our segmentation model and the occluded segmentation mask as guidance, effectively obtaining the complete instance image which will be part of our multi-layer representation;” the re-extraction is read as performing refined masks); and
generating red, green, blue, and alpha (RGBA) layer images based on the completed layer images and the refined layer masks (page 22416, left column, “Finally, the image Reassembly module generates occlusion aware Alpha layers, and builds our RGBA stack such that flattening the stack effectively reconstructs I.”).
Regarding independent claim 9, the rejection of claim 1 applies directly. Additionally, Tudosiu further discloses A system of implementing automatic layer decomposition (page 22415, right column, “We build a pipeline that comprises a sequence of three main modules. First, the Decomposition module encompasses instance discovery and extraction. It focuses on scene understanding, comprising a sequence of object detection, segmentation and depth estimation models.”), comprising:
at least one processor (the pipeline built above is not capable of being ran on pencil and paper, thus requires some form of computer though not explicitly stated); and
at least one memory communicatively coupled to the at least one processor and comprising computer-readable instructions that upon execution by the at least one processor cause the at least one processor to perform operations (the pipeline built above is not capable of being ran on pencil and paper, thus requires some form of computer programmed to perform the desired actions though not explicitly stated) comprising:
receiving an image comprising a plurality of objects (see claim 1 analysis);
generating object detection results based on detecting the plurality of objects in the image in response to receiving the image, the object detection results comprising an object detection result corresponding to each of the plurality of objects, the object detection results including object bounding boxes (see claim 1 analysis);
generating textual descriptions of the image by inputting the image and a prompt into a multi-modal large language model, the textual descriptions comprising a textual description corresponding to each of the plurality of objects (see claim 1 analysis);
associating the object detection result with the textual description corresponding to each of the plurality of objects in response to generating the textual descriptions (see claim 1 analysis);
forming a grounded caption in which each textual description is associated with a corresponding object bounding box, in response to associating the object detection result with the textual description corresponding to each of the plurality of objects (see claim 1 analysis);
generating depth estimation results by predicting a depth map of the image, the depth estimation results comprising a depth estimation result corresponding to each of the plurality of objects (see claim 1 analysis); and
merging the plurality of objects into layers based the grounded caption and the depth estimation result corresponding to each of the plurality of objects in response to forming the grounded caption and generating the depth estimation results (see claim 1 analysis).
Regarding dependent claim 10, the rejection of claim 9 is incorporated herein. Additionally, Tudosiu further discloses wherein at least one of the plurality of objects in the image is occluded by one or more other objects among the plurality of objects (page 22416, right column, “Image instances are represented by graph nodes, and a directed edge from instance A to in stance B (A → B) indicates that A occludes B.”), and wherein the method further comprises:
generating occlusion relation graphs associated with the image ([age 22416, right column, “We further compute depth and occlusion instance graphs for a more comprehensive scene understanding. Specifically we are using the InstaDepthNetod model [17], which is capable of jointly predicting occlusion and depth relationships between instances.”); and
sorting the plurality of objects based on the depth estimation results and the occlusion relation graphs in a foreground-to-background order (page 22417, left column, “First, instances are ordered based on their depth information, from furthest away to closest (according to instance mean depth value).”).
Regarding dependent claim 11, the rejection of claim 10 is incorporated herein. Additionally, Tudosiu further discloses further comprising:
generating inpainting masks corresponding to the layers (page 22417, right column, “we first estimate an inpainting mask using occlusion ordering information and segmentation masks from occluding instances.”); and
generating completed layer images comprising a completed layer image corresponding to each of the layers and a completed background layer image by utilizing an image inpainting model, wherein the completed layer images depict an entirety of the at least one of the plurality of objects as if it was not occluded by the one or more other objects (page 22415, right column, “We build a pipeline that comprises a sequence of three main modules. First, the Decomposition module encompasses instance discovery and extraction. It focuses on scene understanding, comprising a sequence of object detection, segmentation and depth estimation models.;” page 22417, right column, “Third, the instance is inpainted using a state-of-the-art inpainting generative model and automatic captions as prompts. Fourth, we re-extract the completed instance using our segmentation model and the occluded segmentation mask as guidance, effectively obtaining the complete instance image which will be part of our multi-layer representation.” The inpainting is read as being performed on each instance (including the background) using a generative model, to depict the entire object without an occlusion).
Regarding dependent claim 12, the rejection of claim 11 is incorporated herein. Additionally, Tudosiu further discloses further comprising:
generating refined layer masks based on the completed layer images (page 22417, right column, “Third, the instance is inpainted using a state-of-the-art inpainting generative model and automatic captions as prompts. Fourth, we re-extract the completed instance using our segmentation model and the occluded segmentation mask as guidance, effectively obtaining the complete instance image which will be part of our multi-layer representation;” the re-extraction is read as performing refined masks); and
generating red, green, blue, and alpha (RGBA) layer images based on the completed layer images and the refined layer masks (page 22416, left column, “Finally, the image Reassembly module generates occlusion aware Alpha layers, and builds our RGBA stack such that flattening the stack effectively reconstructs I.”).
Regarding independent claim 16, the rejection of claim 1 applies directly. Additionally, Tudosiu further discloses A non-transitory computer-readable storage medium, storing computer-readable instructions that upon execution by a processor cause the processor to implement operations (page 22415, right column, “We build a pipeline that comprises a sequence of three main modules. First, the Decomposition module encompasses instance discovery and extraction. It focuses on scene understanding, comprising a sequence of object detection, segmentation and depth estimation models.” the pipeline built is not capable of being ran on pencil and paper, thus requires some form of computer programmed to perform the desired aactions though not explicitly stated) comprising:
receiving an image comprising a plurality of objects (see claim 1 analysis);
generating object detection results based on detecting the plurality of objects in the image in response to receiving the image, the object detection results comprising an object detection result corresponding to each of the plurality of objects, the object detection results including object bounding boxes (see claim 1 analysis);
generating textual descriptions of the image by inputting the image and a prompt into a multi-modal large language model, the textual descriptions comprising a textual description corresponding to each of the plurality of objects (see claim 1 analysis);
associating the object detection result with the textual description corresponding to each of the plurality of objects in response to generating the textual descriptions (see claim 1 analysis);
forming a grounded caption in which each textual description is associated with a corresponding object bounding box, in response to associating the object detection result with the textual description corresponding to each of the plurality of objects (see claim 1 analysis);
generating depth estimation results by predicting a depth map of the image, the depth estimation results comprising a depth estimation result corresponding to each of the plurality of objects (see claim 1 analysis); and
merging the plurality of objects into layers based the grounded caption and the depth estimation result corresponding to each of the plurality of objects in response to forming the grounded caption and generating the depth estimation results (see claim 1 analysis).
Regarding dependent claim 17, the rejection of claim 16 is incorporated herein. Additionally, Tudosiu further discloses wherein at least one of the plurality of objects in the image is occluded by one or more other objects among the plurality of objects (page 22416, right column, “Image instances are represented by graph nodes, and a directed edge from instance A to in stance B (A → B) indicates that A occludes B.”), and wherein the method further comprises:
generating occlusion relation graphs associated with the image ([age 22416, right column, “We further compute depth and occlusion instance graphs for a more comprehensive scene understanding. Specifically we are using the InstaDepthNetod model [17], which is capable of jointly predicting occlusion and depth relationships between instances.”); and
sorting the plurality of objects based on the depth estimation results and the occlusion relation graphs in a foreground-to-background order (page 22417, left column, “First, instances are ordered based on their depth information, from furthest away to closest (according to instance mean depth value).”).
Regarding dependent claim 18, the rejection of claim 17 is incorporated herein. Additionally, Tudosiu further discloses the operations further comprising:
generating inpainting masks corresponding to the layers (page 22417, right column, “we first estimate an inpainting mask using occlusion ordering information and segmentation masks from occluding instances.”); and
generating completed layer images comprising a completed layer image corresponding to each of the layers and a completed background layer image by utilizing an image inpainting model, wherein the completed layer images depict an entirety of the at least one of the plurality of objects as if it was not occluded by the one or more other objects (page 22415, right column, “We build a pipeline that comprises a sequence of three main modules. First, the Decomposition module encompasses instance discovery and extraction. It focuses on scene understanding, comprising a sequence of object detection, segmentation and depth estimation models.;” page 22417, right column, “Third, the instance is inpainted using a state-of-the-art inpainting generative model and automatic captions as prompts. Fourth, we re-extract the completed instance using our segmentation model and the occluded segmentation mask as guidance, effectively obtaining the complete instance image which will be part of our multi-layer representation.” The inpainting is read as being performed on each instance (including the background) using a generative model, to depict the entire object without an occlusion);
generating refined layer masks based on the completed layer images (page 22417, right column, “Third, the instance is inpainted using a state-of-the-art inpainting generative model and automatic captions as prompts. Fourth, we re-extract the completed instance using our segmentation model and the occluded segmentation mask as guidance, effectively obtaining the complete instance image which will be part of our multi-layer representation;” the re-extraction is read as performing refined masks); and
generating red, green, blue, and alpha (RGBA) layer images based on the completed layer images and the refined layer masks (page 22416, left column, “Finally, the image Reassembly module generates occlusion aware Alpha layers, and builds our RGBA stack such that flattening the stack effectively reconstructs I.”).
Claim(s) 7-8, 15 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Tudosiu, Rasheed, and Cai as applied to claims 1, 9 and 16 respectively above, and further in view of U.S. Publication No. 2020/0380229 to Peruch et al. (hereinafter Peruch).
Regarding dependent claim 7, the rejection of claim 1 is incorporated herein. Additionally, Tudosiu further discloses further comprising:
generating segmentation masks for the plurality of objects based on the object detection results (page 22416, right column, “Our next step is to precisely segment detected instances. In order to handle a large number of categories, domains and image qualities we seek to leverage a robust general purpose segmentation model.”… “The ability to use bounding boxes as grounding for segmentation predictions makes this family of models an excellent candidate to be combined with our detCLIPv2 detector.”); and
generating object instance-level annotations based on(page 22416, left column, “Accurately detecting all relevant in stances in an image is the first step of our pipeline.” page 22416, right column, “Our next step is to precisely segment detected instances. In order to handle a large number of categories, domains and image qualities we seek to leverage a robust general purpose segmentation model.”… “The ability to use bounding boxes as grounding for segmentation predictions makes this family of models an excellent candidate to be combined with our detCLIPv2 detector.”).
Tudosiu, Rasheed and Cai fails to explicitly disclose as further recited. However, Peruch discloses extracting texts from the image by performing Optical Character Recognition (OCR) on the image to generate an OCR result (paragraph 0045, “The resulting rectified image depicting the label can then be supplied to an appropriate text recognition or OCR algorithm or appropriate barcode recognition algorithm to extract the information encoded in the label.”);
generating object instance-level annotations based on the OCR result (paragraph 0045, “The resulting rectified image depicting the label can then be supplied to an appropriate text recognition or OCR algorithm or appropriate barcode recognition algorithm to extract the information encoded in the label;” paragraph 0114, “As discussed above, existing optical character recognition (OCR) technologies for detecting and recognizing text content of images are robust and well-developed if the images of the text are captured in a fronto-parallel view. As such, supplying the candidate rectified images to an OCR algorithm will result in a plurality of potential text recognition outputs, some of which may have higher accuracy or higher confidence than others (e.g., in accordance with whether the supplied candidate rectified image included text having a reading axis aligned with the horizontal axis of the candidate rectified image).”)
As noted above, Tudosiu, Rasheed and Cai are directed toward similar methods of endeavor of image analysis and annotation. Further, Tudosiu is directed toward, “we introduce MuLAn: a novel dataset comprising over 44K MUlti-Layer ANnotations of RGB images as multi layer, instance-wise RGBA decompositions, and over 100K instance images. To build MuLAn, we developed a training free pipeline which decomposes a monocular RGB image into a stack of RGBA layers comprising of background and isolated instances. We achieve this through the use of pre trained general-purpose models, and by developing three modules: image decomposition for instance discovery and extraction, instance completion to reconstruct occluded areas, and image re-assembly (abstract).” Peruch is directed toward, “A method for automatically recognizing content of labels on objects (abstract).” As can be easily seen by one of ordinary skill in the art before the effective filing date of the claimed invention, Tudosiu, Rasheed, Cai and Peruch are directed toward similar methods of endeavor of image analysis and understanding. Further, Peruch allows for performing OCR on textual data within the image. One of ordinary skill in the art would easily understand outputting OCR data can provide a more clear indication of image content. For example, OCR’ing a label may clarify that a box in an image is a specific medication “Tylenol”, as opposed to just “medicine” or that a vehicle is a “Subaru” as opposed to just “vehicle.” Thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Peruch in order to ensure all possible contextual information is extracted from images so that a user can best understand a scene.
Regarding dependent claim 8, the rejection of claim 7 is incorporated herein. Additionally, Tudosiu iin the combination further discloses wherein the merging the plurality of objects into layers comprises:
merging the plurality of objects into the layers based on the object instance-level annotations and the depth estimation results (page 22416, left column, “Finally, the image Reassembly module generates occlusion aware Alpha layers, and builds our RGBA stack such that flattening the stack effectively reconstructs I.” page 22416, right column, “Understanding the relative position ing of instances in an image is crucial towards achieving our RGBA decomposition goal. Depth estimation provides essential information, indicating distance to the camera at capture time. We use the MiDaS model [29], chosen for its robustness: it was trained on 12 different datasets allowing it to be reliable across different type of scenes and image qualities. Once computed, we quantise the depth map into multiple bins of width 250 relative depth units to facilitate cross instance comparisons.”).
Regarding dependent claim 15, the rejection of claim 9 is incorporated herein. Additionally, Tudosiu further discloses the operations further comprising:
generating segmentation masks for the plurality of objects based on the object detection results (page 22416, right column, “Our next step is to precisely segment detected instances. In order to handle a large number of categories, domains and image qualities we seek to leverage a robust general purpose segmentation model.”… “The ability to use bounding boxes as grounding for segmentation predictions makes this family of models an excellent candidate to be combined with our detCLIPv2 detector.”); and
generating object instance-level annotations based on(page 22416, left column, “Accurately detecting all relevant in stances in an image is the first step of our pipeline.” page 22416, right column, “Our next step is to precisely segment detected instances. In order to handle a large number of categories, domains and image qualities we seek to leverage a robust general purpose segmentation model.”… “The ability to use bounding boxes as grounding for segmentation predictions makes this family of models an excellent candidate to be combined with our detCLIPv2 detector.”).
Tudosiu, Rasheed and Cai fails to explicitly disclose as further recited. However, Peruch discloses extracting texts from the image by performing Optical Character Recognition (OCR) on the image to generate an OCR result (paragraph 0045, “The resulting rectified image depicting the label can then be supplied to an appropriate text recognition or OCR algorithm or appropriate barcode recognition algorithm to extract the information encoded in the label.”);
generating object instance-level annotations based on the OCR result (paragraph 0045, “The resulting rectified image depicting the label can then be supplied to an appropriate text recognition or OCR algorithm or appropriate barcode recognition algorithm to extract the information encoded in the label;” paragraph 0114, “As discussed above, existing optical character recognition (OCR) technologies for detecting and recognizing text content of images are robust and well-developed if the images of the text are captured in a fronto-parallel view. As such, supplying the candidate rectified images to an OCR algorithm will result in a plurality of potential text recognition outputs, some of which may have higher accuracy or higher confidence than others (e.g., in accordance with whether the supplied candidate rectified image included text having a reading axis aligned with the horizontal axis of the candidate rectified image).”)
As noted above, Tudosiu, Rasheed and Cai are directed toward similar methods of endeavor of image analysis and annotation. Further, Tudosiu is directed toward, “we introduce MuLAn: a novel dataset comprising over 44K MUlti-Layer ANnotations of RGB images as multi layer, instance-wise RGBA decompositions, and over 100K instance images. To build MuLAn, we developed a training free pipeline which decomposes a monocular RGB image into a stack of RGBA layers comprising of background and isolated instances. We achieve this through the use of pre trained general-purpose models, and by developing three modules: image decomposition for instance discovery and extraction, instance completion to reconstruct occluded areas, and image re-assembly (abstract).” Peruch is directed toward, “A method for automatically recognizing content of labels on objects (abstract).” As can be easily seen by one of ordinary skill in the art before the effective filing date of the claimed invention, Tudosiu, Rasheed, Cai and Peruch are directed toward similar methods of endeavor of image analysis and understanding. Further, Peruch allows for performing OCR on textual data within the image. One of ordinary skill in the art would easily understand outputting OCR data can provide a more clear indication of image content. For example, OCR’ing a label may clarify that a box in an image is a specific medication “Tylenol”, as opposed to just “medicine” or that a vehicle is a “Subaru” as opposed to just “vehicle.” Thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Peruch in order to ensure all possible contextual information is extracted from images so that a user can best understand a scene.
Regarding dependent claim 20, the rejection of claim 16 is incorporated herein. Additionally, Tudosiu further discloses the operations further comprising:
generating segmentation masks for the plurality of objects based on the object detection results (page 22416, right column, “Our next step is to precisely segment detected instances. In order to handle a large number of categories, domains and image qualities we seek to leverage a robust general purpose segmentation model.”… “The ability to use bounding boxes as grounding for segmentation predictions makes this family of models an excellent candidate to be combined with our detCLIPv2 detector.”); and
generating object instance-level annotations based on(page 22416, left column, “Accurately detecting all relevant in stances in an image is the first step of our pipeline.” page 22416, right column, “Our next step is to precisely segment detected instances. In order to handle a large number of categories, domains and image qualities we seek to leverage a robust general purpose segmentation model.”… “The ability to use bounding boxes as grounding for segmentation predictions makes this family of models an excellent candidate to be combined with our detCLIPv2 detector.”).
Tudosiu, Rasheed and Cai fails to explicitly disclose as further recited. However, Peruch discloses extracting texts from the image by performing Optical Character Recognition (OCR) on the image to generate an OCR result (paragraph 0045, “The resulting rectified image depicting the label can then be supplied to an appropriate text recognition or OCR algorithm or appropriate barcode recognition algorithm to extract the information encoded in the label.”);
generating object instance-level annotations based on the OCR result (paragraph 0045, “The resulting rectified image depicting the label can then be supplied to an appropriate text recognition or OCR algorithm or appropriate barcode recognition algorithm to extract the information encoded in the label;” paragraph 0114, “As discussed above, existing optical character recognition (OCR) technologies for detecting and recognizing text content of images are robust and well-developed if the images of the text are captured in a fronto-parallel view. As such, supplying the candidate rectified images to an OCR algorithm will result in a plurality of potential text recognition outputs, some of which may have higher accuracy or higher confidence than others (e.g., in accordance with whether the supplied candidate rectified image included text having a reading axis aligned with the horizontal axis of the candidate rectified image).”)
As noted above, Tudosiu, Rasheed and Cai are directed toward similar methods of endeavor of image analysis and annotation. Further, Tudosiu is directed toward, “we introduce MuLAn: a novel dataset comprising over 44K MUlti-Layer ANnotations of RGB images as multi layer, instance-wise RGBA decompositions, and over 100K instance images. To build MuLAn, we developed a training free pipeline which decomposes a monocular RGB image into a stack of RGBA layers comprising of background and isolated instances. We achieve this through the use of pre trained general-purpose models, and by developing three modules: image decomposition for instance discovery and extraction, instance completion to reconstruct occluded areas, and image re-assembly (abstract).” Peruch is directed toward, “A method for automatically recognizing content of labels on objects (abstract).” As can be easily seen by one of ordinary skill in the art before the effective filing date of the claimed invention, Tudosiu, Rasheed, and Cai and Peruch are directed toward similar methods of endeavor of image analysis and understanding. Further, Peruch allows for performing OCR on textual data within the image. One of ordinary skill in the art would easily understand outputting OCR data can provide a more clear indication of image content. For example, OCR’ing a label may clarify that a box in an image is a specific medication “Tylenol”, as opposed to just “medicine” or that a vehicle is a “Subaru” as opposed to just “vehicle.” Thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Peruch in order to ensure all possible contextual information is extracted from images so that a user can best understand a scene.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Contact
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Courtney J. Windsor whose telephone number is (571)272-3956. The examiner can normally be reached Monday - Friday 8:00 - 4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John Villecco can be reached at 571-272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/COURTNEY JOAN NELSON/Primary Examiner, Art Unit 2661
Prosecution Insights