Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Use of indicates a limitation is not explicitly disclosed by the reference alone.
Claim(s) 1, 5, 16, 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Larlus (US 2014/0037198) in view of Tailang (US 2023/0154075)
Claim 1
Larlus discloses a computer-implemented method comprising:
generating, utilizing a semantic object segmentation model, a set of masks corresponding to objects depicted within a raster image (abstract: “An image segmentation method includes generating a hierarchy of regions by unsupervised segmentation of an input image. Each region is described with a respective region feature vector representative of the region. Hierarchical structures are identified, each including a parent region and its respective child regions in the hierarchy. Each hierarchical structure is described with a respective hierarchical feature vector that is based on the region feature vectors of the respective parent and child regions.”)
determining an intersection between a first mask and a second mask from among the set of masks (e.g. mutual exclusion; Larlus, ¶ 25: “The hierarchy generator 32 arranges all of the identified regions into a hierarchy of regions 34 in which each parent region in the hierarchy is the sum (union) of its children. By this, it means that the parent region includes all of the pixels in all of its children.”);
generating a hierarchical semantic structure comprising a set of nodes corresponding to the set of masks by generating a first node for the first mask and a second node for the second mask arranged according to the intersection (Larlus, ¶ 25: “The instructions 22 include a region detector 30 and a hierarchy generator 32 (which may be separate or combined) that together serve to generate a description of the image in the form of a hierarchy of regions 34,”; abstract: “An image segmentation method includes generating a hierarchy of regions by unsupervised segmentation of an input image. Each region is described with a respective region feature vector representative of the region. Hierarchical structures are identified, each including a parent region and its respective child regions in the hierarchy. Each hierarchical structure is described with a respective hierarchical feature vector that is based on the region feature vectors of the respective parent and child regions.”); and
Larlus does not disclose, but Tailang discloses generating a vector image from the raster image according to the hierarchical semantic structure (Tailang, ¶ 66 “The segmentation map 1108 generated from the semantic classification 208 is leveraged to produce a semantic output 1110 including vector objects that resemble the semantic objects of the raster object 114 as described herein. By leveraging the semantic classification of the raster object 114, the semantic vectorization system 110 generates more semantically relevant and more accurate vector objects 116 as compared to conventional techniques. The sematic output including vector objects reduces user interaction, and thus, computational resources that implement the semantic vectorization techniques are used efficiently. Accordingly, the semantic vectorization system as described herein is an improvement over the conventional techniques”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to generate a vector object.
One of ordinary skill in the art would have motivation because semantic segmentation can be used to improve vector object generation. One of ordinary skill in the art would have had a reasonable expectation of success because both references consider a hierarchy to segment a raster image.
Claim 5
Larlus discloses further comprising: generating a partial order representation for the set of masks; and generating the hierarchical semantic structure based on the partial order representation for the set of masks (Larlus, ¶ 48: “For example, an unsupervised segmentation algorithm 30, 32 is applied to each training image that builds a hierarchy 34 of regions for each training image, for example by transforming (if needed) an output hierarchy into a binary tree. In the exemplary embodiment, the Berkeley segmentation algorithm is used which automatically generates the hierarchy. In the case of another super-pixel method like Mean-Shift or N-cut the identified regions (S104) are transformed into a hierarchy of regions (S106) by agglomerative clustering using a similarity function between regions”).
Claim 16
Examiner’s Interpretation:
Machine readable media can encompass forms of signal transmission media that falls outside of the four statutory categories of invention. MPEP 2106; citing In re Nuijten, 500 F.3d 1346, 84 USPQ2d 1495 (Fed. Cir. 2007). A claim whose BRI covers both statutory and non-statutory embodiments embraces subject matter that is not eligible for patent protection and therefore is directed to non-statutory subject matter. MPEP 2106.
Applicant’s specification defines computer readable media at paragraph 102 as:
Computer-readable media can be any available media that can be accessed by a general purpose or special purpose computer system. Computer-readable media that store computer-executable instructions are non-transitory computer-readable storage media (devices). Computer-readable media that carry computer-executable instructions are transmission media. Thus, by way of example, and not limitation, embodiments of the disclosure can comprise at least two distinctly different kinds of computer-readable media: non-transitory computer-readable storage media (devices) and transmission media.
Claims 16-20 as drafted recite a non-transitory computer readable medium…
Because non-transitory as defined explicitly excludes transmission media, the broadest reasonable interpretation of the claimed medium in view of Applicant’s specification covers only eligible subject matter.
Claim Mapping:
Larlus discloses a non-transitory computer readable medium storing executable instructions which, when executed by a processing device, cause the processing device to perform operations comprising:
PNG
media_image1.png
453
481
media_image1.png
Greyscale
generating, from a raster image, a hierarchical semantic structure comprising a set of nodes corresponding to masks of objects depicted within the raster image (Larlus, ¶ 26: “FIG. 4 graphically illustrates the hierarchy of regions 34 for the example image 10 as a set of nodes connected by edges to form a tree where the root node 50 of the tree represents the entire image (all its pixels) and has exactly two child nodes 52, 54 (each corresponding to a respective hierarchy region of the image 10 which together form the entire image).”;
determining, within the hierarchical semantic structure, nodes among the set of nodes corresponding to vector regions indicating vector paths corresponding to content depicted within the raster image (Larlus, ¶ 32: “The exemplary feature vectors 72, 74, 76 are statistical representations of the pixels forming the respective region. Examples of such representations include Fisher vectors, shape descriptors, and SIFT descriptors, as described below, although the method is not limited to any particular type of region representation. In the exemplary embodiment, the region feature vectors 72, 74, 76 are all of a same fixed length, e.g., of at least 20 or at least 30 vector elements.”);
providing, for display on a client device (Larlus, ¶ 39: “An output device 112 outputs the segmentation map 14 for the image.”).
Larlus does not disclose, but Tailang discloses generating, from the raster image, a vector image including the vector paths of the vector regions (¶ 44: “digital content 112 including a raster object 114 is received as an input by the semantic vectorization system 110 (block 1202). In some instances, the raster object 114 is included in a digital image 134 captured by a camera device 136. The raster object 114 includes pixels 202. In one instance, the raster object 114 depicts a scene, e.g., a woman on a skateboard on a sidewalk with trees in the background as illustrated in FIG. 3 as a raster object 302. This digital content 112 is utilized by the semantic vectorization system 110 to generate digital content 204 that includes one or more vector objects 116 that mimic the visual appearance of the raster object 114.”); and
providing, for display on a client device together with the vector image, a vector hierarchy interface depicting a hierarchical arrangement of the vector regions according to the hierarchical semantic structure (“The segmentation map 1108 generated from the semantic classification 208 is leveraged to produce a semantic output 1110 including vector objects that resemble the semantic objects of the raster object 114 as described herein. By leveraging the semantic classification of the raster object 114, the semantic vectorization system 110 generates more semantically relevant and more accurate vector objects 116 as compared to conventional techniques. The sematic output including vector objects reduces user interaction, and thus, computational resources that implement the semantic vectorization techniques are used efficiently. Accordingly, the semantic vectorization system as described herein is an improvement over the conventional techniques”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to generate a vector hierarchy display.
One of ordinary skill in the art would have motivation because semantic segmentation can be used to improve vector object generation. One of ordinary skill in the art would have had a reasonable expectation of success because both references consider a hierarchy to segment a raster image.
Claim 17
Larlus discloses wherein generating the hierarchical semantic structure further comprises generating hierarchical layers by determining semantic relationships among the objects within the raster image (Larlus, ¶ 29: “the tree structure of the hierarchy is highly branched and has several layers in the tree… While in the exemplary embodiment, the hierarchical structures 56 each consist of only two generations, i.e., a parent and children, in other embodiments, the hierarchical structures may consist of more than two generations, such as three generations, e.g., grandparent, parent, parent's sibling, and two pairs of children.”)
Allowable Subject Matter
Claim(s) 9-15 allowed.
Claim(s) 2-4, 6-8, 18-20 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding claim(s) 2-3, Larlus considers mutual exclusion in masks and would therefore not suggest determining a first position in the hierarchical semantic structure for the first mask and a second position in the hierarchical semantic structure for the second mask based on a comparison of the intersection to an intersection threshold.
Regarding claim(s) 4, Larlus considers an appropriate mask size which could correspond to mask quality, but Larlus does not suggest and a number of points sampled along a side of the raster image
Regarding claim(s) 6, 7, 8 Tailing suggest traced curves, but does not suggest mapping the vector region to a node of the set of nodes based on increasing an intersection of the vector region with the set of nodes.
Regarding claim(s) 9-15, neither reference suggests in the context of the independent claim: extract, utilizing a vector region segmentation model, a set of vector regions corresponding to content depicted in the raster image; and
modify the hierarchical semantic structure by mapping the set of vector regions to the set of nodes according to intersections between the set of vector regions and the set of masks.
Regarding claim(s) 18 Larlus considers mutual exclusion in masks and would therefore not suggest determining an intersection of the region and the node comprising a ratio of an overlap of the region with the node and a size of the region; and determining the intersection exceeds an intersection threshold.
Regarding claim(s) 19, the cited prior art does not disclose a semantically related subset of the vector paths associated with a region of the vector image; and modifying the semantically related subset of the vector paths based on the selection
Regarding claim(s) 20, the cited prior art does not disclose selecting, based on a user interaction with the vector hierarchy interface, a vector path mapped to a node within the hierarchical semantic structure; determining a semantically related subset of the vector paths associated with the vector path based on the hierarchical semantic structure; and modifying, the semantically related subset of the vector paths based on the selection.
Prior Art
Additional prior art relevant to Applicant’s disclosure but not relied upon:
Liu (US 2022/0114698) also considers segmentation:
PNG
media_image2.png
455
366
media_image2.png
Greyscale
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RYAN M GRAY whose telephone number is (571)272-4582. The examiner can normally be reached on Monday through Friday, 9:00am-5:30pm (EST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached on (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN M GRAY/Primary Examiner, Art Unit 2611
Prosecution Insights