DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement(s) (IDS) submitted on 12/16/2025 is/are being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 5, 9-16, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Hsun, U.S. Publication No. 2015/0117777 in view of Lin et al, U.S. Publication No. 2016/0350930 and Jumpasut et al, U.S. Publication No. 2015/0063697.
Regarding claim 15, Hsun teaches an apparatus comprising:
one or more processors (see Hsun paragraph [0005]); and
memory storing instructions that, when executed by the one or more processors, cause the apparatus to (see paragraph [0006]):
access a plurality of images stored in an image repository, wherein the plurality of images comprises a first image (see Figure 1, storage medium 120 and paragraph [0020]);
detect (see Figure 2, input image to depth analyzer 114 and Figure 5 which shows an embodiment of depth analyzer 114), using one or more image processing techniques, a first object in the first image and a second object in the first image (see Figure 5, segmentation module 504 and paragraph [0040]);
identify a first tag associated with the first object and a second tag associated with the second object (see Figure 1, region parser 116 and Figure 6, which is an embodiment of region parser 116, object identifier 610 which outputs object type 622 and paragraph [0045]);
determine, a first depth value associated with the first object and a second depth value associated with the second object (see Figure 5, depth map generator 506 and paragraph [0041]);
determine, based on the first depth value and the second depth value, a spatial relationship between the first object and the second object, wherein the spatial relationship comprises locations of the first object and the second object in the first image (see Figure 6, region processor 606 and paragraph [0044]); and
generate metadata associated with the first image, wherein the metadata indicates the spatial relationship between the first object and the second object (see Figure 6, region data 124 and paragraph [0044]).
Hsun does not expressively teach wherein the apparatus is to
identify, using one or more machine-learning models, [the] first tag associated with the first object and [the] second tag associated with the second object;
determine, using the one or more machine-learning models, [the] first depth value associated with the first object and [the] second depth value associated with the second object;
wherein the one or more image processing techniques comprises determining a plurality of points of the first image and associating each of the plurality of points with a color value.
However, Lin in a similar invention in the same field of endeavor teaches an apparatus configured to identify a first tag associated with a first object in a first image and a second tag associated with a second object in a first image (see Lin Figure 4, images 112 input into module 114 and output semantically labeled image 120); and determine a first depth value associated with the first object and a second depth value associated with the second object (see Figure 4, depth map 122 and paragraph [0035]) as taught in Hsun wherein the apparatus is to
identify, using one or more machine-learning models, [the] first tag associated with the first object and [the] second tag associated with the second object; and
determine, using the one or more machine-learning models, [the] first depth value associated with the first object and [the] second depth value associated with the second object (see paragraph [0038]).
One of ordinary skill in the art before the effective filing date of the invention would have found it obvious to combine the teaching of using machine learning models for tags and depth values in an image as taught in Lin with the apparatus taught in Hsun, the motivation being to utilize the processing power and accuracy of such models for image recognition tasks.
Hsun in view of Lin does not expressively teach
wherein the one or more image processing techniques comprises determining a plurality of points of the first image and associating each of the plurality of points with a color value.
However, Jumpasut in a similar invention in the same field of endeavor teaches a system for using one or more image processing techniques to detect an object in an image (see Jumpasut Figure 1, segmentation unit 130 and paragraph [0078]) as taught in Hsun in view of Lin wherein
the one or more image processing techniques comprises determining a plurality of points of the first image and associating each of the plurality of points with a color value (see Figure 3, which is an embodiment of segmentation unit 130 of Figure 1, preprocessing unit 131 and paragraph [0084]).
One of ordinary skill in the art before the effective filing date of the invention would have found it obvious as a matter of simple substitution to replace the segmentation technique taught in Hsun in view of Lin with the technique taught in Jumpasut to yield the predictable results of successfully detecting objects in images.
Independent claims 1 and 18 recite similar limitations as claim 15, and are rejected under similar rationale.
Regarding claim 2, Hsun in view of Lin and Jumpasut teaches all the limitations of claim 1, and further teaches wherein the spatial relationship indicates that the first object is in a foreground or a background relative to the second object (see Hsun paragraph [0044]).
Regarding claim 5, Hsun in view of Lin and Jumpasut teaches all the limitations of claim 1, but does not expressively teach wherein the spatial relationship indicates that the first object is obstructed by the second object.
However, Hsun goes on to teach that, through calculating the first and second depth values, a spatial relationship indicating that the first object is obstructed by the second object is determined (see Hsun paragraph [0040]). Therefore, one of ordinary skill in the art before the effective filing date of the invention would have found it obvious to combine the teaching of the spatial relationship determined in Hsun paragraph [0040] with the spatial relationship that eventually is generated in metadata taught in Hsun Figure 6, the motivation being to allow more data about the objects to be input into effect selector 119 of Figure 2 of Hsun thereby refining which effects are added to the output image.
Regarding claim 9, Hsun in view of Lin and Jumpasut teaches all the limitations of claim 1, wherein the one or more image processing techniques comprise at least one of: a scale invariant feature transform (SIFT) technique or a histogram of oriented gradients (HOG) technique.
However, one of ordinary skill in the art before the effective filing date of the invention would have found it obvious as a matter of simple substitution to replace the one or more image processing techniques of Hsun in view of Lin and Jumpasut with those claimed to yield the predictable results of successfully processing the image and identifying objects.
Regarding claim 10, Hsun in view of Lin and Jumpasut teaches all the limitations of claim 1, and further teaches wherein the one or more machine-learning models comprise one or more convolutional neural networks (see Lin paragraph [0043]).
Regarding claim 11, Hsun in view of Lin and Jumpasut teaches all the limitations of claim 1, and further teaches wherein the identifying the first tag associated with the first object and the second tag associated with the second object further comprises:
applying a first label to the first object and a second label to the second object (see Lin Figure 4, semantically labeled image 120 and paragraph [0035] as applied to Hsun Figure 6, object type 622).
Regarding claim 12, Hsun in view of Lin and Jumpasut teaches all the limitations of claim 11, and further teaches determining, based on one or more classifications of the first object and the second object, the first label and the second label (see Hsun Figure 6, object identifier 604 and paragraph [0043]); and
writing the first label and the second label to the metadata along with the spatial relationship (see Hsun Figure 2, region data 124 and object characteristics 125 as combined with Lin Figure 4).
Regarding claim 13, Hsun in view of Lin and Jumpasut teaches all the limitations of claim 12, and further teaches wherein the one or more classifications comprise at least one of: a person, an animal, a vehicle, a landmark, a foreground, or a background (see Hsun paragraph [0043]).
Regarding claim 16, Hsun in view of Lin and Jumpasut teaches all the limitations of claim 15, and further teaches wherein the instructions, when executed by the one or more processors, cause the apparatus to modify image data based at least in part on the metadata, wherein the image data is associated with the first image (see Hsun Figure 2, region data 124 to effect selector 119 and paragraph [0025]).
Regarding claim 19, Hsun in view of Lin and Jumpasut teaches all the limitations of claim 18, and further teaches wherein the first image comprises a plurality of points, and wherein the first depth value and the second depth value are associated with portions of the plurality of points (see Hsun paragraph [0041]).
Regarding claim 20, Hsun in view of Lin and Jumpasut teaches all the limitations of claim 18, and further teaches wherein the plurality of images comprises one or more videos (see Hsun paragraph [0024]).
Method claim 14 recites similar limitations as claim 20, and is rejected under similar rationale.
Claims 3, 7, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Hsun, U.S. Publication No. 2015/0117777 in view of Lin et al, U.S. Publication No. 2016/0350930; Jumpasut et al, U.S. Publication No. 2015/0063697 and Fork et al, U.S. Publication No. 2013/0204866.
Regarding claim 3, Hsun in view of Lin and Jumpasut teaches all the limitations of claim 1, but does not expressively teach wherein the spatial relationship indicates one or more proportions of the first image that are occupied by at least one of the first object or the second object.
However, Fork in a similar invention in the same field of endeavor teaches an apparatus comprising: one or more processors (see Fork paragraph [0023]); and memory storing instructions that, when executed by the one or more processors, cause the apparatus to (see paragraph [0024]): access a plurality of images stored in an image repository, wherein the plurality of images comprises a first image (see Figure 2, image repository 141 and paragraph [0033]);
detect, using one or more image processing techniques, a first object in the first image (see Figure 3, step 330 and paragraph [0046]. Since the size or relative size of an object is determined, this implies that the object itself is detected) and a second object in the first image (see Figure 3, step 350 and paragraph [0054] which indicates that each object with an object tag is analyzed); identify a first tag associated with the first object and a second tag associated with the second object (see Figure 3, step 320 and paragraph [0045]); determine a first depth value associated with the first object and a second depth value associated with the second object (see Figure 3, step 330 and paragraph [0035]); generate metadata associated with the first image, wherein the metadata indicates a spatial relationship of the first object and the second object (see Figure 3, step 340 and paragraphs [0051] and [0053]) as taught in Hsun in view of Lin and Jumpasut wherein
the spatial relationship indicates one or more proportions of the first image that are occupied by at least one of the first object or the second object (see paragraph [0046]).
One of ordinary skill in the art before the effective filing date of the invention would have found it obvious to combine the teaching of the spatial relationship including proportions of objects as taught in Fork with the method taught in Hsun in view of Lin and Jumpasut, the motivation being to allow more data about the objects to be input into effect selector 119 of Figure 2 of Hsun thereby refining which effects are added to the output image.
Regarding claim 7, Hsun in view of Lin and Jumpasut teaches all the limitations of claim 1, but does not expressively teach wherein the spatial relationship indicates a size of the first object relative to the second object.
However, Fork in a similar invention in the same field of endeavor teaches an apparatus comprising: one or more processors (see Fork paragraph [0023]); and memory storing instructions that, when executed by the one or more processors, cause the apparatus to (see paragraph [0024]): access a plurality of images stored in an image repository, wherein the plurality of images comprises a first image (see Figure 2, image repository 141 and paragraph [0033]);
detect, using one or more image processing techniques, a first object in the first image (see Figure 3, step 330 and paragraph [0046]. Since the size or relative size of an object is determined, this implies that the object itself is detected) and a second object in the first image (see Figure 3, step 350 and paragraph [0054] which indicates that each object with an object tag is analyzed); identify a first tag associated with the first object and a second tag associated with the second object (see Figure 3, step 320 and paragraph [0045]); determine a first depth value associated with the first object and a second depth value associated with the second object (see Figure 3, step 330 and paragraph [0035]); generate metadata associated with the first image, wherein the metadata indicates a spatial relationship of the first object and the second object (see Figure 3, step 340 and paragraphs [0051] and [0053]) as taught in Hsun in view of Lin and Jumpasut wherein
the spatial relationship indicates a size of the first object relative to the second object (see paragraph [0046]).
One of ordinary skill in the art before the effective filing date of the invention would have found it obvious to combine the teaching of the spatial relationship including relative sizes of objects as taught in Fork with the method taught in Hsun in view of Lin and Jumpasut, the motivation being to allow more data about the objects to be input into effect selector 119 of Figure 2 of Hsun thereby refining which effects are added to the output image.
Regarding claim 17, Hsun in view of Lin and Jumpasut teaches all the limitations of claim 16, but does not expressively teach wherein the modifying the image data comprises adding one or more portions of the metadata to the image data, deleting one or more portions of the metadata associated with the image data, or adjusting one or more portions of the metadata associated with the image data.
However, Fork in a similar invention in the same field of endeavor teaches an apparatus comprising: one or more processors (see Fork paragraph [0023]); and memory storing instructions that, when executed by the one or more processors, cause the apparatus to (see paragraph [0024]): access a plurality of images stored in an image repository, wherein the plurality of images comprises a first image (see Figure 2, image repository 141 and paragraph [0033]);
detect, using one or more image processing techniques, a first object in the first image (see Figure 3, step 330 and paragraph [0046]. Since the size or relative size of an object is determined, this implies that the object itself is detected) and a second object in the first image (see Figure 3, step 350 and paragraph [0054] which indicates that each object with an object tag is analyzed); identify a first tag associated with the first object and a second tag associated with the second object (see Figure 3, step 320 and paragraph [0045]); determine a first depth value associated with the first object and a second depth value associated with the second object (see Figure 3, step 330 and paragraph [0035]); generate metadata associated with the first image, wherein the metadata indicates a spatial relationship of the first object and the second object (see Figure 3, step 340 and paragraphs [0051] and [0053]); and modify image data based at least in part on the metadata, wherein the image data is associated with the first image (see Figure 3, step 340) as taught in Hsun in view of Lin and Jumpasut wherein
the modifying the image data comprises adding one or more portions of the metadata to the image data (see paragraph [0053] and Figure 6 which shows the new metadata with the image and tag).
One of ordinary skill in the art before the effective filing date of the invention would have found it obvious to combine the teaching of adding metadata to image data as taught in Fork with the apparatus taught in Hsun in view of Lin and Jumpasut, the motivation being to allow users to view the metadata at a later time and thereby make decisions based on the metadata.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Hsun, U.S. Publication No. 2015/0117777 in view of Lin et al, U.S. Publication No. 2016/0350930; Jumpasut et al, U.S. Publication No. 2015/0063697; Koivisto et al, U.S. Publication No. 2019/0258878; and Liu et al, U.S. Publication No. 2018/0039853.
Regarding claim 4, Hsun in view of Lin and Jumpasut teaches all the limitations of claim 1, but does not expressively teach excluding a third object from the metadata based on a determination that the third object occupies less than a threshold amount of the first image.
However, Koivisto in a similar invention in the same field of endeavor teaches a method comprising detecting, using one or more image processing techniques, a first object in a first image and a second object in the first image (see Koivisto Figure 1B, object detector 106 and paragraph [0053]) and generating metadata about the first and second objects (see Figure 1B, feature determiner 110 and paragraph [0078]) as taught in Hsun in view of Lin and Jumpasut further comprising
excluding a third object from the metadata based on a determination that the third object is less than a threshold size (see Figure 1B, detected object filter 116A and paragraph [0071]).
One of ordinary skill in the art before the effective filing date of the invention would have found it obvious to combine the teaching of excluding objects from metadata based on a threshold size as taught in Koivisto with the method taught in Hsun in view of Lin and Jumpasut, the motivation being to conserve power in the system by excluding insignificant objects for analysis (see Koivisto paragraph [0070]).
Hsun in view of Lin, Jumpasut and Koivisto does not expressively teach wherein [the]determination [is] that the third object occupies less than a threshold amount of the first image.
However, Liu in a similar invention in the same field of endeavor teaches that determining whether an object is small in an image can be based on if the object occupies less than a threshold amount of the image (see Liu paragraph [0025]).
One of ordinary skill in the art before the effective filing date of the invention would have found it obvious as a matter of simple substitution to replace the determination on if an object is small based on raw size taught in Hsun in view of Lin, Jumpasut and Koivisto with that taught in Liu to yield the predictable results of successfully excluding insignificant objects and thereby saving power in the method.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Hsun, U.S. Publication No. 2015/0117777 in view of Lin et al, U.S. Publication No. 2016/0350930; Jumpasut et al, U.S. Publication No. 2015/0063697 and Zhao et al, U.S. Publication No. 2020/0401835.
Regarding claim 6, Hsun in view of Lin and Jumpasut teaches all the limitations of claim 1, but does not expressively teach excluding a third object from the metadata based on a determination that the third object is obstructed by greater than a threshold amount.
However, Zhao in a similar invention in the same field of endeavor teaches a method comprising detecting, using one or more image processing techniques, a first object in a first image and a second object in the first image (see Zhao Figure 3A, step 304 and paragraph [0050]) and generating metadata about the first and second objects (see Figure 3A, step 310 and paragraph [0060]) as taught in Hsun in view of Lin and Jumpasut further comprising
excluding a third object from the metadata based on a determination that the third object is obstructed by greater than a threshold amount (see paragraph [0051]).
One of ordinary skill in the art before the effective filing date of the invention would have found it obvious to combine the teaching of excluding objects from metadata based on obstruction amount as taught in Zhao with the method taught in Hsun in view of Lin and Jumpasut, the motivation being to conserve power in the system by excluding insignificant objects for analysis.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Hsun, U.S. Publication No. 2015/0117777 in view of Lin et al, U.S. Publication No. 2016/0350930; Jumpasut et al, U.S. Publication No. 2015/0063697 and Koivisto et al, U.S. Publication No. 2019/0258878.
Regarding claim 8, Hsun in view of Lin and Jumpasut teaches all the limitations of claim 1, but does not expressively teach excluding a third object from the metadata based on a determination that the third object is less than a threshold size.
However, Koivisto in a similar invention in the same field of endeavor teaches a method comprising detecting, using one or more image processing techniques, a first object in a first image and a second object in the first image (see Koivisto Figure 1B, object detector 106 and paragraph [0053]) and generating metadata about the first and second objects (see Figure 1B, feature determiner 110 and paragraph [0078]) as taught in Hsun in view of Lin and Jumpasut further comprising
excluding a third object from the metadata based on a determination that the third object is less than a threshold size (see Figure 1B, detected object filter 116A and paragraph [0071]).
One of ordinary skill in the art before the effective filing date of the invention would have found it obvious to combine the teaching of excluding objects from metadata based on a threshold size as taught in Koivisto with the method taught in Hsun in view of Lin and Jumpasut, the motivation being to conserve power in the system by excluding insignificant objects for analysis (see Koivisto paragraph [0070]).
Response to Arguments
Applicant’s arguments with respect to claim(s) 1, 15, and 18 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CASEY L KRETZER whose telephone number is (571)272-5639. The examiner can normally be reached M-F 10:00-7:00 PM Pacific Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Payne can be reached at (571)272-3024. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CASEY L KRETZER/Primary Examiner, Art Unit 2635