Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The new information disclosure statement (IDS) submitted was filed on 11/11/2025. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Arguments
Applicant presents new arguments starting with amendments to overcome the 112b rejections. Examiner agrees with the amendments to claims 41 and 45 and removes the rejection, but another 112b rejection is raised regarding the amendments to claim 30. Applicant also presents arguments regarding the cited references under 103, more specifically the reference Maarek regarding the underlined part of the independent claims. Examiner disagrees and presents reasoning. The added information to the independent claims is a “population statistic” and “population ranking”. This is clearly covered within a BRI (broadest reasonable interpretation) of the claim interpretation, the reference Maarek directly teaches “population statistics” as “the quality may also include values indicating the number of times a user has interacted with the content (e.g., viewed, shared, commented, downloaded, re-blogged, re-posted, favorited, liked, and the like) at or above the quality threshold” on paragraph 115. In addition, Maarek also directly teaches a “population grouping” in the form of ranking the quality. Applicant also presents new claims 46 and 47 which are also rejected by the current reference Maarek and the newly cited reference Yang for the new information presents in claim 47. Therefore for the reasons presented, all of the claims remain rejected.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 30 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 30 recites the limitation " a label based on the population to obtain labeled images grouping ". There is insufficient antecedent basis for this limitation in the claim. It is also unclear, One of ordinary skill in the art would ask “is it the population statistic or population grouping?”.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim 24, 26-37 and 39-46 are rejected under 35 U.S.C. 103 as being unpatentable over Maarek et. al, hereafter Maarek (US Publication No. 20210352030 A1) in view of Hao et. al. (US-10467507-B1)
As per claim 24, Maarek teaches “A computer-implemented method for automated labeling of images based on implicit user signals indicative of image quality, the method comprising:
obtaining, by one or more computing devices, a plurality of images;” (“[0079] Applications 242 may include computer executable instructions which, when executed by Client device 200, transmit, receive, and/or otherwise process audio, video, images, and enable telecommunication with a server and/or another user of another client device.”) Maarek)
“grouping, by the one or more computing devices, each image in the plurality of images into one or more clusters based at least in part on a time metric; and” (See paragraph 112, cluster is interpreted as a group, and the time metric is “the decision tree model can group “near-duplicate” pairs having predetermined time differences”. [0112] In Step 410, all of the photos within the set that are identified as being “near-duplicates” of each other are grouped together. For example, if the identified set of photos includes subsets of photos of a rainbow, a beach and a sky, then these subsets are grouped together by the messaging engine 300. In some embodiments, the grouping of the identified “near-duplicate” photos identified in Step 408 can be grouped according to any known or to be known image classification algorithm, technology or mechanism, for example, using a decision tree model. For example, as mentioned above, the decision tree model can group “near-duplicate” pairs having predetermined time differences (e.g., time deltas of 5 seconds or less).” Maarek)
“for at least one of the one or more clusters:
obtaining, by the one or more computing devices, one or more user signals descriptive of one or more user actions relative to one or more of the images in the cluster;” (See paragraphs 113 and 115. The photos described by the reference have quality, and this quality encompasses a user signal descriptive of one or more user actions (“[0115]… In another non-limiting embodiment, the quality may also include values indicating the number of times a user has interacted with the content (e.g., viewed, shared, commented, downloaded, re-blogged, re-posted, favorited, liked, and the like) at or above the quality threshold.) Examiner interprets “viewed, shared, commented, downloaded, re-blogged, re-posted, favorited, liked, and the like” as user signals descriptive of user actions.” Maarek)
“assigning, by the one or more computing devices, one or more weights to the one or more user signals based on the one or more user actions;” (See paragraph 113, it shows that a photo has quality and shareability is tied to the quality. In paragraph 115 it shows that quality includes values seen in “the quality may also include values indicating the number of times a user has interacted with the content (e.g., viewed, shared, commented, downloaded, re-blogged, re-posted, favorited, liked, and the like)”. The quality taught here is the value of the user actions specified, therefore within the BRI (Broadest Reasonable Interpretation), the weight representing user signals based on user actions is interpreted as a quality value representing user signals based on user actions (viewed, shared, commented, downloaded, re-blogged, re-posted, favorited, liked, and the like). Maarek)
“inferring, by the one or more computing devices, a quality metric for at least one image in the cluster based at least in part on the one or more weights assigned to the one or more user signals descriptive of the user actions relative to the images in the cluster;” (See paragraphs 113 and 115. Photos from a group are chosen based on qualities, which includes user actions relative to the images in the cluster. “[0113] In Step 412, each group of photos is then analyzed and the “highest quality” photo from each group is identified as a representative photo for each group. ”. Paragraph 115 contains “In another non-limiting embodiment, the quality may also include values indicating the number of times a user has interacted with the content (e.g., viewed, shared, commented, downloaded, re-blogged, re-posted, favorited, liked, and the like) at or above the quality threshold.”. .” As seen in the argument paragraph 113 shows that the quality is tied to shareability. The claim discloses that the “quality metric” is based on the “weights” assigned to the user signals. The quality metric, within the BRI (Broadest Reasonable Interpretation), is interpreted as “shareability”, which as seen in paragraphs 115, 113 as presented in the prior limitation , it is based on a quality value (weight) assigned to the user signals. Maarek))
determining, by the one or more computing devices, a population statistic for the cluster based at least in part on the quality metrics determined for the images in the cluster; (See paragraph 115, examiner interprets “a population statistic” as the number of times a user interacts with content, such as share counts. “[0115] In a non-limiting example, “high-quality” or “highest quality” (used interchangeably) can refer to the digital photo being of interest to a user(s), where interest (or user engagement) can be based on… In another example, an image's quality can be based on the latent values of the photo satisfying a quality threshold. In another non-limiting embodiment, the quality may also include values indicating the number of times a user has interacted with the content (e.g., viewed, shared, commented, downloaded, re-blogged, re-posted, favorited, liked, and the like) at or above the quality threshold.” See also paragraphs 116-123. See also fig. 4A. Maarek)
determining, by the one or more computing devices, a population grouping based on the population statistic, wherein the population grouping corresponds to a plurality of images of the images in the cluster; (See paragraphs 119-125, examiner interprets “population grouping” as ranking, “ [0120] In Step 416, the groups of photos (created and/or identified in Step 410) are ranked based on the “shareability” value of each group's representative “high-quality” photo. Thus, the groups are ranked according to how likely a group is determined to include a shareable photo” “[0122] In some embodiments, the groups of photos are ranked such that an entire grouping of photos is placed higher in the ranking than another grouping of photos; and in some embodiments, only individual photos are ranked, either across groups or within a group, such that only the “highest quality” photos are ranked based on their “shareability” value.” See also fig. 4A Maarek)
labeling, by the one or more computing devices, each image of the plurality of images of the images in the cluster with a label based on the population grouping to obtain labeled images; and (Shareability is interpreted as “a label”, and it is generated for at least one image in a group, and the shareability is based on quality (quality metrics). See paragraphs 115-125, [0118] In Step 414, each group's highest quality photo is analyzed and a “shareability” value is determined for each photo, and between each photo. As discussed above, the “shareability” of each photo is tied to the quality of each photo; therefore, the results of Step 412 can be leveraged into the determination of how “shareable” a “high quality” photo actually is. In some embodiments, the higher the photo's “quality”, the higher the “shareability” value. This step is part of fig. 4A where it shows that this leads to step 418 that obtains a set of “highest quality photos”, therefore it is for obtaining labeled images. Then in step 416 the photos are ranked (population grouping). Going into step 418-420, the images are labeled as high quality, examiner interprets the images not labeled as high quality are implicitly low quality, and therefore implicitly also labeled. “[0125] In Step 418, a set of “highest quality” photos (or “shareable”) photos that represent each group within the ranking is selected. An example of the selected set of photos can be seen in FIG. 5, items 510-514, as discussed above, illustrated in a like manner accordingly, and discussed in more detail below in relation to Steps 420-422. In some embodiments, the selection of the photos involves the messaging engine 300 automatically selecting the top-N photos from the top group. In some embodiments, the selection of the photos involves the messaging engine 300 automatically selecting the top-N photos from each of the groups, or across each group.”. See also fig. 4A. Maarek), Maarek also teaches a “training dataset…” (Paragraphs 123-124 show a training dataset, see also paragraphs 45, 106 and 111. Maarek), however Maarek does not directly teach “a training dataset comprising: the labeled images; and the label generated for the labeled images. ”
Hao teaches “generating, by the one or more computing devices, a training dataset comprising:
the labeled images; and the label for the labeled images.” (See page 8 paragraph 3-5 and page 9 paragraph 1. “In some embodiments, the training data sets may be labeled by hand, the labeled images may be labeled as high quality or low quality. A first portion of the training set may be labeled as high quality or good quality to indicate that the first portion of images from the training set contains images with good composition. A second portion of the training set is may be labeled as bad quality or low quality to indicate that the second portion of the training set contains images with poor composition.” See also page 7 paragraph 4 “Based on the image expert feedback data, additional training data sets may be generated, and/or the image scoring model may be further refined.”. This is performed by the processors shown on fig. 2. Hao)
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings of Maarek with the teachings of Hao to generate a training data set comprising labels and labeled images. The modification would have been motivated by the desire to have the ability to further refine the deep learning model as suggested by Hao (See page 7 paragraph 4 which mentions that additional training data sets are generated, and see page 8 paragraph 4 which shows that the training data sets are for refining training (which results in a refined deep learning model). “For example, the labeling of images as “good” may also be described labeling the image as good quality, high quality, or high scoring. Similarly, the labeling of images as “bad” may also be described as labeling the image as bad quality, poor quality, low quality, or low scoring. Additional training data sets may be added, which may add or remove features to extract, to help refine training. Training may be refined until the model is able extract features from images and using the extracted features to classify images within a predetermined precision parameter or precision threshold.” Hao).
Claim 44 is rejected under the same analysis as claim 24, Maarek also teaches computer readable media (See paragraphs 9-10 and 25. Maarek)
As per claim 26 Maarek in view of Hao teaches “The computer-implemented method of claim 24, wherein the one or more user signals descriptive of user actions relative to the images in the cluster comprise user viewing data that indicates a number of times each image has been viewed by a user.” (See paragraph 115. “In another non-limiting embodiment, the quality may also include values indicating the number of times a user has interacted with the content (e.g., viewed, shared, commented, downloaded, re-blogged, re-posted, favorited, liked, and the like) at or above the quality threshold.” Maarek)
As per claim 27 Maarek in view of Hao teaches “The computer-implemented method of claim 24, wherein the one or more user signals descriptive of user actions relative to the images in the cluster comprise user interaction data that indicates a number of times a user has interacted with each image via physical user input controls.” (See paragraph 115. “In another non-limiting embodiment, the quality may also include values indicating the number of times a user has interacted with the content (e.g., viewed, shared, commented, downloaded, re-blogged, re-posted, favorited, liked, and the like) at or above the quality threshold.” Maarek)
As per claim 28, Maarek in view of Hao teaches “The computer-implemented method of claim 24, wherein the one or more user signals descriptive of user actions relative to the images in the cluster comprise user sharing data that indicates a number of times each image has been shared by a user.” (See paragraph 115. “In another non-limiting embodiment, the quality may also include values indicating the number of times a user has interacted with the content (e.g., viewed, shared, commented, downloaded, re-blogged, re-posted, favorited, liked, and the like) at or above the quality threshold.” Maarek)
As per claim 29 Maarek in view of Hao “The computer-implemented method of claim 24, wherein the one or more user signals descriptive of user actions relative to the images in the cluster comprise user favoriting data that indicates a number of times each image has been favorited by a user.” (See paragraph 115. “In another non-limiting embodiment, the quality may also include values indicating the number of times a user has interacted with the content (e.g., viewed, shared, commented, downloaded, re-blogged, re-posted, favorited, liked, and the like) at or above the quality threshold.” Maarek)
As per claim 30, Maarek in view of Hao already teaches “The computer-implemented method of claim 24, wherein labeling, by the one or more computing devices, each image of the plurality of images of the images in the cluster with a label based on the population to obtain labeled images grouping comprises:”, Maarek in view of Hao also teaches
“identifying, by the one or more computing devices based at least in part on the quality metrics, a first set of images from the cluster that have superior quality to a second, different set of images from the cluster;” (Paragraph 122 teaches that photos are ranked based on shareability (which is based on quality) in the group. Examiner interprets photos that are ranked as having “superior quality”. “[0122] In some embodiments, the groups of photos are ranked such that an entire grouping of photos is placed higher in the ranking than another grouping of photos; and in some embodiments, only individual photos are ranked, either across groups or within a group, such that only the “highest quality” photos are ranked based on their “shareability” value. In some embodiments, the “shareability” value of a group is compared to a threshold value, such that if the “shareability” value is at or below the threshold value, it may be dropped from the ranked list/set.” Maarek)
“labelling, by the one or more computing devices, the first set of images with a first label; and” (See paragraph 122, examiner interprets “first label” as highest quality photos (shareable photos). [0122] In some embodiments, the groups of photos are ranked such that an entire grouping of photos is placed higher in the ranking than another grouping of photos; and in some embodiments, only individual photos are ranked, either across groups or within a group, such that only the “highest quality” photos are ranked based on their “shareability” value. In some embodiments, the “shareability” value of a group is compared to a threshold value, such that if the “shareability” value is at or below the threshold value, it may be dropped from the ranked list/set. Maarek)
“labelling, by the one or more computing devices, the second set of images with a second, different label.” (See paragraph 122, examiner interprets “second, different label” as not ranked (not shareable photos). )
As per claim 31, Maarek in view of Hao teaches “The computer-implemented method of claim 24, further comprising: training, by the one or more computing devices and using a learning technique, a machine-learned model on the training dataset.” (See paragraph 124, the training dataset is used to train a machine learning algorithm. “[0124] Given the training data set, the messaging engine 300 can implement a ranker (e.g., via the ranking module 308) in order to rank photos that are likely to be shared higher than photos that are not likely to be shared by applying any known or to be known machine-learning algorithm for supervised ranking, such as, but not limited to, a linear ranking (e.g. SVMrank) or Gradient-boosted Decision Trees (GBDT).” Maarek)
As per claim 32, Maarek in view of Hao teaches “the computer-implemented method of claim 31, wherein the machine-learned model is trained to select one or more superior quality images from a sequence of input images.” (See paragraph 124 and 125, the machine learning algorithm is trained to rank photos (therefore it selects one or more superior quality images) based on having a higher shareability (quality). “[0124] Given the training data set, the messaging engine 300 can implement a ranker (e.g., via the ranking module 308) in order to rank photos that are likely to be shared higher than photos that are not likely to be shared by applying any known or to be known machine-learning algorithm for supervised ranking, such as, but not limited to, a linear ranking (e.g. SVMrank) or Gradient-boosted Decision Trees (GBDT).” “[0125] In Step 418, a set of “highest quality” photos (or “shareable”) photos that represent each group within the ranking is selected.” Maarek )
As per claim 33, Maarek in view of Hao teaches “the computer-implemented method of claim 24, wherein the training dataset does not include ground truth data labeled by a human.” (See paragraphs 114, 115 the quality (ground truth data) can be included by the system, service or platform hosting the content. Examiner interprets “ground truth data” as the data primarily used for the dataset. Maarek)
As per claim 34 Maarek in view of Hao teaches “the computer-implemented method of claim 24, wherein grouping, by the one or more computing devices, each image in the plurality of images into one or more clusters comprises:
identifying, by the one or more computing devices, a timestamp associated with each image; and” (See paragraph 104, the metadata of an image includes a timestamp. “The identification of such set of photos within the collection can be based on metadata of the photos in the collection, which provides timestamps and/or positioning tags indicating when and where the photos originated.” Maarek)
“selecting, by the one or more computing devices, images from the plurality of images to include in each of the one or more clusters such that the timestamp associated with each image within each cluster is within a timespan.” (See paragraph 112, the images can be grouped based on a time difference (interpreted as the timestamp associated with each image within each cluster is within a timespan). Maarek)
As per claim 35, Maarek in view of Hao teaches “the computer-implemented method of claim 34, wherein the plurality of images consists substantially of one or more burst image sets, and wherein each of the one or more burst image sets comprise a video sequence of image frames, and wherein the timestamp associated with each image frame in the video sequence is within the timespan.” (See paragraph 112, examiner interprets burst as group, and the groups can be grouped up based on time difference. Examiner interprets timespan as “the timestamp associated with each image frame in the video sequence is within the timespan”. “[0112]… For example, as mentioned above, the decision tree model can group “near-duplicate” pairs having predetermined time differences (e.g., time deltas of 5 seconds or less).” See also paragraphs 36-37,53, 67, 79 which show the use of videos. Paragraph 104 also shows that the images have timestamps. Maarek)
As per claim 39 Maarek in view of Hao teaches “The computer-implemented method of claim 24, wherein grouping, by the one or more computing devices, each image into one or more clusters comprises:
determining, by the one or more computing devices, a representative image for each cluster;” (See paragraph 113 “[0113] In Step 412, each group of photos is then analyzed and the “highest quality” photo from each group is identified as a representative photo for each group.” Maarek)
“selecting, by the one or more computing devices, a set of images from the plurality of images based in part on the time metric, wherein each image in the set of images meets a threshold of the time metric; and” (See paragraphs 112, 114 and 115. In paragraph 112 the images can be grouped having a predetermined time differences (threshold of the time metric), therefore each image in the set (group) meets the threshold. “[112]… For example, as mentioned above, the decision tree model can group “near-duplicate” pairs having predetermined time differences (e.g., time deltas of 5 seconds or less) Maarek)
“comparing, by the one or more computing devices, each image in the set of images to the representative image.” (See paragraph 122. Examiner interprets “comparing” as a ranking, since to do a ranking a comparison needs to happen. “[0122] In some embodiments, the groups of photos are ranked such that an entire grouping of photos is placed higher in the ranking than another grouping of photos; and in some embodiments, only individual photos are ranked, either across groups or within a group, such that only the “highest quality” photos are ranked based on their “shareability” value.” Maarek)
As per claim 40, Maarek in view of Hao teaches “the computer-implemented method of claim 39, wherein comparing, by the one or more computing devices, each image in the set of images to the representative image comprises:
determining, by the one or more computing devices and using a machine-learned model configured to generate a similarity score between two images, similarity scores for each image in the set of images; and” (See paragraphs 109 and 111. Steps 458 consists of finding the similarity between two photos and steps 458 to 460 are based on a machine learned model (learned classifier). “[109]… Step 458. In some embodiments, using the latent vectors of each photo, the similarity between two photos is taken as the Cosine similarity between each photo's latent vector's values.” [0111] According to some embodiments, the latent vector analysis (Steps 458-460) can be based on a learned classifier, as mentioned above, which can be trained on a training set of photos by analyzing each photo (in some embodiments, in a human supervised manner) using a decision tree model.” Maarek)
“adding, by the one or more computing devices, any images determined to have similarity scores meeting a threshold value to the cluster associated with the representative image.” ( See Paragraphs 109-111 which also teaches the use of a threshold value when taking similarity into account between two images (cluster). Examiner interprets “the representative image” as any of the two images. “Step 460 Thus, when these latent values are at or above a threshold value, the two photos can be determined to be “near-duplicates.” In addition, Paragraph 113-115 shows that there is a representative photo chosen, which is based on quality thresholds. Maarek)
As per claim 41, Maarek in view of Hao already teaches “The computer-implemented method of claim 24, wherein labeling, by the one or more computing devices, each image of the plurality of images of the images in the cluster with a label based on the population grouping to obtain labeled images comprises:”, Maarek in view of Hao also teaches
“creating, by the one or more computing devices, a vector of the quality metrics determined for each image in one of the one or more clusters; and wherein each image of the plurality of images is labeled based on the vector.” (See paragraphs 115, it teaches the quality metrics of each image, also interpreted as image features. See also paragraph 45 “As discussed below, in some embodiments, features for each photo are derived, determined, or otherwise identified based on the photo's contents, including, but not limited to, a latent vector representation based on a deep-learning network as well as semantic binary derivatives, such as the photo aesthetics.”. See also paragraphs 89 “[0089] In some embodiments, the information stored in database 320 can be represented as an n-dimensional vector (or feature vector) for each stored data/metadata item, where the information associated with, for example, the stored data and/or metadata can correspond to a node(s) on the vector. As such, database 320 can store and index stored information in database 320 as linked set of data and metadata, where the data and metadata relationship can be stored as the n-dimensional vector discussed above. ”. See also paragraphs 107-111. See also paragraph 116 “For example, a photo's relevancy can be determined via implementation of a logistic loss function which quantifies a photo's parameters or features and ranks them according to such scoring.” Maarek )
As per claim 42, Maarek in view of Hao teaches “The computer-implemented method of claim 24, wherein obtaining, by the one or more computing devices, the plurality of images comprises: obtaining, by the one or more computing devices, a respective property dataset associated with one or more of the plurality of images, wherein the respective property dataset for each image comprises one or more of: a time the image was taken, a date the image was taken, a place the image was taken, a number of times the image was accessed, a number of times the image was shared, or combinations thereof.” (It is well known in the art that images can include metadata (a respective property dataset associated with one or more images), in addition see paragraph 104 “This information can serve as a part of a query of the photo collection in order to identify photos that where taken (e.g., captured and/or downloaded/uploaded) when the sending user was on vacation. The identification of such set of photos within the collection can be based on metadata of the photos in the collection, which provides timestamps and/or positioning tags indicating when and where the photos originated.” Maarek)
As per claim 43, Maarek in view of Hao teaches “The computer-implemented method of claim 24, wherein obtaining, by the one or more computing devices, the plurality of images comprises:
accessing, by the one or more computing devices, an application configured to process image data on a user device; and” (See fig. 1 and paragraphs 67 and 82-86. Client device 101 (user) is connected to the network which is connected to the servers, which includes the application. “[0067] In some embodiments, users are able to access services provided by servers 106, 108, 120 and/or 130. This may include in a non-limiting example, authentication servers, search servers, email servers, social networking services servers, SMS servers, IM servers, MMS servers, exchange servers, photo-sharing services servers, and travel services servers, via the network 105 using their various devices 101-104.” Maarek)
“enabling, by the one or more computing devices and via an application programming interface, the application to transmit data to a data labeling application configured to determine the quality metric.” (See paragraphs 47,81,82 and fig. 3. The messaging engine 300 (the data labeling application configured to determine the quality metric) can execute (enable) on a user device “[0082] According to some embodiments, messaging engine 300 can be embodied as a stand-alone application that executes on a user device. In some embodiments, the messaging engine 300 can function as an application installed on the user's device, and in some embodiments, such application can be a web-based application accessed by the user device over a network. In some embodiments, the messaging engine 300 can be installed as an augmenting script, program or application to another messaging and/or media content hosting/serving application, such as, for example, Yahoo!® Mail, Yahoo!® Messenger, Yahoo!® Search, Flickr®, Tumblr®, Twitter®, Instagram®, SnapChat®, Facebook®, and the like.” Maarek)
As per claim 45, Maarek teaches “A computing system comprising a machine-learned model, wherein the machine- learned model has been trained by performance of operations, the operations comprising: (Paragraphs 45, 111 and 124 show the use of machine learning trained. Examiner interprets “performance of operations” as simply performing operations, which machine learning already does. Maarek)
obtaining, by one or more computing devices, a plurality of images;” (“[0079] Applications 242 may include computer executable instructions which, when executed by Client device 200, transmit, receive, and/or otherwise process audio, video, images, and enable telecommunication with a server and/or another user of another client device.”) Maarek)
“grouping, by the one or more computing devices, each image in the plurality of images into one or more clusters based at least in part on a time metric; and” (See paragraph 112, cluster is interpreted as a group, and the time metric is “the decision tree model can group “near-duplicate” pairs having predetermined time differences”. [0112] In Step 410, all of the photos within the set that are identified as being “near-duplicates” of each other are grouped together. For example, if the identified set of photos includes subsets of photos of a rainbow, a beach and a sky, then these subsets are grouped together by the messaging engine 300. In some embodiments, the grouping of the identified “near-duplicate” photos identified in Step 408 can be grouped according to any known or to be known image classification algorithm, technology or mechanism, for example, using a decision tree model. For example, as mentioned above, the decision tree model can group “near-duplicate” pairs having predetermined time differences (e.g., time deltas of 5 seconds or less).” Maarek)
“for at least one of the one or more clusters: obtaining, by the one or more computing devices, one or more user signals descriptive of one or more user actions relative to one or more of the images in the cluster;” (See paragraphs 113 and 115. The photos described by the reference have quality, and this quality encompasses a user signal descriptive of one or more user actions (“[0115]… In another non-limiting embodiment, the quality may also include values indicating the number of times a user has interacted with the content (e.g., viewed, shared, commented, downloaded, re-blogged, re-posted, favorited, liked, and the like) at or above the quality threshold.) Examiner interprets “viewed, shared, commented, downloaded, re-blogged, re-posted, favorited, liked, and the like” as user signals descriptive of user actions.” Maarek)
“assigning, by the one or more computing devices, one or more weights to the one or more user signals based on the one or more user actions;” (See paragraph 113, it shows that a photo has quality and shareability is tied to the quality. In paragraph 115 it shows that quality includes values seen in “the quality may also include values indicating the number of times a user has interacted with the content (e.g., viewed, shared, commented, downloaded, re-blogged, re-posted, favorited, liked, and the like)”. The quality taught here is the value of the user actions specified, therefore within the BRI (Broadest Reasonable Interpretation), the weight representing user signals based on user actions is interpreted as a quality value representing user signals based on user actions (viewed, shared, commented, downloaded, re-blogged, re-posted, favorited, liked, and the like). Maarek)
“inferring, by the one or more computing devices, a quality metric for at least one image in the cluster based at least in part on the one or more weights assigned to the one or more user signals descriptive of the user actions relative to the images in the cluster;” (See paragraphs 113 and 115. Photos from a group are chosen based on qualities, which includes user actions relative to the images in the cluster. “[0113] In Step 412, each group of photos is then analyzed and the “highest quality” photo from each group is identified as a representative photo for each group. ”. Paragraph 115 contains “In another non-limiting embodiment, the quality may also include values indicating the number of times a user has interacted with the content (e.g., viewed, shared, commented, downloaded, re-blogged, re-posted, favorited, liked, and the like) at or above the quality threshold.”. .” As seen in the argument paragraph 113 shows that the quality is tied to shareability. The claim discloses that the “quality metric” is based on the “weights” assigned to the user signals. The quality metric, within the BRI (Broadest Reasonable Interpretation), is interpreted as “shareability”, which as seen in paragraphs 115, 113 as presented in the prior limitation , it is based on a quality value (weight) assigned to the user signals. Maarek))
“determining, by the one or more computing devices, a population statistic for the cluster based at least in part on the quality metrics determined for the images in the cluster;” (See paragraph 115, examiner interprets “a population statistic” as the number of times a user interacts with content, such as share counts. “[0115] In a non-limiting example, “high-quality” or “highest quality” (used interchangeably) can refer to the digital photo being of interest to a user(s), where interest (or user engagement) can be based on… In another example, an image's quality can be based on the latent values of the photo satisfying a quality threshold. In another non-limiting embodiment, the quality may also include values indicating the number of times a user has interacted with the content (e.g., viewed, shared, commented, downloaded, re-blogged, re-posted, favorited, liked, and the like) at or above the quality threshold.” See also paragraphs 116-123. See also fig. 4A. Maarek)
“determining, by the one or more computing devices, a population grouping based on the population statistic, wherein the population grouping corresponds to a plurality of images of the images in the cluster;” (See paragraphs 119-125, examiner interprets “population grouping” as ranking, “ [0120] In Step 416, the groups of photos (created and/or identified in Step 410) are ranked based on the “shareability” value of each group's representative “high-quality” photo. Thus, the groups are ranked according to how likely a group is determined to include a shareable photo” “[0122] In some embodiments, the groups of photos are ranked such that an entire grouping of photos is placed higher in the ranking than another grouping of photos; and in some embodiments, only individual photos are ranked, either across groups or within a group, such that only the “highest quality” photos are ranked based on their “shareability” value.” See also fig. 4A Maarek)
labeling, by the one or more computing devices, each image of the plurality of images of the images in the cluster with a label based on the population grouping to obtain labeled images; and (Shareability is interpreted as “a label”, and it is generated for at least one image in a group, and the shareability is based on quality (quality metrics). See paragraphs 115-125, [0118] In Step 414, each group's highest quality photo is analyzed and a “shareability” value is determined for each photo, and between each photo. As discussed above, the “shareability” of each photo is tied to the quality of each photo; therefore, the results of Step 412 can be leveraged into the determination of how “shareable” a “high quality” photo actually is. In some embodiments, the higher the photo's “quality”, the higher the “shareability” value. This step is part of fig. 4A where it shows that this leads to step 418 that obtains a set of “highest quality photos”, therefore it is for obtaining labeled images. Then in step 416 the photos are ranked (population grouping). Going into step 418-420, the images are labeled as high quality, examiner interprets the images not labeled as high quality are implicitly low quality, and therefore implicitly also labeled. “[0125] In Step 418, a set of “highest quality” photos (or “shareable”) photos that represent each group within the ranking is selected. An example of the selected set of photos can be seen in FIG. 5, items 510-514, as discussed above, illustrated in a like manner accordingly, and discussed in more detail below in relation to Steps 420-422. In some embodiments, the selection of the photos involves the messaging engine 300 automatically selecting the top-N photos from the top group. In some embodiments, the selection of the photos involves the messaging engine 300 automatically selecting the top-N photos from each of the groups, or across each group.”. See also fig. 4A. Maarek), Maarek also teaches a “training dataset…” (Paragraphs 123-124 show a training dataset, see also paragraphs 45, 106 and 111. Maarek), however Maarek does not completely teach “generating, by the one or more computing devices, a training dataset comprising: the labeled images; and the label generated for the labeled images.”
Hao teaches “generating, by the one or more computing devices, a training dataset comprising: the labeled images; and the label generated for the labeled images.” (See page 8 paragraph 3-5 and page 9 paragraph 1. “In some embodiments, the training data sets may be labeled by hand, the labeled images may be labeled as high quality or low quality. A first portion of the training set may be labeled as high quality or good quality to indicate that the first portion of images from the training set contains images with good composition. A second portion of the training set is may be labeled as bad quality or low quality to indicate that the second portion of the training set contains images with poor composition.” See also page 7 paragraph 4 “Based on the image expert feedback data, additional training data sets may be generated, and/or the image scoring model may be further refined.”. This is performed by the processors shown on fig. 2. Hao)
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings of Maarek with the teachings of Hao to generate a training data set comprising labels and labeled images. The modification would have been motivated by the desire to have the ability to further refine the deep learning model as suggested by Hao (See page 7 paragraph 4 which mentions that additional training data sets are generated, and see page 8 paragraph 4 which shows that the training data sets are for refining training (which results in a refined deep learning model). “For example, the labeling of images as “good” may also be described labeling the image as good quality, high quality, or high scoring. Similarly, the labeling of images as “bad” may also be described as labeling the image as bad quality, poor quality, low quality, or low scoring. Additional training data sets may be added, which may add or remove features to extract, to help refine training. Training may be refined until the model is able extract features from images and using the extracted features to classify images within a predetermined precision parameter or precision threshold.” Hao).
As per claim 46 Maarek in view of Hao already teaches “The computer-implemented method of claim 24, further comprising:”, however Maarek also teaches “determining, by the one or more computing devices, a second population grouping based on the population statistic, wherein the second population grouping corresponds to one or more images of the images in the cluster; (See paragraphs 119-125, examiner interprets “population grouping” as ranking, “ [0120] In Step 416, the groups of photos (created and/or identified in Step 410) are ranked based on the “shareability” value of each group's representative “high-quality” photo. Thus, the groups are ranked according to how likely a group is determined to include a shareable photo” “[0122] In some embodiments, the groups of photos are ranked such that an entire grouping of photos is placed higher in the ranking than another grouping of photos; and in some embodiments, only individual photos are ranked, either across groups or within a group, such that only the “highest quality” photos are ranked based on their “shareability” value.” See also fig. 4A. See also paragraphs 125 “[0125] In Step 418, a set of “highest quality” photos (or “shareable”) photos that represent each group within the ranking is selected. An example of the selected set of photos can be seen in FIG. 5, items 510-514, as discussed above, illustrated in a like manner accordingly, and discussed in more detail below in relation to Steps 420-422. In some embodiments, the selection of the photos involves the messaging engine 300 automatically selecting the top-N photos from the top group. In some embodiments, the selection of the photos involves the messaging engine 300 automatically selecting the top-N photos from each of the groups, or across each group.” By repeating the grouping for other clusters, it teaches a “second population grouping” within a BRI. Maarek),
and labeling, by the one or more computing devices, each image of the one or more images of the images in the cluster with a label based on the second population grouping.” (Shareability is interpreted as “a label”, and it is generated for at least one image in a group, and the shareability is based on quality (quality metrics). See paragraphs 115-125, [0118] In Step 414, each group's highest quality photo is analyzed and a “shareability” value is determined for each photo, and between each photo. As discussed above, the “shareability” of each photo is tied to the quality of each photo; therefore, the results of Step 412 can be leveraged into the determination of how “shareable” a “high quality” photo actually is. In some embodiments, the higher the photo's “quality”, the higher the “shareability” value. This step is part of fig. 4A where it shows that this leads to step 418 that obtains a set of “highest quality photos”, therefore it is for obtaining labeled images. Then in step 416 the photos are ranked (population grouping). Going into step 418-420, the images are labeled as high quality, examiner interprets the images not labeled as high quality are implicitly low quality, and therefore implicitly also labeled. “[0125] In Step 418, a set of “highest quality” photos (or “shareable”) photos that represent each group within the ranking is selected. An example of the selected set of photos can be seen in FIG. 5, items 510-514, as discussed above, illustrated in a like manner accordingly, and discussed in more detail below in relation to Steps 420-422. In some embodiments, the selection of the photos involves the messaging engine 300 automatically selecting the top-N photos from the top group. In some embodiments, the selection of the photos involves the messaging engine 300 automatically selecting the top-N photos from each of the groups, or across each group.” By repeating the grouping for other clusters, it teaches labeling based on a “second population grouping” within a BRI.. See also fig. 4A. Maarek)
Claim 25 is rejected under 35 U.S.C. 103 as being unpatentable over Maarek in view of Hao and further in view of Szeliski et. al, hereafter Szeliski (US Publication No. 20080028341 A1)
As per claim 25, Maarek in view of Hao already teaches “the computer-implemented method of claim 24, wherein the one or more user signals descriptive of user actions relative to the images in the cluster… on one or more of the images in the cluster”, however Maarek in view of Hao does not teach “comprise user dwell data that indicates an aggregate dwell time of a user on one or more of the images”
Szeliski teaches “comprise user dwell data that indicates an aggregate dwell time of a user on one or more of the images” (Paragraph 32 “For instance, in an embodiment of the present invention, images within a 3D environment may be ranked based on user viewing information, such as the number of users who have viewed a particular image and/or how long users have viewed the image (i.e., a dwell time). Szeliski)
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings of Maarek and Hao with the teachings of Szeliski to include user dwell data. The modification would have been motivated by the desire to have a better ranking or grouping, an improvement, since dwell time is a method of showing user interest as suggested by Szeliski (Paragraph 32 “For instance, in an embodiment of the present invention, images within a 3D environment may be ranked based on user viewing information, such as the number of users who have viewed a particular image and/or how long users have viewed the image (i.e., a dwell time). By ranking images, images that are more representative, pleasing, and/or interesting to users may be identified.” Szeliski).
Claim 38 is rejected under 35 U.S.C. 103 as being unpatentable over Maarek in view of Hao and further in view of Cosman et. al, hereafter Cosman (US Publication No. 20210192078 A1)
As per claim 38, Maarek in view of Hao teaches “The computer-implemented method of claim 31, wherein training, by the one or more computing devices and using a learning technique, the machine-learned model on the training dataset comprises”, however Maarek in view of Hao does not teach “participating in a federated learning framework, and wherein participating in the federated learning framework comprises: training or retraining a local model based at least in part on the training dataset; and providing data descriptive of a model update from training or retraining the local model to a central computing system for aggregation with model updates from other users.”
Cosman teaches ““participating in a federated learning framework, and wherein participating in the federated learning framework comprises:” (See Fig. 4A. “[0018] FIG. 4A is a flow diagram of a method of performing private federated learning using the computing components and privatization techniques described herein;” Cosman)
“training or retraining a local model based at least in part on the training dataset;” (See 402 in Fig. 4A. Cosman)
“and providing data descriptive of a model update from training or retraining the local model to a central computing system for aggregation with model updates from other users.” (See 405 and 406 in Fig. 4A. Cosman)
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings of Maarek and Hao with the teachings of Cosman to use a federated learning framework. The modification would have been motivated by the desire to have better privacy (an improvement) as suggested by Cosman (“[0037] This model of privacy is well suited to federated learning scenarios that use distributed model training. Separated differential privacy enables learning models to be trained in a decentralized setting while providing local privacy guarantees for the transmitted model updates from the devices.” Cosman).
Claim 47 is rejected under 35 U.S.C. 103 as being unpatentable over Maarek in view of Hao and further in view of Yang et. al, hereafter Yang (CN 111242040 A)
As per claim 47, Maarek in view of Hao already teaches “(New) The computer-implemented method of claim 24, wherein the population grouping comprises one of:”, however Maarek in view of Hao does not teach “an upper quartile, a middle quartile, or a lower quartile.”
Yang teaches “an upper quartile, a middle quartile, or a lower quartile.” (See page 8 paragraphs 5-11 “S303: the average similarity in each cluster to order, determining an average similarity quartile and lower quartile corresponding. In one embodiment, determining each face picture in each cluster in the cluster average similarity of other human face picture, the average similarity in each cluster to order, and calculating the upper quartile Q3 and lower quartile Q1 corresponding to the average similarity S3 and S1 order from small to large according to average similarity” “wherein the four-digit also called four-site, is refers to the all numerical values from small to big are arranged and divided into four equal parts, in a numerical value of three division point location in the statistics. It is a group of data sequence after the value on the 25% and 75% positions. quartile is the all data is equally divided into four parts by 3 points data wherein each part comprises 25%, quartile wherein quartile intermediate is the median, usually said is located on 25% position of numerical value (referred to as a lower quartile) and at position 75% of value (called upper quartile).” Yang)
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings of Maarek and Hao with the teachings of Yang to use quartiles as part of the population grouping. The modification would have been motivated by the desire to improve the efficiency of clustering by presenting similarity , therefore it is an improvement, as suggested by Yang (See page 1 abstract “performing static clustering processing for batches of face images to obtain a plurality of clusters comprises neighbour face set, according to each face image in each cluster in the cluster average similarity of other human face picture, for each cluster in the human face picture for screening. selecting multiple face images from each cluster respectively establishing file, according to the neighbour of the file similarity to combine the cluster satisfies the neighbour combining condition, and newly establishing the archives based on cluster after merging, according to the average similarity of the new face image and each file in the human face picture, determining archives the new face image corresponding to the new face images corresponding to adding the file cluster. The solution improves the efficiency of face clustering.” See page 8 paragraphs 5-11 “S303: the average similarity in each cluster to order, determining an average similarity quartile and lower quartile corresponding. In one embodiment… S304: according to the difference of the average similarity upper quartile and lower quartile corresponding to each cluster to obtain the similarity tolerance of each cluster. In one embodiment, after calculating quartile Q3 and lower quartile Q1 corresponding to the average similarity S3 and S1, subtracting and lower average similarity S1 quartile Q1 corresponding to the same cluster in the quartile Q3 corresponding to the average similarity S3, obtain the similarity tolerance tolerance, i.e., tolerance = S3-S1 corresponding to the cluster. repeating the above steps, further calculates the similarity tolerance of each cluster” Yang).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DYLAN J MENDEZ MUNIZ whose telephone number is (703)756-5672. The examiner can normally be reached M-F, 8AM - 5PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Moyer can be reached at (571) 272-9523. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DYLAN JOHN MENDEZ MUNIZ/Examiner, Art Unit 2675
/ANDREW M MOYER/Supervisory Patent Examiner, Art Unit 2675
Prosecution Insights