DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Claims 1-20 are pending under this Office action.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 10, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Cerosaletti, etc. (US 20110075917 A1) in view of Lin, etc. (US 20190244327 A1), further in view of Li, etc. (US 20250378666 A1).
Regarding claim 1, Cerosaletti teaches that a method for obtaining a cover image (See Cerosaletti: Fig. 1, and [0039], “FIG. 1 is a block diagram of a digital camera phone 10 based imaging system that can be used to implement the present invention. The digital camera phone 10 is one type of digital camera. The present invention can also be implemented for use with any other type of digital imaging device, such as other types of digital still camera or digital video cameras, or with any system that receives digital images”), comprising:
obtaining a plurality of first cropped images of an original image corresponding to a candidate resource (See Cerosaletti: Figs. 1-3, and [0007], “disclose a digital camera system which allows a user to revise a captured image relative to a set of editorial suggestions which include cropping and recentering the main subject of the image”; [0047], “The digital processor 100 can also create a low-resolution "thumbnail" size image, as described in commonly-assigned U.S. Pat. No. 5,164,831”; and [0053], “Referring now to FIG. 2, a method is described for determining an aesthetic quality parameter 390 for a media asset 310 according to an embodiment of the present invention. According to this embodiment, a collection of media assets 310 is present and an aesthetic quality parameter 390 is determined for each one. A variety of different person and main subject features (e.g., face location, face size, face contrast, face brightness, location of main subject, and size of main subject) are known to those skilled in the art and can be computed successfully with respect to the media assets 310 in accordance with the present invention. In the FIG. 2 embodiment, a person detector 320 is utilized to find detected people 322 in media assets 310”. Note that the media asset 310 is mapped to the original images, and the best or highest aesthetic score image is mapped to the candidate resource);
obtaining an aesthetic score of each of the plurality of first cropped images (See Cerosaletti: Fig. 2, and [0053], “Referring now to FIG. 2, a method is described for determining an aesthetic quality parameter 390 for a media asset 310 according to an embodiment of the present invention. According to this embodiment, a collection of media assets 310 is present and an aesthetic quality parameter 390 is determined for each one. A variety of different person and main subject features (e.g., face location, face size, face contrast, face brightness, location of main subject, and size of main subject) are known to those skilled in the art and can be computed successfully with respect to the media assets 310 in accordance with the present invention. In the FIG. 2 embodiment, a person detector 320 is utilized to find detected people 322 in media assets 310. Preferably, detected people 322 are found using a face detection algorithm. Methods for detecting human faces are well known in the art of digital image processing. For example, a face detection method for finding human faces in images is described in the article "Robust real-time face detection" by Viola, et al. (Int. Journal of Computer Vision, Vol. 57, pp. 137-154, 2004). This method utilizes an "integral image" representation that consists of the immediate horizontal and vertical sums of pixels above a specific pixel location. Then, the full integral image can be computed as a successive summation over any number of array references. These rectangular features are input to a classifier built using the AdaBoost learning algorithm to select a small number of critical features. Finally, the classifiers are combined in a "cascade" so that the image background regions are discarded so that algorithms can operate only on face-like regions”); and
determining a target cover image of the candidate resource from the plurality of first cropped images based on the aesthetic score of each first cropped image (See Cerosaletti: Fig. 7, and [0079], “In a preferred embodiment of the present invention, the aesthetic quality parameter 390 is a single one-dimensional value, since this allows simpler comparisons between media assets. The resulting aesthetic quality parameters 390 can be associated with the media assets 310 by use of a database or can be stored as metadata in the media asset digital file”; [0098], “As another means to display aesthetic quality values 660, FIG. 7 shows a graph 700 which is a plot of aesthetic quality as a function of time. A curve 720 plotting the aesthetic quality as a function of time shows that aesthetic quality is generally increasing over time. To reduce randomness in the curve 720 the mean aesthetic quality for images within specified time intervals (e.g., months) can be plotted rather than the aesthetic quality for individual images. An indication of the variation in aesthetic quality at selected time intervals can be represented by variation bars 740 which, in this embodiment, show the coefficient of variation every six months. Representative images 760 can also be shown at selected time intervals. This plot of aesthetic quality as a function of time can be created for one particular photographer's media assets, as a composite of any number of photographers' media assets, or as a composite of media assets displayed on an image sharing website or through an online social network”; and Fig. 8 and [0103], “FIG. 8 shows a flowchart of a method for selecting images for sharing based on identifying images that satisfy a threshold aesthetic quality criteria. Initially, aesthetic quality parameters 810 are computed for a media asset collection 800 using the method described above relative to FIG. 2. Next, an asset selector 830 compares the aesthetic quality parameters 810 to a specified aesthetic quality threshold 820 to determine a set of selected media assets 840 having aesthetic quality parameters 810 higher than the aesthetic quality threshold 820. For example, in one embodiment, the aesthetic quality threshold 820 could be an aesthetic quality value of "83." Then, the asset selector 830 will select the media assets in the media asset collection 800 having aesthetic quality parameters 810 larger than "83." Finally, the selected media assets 840 are shared using the image sharer 850. In one embodiment of the present invention, the asset selector 830 places the selected media assets 840 into a holding area such as an image data memory. The image sharer 850 can share the selected media assets 840 using any number of different methods for electronic sharing such as E-mailing them to a particular user or group of users, or uploading the selected media assets 840 to an image sharing website. Image sharing websites include online social networks. Those skilled in the art will recognize other means of sharing images that can be used successfully with this invention”. Note that the aesthetic quality is calculated for each media asset, and comparing with an threshold to select the high quality images (assets) which is mapped the candidate resource).
However, Cerosaletti fails to explicitly disclose that obtaining a plurality of first cropped images of an original image; and determining a target cover image of the candidate resource.
However, Lin teaches that obtaining a plurality of first cropped images of an original image (See Lin: Fig. 5, and [0084], “FIG. 5 illustrates at 500 an example of an image and croppings that may be derived from the image based on composition quality characteristics. In particular, FIG. 5 depicts image 502, and croppings 504, 506, 508. Utilizing a classifier such as that discussed above, the composition quality module 204 may compute composition scores for multiple croppings derived from the image 502. The model to which the multiple croppings are compared may indicate that croppings 504, 506, 508 exhibit features indicative of well-composed photos. For example, the croppings 504, 506, 508 may adhere generally to rules learned from a training data set of photos determined to be well-composed. Although the croppings 504, 506, 508 may adhere to the rules learned from the training set, and thus be considered “well-composed”, each of the croppings 504, 506, 508 is missing person 510, which may be a salient feature of image 502. To ensure salient features of an image are included in croppings, content preservation techniques may be employed”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Cerosaletti to have obtaining a plurality of first cropped images of an original image target as taught by Lin in order to enable adopting a composition quality module to utilize a classifier that is trained to recognize features of well-composed images for determining composition quality of a cropping (See Lin: Fig. 2, and [0077], “The composition quality module 204 represents functionality to determine a composition quality of a respective cropping. To determine a composition quality of a cropping, the composition quality module 204 may utilize a classifier that is trained to recognize features of well-composed images. Rather than hard-coding general cropping rules (e.g., the rule-of-thirds, balancing elements, leading lines, and so on), rules may be mined from a data set of professional photographs that are already determined to be well-composed. Given a data set D of well-composed professional photos, the classifier may extract not only rules (e.g., the rule-of-thirds, balancing elements, leading lines, and so on) from those photos, but also knowledge as to when those rules may be broken. Using a large set of photos that are already determined to be well-composed to train the classifier also has the advantage of being able to avoid using human cropped photos, which are expensive to acquire in terms of both time and resources”). Cerosaletti teaches a method and system that may determine the high-quality (with highest aesthetic score) of the input media assets (plurality of input images); while Lin teaches a system and method that may crop the original input image into a plurality of cropped images and select the best quality cropped image as the cropped image. Therefore, it is obvious to one of ordinary skill in the art to modify Cerosaletti by Lin to crop the original image and evaluate the aesthetic quality of each cropped image (substituting the cropped images as input media assets for aesthetic evaluation). The motivation to modify Cerosaletti by Lin is “Use of known technique to improve similar devices (methods, or products) in the same way”.
However, Cerosaletti, modified by Lin, fails to explicitly disclose that determining a target cover image of the candidate resource.
However, Li teaches that determining a target cover image of the candidate resource (See Li: Figs. 1A-D, and [0237], “At block S1104, the cover image corresponding to the virtual object is generated according to the image of the virtual object shot by the virtual camera”; and [0239], “In addition, in this embodiment, when the virtual object is shot, only the virtual object is rendered, and other objects outside the virtual object are not rendered, such that the shot cover image includes only the virtual object and does not include other objects, thereby making the shot cover image more complete and aesthetically pleasing”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Cerosaletti to have determining a target cover image of the candidate resource as taught by Li in order to generate the cover images of the virtual objects more complete and beautiful (See Li: Figs. 1A-D, and [0240], “In this embodiment, the generation instruction of the cover image of the virtual object is received, the position of the virtual camera for shooting the virtual object is determined; the virtual object is shot according to the position of the virtual camera; and the cover image corresponding to the virtual object is generated according to the image of the virtual object shot by the virtual camera. The cover image shot by this method is more complete and aesthetically pleasing”). Cerosaletti teaches a method and system that may determine the high-quality (with highest aesthetic score) of the input media assets (plurality of input images); while Li teaches a system and method that may adjust the position and orientation of the camera to shoot the object with the most complete and beautiful image as the cover image for the object. Therefore, it is obvious to one of ordinary skill in the art to modify Cerosaletti by Li to use the highest aesthetic score image as the cover image for the media assets. The motivation to modify Cerosaletti by Li is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Regarding claim 2, Cerosaletti, Lin, and Li teach all the features with respect to claim 1 as outlined above. Further, Cerosaletti teaches that the method according to claim 1, wherein obtaining the aesthetic score of each of the plurality of first cropped images comprises:
inputting each first cropped image into an aesthetic scoring model, wherein the aesthetic scoring model comprises a visual encoder and a first large model (See Cerosaletti: Fig. 2, and [0075], “] A structure feature detector 350 is used to detect structure features 352 in the media asset 310. A variety of different structure features such as sharpness, and spatial distribution of edges are known to those skilled in the art and can be computed for media assets 310 in accordance with the present invention. One method of calculating a spatial distribution of edges is described in the aforementioned article "The design of high-level features for photo quality assessment," by Ke, et al. In this method, an edge spatial distribution feature extractor is implemented. A 3.times.3 Laplacian filter with .alpha.=0.2 is applied separately to each of the red, green, and blue channels. Then, the mean is taken across the three channels. The Laplacian image is resized to 100.times.100 pixels and the image sum is normalized to 1 to allow for absolute comparison between images and within groups of media asset content categories such as images with people and images without people. The L.sub.1 statistical metric can be used to calculate the distance between pairs of Laplacian images. Alternatively, the Laplacian image can be used to compute an image structure feature by measuring the amount of area that the edges occupy by computing the area of a bounding box that encloses a certain percentage (e.g., the top 96.04%) of the edge energy”; and [0094], “(2) Using Contingency Tables: This is a sampling and correlation method. Multiple observations of each feature detector are recorded along with information about the emphasis or appeal. These observations are then compiled together to create contingency tables which, when normalized, can then be used. This method is similar to neural network type of training (learning)”. Note that the feature extraction (detectors) modules and the aesthetic score calculation module is mapped to the aesthetic scoring model, and the neural networks with training are mapped to the large model);
obtaining a first image feature by extracting a feature of the each first cropped image via the visual encoder (See Cerosaletti: Fig 2, and [0077], “A capture information extractor 370 is utilized to identify capture features 372 for a media asset 310. The capture information extractor 370 determines the capture features 372 related to the capture of the media asset 310, and outputs the resulting capture features 372. The capture features 372 can include, for example, the time the media asset 310 was captured, the focal length, the subject distance, the magnification, whether the flash was fired, whether the self-timer was activated, and the image resolution. Those skilled in the art will recognize a variety of different possible methods for the capture information extractor 370 to determine capture features 372. Often times, capture features 372 for are embedded in the file header of the media asset 310. For example, EXIF metadata can be used by the media capture device to store information associated with the capture of the media asset 310. For example, the "Date/Time" metadata entry is associated with the date and time the media asset 310 is captured. The capture information extractor 370 uses the most appropriate method for extracting the capture features 372 for the media assets 310. The capture feature of image resolution (i.e., the number of rows and columns of image pixels) is used as capture features 372 in a preferred embodiment of the present invention”); and
obtaining the aesthetic score of the each first cropped image based on the first image feature via the first large model (See Cerosaletti: Fig 2, and [0080], “A simplistic approach that the quality computer 380 can use to determine the aesthetic quality parameter 390 is to simply sum up equally valued or weighted inputs. However, preferably more sophisticated methods are used to determine the aesthetic quality parameter 390. In a particular embodiment, the quality computer 380 is a reasoning engine that has been trained to generate aesthetic quality parameters 390 through a classification process. In this embodiment, a separate model is trained for each composition expressed within the compositional model 336. In the reasoning engine, different input values, identified by respective inputs, can compete or reinforce each other according to knowledge derived from the results of the true aesthetic quality values from human observers-evaluations of real images. Competition and reinforcement are resolved by the inference network of the reasoning engine. A currently preferred reasoning engine is a Bayes net”).
Regarding claim 10, Cerosaletti, Lin, and Li teach all the features with respect to claim 1 as outlined above. Further, Cerosaletti teaches that the method according to claim 1, wherein determining the target cover image of the candidate resource from the plurality of first cropped images based on the aesthetic score of each first cropped image comprises:
determining a plurality of candidate cover images of the candidate resource from the plurality of first cropped images based on the aesthetic score of each first cropped image (See Cerosaletti: Fig. 2, and [0053], “Referring now to FIG. 2, a method is described for determining an aesthetic quality parameter 390 for a media asset 310 according to an embodiment of the present invention. According to this embodiment, a collection of media assets 310 is present and an aesthetic quality parameter 390 is determined for each one. A variety of different person and main subject features (e.g., face location, face size, face contrast, face brightness, location of main subject, and size of main subject) are known to those skilled in the art and can be computed successfully with respect to the media assets 310 in accordance with the present invention. In the FIG. 2 embodiment, a person detector 320 is utilized to find detected people 322 in media assets 310. Preferably, detected people 322 are found using a face detection algorithm. Methods for detecting human faces are well known in the art of digital image processing. For example, a face detection method for finding human faces in images is described in the article "Robust real-time face detection" by Viola, et al. (Int. Journal of Computer Vision, Vol. 57, pp. 137-154, 2004). This method utilizes an "integral image" representation that consists of the immediate horizontal and vertical sums of pixels above a specific pixel location. Then, the full integral image can be computed as a successive summation over any number of array references. These rectangular features are input to a classifier built using the AdaBoost learning algorithm to select a small number of critical features. Finally, the classifiers are combined in a "cascade" so that the image background regions are discarded so that algorithms can operate only on face-like regions”); and
determining the target cover image from the plurality of candidate cover images based on feedback data provided by a user group on the plurality of candidate cover images (See Cerosaletti: Fig. 2, and [0105], “When an input digital image is captured using a digital camera, feedback can be provided to the photographer regarding the aesthetic quality of the input digital image by displaying a quality indicator on a user interface of the digital camera. Continuous feedback in the form of a quality indicator could also be given while the photographer is framing the digital image such that the photographer could dynamically modify the framing to create the most aesthetically pleasing image”).
Regarding claim 19, Cerosaletti, Lin, and Li teach all the features with respect to claim 1 as outlined above. Further, Cerosaletti, Lin, and Li teach that an apparatus for obtaining a cover image (See Cerosaletti: Fig. 1, and [0039], “FIG. 1 is a block diagram of a digital camera phone 10 based imaging system that can be used to implement the present invention. The digital camera phone 10 is one type of digital camera. The present invention can also be implemented for use with any other type of digital imaging device, such as other types of digital still camera or digital video cameras, or with any system that receives digital images”), comprising:
at least one processor (See Cerosaletti: Fig. 1, and [0043], “] The analog output signals from the image sensor array 40 are amplified and converted to digital data by the analog-to-digital (A/D) converter 80 on the CMOS sensor 50. The digital data is stored in a DRAM buffer memory 90 and subsequently processed by a digital processor 100 controlled by the firmware stored in firmware memory 110, which can be flash EPROM memory. The digital processor 100 includes a real-time clock 120, which keeps the date and time even when the digital camera phone 10 and digital processor 100 are in their low power state”); and
a memory communicatively coupled to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions causes the at least one processor (See Cerosaletti: Fig. 1, and [0044], “The processed digital image files are stored in the image/data memory 20. The image/data memory 20 can also be used to store aesthetic quality parameters determined using the method of the present invention. The image/data memory 20 can also store other types of data, such as photographer id, image of the photographer, rankings of photographers, and phone numbers”) to:
obtain a plurality of first cropped images of an original image (See Lin: Fig. 5, and [0084], “FIG. 5 illustrates at 500 an example of an image and croppings that may be derived from the image based on composition quality characteristics. In particular, FIG. 5 depicts image 502, and croppings 504, 506, 508. Utilizing a classifier such as that discussed above, the composition quality module 204 may compute composition scores for multiple croppings derived from the image 502. The model to which the multiple croppings are compared may indicate that croppings 504, 506, 508 exhibit features indicative of well-composed photos. For example, the croppings 504, 506, 508 may adhere generally to rules learned from a training data set of photos determined to be well-composed. Although the croppings 504, 506, 508 may adhere to the rules learned from the training set, and thus be considered “well-composed”, each of the croppings 504, 506, 508 is missing person 510, which may be a salient feature of image 502. To ensure salient features of an image are included in croppings, content preservation techniques may be employed”) corresponding to a candidate resource (See Cerosaletti: Figs. 1-3, and [0007], “disclose a digital camera system which allows a user to revise a captured image relative to a set of editorial suggestions which include cropping and recentering the main subject of the image”; [0047], “The digital processor 100 can also create a low-resolution "thumbnail" size image, as described in commonly-assigned U.S. Pat. No. 5,164,831”; and [0053], “Referring now to FIG. 2, a method is described for determining an aesthetic quality parameter 390 for a media asset 310 according to an embodiment of the present invention. According to this embodiment, a collection of media assets 310 is present and an aesthetic quality parameter 390 is determined for each one. A variety of different person and main subject features (e.g., face location, face size, face contrast, face brightness, location of main subject, and size of main subject) are known to those skilled in the art and can be computed successfully with respect to the media assets 310 in accordance with the present invention. In the FIG. 2 embodiment, a person detector 320 is utilized to find detected people 322 in media assets 310”. Note that the media asset 310 is mapped to the original images, and the best or highest aesthetic score image is mapped to the candidate resource);
obtain an aesthetic score of each of the plurality of first cropped images (See Cerosaletti: Fig. 2, and [0053], “Referring now to FIG. 2, a method is described for determining an aesthetic quality parameter 390 for a media asset 310 according to an embodiment of the present invention. According to this embodiment, a collection of media assets 310 is present and an aesthetic quality parameter 390 is determined for each one. A variety of different person and main subject features (e.g., face location, face size, face contrast, face brightness, location of main subject, and size of main subject) are known to those skilled in the art and can be computed successfully with respect to the media assets 310 in accordance with the present invention. In the FIG. 2 embodiment, a person detector 320 is utilized to find detected people 322 in media assets 310. Preferably, detected people 322 are found using a face detection algorithm. Methods for detecting human faces are well known in the art of digital image processing. For example, a face detection method for finding human faces in images is described in the article "Robust real-time face detection" by Viola, et al. (Int. Journal of Computer Vision, Vol. 57, pp. 137-154, 2004). This method utilizes an "integral image" representation that consists of the immediate horizontal and vertical sums of pixels above a specific pixel location. Then, the full integral image can be computed as a successive summation over any number of array references. These rectangular features are input to a classifier built using the AdaBoost learning algorithm to select a small number of critical features. Finally, the classifiers are combined in a "cascade" so that the image background regions are discarded so that algorithms can operate only on face-like regions”); and
determine a target cover image of the candidate resource (See Li: Figs. 1A-D, and [0237], “At block S1104, the cover image corresponding to the virtual object is generated according to the image of the virtual object shot by the virtual camera”; and [0239], “In addition, in this embodiment, when the virtual object is shot, only the virtual object is rendered, and other objects outside the virtual object are not rendered, such that the shot cover image includes only the virtual object and does not include other objects, thereby making the shot cover image more complete and aesthetically pleasing”) from the plurality of first cropped images based on the aesthetic score of each first cropped image (See Cerosaletti: Fig. 7, and [0079], “In a preferred embodiment of the present invention, the aesthetic quality parameter 390 is a single one-dimensional value, since this allows simpler comparisons between media assets. The resulting aesthetic quality parameters 390 can be associated with the media assets 310 by use of a database or can be stored as metadata in the media asset digital file”; [0098], “As another means to display aesthetic quality values 660, FIG. 7 shows a graph 700 which is a plot of aesthetic quality as a function of time. A curve 720 plotting the aesthetic quality as a function of time shows that aesthetic quality is generally increasing over time. To reduce randomness in the curve 720 the mean aesthetic quality for images within specified time intervals (e.g., months) can be plotted rather than the aesthetic quality for individual images. An indication of the variation in aesthetic quality at selected time intervals can be represented by variation bars 740 which, in this embodiment, show the coefficient of variation every six months. Representative images 760 can also be shown at selected time intervals. This plot of aesthetic quality as a function of time can be created for one particular photographer's media assets, as a composite of any number of photographers' media assets, or as a composite of media assets displayed on an image sharing website or through an online social network”; and Fig. 8 and [0103], “FIG. 8 shows a flowchart of a method for selecting images for sharing based on identifying images that satisfy a threshold aesthetic quality criteria. Initially, aesthetic quality parameters 810 are computed for a media asset collection 800 using the method described above relative to FIG. 2. Next, an asset selector 830 compares the aesthetic quality parameters 810 to a specified aesthetic quality threshold 820 to determine a set of selected media assets 840 having aesthetic quality parameters 810 higher than the aesthetic quality threshold 820. For example, in one embodiment, the aesthetic quality threshold 820 could be an aesthetic quality value of "83." Then, the asset selector 830 will select the media assets in the media asset collection 800 having aesthetic quality parameters 810 larger than "83." Finally, the selected media assets 840 are shared using the image sharer 850. In one embodiment of the present invention, the asset selector 830 places the selected media assets 840 into a holding area such as an image data memory. The image sharer 850 can share the selected media assets 840 using any number of different methods for electronic sharing such as E-mailing them to a particular user or group of users, or uploading the selected media assets 840 to an image sharing website. Image sharing websites include online social networks. Those skilled in the art will recognize other means of sharing images that can be used successfully with this invention”. Note that the aesthetic quality is calculated for each media asset, and comparing with an threshold to select the high quality images (assets) which is mapped the candidate resource).
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Cerosaletti, etc. (US 20110075917 A1) in view of Lin, etc. (US 20190244327 A1), further in view of Li, etc. (US 20250378666 A1) and McMichael, etc. (US 20160259494 A1).
Regarding claim 13, Cerosaletti, Lin, and Li teach all the features with respect to claim 1 as outlined above. However, Cerosaletti, modified by Lin and Li, fails to explicitly disclose that the method according to claim 1, further comprising: identifying a first probability that the target cover image falls into a preference style of a user group; and determining a recommended resource corresponding to the user group from a plurality of candidate resources based on the first probability.
However, McMichael teaches that the method according to claim 1, further comprising: identifying a first probability that the target cover image falls into a preference style of a user group (See McMichael: Figs. 5-6, and [0010], “According to a preferred embodiment of the invention, a system for controlling video thumbnail images, comprising an analytics engine comprising at least a plurality of programming instructions stored in a memory operating on a network-connected computing device and adapted to collect at least a plurality of statistics related to at least a user behavior regarding a plurality of electronic media content, the media content comprising at least a video clip; and a distribution engine comprising at least a plurality of programming instructions stored in a memory operating on a network-connected computing device and adapted to receive at least a plurality of electronic media content and determine an optimal distribution of content to a plurality of network-connected user devices, is disclosed”; and [0047], “FIG. 5 illustrates an exemplary computer system 500 that utilizes machine learning to select thumbnails according to an embodiment of the invention. According to the embodiment, a plurality of image frames 501 comprising still imagery may be associated with a plurality of video portions 502 that may comprise segments or clips of video-based media content. Associated groupings of video and image content may be processed by a distribution engine 520 that may comprise at least a plurality of programming instructions stored in a memory and adapted to receive media content and determine an optimal distribution of content to end user devices 504 or network endpoints 503 that may be considered endpoints of a media content network (that is, network-connected devices that may receive media content or other network communication), where media may be presented for viewing but is no longer processed or changed. In this manner, network endpoints may be considered devices for content consumption, whereas content modification or creation functions are performed elsewhere according to the embodiment). Distribution engine 520 may perform functions such as selecting content for presentation to users based on known demographic or analytics information, such information being collected and provided by an analytics engine 510 that may comprise at least a plurality of programming instructions stored in a memory and adapted to collect statistics, demographic information, network statistics, device information, behavior tracking information, or other such measureable or quantifiable information that may be relevant to user devices 504, network endpoints 503, or user behaviors while interacting with a network (such as, for example, what media a user views or other behavior tracking information). For example, an analytics engine may track user behavior through device-based or software-based means, such as location or hardware device monitoring or browser “cookies” that may track a user's web browsing behavior or preferences”); and
determining a recommended resource corresponding to the user group from a plurality of candidate resources based on the first probability (See McMichael: Figs. 5-6, and [0048], “FIG. 6 is an illustration of an exemplary frame selection process 600 for selecting a portion of frames from a video clip for segmenting, according to an embodiment of the invention. According to the embodiment, a video clip may comprise a plurality of still image frames 610a-n, for example a 30-second clip comprising 900 still image frames as illustrated. Still frames 610a-n may be grouped or segmented into divided portions, for example a quantity of 900 frames may be grouped into 6 equal segments comprising 150 frames each. It should be appreciated that while a specific segmenting arrangement is illustrated for clarity, such an arrangement is intended to be exemplary and segments need not necessarily be equal in distribution, for example unequal segments of 600, 200, 75, and 25 still frames each might be utilized when segmenting a quantity of 900 frames, and it should be further appreciated that any other such arrangement or distribution may be utilized according to the embodiment. A particular segment 611 may be further divided into smaller segments 612a-n, for example a group of four smaller segments “3a” through “3d” may each comprise approximately ¼ of a selected video segment as shown. Again, it should be appreciated that while an equal distribution of frames into smaller segments 612a-n is shown, such an arrangement is exemplary and alternate arrangements or distributions of frames may be utilized according to the embodiment. In this manner, arbitrary video portions may be selected and divided into discrete segments or still frames for use according to the invention”; and 0060], “In an exemplary use case, a one-second portion of video may contain 30 still images (referred to as 30 frames per second, a common video frame rate). Each frame in the exemplary video may have the potential to be the “most appealing” to a specific market segment or region on a given target device. The “appeal” of each frame may be quantified according to user interactions such as video plays (the number of times a user or market segment views the video), the number of plays within a given time frame, plays from a specific network on which the video is being hosted or presented, within an app such as a game or native advertising to a particular device such as a user's phone, or part of published content supporting video. In this manner it may be appreciated that a wide variety of metrics may be monitored for use in determining which thumbnail may be ideal for a given video segment, and in what context the selected thumbnail may be ideal. For example, a thumbnail that appeals to one market segment may not have the same appeal to another market segment (such as appealing to specific demographic groups), or appeal on one network may not be reflected on other networks”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Cerosaletti to have the method according to claim 1, further comprising: identifying a first probability that the target cover image falls into a preference style of a user group; and determining a recommended resource corresponding to the user group from a plurality of candidate resources based on the first probability as taught by McMichael in order to optimize video and thumbnails to generate user behavior or traffic in simple and easy manner so as to review behavior patterns or other monitored statistics related to the content in an efficient manner (See McMichael: Fig. 9, and [0051], “In this manner, a user may be provided with the means to optimize their video and thumbnails to generate user behavior or traffic, and they may be provided with a simple and easily understood interface for reviewing behavior patterns or other monitored statistics related to their content, so they may make more informed and effective decisions regarding the configuration of their video and thumbnail content”). Cerosaletti teaches a method and system that may determine the high-quality (with highest aesthetic score) of the input media assets (plurality of input images); while McMichael teaches a system and method that may control video thumbnail images using the system of the invention based on the statistical behaviors of the user. Therefore, it is obvious to one of ordinary skill in the art to modify Cerosaletti by McMichael to generate the cover image based on the statistics of the user behaviours. The motivation to modify Cerosaletti by McMichael is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Claims 16-17 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Zeng, etc. (US 20250022136 A1) in view of Cerosaletti, etc. (US 20110075917 A1).
Regarding claim 16, Zeng teach a method for training an image scoring model (See Zeng: Fig. 10, and [0141], “Reference is made to FIG. 10 below, which is a schematic diagram of a structure of an electronic device (such as a terminal device or a server in FIG. 10) 1000 suitable for implementing an embodiment of the present disclosure. The terminal device in this embodiment of the present disclosure may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a personal digital assistant (PDA), a portable android device (PAD), a portable media player (PMP), and a vehicle-mounted terminal (such as a vehicle navigation terminal), and a fixed terminal such as a television (TV) and a desktop computer. The electronic device 1000 shown in FIG. 10 is merely an example, and shall not impose any limitation on the function and scope of use of the embodiments of the present disclosure”; and [0030], “the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the image cropping method described above, or to implement the model training method described above”), comprising:
obtaining a reference cropped image and a sample cropped image of a sample image (See Zeng: Fig. 1, and [0049], “S110: Segment an image to be cropped to obtain a first segmented image, and determine a bounding box of a target object in the image to be cropped based on the first segmented image”; [0047], “The digital processor 100 can also create a low-resolution "thumbnail" size image, as described in commonly-assigned U.S. Pat. No. 5,164,831”; and [0055], “S120: Generate a plurality of first candidate boxes within the bounding box, and select a first target box from the plurality of first candidate boxes based on an aesthetic score of a first feature map corresponding to each first candidate box”. Note that the a bounding box of a target object in the image to be cropped is mapped to a reference cropped image, and the cropped image from the plurality of first candidate boxes is mapped to a sample cropped image of a sample image);
obtaining a coincidence parameter between each of the plurality of sample cropped images and the reference cropped image (See Zeng: Fig. 1, and [0060], “Compared with the conventional image cropping solution where candidate boxes are generated in a global image, the number of the candidate boxes can be greatly reduced in this embodiment by first determining the bounding box of the target object and then generating the plurality of candidate boxes within the bounding box. It has been verified by experiments that the number of the candidate boxes can be reduced by 10 to 20 times. This embodiment not only reduces the extraction time and storage space of the corresponding features of the candidate boxes, but also reduces the time for performing aesthetic scoring by the aesthetic scoring model, such that the real-time performance of image cropping can be improved. Furthermore, the preference of the aesthetic scoring model for a salient object in the image to be cropped may be enhanced when the bounding box is determined based on the salient object. In addition, generating the candidate boxes within the bounding box can avoid cropping an object in a wrong position”. Note that the plurality of candidate boxes within the bounding box is mapped to a coincidence parameter);
obtaining a slicing detection result of the sample cropped image by performing object slicing detection on the sample cropped image (See Zeng: Fig. 1, and [0056], “In this embodiment of the present disclosure, the first candidate boxes may be represented as candidate cropping ranges determined for completing image cropping, and the first target box may be represented as a final cropping range determined for completing image cropping. The plurality of first candidate boxes may be generated in a sliding window manner within the bounding box in the image to be cropped, and feature extraction is performed on the image in each first candidate box to obtain a corresponding first feature map. Each first feature map may be input into a pre-trained aesthetic scoring model such that the aesthetic scoring model outputs the aesthetic score of the first feature map”; and [0151], “segment an image to be cropped to obtain a first segmented image, and determine a bounding box of a target object in the image to be cropped based on the first segmented image; generate a plurality of first candidate boxes within the bounding box, and select a first target box from the plurality of first candidate boxes based on an aesthetic score of a first feature map corresponding to each first candidate box; and use an image located within the first target box in the image to be cropped as a cropping result”);
obtaining a sample aesthetic score of the sample cropped image (See Zeng: Fig. 1, and [0056], “In this embodiment of the present disclosure, the first candidate boxes may be represented as candidate cropping ranges determined for completing image cropping, and the first target box may be represented as a final cropping range determined for completing image cropping. The plurality of first candidate boxes may be generated in a sliding window manner within the bounding box in the image to be cropped, and feature extraction is performed on the image in each first candidate box to obtain a corresponding first feature map. Each first feature map may be input into a pre-trained aesthetic scoring model such that the aesthetic scoring model outputs the aesthetic score of the first feature map”. Note that the first feature aesthetic score is mapped to the sample aesthetic score);
obtaining a sample target score of the sample cropped image based on at least one of the coincidence parameter, the slicing detection result, or the sample aesthetic score (See Zeng: Fig. 1, and [0057], “The first candidate box corresponding to the highest aesthetic score may be determined as the first target box. Alternatively, first candidate boxes corresponding to the top N high aesthetic scores may be presented, and the user may be prompted to select a desired cropping range. Further, the first candidate box corresponding to the cropping range selected by the user may be determined as the first target box, where N may be an integer greater than or equal to 1”); and
training an image scoring model (See Zeng: Figs. 1-2, and [0186], “the trained segmentation model is used for determining the first segmented image described in the image cropping method according to any one of claims 1 to 6; and the trained aesthetic scoring model is used for determining the aesthetic score described in the image cropping method according to any one of claims 1 to 6”; and [0030], “the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the image cropping method described above, or to implement the model training method described above”) based on the sample cropped image and the sample target score.
However, Zeng fails to explicitly disclose that (training an image scoring model ) based on the sample cropped image and the sample target score.
However, Cerosaletti teaches that (training an image scoring model ) based on the sample cropped image and the sample target score (See Cerosaletti: Fig. 2, and [0065], “The image composition is represented as a set of numbers such that person features 326, such as face size and face location, and vanishing point locations 332 that can be directly used by the compositional modeler 334 to create a compositional model 336. Alternatively, the compositional modeler 334 can transform the person features 326, the vanishing point locations 332, or both, into descriptive categories as a preliminary step in the creation of the compositional model 336. For example, the vanishing point locations 332 can be mapped to a set of vanishing point location categories such as horizontal, vertical, up, down, and center”;p and [0081], “The true aesthetic quality values are gathered from human observers-evaluations of real images. By using empirical data collection methods, a psychometric experiment can be conducted in which human observers evaluate a variety of different images. For example, images can be rated using a 0 to 100-point scale bi-anchored with "lowest imaginable" and "highest imaginable" for aesthetically pleasing. The aforementioned features can then computed for all of these images. The image ratings are considered true aesthetic quality values and can then be provided as a training data set to the reasoning engine. The image ratings can also be clustered by the patterns of human observer responses utilizing techniques such as k-means clustering as described by Duda et al. in "Pattern Classification" (John Wiley and Sons, New York, 2001). These clusters can also be provided as a training data set to the reasoning engine”. Note that the compositional model for aesthetic score model is mapped to (training the image scoring model) based on sample cropped image and the sample target score).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Zeng to have (training an image scoring model ) based on the sample cropped image and the sample target score as taught by Cerosaletti in order to enable determined esthetic quality parameters to be used to measure photographer's progress over time toward producing images with high level of esthetic quality (See Cerosaletti: Fig. 2, and [0017], “The present invention has the advantage that improved aesthetic quality parameters are determined by using a compositional model that includes vanishing point locations”). Zeng teaches a method and system that may crop the input images and train the aesthetic scoring model using the cropped images; while Cerosaletti teaches a system and method that may train and generate compositional image aesthetic scoring model based on various features of the input images and quality score of the images. Therefore, it is obvious to one of ordinary skill in the art to modify Zeng by Cerosaletti to use image features and quality scores to train and generate the image aesthetic scoring model. The motivation to modify Zeng by Cerosaletti is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Regarding claim 17, Zeng and Cerosaletti teach all the features with respect to claim 16 as outlined above. Further, Zeng teaches that the method according to claim 16, wherein obtaining the coincidence parameter between each of the plurality of sample cropped images and the reference cropped image comprises:
obtaining an intersection over union (IoU) between a crop box of the sample cropped image and a crop box of the reference cropped image, and taking the IoU as the coincidence parameter (See Zeng: Fig. 1, and [0056], “In this embodiment of the present disclosure, the first candidate boxes may be represented as candidate cropping ranges determined for completing image cropping, and the first target box may be represented as a final cropping range determined for completing image cropping. The plurality of first candidate boxes may be generated in a sliding window manner within the bounding box in the image to be cropped, and feature extraction is performed on the image in each first candidate box to obtain a corresponding first feature map. Each first feature map may be input into a pre-trained aesthetic scoring model such that the aesthetic scoring model outputs the aesthetic score of the first feature map”).
Regarding claim 20, Cerosaletti, Lin, and Li teach all the features with respect to claim 1 as outlined above. Further, Cerosaletti and Zeng teach that an apparatus for training an image scoring model, comprising:
at least one processor (See Cerosaletti: Fig. 1, and [0043], “] The analog output signals from the image sensor array 40 are amplified and converted to digital data by the analog-to-digital (A/D) converter 80 on the CMOS sensor 50. The digital data is stored in a DRAM buffer memory 90 and subsequently processed by a digital processor 100 controlled by the firmware stored in firmware memory 110, which can be flash EPROM memory. The digital processor 100 includes a real-time clock 120, which keeps the date and time even when the digital camera phone 10 and digital processor 100 are in their low power state”); and
a memory communicatively coupled to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions causes the at least one processor to implement the method of claim 16 (See Zeng: Fig. 10, and [0142], “As shown in FIG. 10, the electronic device 1000 may include a processing apparatus (e.g., a central processor, a graphics processor) 1001 that may perform a variety of appropriate actions and processing in accordance with a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage apparatus 1008 into a random access memory (RAM) 1003. The RAM 1003 further stores various programs and data required for the operation of the electronic device 1000. The processing apparatus 1001, the ROM 1002, and the RAM 1003 are connected to each other through a bus 1004. An input/output (I/O) interface 1005 is also connected to the bus 1004”).
Allowable Subject Matter
Claims 3-9 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The best arts search do not teach the limitations of “the method according to claim 1, wherein obtaining the plurality of first cropped images of the original image corresponding to the candidate resource comprises: obtaining a plurality of second cropped images of the original image; obtaining a target score of each of the plurality of second cropped images; and determining the first cropped images from the plurality of second cropped images based on the target score of each second cropped image.”
Claims 11-12 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The best arts search do not teach the limitations of “the method according to claim 10, wherein determining the target cover image from the plurality of candidate cover images based on the feedback data provided by the user group on the plurality of candidate cover images comprises: obtaining an average reward of each of the plurality of candidate cover images based on feedback data corresponding to each candidate cover image; obtaining an upper confidence bound score of the candidate cover image based on an exploration parameter, the average reward and a selection times of the candidate cover image; and taking a candidate cover image corresponding to a maximum upper confidence bound score as the target cover image.”
Claims 14-15 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The best arts search do not teach the limitations of “the method according to claim 13, wherein identifying the first probability that the target cover image falls into the preference style of the user group comprises: inputting the target cover image into a style recognition model, wherein the style recognition model comprises a second large model and a classification model; obtaining a second probability of the target cover image under each of a plurality of style dimensions via the second large model; and obtaining the first probability based on the second probability of the target cover image under each of the plurality of style dimensions via the classification model.”
Claim 18 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The best arts search do not teach the limitations of “the method according to claim 16, wherein a plurality of sample cropped images comprise a plurality of sample horizontal cropped images and a plurality of sample vertical cropped images, and the reference cropped image comprises a reference horizontal cropped image and a reference vertical cropped image, wherein obtaining the coincidence parameter between each of the plurality of sample cropped images and the reference cropped image comprises: obtaining a coincidence parameter between each of the plurality of sample horizontal cropped images and the reference horizontal cropped image, and taking the coincidence parameter as a coincidence parameter corresponding to each sample horizontal cropped image; and obtaining a coincidence parameter between each of the plurality of sample vertical cropped images and the reference vertical cropped image, and taking the coincidence parameter as a coincidence parameter corresponding to each sample vertical cropped image.”
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GORDON G LIU whose telephone number is (571)270-0382. The examiner can normally be reached Monday - Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona E Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GORDON G LIU/ Primary Examiner, Art Unit 2618
Prosecution Insights