DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant’s amendments filed on 07/09/2026 to the specification and claims have overcome specification objection, claim rejections under 35 U.S.C. 112(b), claim rejections under 35 U.S.C. 101, and prior art rejections as preciously set forth in the Non-Final Rejection Office Action mailed on 04/09/2026.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 4-7 and 34-35 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tsuji et al (WO 2019171803 A1), hereinafter Tsuji in view of Li et al (US 20190286892 A1), hereinafter Li, and further in view of Watanabe et al (JP 2019091138 A), hereinafter Watanabe.
-Regarding claim 1, Tsuji discloses an image processing system comprising (Abstract; FIGS. 1-8): at least one memory storing instructions, and at least one processor configured to execute the instructions stored in the at least one memory to (FIGS. 1-2; Page 4, 4th paragraph, ”CPU (processor), memory …”; Page 5, 3rd paragraph); acquire an estimation result of estimating a posture of a person (a posture of a person has to be estimated in order to perform comparison of subject posture and posing in Tsuji’s FIG. 3 and FIG. 5 (see also page 7, 3rd paragraph and page 9, 2nd paragraph)) included in a first image and a person included in a second image (FIG. 3; Page 3, 4th paragraph, “recognizes elements appearing in the image”, 5th paragraph, “recognize elements such as a person, an object, …”; Page 6, 1st paragraph, “… human recognition … determines the category of the image based on the element of the image … Subject (person) attributes: "Female", "Couple", "Baby", "Businessman" … may be classified into a plurality of categories …”; Page 7, 3rd paragraph, “comparison of subject posture and posing”; FIG. 5, step 521; Page 8, “performs various analysis processes such as human recognition, general object recognition”; Page 9, 2nd paragraph, “similar person's posture and posing”); acquire a recognition result of recognizing an object, other than the persons (FIG. 3; Page 3, 4th paragraph, “recognizes elements appearing in the image”, 5th paragraph, “recognize elements such as a person, an object, …”; Page 6, 1st paragraph, “… object recognition … Subject (general object) ... "car", "mountain", "cherry blossom", "viewer", "temple"…”; FIG. 5, step 521; Page 8, “performs various analysis processes such as human recognition, general object recognition”), included in the first image and an object included in the second image (FIGS 1-2); and perform a similarity determination of the similarity of the first image to the second image based on the estimation results of the postures of the persons and the recognition results of the objects (Abstract; FIG. 1; FIG. 2, units 101-106; FIGS. 5-8; Page 3, 4th paragraph, “Using these recognition results, … selects an image matching the search condition from the image database 100 (first selection) and then selects an SNS from the selected image group (first image group)… are highly evaluated in the above and have a low similarity to other images in the same category are selected (second selection) …”; Page 4, 2nd and 3rd paragraphs; Page 7, 1st -4th paragraphs, “… similarity … comparison of feature amounts (color, brightness, etc.), scene comparison, comparison of subject posture and posing”).
Tsuji does not disclose performing the similarity determination based on a degree of similarity between relationship feature values related to relationships between the persons and the objects.
In the same field of endeavor, Li teaches a method to identify human-object interactions in an image content (Li: Abstract; FIGS. 1-7). Li further teaches performing the similarity determination based on a degree of similarity between relationship feature values related to relationships between the persons and the objects (Li: FIGS. 1-3, 6-7; [0003], “person is riding a bicycle … most common relationship between the two objects … identify interactions based on the person and object identified … compare the current image to other images of the same action … a person is next to a bicycle, riding a bicycle and washing a bicycle may both be possible interactions between the two objects … evaluate the image for similarities to other images involving riding and washing”; [0026], “… human-object interaction metadata …”; [0048]; [0050], “joint-location heat map … determine whether the joint location and the object contact point are in sufficiently similar locations”; [0051]; [0053], “… match score based on the degree of match between the search terms and the information contained in searchable fields and the human object interaction metadata …”; [0056]-[0057]).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of Tsuji with the teaching of Li by using the relationship between the subjects in order to achieve robust and accurate similarity determination.
Tsuji in view of Li does not teach performing the similarity determination based on weights of the degrees of similarity. However, this is a common practice in the field (See Zhang (CN 109858308 A)).
Watanabe is an analogous art pertinent to the problem to be solved in this application and teaches an image retrieving method using pose information of a person itself as an image retrieval query (Watanabe: FIGS. 1-20). Watanabe further teaches performing the similarity determination based on weights of the degrees of similarity (Watanabe: Page 11, 5th paragraph, “weighting may be performed according to the degree of similarity”; Page 12, 5th paragraph, “searches for a similar image … image feature amount distance and the posture feature amount distance are integrated … normalize or weight the distances”).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify the teaching of Tsuji in view of Li with the teaching of Watanabe by performing the similarity determination based on weights of the degrees of similarity in order to achieve higher reliability of similarity determination.
-Regarding claim 2, Tsuji in view of Li, and further in view of Watanabe teaches the image processing system of claim 1. Tsuji further discloses performing the similarity determination based on a degree of similarity between posture feature values that are based on the estimation results of the postures of the persons and a degree of similarity between object feature values that are based on the recognition results of the objects (Tsuji: Abstract; FIG. 1; FIG. 2, units 101-106; FIGS. 5-8; Page 3, 4th paragraph; Page 4, 2nd and 3rd paragraphs, “having a high score … calculated by a predetermined evaluation formula”; Page 6, last paragraph – Page 7, 4th paragraph).
-Regarding claim 4, Tsuji in view of Li, and further in view of Watanabe teaches the image processing system of claim 1.
Tsuji in view of Li does not teaches performing the similarity determination based on reliabilities of the subject estimation.
However, Watanabe is an analogous art pertinent to the problem to be solved in this application and teaches an image retrieving method using pose information of a person itself as an image retrieval query (Watanabe: FIGS. 1-20). Watanabe further teaches performing the similarity determination based on reliabilities of the subject estimation (Watanabe: Page 3, 2nd paragraph, The feature points are detected … and have information on coordinates in the image and reliability … a value indicating the probability that the corresponding feature point exists in the detected coordinates”; Page 4, 3rd paragraph, “The posture information … represented by coordinates in the image and a numerical value of reliability”).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify the teaching of Tsuji in view of Li with the teaching of Watanabe by performing the similarity determination based on reliabilities of the subject estimation in order to achieve robust similarity determination.
-Regarding claim 5, Tsuji in view of Li, and further in view of Watanabe teaches the image processing system of claim 1.
Tsuji in view of Li does not teaches performing similarity determination based on a change in the subjects.
However, Watanabe is an analogous art pertinent to the problem to be solved in this application and teaches an image retrieving method using pose information of a person itself as an image retrieval query (Watanabe: FIGS. 1-20). Watanabe further teaches performing similarity determination based on a change in the subjects (Watanabe: Page 4, 3rd paragraph, “extracted from … the moving image data …”; Page 17, last paragraph, “extracts the coordinates of the corresponding feature point from the plurality of pieces of posture information arranged in time series, and generates a flow line …”; FIG. 18). Watanabe also teaches the images including plurality of images in a chronologically consecutive order (Watanabe: Page 3, 1st paragraph, “surveillance video analysis application: … search for images and videos of people who are taking a specific pose”; Page 17, 4th paragraph, “consecutive frames”, last paragraph, “time series”).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify the teaching of Tsuji in view of Li with the teaching of Watanabe by performing the similarity determination based on a change in the subjects in order to achieve robust similarity determination.
-Regarding claim 6, Tsuji in view of Li, and further in view of Watanabe teaches the image processing system of claim 1.
Tsuji does not disclose performing the similarity determination based on relationships between the persons and the objects, the relationships being based on the estimation results of the postures of the persons and the recognition results of the objects.
In the same field of endeavor, Li teaches a method to identify human-object interactions in an image content (Li: Abstract; FIGS. 1-7). Li further teaches performing the similarity determination based on relationships between the persons and the objects, the relationships being based on the estimation results of the postures of the persons and the recognition results of the objects (Li: FIGS. 1, network 106, subnets 108, 110, 112, metadata 114; FIG. 2).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of Tsuji with the teaching of Li by using the relationship between the subjects in order to achieve robust and accurate similarity determination.
-Regarding claim 7, Tsuji in view of Li, and further in view of Watanabe teaches the image processing system of claim 6.
Tsuji does not disclose performing the similarity determination based on a degree of similarity between relationship feature values that are based on the relationships between the persons and the objects.
In the same field of endeavor, Li teaches a method to identify human-object interactions in an image content (Li: Abstract; FIGS. 1-7). Li further teaches performing the similarity determination based on a degree of similarity between relationship feature values that are based on the relationships between the persons and the objects (Li: FIGS. 1-3, 6-7; [0003], “person is riding a bicycle … most common relationship between the two objects … identify interactions based on the person and object identified … compare the current image to other images of the same action … a person is next to a bicycle, riding a bicycle and washing a bicycle may both be possible interactions between the two objects … evaluate the image for similarities to other images involving riding and washing”; [0026], “… human-object interaction metadata …”; [0048]; [0050], “joint-location heat map … determine whether the joint location and the object contact point are in sufficiently similar locations”; [0051]; [0053], “… match score based on the degree of match between the search terms and the information contained in searchable fields and the human object interaction metadata …”; [0056]-[0057]).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of Tsuji with the teaching of Li by using the relationship between the subjects in order to achieve robust and accurate similarity determination.
-Regarding claim 34, Tsuji discloses an image processing method comprising (Abstract; FIGS. 1-8): acquiring an estimation result of estimating a posture of a person included in a first image and a person included in a second image (FIG. 3; Page 3, 4th paragraph, “recognizes elements appearing in the image”, 5th paragraph, “recognize elements such as a person, an object, …”; Page 6, 1st paragraph, “… human recognition … determines the category of the image based on the element of the image … Subject (person) attributes: "Female", "Couple", "Baby", "Businessman" … may be classified into a plurality of categories …”; Page 7, 3rd paragraph, “comparison of subject posture and posing”; FIG. 5, step 521; Page 8, “performs various analysis processes such as human recognition, general object recognition”; Page 9, 2nd paragraph, “similar person's posture and posing”); acquiring a recognition result of recognizing an object, other than the persons (FIG. 3; Page 3, 4th paragraph, “recognizes elements appearing in the image”, 5th paragraph, “recognize elements such as a person, an object, …”; Page 6, 1st paragraph, “… object recognition … Subject (general object) ... "car", "mountain", "cherry blossom", "viewer", "temple"…”; FIG. 5, step 521; Page 8, “performs various analysis processes such as human recognition, general object recognition”), included in the first image and an object included in the second image (FIGS 1-2); and performing a similarity determination of the similarity of the first image to the second image based on the estimation results of the postures of the persons and the recognition results of the objects (Abstract; FIG. 1; FIG. 2, units 101-106; FIGS. 5-8; Page 3, 4th paragraph, “Using these recognition results, … selects an image matching the search condition from the image database 100 (first selection) and then selects an SNS from the selected image group (first image group)… are highly evaluated in the above and have a low similarity to other images in the same category are selected (second selection) …”; Page 4, 2nd and 3rd paragraphs; Page 7, 1st -4th paragraphs, “… similarity … comparison of feature amounts (color, brightness, etc.), scene comparison, comparison of subject posture and posing”).
Tsuji does not disclose performing the similarity determination based on a degree of similarity between relationship feature values related to relationships between the persons and the objects.
In the same field of endeavor, Li teaches a method to identify human-object interactions in an image content (Li: Abstract; FIGS. 1-7). Li further teaches performing the similarity determination based on a degree of similarity between relationship feature values related to relationships between the persons and the objects (Li: FIGS. 1-3, 6-7; [0003], “person is riding a bicycle … most common relationship between the two objects … identify interactions based on the person and object identified … compare the current image to other images of the same action … a person is next to a bicycle, riding a bicycle and washing a bicycle may both be possible interactions between the two objects … evaluate the image for similarities to other images involving riding and washing”; [0026], “… human-object interaction metadata …”; [0048]; [0050], “joint-location heat map … determine whether the joint location and the object contact point are in sufficiently similar locations”; [0051]; [0053], “… match score based on the degree of match between the search terms and the information contained in searchable fields and the human object interaction metadata …”; [0056]-[0057]).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of Tsuji with the teaching of Li by using the relationship between the subjects in order to achieve robust and accurate similarity determination.
Tsuji in view of Li does not teach performing the similarity determination based on weights of the degrees of similarity. However, this is a common practice in the field (See Zhang (CN 109858308 A)).
Watanabe is an analogous art pertinent to the problem to be solved in this application and teaches an image retrieving method using pose information of a person itself as an image retrieval query (Watanabe: FIGS. 1-20). Watanabe further teaches performing the similarity determination based on weights of the degrees of similarity (Watanabe: Page 11, 5th paragraph, “weighting may be performed according to the degree of similarity”; Page 12, 5th paragraph, “searches for a similar image … image feature amount distance and the posture feature amount distance are integrated … normalize or weight the distances”).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify the teaching of Tsuji in view of Li with the teaching of Watanabe by performing the similarity determination based on weights of the degrees of similarity in order to achieve higher reliability of similarity determination.
-Regarding claim 35, Tsuji discloses a non-transitory computer-readable medium storing an image processing program (FIGS. 1-2; Page 4, 4th paragraph, ”CPU (processor), memory …”; Page 5, 3rd paragraph) that causes a computer to execute the processes of (Abstract; FIGS. 1-8): acquiring an estimation result of estimating a posture of a person included in a first image and a person included in a second image (FIG. 3; Page 3, 4th paragraph, “recognizes elements appearing in the image”, 5th paragraph, “recognize elements such as a person, an object, …”; Page 6, 1st paragraph, “… human recognition … determines the category of the image based on the element of the image … Subject (person) attributes: "Female", "Couple", "Baby", "Businessman" … may be classified into a plurality of categories …”; Page 7, 3rd paragraph, “comparison of subject posture and posing”; FIG. 5, step 521; Page 8, “performs various analysis processes such as human recognition, general object recognition”; Page 9, 2nd paragraph, “similar person's posture and posing”); acquiring a recognition result of recognizing an object, other than the persons (FIG. 3; Page 3, 4th paragraph, “recognizes elements appearing in the image”, 5th paragraph, “recognize elements such as a person, an object, …”; Page 6, 1st paragraph, “… object recognition … Subject (general object) ... "car", "mountain", "cherry blossom", "viewer", "temple"…”; FIG. 5, step 521; Page 8, “performs various analysis processes such as human recognition, general object recognition”), included in the first image and an object included in the second image (FIGS 1-2); and performing a similarity determination of the similarity of the first image to the second image based on the estimation results of the postures of the persons and the recognition results of the objects (Abstract; FIG. 1; FIG. 2, units 101-106; FIGS. 5-8; Page 3, 4th paragraph, “Using these recognition results, … selects an image matching the search condition from the image database 100 (first selection) and then selects an SNS from the selected image group (first image group)… are highly evaluated in the above and have a low similarity to other images in the same category are selected (second selection) …”; Page 4, 2nd and 3rd paragraphs; Page 7, 1st -4th paragraphs, “… similarity … comparison of feature amounts (color, brightness, etc.), scene comparison, comparison of subject posture and posing”).
Tsuji does not disclose performing the similarity determination based on a degree of similarity between relationship feature values related to relationships between the persons and the objects.
In the same field of endeavor, Li teaches a method to identify human-object interactions in an image content (Li: Abstract; FIGS. 1-7). Li further teaches performing the similarity determination based on a degree of similarity between relationship feature values related to relationships between the persons and the objects (Li: FIGS. 1-3, 6-7; [0003], “person is riding a bicycle … most common relationship between the two objects … identify interactions based on the person and object identified … compare the current image to other images of the same action … a person is next to a bicycle, riding a bicycle and washing a bicycle may both be possible interactions between the two objects … evaluate the image for similarities to other images involving riding and washing”; [0026], “… human-object interaction metadata …”; [0048]; [0050], “joint-location heat map … determine whether the joint location and the object contact point are in sufficiently similar locations”; [0051]; [0053], “… match score based on the degree of match between the search terms and the information contained in searchable fields and the human object interaction metadata …”; [0056]-[0057]).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of Tsuji with the teaching of Li by using the relationship between the subjects in order to achieve robust and accurate similarity determination.
Tsuji in view of Li does not teach performing the similarity determination based on weights of the degrees of similarity. However, this is a common practice in the field (See Zhang (CN 109858308 A)).
Watanabe is an analogous art pertinent to the problem to be solved in this application and teaches an image retrieving method using pose information of a person itself as an image retrieval query (Watanabe: FIGS. 1-20). Watanabe further teaches performing the similarity determination based on weights of the degrees of similarity (Watanabe: Page 11, 5th paragraph, “weighting may be performed according to the degree of similarity”; Page 12, 5th paragraph, “searches for a similar image … image feature amount distance and the posture feature amount distance are integrated … normalize or weight the distances”).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify the teaching of Tsuji in view of Li with the teaching of Watanabe by performing the similarity determination based on weights of the degrees of similarity in order to achieve higher reliability of similarity determination.
Allowable Subject Matter
Claims 8-18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims, and overcome claim rejections in above section of “Claim Rejections - 35 USC § 112”.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1 and 34-35 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XIAO LIU whose telephone number is (571)272-4539. The examiner can normally be reached Monday-Thursday and Alternate Fridays 8:30-4:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/XIAO LIU/Primary Examiner, Art Unit 2664
Prosecution Insights