DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-3, 5-8 and 10-15 are pending.
Claims 4 and 9 are canceled.
Claim Rejections - 35 USC § 103
The following is a quotation of pre-AIA 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claim(s) 1-3, 5-8 and 10-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Jain et al (US20210357644A1) in view of Alismail (US20230068848A1).
Regarding claims 1 and 10, Jain teaches a computer-implemented method for a technical system, the method comprising the following steps:
applying a plurality of masks to an image, the plurality of masks masking different regions of the image than one another, wherein the application of the plurality of masks generates a plurality of respective masked images;
(Jain, Figs. 2-3; "applying the series of masks to a digital image to produce masked versions of the digital image", [0026]; "the mask creation module 160 may apply the series of masks to the digital image to create masked versions of the digital image", [0041]; "The editing platform can sample N number of binary masks by independently setting each element to a value of one with probability p and to a value of zero with the remaining probability", [0065]; "all masks may be shifted by a random number of pixels in both spatial directions", [0067]; Jain teaches a series of masks applied to one image producing multiple masked versions, each mask zeroing different elements/regions and spatially shifted so the masks cover different regions than one another)
for each of the plurality of masked images respectively:
(a) reconstructing a respective masked-out region of the respective masked image based on a context provided by an unmasked region of the respective masked image, thereby generating a respective reconstructed image; and
(Jain, Fig. 2; "Another approach focuses on performing perturbations (e.g., inpainting, blurring, occluding) on regions of the digital image that serves as the input for a neural network", [0050]; "Neural networks may learn to utilize the surrounding pixels to determine whether an object is present", [0023]; Jain masks regions and names inpainting (region reconstruction) as a known perturbation, using surrounding unmasked context; Alismail, Figs. 2A–2C; "an obstructed view may have one or more environmental factors partially or entirely obstructed from the field of view of the image capturing device", [0035]; Alismail establishes the recognized problem of obstructed/missing image regions in AV perception)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the teachings of Alismail into the system or method of Jain in order to reconstruct each masked-out region from the surrounding context. The combination represents a simple substitution of one known, interchangeable perturbation (inpainting) for plain zero-masking among Jain's finite identified set, and utilizes a known technique to solve Alismail's recognized obstructed-region problem, yielding a complete reconstructed image for downstream classification with a reasonable expectation of success. The combination of Jain and Alismail also teaches other enhanced capabilities.
(b) performing a classification on the respective reconstructed image that assigns to each pixel of the respective reconstructed image a respective one of a plurality of classifications, each of the classifications corresponding to a respective one of a plurality of object types, wherein, for each of the classified pixels, the classification of the respective pixel identifies the respective pixel as being part of a depiction of an object of the object type to which its respective classification corresponds;
(Jain, Fig. 5; "The editing platform can apply the model to the masked digital images to generate a series of outputs indicative of separate instances of objects", [0081]; "trained to identify a single class of objects, such as humans, animals, buildings, vehicles, and the like", [0019]; "establishing, on a per-pixel basis, importance of the digital image", [0087]; Alismail, "the system will use a classifier to assign a classification to each ROI", [0005]; "type (e.g., vehicle vs. pedestrian vs. bicycle vs. static object or obstacle)", [0083]; Jain applies a model to each image and classifies by object type (humans, vehicles, etc.), operating on a per-pixel basis for saliency; Alismail assigns an object-type classification to each image region (ROI). The recited "plurality of object types" reads on Jain's/Alismail's object-type classes generally)
generating a classification map comprising a plurality of pixel positions, wherein each of the reconstructed images includes a respective pixel for each of the pixel positions of the classification map, and wherein the classification map assign each of the pixel positions of the classification map a respective one of the plurality of classifications based on an aggregate of all of the classifications of the pixels of the reconstructed images that correspond to the respective pixel position of the classification map; and
(Jain, Figs. 2 and 6; "compute a weighted sum of the masks. Such an approach will cause each pixel in the digital image to be associated with a weighted sum of the corresponding elements in the series of masks. These values may be loaded into a matrix having the same dimensions as the digital image", [0046]; "the editing platform can generate a saliency map by establishing, on a per-pixel basis, importance of the digital image based on the values in the matrix", [0087]; Alismail, "The system will aggregate the classifications for each ROI to determine an aggregate classification for each ROI", [0005]; Jain generates a per-pixel map (same-dimension matrix) where each position is assigned a value derived from an aggregate (weighted sum) of all masked-version outputs at that position; Alismail confirms aggregating per-region classifications into an aggregate classification)
controlling the technical system to perform an operation that is selected based on a characterization of an environment surrounding the technical system as including one or more objects, the characterization being based on the classification map.
(Jain, "outputs indicative of separate instances of objects", [0081]; Alismail, Figs. 3 and 9; "it will generate a function request that, when executed, will cause a system of which the image capturing device is a component to perform a function", [0005]; "cause a motion planning system of the vehicle to move to a parking location; or (ii) alert a human operator to take over operation of the autonomous vehicle", [0006]; "the function may include … altering a trajectory of the vehicle 105, altering a speed of the vehicle 105", [0038]; "The objects may include traffic signals, roadway boundaries, other vehicles, pedestrians, and/or obstacles", [0082]; Jain supplies the object/environment characterization from the per-pixel map; Alismail supplies controlling the technical system to perform a selected operation based on that characterization. Incorporating Alismail's function-request control into Jain would operate the autonomous-vehicle technical system based on Jain's environment characterization; that is, applying a known control response to a known perception result for the predictable benefit of safe operation)
Regarding claim 2, the combination of Jain and Alismail teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, wherein each of the pixel positions of the classification map is assigned the respective one of the plurality of classifications based on which classification among the classifications of the pixels of the reconstructed images that correspond to the respective pixel position occurs more frequently than at least one other of the classification of the corresponding pixels.
(Jain, Fig. 6; "For each masked digital image, the editing platform can assign the highest similarity metric to the corresponding mask as a weight metric", [0085]; "compute a weighted sum of the masks. Such an approach will cause each pixel in the digital image to be associated with a weighted sum of the corresponding elements in the series of masks", [0046]; aggregating the per-masked-version classifications at each pixel position and assigns the dominant value. It would have been obvious to select, from among a finite set of predictable aggregation schemes, the classification that occurs more frequently than at least one other (plurality voting), because this is a known technique that yields a predictable result when combining multiple per-position predictions)
Regarding claim 3, the combination of Jain and Alismail teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, wherein each of the pixel positions of the classification map is assigned the respective one of the plurality of classifications based on which classification among the classifications of the pixels of the reconstructed images that correspond to the respective pixel position occurs most frequently among all of the classifications of the corresponding pixels.
(Jain, Fig. 6; "compute a weighted sum of the masks. Such an approach will cause each pixel in the digital image to be associated with a weighted sum of the corresponding elements in the series of masks", [0046]; "establishing, on a per-pixel basis, importance of the digital image based on the values in the matrix", [0087]; assigning each pixel position a value from the aggregate of all corresponding per-masked-version classifications; assigning the most-frequently-occurring classification (majority voting) is the canonical, predictable aggregation among a finite set of known options (averaging, weighted sum, majority vote))
Regarding claim 5, the combination of Jain and Alismail teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, further comprising determining an indication of whether the classifications of pixels of the plurality of reconstructed images that correspond to a same pixel position of the classification map assign the same object type or different object types to the respective pixel position.
(Jain, Fig. 4; "examine the cosine similarities of the class probabilities associated with those regions", [0070]; "The editing platform can infer the disturbance caused by each mask based on the similarity metrics", [0062]; measuring, per position, the similarity (and conversely the disturbance) between the classification information produced across the masked versions, which inherently indicates whether the masked versions assign the same or different classification/object type to that position)
Regarding claim 6, the combination of Jain and Alismail teaches its/their respective base claim(s).
The combination further teaches the method according to claim 5, further comprising generating a consistency map that, for each of the pixel positions of the classification map, includes the indication of whether the classifications of the pixels of the reconstructed images that correspond to the respective pixel position assign the same object type or different object types."
(Jain, Fig. 6; "compute a weighted sum of the masks. Such an approach will cause each pixel in the digital image to be associated with a weighted sum of the corresponding elements in the series of masks. These values may be loaded into a matrix having the same dimensions as the digital image", [0046]; "the editing platform can generate a saliency map by establishing, on a per-pixel basis, importance of the digital image based on the values in the matrix", [0087]; generating a per-pixel map (matrix of same dimensions) that includes, at each position, the similarity-based indication of whether the masked-version classifications agree (same) or differ (different object types))
Regarding claim 7, the combination of Jain and Alismail teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, further comprising outputting an alarm and/or controlling the technical system in response to determining that at least one classification of one or more of the reconstructed images differs from a classification of another of the reconstructed images for a corresponding pixel position."
(Jain, "The editing platform can infer the disturbance caused by each mask based on the similarity metrics", [0062]; Jain determines that classifications differ across masked versions; Alismail, Figs. 3 and 9; "generating a notification (e.g., a visual and/or audible notification) that an environmental obstruction has been detected", [0038]; "the system may generate an alternate function request, such as a command to cause the vehicle to move into a parking space, or a command to signal a human operator to take over or augment vehicle operation", [0050]; Jain teaches the difference determination; Alismail teaches outputting an alarm (notification) and/or controlling the technical system in response; Incorporating Alismail's notification/control response into Jain would enable a detected inconsistency to trigger an alarm or vehicle action, a typical example of applying a known response to a known degraded-perception condition)
Regarding claim 8, the combination of Jain and Alismail teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, further comprising:
determining each of the plurality of masks to indicate a group of masked elements representing a region of the image, wherein the region matches a predetermined patch in size and shape or that is larger than a patch that has predetermined dimensions in at least one of dimension."
(Jain, Fig. 3; "The editing platform can sample N number of binary masks by independently setting each element to a value of one with probability p and to a value of zero with the remaining probability", [0065]; "The dimensions of the masks, as defined by height (h) and width (w), are normally smaller than the dimensions of the digital image, as defined by height (H) and width (W)", [0065]; "the editing platform may crop areas H×W with uniformly random offsets ranging from (0,0) to (CH, CW)", [0067]; determining each mask to indicate a group of masked elements (cells) representing a region of predetermined dimensions/shape)
Regarding claim 11, the combination of Jain and Alismail teaches its/their respective base claim(s).
The combination further teaches the device according to claim 10, wherein the technical system includes an at least partially autonomous computer-controlled machine, the at least partially autonomous computer-controlled machine including a robot, or a vehicle, or a domestic appliance, or a power tool, or a manufacturing machine, or a personal assistant, or an access control system."
(Jain, Alismail; see comments on claim 1; Alismail; "a vehicle having a processor, programming instructions and drivetrain components that are controllable by the processor without requiring a human operator", [0028]; "the on-board computing device 912 may determine a motion plan for the autonomous vehicle", [0085])
Regarding claim 12, the combination of Jain and Alismail teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, further comprising:
determining the plurality of masks to indicate several groups of masked elements representing several different regions of the image, wherein the different regions individually match a predetermined patch in size and shape or are larger than a patch having predetermined dimensions in at least one dimension."
(Jain, Fig. 3; "The editing platform can sample N number of binary masks by independently setting each element to a value of one with probability p and to a value of zero with the remaining probability", [0065]; "all masks may be shifted by a random number of pixels in both spatial directions", [0067]; the masks set many elements to zero throughout the image, thereby indicating several groups of masked elements representing several different regions of predetermined dimensions, and shifting distributes those regions)
Regarding claim 13, the combination of Jain and Alismail teaches its/their respective base claim(s).
The combination further teaches the method of claim 1, further comprising:
generating a consistency map that, for each of the pixel positions of the classification map, provides a quantification of a level of consistency between the classification of the pixels of the reconstructed images that correspond to the respective pixel position;
determining a level of image consistency between the reconstructed images based on the consistency map; and
determining whether to enter a safe mode based on the determined level of image consistency."
(Jain, Figs. 4 and 6; "examine the cosine similarities of the class probabilities associated with those regions", [0070]; "establishing, on a per-pixel basis, importance of the digital image based on the values in the matrix", [0087]; "infer the disturbance caused by each mask based on the similarity metrics", [0062]; Jain teaches generating a per-pixel consistency map and determining a consistency/disturbance level; Alismail; "the system may generate an alternate function request, such as a command to cause the vehicle to move into a parking space, or a command to signal a human operator to take over or augment vehicle operation", [0050]; "Based on the severity level of the obstruction, the system is configured to perform the one or more functions", [0067]; Alismail teaches determining, from the severity/level, whether to enter a safe mode)
Regarding claim 14, the combination of Jain and Alismail teaches its/their respective base claim(s).
The combination further teaches the method of claim 1, further comprising:
generating a consistency map that, for each of the pixel positions of the classification map, provides a quantification of a level of consistency between the classifications of the pixels of the reconstructed images that correspond to the respective pixel position;
determining a level of image consistency between the reconstructed images based on the consistency map; and
determining whether to output an alarm based on the determined level of image consistency.
(Jain, Figs. 4, 6; "examine the cosine similarities of the class probabilities associated with those regions", [0070]; "establishing, on a per-pixel basis, importance of the digital image based on the values in the matrix", [0087], Jain teaches generating the per-pixel consistency map and a consistency level; Alismail, "generating a notification (e.g., a visual and/or audible notification) that an environmental obstruction has been detected", [0038]; "Based on the severity level of the obstruction, the system is configured to perform the one or more functions", [0067]; Alismail determining, from the level/severity, whether to output an alarm (notification))
Regarding claim 15, the combination of Jain and Alismail teaches a computer-implemented method for operating a technical system, the method comprising:"
applying a plurality of masks to an image, the plurality of masks masking different regions of the image than one another, wherein the application of the plurality of masks generates a plurality of respective masked images;
(Jain, see comments on claim 1)
for each of the plurality of masked images respectively:
(a) reconstructing a respective masked-out region of the respective masked image based on a context provided by an unmasked region of the respective masked image, thereby generating a respective reconstructed image; and"
(Jain, Alismail, see comments on claim 1)
(b) performing a classification on the respective reconstructed image that assigns to each pixel of the respective reconstructed image a respective one of a plurality of classifications, each of the classifications corresponding to a respective one of a plurality of object types …"
(Jain, Alismail, see comments on claim 1)
generating a consistency map that: (a) includes a plurality of pixel positions for each of which each of the reconstructed images includes a respective pixel; and (b) for each of the pixel positions of the consistency map, provides a quantification of a level of consistency between the classifications of the pixels of the reconstructed images that correspond to the respective pixel position;
(Jain, Figs. 4, 6; "examine the cosine similarities of the class probabilities associated with those regions", [0070]; "compute a weighted sum of the masks. Such an approach will cause each pixel in the digital image to be associated with a weighted sum of the corresponding elements in the series of masks", [0046]; "establishing, on a per-pixel basis, importance of the digital image based on the values in the matrix", [0087]; Jain computes a per-masked-version similarity (cosine similarity of classification information) and aggregates these into a per-pixel map, thereby quantifying, at each pixel position, the level of consistency between the classifications across the masked versions)
determining a level of image inconsistency between the reconstructed images based on the consistency map; and
(Jain; "determine not only how the disturbance in the output is measured but also where the disturbance is measured", [0061]; "The editing platform can infer the disturbance caused by each mask based on the similarity metrics", [0062]; determining a level of disturbance/dissimilarity (i.e., inconsistency) among the outputs of the masked versions from the similarity values underlying the map)
based on the determined level of image inconsistency, entering the technical system into a safe mode.
(Alismail, Figs. 3 and 9; "the system may generate an alternate function request, such as a command to cause the vehicle to move into a parking space, or a command to signal a human operator to take over or augment vehicle operation", [0050]; "cause a motion planning system of the vehicle to move to a parking location; or (ii) alert a human operator to take over operation of the autonomous vehicle", [0006]; "Based on the severity level of the obstruction, the system is configured to perform the one or more functions", [0067]; Alismail teaches, upon a degradation/severity determination derived from the aggregated per-region image classifications, entering the technical system into a safe mode (pulling over to a parking space / handing control to a human operator); Incorporating Alismail's safe-mode response into Jain so that, when Jain determines a high level of image inconsistency, the autonomous-vehicle technical system enters a safe mode, that is, applying a known fail-safe response to a known degraded-perception condition for predictable safe operation)
Response to Arguments
Applicant's arguments filed on 6/8/2026 with respect to one or more of the pending claims have been fully considered but are moot in view of the new ground(s) of rejection.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIANXUN YANG whose telephone number is (571)272-9874. The examiner can normally be reached on MON-FRI: 8AM-5PM Pacific Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on (571)272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272- 1000.
/JIANXUN YANG/
Primary Examiner, Art Unit 2662 6/28/2026
Prosecution Insights