DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-11 are pending.
Election by Original Presentation
Newly submitted dependent claims 8-9 and 11 are withdrawn from further consideration pursuant to 37 CFR 1.142(b), as being drawn to new species which are otherwise restrictable by original presentation, directed to methods of
determining a shape of road based on a curvature of the road (claim 8) and on a brightness of pixels in the captured image and depth information Claim 9); and
identifying the area-of-interest based on a position displacement between a plurality of captured images (claim 11).
These new species are mutually exclusive with all species originally elected in the Non-Final rejection mailed on 8/19/2025.
Claim Rejections - 35 USC § 103
The following is a quotation of pre-AIA 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claim(s) 1-7 and 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Motoyama et al (US20210297633) in view of Uchida et al (US20160217583).
Regarding claims 1 and 6-7, Motoyama teaches an image processing device comprising: at least one memory configured to store instructions; and at least one processor configured to execute the instructions to:
recognize, among multiple different area classes designated regarding objects appearing in a captured image, which of the multiple different area classes each pixel in the captured image belongs to;
(Motoyama, Fig. 3; “acquires an image captured by the camera mounted on the moving body 1”, [0067]; Fig. 6, “the recognition unit 182 executes the image recognition of the image captured by the imaging unit 13 (Step S103). For example, the recognition unit 182 recognizes the attribute of the object (image area) shown in the image by executing the semantic segmentation on the image captured by the imaging unit 13. ... the output label of the learning model include road surfaces, sidewalks, vegetation, stationary structures (buildings, walls, fences, guardrails, and the like), vehicles (automobiles, trucks, buses, and the like), a two-wheel vehicles (motorcycles, bicycles, and the like), person, traffic lights, signs, white lines, the sky, and the like”, [0101]; “As illustrated in FIG. 17, the generation unit 183 generates a semantic map by injecting the label data included in the image recognition result into the grid map on the basis of the depth value ((X,Y,Z) illustrated in FIG. 17) included in the depth data”, [0107]; Fig. 3, with the depth data, part of the pixels in the image may be semantically classified into different area classes belonging to different objects such as person, telephone pole, wall, etc., which have 3D structures and may be considered as typical area classes related to obstacles in a road for the automobile)
identify, as an area-of-interest candidate, an unrecognized-class area not belonging to any of the multiple area classes, in an area in the captured image representing a prescribed area class among the multiple different area classes;
(Motoyama, Figs. 13 and 16; “the dirt area in the image is an area of unknown attribute”, [0105]; “The dirt on the road surface and the small falling objects are not a problem for traveling of the moving body but are one of the things that a user wants to avoid when traveling”, [0056]; “in a case where the user discovers dirt or small falling objects, it is desirable to enable the user to immediately change the route of the autonomous moving body”, [0057]; “In FIG. 1, dirt (for example, a puddle) on the road surface is shown in the image”, [0059]; identifying "dirt" or a "puddle," which is described as an "area of unknown attribute" (unrecognized-class area). This dirt is located "on the road surface" (in an area representing a prescribed area class, i.e., the road)
determine an area-of-interest from among area-of-interest candidates; and
(Motoyama, Figs. 13-15, areas of interests on the road may be obstacles (e.g., person, stationary structures, ...) and unknown objects such as a dirt (Figs. 27-29), which are “area-of-interest candidates”; “In a case where the dirt information is received (Step S401: Yes), the recognition unit 182 determines whether there is an obstacle on the route of the moving body 1 (Step S402). In a case where the machine can correctly recognize an obstacle on the route, the machine does not need to ask the user for decision”, [0149]; dirt is identified from “area-of-interest candidates” (dirt, various type of obstacles))
determine whether the area-of-interest candidate represents a stationary object,
(Motoyama, Fig. 5, Block 18f; “The block 18f is a block which executes moving object tracking... The block 18f generates a moving object list on the basis of the attributes of the object recognized by the block 18a, the depth data acquired by the block 18b, and the self-position estimated by the block 18c”, [0092]; “The tracking unit 184 executes tracking of the moving object on the basis of the image recognition result of the recognition unit 182 and the measurement result of the measuring unit 14. The tracking unit 184 acquires the tracking result of the moving object as moving object tracking information”, [0112]; Figs. 7-10; “Examples of the output label of the learning model include road surfaces, sidewalks, vegetation, stationary structures (buildings, walls, fences, guardrails, and the like), vehicles (automobiles, trucks, buses, and the like), a two-wheel vehicles (motorcycles, bicycles, and the like), person, traffic lights, signs, white lines, the sky, and the like”, [0101]; the system can recognize stationary structures (e.g., buildings, walls, fences, guardrails, ...) and moving objects (e.g., automobiles, person,...))
Motoyama does not expressly disclose but Uchida teaches:
... based on a difference between
an absolute value of a change in distance of pixels (∆d1) in the area-of-interest candidate during a predetermined period, which is calculated based on a change in depth information indicated by the pixels, and
an absolute value of a change in distance of pixels (∆d2) in an area determined to be a moving body during the predetermined period.
(Uchida, Fig. 2; “calculate an amount of change over time in the depth distance of the at least one potential object; and detect an object around the vehicle from among the at least one potential object using the amount of change over time”, [0008]; “The amount of change over time in the depth distance of the potential objects corresponds to the relative speeds of the potential objects with respect to the object detection apparatus 1 or the host vehicle”, [0040]; “That is, a camera, such as a stereo camera, for example, that is able to obtain ... depth distance information for each pixel area may be used”, [0023]; the depth changes of the potential objects may be measured using the depth changes of the pixels corresponding to the potential objects in the captured images; obviously, (1) if the distance change of the pixels in the potential object, relative to the host vehicle, is a same value (∆d1 => first “an absolute value) as the travel distance of the host vehicle (calculated by host vehicle speed x travel time), the potential object is a stationary object as referenced to the road; on the other hand, (2) if the distance change of the pixels in the potential object, relative to the host vehicle, is a different value (∆d2 => second “an absolute value) than the travel distance of the host vehicle (calculated by host vehicle speed x travel time), the potential object is a moving object as referenced to the road; that is, there is a difference between ∆d1 and ∆d2; in one extreme example, when the host vehicle speed is 0, ∆d1=0 for potential object 1 in the captured image (such as the unknown object “dirt” in Fig. 13 of Motoyama) indicates that the potential object 1 is a stationary object in the image; similarly, when the host vehicle speed is 0, ∆d2≠0 for potential object 2 in the captured image (such as the person object in Fig. 13 of Motoyama) indicates that the potential object 2 is a moving object in the image; without knowing the host vehicle speed, using ∆d1 and ∆d2 alone cannot determine if an object is stationary or not)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the teachings of Uchida into the system or method of Motoyama in order to determine if an object is stationary based on the depth changes of its pixels and the host vehicle's speed for accurately and robustly modeling the surrounding environment for autonomous driving. The combination of Motoyama and Uchida also teaches other enhanced capabilities.
Regarding claim 2, the combination of Motoyama and Uchida teaches its/their respective base claim(s).
The combination further teaches the image processing device according to claim 1, wherein the at least one processor is configured to execute the instructions to:
determine whether the area-of-interest candidate represents a three-dimensional object based on the depth information, and
determine that the area-of-interest candidate is the area-of-interest when the area-of-interest candidate represents the three-dimensional object.
(Motoyama, “FIG. 2 is an Occupancy Grid Map using a depth sensor mounted on the moving body 1. The Occupancy Grid Map is a grid map for the presence or absence of obstacles. FIG. 2 is a grid map illustrating the vicinity of the moving body 1 when vertically viewed down from the sky. A place where there is an obstacle is black, a place where there is no obstacle is white, and a place where the presence or absence of obstacles is not determined is gray”, [0060], Fig. 3, with the depth data ([0107]), part of the pixels in the image may be semantically classified into different area classes belonging to different objects such as person, telephone pole, wall, etc., which have 3D structures and may be considered as typical area classes relating to obstacles in a road for the automobile, => ”area-of-interest”)
Regarding claim 3, the combination of Motoyama and Uchida teaches its/their respective base claim(s).
The combination further teaches the image processing device according to claim 1, wherein the at least one processor is configured to execute the instructions to:
determine whether the area-of-interest candidate represents the stationary object based on the change in depth information, and
determine that the area-of-interest candidate is the area-of-interest when the area-of-interest candidate represents the stationary object.
(Motoyama, Fig. 13, “stationary structure” such as a telephone pole (Fig. 3))
Regarding claim 4, the combination of Motoyama and Uchida teaches its/their respective base claim(s).
The combination further teaches its/their respective base claim(s).
Motoyama further teaches the image processing device according to claim 1, wherein the at least one processor is configured to execute the instructions to:
recognize, among the multiple different area classes designated regarding objects appearing in the captured image, which of the multiple different area classes each pixel in the captured image, including a road and moving bodies traveling on the road, belongs to; and
(Motoyama, see comments on claim 2; Fig. 13, “stationary structure” and “person” which are detectable by the depth sensors (i.e., having 3D structures”) and can be recognized as road obstacles)
identify, as the area-of-interest candidate, an unrecognized-class area at least partially adjacent to an area in the captured image belonging to a road class representing a road area among the multiple different area classes.
(Motoyama, Fig. 13, an area of unknown attribute (e.g., a dirt) is adjacent to the detectable 3D objects such as “stationary structure” and “person”)
Regarding claim 5, the combination of Motoyama and Uchida teaches its/their respective base claim(s).
The combination further teaches the image processing device according to claim 1, wherein the at least one processor is configured to execute the instructions to:
determine the area-of-interest, from among the area-of-interest candidates in areas other than an area in the captured image belonging to the moving body among the multiple different area classes.
(Motoyama, see comments on claim 2; Fig. 13, other than the area indicating a “person” as a road obstacle, the area of “stationary structure” such as telephone poles (Fig. 3) may be determined also as another road obstacle)
Regarding claim 10, the combination of Motoyama and Uchida teaches its/their respective base claim(s).
The combination further teaches the image processing device according to claim 1, wherein the at least one processor is configured to execute the instructions to:
determine that the area-of-interest candidate represents the stationary object, when the difference between the absolute value of the change in distance of the pixels in the area-of-interest candidate and the absolute value of the change in distance of the pixels in the area determined to be the moving body, is equal to or greater than a change threshold value.
(Motoyama, Uchida, see comments on claim 1)
Response to Arguments
Applicant's arguments filed on 11/19/2025 with respect to one or more of the pending claims have been fully considered but are moot in view of the new ground(s) of rejection.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIANXUN YANG whose telephone number is (571)272-9874. The examiner can normally be reached on MON-FRI: 8AM-5PM Pacific Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on (571)272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center. for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272- 1000.
/JIANXUN YANG/
Primary Examiner, Art Unit 2662 1/3/2026