DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claim 17 is rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. Claim 17 — Potential §112(a): Written description Claim 17 recites controlling the communication interface to transmit, to a server, "at least one new second recognition model" and subsequently receiving the new second recognition model from the server. However, the specification describes the electronic device transmitting a request for a second recognition model, or characteristic information used to determine or generate the model, and receiving the corresponding recognition model from the server. The presently identified disclosure does not establish possession of the claimed operation in which the electronic device itself transmits the new second recognition model to the server as recited. Accordingly, the limitation requires further consideration under 35 U.S.C. §112(a) for adequate written-description support.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-19 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 1, 10, and 19 recite "with each second recognition model allocated to each corresponding subset space." It is unclear whether the recited limitation requires a one-to-one correspondence between each second recognition model and a respective subset space or requires each second recognition model to be allocated to each of the plurality of subset spaces. The different interpretations establish materially different allocation relationships.
The ambiguity is further demonstrated by dependent claims 9 and 18, which recite allocating multiple second recognition models to a single subset space.
Accordingly, the scope of the required relationship between the plurality of second recognition models and the plurality of subset spaces is unclear and indefinite.
Claims 4 and 13 recite determining the shape, area, and boundary line such that "object classes belonging to each of the plurality of subset spaces are present in a proportion above a certain threshold." However, the claim does not clearly establish which object classes are subject to the threshold or the basis for calculating the recited proportion. Specifically, it is unclear whether the proportion refers to objects belonging to a particular class relative to the total objects within a subset space, the number of different classes represented within the subset space, or another relationship. Further, it is unclear whether each object class within a subset space must exceed the threshold or whether the presence of at least one object class exceeding the threshold is sufficient.
Accordingly, the scope of the claimed threshold-based determination is unclear and indefinite.
Claims 7 and 16 recite "checking for changes in the spatial structure information as the electronic device moves through the space." Claims 1 and 10 recite initially obtaining spatial structure information by scanning the space. However, claims 7 and 16 do not establish the relationship between that previously obtained information and the information used to determine whether a change has occurred. It is unclear whether the recited changes refer to modifications of previously stored spatial structure information or physical changes within the space detected through additional sensor observations. Further, the claim does not clearly identify the information against which the previously obtained spatial structure information is evaluated to determine whether a change has occurred.
Accordingly, the scope of the recited checking operation is unclear and indefinite.
Claims 9 and 18 recite allocating multiple second recognition models to a single subset space. However, claims 1 and 10 recite each second recognition model being allocated to each corresponding subset space. Under a one-to-one interpretation, the limitation of claims 9 and 18 appear inconsistent with the previously established allocation relationship. Under an interpretation requiring each recognition model to be allocated to each subset space, the additional allocation limitation of claims 9 and 18 appear to be already required by claims 1 and 10. Accordingly, the inconsistent allocation language creates uncertainty regarding the scope of the claim and the scope of the relationship inherited, and is therefore indefinite.
Claims 2-3, 5-6, 8, 11-12, and 14-15 are rejected as indefinite for the same reasons by virtue of their dependency and failure to cure the cited deficiencies of the claims 1 and 10 above.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 9-12, and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Tian et al (US 2021/0142110) in view of Paula et al (WO 2019/164484).
Regarding Claim 1, Tian teaches a method, performed by an electronic device ([0091], Fig. 6, Step 602, automated machine setting selection based on image detection and classification results produced through local feature extraction), of performing object recognition (Fig. 6, [0085]), the method comprising:
obtaining spatial structure information and images through scanning (Tian teaches obtaining sensor information, including image data and thermal-map data, from sensors associated with an appliance ([0032]). Tian further teaches capturing images of objects occupying a physical space within the machine using a camera (FIG. 6, step 604; [0085]–[0086]));
generating the spatial map based on structural information (Tian teaches representing the imaged physical space using a spatially arranged plurality of image grids, each representing a corresponding region of the space (FIG. 5; [0088]). Tian further teaches obtaining sensor information that includes image and thermal-map data ([0032]));
first recognition model and first object information (Tian teaches recognizing objects and obtaining object classification information using a deep-learning feature extraction model and trained classifiers (FIG. 6, steps 610–614; [0089]–[0092]))
dividing the space based on the map and first object information (Tian teaches dividing the imaged space into a plurality of sub-regions corresponding to image grids, wherein the division may be performed based on the presence of local object features (FIG. 6, step 608; [0088])).
Tian fails to teach the following, which in the same field of endeavor, Paula further teaches applying a first scene-recognition model to an image to identify object types and generate a segmented image corresponding to the recognized objects (FIG. 4; [0035]).
using the output of a first recognition model to divide an image into a plurality of segments associated with different recognized object types (FIG. 4; [0035]).
determining and allocating second recognition models based on subset-space characteristics (Paula teaches selecting a plurality of specialized recognition models based on the object types or environmental characteristics identified by a first recognition model. Different sub-models are selected for different image segments according to their corresponding characteristics (FIGS. 2–4; [0026]–[0027], [0035])).
performing subsequent object recognition using allocated models during movement (Paula teaches applying the selected second recognition models to their corresponding image segments to obtain additional object information (FIG. 4; [0035]). WO further teaches capturing environmental images at different times or locations, including when an electronic assistant is moved to another room ([0016], [0025])).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the hierarchical recognition and model-selection techniques of Paula into the regional image-processing system of Tian, such that initial recognition information is used to identify regions of interest and select specialized recognition models for subsequent processing of those regions. One of ordinary skill would have been motivated to make the modification to improve object-recognition efficiency by applying specialized recognition models to relevant regions while avoiding unnecessary processing using models that are unlikely to provide useful information, as expressly recognized by Paula ([0034]–[0035]).
Regarding Claim 10, Tian, modified by Paula, teaches an electronic device comprising: one or more sensors; memory storing one or more instructions; and a processor configured to execute the one or more instructions stored in the memory to: perform substantially the same method as recited in claim 1 above ([0091], FIG. 6 In some embodiments, the machine is a first machine (e.g., a laundry machine (e.g., washing machine or dryer), a smart oven for cooking various food items or food ingredients in a dish, a smart refrigerator for storing various food items, etc.) having one or more processors, a camera, and memory (602)).
Regarding Claim 19, Tian, modified by Paula, teaches a non-transitory computer-readable recording medium configured to store instructions which, when executed by a processor of a device for performing object recognition, cause the device to: perform substantially the same method as recited in claim 1 above ([0103], [0091], FIG. 6 In some embodiments, the machine is a first machine (e.g., a laundry machine (e.g., washing machine or dryer), a smart oven for cooking various food items or food ingredients in a dish, a smart refrigerator for storing various food items, etc.) having one or more processors, a camera, and memory (602)).
Regarding Claims 2 and 11, Tian, modified by Paula, teach all aspects of the invention according to Claims 1 and 10 above, wherein the first object information includes location information and large category classification information, and the second object information includes location information and small category classification information (Paula further teaches hierarchical recognition in which an initial model identifies a broader category and subsequent models identify more specific categories, including location, room, and object types (FIG. 3; [0033]–[0034])).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the hierarchical recognition and model-selection techniques of Paula into the regional image-processing system of Tian, such that initial recognition information is used to identify regions of interest and select specialized recognition models for subsequent processing of those regions. One of ordinary skill would have been motivated to make the modification to improve object-recognition efficiency by applying specialized recognition models to relevant regions while avoiding unnecessary processing using models that are unlikely to provide useful information, as expressly recognized by Paula ([0034]–[0035]).
Regarding Claims 3 and 12, Tian, modified by Paula, teach all aspects of the invention according to Claims 1 and 10 above, wherein the dividing of the space into the plurality of subset spaces comprises determining a shape and an area of each of the plurality of subset spaces, and a boundary line between the plurality of subset spaces (Tian teaches dividing an image into a plurality of image grids having defined geometric regions (FIGS. 5–6; [0088]). The division establishes the shape, area, and boundaries of each resulting image sub-region).
Regarding Claims 9 and 18, Tian, modified by Paula, teach all aspects of the invention according to Claims 1 and 10 above, wherein determining of the plurality of second recognition models for the plurality of subset spaces comprises allocating multiple second recognition models to a single subset space (Paula teaches using multiple recognition models within a hierarchical recognition framework, including applying multiple models when the confidence of a parent model is insufficient (FIG. 1; [0017]). The hierarchical arrangement further permits subsequent specialized models to process information associated with an initially recognized region ([0019], [0035])).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the hierarchical recognition and model-selection techniques of Paula into the regional image-processing system of Tian, such that initial recognition information is used to identify regions of interest and select specialized recognition models for subsequent processing of those regions. One of ordinary skill would have been motivated to make the modification to improve object-recognition efficiency by applying specialized recognition models to relevant regions while avoiding unnecessary processing using models that are unlikely to provide useful information, as expressly recognized by Paula ([0034]–[0035]).
Claims 4 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Tian et al (US 2021/0142110) in view of Paula et al (WO 2019/164484), and further in view of Shi (US 2018/0144477).
Regarding Claims 4 and 13, Tian, modified by Paula, teach all aspects of the invention according to Claims 1 and 10 above. Tian teaches dividing an image into a plurality of spatial sub-regions, including image grids selected according to local object features (FIGS. 5–6; [0088]–[0089]). Tian further teaches determining object classifications using trained classifiers ([0090]–[0092]).
Shi teaches evaluating local image regions using predetermined proportional thresholds, including determining whether the area occupied by an object within a candidate region exceeds a threshold proportion of the candidate region (FIG. 7, steps S203–S204). The reference further teaches merging adjacent regions according to their object classifications and segmentation masks (FIG. 9, steps S2061–S2068).
It would have been obvious to one of ordinary skill in the art to incorporate the proportional region-evaluation and class-based segmentation techniques of US 2018/0144477 A1 into the spatial subdivision process of Tian in view of WO 2019/164484, thereby determining region boundaries according to the relative presence of recognized object classes to improve object segmentation accuracy.
Claims 5 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Tian et al (US 2021/0142110) in view of Paula et al (WO 2019/164484), and further in view of Shi (US 2018/0144477), and further in view of Artes et al (US 2019/0025838).
Regarding Claims 5 and 14, Tian, modified by Paula, teach all aspects of the invention according to Claims 1 and 10 above. Tian teaches dividing an image representing a physical space into a plurality of unit regions corresponding to image grids (FIGS. 5–6; [0088]). Tian further teaches extracting regional object features and identifying object types using trained classifiers (FIGS. 5–6; [0089]–[0092]).
Shi further teaches grouping adjacent image regions based on object classification and segmentation information, including combining regions associated with the same object class (FIG. 9, steps S2061–S2068).
Artes teaches detecting and identifying objects or obstacles within spatial regions, determining characteristics including object size and location, and identifying spatial clusters based on the number and distribution of detected obstacles (FIGS. 4–5). Artes further teaches recognizing clusters when the number of obstacles exceeds a predetermined minimum value (discussion of FIG. 5).
It would have been obvious to one of ordinary skill in the art to incorporate the object-count-based clustering of Artes and class-based region-grouping techniques of Shi into Tian's grid-based object recognition process, thereby grouping spatial regions according to the number and classifications of recognized objects to generate subset spaces having related object characteristics and improve subsequent recognition processing.
Claims 6-7 and 15-16 are rejected under 35 U.S.C. 103 as being unpatentable over Tian et al (US 2021/0142110) in view of Paula et al (WO 2019/164484), and further in view of Artes et al (US 2019/0025838).
Regarding Claims 6 and 15, Tian, modified by Paula, teach all aspects of the invention according to Claims 1 and 10 above. Tian teaches dividing an image into sub-images based on local object features, including arranging the sub-images such that each contains at least one local feature (FIG. 5; [0088]).
Artes teaches defining spatial zones having sizes, shapes, and boundaries determined according to detected obstacle locations and spatial distributions, including defining rectangular zones around clusters of objects and adjusting zone boundaries based on the locations of detected obstacles (FIGS. 5, 8A–8E, and 9A–9F).
It would have been obvious to one of ordinary skill in the art to apply the location-dependent spatial-boundary techniques of Artes to Tian's image-grid subdivision process to adjust the size or shape of the regions according to the locations of recognized object features, thereby improving correspondence between the resulting regions and the spatial arrangement of detected objects.
Regarding Claims 7 and 16, Tian, modified by Paula, teach all aspects of the invention according to Claims 1 and 10 above. Artes teaches a mobile robot obtaining sensor measurements of a physical environment, generating a spatial map, and identifying objects and obstacles using the obtained measurements (FIGS. 1–5).
Artes further teaches using observations obtained during robot operations, including information concerning the positions of obstacles at different points in time, to establish and adjust spatial zones. Artes also teaches updating the sectoring of a map in response to additional information concerning the environment (discussion of FIGS. 4–5 and subsequent exploratory operations).
It would have been obvious to one of ordinary skill in the art to incorporate the environmental monitoring and map-adjustment techniques of Artes into the object recognition system of Tian in view of Paula, thereby maintaining an accurate spatial representation when environmental conditions change and improving the accuracy of subsequent region-based object recognition.
Claims 8 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Tian et al (US 2021/0142110) in view of Paula et al (WO 2019/164484), and further in view of Konolige et al (US 2018/0349725).
Regarding Claims 8 and 17, Tian, modified by Paula, teach all aspects of the invention according to Claims 1 and 10 above.
Konolige teaches a robot transmitting a request to a remote model-generation system for a recognition model and receiving the generated model for subsequent object recognition (FIGS. 1 and 6). The reference further teaches generating a training-image dataset using rendered images of objects, including backgrounds and additional environmental objects selected based on characteristics of the robot's operating environment (FIG. 3B; FIG. 5, blocks 556–558). The generated training examples are used to train a machine-learning recognition model, which is subsequently provided to the robot (FIG. 5, blocks 560–562).
It would have been obvious to one of ordinary skill in the art to incorporate the server-based model-generation technique of Konolige into the system of Tian in view of WO 2019/164484, such that characteristics associated with the identified subset spaces are used to generate environment-specific training images and train additional recognition models. The modification would improve recognition performance by providing models trained using examples representative of the environments in which the models are employed.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MARGARET G WEBB whose telephone number is (571)270-7803. The examiner can normally be reached M-F 9:00-6:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Charles Appiah can be reached at (571) 272-7904. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MARGARET G WEBB/ Primary Examiner, Art Unit 2641
